Hey Eliran
I humbly think To store elements of different types (e.g., Double, Int,
String) in a single array, you can use the StructArray or UnionArray in
Apache Arrow. These types allow for the combination of different types in a
structured format
StructArray: This allows you to define a structure where each field can
have a different type.
UnionArray: This is more flexible than StructArray and allows for storing
elements of different types in a more heterogeneous manner.
These approaches are also useful when Modeling data as rows is still needed
while using Apache arrow columnar
Here is a simple example usage of StructArray and Row Modeling together:
import pyarrow as pa # Define a schema for the struct where each field can
be of a different type schema = pa.schema([ ('field_double', pa.float64()),
('field_int', pa.int32()), ('field_string', pa.string()), ]) # Create a
StructArray data = [ {'field_double': 1.1, 'field_int': 10, 'field_string':
"example1"}, {'field_double': 2.2, 'field_int': 20, 'field_string':
"example2"}, {'field_double': 3.3, 'field_int': 30, 'field_string':
"example3"}, ] # Convert data to StructArray struct_array = pa.array(data,
type=pa.struct([ ('field_double', pa.float64()), ('field_int', pa.int32()),
('field_string', pa.string()), ]))
Best regards
*MG <https://linkedin.com/in/emg110>*
-------------------------
Technology Architect
-------------------------
GoPlausible Co-Founder <https://goplausible.com/>
On Fri, Mar 1, 2024 at 5:29 AM Eliran Noach (BLOOMBERG/ 120 PARK) <
eno...@bloomberg.net> wrote:
> Hello,
>
> our team is utilizing apache arrow in one of our projects.
> We were wondering if it is possible to create an array and store elements
> of different types.
> We basically looking for a way to store different types say Double, Int,
> String in a single array and be able to parse it when received by a
> different application.
>
> we looked at all the array types and they all seem to be strongly typed
> DoubleArray, Int32Array etc ... Same is true for scalar types
>
> ** we are also wondering if it is possible to model the array as rows and
> not as columns ? I know this sounds like it goes against the apache arrow
> design of columnar format, we are would still like to know if it is
> possible and if so how it can be achieved.
>
> Thanks in Advanced,
> Eliran
>
>
>
