xitology commented on issue #503:
URL: https://github.com/apache/arrow-julia/issues/503#issuecomment-2005637391
@Moelf, you made a point that IO with Arrow requires materializing exact
field types, otherwise it would be slow. This is not so. Rust is capable of
reading and writing Arrow data of arbitrary form, and can do it fast despite
being a statically compiled language. People write SQL engines over Arrow in
Rust, and they certainly don't know the type of each table and each column in
advance. They can do it because in a column-oriented layout any data is
ultimately stored as a collection of arrays of primitive types. By reducing any
operation on composite data to operations on the underlying primitive arrays,
they can process data effectively even if they don't know the exact layout of
the composite fields during compilation.
DataFrames API does it exactly right by making the step of materializing the
row type explicit. The user can request it and pay the compilation cost if they
want to process a DataFrame as a vector of tuples, but that's their choice. In
fact, Arrow.jl works exactly the same, but only with top-level Table objects,
and not with fields of composite types. Which is a problem, as my bug report
demonstrates.
What I would propose... support two variants of each composite array type:
transparent and opaque. The transparent is the current one while the opaque
would hide the type of its content. The current definitions almost work, if we
represent, say, an opaque Struct array as `Arrow.Struct{Any, Any, Any}`. We
will only need to add a field to store column names (and allow `data` to be a
vector). The API could provide a way to switch between transparent and opaque
forms, but I'd argue the opaque form should be the default: this way you could
read and process any Arrow data compilation-free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
