Moelf commented on issue #503: URL: https://github.com/apache/arrow-julia/issues/503#issuecomment-2004544872
It's similar to why naive `for row in eachrow(df::DataFrame)` is slow, but if you use `DataFramesMeta.jl` or convert to a `TypedTable` from `TypedTables.jl` the row iteration would be faster again. In C or Rust, you need to know the column type at compile time, you can just do: ``` auto df = openarrow(path) auto col = df.mycol1 auto val = col[1] ``` but you can do the moral equivalent of this in Julia. Fundamentally, the type information (schema) of the Arrow table lives as "values" (bytes) on-disk, you have to somehow process that info and at run-time generate a type tree -- using Julia's typing system or using Dict is implementation detail. For Rust and C, it's still doing this type of thing, but in more staged-fashion and probably more efficient, wheres in Arrow.jl I think it really leans heavily on Julia's typing system and compose everything using Julia type composition -- which may not have been smart from the result. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
