alamb commented on issue #26517: URL: https://github.com/apache/datafusion/issues/26517#issuecomment-6418340414
We had a good discussion about this today -- in general I think the technical items listed above all make sense and would make good API additions to DataFusion. In terms of gathering sufficient maintainer interest / attention to add the new APIs, I suggested that we focus on the usecases. For example, for > 1. Read only the leaves a UDF needs I think it would really help to spell out the high level usecase, something like -------- We reading variant data stored in parquet. When we run a query like this to extract field `f` from a variant column, even if `f` is shredded, DataFusion currently reads the entire variant column ```sql SELECT variant_col:f from my_table; ``` Here is the full reproducer ```rust // TODO: rust code that shows how to reproduce this ``` DataFusion already does this for struct fields (TODO links to examples / showing how this works), I want the same thing for variant -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
