alamb commented on issue #15780:
URL: https://github.com/apache/datafusion/issues/15780#issuecomment-2976060769
> variability comes from reading the data, parsing parquet, etc. and once
it's in memory as arrow converting from Int32 to Int64 is tri
This makes a lot of sense to me
> It only deals with the View types, which I think are a special case of the
general idea.
I agree -- this is a nice observation
> What I am thinking is:
> For any cases where we can read from parquet directly into the table type
do that (e.g. StringView and Utf8 are stored the sa me in Parquet, both UInt8
and UInt16 are stored as INT32 in Parquet). This is basically what is happening
with the view types already.
> If a cast is required prefer to cast the filter / scalar values or the
smaller column (we can use similar logic as what we use to reorder filters).
This also makes sense to me
I still think the major / critical usecase is going to be "extract a field
from this struct / variant" but anything else we do will be bonus
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
