etseidl commented on issue #10941: URL: https://github.com/apache/arrow-rs/issues/10941#issuecomment-5628931045
> > readers should ignore both the logical type annotation and **column order** for that column. > > I understood this as ignoring the `FileMetaData::column_orders` property for such columns. As far as I can see, that is actually not used inside arrow-rs itself, but might be used by datafusion or other consumers. We would need to indicate to such consumers that the column order and statistics of a column cannot be trusted, or postprocess the whole `FileMetaData` struct to update those column orders. Thanks @jhorstmann, I'd actually forgotten about the `column_orders` vec in the file metadata. This needs checking, but I think that what happens will depend on what's already in the thrift. If the column order for an unknown logical type is set to `TYPE_ORDER`, then when decoding the metadata that will get converted to the rust `ColumnOrder::TypeDefined(SortOrder::UNDEFINED)` variant. But if any of the newer column orders are used, then those will remain unchanged. I think we need to think through what we want to do here. I'm currently inclined to always set the column order to `ColumnOrder::UNKNOWN`, since that variant seems to be the one that indicates stats should be ignored (although I think UNDEFINED should also result in the same behavior). I don't think datafusion currently looks at column_orders. Back when I was trying to fix NaN handling, @adriangb helped me with the plumbing to get the column order pushed down to the predicate handling code. At the time I don't think I found a single read of the column_order anywhere in Datafusion. In the linked PR, I think the clearest signal to a user that something is amiss is setting the logical type on the column descriptor to _Unknown. TBH I was happier when we could simply error on invalid combinations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
