adriangb commented on issue #21308: URL: https://github.com/apache/datafusion/issues/21308#issuecomment-5196307187
I've opened #24119 as an EPIC for the Parquet-side struct access work — projection pruning, filter pushdown, row-group pruning and the statistics that feed them — since a batch of concrete follow-ups just came out of #24090 (nested schema pruning for narrowed declared schemas) and they needed a home. That leaves some overlap with this issue, so a suggestion: would it make sense to reframe this EPIC to focus specifically on **Variant**? The two linked issues here point in slightly different directions: - #21306 (struct-aware optimizations are hardcoded to `GetFieldFunc`, so `variant_get` gets none of them) is squarely the Variant blocker, and it's the one that needs a general capability API rather than more struct-specific machinery. It's relevant to both EPICs; I've listed it under #24119 as shared rather than claiming it. - #21307 (`ProjectionExec` produces unknown statistics for all `ScalarFunctionExpr` outputs) is really a physical-expr / statistics issue that happens to bite struct and Variant queries hardest — arguably it belongs to neither EPIC exclusively. Splitting along "Variant as a first-class type" vs "make Parquet read less for struct columns" seems like the cleaner line to me, but you filed this one and have the fuller picture of where Variant is heading — happy to fold #24119 back in here instead if you'd rather keep it as one umbrella. cc @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
