emaynardigs commented on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields URL: https://github.com/apache/spark/pull/27155#issuecomment-586352041 >so I'm thinking to submit a PR based on our internal version (a modified version of #22535) which is proven to be stable and in production for awhile. My main reservation is that #22535 relies on a dot in the name of the field, and so cannot support ORC. The key difference in this PR is that it actually inspects the type of the field and extends the same functionality to Parquet and ORC. It's also rebased for the current master already and so merges cleanly. I also added more tests. No, I intended to cherry pick this back after it merges, but if it doesn't get merge we'll probably end up using it on our fork much like you've done.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
