IvanVergiliev commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-499470485 @cloud-fan I applied the changes to the `hive/` subtree as well. It seems like it wasn't updated to reflect some of the recent changes to the other files (for example, it doesn't seem to have the quoted attribute name support) - I only applied the minimum amount of changed required to get the same algorithmic behaviour. I haven't applied the quoted attribute name support and other differences. @gatorsmile I see what you're saying. I got rid of the benchmark that only does the filter conversion step since that indeed seemed a bit out of place with the rest. The remaining new benchmark is `Pushdown benchmark with unbalanced Column` which tests the full filter-with-pushdown sequence that the rest of the benchmarks cover. This remaining benchmark is almost the same as `Pushdown benchmark with many filters` which was added in https://github.com/apache/spark/pull/22313 with the introduction of the `buildTree` function. I think it makes sense to have these two benchmarks next to each other. Let me know if you'd like me to move both of them to a new benchmark file. (I have a small concern that if we move them to a separate file, it'll be easier to forget to run them when making changes - because, as you say, the process is not fully automated at the moment. Thus, I slightly prefer keeping them where they are, but I'm open to moving them too.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
