IvanVergiliev commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-475720528 @cloud-fan interesting! A couple of questions to make sure we're on the same page: 1. If I understand your suggestion correctly, on a high-level it would roughly result in trimming the Spark `Filter` to only the parts of the tree that are convertible to `SearchArgument`s, and then converting these directly without the need for checking. Is that an accurate representation of your idea, or do you have something else in mind? If it's on the same page, I can think about it some more and take a stab at implementing. 2. What's your optimization function between the different ideas here? Are you thinking about ways to decrease code length, code duplication, complexity / maintainability? Want to make sure we're on the same page about this as well so we can evaluate the ideas similarly.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
