IvanVergiliev commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-475765931 To clarify with regards to 2. - this is exactly the behavior that this PR introduces. There’s effectively a single pass throughout the tree to check for convertibility, and then a second pass to build the ORC filter. While the build phase calls `isConvertible` for each node, this doesn't cause additional passes across the tree since the results have already been computed in the initial checking pass. I’ll think about the details of implementing the trimming and take a stab at an implementation if it seems like it can be made cleaner.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
