IvanVergiliev commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-490417987 @cloud-fan after some noticeable delay, I managed to find the time to implement the tree trimming idea yesterday. I think the result looks pretty nice - thanks for the suggestion! I ran the relevant parts of the benchmark and the performance appears to be comparable - maybe ~10% slower than the previous version. I imagine this is due to external factors, but since it's such a small difference compared to going from exponential to linear, I don't think it's worth looking into it. I haven't updated the benchmark results in the commit with the most recent code since doing so took a couple hours last time I tried, and the differences are not huge. Let me know if you'd like me to update this. Looking forward to hearing your thoughts!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
