gatorsmile edited a comment on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-498352686 @IvanVergiliev I agree we should automate running these micro benchmark tests to ensure we do not have the regressions. However, Apache Spark does not do it now. In my company, we have a different framework that run the microbenchmark suites for Spark to ensure we do not have perf regressions. For more details, you can see the talk: https://databricks.com/session/fast-and-reliable-apache-spark-sql-engine The micro benchmark suites this PR changed are used to compare the perf difference among the similar but related cases. For example, turning on/off a specific conf. You can create a new perf suite for showing the cost per filter and put them in a single test case? We can check `Relative` to know whether it has a regression?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
