berkaysynnada commented on issue #15511: URL: https://github.com/apache/datafusion/issues/15511#issuecomment-2774860687
> Instead of running the benchmark, how about adding those benchmark query to tests, I don't think we need to actually "benchmark" the code for each merge. Keeping all benchmark coverage with their replicas is challenging (as @Omega359 said not all benchmarks are sql queries) and very prone to go out of sync. > Adding it to the extended test suite and running it on every merge isn’t a viable solution, as it’s costly and often unnecessary You are also right here about the cost, but what if we can have 2 modes for benchmarks, one for the actual benchmarking purpose, and one with just to validate. If it is in validation mode, it will work with very low loads -- e.g. min sampling count, min batch sizes etc. cargo test (or extended tests) would run the validation mode, and normal benchmarking goes with the standard mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
