etseidl commented on PR #11267: URL: https://github.com/apache/arrow-rs/pull/11267#issuecomment-6085972777
> My :2c: is that the missing piece is some configuration in this repo declaring that, for example, the `arrow_writer` suite needs at least 4 shards to finish within the 2h deadline. As of right now you must specify it on each and every invocation. > > My reasoning is that benchmarks are only useful if we actually run them systematically. Narrowing the suite through filters implicitly assumes we already know where performance will change, which risks hiding regressions in places we did not anticipate. I don't disagree. I recently tracked down a phantom regression in a benchmark that was actually a consequence of fixing a hidden gotcha with dictionary encoding FLBA with the V1 writer. But filtering is needed during development, esp as I see more workflows that are LLM driven modify-bench-repeat loops. Those loops can consume a lot of resources if not properly targeted. The broad suite should at least be limited to a pre-merge check. Or we could have even a weekly job that runs the full suite of benchmarks to find any missed regressions, then bisect to find the culprit. Maybe I'll turn Claude loose on the benches and generate a list of all the available benches and what coverage they provide. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
