DanielLeens commented on issue #11598: URL: https://github.com/apache/seatunnel/issues/11598#issuecomment-5137560662
Thanks for writing this as a long-term umbrella/STIP proposal. I think this belongs on the design/STIP track rather than being treated as only a small feature request. The important part is that the proposal separates the first runnable baseline from the later regression-gate ambition. That boundary should stay firm: #11580 can be the first concrete implementation slice, but the project should not turn benchmark results into a PR-blocking signal until the workload variance, hardware isolation, rerun policy, and false-positive risk are understood from real data. For Phase 1, I would keep the acceptance criteria very explicit: 1. run against a real Zeta runtime, not only an embedded/mock path; 2. use deterministic benchmark Source/Sink implementations so external I/O does not dominate the first result; 3. record the tested SeaTunnel commit, JVM/runtime parameters, workload size, parallelism, warm-up/measurement rules, and environment metadata in every JSON result; 4. keep the initial workload small enough that reviewers can understand and reproduce it; 5. document that Phase 1 is observational only and not a merge gate. For the later phases, one design point is especially important: benchmarks that need engine-internal classes should not force those internals into public APIs just for measurement. Keeping internal microbenchmarks close to their owning engine modules, with only thin runnable wrappers where needed, will make the framework easier to maintain. So my recommendation is to keep this issue open as the long-term benchmark framework tracker, link #11580 as the first delivery slice, and require each benchmark PR to state which phase and which workload boundary it implements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
