nzw921rx opened a new issue, #11598: URL: https://github.com/apache/seatunnel/issues/11598
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### Describe the proposal ## Background The [Apache SeaTunnel 3.0 Roadmap](https://github.com/apache/seatunnel/issues/11021) introduces a number of substantial and performance-sensitive changes to the Zeta engine. Representative changes affecting the core execution path include: - [#10800](https://github.com/apache/seatunnel/pull/10800): introduces timer scheduling, `FlushSignal` propagation, and signal handling across the Source, intermediate queue, Transform, and Sink execution path. - [#10572](https://github.com/apache/seatunnel/pull/10572): adds real-time task busyness, queue occupancy, backpressure, and processing-time observability across Source, Transform, Sink, and intermediate queues. - [#10491](https://github.com/apache/seatunnel/pull/10491): instruments the end-to-end record path with tracing stages from source emission to sink completion. - [#10306](https://github.com/apache/seatunnel/pull/10306): introduces row-level error handling and Dead Letter Queue support into the Transform and Sink execution paths. - [#10600](https://github.com/apache/seatunnel/pull/10600): adds table-level fault isolation involving multi-table writers, physical plans, and `JobMaster`. - [#10812](https://github.com/apache/seatunnel/pull/10812): introduces a StateStore abstraction across runtime state, checkpoint state, metrics snapshots, and engine services. - [#10860](https://github.com/apache/seatunnel/pull/10860): adds observability metrics for Zeta engine state stores. - [#10489](https://github.com/apache/seatunnel/pull/10489): adds lifecycle hooks and execution context support to the Zeta SQL UDF path. These changes are valuable for SeaTunnel 3.0, but they also make one existing gap increasingly visible: we currently have no standardized and reproducible way to evaluate their impact on Zeta engine performance. ## Problems ### 1. We cannot quantify the performance impact of major engine changes SeaTunnel 3.0 contains changes across task execution, intermediate queues, Transform and Sink lifecycle, tracing, state management, and multi-table execution. Correctness tests can verify whether these features work, but they cannot tell us: - whether throughput improved or regressed; - whether end-to-end latency changed; - whether CPU, memory, or GC costs increased; - whether a new control path adds overhead when enabled or disabled; - whether an optimization simply moves the bottleneck to another stage. Without a stable baseline, we cannot evaluate the cumulative performance impact of the 3.0 engine changes. ### 2. Zeta cannot provide evidence-based performance expectations Without a standard benchmark environment and workload definition, Zeta cannot provide credible and repeatable performance results across commits or releases. Individual test results produced with different hardware, JVM parameters, data volumes, payloads, parallelism, connectors, or measurement methods are not directly comparable. This makes it difficult for the project to clearly explain: - what level of throughput Zeta can provide under a defined workload; - how performance changes between releases; - which scenarios Zeta performs well in; - what performance cost is introduced by enabling a specific engine feature. ### 3. Code review alone cannot validate performance-sensitive optimizations Some engine changes involve lock scope, checkpoint locking, concurrent queues, Disruptor usage, scheduling, serialization, backpressure, or thread coordination. During review, we can reason about correctness and identify possible concurrency risks, but we usually cannot determine whether a change actually improves performance. For example, reducing one lock scope may improve concurrency, but it may also increase context switching, allocation, queue contention, or downstream pressure. Without before-and-after benchmark evidence, reviewers cannot reliably determine whether the change improves or reduces overall engine performance. ### 4. We cannot track whether Zeta is becoming faster or slower over time Zeta is continuously evolving, but the project currently has no long-term performance history. As a result, we cannot answer a basic question: > After months or years of engine development, is Zeta actually becoming faster, more resource-efficient, and more stable, or are performance regressions accumulating unnoticed? A single benchmark result is not enough. We need stable workloads, machine-readable results, and historical comparisons across commits and releases. ## Proposal Establish a long-term, standardized benchmark and performance regression framework for the Zeta engine. The framework should gradually provide: 1. reproducible benchmark workloads for the core Zeta execution path; 2. fixed and documented environment assumptions; 3. comparable throughput, latency, CPU, memory, GC, queue, and backpressure metrics; 4. machine-readable benchmark results; 5. before-and-after comparison for engine pull requests; 6. historical performance trends across commits and releases; 7. specialized benchmarks for performance-sensitive engine components where necessary. The framework should be developed incrementally. The initial implementation should remain intentionally small and focus on establishing one reliable Zeta baseline before expanding into more scenarios. ## Phase 1 The first phase should provide: - a real Zeta runtime rather than an embedded or mocked execution path; - deterministic in-memory benchmark Source and Sink implementations; - at least one core `Source -> Transform -> Sink` workload; - fixed data volume, payload size, parallelism, JVM parameters, and runtime configuration; - explicit warm-up and measurement stages; - repeated executions with a documented aggregation method; - machine-readable JSON results; - execution against the `dev` branch first; - no automatic PR blocking until variance and false-positive risks are understood. The implementation tracked by [#11580](https://github.com/apache/seatunnel/issues/11580) can serve as the first concrete delivery under this long-term initiative. ## Long-Term Goal The long-term goal is not only to run performance tests, but to establish a trusted evaluation framework for Zeta engine development. For every performance-sensitive engine change, contributors and reviewers should eventually be able to answer: - What scenario is affected? - What was the baseline? - What changed after the patch? - Is the result outside the normal benchmark variance? - Did throughput, latency, or resource usage improve? - Did the optimization introduce a regression in another part of the execution path? Once the framework has accumulated enough stable data, it may gradually evolve into a reliable performance regression signal for pull requests and releases. ### Task list ### Phase 1: Initial benchmark framework and baseline - [ ] #11580 - [ ] Define the initial architecture and repository boundaries for the Zeta benchmark framework - [ ] Bootstrap the independent `seatunnel-benchmarks` project - [ ] Support selecting the tested SeaTunnel version, distribution, image, and commit - [ ] Implement a deterministic `BenchmarkSource` based only on the public SeaTunnel API - [ ] Implement a lightweight `PassThroughTransform` - [ ] Implement a `BlackholeSink` that does not introduce external I/O overhead - [ ] Implement the initial Zeta benchmark runner using a real Zeta runtime - [ ] Add the first `BenchmarkSource -> PassThroughTransform -> BlackholeSink` baseline workload - [ ] Define the benchmark scenario configuration model - [ ] Define warm-up, measurement, repetition, timeout, and result aggregation rules - [ ] Define a machine-readable JSON result format - [ ] Record the tested commit, environment, JVM parameters, workload parameters, and runtime configuration in every result - [ ] Document how to build and run the first benchmark locally - [ ] Run the initial benchmark against the `dev` branch without using it as a PR gate ### Phase 2: Core Zeta workload matrix - [ ] Add core record-processing throughput benchmarks - [ ] `Source -> Sink` - [ ] `Source -> PassThroughTransform -> Sink` - [ ] Single Transform and chained Transform workloads - [ ] Narrow-row, wide-row, string-heavy, and binary-heavy workloads - [ ] Add intermediate queue and backpressure benchmarks - [ ] Fast Source with a controllable slow Sink - [ ] Queue occupancy and backpressure propagation - [ ] Blocking queue and Disruptor execution paths - [ ] Different parallelism and queue-capacity configurations - [ ] Add serialization and data-transfer benchmarks - [ ] `SeaTunnelRow` serialization and deserialization - [ ] Cross-task record transfer - [ ] Different payload sizes and field counts - [ ] Add job lifecycle benchmarks - [ ] Job parsing and DAG generation - [ ] Physical plan generation - [ ] Task deployment - [ ] Job startup and shutdown - [ ] Add state and checkpoint benchmarks - [ ] Checkpoint duration and alignment cost - [ ] StateStore read and write performance - [ ] Checkpoint restore performance - [ ] State-store growth and cleanup behavior ### Phase 3: Engine-internal microbenchmarks - [ ] Define the Zeta internal microbenchmark development model - [ ] Keep benchmark implementations that depend on internal classes in the corresponding `seatunnel-engine-*` module under `src/test` - [ ] Add ordinary unit tests to ensure each internal benchmark remains runnable - [ ] Keep only thin JMH executors in the `seatunnel-benchmarks` project - [ ] Avoid exposing internal Engine classes as public APIs only for benchmarking - [ ] Add initial Engine microbenchmarks - [ ] Lock and synchronization paths - [ ] Intermediate queue operations - [ ] Disruptor event publication and consumption - [ ] Record serialization - [ ] DAG and physical-plan generation - [ ] Task scheduling and deployment - [ ] StateStore operations - [ ] Checkpoint coordination - [ ] Produce an executable benchmark JAR - [ ] Support selecting individual benchmarks - [ ] Support benchmark parameters - [ ] Support JMH profilers and flame-graph generation - [ ] Support JSON and CSV output ### Phase 4: Automation and historical performance tracking - [ ] Add a dedicated benchmark execution environment - [ ] Use fixed hardware, JDK, JVM parameters, container configuration, and operating-system settings - [ ] Prevent unrelated workloads from running on the benchmark machine - [ ] Separate build time, startup time, warm-up time, and steady-state measurement time - [ ] Add scheduled benchmark execution for the `dev` branch - [ ] Build the tested SeaTunnel commit - [ ] Run the standard benchmark suite - [ ] Upload raw results and environment metadata - [ ] Retain logs and profiling artifacts - [ ] Add historical benchmark result storage and visualization - [ ] Track results by commit and release - [ ] Display throughput, latency, CPU, memory, GC, and backpressure trends - [ ] Mark environment or workload-definition changes - [ ] Prevent incomparable results from being placed in the same trend line ### Phase 5: Pull request comparison and regression detection - [ ] Add base-versus-head benchmark comparison - [ ] Run the same workloads against the base and head commits - [ ] Repeat measurements and report median and variance - [ ] Report throughput, latency, and resource-usage differences - [ ] Produce a machine-readable and human-readable comparison report - [ ] Add non-blocking benchmark reports for selected Engine pull requests - [ ] Trigger only for performance-sensitive Zeta changes - [ ] Publish results without blocking merges during the initial observation period - [ ] Mark unstable results as `INCONCLUSIVE` - [ ] Collect enough historical data to understand normal variance - [ ] Define the performance regression policy - [ ] Define benchmark stability requirements - [ ] Define acceptable variance and regression thresholds - [ ] Define rerun and investigation rules - [ ] Define benchmark ownership and failure-triage responsibilities - [ ] Evaluate introducing a Zeta performance regression gate - [ ] Enable it only for stable workloads - [ ] Start with a small set of trusted Engine benchmarks - [ ] Require evidence and manual review before treating a result as a regression ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
