nzw921rx opened a new pull request, #11671:
URL: https://github.com/apache/seatunnel/pull/11671
### Purpose of this pull request
Relates to #11598.
This PR delivers the Phase 1 MVP of STIP-32: a reproducible benchmark
framework for
SeaTunnelRow hot paths and complete single-node Zeta pipelines. The first
version focuses on a
fixed, observable baseline and machine-readable artifacts. It intentionally
does not turn a single
GitHub-hosted runner result into a blocking performance gate.
The framework answers three concrete questions:
1. Did a change alter a focused SeaTunnelRow hot path?
2. Can the current Zeta code sustain the fixed workload without losing rows
or accumulating
latency?
3. What is the incremental cost of Transform work, backpressure
observability, and StainTrace under
the same resources and workload?
#### Project structure
```text
seatunnel/
├── pom.xml # opt-in benchmark Maven profile
├── seatunnel-benchmarks/
│ ├── pom.xml # JMH runner and engine
dependencies
│ ├── README.md
│ └── src/
│ ├── main/java/org/apache/seatunnel/benchmark/
│ │ ├── BenchmarkBase.java
│ │ ├── SeaTunnelRowBenchmark.java
│ │ ├── SeaTunnelPipelineBenchmark.java
│ │ ├── SeaTunnelEnvironmentContext.java
│ │ ├── SeaTunnel*EnvironmentContext.java
│ │ ├── connector/source/ # deterministic in-memory
Source
│ │ ├── connector/sink/ # blackhole Sink and latency
result
│ │ └── transform/ # deterministic CPU work and
checksum
│ └── test/java/org/apache/seatunnel/benchmark/
├── tools/benchmarks/
│ ├── run_benchmarks.sh
│ ├── save_jmh_result.py
│ └── regression_report.py
├── .github/workflows/benchmarks.yml
└── docs/{en,zh}/engines/zeta/benchmark.md
```
`seatunnel-benchmarks` is included only when `-Pbenchmark` is enabled. JMH
and the engine runtime
dependencies therefore do not enter the normal SeaTunnel build or runtime
classpath.
#### Measurement architecture
```mermaid
%%{init: {"theme": "base", "themeVariables": {"background": "#0f1d33",
"primaryColor": "#0c2530", "primaryBorderColor": "#2dd4bf", "primaryTextColor":
"#f8fbff", "actorBkg": "#0c2530", "actorBorder": "#2dd4bf", "actorTextColor":
"#f8fbff", "activationBkgColor": "#1f1a34", "activationBorderColor": "#8d7cf6",
"noteBkgColor": "#1f1a34", "noteBorderColor": "#8d7cf6", "noteTextColor":
"#f8fbff", "signalColor": "#5db8e2", "signalTextColor": "#f8fbff",
"labelBoxBkgColor": "#0f1d33", "labelBoxBorderColor": "#5db8e2",
"labelTextColor": "#f8fbff", "loopTextColor": "#f8fbff"}}}%%
flowchart LR
JMH["JMH fork"] --> Context["SeaTunnelEnvironmentContext"]
Context --> Client["SeaTunnelClient"]
Client --> Zeta["single-node Zeta Master + Worker"]
Zeta --> Source["BenchmarkSource"]
Source --> Transform["BenchmarkTransform (optional)"]
Transform --> Sink["BenchmarkSink"]
Sink --> Pipeline["per-invocation Pipeline JSON"]
JMH --> Raw["raw JMH JSON"]
Pipeline --> Report["normalized JSON + Markdown summary"]
Raw --> Report
```
`SeaTunnelEnvironmentContext` starts a real embedded Zeta server and client
in
`@Setup(Level.Trial)` and closes them in `@TearDown(Level.Trial)`. Cluster
startup is outside JMH
timing. Job configuration, submission, scheduling, Source, optional
Transform, Sink, and completion
are inside each benchmark invocation.
No SeaTunnel distribution, shell launcher, external broker, or installed
connector is required.
The benchmark runs the engine code from the same commit as the benchmark.
#### Workload and scenarios
The default pipeline workload is fixed in code:
| Setting | Default |
|---|---:|
| JVM heap | `-Xms4g -Xmx4g` |
| JVM-visible processors | 4 |
| Garbage collector | G1 with pre-touch |
| Zeta slots / pipeline parallelism | 12 / 4 |
| Rows per invocation | 1,000,000 |
| Offered rate | 600,000 rows/s |
| Payload | 256 characters |
| Transform work | 64 hash operations/row |
| JMH forks | 3 |
| Warmup / measurement | 3 / 5 iterations |
The five pipeline methods keep the workload constant while selecting the
path under test:
| JMH method | Purpose |
|---|---|
| `sourceSink` | Baseline `Source -> Sink` data path. |
| `sourceTransformSink` | Adds row copy and deterministic Transform work. |
| `sourceTransformSinkWithBackpressure` | Enables backpressure
observability. |
| `sourceTransformSinkWithTrace` | Enables StainTrace. |
| `sourceTransformSinkWithBackpressureAndTrace` | Enables both observability
features. |
The backpressure scenario does not artificially slow the Sink. Overload is
produced by setting the
offered rate above current capacity and is detected from achieved throughput
and latency growth.
#### Correctness and metric design
Performance is interpreted only after correctness validation:
- `processed_rows == expected_rows`;
- direct `Source -> Sink` has a zero checksum;
- every Transform pipeline has a deterministic non-zero checksum;
- the Zeta job finishes successfully and produces its result file.
JMH and the Sink intentionally expose different measurement boundaries:
| Metric | Boundary | Meaning |
|---|---|---|
| JMH `Score` | Job submission through complete job termination | Complete
bounded-job rows/s, including scheduling and lifecycle cost. |
| `throughput_rows_per_second` | First-to-last Sink receipt | Data-path
completion rate while records are arriving. |
| `event_time_latency_p50/p95/p99/max_ms` | Planned Source time to Sink
receipt | End-to-end delay including open-loop backlog. |
| `first_half_p99_ms` / `second_half_p99_ms` | First and second half of the
sequence | Whether latency keeps growing during the invocation. |
| `latency_growth_ratio` | `(second-half P99 + 1) / (first-half P99 + 1)` |
Values above 1 indicate worsening latency. |
The Source uses an absolute open-loop schedule. Its planned time continues
to advance when Zeta is
backpressured, so waiting time remains visible instead of being omitted by a
coordinated-omission
style closed loop.
The current convenience guardrail marks a sample sustainable when output is
complete, P99 is at
most 1,000 ms, and the latency growth ratio is at most 1.20. This is not a
universal SLA and does not
create a CI failure in Phase 1.
#### Detailed execution flow
1. JMH creates an independent fork with the fixed JVM settings.
2. The environment context creates a temporary SeaTunnel home and starts one
Zeta Master + Worker.
3. The benchmark method generates a unique run ID and normal SeaTunnel job
configuration.
4. `SeaTunnelClient` submits a bounded batch job through the normal parser
and client APIs.
5. Source readers emit deterministic interleaved sequence splits according
to the absolute schedule.
6. The optional Transform copies the row, performs deterministic work, and
writes the checksum.
7. Sink writers discard payloads after recording row count, checksum,
throughput, and latency.
8. Sink writers merge their state and atomically write one Pipeline JSON
result.
9. The context waits for `FINISHED`, reads the result, and rejects
incomplete rows or an invalid
checksum.
10. JMH records the completed invocation; report tools preserve raw data and
build a concise summary.
#### Local execution
Build the current repository code and benchmark runner:
```bash
./mvnw -Pbenchmark -pl seatunnel-benchmarks -am -DskipTests package
```
Run the complete fixed pipeline suite:
```bash
java -jar seatunnel-benchmarks/target/benchmarks.jar
SeaTunnelPipelineBenchmark
```
Run one exact pipeline and save standard JMH JSON:
```bash
java -jar seatunnel-benchmarks/target/benchmarks.jar \
'sourceTransformSink$' \
-p offeredRatePerSecond=600000 \
-p parallelism=4 \
-p payloadSize=256 \
-p transformOperations=64 \
-rf json \
-rff seatunnel-benchmarks/target/zeta-pipeline-result.json
```
Run the SeaTunnelRow microbenchmarks:
```bash
java -jar seatunnel-benchmarks/target/benchmarks.jar SeaTunnelRowBenchmark \
-rf json \
-rff seatunnel-benchmarks/target/seatunnel-row-result.json
```
#### GitHub Actions execution
The `Benchmarks` workflow supports Java 8 and Java 11 and has three modes:
1. **Scheduled:** runs at `00:17 UTC` every day against `dev`. The non-zero
minute avoids the common
top-of-hour scheduling peak.
2. **Manual:** selects a branch/tag/SHA and a common JMH selector. An
optional custom selector can
override the dropdown.
3. **PR comparison:** supplies a PR number and runs `base -> PR -> PR ->
base` on the same worker.
Each side uses the median of its two samples. The report is observational
and has no failure
threshold.
Each Java job uploads raw JMH JSON, per-invocation Pipeline JSON, normalized
JSON, Markdown summary,
and CPU/JDK/kernel/memory fingerprints for 90 days.
GitHub-hosted runner hardware varies. Daily absolute values are suitable for
smoke checks, history,
and large changes, but a single run is not a precise regression gate. The
same-worker PR comparison
reduces host differences; a future hard gate still requires a controlled
runner and an empirically
derived variance model.
#### Future extensions
Phase 1 establishes the fixed workload and result schema. Follow-up work can
add:
- a separate offered-rate capacity sweep and an unpaced saturation
diagnostic;
- measurement-phase tagging so Pipeline history excludes warmup invocations;
- checkpoint, state, recovery, and controlled slow-Sink contexts;
- allocation, GC, CPU, and async-profiler diagnostics as separate runs;
- multi-node and production-connector suites with an external driver;
- long-term storage, rolling median/MAD anomaly detection, and a controlled
performance gate.
### Does this PR introduce any user-facing change?
Yes. It adds an opt-in benchmark module, a `Benchmarks` GitHub Actions
workflow, downloadable result
artifacts, and English/Chinese Zeta benchmark documentation. Normal
SeaTunnel builds and runtime
behavior are unchanged unless the `benchmark` Maven profile is explicitly
enabled.
### How was this patch tested?
```bash
./mvnw -Pbenchmark -pl seatunnel-benchmarks -am -DskipTests package
./mvnw spotless:apply
```
A local Java 8 visualization run also executed every
`SeaTunnelPipelineBenchmark` method with the
fixed `600000:4:256:64` workload using one fork, one warmup iteration, and
two measurement
iterations. It generated standard JMH JSON and 71 Pipeline result files.
Every invocation processed
all 1,000,000 expected rows and every Transform checksum was non-zero. This
shortened run verifies
execution and report compatibility; the default 3/3/5 JMH configuration
remains the performance
baseline configuration.
### Check list
- [x] No new runtime Jar binary package is added to the SeaTunnel
distribution.
- [x] English and Chinese documentation are included.
- [x] No incompatible user-facing configuration or API change is introduced.
- [x] This is not a connector contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
