Rangsh commented on PR #12172: URL: https://github.com/apache/seatunnel/pull/12172#issuecomment-5570910544
@nzw921rx Follow-up ABBA results on Java 11, as discussed. I could not re-trigger the Apache `Benchmarks` workflow from this account (`403 Must have admin rights`), so I ran an equivalent same-worker ABBA locally with the same outer sequence and JMH settings used by `tools/benchmarks/run_benchmarks.sh` for PR comparisons: - Sequence: `baseline -> PR -> PR -> baseline` - JDK: Temurin / Corretto **11.0.26** - Selector: `DebeziumJsonFormatBenchmark` - Settings: `-f 1 -wi 3 -i 5` (10 s warmup / measurement iterations) - Baseline commit: `72acda570` (`dev`) - Candidate commit: `f28464c21` (this PR) - Host: Apple M1 (local macOS; absolute Score is not comparable to GitHub-hosted Ubuntu, but Error/CV comparisons are on the same worker) ### Median of the two runs per revision | Benchmark | Baseline Score | Baseline Error% | Baseline CV | Candidate Score | Candidate Error% | Candidate CV | Score change | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | `deserializeInsertEvent` | 599.946 | 9.35% | 2.43% | 852.924 | 2.38% | 0.62% | **+42.17%** | | `deserializeUpdateEvent` | 435.901 | 3.36% | 0.87% | 577.224 | 2.04% | 0.53% | **+32.42%** | | `serializeInsertEvent` | 908.539 | 3.27% | 0.85% | 975.347 | 8.29% | 2.15% | **+7.35%** | | `serializeMergedUpdateEvent` | 544.952 | 2.77% | 0.72% | 665.885 | 6.04% | 1.57% | **+22.19%** | ### Per-run Score / Error / CV | Benchmark | Run | Score | Error% | CV | | --- | --- | ---: | ---: | ---: | | `deserializeInsertEvent` | baseline-1 | 589.952 | 15.45% | 4.01% | | `deserializeInsertEvent` | candidate-1 | 809.595 | 3.75% | 0.97% | | `deserializeInsertEvent` | candidate-2 | 896.253 | 1.02% | 0.27% | | `deserializeInsertEvent` | baseline-2 | 609.940 | 3.24% | 0.84% | | `deserializeUpdateEvent` | baseline-1 | 437.768 | 2.45% | 0.64% | | `deserializeUpdateEvent` | candidate-1 | 574.538 | 1.40% | 0.36% | | `deserializeUpdateEvent` | candidate-2 | 579.910 | 2.69% | 0.70% | | `deserializeUpdateEvent` | baseline-2 | 434.034 | 4.28% | 1.11% | | `serializeInsertEvent` | baseline-1 | 913.463 | 3.74% | 0.97% | | `serializeInsertEvent` | candidate-1 | 979.313 | 2.61% | 0.68% | | `serializeInsertEvent` | candidate-2 | 971.380 | 13.98% | 3.63% | | `serializeInsertEvent` | baseline-2 | 903.614 | 2.80% | 0.73% | | `serializeMergedUpdateEvent` | baseline-1 | 540.966 | 4.00% | 1.04% | | `serializeMergedUpdateEvent` | candidate-1 | 658.190 | 3.20% | 0.83% | | `serializeMergedUpdateEvent` | candidate-2 | 673.579 | 8.87% | 2.30% | | `serializeMergedUpdateEvent` | baseline-2 | 548.938 | 1.54% | 0.40% | ### Reading of the CV / Error trend On this same-worker ABBA: 1. Throughput improved for all four methods. 2. Deserialize CV / Error% were **lower** on the candidate than on baseline (not higher). 3. Serialize CV / Error% were somewhat higher on the candidate medians, but the absolute CV remains low (`2.15%` / `1.57%`). Looking at the per-run table, that increase is concentrated in **candidate-2** (one noisier serialize pair), while **candidate-1** was as stable as or more stable than baseline. Baseline itself also showed a noisy `baseline-1` for `deserializeInsertEvent` (`Error% 15.45%`, `CV 4.01%`). So on this local ABBA, the earlier Java 11 CV/Error increase does **not** reproduce as a consistent candidate-wide stability regression. It looks closer to normal run-to-run measurement variation. If it would help, I can: - ask a committer to re-run the official GitHub-hosted `Benchmarks` workflow with `pr_number=12172` and `benchmarks=DebeziumJsonFormatBenchmark` for another ABBA on Ubuntu, and/or - add a focused Java 11 GC/CPU diagnostic comparison for the serialize methods. Happy to follow whichever next step you prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
