ai-yang opened a new pull request, #10938:
URL: https://github.com/apache/rocketmq/pull/10938
### Which Issue(s) This PR Fixes
- Fixes #10936
- Related to #2931, #2932, #10442, #10444, and #10588
### Brief Description
`MessageDecoder.encodeMessages` currently encodes every message into its own
complete `byte[]`, retains those arrays, then allocates a final result and
copies every message again.
This change removes those temporary full-message buffers:
- the first pass caches each body reference and the serialized property
bytes while computing the exact aggregate size;
- one final `ByteBuffer` is allocated;
- the second pass writes every message directly into that buffer;
- single-message and batch encoding share internal size and field-writing
helpers;
- a byte-for-byte compatibility test covers an empty body, Unicode
properties, a 4 KiB body, edge flag values and an empty list;
- an independent fixed golden vector verifies the wire field layout without
relying on the shared helper as its oracle.
The public API and encoded wire format are unchanged.
### How Did You Test This Change?
#### Correctness
The new tests compare `encodeMessages(messages)` with the concatenation of
`encodeMessage(message)` for every input and verify an independent fixed wire
vector. The complete `common` module suite passes:
```text
Tests run: 245, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
Command:
```bash
mvn -o -Dmaven.repo.local=/developer/wangrui/.m2/repository \
-Dspotbugs-plugin.version=4.2.2 -Dspotbugs.skip=true \
-Dcheckstyle.skip=true -Drat.skip=true \
-pl common test
```
Relevant downstream tests also pass:
```text
AppendCallbackTest + BatchPutMessageTest + AppendPropCRCTest:
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
ProduceAccumulatorTest:
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
```
#### JMH benchmark
JMH 1.36 configuration:
- throughput mode with GC profiler;
- one thread pinned to CPU 5;
- 3 forks;
- 5 warm-up and 7 measurement iterations per fork, 500 ms each;
- fixed `-Xms512m -Xmx512m` heap;
- OpenJDK 11.0.31, Ubuntu 22.04.4, Intel Xeon Gold 6133.
The benchmark's `baseline()` method reconstructs the previous aggregation
algorithm in the same binary (per-message full encodings followed by a final
copy), so the table labels it as a reconstructed baseline.
| Scenario | Reconstructed baseline ops/s | This PR ops/s | Change |
Baseline B/op | This PR B/op | Change |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 10 x 128 B | 340,743 +/- 6,348 | 381,842 +/- 9,236 | +12.06% | 6,760 |
4,552 | -32.66% |
| 100 x 128 B | 32,247 +/- 1,099 | 36,246 +/- 242 | +12.40% | 67,152 |
45,192 | -32.70% |
| 10 x 1 KiB | 179,223 +/- 3,518 | 240,718 +/- 10,326 | +34.31% | 24,680 |
13,512 | -45.25% |
| 100 x 1 KiB | 16,954 +/- 594 | 22,411 +/- 594 | +32.19% | 246,352 |
134,792 | -45.28% |
Values are means plus or minus JMH's 99.9% confidence-interval half-width.
The baseline and optimized throughput confidence intervals do not overlap in
any scenario.
The 100 x 1 KiB case was repeated on OpenJDK 8u492:
```text
Throughput: 14,472.346 +/- 329.526 -> 18,954.609 +/- 113.257 ops/s (+30.97%)
Allocation: 288,568 -> 173,832 B/op (-39.76%)
```
Its throughput confidence intervals are also disjoint.
I also benchmarked the actual production method from separate baseline and
optimized checkouts in A-B-A order for 100 x 1 KiB:
| Production checkout | Throughput ops/s | 99.9% CI | Allocation B/op |
| --- | ---: | ---: | ---: |
| Baseline A1 | 17,352.526 +/- 400.418 | [16,952.108, 17,752.943] |
246,352.044 |
| This PR | 22,600.219 +/- 880.650 | [21,719.569, 23,480.870] | 134,792.035 |
| Baseline A2 | 16,876.707 +/- 671.661 | [16,205.046, 17,548.368] |
246,352.045 |
This is +30.24%/+33.91% throughput versus the two baseline runs and -45.28%
allocation. The confidence intervals are disjoint.
Benchmark source, methodology, checksums, and all raw per-fork/per-iteration
JSON results: https://gist.github.com/ai-yang/c3e3f65351054263dec6f7b8d67ad3cd
`git diff --check` also passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
