juntaozhang opened a new pull request, #10226:
URL: https://github.com/apache/paimon/pull/10226
### Purpose
Optimize variant metadata decoding by caching a batch's (usually
identical) metadata bytes in a
single-slot `VariantMetadata`, so the dictionary is parsed only once per
distinct buffer instead of
on every row.
### Tests
**Benchmark**:
https://github.com/juntaozhang/paimon/commit/07b0d2fcb2940bdc00bef0492df7db94d47767d7
Compares the optimized path (WITH cache) against the baseline (WO, no cache)
on `OBJECT_LEFT_L64`, R=500k, G1 6g. All points show only **minor GC, 0
MAJOR** , median of 2 runs, JDK 8,11, the `VariantMetadata` cache speeds up
variant metadata decoding by roughly ~1.0–1.3x on both JDK 8 and JDK 11 :
| reuse | WO ns (JDK 8) | WITH ns (JDK 8) | spd JDK 8 | WO ns (JDK 11) |
WITH ns (JDK 11) | spd JDK 11 |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| 0% | 12868.2 | 12584.5 | 1.02x | 10700.7 | 10741.7 | 1.00x |
| 10% | 11030.9 | 10560.4 | 1.04x | 8744.2 | 9488.3 | 0.92x |
| 20% | 10997.8 | 10322.8 | 1.07x | 8761.3 | 8764.5 | 1.00x |
| 30% | 10773.0 | 10114.2 | 1.07x | 8794.5 | 8519.2 | 1.03x |
| 40% | 10455.6 | 9710.9 | 1.08x | 8464.6 | 8224.8 | 1.03x |
| 50% | 10268.3 | 9424.4 | 1.09x | 8267.6 | 7896.8 | 1.05x |
| 60% | 10056.1 | 9098.1 | 1.11x | 8062.6 | 7510.4 | 1.07x |
| 70% | 9844.0 | 8721.1 | 1.13x | 7745.0 | 7184.2 | 1.08x |
| 80% | 9627.2 | 8276.0 | 1.16x | 7541.3 | 6780.3 | 1.11x |
| 90% | 9561.5 | 7815.1 | 1.22x | 7288.7 | 6252.8 | 1.17x |
| 100% | 12040.9 | 9371.0 | 1.29x | 9163.1 | 7428.6 | 1.23x |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
