Gabriel39 opened a new pull request, #66244:
URL: https://github.com/apache/doris/pull/66244
### What problem does this PR solve?
Issue Number: N/A
Related PR: #66223
Problem Summary:
The Paimon JNI read path created a new `PaimonColumnValue` for every array
element, map key/value, and struct field on every row. Timestamp conversions
also resolved the session time zone per value and built redundant intermediate
temporal objects. On wide nested data or large scan batches, these short-lived
objects increase allocation rate and GC pressure.
For example, consider two consecutive `ARRAY<INT>` rows, `[10, 20]` and
`[30, 40]`. Previously, reading the second row allocated two new wrappers even
though the first row's wrappers had already been consumed. This PR keeps two
position-specific wrappers and resets their record, index, Doris type, Paimon
type, and time zone before reading the second row. This is safe because
`VectorColumn` consumes the unpacked list synchronously before the scanner
advances to another value.
The same invariant is applied separately to array elements, map keys, map
values, and struct fields. Caches are lazy, grow only when needed, and only the
current collection size or projected struct indexes are emitted, so shrinking
collections cannot expose stale entries. Struct wrappers are cached by the
original field index, preserving sparse/non-contiguous projection behavior
fixed by #66223.
For temporal values, the session time-zone string is now parsed once into a
`ZoneId`. A Paimon `TIMESTAMP_LTZ` epoch instant is converted directly into
that zone instead of constructing UTC and target-zone intermediates.
`TIMESTAMPTZ` uses Paimon's equivalent local representation directly instead of
converting through `Instant` and UTC.
### What changed?
- Lazily reuse nested `PaimonColumnValue` wrappers across rows for arrays,
maps, structs, and recursively nested values.
- Reset all row-dependent state before a cached wrapper is reused.
- Cache the parsed session `ZoneId` instead of resolving it for every value.
- Remove redundant temporal conversion objects while preserving
negative-epoch, nanosecond, and DST behavior.
- Add unit coverage for wrapper identity reuse, growing/shrinking arrays,
maps, sparse struct projections, nested arrays, DST, negative epoch, and
nanosecond precision.
### Local allocation and latency microbenchmark
Environment: Linux x86_64, Oracle JDK 17.0.16, Paimon 1.3.1. The benchmark
ran latest `master` and this commit in separate worktrees on the same machine.
Each case used 100,000 warmup iterations and 5 measured trials; the table
reports medians. Collection cases ran 300,000 operations per trial, timestamp
cases ran 1,000,000. Allocation was measured with `ThreadMXBean`; GC count/time
was measured with `GarbageCollectorMXBean`. A full GC was requested before each
measured trial and excluded from the reported GC delta.
The collection output lists were preallocated and cleared between operations
to isolate allocation inside the Paimon value conversion path. Therefore these
numbers are a focused Java microbenchmark, not an end-to-end query throughput
claim.
| Case | master alloc (B/op) | PR alloc (B/op) | Allocation reduction |
master time (ns/op) | PR time (ns/op) | Time reduction | Median GC
collections/trial |
|---|---:|---:|---:|---:|---:|---:|---:|
| `ARRAY<INT>[16]` | 512.03 | 0.03 | 99.99% | 368.38 | 141.61 | 61.6% | 2 →
0 |
| `MAP<INT,BIGINT>[16]` | 1232.03 | 208.03 | 83.1% | 668.39 | 508.57 | 23.9%
| 2 → 1 |
| `STRUCT<INT,STRING,BIGINT>` | 128.03 | 32.03 | 75.0% | 63.74 | 63.73 | ~0%
| 0 → 0 |
| `TIMESTAMP_LTZ(9)` | 384.01 | 64.01 | 83.3% | 156.25 | 57.88 | 63.0% | 2 →
1 |
| `TIMESTAMPTZ(9)` | 200.01 | 48.01 | 76.0% | 44.45 | 21.65 | 51.3% | 1 → 0 |
The remaining map allocation comes primarily from Paimon's
`GenericMap.keyArray()` / `valueArray()` materialization, which is unchanged by
this PR. Struct latency is effectively unchanged while allocation drops by 75%.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (the local allocation/latency microbenchmark above)
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
Validation command:
```text
mvn -Dmaven.build.cache.enabled=false -pl be-java-extensions/paimon-scanner
-am -Dtest='org.apache.doris.paimon.*Test'
-Dsurefire.failIfNoSpecifiedTests=false test
```
Result: 20 tests, 0 failures, 0 errors, 0 skipped.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
