zhangyue19921010 opened a new pull request, #11160:
URL: https://github.com/apache/paimon/pull/11160
### Purpose
Fixes #11159.
Sparse BLOB reads allocate a placeholder row for each missing position and
another row when no version supplies a value. Reuse these rows within each
single-batch reader, preserving row IDs, sequence numbers, explicit NULLs and
the placeholder mode used by incremental compaction.
Local Data Evolution table reads on `6d50c1ab0`, 500,000 normal rows,
projecting only the BLOB column:
| BLOB coverage | Reader-thread allocation, before → after | Warm elapsed
time, before → after |
| --- | --- | --- |
| 1% | 48.54 → 1.02 MB | 41.1 → 27.1 ms |
| 100% | 64.88 → 64.88 MB | 544.9 → 552.2 ms |
JDK 11, 8-vCPU Xeon Platinum 8260 VM, local filesystem, G1 with a 2 GiB
maximum heap. Three fresh JVMs per version/workload, alternating order; five
warmups and nine measurements per JVM. Values are medians of JVM medians.
Sparse per-JVM elapsed medians were 41.0–43.5 ms before and 27.0–27.1 ms after;
reader-thread CPU was 37.1 → 26.6 ms. Full-coverage elapsed samples overlapped
(518.4–594.2 ms before, 538.5–579.1 ms after), with no demonstrated benefit.
This measures opening and consuming a preplanned scan; writes and planning are
excluded. Live heap, RSS and physical I/O were not measured. Each fallback
reader now retains one synthetic row.
### Tests
Extended the two-BLOB table test with disjoint populated ranges. Expanded
the fallback test to cover boundary gaps, explicit NULLs, placeholders, row IDs
and sequence numbers in ordinary-read and compaction modes for BLOB, ARRAY and
MAP types. Existing tests cover skip/range iteration, older-version fallback,
deletion vectors and incremental dedicated-file compaction.
Every benchmark iteration checks total rows, populated rows and payload
checksum.
87 related tests passed in each of the two Maven `verify` test executions on
JDK 11, with Checkstyle, Spotless, Enforcer and RAT enabled and without
`fast-build`.
```sh
mvn -pl paimon-core -am -DwildcardSuites=none -DfailIfNoTests=false \
-Dtest=AllPlaceholdersRecordReaderTest,BlobFallbackRecordReaderTest,BlobUpdateTest,MultipleBlobTableTest,DataEvolutionDeletionVectorTest,DataEvolutionDedicatedIncrementalCompactionTest
verify
```
AI-generated with OpenAI Codex; independently reviewed by an AI agent. No
human review is claimed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]