zhangyue19921010 opened a new issue, #11159:
URL: https://github.com/apache/paimon/issues/11159

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Motivation
   
   Reading a partially backfilled BLOB column creates placeholder rows for gaps 
in its files, then NULL rows where no version supplies a value. 
`AllPlaceholdersRecordReader` and `BlobFallbackRecordReader` allocate a 
`GenericRow` and backing array for each synthetic row.
   
   On master `6d50c1ab0`, a local Data Evolution table with 500,000 normal rows 
and 5,000 populated BLOB rows allocates 48.54 MB on the reader thread per 
BLOB-only scan.
   
   ### Solution
   
   Reuse one synthetic row in each reader, initializing invariant fields once 
and updating the row ID during iteration. Both readers emit at most one batch. 
Preserve the placeholder mode added for incremental compaction in #11139, 
including ARRAY/MAP BLOBs and explicit NULLs.
   
   The same sparse workload allocates 1.02 MB after the change. Warm elapsed 
time is 41.1 → 27.1 ms. A fully populated control remains near 64.88 MB; its 
elapsed samples overlap, with no demonstrated benefit.
   
   ### Anything else?
   
   JDK 11, 8-vCPU Xeon Platinum 8260 VM, local filesystem, G1 and a 2 GiB 
maximum heap. Three fresh JVMs per version and workload, alternating base/head 
order; five warmups and nine measured reads per JVM. Reported values are 
medians of the three JVM medians. Every read checks row count, populated count 
and payload checksum. Timing covers opening and consuming a preplanned scan, 
excluding writes and planning. Allocation uses reader-thread `ThreadMXBean`; 
live heap, RSS and physical I/O were not measured.
   
   AI-generated with OpenAI Codex.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to