gripleaf commented on issue #240:
URL: https://github.com/apache/paimon-cpp/issues/240#issuecomment-5392064310
A follow-up PR has been opened for this issue: #242
After profiling the updated implementation, several additional sources of
Arrow-side overhead remained.
This follow-up PR focuses on three areas:
1. Avoid redundant Arrow array validation
Arrays produced by controlled Avro builders were validated again after
`Finish()`. For nested Struct, List, and Map types, this validation recursively
compares data types and computes type fingerprints.
The PR skips this validation in release builds while retaining it in
debug builds for invariant checking.
2. Reuse `ColumnarRow` within a manifest batch
The manifest bucket scan previously constructed and destroyed one
`ColumnarRow` per row, although the underlying Arrow fields and memory pool
remain unchanged throughout the batch.
The PR creates one row view per batch and updates only its row index
during iteration.
3. Reserve Avro builder capacity
The target batch size is known before decoding, but Arrow builders
previously relied on progressive capacity growth.
The PR reserves capacity for the root builder and same-cardinality
Struct children using the configured batch size. List and Map child builders
are reserved using their decoded Avro block sizes, since their element
cardinality is not known in advance.
These changes target management overhead rather than Avro decoding or
decompression itself, reducing repeated validation, object lifecycle
operations, memory reallocations, and data copying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
