JingsongLi opened a new pull request, #9371: URL: https://github.com/apache/paimon/pull/9371
### Purpose Avoid materializing Variant metadata and value as byte arrays while moving data through columnar, Arrow, serialization, shredding, and Parquet write paths. ### Changes - Add mandatory ByteBuffer accessors to Variant and implement a ByteBuffer-backed ColumnarVariant. - Expose ByteBuffer views from column vectors, including direct ArrowBuf views for Arrow variable-binary vectors. - Read and write Arrow Variant values without eagerly copying their metadata and value buffers. - Update Variant readers, builders, serializers, shredding, and row-format writers to consume buffers. - Reuse writer-owned scratch arrays when Parquet requires a heap-backed Binary, avoiding a large temporary allocation per row. - Add regression coverage using direct, padded, and buffer-only Variant inputs. ### Verification - 18 targeted tests passed under the normal Maven profile across paimon-common, paimon-format, and paimon-arrow. - Local end-to-end Arrow batch to Parquet measurements with approximately 1 MiB Variant values showed 1.24x throughput and 23.8% less allocation without compression. With Zstd, throughput was compression-dominated while allocation decreased by 15.4%. The local microbenchmark changes are intentionally not included in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
