yuzelin opened a new pull request, #8691: URL: https://github.com/apache/paimon/pull/8691
This reverts commit 6509987c60321892d8c8acb00e9b5281fd61d78d. ### Purpose AvroBytesArray does not retain the original Avro payload directly. It first decodes every element and re-encodes it into a new byte buffer to build element offsets. When any element is later accessed, the entire array is decoded again, leaving both the raw and decoded representations in memory. Microbenchmarks showed: ARRAY<INT> element access became approximately 1.8–2.3x slower, with 2.2–2.8x more allocations. ARRAY<STRING> element access became approximately 1.6–1.7x slower, with 1.37–1.43x more allocations. Raw rewriting was faster for large integer arrays and string arrays, but this benefit does not justify the regression in the general read path. A prototype that copied raw varints through BinaryDecoder.inputStream() was also tested. It became 5–29% slower when creating arrays and 9–25% slower when rewriting medium and large integer arrays, mainly because each varint byte had to pass through InputStream.read() instead of Avro’s optimized internal buffer access. The intended use case requires extracting an arbitrary range of array elements. Avro elements, especially varint-encoded integers, do not have fixed-width positions or a built-in element index. So the cost is hard to reduce. ### Tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
