rangareddy commented on issue #17396: URL: https://github.com/apache/hudi/issues/17396#issuecomment-5351244822
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9462). **Findings: not done on `master`.** The review comment that created this ticket ([#12866 (comment)](https://github.com/apache/hudi/pull/12866#discussion_r2105605442)) asks simply: *"Take the output stream and directly write to that?"* The code still buffers instead. `hudi-io/src/main/java/org/apache/hudi/io/hfile/HFileBlock.java:270`: ```java public ByteBuffer serialize() throws IOException { // Block payload. ByteBuffer uncompressedBlockData = getUncompressedBlockDataToWrite(); // Compress if specified. ByteBuffer compressedBlockData = context.getCompressor().compress(uncompressedBlockData); // Buffer for building block. ByteArrayOutputStream baos = new ByteArrayOutputStream(Math.max( context.getBlockSize(), compressedBlockData.limit() + HFILEBLOCK_HEADER_SIZE * 2)); try (DataOutputStream dataOutputStream = new DataOutputStream(baos)) { ... ``` so the header, payload and checksum are assembled into an in-method `ByteArrayOutputStream` and handed back as a `ByteBuffer`. Worth noting the abstract method it depends on has the same shape - `:266`, `protected abstract ByteBuffer getUncompressedBlockDataToWrite()` - so each block is materialised in memory twice (uncompressed, then compressed) before anything is written. Changing `serialize()` alone would only remove the third copy; taking the output stream all the way down means changing that abstract method and its implementations too. That is worth capturing in the ticket so the scope is not underestimated. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
