XiaoHongbo-Hope opened a new pull request, #8725:
URL: https://github.com/apache/paimon/pull/8725
### Purpose
BLOB compaction is slow because `BlobFormatWriter` opens a fresh source
stream (a
file open or HTTP GET) for every referenced blob — when a compacted file
draws from
one source file with thousands of `BlobRef`s, that's thousands of
redundant opens.
This reuses one source stream across consecutive `BlobRef`s that share it
— seek to
each offset, copy exactly `descriptor.length` bytes — so N references into
one file
cost one open instead of N. The stream is closed on source switch,
on failure, and on writer close. Non-`BlobRef` and unknown-length (HTTP
`-1`)
references keep the old open-per-blob path, so output bytes and CRC are
unchanged.
### Tests
`BlobFormatWriterTest`:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
