discivigour opened a new pull request, #8625: URL: https://github.com/apache/paimon/pull/8625
## What is the purpose of the change? Improve Python BLOB write performance by reducing CRC and Arrow row-selection overhead. ## Brief change log - Use `isal_zlib.crc32` when available, with `zlib` as the fallback. - Reuse the original Arrow batch when all rows belong to one partition/bucket. - Use zero-copy Arrow `slice` for contiguous partition/bucket groups. - Keep Arrow `take` for non-contiguous groups. - Add coverage for CRC compatibility and all three Arrow grouping paths. ## Performance A 60 GB BLOB write test improved as follows: | Data size | Before | After | Elapsed-time reduction | | --- | ---: | ---: | ---: | | 60 GB | 178.984 s | 132.871 s | 25.76% | The elapsed time decreased by 46.113 seconds, from 178.984 seconds to 132.871 seconds. ## Tests - Verify the fallback CRC written to BLOB records matches standard `zlib.crc32`. - Verify `isal_zlib` and `zlib` produce byte-identical BLOB records. - Verify full batches are passed through without Arrow `take`. - Verify contiguous groups use zero-copy Arrow `slice`. - Verify non-contiguous groups continue to use Arrow `take`. - Verify partitioned BLOB data can be written and read end to end. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
