azecevic000 commented on code in PR #10959:
URL: https://github.com/apache/arrow-rs/pull/10959#discussion_r3917704450
##########
arrow-ipc/src/compression.rs:
##########
@@ -375,14 +386,17 @@ fn decompress_zstd(
))
}
-/// Get the uncompressed length
-/// Notes:
-/// LENGTH_NO_COMPRESSED_DATA: indicate that the data that follows is not
compressed
-/// 0: indicate that there is no data
-/// positive number: indicate the uncompressed length for the following data
-/// Returns an error if the input buffer is shorter than 8 bytes
+/// Reads the uncompressed length from a compressed IPC buffer.
+///
+/// A positive value is the exact expected uncompressed length, `0` indicates
+/// that there is no data, and `-1` indicates that the bytes following the
+/// prefix are not compressed.
+/// IPC decompression limits its output buffer to a positive advertised length
+/// and rejects output whose length differs.
+///
+/// Returns an error if the input buffer is shorter than the 8-byte prefix.
#[inline]
-fn read_uncompressed_size(buffer: &[u8]) -> Result<i64, ArrowError> {
+pub fn read_uncompressed_size(buffer: &[u8]) -> Result<i64, ArrowError> {
Review Comment:
@Rich-T-kid Thank you for the prompt review. I have added a test that
returns an error on an underreported size for LZ4.
To answer your question, this function can be very useful for downstream
users. It allows effective resource governance, where instead of
unconditionally doing decompress and possibly getting an OOM issue during the
allocation, you check if resources (memory in this case) in your system are
available and make a decision based on that. That is also why LZ4 fix is
important, since it gives us a much stronger guarantee.
I don't see any other way to achieve this currently that is exposed through
the public API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
