azecevic000 commented on code in PR #10959:
URL: https://github.com/apache/arrow-rs/pull/10959#discussion_r3924067141
##########
arrow-ipc/src/compression.rs:
##########
@@ -375,14 +386,17 @@ fn decompress_zstd(
))
}
-/// Get the uncompressed length
-/// Notes:
-/// LENGTH_NO_COMPRESSED_DATA: indicate that the data that follows is not
compressed
-/// 0: indicate that there is no data
-/// positive number: indicate the uncompressed length for the following data
-/// Returns an error if the input buffer is shorter than 8 bytes
+/// Reads the uncompressed length from a compressed IPC buffer.
+///
+/// A positive value is the exact expected uncompressed length, `0` indicates
+/// that there is no data, and `-1` indicates that the bytes following the
+/// prefix are not compressed.
+/// IPC decompression limits its output buffer to a positive advertised length
+/// and rejects output whose length differs.
+///
+/// Returns an error if the input buffer is shorter than the 8-byte prefix.
#[inline]
-fn read_uncompressed_size(buffer: &[u8]) -> Result<i64, ArrowError> {
+pub fn read_uncompressed_size(buffer: &[u8]) -> Result<i64, ArrowError> {
Review Comment:
Thanks for pointing out to the issue, though the proposition is more about
memory optimization if some kind of projection is done (which I agree can be
significant for some workloads), the issue I raised is more concerned with
having a guaranteed upper bound on memory allocation and knowing it before
allocation happens.
Thanks for the review, I totally understand general hesitation when
expansion of the public API surface area is considered. In this case, though, I
believe the decompressed size is more than internal detail, as downstream users
can generally make decisions based on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
