iemejia commented on code in PR #3851:
URL: https://github.com/apache/avro/pull/3851#discussion_r3567522261
##########
lang/ruby/lib/avro/data_file.rb:
##########
@@ -319,10 +340,22 @@ def decompress(compressed)
# (without the RFC1950 header & checksum). See the docs for
# inflateInit2 in https://www.zlib.net/manual.html
zstream = Zlib::Inflate.new(-Zlib::MAX_WBITS)
- data = zstream.inflate(compressed)
- data << zstream.finish
+ limit = DataFile.max_decompress_length
+ data = "".b
+ # Inflate in chunks so an over-large (or malicious) block is rejected
+ # before its full decompressed form is materialized in memory.
+ append = lambda do |chunk|
+ if data.bytesize + chunk.bytesize > limit
+ raise DecompressionSizeError, "Decompressed block size exceeds the
maximum allowed of #{limit} bytes"
+ end
+ data << chunk
+ end
+ zstream.inflate(compressed, &append)
+ remaining = zstream.finish
+ append.call(remaining) unless remaining.empty?
+ data
Review Comment:
Good catch — the block form of Zlib::Inflate#inflate is Ruby 3.1+. Reworked
to feed the compressed input in chunks and check the accumulated output after
each inflate() call (then finish), which preserves the size cap and works on
Ruby 2.7+.
##########
lang/ruby/lib/avro/data_file.rb:
##########
@@ -374,11 +439,30 @@ def load_snappy!
end
class ZstandardCodec
+ # Size of the compressed input fed to the streaming decompressor per
step,
+ # so the decompressed output can be bounded incrementally.
+ ZSTD_DECOMPRESS_CHUNK_SIZE = 16 * 1024
+
def codec_name; 'zstandard'; end
def decompress(data)
load_zstandard!
- Zstd.decompress(data)
+ limit = DataFile.max_decompress_length
+ stream = Zstd::StreamingDecompress.new
+ uncompressed = +''.b
+ offset = 0
+ size = data.bytesize
+ # Decompress the block in chunks so an over-large (or malicious) block
is
+ # rejected before its full decompressed form is materialized in memory.
+ while offset < size
+ out = stream.decompress(data.byteslice(offset,
ZSTD_DECOMPRESS_CHUNK_SIZE))
+ offset += ZSTD_DECOMPRESS_CHUNK_SIZE
+ if uncompressed.bytesize + out.bytesize > limit
+ raise DecompressionSizeError, "Decompressed block size exceeds the
maximum allowed of #{limit} bytes"
+ end
+ uncompressed << out
+ end
+ uncompressed
end
Review Comment:
Verified the gem API: `Zstd::StreamingDecompress` exposes only
`decompress`/`decompress_with_pos` — there is no `finish` method (calling it
would raise NoMethodError). zstd frames are self-delimiting, so `decompress`
returns all output as the frame's input is consumed; the within-limit
round-trip test confirms full output. So no finish call is applicable/needed
here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
