iemejia commented on code in PR #3855:
URL: https://github.com/apache/avro/pull/3855#discussion_r3564818348
##########
lang/perl/lib/Avro/DataFileReader.pm:
##########
@@ -28,17 +28,23 @@ use Object::Tiny qw{
use constant MARKER_SIZE => 16;
-# TODO: refuse to read a block more than block_max_size, instead
-# do partial reads
+# A data-file block is decompressed according to the file's codec. A block with
+# a very high compression ratio (or a malformed block) can expand to far more
+# memory than its compressed size. To guard against unbounded allocation, the
+# decompressed size of a single block is capped. This mirrors the Java SDK's
+# decompression limit (AVRO-4247). The default can be overridden with the
+# AVRO_MAX_DECOMPRESS_LENGTH environment variable.
+use constant DEFAULT_MAX_DECOMPRESS_LENGTH => 200 * 1024 * 1024; # 200 MiB
use Avro::DataFile;
use Avro::BinaryDecoder;
use Avro::Schema;
use Carp;
-use Compress::Zstd;
-use IO::Uncompress::Bunzip2 qw(bunzip2);
+use Compress::Zstd::Decompressor;
+use IO::Uncompress::Bunzip2 ();
use IO::Uncompress::RawInflate ;
Review Comment:
Fixed via option (2): Compress::Zstd::Decompressor is now loaded lazily
inside the zstandard decompression path rather than with a compile-time `use`.
The reader loads and works for the other codecs even when the Decompressor
submodule is unavailable, and reading a zstandard block in that case throws
Avro::DataFile::Error::UnsupportedCodec with a clear message. Pushed in 5b1ba03.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
