[
https://issues.apache.org/jira/browse/AVRO-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080096#comment-18080096
]
ASF subversion and git services commented on AVRO-4:
----------------------------------------------------
Commit 27b3f13c1c46e8ecbec8356522cca8d78e87200d in avro's branch
refs/heads/branch-1.12 from Steve Loughran
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=27b3f13c1c ]
AVRO-4247: Enforce decompression size limits (#3745)
* [AVRO-4081] Add decompression size limit to prevent decompression bomb DoS
Add maximum decompression size limit in DeflateCodec to prevent
OutOfMemoryError when processing maliciously crafted Avro files
with high compression ratios (decompression bombs).
The limit defaults to 200MB and can be configured via system property:
org.apache.avro.limits.decompress.maxLength
* Update lang/java/avro/src/main/java/org/apache/avro/file/DeflateCodec.java
Thanks!
Co-authored-by: Martin Grigorov <[email protected]>
* Update lang/java/avro/src/main/java/org/apache/avro/file/DeflateCodec.java
Co-authored-by: Martin Grigorov <[email protected]>
* Address code review feedback for decompression bomb fix
- Move MAX_DECOMPRESS_LENGTH initialization to static block (read once at class
load)
- Add WARNING log for invalid property values (NumberFormatException)
- Validate negative and zero values, reject with warning
- Add "(bytes)" to error message for clarity
- Add quotes around property name in error message
Test command:
java -Xmx64m -Dorg.apache.avro.limits.decompress.maxLength=1048576 \
-jar avro-tools-1.13.0-SNAPSHOT.jar tojson poc.avro
Expected behavior:
Exception in thread "main" org.apache.avro.AvroRuntimeException:
Decompressed size 1056768 (bytes) exceeds maximum allowed size 1048576.
This can be configured by setting the system property
'org.apache.avro.limits.decompress.maxLength'
* [AVRO-4] Move limit checks into NonCopyingByteArrayOutputStream
* Automatically available to all codecs
* Does need an explicit constructor with no limit, used in DataFileWriter
* No tests, though that new constructor makes it trivial
Note: merged in main as DataFileWriter changes would otherwise stop merging
Change-Id: Ifc5b8921a00425df331a4889472b3e78c6677bde
* [AVRO-4247] review feedback
* Tests
* Option read moved to SystemLimitException
Change-Id: I01d4203ad65e0e09dde88b33f603879323b06425
* [AVRO-4247] Rework invocation
* NonCopyingByteArrayOutputStream ctor reverts to unlimited
* moved the check logic to SystemLimitException
* new static method restrictedCapacityOutputStream() is invoked to
get capacity limited streams
* All Codecs use this
* Except Snappy which calls the SystemLimitException check directly
* And there's a test for that too
Change-Id: I0aa355e258c03d452c4e5a4a18605e8e2b357b2d
* spotless
Change-Id: Ie76e91f9e00ff799d2c0d82258802e6a7f4ef756
* getLongLimitFromProperty() to use parameters over inline constants
Change-Id: I5905dac4ac8a9f838cd7b3025906ef278a5cfb58
* change static factory method to "capacityLimitedOutputStream()"
Change-Id: I24dcd2a023c544370c347bdcd17e6d68abeaf260
* Apply compressor changes from iemejia
+ limit the size of a single output stream to max memory/4.
Change-Id: I0344e4c586f982f37eead7d4a6246bd1af5b4e81
* Beautify with spotless
---------
Co-authored-by: OwenSanzas <[email protected]>
Co-authored-by: Ze Sheng <[email protected]>
Co-authored-by: Martin Grigorov <[email protected]>
Co-authored-by: Ryan Skraba <[email protected]>
> difficult to develop Avro in Eclipse
> ------------------------------------
>
> Key: AVRO-4
> URL: https://issues.apache.org/jira/browse/AVRO-4
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.0.0
> Reporter: Patrick D. Hunt
> Assignee: Patrick D. Hunt
> Priority: Major
> Fix For: 1.0.0
>
> Attachments: AVRO-4.patch
>
>
> A fresh checkout of Avro cannot be compiled in eclipse due to tests relying
> on generated code.
> You can generate the needed code using ant, however eclipse still has issues
> due to the fact that the generated
> code is not placed in a directory hierarchy that mimics the package structure
> of the generated class(es).
> I was able to resolve by copying the generated Test.java/Interop.java into a
> dir struct that mimics the
> package struct, but otw I couldn't get Eclipse to compile the code fully.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
