[jira] [Commented] (COMPRESS-542) Corrupt 7z allocates huge amount of SevenZEntries
[
https://issues.apache.org/jira/browse/COMPRESS-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379573#comment-17379573
]
Stefan Bodewig commented on COMPRESS-542:
-
This issue has been assigned the name
[CVE-2021-35516|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-35516]
> Corrupt 7z allocates huge amount of SevenZEntries
> -
>
> Key: COMPRESS-542
> URL: https://issues.apache.org/jira/browse/COMPRESS-542
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Priority: Major
> Fix For: 1.21
>
> Attachments:
> Reduced_memory_allocation_for_corrupted_7z_archives.patch,
> endheadercorrupted.7z, endheadercorrupted2.7z
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We ran into a problem where a 1.43GB corrupt 7z file tried to allocate about
> 138 million SevenZArchiveEntries which will use about 12GB of memory. Sadly
> I'm unable to share the file. If you have enough Memory available the
> following exception is thrown.
> {code:java}
> java.io.IOException: Start header corrupt and unable to guess end Header
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.tryToLocateEndHeader(SevenZFile.java:511)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:470)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:336)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:128)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:369)
> {code}
> 7z itself aborts really quick when I'm trying to list the content of the file.
> {code:java}
> 7z l "corrupt.7z"
> 7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28
> Scanning the drive for archives:
> 1 file, 1537752212 bytes (1467 MiB)
> Listing archive: corrupt.7z
> ERROR: corrupt.7z : corrupt.7z
> Open ERROR: Can not open the file as [7z] archive
> ERRORS:
> Is not archive
> Errors: 1
> {code}
> I hacked together the attached patch which will reduce the memory allocation
> to about 1GB. So lazy instantiation of the entries could be a good solution
> to the problem. Optimal would be to only create the entries if the headers
> could be parsed correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-565.
-
Resolution: Fixed
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Fix For: 1.21
>
> Attachments: commons-compress-1.21-SNAPSHOT.jar,
> image-2021-02-20-15-51-21-747.png
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-542) Corrupt 7z allocates huge amount of SevenZEntries
[
https://issues.apache.org/jira/browse/COMPRESS-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-542.
-
Fix Version/s: 1.21
Resolution: Fixed
And with current master your broken archives will be rejected even earlier as
we now hide recovering a certain type of broken archive introduced with
COMPRESS-497 behind a new option that is off by default.
I think we can close this as fixed in 1.21 by now.
> Corrupt 7z allocates huge amount of SevenZEntries
> -
>
> Key: COMPRESS-542
> URL: https://issues.apache.org/jira/browse/COMPRESS-542
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Priority: Major
> Fix For: 1.21
>
> Attachments:
> Reduced_memory_allocation_for_corrupted_7z_archives.patch,
> endheadercorrupted.7z, endheadercorrupted2.7z
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We ran into a problem where a 1.43GB corrupt 7z file tried to allocate about
> 138 million SevenZArchiveEntries which will use about 12GB of memory. Sadly
> I'm unable to share the file. If you have enough Memory available the
> following exception is thrown.
> {code:java}
> java.io.IOException: Start header corrupt and unable to guess end Header
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.tryToLocateEndHeader(SevenZFile.java:511)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:470)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:336)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:128)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:369)
> {code}
> 7z itself aborts really quick when I'm trying to list the content of the file.
> {code:java}
> 7z l "corrupt.7z"
> 7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28
> Scanning the drive for archives:
> 1 file, 1537752212 bytes (1467 MiB)
> Listing archive: corrupt.7z
> ERROR: corrupt.7z : corrupt.7z
> Open ERROR: Can not open the file as [7z] archive
> ERRORS:
> Is not archive
> Errors: 1
> {code}
> I hacked together the attached patch which will reduce the memory allocation
> to about 1GB. So lazy instantiation of the entries could be a good solution
> to the problem. Optimal would be to only create the entries if the headers
> could be parsed correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-542) Corrupt 7z allocates huge amount of SevenZEntries
[
https://issues.apache.org/jira/browse/COMPRESS-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370314#comment-17370314
]
Stefan Bodewig commented on COMPRESS-542:
-
OK, my benchmarks
https://github.com/bodewig/commons-compress-benchmarks/wiki/COMPRESS-542 show
there is a certain performance hit if there are very many entries - then again
I believe this is acceptable as we need to do the checks anyway and it is
better to perform them before we allocate too much memory.
> Corrupt 7z allocates huge amount of SevenZEntries
> -
>
> Key: COMPRESS-542
> URL: https://issues.apache.org/jira/browse/COMPRESS-542
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Priority: Major
> Attachments:
> Reduced_memory_allocation_for_corrupted_7z_archives.patch,
> endheadercorrupted.7z, endheadercorrupted2.7z
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We ran into a problem where a 1.43GB corrupt 7z file tried to allocate about
> 138 million SevenZArchiveEntries which will use about 12GB of memory. Sadly
> I'm unable to share the file. If you have enough Memory available the
> following exception is thrown.
> {code:java}
> java.io.IOException: Start header corrupt and unable to guess end Header
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.tryToLocateEndHeader(SevenZFile.java:511)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:470)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:336)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:128)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:369)
> {code}
> 7z itself aborts really quick when I'm trying to list the content of the file.
> {code:java}
> 7z l "corrupt.7z"
> 7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28
> Scanning the drive for archives:
> 1 file, 1537752212 bytes (1467 MiB)
> Listing archive: corrupt.7z
> ERROR: corrupt.7z : corrupt.7z
> Open ERROR: Can not open the file as [7z] archive
> ERRORS:
> Is not archive
> Errors: 1
> {code}
> I hacked together the attached patch which will reduce the memory allocation
> to about 1GB. So lazy instantiation of the entries could be a good solution
> to the problem. Optimal would be to only create the entries if the headers
> could be parsed correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-566) make gzip deflate buffer size configurable
[ https://issues.apache.org/jira/browse/COMPRESS-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-566. - Fix Version/s: 1.21 Resolution: Fixed > make gzip deflate buffer size configurable > -- > > Key: COMPRESS-566 > URL: https://issues.apache.org/jira/browse/COMPRESS-566 > Project: Commons Compress > Issue Type: Improvement > Components: Compressors >Reporter: Brett Okken >Priority: Minor > Fix For: 1.21 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > The deflateBuffer in GzipCompressorOutputStream is hardcoded to 512. > It would be good if this could be configurable in GzipParameters. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-562) ZipArchiveInputStream fails with unexpected record signature while ZipInputStream from java.util.zip succeeds
[
https://issues.apache.org/jira/browse/COMPRESS-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349780#comment-17349780
]
Stefan Bodewig commented on COMPRESS-562:
-
well the zero bytes look like padding, unfortunately padding isn't mentioned
anywhere. But it looks as if the signing header wanted to start at a 512 byte
block boundary. We could try to skip all-0 bytes until we reach a block
boundary if we don't recognize the suspect header and it starts with 0s. There
we could try to see whether we find a good signature. But like so often we'd
only be fighting against limitations of trying to stream a ZIP archive.
In general {{ZipArchiveInputStream}} really is a very limited tool. The ZIP
archive format is not one that lends itself to streaming, it works best with
random access this is why {{ZipFile}} is so much superior to it. If you can use
{{ZipFile}}, do so. Not just because the APK signature block problem goes away,
it is so much "more correct", see
https://commons.apache.org/proper/commons-compress/zip.html#ZipArchiveInputStream_vs_ZipFile
> ZipArchiveInputStream fails with unexpected record signature while
> ZipInputStream from java.util.zip succeeds
> -
>
> Key: COMPRESS-562
> URL: https://issues.apache.org/jira/browse/COMPRESS-562
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
> Environment: Zip 3.0 (July 5th 2008), by Info-ZIP, Compiled with gcc
> 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14) for Unix (Mac OS X)
> on Feb 22 2019.
> osx 10.14.6, AdoptOpenJDK 11.0.7
>Reporter: Oleksii Khomchenko
>Priority: Major
> Attachments: apk.PNG, test-services-1.1.0.apk
>
>
> Thank you a lot for the library.
>
> I recently encountered next issue:
> {code:java}
> Exception in thread "main" java.util.zip.ZipException: Unexpected record
> signature: 0X0
> {code}
> is thrown when reading test-services-1.1.0.apk from
> [https://maven.google.com/web/index.html?q=test-ser#androidx.test.services:test-services:1.1.0]
> via commons-compress:1.20 while java.util.zip reads it without the exception.
>
> {code:java}
> public class UnzipTestServicesSample {
> public static void main(String[] args) throws Exception {
> Path p = Paths.get("test-services-1.1.0.apk");
> System.out.println("\n=== java std zip ===\n");
> try (InputStream is = Files.newInputStream(p); ZipInputStream zis =
> new ZipInputStream(is)) {
> ZipEntry entry;
> while ((entry = zis.getNextEntry()) != null) {
> System.out.println("entry: " + entry.getName());
> }
> }
> System.out.println("\n=== apache compress zip ===\n");
> try (InputStream is = Files.newInputStream(p); ArchiveInputStream ais
> = new ZipArchiveInputStream(is)) {
> ArchiveEntry entry;
> while ((entry = ais.getNextEntry()) != null) {
> System.out.println("entry: " + entry.getName());
> }
> }
> }
> }{code}
>
> zip -T says that archive is fine:
>
> {code:java}
> $ zip -T test-services-1.1.0.apk
> test of test-services-1.1.0.apk OK{code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-574) Byte range support in archive creation
[
https://issues.apache.org/jira/browse/COMPRESS-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349775#comment-17349775
]
Stefan Bodewig commented on COMPRESS-574:
-
[~gaellalire] many thanks for your code. This is not really a patch to this
library as such, you are providing a completely different API that doesn't use
anything provided by Commons Compress internally. I've got a hard time
envisioning how we'd integrate the two with each other.
Are you aware of {{ZipArchiveOutputStream}}'s {{addRawArchiveEntry}} method?
Yes, I've read you want the final result as an {{InputStream}} without holding
the archive in memory, but I must admit I don't understand why. The server
use-case you are talking about could be writing to the ServletOutputStream (or
something similar) directly.
Your requirements sound very specific and so far you are the first person I've
heard of with exactly the constraints you describe.
> Byte range support in archive creation
> --
>
> Key: COMPRESS-574
> URL: https://issues.apache.org/jira/browse/COMPRESS-574
> Project: Commons Compress
> Issue Type: Improvement
> Components: Archivers
>Reporter: Gaël Lalire
>Priority: Minor
> Attachments: DynamicZip.java, DynamicZipTest.java
>
>
> When you have a ZIP which contains _N_ components and you want to let the
> user choose which components it needs, you need to create _2^N - 1_ ZIP.
> So the idea is to store each component once (or twice if you want both
> deflated and stored version), and create the ZIP on the fly.
> For the moment you can stream with a ZipOutputStream but if you need an
> InputStream things get a lot harder. I guess programs are writing the ZIP to
> a file system and read from it after, so not really a streaming anymore.
> Also ZipOutputStream will never allow you to resume from a byte range, you
> need to generate all previous data.
> So I made a class to do that, I think such functionality has its place in
> commons compress.
> You can see my code attached and adapt it for better integration / other
> archive type support or simply to get inspired.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-578) Java 8 improvements
[ https://issues.apache.org/jira/browse/COMPRESS-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-578. - Fix Version/s: 1.21 Resolution: Fixed PR merged, thank you > Java 8 improvements > --- > > Key: COMPRESS-578 > URL: https://issues.apache.org/jira/browse/COMPRESS-578 > Project: Commons Compress > Issue Type: Sub-task >Reporter: Arturo Bernal >Priority: Major > Fix For: 1.21 > > Time Spent: 2h > Remaining Estimate: 0h > > * Use lambda alternative > * Use reference > * Replace Loop with Collection.removeIf() > * Use mapToLong().sum()' > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-580) Remove redundant operation
[ https://issues.apache.org/jira/browse/COMPRESS-580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-580. - Fix Version/s: 1.21 Resolution: Fixed PR merged, thank you. > Remove redundant operation > -- > > Key: COMPRESS-580 > URL: https://issues.apache.org/jira/browse/COMPRESS-580 > Project: Commons Compress > Issue Type: Sub-task >Reporter: Arturo Bernal >Priority: Major > Fix For: 1.21 > > Time Spent: 40m > Remaining Estimate: 0h > > * Redundant array creation > * Redundant type arguments > * Redundant type cast > * Redundant String operation > * Unnecessarily escaped character -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-579) Remove redundant local variable
[ https://issues.apache.org/jira/browse/COMPRESS-579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-579. - Fix Version/s: 1.21 Resolution: Fixed PR merged, thank you. > Remove redundant local variable > --- > > Key: COMPRESS-579 > URL: https://issues.apache.org/jira/browse/COMPRESS-579 > Project: Commons Compress > Issue Type: Sub-task >Reporter: Arturo Bernal >Priority: Major > Fix For: 1.21 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-577) Simplify assertion
[ https://issues.apache.org/jira/browse/COMPRESS-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-577. - Fix Version/s: 1.21 Resolution: Fixed PR merged, thank you. > Simplify assertion > -- > > Key: COMPRESS-577 > URL: https://issues.apache.org/jira/browse/COMPRESS-577 > Project: Commons Compress > Issue Type: Sub-task >Reporter: Arturo Bernal >Priority: Major > Fix For: 1.21 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Replace assert calls with simpler but equivalent calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-552) OSGI check broken - try to load class BundleEvent always fails
[
https://issues.apache.org/jira/browse/COMPRESS-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-552.
-
Resolution: Fixed
Great, thank you [~mtomcat_daobi]
> OSGI check broken - try to load class BundleEvent always fails
> --
>
> Key: COMPRESS-552
> URL: https://issues.apache.org/jira/browse/COMPRESS-552
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.20
>Reporter: Björn Michael
>Priority: Major
> Fix For: 1.21
>
> Attachments: commons-compress-1.21-SNAPSHOT.jar
>
>
> There is a check for running in OSGI env. in {{ZstdUtils}}, {{BrotliUtils}},
> {{LZMAUtils}} and {{XZUtils}} like this:
> {code:java}
> static {
> cachedZstdAvailability = CachedAvailability.DONT_CACHE;
> try {
> Class.forName("org.osgi.framework.BundleEvent");
> } catch (final Exception ex) { // NOSONAR
> setCacheZstdAvailablity(true);
> }
> }
> {code}
> Loading the class {{org.osgi.framework.BundleEvent}} always fails because
> {{Import-Package}} directive for {{org.osgi.framework}} is missing in
> _MANIFEST.MF_. Or it requires another more sophisticated approaches
> ([https://stackoverflow.com/q/5879040/1061929]).
> Tested with Eclipse 4.14 (org.eclipse.osgi_3.15.100.v20191114-1701.jar)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (COMPRESS-552) OSGI check broken - try to load class BundleEvent always fails
[
https://issues.apache.org/jira/browse/COMPRESS-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-552:
Attachment: commons-compress-1.21-SNAPSHOT.jar
> OSGI check broken - try to load class BundleEvent always fails
> --
>
> Key: COMPRESS-552
> URL: https://issues.apache.org/jira/browse/COMPRESS-552
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.20
>Reporter: Björn Michael
>Priority: Major
> Fix For: 1.21
>
> Attachments: commons-compress-1.21-SNAPSHOT.jar
>
>
> There is a check for running in OSGI env. in {{ZstdUtils}}, {{BrotliUtils}},
> {{LZMAUtils}} and {{XZUtils}} like this:
> {code:java}
> static {
> cachedZstdAvailability = CachedAvailability.DONT_CACHE;
> try {
> Class.forName("org.osgi.framework.BundleEvent");
> } catch (final Exception ex) { // NOSONAR
> setCacheZstdAvailablity(true);
> }
> }
> {code}
> Loading the class {{org.osgi.framework.BundleEvent}} always fails because
> {{Import-Package}} directive for {{org.osgi.framework}} is missing in
> _MANIFEST.MF_. Or it requires another more sophisticated approaches
> ([https://stackoverflow.com/q/5879040/1061929]).
> Tested with Eclipse 4.14 (org.eclipse.osgi_3.15.100.v20191114-1701.jar)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-552) OSGI check broken - try to load class BundleEvent always fails
[
https://issues.apache.org/jira/browse/COMPRESS-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17345773#comment-17345773
]
Stefan Bodewig commented on COMPRESS-552:
-
TBH I have no idea which job publishes the snapshots and how often it runs -
and whether it actually still does (it doesn't look that way). I was hoping you
could build the master branch of Commons Compress yourself. I'll attach a
SNAPSHOT here in a few minutes.
> OSGI check broken - try to load class BundleEvent always fails
> --
>
> Key: COMPRESS-552
> URL: https://issues.apache.org/jira/browse/COMPRESS-552
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.20
>Reporter: Björn Michael
>Priority: Major
> Fix For: 1.21
>
>
> There is a check for running in OSGI env. in {{ZstdUtils}}, {{BrotliUtils}},
> {{LZMAUtils}} and {{XZUtils}} like this:
> {code:java}
> static {
> cachedZstdAvailability = CachedAvailability.DONT_CACHE;
> try {
> Class.forName("org.osgi.framework.BundleEvent");
> } catch (final Exception ex) { // NOSONAR
> setCacheZstdAvailablity(true);
> }
> }
> {code}
> Loading the class {{org.osgi.framework.BundleEvent}} always fails because
> {{Import-Package}} directive for {{org.osgi.framework}} is missing in
> _MANIFEST.MF_. Or it requires another more sophisticated approaches
> ([https://stackoverflow.com/q/5879040/1061929]).
> Tested with Eclipse 4.14 (org.eclipse.osgi_3.15.100.v20191114-1701.jar)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-552) OSGI check broken - try to load class BundleEvent always fails
[
https://issues.apache.org/jira/browse/COMPRESS-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17345664#comment-17345664
]
Stefan Bodewig commented on COMPRESS-552:
-
[~mtomcat_daobi] could you give commit 23b758fb a try please?
> OSGI check broken - try to load class BundleEvent always fails
> --
>
> Key: COMPRESS-552
> URL: https://issues.apache.org/jira/browse/COMPRESS-552
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.20
>Reporter: Björn Michael
>Priority: Major
> Fix For: 1.21
>
>
> There is a check for running in OSGI env. in {{ZstdUtils}}, {{BrotliUtils}},
> {{LZMAUtils}} and {{XZUtils}} like this:
> {code:java}
> static {
> cachedZstdAvailability = CachedAvailability.DONT_CACHE;
> try {
> Class.forName("org.osgi.framework.BundleEvent");
> } catch (final Exception ex) { // NOSONAR
> setCacheZstdAvailablity(true);
> }
> }
> {code}
> Loading the class {{org.osgi.framework.BundleEvent}} always fails because
> {{Import-Package}} directive for {{org.osgi.framework}} is missing in
> _MANIFEST.MF_. Or it requires another more sophisticated approaches
> ([https://stackoverflow.com/q/5879040/1061929]).
> Tested with Eclipse 4.14 (org.eclipse.osgi_3.15.100.v20191114-1701.jar)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-542) Corrupt 7z allocates huge amount of SevenZEntries
[
https://issues.apache.org/jira/browse/COMPRESS-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17345655#comment-17345655
]
Stefan Bodewig commented on COMPRESS-542:
-
I've added a few more counts during the sanity check phase and the result is
convincing IMHO. The unit test {{testNoOOMOnCorruptedHeader}} added earlier
initially took something like 15s on my machine and now finishes with less than
half a second (and almost no memory being consumed at all).
I'll run a few benchmarks to see whether - and if how much - the additional
parser run hurts for valid archives to get some data on whether to enable
sanity checking for each archive (it is right now) or only for suspect broken
archives.
> Corrupt 7z allocates huge amount of SevenZEntries
> -
>
> Key: COMPRESS-542
> URL: https://issues.apache.org/jira/browse/COMPRESS-542
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Priority: Major
> Attachments:
> Reduced_memory_allocation_for_corrupted_7z_archives.patch,
> endheadercorrupted.7z, endheadercorrupted2.7z
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We ran into a problem where a 1.43GB corrupt 7z file tried to allocate about
> 138 million SevenZArchiveEntries which will use about 12GB of memory. Sadly
> I'm unable to share the file. If you have enough Memory available the
> following exception is thrown.
> {code:java}
> java.io.IOException: Start header corrupt and unable to guess end Header
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.tryToLocateEndHeader(SevenZFile.java:511)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:470)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:336)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:128)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:369)
> {code}
> 7z itself aborts really quick when I'm trying to list the content of the file.
> {code:java}
> 7z l "corrupt.7z"
> 7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28
> Scanning the drive for archives:
> 1 file, 1537752212 bytes (1467 MiB)
> Listing archive: corrupt.7z
> ERROR: corrupt.7z : corrupt.7z
> Open ERROR: Can not open the file as [7z] archive
> ERRORS:
> Is not archive
> Errors: 1
> {code}
> I hacked together the attached patch which will reduce the memory allocation
> to about 1GB. So lazy instantiation of the entries could be a good solution
> to the problem. Optimal would be to only create the entries if the headers
> could be parsed correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-539) TarArchiveInputStream allocates a lot of memory when iterating through an archive
[
https://issues.apache.org/jira/browse/COMPRESS-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-539.
-
Resolution: Won't Fix
> TarArchiveInputStream allocates a lot of memory when iterating through an
> archive
> -
>
> Key: COMPRESS-539
> URL: https://issues.apache.org/jira/browse/COMPRESS-539
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Assignee: Peter Lee
>Priority: Major
> Attachments: Don't_call_InputStream#skip.patch,
> Reuse_recordBuffer.patch, image-2020-06-21-10-58-07-917.png,
> image-2020-06-21-10-58-43-255.png, image-2020-06-21-10-59-10-825.png,
> image-2020-07-05-22-10-07-402.png, image-2020-07-05-22-11-25-526.png,
> image-2020-07-05-22-32-15-131.png, image-2020-07-05-22-32-31-511.png
>
>
> I iterated through the linux source tar and noticed some unneeded
> allocations happen without extracting any data.
> Reproducing code
> {code:java}
> File tarFile = new File("linux-5.7.1.tar");
> try (TarArchiveInputStream in = new
> TarArchiveInputStream(Files.newInputStream(tarFile.toPath( {
> TarArchiveEntry entry;
> while ((entry = in.getNextTarEntry()) != null) {
> }
> }
> {code}
> The measurement was done on Java 11.0.7 with the Java Flight Recorder.
> Options used:
> -XX:StartFlightRecording=settings=profile,filename=allocations.jfr
> Baseline with the current master implementation:
> Estimated TLAB allocation: 293MiB
> !image-2020-06-21-10-58-07-917.png!
> 1. IOUtils.skip -> input.skip(numToSkip)
> This delegates in my test scenario to the InputStream.skip implementation
> which allocates a new byte[] for every invocation. By simply commenting out
> the while loop which calls the skip method the estimated TLAB allocation
> drops to 164MiB (-129MiB).
> !image-2020-06-21-10-58-43-255.png!
> Commenting out the skip call does not seem to be the best solution but it
> was quick for me to see how much memory can be saved. Also no unit tests
> where failing for me.
> 2. TarArchiveInputStream.readRecord
> For every read of the record a new byte[] is created. Since the record size
> does not change the byte[] can be reused and created when instantiating the
> TarStream. This optimization is already present in the
> TarArchiveOutputStream. Reusing the buffer reduces the estimated TLAB
> allocations further to 128MiB (-36MiB).
> !image-2020-06-21-10-59-10-825.png!
> I attached the patches I used so the results can be verified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-552) OSGI check broken - try to load class BundleEvent always fails
[
https://issues.apache.org/jira/browse/COMPRESS-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-552.
-
Fix Version/s: 1.21
Resolution: Fixed
assuming it works in a real environment just like inside our tests
> OSGI check broken - try to load class BundleEvent always fails
> --
>
> Key: COMPRESS-552
> URL: https://issues.apache.org/jira/browse/COMPRESS-552
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.20
>Reporter: Björn Michael
>Priority: Major
> Fix For: 1.21
>
>
> There is a check for running in OSGI env. in {{ZstdUtils}}, {{BrotliUtils}},
> {{LZMAUtils}} and {{XZUtils}} like this:
> {code:java}
> static {
> cachedZstdAvailability = CachedAvailability.DONT_CACHE;
> try {
> Class.forName("org.osgi.framework.BundleEvent");
> } catch (final Exception ex) { // NOSONAR
> setCacheZstdAvailablity(true);
> }
> }
> {code}
> Loading the class {{org.osgi.framework.BundleEvent}} always fails because
> {{Import-Package}} directive for {{org.osgi.framework}} is missing in
> _MANIFEST.MF_. Or it requires another more sophisticated approaches
> ([https://stackoverflow.com/q/5879040/1061929]).
> Tested with Eclipse 4.14 (org.eclipse.osgi_3.15.100.v20191114-1701.jar)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-542) Corrupt 7z allocates huge amount of SevenZEntries
[
https://issues.apache.org/jira/browse/COMPRESS-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344081#comment-17344081
]
Stefan Bodewig commented on COMPRESS-542:
-
Commit 26924e96 contains an extended sanity check which gets away with lot less
memory allocations (basically a smallish constant number and a few bytes - up
to two longs plus a bit - times the claimed number of folders inside of the
archive currently). Right now it could check a few more things and I'll work on
that. I've enabled the check unconditionally but will revisit that once done as
it seems as if it would slow down normal operation - we'll have to see how
significant the effect is, and how much it will be reduced by removing the
checks that are now performed for a second time when actually filling the
metadata structures.
> Corrupt 7z allocates huge amount of SevenZEntries
> -
>
> Key: COMPRESS-542
> URL: https://issues.apache.org/jira/browse/COMPRESS-542
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Priority: Major
> Attachments:
> Reduced_memory_allocation_for_corrupted_7z_archives.patch,
> endheadercorrupted.7z, endheadercorrupted2.7z
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We ran into a problem where a 1.43GB corrupt 7z file tried to allocate about
> 138 million SevenZArchiveEntries which will use about 12GB of memory. Sadly
> I'm unable to share the file. If you have enough Memory available the
> following exception is thrown.
> {code:java}
> java.io.IOException: Start header corrupt and unable to guess end Header
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.tryToLocateEndHeader(SevenZFile.java:511)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:470)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:336)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:128)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:369)
> {code}
> 7z itself aborts really quick when I'm trying to list the content of the file.
> {code:java}
> 7z l "corrupt.7z"
> 7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28
> Scanning the drive for archives:
> 1 file, 1537752212 bytes (1467 MiB)
> Listing archive: corrupt.7z
> ERROR: corrupt.7z : corrupt.7z
> Open ERROR: Can not open the file as [7z] archive
> ERRORS:
> Is not archive
> Errors: 1
> {code}
> I hacked together the attached patch which will reduce the memory allocation
> to about 1GB. So lazy instantiation of the entries could be a good solution
> to the problem. Optimal would be to only create the entries if the headers
> could be parsed correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-542) Corrupt 7z allocates huge amount of SevenZEntries
[
https://issues.apache.org/jira/browse/COMPRESS-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337996#comment-17337996
]
Stefan Bodewig commented on COMPRESS-542:
-
As [~akelday] points out in the github PR one problem is that {{SevnZFile}}
tries to recover from a corrupted archive and gives up an important CRC check
that way. Of course an OOM could happen on a legitimate archive that is really
big. For the latter case we could reuse the memory limit of the
{{SevenZFileOption}} we currently only use when decoding streams.
One idea I'm currently pondering is if either a limit has been set or we know
the archive is corrupt then we parse the metadata in two passes. The first pass
would only collect enough information to guess the amount of memory we'd need
to allocate and at the same time sanity check values - like array sizes must be
positive, positions of other "places" must be inside of the bounds of the
archive and so on.
This would probably slow down opening a new archive in certain cases but would
allow us to reject corrupted archives without running out of memory - and would
allow users to commit to a certain amount of memory. Of course we'd only be
guessing the amount of memory we require but this sounds better than not
providing any control at all.
> Corrupt 7z allocates huge amount of SevenZEntries
> -
>
> Key: COMPRESS-542
> URL: https://issues.apache.org/jira/browse/COMPRESS-542
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Priority: Major
> Attachments:
> Reduced_memory_allocation_for_corrupted_7z_archives.patch,
> endheadercorrupted.7z, endheadercorrupted2.7z
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We ran into a problem where a 1.43GB corrupt 7z file tried to allocate about
> 138 million SevenZArchiveEntries which will use about 12GB of memory. Sadly
> I'm unable to share the file. If you have enough Memory available the
> following exception is thrown.
> {code:java}
> java.io.IOException: Start header corrupt and unable to guess end Header
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.tryToLocateEndHeader(SevenZFile.java:511)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:470)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:336)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:128)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:369)
> {code}
> 7z itself aborts really quick when I'm trying to list the content of the file.
> {code:java}
> 7z l "corrupt.7z"
> 7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28
> Scanning the drive for archives:
> 1 file, 1537752212 bytes (1467 MiB)
> Listing archive: corrupt.7z
> ERROR: corrupt.7z : corrupt.7z
> Open ERROR: Can not open the file as [7z] archive
> ERRORS:
> Is not archive
> Errors: 1
> {code}
> I hacked together the attached patch which will reduce the memory allocation
> to about 1GB. So lazy instantiation of the entries could be a good solution
> to the problem. Optimal would be to only create the entries if the headers
> could be parsed correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-575) Harden handling of sparse TAR entries
[ https://issues.apache.org/jira/browse/COMPRESS-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-575. - Fix Version/s: 1.21 Resolution: Fixed > Harden handling of sparse TAR entries > - > > Key: COMPRESS-575 > URL: https://issues.apache.org/jira/browse/COMPRESS-575 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.20 >Reporter: Stefan Bodewig >Priority: Major > Fix For: 1.21 > > > Currently the handling of sparse tar entries works well for the blue-sky case > where the archive is not corrupted in any way. But if there are problems like > non-numeric values in the wrong places, negative offsets and so on, Compress > may throw some hard to understand RuntimeExceptions or fail in other > difficult to trace ways. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-567) IllegalArgumentException in ZipFile.positionAtCentralDirectory
[
https://issues.apache.org/jira/browse/COMPRESS-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-567.
-
Resolution: Fixed
> IllegalArgumentException in ZipFile.positionAtCentralDirectory
> --
>
> Key: COMPRESS-567
> URL: https://issues.apache.org/jira/browse/COMPRESS-567
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Fabian Meumertzheim
>Priority: Major
> Fix For: 1.21
>
> Attachments: crash.zip
>
>
> The following snippet of code throws an undeclared IllegalArgumentException:
> {code:java}
> byte[] bytes = Base64.getDecoder().decode("UEsFBgAAAQD//1AAJP9QAA==");
> SeekableInMemoryByteChannel input = new SeekableInMemoryByteChannel(bytes);
> try {
> ZipFile file = new ZipFile(input);
> } catch (IOException ignored) {}
> {code}
> The stack trace is:
> {noformat}
> java.lang.IllegalArgumentException: Position has to be in range 0.. 2147483647
> at
> org.apache.commons.compress.utils.SeekableInMemoryByteChannel.position(SeekableInMemoryByteChannel.java:94)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory32(ZipFile.java:1128)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:1037)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:702)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:371)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:318)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:274)
> {noformat}
> I also attached the input as a ZIP file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (COMPRESS-575) Harden handling of sparse TAR entries
[ https://issues.apache.org/jira/browse/COMPRESS-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig updated COMPRESS-575: Issue Type: Bug (was: Task) > Harden handling of sparse TAR entries > - > > Key: COMPRESS-575 > URL: https://issues.apache.org/jira/browse/COMPRESS-575 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.20 >Reporter: Stefan Bodewig >Priority: Major > > Currently the handling of sparse tar entries works well for the blue-sky case > where the archive is not corrupted in any way. But if there are problems like > non-numeric values in the wrong places, negative offsets and so on, Compress > may throw some hard to understand RuntimeExceptions or fail in other > difficult to trace ways. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (COMPRESS-575) Harden handling of sparse TAR entries
Stefan Bodewig created COMPRESS-575: --- Summary: Harden handling of sparse TAR entries Key: COMPRESS-575 URL: https://issues.apache.org/jira/browse/COMPRESS-575 Project: Commons Compress Issue Type: Task Affects Versions: 1.20 Reporter: Stefan Bodewig Currently the handling of sparse tar entries works well for the blue-sky case where the archive is not corrupted in any way. But if there are problems like non-numeric values in the wrong places, negative offsets and so on, Compress may throw some hard to understand RuntimeExceptions or fail in other difficult to trace ways. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-573) Configuration error in Remote Desktop with Apache guacamole
[ https://issues.apache.org/jira/browse/COMPRESS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305663#comment-17305663 ] Stefan Bodewig commented on COMPRESS-573: - [~junaidmehmood] this issue tracker is about Apache Commons Compress, you seem to have selected the wrong project. Apart from that I agree with Michael your issue seems to be one that should use the project's user support resources, see https://guacamole.apache.org/support/ > Configuration error in Remote Desktop with Apache guacamole > --- > > Key: COMPRESS-573 > URL: https://issues.apache.org/jira/browse/COMPRESS-573 > Project: Commons Compress > Issue Type: Bug >Reporter: Junaid Mehmood >Priority: Critical > > I cant connect my virtual machine with Apache guacamole through remote > desktop. please help me in this regards. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-571) 7z random access failes on shuffled entry list
[
https://issues.apache.org/jira/browse/COMPRESS-571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303144#comment-17303144
]
Stefan Bodewig commented on COMPRESS-571:
-
+1 to returning a copy. We already do in other places (for example when
returning ZIP extra fields). I wonder why findbugs doesn't complain about the
current code - at least it flags cases where you return the array directly.
> 7z random access failes on shuffled entry list
> --
>
> Key: COMPRESS-571
> URL: https://issues.apache.org/jira/browse/COMPRESS-571
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Priority: Major
>
> I noticed a problem on a 7z file and could reproduce the error if the
> InputStream is retrieved after shuffling the entries.
> This test fails with a checksum verification error
> {code:java}
> @Test
> public void retrieveInputStreamForShuffledEntries() throws IOException {
> try (final SevenZFile sevenZFile = new
> SevenZFile(getFile("COMPRESS-256.7z"))) {
> List entries = (List)
> sevenZFile.getEntries();
> Collections.shuffle(entries);
> for (final SevenZArchiveEntry entry : entries) {
> IOUtils.toByteArray(sevenZFile.getInputStream(entry));
> }
> }
> }
> {code}
> This is the exception
> {code:java}
> java.io.IOException: Checksum verification failed
> at
> org.apache.commons.compress.utils.ChecksumVerifyingInputStream.read(ChecksumVerifyingInputStream.java:94)
> at
> org.apache.commons.compress.utils.ChecksumVerifyingInputStream.read(ChecksumVerifyingInputStream.java:74)
> at org.apache.commons.compress.utils.IOUtils.copy(IOUtils.java:87)
> at org.apache.commons.compress.utils.IOUtils.copy(IOUtils.java:62)
> at
> org.apache.commons.compress.utils.IOUtils.toByteArray(IOUtils.java:247)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFileTest.retrieveInputStreamForShuffledEntries(SevenZFileTest.java:616)
> {code}
> This also fails on the current master with the same error
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-510) Multiple retrievals of InputStream for same SevenZFile entry fails
[
https://issues.apache.org/jira/browse/COMPRESS-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-510.
-
Resolution: Fixed
> Multiple retrievals of InputStream for same SevenZFile entry fails
> --
>
> Key: COMPRESS-510
> URL: https://issues.apache.org/jira/browse/COMPRESS-510
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Assignee: Peter Lee
>Priority: Major
> Fix For: 1.21
>
> Attachments: image-2020-04-22-16-55-08-369.png
>
>
> I was trying out the new random access for the 7z files and have one of our
> tests failing where we are trying to read the same entry multiple times
> without closing the archive.
> Reproducing test case (I added this locally to the SevenZFileTest class)
> {code:java}
> @Test
> public void retrieveInputStreamForEntryMultipleTimes() throws IOException {
> try (SevenZFile sevenZFile = new SevenZFile(getFile("bla.7z"))) {
> for (SevenZArchiveEntry entry : sevenZFile.getEntries()) {
> byte[] firstRead =
> IOUtils.toByteArray(sevenZFile.getInputStream(entry));
> byte[] secondRead =
> IOUtils.toByteArray(sevenZFile.getInputStream(entry));
> assertArrayEquals(firstRead, secondRead);
> }
> }
> }
> {code}
> The Exception thrown is
> {code:java}
> java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 2
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.buildDecodingStream(SevenZFile.java:1183)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.getInputStream(SevenZFile.java:1436)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFileTest.retrieveInputStreamForEntryMultipleTimes(SevenZFileTest.java:688)
> ...
> {code}
> A similar test case for e.g. zip works fine
> {code:java}
> @Test
> public void retrieveInputStreamForEntryMultipleTimes() throws IOException {
> try (ZipFile zipFile = new ZipFile(getFile("bla.zip"))) {
> Enumeration entry = zipFile.getEntries();
> while (entry.hasMoreElements()) {
> ZipArchiveEntry e = entry.nextElement();
> byte[] firstRead = IOUtils.toByteArray(zipFile.getInputStream(e));
> byte[] secondRead =
> IOUtils.toByteArray(zipFile.getInputStream(e));
> assertArrayEquals(firstRead, secondRead);
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-540) Random access on Tar archive
[ https://issues.apache.org/jira/browse/COMPRESS-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-540. - Resolution: Fixed Thank you [~rschimpf] > Random access on Tar archive > > > Key: COMPRESS-540 > URL: https://issues.apache.org/jira/browse/COMPRESS-540 > Project: Commons Compress > Issue Type: Improvement >Reporter: Robin Schimpf >Priority: Major > Fix For: 1.21 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The TarArchiveInputStream only provides sequential access. If only a small > amount of files from the archive is needed large amount of data in the input > stream needs to be skipped. > Therefore I was working on a implementation to provide random access to > TarFiles equal to the ZipFile api. The basic idea behind the implementation > is the following > * Random access is backed by a SeekableByteChannel > * Read all headers of the tar file and save the place to the data of every > header > * User can request an input stream for any entry in the archive multiple > times -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-552) OSGI check broken - try to load class BundleEvent always fails
[
https://issues.apache.org/jira/browse/COMPRESS-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296837#comment-17296837
]
Stefan Bodewig commented on COMPRESS-552:
-
Commits 30a77505 seems to work, it would be good if you could test it with a
real OSGi setup.
> OSGI check broken - try to load class BundleEvent always fails
> --
>
> Key: COMPRESS-552
> URL: https://issues.apache.org/jira/browse/COMPRESS-552
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.20
>Reporter: Björn Michael
>Priority: Major
>
> There is a check for running in OSGI env. in {{ZstdUtils}}, {{BrotliUtils}},
> {{LZMAUtils}} and {{XZUtils}} like this:
> {code:java}
> static {
> cachedZstdAvailability = CachedAvailability.DONT_CACHE;
> try {
> Class.forName("org.osgi.framework.BundleEvent");
> } catch (final Exception ex) { // NOSONAR
> setCacheZstdAvailablity(true);
> }
> }
> {code}
> Loading the class {{org.osgi.framework.BundleEvent}} always fails because
> {{Import-Package}} directive for {{org.osgi.framework}} is missing in
> _MANIFEST.MF_. Or it requires another more sophisticated approaches
> ([https://stackoverflow.com/q/5879040/1061929]).
> Tested with Eclipse 4.14 (org.eclipse.osgi_3.15.100.v20191114-1701.jar)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-552) OSGI check broken - try to load class BundleEvent always fails
[
https://issues.apache.org/jira/browse/COMPRESS-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296822#comment-17296822
]
Stefan Bodewig commented on COMPRESS-552:
-
I'd prefer a solution that doesn't require a compile time dependency on OSGi.
Checking the classloader looks good, we could iterate the super interfaces and
classes until the classname matches without performing an instanceof check.
We've already got a single PAX Exam test which only asserts Compress can be
loaded as a bundle (this is easier to break, than I would have thought). I'll
try to figure out whether this can be used to verify a certain check works when
run inside of the bundle. We'll see.
> OSGI check broken - try to load class BundleEvent always fails
> --
>
> Key: COMPRESS-552
> URL: https://issues.apache.org/jira/browse/COMPRESS-552
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.20
>Reporter: Björn Michael
>Priority: Major
>
> There is a check for running in OSGI env. in {{ZstdUtils}}, {{BrotliUtils}},
> {{LZMAUtils}} and {{XZUtils}} like this:
> {code:java}
> static {
> cachedZstdAvailability = CachedAvailability.DONT_CACHE;
> try {
> Class.forName("org.osgi.framework.BundleEvent");
> } catch (final Exception ex) { // NOSONAR
> setCacheZstdAvailablity(true);
> }
> }
> {code}
> Loading the class {{org.osgi.framework.BundleEvent}} always fails because
> {{Import-Package}} directive for {{org.osgi.framework}} is missing in
> _MANIFEST.MF_. Or it requires another more sophisticated approaches
> ([https://stackoverflow.com/q/5879040/1061929]).
> Tested with Eclipse 4.14 (org.eclipse.osgi_3.15.100.v20191114-1701.jar)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-569) OutOfMemoryError on a crafted tar file
[ https://issues.apache.org/jira/browse/COMPRESS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-569. - Fix Version/s: 1.21 Resolution: Fixed this specific issue can be closed now. > OutOfMemoryError on a crafted tar file > -- > > Key: COMPRESS-569 > URL: https://issues.apache.org/jira/browse/COMPRESS-569 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.21 >Reporter: Fabian Meumertzheim >Priority: Blocker > Fix For: 1.21 > > Attachments: TarFileTimeout.java, timeout.tar > > > Apache Commons Compress at commit > https://github.com/apache/commons-compress/commit/1b7528fbd6295a3958daf1b1114621ee5e40e83c > throws an OutOfMemoryError after consuming ~5 minutes of CPU on my > machine on a crafted tar archive that is less than a KiB in size: > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at java.base/sun.nio.cs.UTF_8.newDecoder(UTF_8.java:70) > at > org.apache.commons.compress.archivers.zip.NioZipEncoding.newDecoder(NioZipEncoding.java:182) > at > org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:135) > at > org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:311) > at > org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:275) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1550) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:554) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:570) > at > org.apache.commons.compress.archivers.tar.TarFile.getNextTarEntry(TarFile.java:250) > at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:211) > at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:94) > at TarFileTimeout.main(TarFileTimeout.java:22) > I attached both the tar file and a Java reproducer for this issue. > Citing Stefan Bodewig's analysis of this issue: > Your archive contains an entry with a claimed size of -512 bytes. When > TarFile reads entries it tries to skip the content of th entry as it is > only interested in meta data on a first scan and does so by positioning > the input stream right after the data of the current entry. In this case > it positions it 512 bytes backwards right at the start of the current > entry's meta data again. This leads to an infinite loop that reads a new > entry, stores the meta data, repositions the stream and starts over > again. Over time the list of collected meta data eats up all available > memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-569) OutOfMemoryError on a crafted tar file
[ https://issues.apache.org/jira/browse/COMPRESS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296819#comment-17296819 ] Stefan Bodewig commented on COMPRESS-569: - Thank you. I've added the test in 5aa752ab > OutOfMemoryError on a crafted tar file > -- > > Key: COMPRESS-569 > URL: https://issues.apache.org/jira/browse/COMPRESS-569 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.21 >Reporter: Fabian Meumertzheim >Priority: Blocker > Attachments: TarFileTimeout.java, timeout.tar > > > Apache Commons Compress at commit > https://github.com/apache/commons-compress/commit/1b7528fbd6295a3958daf1b1114621ee5e40e83c > throws an OutOfMemoryError after consuming ~5 minutes of CPU on my > machine on a crafted tar archive that is less than a KiB in size: > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at java.base/sun.nio.cs.UTF_8.newDecoder(UTF_8.java:70) > at > org.apache.commons.compress.archivers.zip.NioZipEncoding.newDecoder(NioZipEncoding.java:182) > at > org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:135) > at > org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:311) > at > org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:275) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1550) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:554) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:570) > at > org.apache.commons.compress.archivers.tar.TarFile.getNextTarEntry(TarFile.java:250) > at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:211) > at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:94) > at TarFileTimeout.main(TarFileTimeout.java:22) > I attached both the tar file and a Java reproducer for this issue. > Citing Stefan Bodewig's analysis of this issue: > Your archive contains an entry with a claimed size of -512 bytes. When > TarFile reads entries it tries to skip the content of th entry as it is > only interested in meta data on a first scan and does so by positioning > the input stream right after the data of the current entry. In this case > it positions it 512 bytes backwards right at the start of the current > entry's meta data again. This leads to an infinite loop that reads a new > entry, stores the meta data, repositions the stream and starts over > again. Over time the list of collected meta data eats up all available > memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-539) TarArchiveInputStream allocates a lot of memory when iterating through an archive
[
https://issues.apache.org/jira/browse/COMPRESS-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296643#comment-17296643
]
Stefan Bodewig commented on COMPRESS-539:
-
[~peterlee] do you still intend to change anything or can we close this?
> TarArchiveInputStream allocates a lot of memory when iterating through an
> archive
> -
>
> Key: COMPRESS-539
> URL: https://issues.apache.org/jira/browse/COMPRESS-539
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Assignee: Peter Lee
>Priority: Major
> Attachments: Don't_call_InputStream#skip.patch,
> Reuse_recordBuffer.patch, image-2020-06-21-10-58-07-917.png,
> image-2020-06-21-10-58-43-255.png, image-2020-06-21-10-59-10-825.png,
> image-2020-07-05-22-10-07-402.png, image-2020-07-05-22-11-25-526.png,
> image-2020-07-05-22-32-15-131.png, image-2020-07-05-22-32-31-511.png
>
>
> I iterated through the linux source tar and noticed some unneeded
> allocations happen without extracting any data.
> Reproducing code
> {code:java}
> File tarFile = new File("linux-5.7.1.tar");
> try (TarArchiveInputStream in = new
> TarArchiveInputStream(Files.newInputStream(tarFile.toPath( {
> TarArchiveEntry entry;
> while ((entry = in.getNextTarEntry()) != null) {
> }
> }
> {code}
> The measurement was done on Java 11.0.7 with the Java Flight Recorder.
> Options used:
> -XX:StartFlightRecording=settings=profile,filename=allocations.jfr
> Baseline with the current master implementation:
> Estimated TLAB allocation: 293MiB
> !image-2020-06-21-10-58-07-917.png!
> 1. IOUtils.skip -> input.skip(numToSkip)
> This delegates in my test scenario to the InputStream.skip implementation
> which allocates a new byte[] for every invocation. By simply commenting out
> the while loop which calls the skip method the estimated TLAB allocation
> drops to 164MiB (-129MiB).
> !image-2020-06-21-10-58-43-255.png!
> Commenting out the skip call does not seem to be the best solution but it
> was quick for me to see how much memory can be saved. Also no unit tests
> where failing for me.
> 2. TarArchiveInputStream.readRecord
> For every read of the record a new byte[] is created. Since the record size
> does not change the byte[] can be reused and created when instantiating the
> TarStream. This optimization is already present in the
> TarArchiveOutputStream. Reusing the buffer reduces the estimated TLAB
> allocations further to 128MiB (-36MiB).
> !image-2020-06-21-10-59-10-825.png!
> I attached the patches I used so the results can be verified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-540) Random access on Tar archive
[ https://issues.apache.org/jira/browse/COMPRESS-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296642#comment-17296642 ] Stefan Bodewig commented on COMPRESS-540: - I think this is complete by now, isn't it? > Random access on Tar archive > > > Key: COMPRESS-540 > URL: https://issues.apache.org/jira/browse/COMPRESS-540 > Project: Commons Compress > Issue Type: Improvement >Reporter: Robin Schimpf >Priority: Major > Fix For: 1.21 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The TarArchiveInputStream only provides sequential access. If only a small > amount of files from the archive is needed large amount of data in the input > stream needs to be skipped. > Therefore I was working on a implementation to provide random access to > TarFiles equal to the ZipFile api. The basic idea behind the implementation > is the following > * Random access is backed by a SeekableByteChannel > * Read all headers of the tar file and save the place to the data of every > header > * User can request an input stream for any entry in the archive multiple > times -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (COMPRESS-540) Random access on Tar archive
[ https://issues.apache.org/jira/browse/COMPRESS-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig updated COMPRESS-540: Fix Version/s: 1.21 > Random access on Tar archive > > > Key: COMPRESS-540 > URL: https://issues.apache.org/jira/browse/COMPRESS-540 > Project: Commons Compress > Issue Type: Improvement >Reporter: Robin Schimpf >Priority: Major > Fix For: 1.21 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The TarArchiveInputStream only provides sequential access. If only a small > amount of files from the archive is needed large amount of data in the input > stream needs to be skipped. > Therefore I was working on a implementation to provide random access to > TarFiles equal to the ZipFile api. The basic idea behind the implementation > is the following > * Random access is backed by a SeekableByteChannel > * Read all headers of the tar file and save the place to the data of every > header > * User can request an input stream for any entry in the archive multiple > times -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-545) Decompression fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/COMPRESS-545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296641#comment-17296641
]
Stefan Bodewig commented on COMPRESS-545:
-
I just confirmed both archives lead to IOExceptions by now.
> Decompression fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: COMPRESS-545
> URL: https://issues.apache.org/jira/browse/COMPRESS-545
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
> Attachments: ArrayIndexOutOfBoundsException1.zip,
> ArrayIndexOutOfBoundsException2.zip
>
>
> This Kotlin code fails with exception(ArrayIndexOutOfBoundsException1.zip is
> in the attachments)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 30
> out of bounds for length 29Exception in thread "main"
> java.lang.ArrayIndexOutOfBoundsException: Index 30 out of bounds for length
> 29 at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.decodeNext(HuffmanDecoder.java:321)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.read(HuffmanDecoder.java:307)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:152)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396) at
> java.base/java.io.InputStream.readAllBytes(InputStream.java:333) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt:86) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> import java.io.File
> fun main() {
> val bytes = File("ArrayIndexOutOfBoundsException1.zip").readBytes()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> Expected some other exception as IOException is the only declared.
>
> This Kotlin code fails with exception(ArrayIndexOutOfBoundsException2.zip is
> in the attachments)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index -1
> out of bounds for length 8192Exception in thread "main"
> java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length
> 8192 at
> org.apache.commons.compress.compressors.lzw.LZWInputStream.expandCodeToOutputStack(LZWInputStream.java:232)
> at
> org.apache.commons.compress.archivers.zip.UnshrinkingInputStream.decompressNextSymbol(UnshrinkingInputStream.java:124)
> at
> org.apache.commons.compress.compressors.lzw.LZWInputStream.read(LZWInputStream.java:80)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396) at
> java.base/java.io.InputStream.readAllBytes(InputStream.java:333) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt:86) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> import java.io.File
> fun main() {
> val bytes = File("ArrayIndexOutOfBoundsException2.zip").readBytes()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> Expected some other exception as IOException is the only declared.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (COMPRESS-545) Decompression fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/COMPRESS-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-545:
Fix Version/s: 1.21
> Decompression fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: COMPRESS-545
> URL: https://issues.apache.org/jira/browse/COMPRESS-545
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
> Attachments: ArrayIndexOutOfBoundsException1.zip,
> ArrayIndexOutOfBoundsException2.zip
>
>
> This Kotlin code fails with exception(ArrayIndexOutOfBoundsException1.zip is
> in the attachments)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 30
> out of bounds for length 29Exception in thread "main"
> java.lang.ArrayIndexOutOfBoundsException: Index 30 out of bounds for length
> 29 at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.decodeNext(HuffmanDecoder.java:321)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.read(HuffmanDecoder.java:307)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:152)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396) at
> java.base/java.io.InputStream.readAllBytes(InputStream.java:333) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt:86) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> import java.io.File
> fun main() {
> val bytes = File("ArrayIndexOutOfBoundsException1.zip").readBytes()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> Expected some other exception as IOException is the only declared.
>
> This Kotlin code fails with exception(ArrayIndexOutOfBoundsException2.zip is
> in the attachments)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index -1
> out of bounds for length 8192Exception in thread "main"
> java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length
> 8192 at
> org.apache.commons.compress.compressors.lzw.LZWInputStream.expandCodeToOutputStack(LZWInputStream.java:232)
> at
> org.apache.commons.compress.archivers.zip.UnshrinkingInputStream.decompressNextSymbol(UnshrinkingInputStream.java:124)
> at
> org.apache.commons.compress.compressors.lzw.LZWInputStream.read(LZWInputStream.java:80)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396) at
> java.base/java.io.InputStream.readAllBytes(InputStream.java:333) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt:86) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> import java.io.File
> fun main() {
> val bytes = File("ArrayIndexOutOfBoundsException2.zip").readBytes()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> Expected some other exception as IOException is the only declared.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-545) Decompression fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/COMPRESS-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-545.
-
Resolution: Fixed
> Decompression fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: COMPRESS-545
> URL: https://issues.apache.org/jira/browse/COMPRESS-545
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
> Attachments: ArrayIndexOutOfBoundsException1.zip,
> ArrayIndexOutOfBoundsException2.zip
>
>
> This Kotlin code fails with exception(ArrayIndexOutOfBoundsException1.zip is
> in the attachments)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 30
> out of bounds for length 29Exception in thread "main"
> java.lang.ArrayIndexOutOfBoundsException: Index 30 out of bounds for length
> 29 at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.decodeNext(HuffmanDecoder.java:321)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.read(HuffmanDecoder.java:307)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:152)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396) at
> java.base/java.io.InputStream.readAllBytes(InputStream.java:333) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt:86) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> import java.io.File
> fun main() {
> val bytes = File("ArrayIndexOutOfBoundsException1.zip").readBytes()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> Expected some other exception as IOException is the only declared.
>
> This Kotlin code fails with exception(ArrayIndexOutOfBoundsException2.zip is
> in the attachments)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index -1
> out of bounds for length 8192Exception in thread "main"
> java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length
> 8192 at
> org.apache.commons.compress.compressors.lzw.LZWInputStream.expandCodeToOutputStack(LZWInputStream.java:232)
> at
> org.apache.commons.compress.archivers.zip.UnshrinkingInputStream.decompressNextSymbol(UnshrinkingInputStream.java:124)
> at
> org.apache.commons.compress.compressors.lzw.LZWInputStream.read(LZWInputStream.java:80)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396) at
> java.base/java.io.InputStream.readAllBytes(InputStream.java:333) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt:86) at
> kotlinx.fuzzer.tests.apache.zip.ApacheZipTestKt.main(ApacheZipTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> import java.io.File
> fun main() {
> val bytes = File("ArrayIndexOutOfBoundsException2.zip").readBytes()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> Expected some other exception as IOException is the only declared.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-549) Inconsistency with latest PKZip standard
[
https://issues.apache.org/jira/browse/COMPRESS-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296640#comment-17296640
]
Stefan Bodewig commented on COMPRESS-549:
-
If you encounter any file that combines a data descriptor with the stored
method, then your only safe way is to use {{ZipFile}} - either the java.util
variety or ours. Anything using a non-seekable stream can run into situations
where reading fails.
> Inconsistency with latest PKZip standard
>
>
> Key: COMPRESS-549
> URL: https://issues.apache.org/jira/browse/COMPRESS-549
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Sebastian Kürten
>Priority: Major
>
> I came across some Zip archives that cannot be read using
> ZipArchiveInputStream. Investigating the issue, I found that
> java.util.zip.ZipInputStream from the JDK shows the same behavior while
> java.util.zip.ZipFile seems to do fine. For java.util.zip.ZipInputStream, the
> issue has been reported here:
> [https://bugs.openjdk.java.net/browse/JDK-8143613] and an example file is
> provided. I copied the testing file into a repository for testing. Here's the
> example file:
> [https://github.com/sebkur/test-zip-impls/blob/master/src/test/java/de/topobyte/zip/tests/jdk8143613/TestData.java]
> and the test that fails reading that data using ZipArchiveInputStream:
> [https://github.com/sebkur/test-zip-impls/blob/master/src/test/java/de/topobyte/zip/tests/jdk8143613/TestCommonsZipInputStream.java]
>
>
> If this file is indeed a ZIP archive consistent with the PKZip spec, I think
> commons-compress does not work according to the spec. I would appreciate if
> someone could look into this and verify if this is indeed a bug in
> commons-compress. Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-551) GeneralPurposeBit has no compression level information
[ https://issues.apache.org/jira/browse/COMPRESS-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296639#comment-17296639 ] Stefan Bodewig commented on COMPRESS-551: - [~kaladhar] do you by chance know which level corresponds to which set of bits for deflate? Level 6 is normal, 9 probably is maximum compression and 1probably is "super fast", but which one is fast? > GeneralPurposeBit has no compression level information > -- > > Key: COMPRESS-551 > URL: https://issues.apache.org/jira/browse/COMPRESS-551 > Project: Commons Compress > Issue Type: Bug >Reporter: Kaladhar Reddy Mummadi >Priority: Major > > [https://github.com/apache/commons-compress/blob/rel/1.20/src/main/java/org/apache/commons/compress/archivers/zip/GeneralPurposeBit.java] > When _putArchiveEntry_ is called and _localHeaders_ are generated, > *Compression level* information is always *0.* > According to section 4.4.4 in > [https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT] *Bit 1 and 2* > should consists of compression level info. > But zips generated has level `0` always even when I change level using > *setLevel* API on *ZipArchiveOutputStream* > Expected behavior is to have level information in local headers of an entry. > > Compressing of data works fine, just that correct level is not baked into > localHeader. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-552) OSGI check broken - try to load class BundleEvent always fails
[
https://issues.apache.org/jira/browse/COMPRESS-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296638#comment-17296638
]
Stefan Bodewig commented on COMPRESS-552:
-
I'm afraid the accepted answer of the StackOverflow question would suffer from
the same problem ({{FrameworkUtil}} probably isn't available either). So the
only way out would be a Bundle-Activator?
> OSGI check broken - try to load class BundleEvent always fails
> --
>
> Key: COMPRESS-552
> URL: https://issues.apache.org/jira/browse/COMPRESS-552
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.20
>Reporter: Björn Michael
>Priority: Major
>
> There is a check for running in OSGI env. in {{ZstdUtils}}, {{BrotliUtils}},
> {{LZMAUtils}} and {{XZUtils}} like this:
> {code:java}
> static {
> cachedZstdAvailability = CachedAvailability.DONT_CACHE;
> try {
> Class.forName("org.osgi.framework.BundleEvent");
> } catch (final Exception ex) { // NOSONAR
> setCacheZstdAvailablity(true);
> }
> }
> {code}
> Loading the class {{org.osgi.framework.BundleEvent}} always fails because
> {{Import-Package}} directive for {{org.osgi.framework}} is missing in
> _MANIFEST.MF_. Or it requires another more sophisticated approach
> (https://stackoverflow.com/q/5879040/1061929).
> Tested with Eclipse 4.14 (org.eclipse.osgi_3.15.100.v20191114-1701.jar)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-557) Have some kind of reset() in LZ77Compressor
[ https://issues.apache.org/jira/browse/COMPRESS-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296634#comment-17296634 ] Stefan Bodewig commented on COMPRESS-557: - what would be the desired behavior? > Have some kind of reset() in LZ77Compressor > --- > > Key: COMPRESS-557 > URL: https://issues.apache.org/jira/browse/COMPRESS-557 > Project: Commons Compress > Issue Type: New Feature >Reporter: Peter Lee >Priority: Minor > > It would be useful if we have a _reset()_ in LZ77Compressor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (COMPRESS-567) IllegalArgumentException in ZipFile.positionAtCentralDirectory
[
https://issues.apache.org/jira/browse/COMPRESS-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-567:
Fix Version/s: 1.21
> IllegalArgumentException in ZipFile.positionAtCentralDirectory
> --
>
> Key: COMPRESS-567
> URL: https://issues.apache.org/jira/browse/COMPRESS-567
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Fabian Meumertzheim
>Priority: Major
> Fix For: 1.21
>
> Attachments: crash.zip
>
>
> The following snippet of code throws an undeclared IllegalArgumentException:
> {code:java}
> byte[] bytes = Base64.getDecoder().decode("UEsFBgAAAQD//1AAJP9QAA==");
> SeekableInMemoryByteChannel input = new SeekableInMemoryByteChannel(bytes);
> try {
> ZipFile file = new ZipFile(input);
> } catch (IOException ignored) {}
> {code}
> The stack trace is:
> {noformat}
> java.lang.IllegalArgumentException: Position has to be in range 0.. 2147483647
> at
> org.apache.commons.compress.utils.SeekableInMemoryByteChannel.position(SeekableInMemoryByteChannel.java:94)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory32(ZipFile.java:1128)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:1037)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:702)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:371)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:318)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:274)
> {noformat}
> I also attached the input as a ZIP file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-565:
Fix Version/s: 1.21
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Fix For: 1.21
>
> Attachments: commons-compress-1.21-SNAPSHOT.jar,
> image-2021-02-20-15-51-21-747.png
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-569) OutOfMemoryError on a crafted tar file
[
https://issues.apache.org/jira/browse/COMPRESS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296633#comment-17296633
]
Stefan Bodewig commented on COMPRESS-569:
-
your test now quickly throws an {{IOException}} with commit 5c5f8a89
Interestingly for dump archives some of our valid test cases seem to contain
negative sizes, I'll have to look into this more closely.
> OutOfMemoryError on a crafted tar file
> --
>
> Key: COMPRESS-569
> URL: https://issues.apache.org/jira/browse/COMPRESS-569
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.21
>Reporter: Fabian Meumertzheim
>Priority: Blocker
> Attachments: TarFileTimeout.java, timeout.tar
>
>
> Apache Commons Compress at commit
> https://github.com/apache/commons-compress/commit/1b7528fbd6295a3958daf1b1114621ee5e40e83c
> throws an OutOfMemoryError after consuming ~5 minutes of CPU on my
> machine on a crafted tar archive that is less than a KiB in size:
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.base/sun.nio.cs.UTF_8.newDecoder(UTF_8.java:70)
> at
> org.apache.commons.compress.archivers.zip.NioZipEncoding.newDecoder(NioZipEncoding.java:182)
> at
> org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:135)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:311)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:275)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1550)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:554)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:570)
> at
> org.apache.commons.compress.archivers.tar.TarFile.getNextTarEntry(TarFile.java:250)
> at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:211)
> at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:94)
> at TarFileTimeout.main(TarFileTimeout.java:22)
> I attached both the tar file and a Java reproducer for this issue.
> Citing Stefan Bodewig's analysis of this issue:
> Your archive contains an entry with a claimed size of -512 bytes. When
> TarFile reads entries it tries to skip the content of th entry as it is
> only interested in meta data on a first scan and does so by positioning
> the input stream right after the data of the current entry. In this case
> it positions it 512 bytes backwards right at the start of the current
> entry's meta data again. This leads to an infinite loop that reads a new
> entry, stores the meta data, repositions the stream and starts over
> again. Over time the list of collected meta data eats up all available
> memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-569) OutOfMemoryError on a crafted tar file
[ https://issues.apache.org/jira/browse/COMPRESS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296614#comment-17296614 ] Stefan Bodewig commented on COMPRESS-569: - [~Meumertzheim] can I add your tar archive as a test case to our source tree? > OutOfMemoryError on a crafted tar file > -- > > Key: COMPRESS-569 > URL: https://issues.apache.org/jira/browse/COMPRESS-569 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.21 >Reporter: Fabian Meumertzheim >Priority: Blocker > Attachments: TarFileTimeout.java, timeout.tar > > > Apache Commons Compress at commit > https://github.com/apache/commons-compress/commit/1b7528fbd6295a3958daf1b1114621ee5e40e83c > throws an OutOfMemoryError after consuming ~5 minutes of CPU on my > machine on a crafted tar archive that is less than a KiB in size: > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at java.base/sun.nio.cs.UTF_8.newDecoder(UTF_8.java:70) > at > org.apache.commons.compress.archivers.zip.NioZipEncoding.newDecoder(NioZipEncoding.java:182) > at > org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:135) > at > org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:311) > at > org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:275) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1550) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:554) > at > org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:570) > at > org.apache.commons.compress.archivers.tar.TarFile.getNextTarEntry(TarFile.java:250) > at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:211) > at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:94) > at TarFileTimeout.main(TarFileTimeout.java:22) > I attached both the tar file and a Java reproducer for this issue. > Citing Stefan Bodewig's analysis of this issue: > Your archive contains an entry with a claimed size of -512 bytes. When > TarFile reads entries it tries to skip the content of th entry as it is > only interested in meta data on a first scan and does so by positioning > the input stream right after the data of the current entry. In this case > it positions it 512 bytes backwards right at the start of the current > entry's meta data again. This leads to an infinite loop that reads a new > entry, stores the meta data, repositions the stream and starts over > again. Over time the list of collected meta data eats up all available > memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-569) OutOfMemoryError on a crafted tar file
[
https://issues.apache.org/jira/browse/COMPRESS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296604#comment-17296604
]
Stefan Bodewig commented on COMPRESS-569:
-
I see two issues here:
* we should never accept a negative size for a {{TarArchiveEntry}} we read.
Instead we should throw an exception because of a broken archive. I'll take the
opportunity to verify we do deal with negative sizes (and potentially other
numeric values that are supposed to fall into a specific range) for the other
archive types as well.
* we should check we are not moving backwards in {{TarFile}} - this one can be
fixed quickly.
> OutOfMemoryError on a crafted tar file
> --
>
> Key: COMPRESS-569
> URL: https://issues.apache.org/jira/browse/COMPRESS-569
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.21
>Reporter: Fabian Meumertzheim
>Priority: Blocker
> Attachments: TarFileTimeout.java, timeout.tar
>
>
> Apache Commons Compress at commit
> https://github.com/apache/commons-compress/commit/1b7528fbd6295a3958daf1b1114621ee5e40e83c
> throws an OutOfMemoryError after consuming ~5 minutes of CPU on my
> machine on a crafted tar archive that is less than a KiB in size:
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.base/sun.nio.cs.UTF_8.newDecoder(UTF_8.java:70)
> at
> org.apache.commons.compress.archivers.zip.NioZipEncoding.newDecoder(NioZipEncoding.java:182)
> at
> org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:135)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:311)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:275)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1550)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:554)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:570)
> at
> org.apache.commons.compress.archivers.tar.TarFile.getNextTarEntry(TarFile.java:250)
> at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:211)
> at org.apache.commons.compress.archivers.tar.TarFile.(TarFile.java:94)
> at TarFileTimeout.main(TarFileTimeout.java:22)
> I attached both the tar file and a Java reproducer for this issue.
> Citing Stefan Bodewig's analysis of this issue:
> Your archive contains an entry with a claimed size of -512 bytes. When
> TarFile reads entries it tries to skip the content of th entry as it is
> only interested in meta data on a first scan and does so by positioning
> the input stream right after the data of the current entry. In this case
> it positions it 512 bytes backwards right at the start of the current
> entry's meta data again. This leads to an infinite loop that reads a new
> entry, stores the meta data, repositions the stream and starts over
> again. Over time the list of collected meta data eats up all available
> memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-567) IllegalArgumentException in ZipFile.positionAtCentralDirectory
[
https://issues.apache.org/jira/browse/COMPRESS-567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292369#comment-17292369
]
Stefan Bodewig commented on COMPRESS-567:
-
[~peterlee] I've gone a step further and made all our internally used stream
classes throw {{IOException}} with commit 32509ee9 - I'd appreciate a second
set of eyes.
> IllegalArgumentException in ZipFile.positionAtCentralDirectory
> --
>
> Key: COMPRESS-567
> URL: https://issues.apache.org/jira/browse/COMPRESS-567
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Fabian Meumertzheim
>Priority: Major
> Attachments: crash.zip
>
>
> The following snippet of code throws an undeclared IllegalArgumentException:
> {code:java}
> byte[] bytes = Base64.getDecoder().decode("UEsFBgAAAQD//1AAJP9QAA==");
> SeekableInMemoryByteChannel input = new SeekableInMemoryByteChannel(bytes);
> try {
> ZipFile file = new ZipFile(input);
> } catch (IOException ignored) {}
> {code}
> The stack trace is:
> {noformat}
> java.lang.IllegalArgumentException: Position has to be in range 0.. 2147483647
> at
> org.apache.commons.compress.utils.SeekableInMemoryByteChannel.position(SeekableInMemoryByteChannel.java:94)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory32(ZipFile.java:1128)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:1037)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:702)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:371)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:318)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:274)
> {noformat}
> I also attached the input as a ZIP file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-567) IllegalArgumentException in ZipFile.positionAtCentralDirectory
[
https://issues.apache.org/jira/browse/COMPRESS-567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291742#comment-17291742
]
Stefan Bodewig commented on COMPRESS-567:
-
most likely we should be throwing {{IOExceptions}} in {{SeekableByteChannel}}'s
{{position}} - and fix {{truncate}} at the same time.
> IllegalArgumentException in ZipFile.positionAtCentralDirectory
> --
>
> Key: COMPRESS-567
> URL: https://issues.apache.org/jira/browse/COMPRESS-567
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Fabian Meumertzheim
>Priority: Major
> Attachments: crash.zip
>
>
> The following snippet of code throws an undeclared IllegalArgumentException:
> {code:java}
> byte[] bytes = Base64.getDecoder().decode("UEsFBgAAAQD//1AAJP9QAA==");
> SeekableInMemoryByteChannel input = new SeekableInMemoryByteChannel(bytes);
> try {
> ZipFile file = new ZipFile(input);
> } catch (IOException ignored) {}
> {code}
> The stack trace is:
> {noformat}
> java.lang.IllegalArgumentException: Position has to be in range 0.. 2147483647
> at
> org.apache.commons.compress.utils.SeekableInMemoryByteChannel.position(SeekableInMemoryByteChannel.java:94)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory32(ZipFile.java:1128)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:1037)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:702)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:371)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:318)
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:274)
> {noformat}
> I also attached the input as a ZIP file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287878#comment-17287878
]
Stefan Bodewig commented on COMPRESS-565:
-
[~peterlee]no problem if you want to port it, but I can do so as well.
What I would do:
* introduce another constant to Zip64Mode that works like ALWAYS for the LFH
and AS_NEEDED for CFH. This would be a compromise that would allow Powershell
(and likely Excel) to read the archive.
* ask [~evgenii.bovykin] to try a new snapshot using that new value rather
than ALWAYS.
* document that at the zip page on the site
* not change any defaults WRT Zip64Mode
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: commons-compress-1.21-SNAPSHOT.jar,
> image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287800#comment-17287800
]
Stefan Bodewig edited comment on COMPRESS-565 at 2/20/21, 9:24 PM:
---
[~evgenii.bovykin] could you please give the jar I've attached a try? This is
identical to current git master but handles the ZIP64 entry of the central
directory in a different way - this wouldn't be sensitive to the unicode extra
field, but I really suspect the lfh offset inside of the central directory to
be what causes the problem.
If oyu prefer t build the jar yourself, it is the branch
compress-565-experiment I've just pushed.
was (Author: bodewig):
[~evgenii.bovykin] could you please give the jar I've attached a try? This is
identical to current git master but handles the ZIP64 entry of the central
directory in a different way - this wouldn't be sensitive to the unicode extra
field, but I really suspect the lfh offset inside of the central directory to
be what causes the problem.
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: commons-compress-1.21-SNAPSHOT.jar,
> image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287800#comment-17287800
]
Stefan Bodewig edited comment on COMPRESS-565 at 2/20/21, 9:24 PM:
---
[~evgenii.bovykin] could you please give the jar I've attached a try? This is
identical to current git master but handles the ZIP64 entry of the central
directory in a different way - this wouldn't be sensitive to the unicode extra
field, but I really suspect the lfh offset inside of the central directory to
be what causes the problem.
If you prefer to build the jar yourself, it is the branch
compress-565-experiment I've just pushed.
was (Author: bodewig):
[~evgenii.bovykin] could you please give the jar I've attached a try? This is
identical to current git master but handles the ZIP64 entry of the central
directory in a different way - this wouldn't be sensitive to the unicode extra
field, but I really suspect the lfh offset inside of the central directory to
be what causes the problem.
If oyu prefer t build the jar yourself, it is the branch
compress-565-experiment I've just pushed.
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: commons-compress-1.21-SNAPSHOT.jar,
> image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287800#comment-17287800
]
Stefan Bodewig commented on COMPRESS-565:
-
[~evgenii.bovykin] could you please give the jar I've attached a try? This is
identical to current git master but handles the ZIP64 entry of the central
directory in a different way - this wouldn't be sensitive to the unicode extra
field, but I really suspect the lfh offset inside of the central directory to
be what causes the problem.
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: commons-compress-1.21-SNAPSHOT.jar,
> image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-565:
Attachment: commons-compress-1.21-SNAPSHOT.jar
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: commons-compress-1.21-SNAPSHOT.jar,
> image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287771#comment-17287771
]
Stefan Bodewig commented on COMPRESS-565:
-
Too bad, thank you.
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287727#comment-17287727
]
Stefan Bodewig commented on COMPRESS-565:
-
If I read the command line options correctly, throwing in -bb3 as switch might
provide more extensive information.
{{7z t -bb3 ..}}.
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287718#comment-17287718
]
Stefan Bodewig commented on COMPRESS-565:
-
Does 7z on Windows provide you with more details? I'm not using Windows
normally, so checking would take some time ...
What we have changed between 1.9 and 1.20 is the handling of the "relative
offset of local header" inside the central directory entry - at least this is
where I see the created archives differ. This is one of the problems Excel has
had with our archives as well. To be honest I believe our implementation is
correct and Excel is wrong - this is why I'd really like to know what bothers
7z.
Finally are any of the consumers of your archive actually using the Unicode
extra fields? If not and if leaving this out fixes the problem then I'd just
drop it. But I must admit I'm not really sure how/why the unicode extra fields
changes the outcome in any way.
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287705#comment-17287705
]
Stefan Bodewig commented on COMPRESS-565:
-
just as a data-point, p7z 16 for Linux (the latest version available as apt
package) doesn't see any problems with the archive.
I've run Peter's test code in Evgenii's repo with both 1.20 and 1.9 and the
LFHs written for archive/input.bin are identical (except for the timestamps
contained within).
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287700#comment-17287700
]
Stefan Bodewig commented on COMPRESS-565:
-
Yes, it might be open source (actually it most probably leaves the archive
handling to the classes in .NET Framework or Core which may delegate to to
Windows native libraries on Windows and zlib on non-Windows platforms), but
that doesn't mean figuring out where the errors is would be easy ;)
Unfortunately the full error isn't really helpful, I was hoping it would say
what exactly it doesn't like about the local file header. Sorry for bothering
you.
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287682#comment-17287682
]
Stefan Bodewig commented on COMPRESS-565:
-
There are known incompatibilities with our Zip64 implementation and what Excel
expects, see https://issues.apache.org/jira/browse/COMPRESS-474 and I wouldn't
be surprised if PowerShell used the same code. But this should only cause
problems if the archive contains files smaller than 4GB and not be related to
Unicode Extra Fields at all.
Please include the exact error messages when reporting an interoperability
issue not just "error about corrupted headers". If a tool fails to extract
archives created by Commons Compress than it could be the fault of either side
- given the spec is vague enough and implementations differ in their
interpretation of it.
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-565) Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
[
https://issues.apache.org/jira/browse/COMPRESS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287676#comment-17287676
]
Stefan Bodewig commented on COMPRESS-565:
-
[~peterlee] can you post the warnings 7z emits here, please?
> Regression - Corrupted headers when using 64 bit ZipArchiveOutputStream
> ---
>
> Key: COMPRESS-565
> URL: https://issues.apache.org/jira/browse/COMPRESS-565
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Evgenii Bovykin
>Assignee: Peter Lee
>Priority: Major
> Attachments: image-2021-02-20-15-51-21-747.png
>
>
> We've recently updated commons-compress library from version 1.9 to 1.20 and
> now experiencing the problem that didn't occur before.
>
> When using ZipArchiveOutputStream to archive 5Gb file and setting the
> following fields
> {{output.setUseZip64(Zip64Mode.Always)}}
>
> {{output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)}}
> resulting archive contains corrupted headers.
> *Expand-Archive Powershell utility cannot extract the archive at all with the
> error about corrupted header. 7zip also complains about it, but can extract
> the archive.*
>
> The problem didn't appear when using library version 1.9.
>
> I've created a sample project that reproduces the error -
> [https://github.com/missingdays/commons-compress-example]
> Issue doesn't reproduce if you do any of the following:
>
> # Downgrade library to version 1.9
> # Remove
> output.setCreateUnicodeExtraFields(ZipArchiveOutputStream.UnicodeExtraFieldPolicy.ALWAYS)
> # Remove output.setUseZip64(Zip64Mode.Always) and zip smaller file (e.g. 1Gb)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-555) ZipArchiveInputStream should allow stored entries with data descriptor by default
[
https://issues.apache.org/jira/browse/COMPRESS-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194010#comment-17194010
]
Stefan Bodewig commented on COMPRESS-555:
-
Unfortunately trying to read STORED entries that use a data descriptor is
unreliable to say the least. It is very easy to do if you can read the central
directory at the end of the archive - and thus ZipFile handles them just fine,
but reading the archive as a stream is a very different issue.
The default right now will tell you "I don't think I can handle this entry" if
you use the {{canReadEntryData}} method. The non-default option will read
forward until it finds something that looks like the signature of the next ZIP
entry. This will completely break down if the STORED entry contains such a
sequence of bytes - ZIPs in ZIPs is the primary example for this (think WARs
containing JARs for example). In recent versions we'll try to verify the
claimed size we read from what we believe to be the data descriptor matches the
length we've read, but then you are faced with an IOException for reading an
entry that the stream claimed to be able to handle.
Personally I believe the option will lead to too much confusion to enable it by
default. I prefer to have users take the deliberate choice and realize what
they are signing up for. Even better they would find a way to make the initial
stream seekable and use Zipfile rather than ZipArchiveInputStream.
> ZipArchiveInputStream should allow stored entries with data descriptor by
> default
> -
>
> Key: COMPRESS-555
> URL: https://issues.apache.org/jira/browse/COMPRESS-555
> Project: Commons Compress
> Issue Type: Improvement
> Components: Archivers
>Affects Versions: 1.20
>Reporter: Trevor Bentley
>Priority: Major
> Fix For: 1.21
>
>
> We are currently using tika for text extraction which uses commons-compress
> for handling zips. Currently some sites are returning zips that have entries
> with stored data descriptors which fail to extract due to the
> ZipArchiveInputStream defaulting to false for
> 'allowStoredEntriesWithDataDescriptor'.
> Allowing the reading of stored entries on Zip archives should be enabled by
> default.
> PR: https://github.com/apache/commons-compress/pull/137
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-463) ZipArchiveInputStream should throw if stored entry ends prematurely
[
https://issues.apache.org/jira/browse/COMPRESS-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163635#comment-17163635
]
Stefan Bodewig commented on COMPRESS-463:
-
With Compress 1.7 the implementation for reading stored entries has changed and
introduced a flaky behavior where {{read}} would alternate between -1 and a
non-negative number returned which can throw {{InputStreamReader}} into an
infinite loop. Compress 1.6 and earlier would have returned -1 consistently and
thus not cause an infinite loop - but it would also not properly signal the
archive as corrupted and give you the false impression you'd have read the
entry completely.
> ZipArchiveInputStream should throw if stored entry ends prematurely
> ---
>
> Key: COMPRESS-463
> URL: https://issues.apache.org/jira/browse/COMPRESS-463
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.17
>Reporter: Stefan Bodewig
>Priority: Major
> Labels: zip
> Fix For: 1.18
>
>
> If the archive stream ends before the claimed size of a stored entry has been
> hit the stream returns -1 rather than throwing a "truncated archive"
> exception like it would do for deflated entries.
> Because of this it is impossible to detect a truncated stored entry.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-539) TarArchiveInputStream allocates a lot of memory when iterating through an archive
[
https://issues.apache.org/jira/browse/COMPRESS-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152153#comment-17152153
]
Stefan Bodewig commented on COMPRESS-539:
-
I'm afraid we've never run any measurements but relied on gut feeling.
The assumption here is that {{skip}} on a seekable stream should be faster if
skipping more than just a few bytes. I'm not sure how you have constructed the
example [~rschimpf] but I guess the entries are pretty small (a few kilobytes).
Our (well, my) expectation is that {{read}} in such a case might return cached
data as we've just read a tar header from the stream, but if the entry in
question was bigger - a few megabytes, maybe - then reading it should be
measurably slower than seeking ahead for the same amount of bytes.
> TarArchiveInputStream allocates a lot of memory when iterating through an
> archive
> -
>
> Key: COMPRESS-539
> URL: https://issues.apache.org/jira/browse/COMPRESS-539
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Assignee: Peter Lee
>Priority: Major
> Attachments: Don't_call_InputStream#skip.patch,
> Reuse_recordBuffer.patch, image-2020-06-21-10-58-07-917.png,
> image-2020-06-21-10-58-43-255.png, image-2020-06-21-10-59-10-825.png,
> image-2020-07-05-22-10-07-402.png, image-2020-07-05-22-11-25-526.png,
> image-2020-07-05-22-32-15-131.png, image-2020-07-05-22-32-31-511.png
>
>
> I iterated through the linux source tar and noticed some unneeded
> allocations happen without extracting any data.
> Reproducing code
> {code:java}
> File tarFile = new File("linux-5.7.1.tar");
> try (TarArchiveInputStream in = new
> TarArchiveInputStream(Files.newInputStream(tarFile.toPath( {
> TarArchiveEntry entry;
> while ((entry = in.getNextTarEntry()) != null) {
> }
> }
> {code}
> The measurement was done on Java 11.0.7 with the Java Flight Recorder.
> Options used:
> -XX:StartFlightRecording=settings=profile,filename=allocations.jfr
> Baseline with the current master implementation:
> Estimated TLAB allocation: 293MiB
> !image-2020-06-21-10-58-07-917.png!
> 1. IOUtils.skip -> input.skip(numToSkip)
> This delegates in my test scenario to the InputStream.skip implementation
> which allocates a new byte[] for every invocation. By simply commenting out
> the while loop which calls the skip method the estimated TLAB allocation
> drops to 164MiB (-129MiB).
> !image-2020-06-21-10-58-43-255.png!
> Commenting out the skip call does not seem to be the best solution but it
> was quick for me to see how much memory can be saved. Also no unit tests
> where failing for me.
> 2. TarArchiveInputStream.readRecord
> For every read of the record a new byte[] is created. Since the record size
> does not change the byte[] can be reused and created when instantiating the
> TarStream. This optimization is already present in the
> TarArchiveOutputStream. Reusing the buffer reduces the estimated TLAB
> allocations further to 128MiB (-36MiB).
> !image-2020-06-21-10-59-10-825.png!
> I attached the patches I used so the results can be verified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-539) TarArchiveInputStream allocates a lot of memory when iterating through an archive
[
https://issues.apache.org/jira/browse/COMPRESS-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151287#comment-17151287
]
Stefan Bodewig commented on COMPRESS-539:
-
I'm sorry we crossed with our actions mid-flight [~peterlee] . I'm afraid I
have to ask you to revert your change to {{IOUtils.skip}}.
> TarArchiveInputStream allocates a lot of memory when iterating through an
> archive
> -
>
> Key: COMPRESS-539
> URL: https://issues.apache.org/jira/browse/COMPRESS-539
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Assignee: Peter Lee
>Priority: Major
> Attachments: Don't_call_InputStream#skip.patch,
> Reuse_recordBuffer.patch, image-2020-06-21-10-58-07-917.png,
> image-2020-06-21-10-58-43-255.png, image-2020-06-21-10-59-10-825.png
>
>
> I iterated through the linux source tar and noticed some unneeded
> allocations happen without extracting any data.
> Reproducing code
> {code:java}
> File tarFile = new File("linux-5.7.1.tar");
> try (TarArchiveInputStream in = new
> TarArchiveInputStream(Files.newInputStream(tarFile.toPath( {
> TarArchiveEntry entry;
> while ((entry = in.getNextTarEntry()) != null) {
> }
> }
> {code}
> The measurement was done on Java 11.0.7 with the Java Flight Recorder.
> Options used:
> -XX:StartFlightRecording=settings=profile,filename=allocations.jfr
> Baseline with the current master implementation:
> Estimated TLAB allocation: 293MiB
> !image-2020-06-21-10-58-07-917.png!
> 1. IOUtils.skip -> input.skip(numToSkip)
> This delegates in my test scenario to the InputStream.skip implementation
> which allocates a new byte[] for every invocation. By simply commenting out
> the while loop which calls the skip method the estimated TLAB allocation
> drops to 164MiB (-129MiB).
> !image-2020-06-21-10-58-43-255.png!
> Commenting out the skip call does not seem to be the best solution but it
> was quick for me to see how much memory can be saved. Also no unit tests
> where failing for me.
> 2. TarArchiveInputStream.readRecord
> For every read of the record a new byte[] is created. Since the record size
> does not change the byte[] can be reused and created when instantiating the
> TarStream. This optimization is already present in the
> TarArchiveOutputStream. Reusing the buffer reduces the estimated TLAB
> allocations further to 128MiB (-36MiB).
> !image-2020-06-21-10-59-10-825.png!
> I attached the patches I used so the results can be verified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-539) TarArchiveInputStream allocates a lot of memory when iterating through an archive
[
https://issues.apache.org/jira/browse/COMPRESS-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151285#comment-17151285
]
Stefan Bodewig commented on COMPRESS-539:
-
COMPRESS-449 rather than 443, sorry.
> TarArchiveInputStream allocates a lot of memory when iterating through an
> archive
> -
>
> Key: COMPRESS-539
> URL: https://issues.apache.org/jira/browse/COMPRESS-539
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Assignee: Peter Lee
>Priority: Major
> Attachments: Don't_call_InputStream#skip.patch,
> Reuse_recordBuffer.patch, image-2020-06-21-10-58-07-917.png,
> image-2020-06-21-10-58-43-255.png, image-2020-06-21-10-59-10-825.png
>
>
> I iterated through the linux source tar and noticed some unneeded
> allocations happen without extracting any data.
> Reproducing code
> {code:java}
> File tarFile = new File("linux-5.7.1.tar");
> try (TarArchiveInputStream in = new
> TarArchiveInputStream(Files.newInputStream(tarFile.toPath( {
> TarArchiveEntry entry;
> while ((entry = in.getNextTarEntry()) != null) {
> }
> }
> {code}
> The measurement was done on Java 11.0.7 with the Java Flight Recorder.
> Options used:
> -XX:StartFlightRecording=settings=profile,filename=allocations.jfr
> Baseline with the current master implementation:
> Estimated TLAB allocation: 293MiB
> !image-2020-06-21-10-58-07-917.png!
> 1. IOUtils.skip -> input.skip(numToSkip)
> This delegates in my test scenario to the InputStream.skip implementation
> which allocates a new byte[] for every invocation. By simply commenting out
> the while loop which calls the skip method the estimated TLAB allocation
> drops to 164MiB (-129MiB).
> !image-2020-06-21-10-58-43-255.png!
> Commenting out the skip call does not seem to be the best solution but it
> was quick for me to see how much memory can be saved. Also no unit tests
> where failing for me.
> 2. TarArchiveInputStream.readRecord
> For every read of the record a new byte[] is created. Since the record size
> does not change the byte[] can be reused and created when instantiating the
> TarStream. This optimization is already present in the
> TarArchiveOutputStream. Reusing the buffer reduces the estimated TLAB
> allocations further to 128MiB (-36MiB).
> !image-2020-06-21-10-59-10-825.png!
> I attached the patches I used so the results can be verified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-539) TarArchiveInputStream allocates a lot of memory when iterating through an archive
[
https://issues.apache.org/jira/browse/COMPRESS-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151283#comment-17151283
]
Stefan Bodewig commented on COMPRESS-539:
-
not calling skip may reduce the amount of memory consumed but will slow things
down considerably as skip is usually far more efficient than read. In your case
I'd recommend wrapping your input stream into a {{SkipShieldingInputStream}}
instead, which will have the same effect as commenting out the skip calls.
Re-using the buffer in {{TarArchiveInputStream}} sounds right. No idea why this
is different from TarArchiveOutputStream where it seems we've used a shared
buffer since forever.
[~peterlee] the team has had this discussion around COMPRESS-443 - a while
before you joined - and back then explicitly decided we wanted to keep using
{{skip}} for performance reasons.
> TarArchiveInputStream allocates a lot of memory when iterating through an
> archive
> -
>
> Key: COMPRESS-539
> URL: https://issues.apache.org/jira/browse/COMPRESS-539
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Assignee: Peter Lee
>Priority: Major
> Attachments: Don't_call_InputStream#skip.patch,
> Reuse_recordBuffer.patch, image-2020-06-21-10-58-07-917.png,
> image-2020-06-21-10-58-43-255.png, image-2020-06-21-10-59-10-825.png
>
>
> I iterated through the linux source tar and noticed some unneeded
> allocations happen without extracting any data.
> Reproducing code
> {code:java}
> File tarFile = new File("linux-5.7.1.tar");
> try (TarArchiveInputStream in = new
> TarArchiveInputStream(Files.newInputStream(tarFile.toPath( {
> TarArchiveEntry entry;
> while ((entry = in.getNextTarEntry()) != null) {
> }
> }
> {code}
> The measurement was done on Java 11.0.7 with the Java Flight Recorder.
> Options used:
> -XX:StartFlightRecording=settings=profile,filename=allocations.jfr
> Baseline with the current master implementation:
> Estimated TLAB allocation: 293MiB
> !image-2020-06-21-10-58-07-917.png!
> 1. IOUtils.skip -> input.skip(numToSkip)
> This delegates in my test scenario to the InputStream.skip implementation
> which allocates a new byte[] for every invocation. By simply commenting out
> the while loop which calls the skip method the estimated TLAB allocation
> drops to 164MiB (-129MiB).
> !image-2020-06-21-10-58-43-255.png!
> Commenting out the skip call does not seem to be the best solution but it
> was quick for me to see how much memory can be saved. Also no unit tests
> where failing for me.
> 2. TarArchiveInputStream.readRecord
> For every read of the record a new byte[] is created. Since the record size
> does not change the byte[] can be reused and created when instantiating the
> TarStream. This optimization is already present in the
> TarArchiveOutputStream. Reusing the buffer reduces the estimated TLAB
> allocations further to 128MiB (-36MiB).
> !image-2020-06-21-10-59-10-825.png!
> I attached the patches I used so the results can be verified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-538) ZipEntry created using ZipArchiveOutputStream(File) constructor adds 'ZIP64 Extended Information Extra Field' even when zip64 is not required.
[
https://issues.apache.org/jira/browse/COMPRESS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151045#comment-17151045
]
Stefan Bodewig commented on COMPRESS-538:
-
We can not remove the extra header from the LFH as we have already written the
content of the entry to the stream after that. I think this is the best we can
do.
> ZipEntry created using ZipArchiveOutputStream(File) constructor adds 'ZIP64
> Extended Information Extra Field' even when zip64 is not required.
> --
>
> Key: COMPRESS-538
> URL: https://issues.apache.org/jira/browse/COMPRESS-538
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.8, 1.9, 1.20
>Reporter: Pritesh
>Assignee: Peter Lee
>Priority: Major
>
> When creating a zip file using
> [ZipArchiveOutputStream(File)|https://commons.apache.org/proper/commons-compress/apidocs/org/apache/commons/compress/archivers/zip/ZipArchiveOutputStream.html#ZipArchiveOutputStream-java.io.File-]
> constructor each entry get an extra `Zip64ExtendedInformationExtraField`
> header which shouldn't be there.
>
> The issue can be reproduced using below code sample
> {code:java}
> public static void main(String[] args) throws IOException {
> // Create file
> final File file = File.createTempFile("apache_", ".zip");
> try ( ZipArchiveOutputStream zos = new ZipArchiveOutputStream(file)) {
> zos.putArchiveEntry(new ZipArchiveEntry("TextFile1.txt"));
> zos.closeArchiveEntry();
> }
> // List Headers File
> final ZipFile zipFile = new ZipFile(file);
> final Enumeration entries =
> zipFile.getEntries();
> while (entries.hasMoreElements()) {
> final ZipArchiveEntry entry = entries.nextElement();
> for (ZipExtraField zipExtraField : entry.getExtraFields()) {
> System.out.println("EntryName:" + entry.getName() + " Header:
> "+ zipExtraField.getHeaderId().getValue()); // Header shouldn't be present.
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-507) Commons Compress cannot be built with JDK14 due to Pack200 removal
[ https://issues.apache.org/jira/browse/COMPRESS-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151043#comment-17151043 ] Stefan Bodewig commented on COMPRESS-507: - I'm not familiar with Proguard and am afraid we cannot offer a workaround - other than not using Java 14+ - right now. > Commons Compress cannot be built with JDK14 due to Pack200 removal > -- > > Key: COMPRESS-507 > URL: https://issues.apache.org/jira/browse/COMPRESS-507 > Project: Commons Compress > Issue Type: Bug > Components: Build, Compressors >Affects Versions: 1.20 >Reporter: Vincent Privat >Priority: Major > Fix For: 1.21 > > > Oracle dropped Pack200 in JDK14 as per > [https://bugs.openjdk.java.net/browse/JDK-8232022] > Someone expressed his will to maintain Pack200 here: > [https://mail.openjdk.java.net/pipermail/jdk-dev/2020-March/003979.html] > [https://github.com/pfirmstone/Pack200-ex-openjdk] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-507) Commons Compress cannot be built with JDK14 due to Pack200 removal
[ https://issues.apache.org/jira/browse/COMPRESS-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149329#comment-17149329 ] Stefan Bodewig commented on COMPRESS-507: - AFAICT there still is no replacement library. So one possible solution for 1.21 still may be to completely remove pack200 support (and maybe move it to a module distributed separately), but we haven't decided anything, yet. In fact the current master still doesn't compile with JDK 14+ I'm not sure I understand why this build time problem causes issues for you - assuming you use a binary of Commons Compress built on JDK earlier than 14 (the one in Maven Central has been built with JDK 8). If you try to use Pack200 on JDK14+ it would probably fail even if there was a replacement library as the existing pack200 code (before it was removed) didn't work with Java14 bytecode AFAIU. > Commons Compress cannot be built with JDK14 due to Pack200 removal > -- > > Key: COMPRESS-507 > URL: https://issues.apache.org/jira/browse/COMPRESS-507 > Project: Commons Compress > Issue Type: Bug > Components: Build, Compressors >Affects Versions: 1.20 >Reporter: Vincent Privat >Priority: Major > Fix For: 1.21 > > > Oracle dropped Pack200 in JDK14 as per > [https://bugs.openjdk.java.net/browse/JDK-8232022] > Someone expressed his will to maintain Pack200 here: > [https://mail.openjdk.java.net/pipermail/jdk-dev/2020-March/003979.html] > [https://github.com/pfirmstone/Pack200-ex-openjdk] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-537) SevenZipOutputFile with DEFLATE compression is incorrectly decompressed
[
https://issues.apache.org/jira/browse/COMPRESS-537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127586#comment-17127586
]
Stefan Bodewig commented on COMPRESS-537:
-
You are ignoring the return value of {{read}}. Juts like the {{read}} method of
{{InputStream}} there is no guarantee the read call will fill the array, it may
read less bytes and another call (or more of them) may be necessary until
{{read}} returns -1.
In your tests you seem to be lucky for some of the compression methods but you
must ensure you really read the contents completely by looping over the
{{read}} method.
> SevenZipOutputFile with DEFLATE compression is incorrectly decompressed
> ---
>
> Key: COMPRESS-537
> URL: https://issues.apache.org/jira/browse/COMPRESS-537
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Frotty
>Priority: Major
> Attachments: roxpack.atlas
>
>
> I create a 7zip archive and then load files from it. Using LZMA or BZIP2
> works, but when I change it to DEFLATE the files aren't loaded correctly,
> parts of them are missing.
> Example kotlin code:
>
> {code:java}
> public fun test() {
>// SAVE
>val target = File("target.7z")
>val sevenZOutput = SevenZOutputFile(target)
>sevenZOutput.setContentCompression(SevenZMethod.DEFLATE)
>val file = File("roxpack.atlas")
>val entry = sevenZOutput.createArchiveEntry(file, "roxpack.atlas")
>sevenZOutput.putArchiveEntry(entry)
>sevenZOutput.write(Files.readAllBytes(file.toPath()))
>sevenZOutput.closeArchiveEntry()
>sevenZOutput.close()
>// LOAD
>val s7f = SevenZFile(target)
>s7f.use {
> var nextEntry = it.nextEntry
> do {
> if (nextEntry.name == entry.name) {
> val content = ByteArray(nextEntry.size.toInt())
> s7f.read(content)
> // with DEFLATE content array contains only 0's at the end,
> content string is cut off
> println(String(content))
> return
> }
> nextEntry = it.nextEntry
> } while (nextEntry != null)
>}
> }
> {code}
> When I extract the file using 7zip GUI tool, the file is intact, so the
> problem lies within the decompression. And again, this all works fine with
> LZMA and BZIP2.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-536) maybe we can make it static
[ https://issues.apache.org/jira/browse/COMPRESS-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-536. - Fix Version/s: 1.21 Resolution: Fixed > maybe we can make it static > --- > > Key: COMPRESS-536 > URL: https://issues.apache.org/jira/browse/COMPRESS-536 > Project: Commons Compress > Issue Type: Improvement >Reporter: JIN XU >Priority: Minor > Fix For: 1.21 > > Time Spent: 0.5h > Remaining Estimate: 0h > > [https://github.com/apache/commons-compress/pull/107] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-535) maybe we can add a break here
[ https://issues.apache.org/jira/browse/COMPRESS-535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-535. - Fix Version/s: 1.21 Resolution: Fixed > maybe we can add a break here > -- > > Key: COMPRESS-535 > URL: https://issues.apache.org/jira/browse/COMPRESS-535 > Project: Commons Compress > Issue Type: Improvement >Reporter: JIN XU >Priority: Minor > Fix For: 1.21 > > Time Spent: 40m > Remaining Estimate: 0h > > [https://github.com/apache/commons-compress/pull/106] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-534) use StandardCharsets.
[ https://issues.apache.org/jira/browse/COMPRESS-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-534. - Fix Version/s: 1.21 Resolution: Fixed > use StandardCharsets. > - > > Key: COMPRESS-534 > URL: https://issues.apache.org/jira/browse/COMPRESS-534 > Project: Commons Compress > Issue Type: Improvement >Reporter: JIN XU >Priority: Minor > Fix For: 1.21 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://github.com/apache/commons-compress/pull/105] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-533) Refine javadoc and fix typos
[ https://issues.apache.org/jira/browse/COMPRESS-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Bodewig resolved COMPRESS-533. - Fix Version/s: 1.21 Resolution: Fixed > Refine javadoc and fix typos > > > Key: COMPRESS-533 > URL: https://issues.apache.org/jira/browse/COMPRESS-533 > Project: Commons Compress > Issue Type: Improvement >Reporter: JIN XU >Priority: Minor > Fix For: 1.21 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > [https://github.com/apache/commons-compress/pull/104] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (COMPRESS-529) Tar decompression fails with runtime exceptions
[
https://issues.apache.org/jira/browse/COMPRESS-529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-529.
-
Fix Version/s: 1.21
Assignee: (was: Peter Lee)
Resolution: Fixed
> Tar decompression fails with runtime exceptions
> ---
>
> Key: COMPRESS-529
> URL: https://issues.apache.org/jira/browse/COMPRESS-529
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
> Attachments: LongName.tar, NumberFormatException.tar, japicmp.html,
> japicmp.html
>
>
> This Kotlin code fails with exception(NumberFormatException.tar is in the
> attachment)
> Exception in thread "main" java.lang.NumberFormatException: For input string:
> "143266�921.098285006"
> at
> java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)
> at
> java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.base/java.lang.Double.parseDouble(Double.java:543)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.processPaxHeader(TarArchiveEntry.java:1161)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.updateEntryFromPaxHeaders(TarArchiveEntry.java:1093)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.applyPaxHeadersToCurrentEntry(TarArchiveInputStream.java:757)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.paxHeaders(TarArchiveInputStream.java:562)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:404)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:799)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:69)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("NumberFormatException.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> This Kotlin code fails with exception(LongName.tar is in the attachment)
> Exception in thread "main" java.lang.RuntimeException: file name 'asidhuasih a
> sdjn osdn
> sdvs ndv
> asdjbhasdb asb iasbfi basdigf basduio
> asdkhasjdhasd
> asdjkhasnjddjasjdas
> /?' is too long ( > 100 bytes)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.handleLongName(TarArchiveOutputStream.java:683)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:358)
> at ru.example.kotlinfuzzer.tests.MainKt.test(main.kt:20)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:8)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import org.apache.commons.compress.archivers.tar.TarArchiveEntry
> import java.io.File
> fun main() {
> test(File("LongName.tar"))
> }
> fun test(tar: File) {
> val (decompressed, name) =
> ArchiveStreamFactory().createArchiveInputStream("tar", tar.inputStream()).use
> { ais ->
> val entry = ais.nextEntry
> ais.readAllBytes() to entry.name
> }
> File.createTempFile("apache_", ".tar").also {
> ArchiveStreamFactory().createArchiveOutputStream("tar",
> it.outputStream()).use { aos ->
> val entry = TarArchiveEntry(name)
> entry.size = decompressed.size.toLong()
> aos.putArchiveEntry(entry)
> try {
> aos.write(decompressed)
> } finally {
> aos.closeArchiveEntry()
> }
> }
> }
> }
> {code}
>
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-532) Decompression fails with IllegalArgumentException(BinaryTree.java:72)
[
https://issues.apache.org/jira/browse/COMPRESS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-532.
-
Fix Version/s: 1.21
Resolution: Fixed
fixed with commit 6bbc7cb1
> Decompression fails with IllegalArgumentException(BinaryTree.java:72)
> -
>
> Key: COMPRESS-532
> URL: https://issues.apache.org/jira/browse/COMPRESS-532
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
> Attachments: IllegalArgumentException.zip
>
>
> This Kotlin code fails with exception(IllegalArgumentException.zip is in the
> attachments)
> Exception in thread "main" java.lang.IllegalArgumentException: Tree value at
> index 3 has already been assigned (-2)
> at
> org.apache.commons.compress.archivers.zip.BinaryTree.addLeaf(BinaryTree.java:72)
> at
> org.apache.commons.compress.archivers.zip.BinaryTree.addLeaf(BinaryTree.java:80)
> at
> org.apache.commons.compress.archivers.zip.BinaryTree.addLeaf(BinaryTree.java:80)
> at
> org.apache.commons.compress.archivers.zip.BinaryTree.decode(BinaryTree.java:199)
> at
> org.apache.commons.compress.archivers.zip.ExplodingInputStream.init(ExplodingInputStream.java:104)
> at
> org.apache.commons.compress.archivers.zip.ExplodingInputStream.fillBuffer(ExplodingInputStream.java:158)
> at
> org.apache.commons.compress.archivers.zip.ExplodingInputStream.read(ExplodingInputStream.java:119)
> at java.base/java.io.InputStream.read(InputStream.java:271)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at kotlinx.fuzzer.tests.MainKt.main(main.kt:10)
> at kotlinx.fuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("IllegalArgumentException.zip")
> ArchiveStreamFactory().createArchiveInputStream("zip",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> Expected some other exception as IOException is the only declared.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-531) Tar decompression fails with NullPointerException
[
https://issues.apache.org/jira/browse/COMPRESS-531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-531.
-
Fix Version/s: 1.21
Resolution: Fixed
> Tar decompression fails with NullPointerException
> -
>
> Key: COMPRESS-531
> URL: https://issues.apache.org/jira/browse/COMPRESS-531
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
> Attachments: NPE.tar
>
>
> This Kotlin code fails with exception(NPE.tar is in the attachment)
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.applyPaxHeadersToCurrentEntry(TarArchiveInputStream.java:757)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.paxHeaders(TarArchiveInputStream.java:562)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:404)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:799)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:20)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("NPE.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-531) Tar decompression fails with NullPointerException
[
https://issues.apache.org/jira/browse/COMPRESS-531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120514#comment-17120514
]
Stefan Bodewig commented on COMPRESS-531:
-
Thanks to the fix of COMPRESS-530 the archive now causes an {{IOException}}
anyway. With commit e4eccde3 I've also removed the cause of the NPE.
> Tar decompression fails with NullPointerException
> -
>
> Key: COMPRESS-531
> URL: https://issues.apache.org/jira/browse/COMPRESS-531
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Attachments: NPE.tar
>
>
> This Kotlin code fails with exception(NPE.tar is in the attachment)
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.applyPaxHeadersToCurrentEntry(TarArchiveInputStream.java:757)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.paxHeaders(TarArchiveInputStream.java:562)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:404)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:799)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:20)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("NPE.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-528) Tar decompression - Inconsistent entry size
[
https://issues.apache.org/jira/browse/COMPRESS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120508#comment-17120508
]
Stefan Bodewig commented on COMPRESS-528:
-
I'm not sure your expectation is correct. {{getSize}} returns what an entry's
metadata says, not necessarily what can be read. I'm not sure we are consistent
here across the formats. In some of them we throw exceptions if the actual
numbers don't match, in others - like tar - we silently accept it.
Changing something here could mean we'd start throwing exceptions for archives
where we have not done so, so far. Also one needs to think about what happens
if the content is never read like in code just invoking {{getNextEntry}}
repeatedly in order to list the contents, for example.
> Tar decompression - Inconsistent entry size
> ---
>
> Key: COMPRESS-528
> URL: https://issues.apache.org/jira/browse/COMPRESS-528
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Attachments: InconsistentSize.tar
>
>
> Expected that
> {{[getSize|https://commons.apache.org/proper/commons-compress/javadocs/api-1.20/org/apache/commons/compress/archivers/tar/TarArchiveEntry.html#getSize--]()
> }}returns the size of byte array read from ArchiveInputStream
> To reproduce run this Kotlin code(InconsistentSize.tar is in the attachment).
> It fails with exception
> Exception in thread "main" java.lang.IllegalStateException: 19 expected but 0
> found
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:82)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("InconsistentSize.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> val expected = ais.nextEntry.size
> val actual = ais.readAllBytes().size.toLong()
> check(expected == actual) { "$expected expected but $actual found" }
> }
> }
> {code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-530) Tar decompression fails with NegativeArraySizeException
[
https://issues.apache.org/jira/browse/COMPRESS-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-530.
-
Fix Version/s: 1.21
Resolution: Fixed
> Tar decompression fails with NegativeArraySizeException
> ---
>
> Key: COMPRESS-530
> URL: https://issues.apache.org/jira/browse/COMPRESS-530
> Project: Commons Compress
> Issue Type: Bug
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
> Attachments: NegativeArraySizeException.tar
>
>
> This Kotlin code fails with exception(NegativeArraySizeException.tar is in
> the attachment)
> Exception in thread "main" java.lang.NegativeArraySizeException: -5332783
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.parsePaxHeaders(TarArchiveInputStream.java:703)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.paxHeaders(TarArchiveInputStream.java:555)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:404)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:799)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:31)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("NegativeArraySizeException.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-529) Tar decompression fails with runtime exceptions
[
https://issues.apache.org/jira/browse/COMPRESS-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120506#comment-17120506
]
Stefan Bodewig commented on COMPRESS-529:
-
At least the two of us share the same misunderstanding, then :)
If anything is wrong with a stream we read, we should always try to turn this
into an {{IOException}}. This is something we don't always do, obviously, as
[~zuevmaxim] 's issues show. When writing things are a bit different as we can
expect the user to only hand us valid inputs IMHO.
The second issue here must expect errors as this is what
{{TarArchiveOutputStream}} does for file names that are too long or numbers
that are too big unless you tell it otherwise - and this is documented.
I agree we should turn the exceptions into {{IllegalArgumentException}}s - and
document them at the public APIs that may throw them. Any client code that
catches {{RuntimeException}}s now will automatically catch
{{IllegalArgumentException}} as well, so we are on the safe side.
> Tar decompression fails with runtime exceptions
> ---
>
> Key: COMPRESS-529
> URL: https://issues.apache.org/jira/browse/COMPRESS-529
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Attachments: LongName.tar, NumberFormatException.tar, japicmp.html,
> japicmp.html
>
>
> This Kotlin code fails with exception(NumberFormatException.tar is in the
> attachment)
> Exception in thread "main" java.lang.NumberFormatException: For input string:
> "143266�921.098285006"
> at
> java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)
> at
> java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.base/java.lang.Double.parseDouble(Double.java:543)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.processPaxHeader(TarArchiveEntry.java:1161)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.updateEntryFromPaxHeaders(TarArchiveEntry.java:1093)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.applyPaxHeadersToCurrentEntry(TarArchiveInputStream.java:757)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.paxHeaders(TarArchiveInputStream.java:562)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:404)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:799)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:69)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("NumberFormatException.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> This Kotlin code fails with exception(LongName.tar is in the attachment)
> Exception in thread "main" java.lang.RuntimeException: file name 'asidhuasih a
> sdjn osdn
> sdvs ndv
> asdjbhasdb asb iasbfi basdigf basduio
> asdkhasjdhasd
> asdjkhasnjddjasjdas
> /?' is too long ( > 100 bytes)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.handleLongName(TarArchiveOutputStream.java:683)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:358)
> at ru.example.kotlinfuzzer.tests.MainKt.test(main.kt:20)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:8)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import org.apache.commons.compress.archivers.tar.TarArchiveEntry
> import java.io.File
> fun main() {
> test(File("LongName.tar"))
> }
> fun test(tar: File) {
> val (decompressed, name) =
> ArchiveStreamFactory().createArchiveInputStream("tar", tar.inputStream()).use
> { ais ->
> val entry = ais.nextEntry
> ais.readAllBytes() to entry.name
> }
> File.createTempFile("apache_", ".tar").also {
> ArchiveStreamFactory().createArchiveOutputStream("tar",
> it.outputStream()).use { aos ->
> val entry = TarArchiveEntry(name)
> entry.size = decompressed.size.toLong()
> aos.putArchiveEntry(entry)
> try {
> aos.write(decompressed)
> } finally {
> aos.closeArchiveEntry()
> }
> }
> }
> }
> {code}
>
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-529) Tar decompression fails with runtime exceptions
[
https://issues.apache.org/jira/browse/COMPRESS-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119333#comment-17119333
]
Stefan Bodewig commented on COMPRESS-529:
-
The point of this bug report is that it happens when you read an archive. The
exception occurs in {{getNextEntry}} . Here the user expects an {{IOException}}
and probably catches it, but will not catch a {{NumberFormatException}}. So
while wrapping the NFE into an IOE may not provide useful information, it will
make the user code hande the exception and use a controlled path of exception.
This is a benefit in itself.
> Tar decompression fails with runtime exceptions
> ---
>
> Key: COMPRESS-529
> URL: https://issues.apache.org/jira/browse/COMPRESS-529
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Attachments: LongName.tar, NumberFormatException.tar, japicmp.html,
> japicmp.html
>
>
> This Kotlin code fails with exception(NumberFormatException.tar is in the
> attachment)
> Exception in thread "main" java.lang.NumberFormatException: For input string:
> "143266�921.098285006"
> at
> java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)
> at
> java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.base/java.lang.Double.parseDouble(Double.java:543)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.processPaxHeader(TarArchiveEntry.java:1161)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.updateEntryFromPaxHeaders(TarArchiveEntry.java:1093)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.applyPaxHeadersToCurrentEntry(TarArchiveInputStream.java:757)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.paxHeaders(TarArchiveInputStream.java:562)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:404)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:799)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:69)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("NumberFormatException.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> This Kotlin code fails with exception(LongName.tar is in the attachment)
> Exception in thread "main" java.lang.RuntimeException: file name 'asidhuasih a
> sdjn osdn
> sdvs ndv
> asdjbhasdb asb iasbfi basdigf basduio
> asdkhasjdhasd
> asdjkhasnjddjasjdas
> /?' is too long ( > 100 bytes)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.handleLongName(TarArchiveOutputStream.java:683)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:358)
> at ru.example.kotlinfuzzer.tests.MainKt.test(main.kt:20)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:8)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import org.apache.commons.compress.archivers.tar.TarArchiveEntry
> import java.io.File
> fun main() {
> test(File("LongName.tar"))
> }
> fun test(tar: File) {
> val (decompressed, name) =
> ArchiveStreamFactory().createArchiveInputStream("tar", tar.inputStream()).use
> { ais ->
> val entry = ais.nextEntry
> ais.readAllBytes() to entry.name
> }
> File.createTempFile("apache_", ".tar").also {
> ArchiveStreamFactory().createArchiveOutputStream("tar",
> it.outputStream()).use { aos ->
> val entry = TarArchiveEntry(name)
> entry.size = decompressed.size.toLong()
> aos.putArchiveEntry(entry)
> try {
> aos.write(decompressed)
> } finally {
> aos.closeArchiveEntry()
> }
> }
> }
> }
> {code}
>
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-529) Tar decompression fails with runtime exceptions
[
https://issues.apache.org/jira/browse/COMPRESS-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116590#comment-17116590
]
Stefan Bodewig commented on COMPRESS-529:
-
{{LONGFILE_ERROR}} is the default, we may need to document more properly what
kind of error will be thrown.
> Tar decompression fails with runtime exceptions
> ---
>
> Key: COMPRESS-529
> URL: https://issues.apache.org/jira/browse/COMPRESS-529
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Attachments: LongName.tar, NumberFormatException.tar
>
>
> This Kotlin code fails with exception(NumberFormatException.tar is in the
> attachment)
> Exception in thread "main" java.lang.NumberFormatException: For input string:
> "143266�921.098285006"
> at
> java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)
> at
> java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.base/java.lang.Double.parseDouble(Double.java:543)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.processPaxHeader(TarArchiveEntry.java:1161)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.updateEntryFromPaxHeaders(TarArchiveEntry.java:1093)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.applyPaxHeadersToCurrentEntry(TarArchiveInputStream.java:757)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.paxHeaders(TarArchiveInputStream.java:562)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:404)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:799)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:69)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("NumberFormatException.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> This Kotlin code fails with exception(LongName.tar is in the attachment)
> Exception in thread "main" java.lang.RuntimeException: file name 'asidhuasih a
> sdjn osdn
> sdvs ndv
> asdjbhasdb asb iasbfi basdigf basduio
> asdkhasjdhasd
> asdjkhasnjddjasjdas
> /?' is too long ( > 100 bytes)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.handleLongName(TarArchiveOutputStream.java:683)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:358)
> at ru.example.kotlinfuzzer.tests.MainKt.test(main.kt:20)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:8)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import org.apache.commons.compress.archivers.tar.TarArchiveEntry
> import java.io.File
> fun main() {
> test(File("LongName.tar"))
> }
> fun test(tar: File) {
> val (decompressed, name) =
> ArchiveStreamFactory().createArchiveInputStream("tar", tar.inputStream()).use
> { ais ->
> val entry = ais.nextEntry
> ais.readAllBytes() to entry.name
> }
> File.createTempFile("apache_", ".tar").also {
> ArchiveStreamFactory().createArchiveOutputStream("tar",
> it.outputStream()).use { aos ->
> val entry = TarArchiveEntry(name)
> entry.size = decompressed.size.toLong()
> aos.putArchiveEntry(entry)
> try {
> aos.write(decompressed)
> } finally {
> aos.closeArchiveEntry()
> }
> }
> }
> }
> {code}
>
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-529) Tar decompression fails with runtime exceptions
[
https://issues.apache.org/jira/browse/COMPRESS-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116586#comment-17116586
]
Stefan Bodewig commented on COMPRESS-529:
-
In this particular case I disagree. I think an {{IllegalArgumentException}} is
fine if you invoke one of the {{putArchiveEntry}} variants. We might need to
add more documentation, granted, but this is not an I/O error of any kind and
the caller is in full control of what it passes in as argument.
> Tar decompression fails with runtime exceptions
> ---
>
> Key: COMPRESS-529
> URL: https://issues.apache.org/jira/browse/COMPRESS-529
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Attachments: LongName.tar, NumberFormatException.tar
>
>
> This Kotlin code fails with exception(NumberFormatException.tar is in the
> attachment)
> Exception in thread "main" java.lang.NumberFormatException: For input string:
> "143266�921.098285006"
> at
> java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)
> at
> java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.base/java.lang.Double.parseDouble(Double.java:543)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.processPaxHeader(TarArchiveEntry.java:1161)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.updateEntryFromPaxHeaders(TarArchiveEntry.java:1093)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.applyPaxHeadersToCurrentEntry(TarArchiveInputStream.java:757)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.paxHeaders(TarArchiveInputStream.java:562)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:404)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:799)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:69)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.File
> fun main() {
> val tar = File("NumberFormatException.tar")
> ArchiveStreamFactory().createArchiveInputStream("tar",
> tar.inputStream()).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> This Kotlin code fails with exception(LongName.tar is in the attachment)
> Exception in thread "main" java.lang.RuntimeException: file name 'asidhuasih a
> sdjn osdn
> sdvs ndv
> asdjbhasdb asb iasbfi basdigf basduio
> asdkhasjdhasd
> asdjkhasnjddjasjdas
> /?' is too long ( > 100 bytes)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.handleLongName(TarArchiveOutputStream.java:683)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:358)
> at ru.example.kotlinfuzzer.tests.MainKt.test(main.kt:20)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:8)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import org.apache.commons.compress.archivers.tar.TarArchiveEntry
> import java.io.File
> fun main() {
> test(File("LongName.tar"))
> }
> fun test(tar: File) {
> val (decompressed, name) =
> ArchiveStreamFactory().createArchiveInputStream("tar", tar.inputStream()).use
> { ais ->
> val entry = ais.nextEntry
> ais.readAllBytes() to entry.name
> }
> File.createTempFile("apache_", ".tar").also {
> ArchiveStreamFactory().createArchiveOutputStream("tar",
> it.outputStream()).use { aos ->
> val entry = TarArchiveEntry(name)
> entry.size = decompressed.size.toLong()
> aos.putArchiveEntry(entry)
> try {
> aos.write(decompressed)
> } finally {
> aos.closeArchiveEntry()
> }
> }
> }
> }
> {code}
>
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-526) Decompression fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/COMPRESS-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-526.
-
Fix Version/s: 1.21
Resolution: Fixed
fixed with commit e176d4b7
> Decompression fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: COMPRESS-526
> URL: https://issues.apache.org/jira/browse/COMPRESS-526
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index -1
> out of bounds for length 65
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.getCodes(HuffmanDecoder.java:479)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.buildTree(HuffmanDecoder.java:453)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.access$800(HuffmanDecoder.java:31)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.(HuffmanDecoder.java:293)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:146)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:44)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x08, 0x00, 0x09, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x6f, 0x00, 0x00, 0x00,
> 0x61, 0x4a, 0x04, 0x04, 0x00, 0x00, 0xff, 0xff, 0x50, 0x53,
> 0x07, 0x08, 0x43, 0xbe, 0xb7, 0xe8, 0x07, 0x00, 0x00, 0x00,
> 0x01, 0x00, 0x00, 0x00, 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00,
> 0x08, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x01, 0x00, 0x00, 0x00, 0x62, 0x4a, 0x02, 0x04, 0x00, 0x00,
> 0xff, 0xff, 0x50, 0x4b, 0x7f, 0x08, 0xf9, 0xef, 0xbe, 0x71,
> 0x07, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x50, 0x4b,
> 0x03, 0x04, 0x14, 0x00, 0x08, 0x00, 0x08, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x63, 0x4a,
> 0x06, 0x04, 0x00, 0x00, 0xff, 0xff, 0x50, 0x4b, 0x07, 0x08,
> 0x01, 0xdf, 0xb9, 0x06, 0x07, 0x00, 0x00, 0x00, 0x01, 0x00,
> 0x00, 0x00, 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x08, 0x00,
> 0x08
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index -1
> out of bounds for length 32
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.decodeNext(HuffmanDecoder.java:328)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.read(HuffmanDecoder.java:307)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:152)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:35)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x00, 0x00, 0x09, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0xb6, 0xfe, 0x77, 0x4a, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x55, 0x54, 0x09, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x64, 0x55,
[jira] [Resolved] (COMPRESS-527) Decompression fails with NullPointerException
[
https://issues.apache.org/jira/browse/COMPRESS-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-527.
-
Fix Version/s: 1.21
Resolution: Fixed
fixed with commit e176d4b7
> Decompression fails with NullPointerException
> -
>
> Key: COMPRESS-527
> URL: https://issues.apache.org/jira/browse/COMPRESS-527
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.buildTree(HuffmanDecoder.java:466)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.access$800(HuffmanDecoder.java:31)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.(HuffmanDecoder.java:293)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:146)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:35)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x00, 0x00, 0x09, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0xb6, 0xfe, 0x77, 0x4a, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x55, 0x54, 0x09, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x64, 0x55, 0x75, 0x78, 0x0b, 0x00,
> 0x01, 0x04, 0x5c, 0xf9, 0x01, 0x00, 0x04, 0x88, 0x13, 0x00,
> 0x00, 0x1d, 0x8b, 0xc1, 0x0d, 0xc0, 0x30, 0x08, 0x03, 0xff,
> 0x99, 0xc2, 0xab, 0x81, 0x50, 0x1a, 0xa8, 0x44, 0x1e, 0x56,
> 0x30, 0x7f, 0x21, 0x1f, 0x5b, 0x3e, 0x9d, 0x85, 0x6e
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.buildTree(HuffmanDecoder.java:464)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.populateDynamicTables(HuffmanDecoder.java:389)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.readDynamicTables(HuffmanDecoder.java:187)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:145)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:32)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x00, 0x00, 0x09, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0xb6, 0xfe, 0x77, 0x4a, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x55, 0x54, 0x09, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x64, 0x55, 0x75, 0x78, 0x0b, 0x00,
> 0x01, 0x04, 0x5c, 0xf9, 0x01, 0x00, 0x04, 0x88, 0x13, 0x00,
> 0x00, 0x1d, 0x8b, 0xc1, 0x0d, 0xc0, 0x30, 0x88, 0x03, 0xff
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-525) Decompression fails with IllegalStateException: Unsupported compression: 3
[
https://issues.apache.org/jira/browse/COMPRESS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-525.
-
Resolution: Fixed
fixed with commit aa1bc872
> Decompression fails with IllegalStateException: Unsupported compression: 3
> --
>
> Key: COMPRESS-525
> URL: https://issues.apache.org/jira/browse/COMPRESS-525
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.IllegalStateException: Unsupported
> compression: 3
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:149)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:29)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x08, 0x00, 0x09, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x78, 0x00,
> 0x61, 0x4a, 0x04, 0x04, 0x00, 0x00, 0xff, 0xff, 0x50, 0x53,
> 0x07, 0x08, 0x43, 0xbe, 0xb7, 0xe8, 0x07, 0x00, 0x00, 0x00,
> 0x01, 0x00, 0x00, 0x00, 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00,
> 0x08, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x01, 0x00, 0x00, 0x00, 0x62, 0x4a, 0x02, 0x04, 0x00, 0x00,
> 0xff, 0xff, 0x50, 0x4b, 0x7f, 0x08, 0xf9, 0xef, 0xbe, 0x71,
> 0x07, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x50, 0x4b,
> 0x03, 0x04, 0x14, 0x00, 0x08, 0x00, 0x08, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x63, 0x4a,
> 0x06, 0x04, 0x00, 0x00, 0xff, 0xff, 0x50, 0x4b, 0x07, 0x08,
> 0x6f, 0xdf
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-520) Implement ZCompressorInputStreamTest without powermock
[
https://issues.apache.org/jira/browse/COMPRESS-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-520.
-
Resolution: Fixed
> Implement ZCompressorInputStreamTest without powermock
> --
>
> Key: COMPRESS-520
> URL: https://issues.apache.org/jira/browse/COMPRESS-520
> Project: Commons Compress
> Issue Type: Improvement
>Affects Versions: 1.20
>Reporter: Robin Schimpf
>Priority: Trivial
> Fix For: 1.21
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The ZCompressorInputStreamTest uses powermock to mock an empty enumeration.
> This prevents the test to be run on Java 9+.
> The same behaviour can be achieved by using {{Collections.emptyEnumeration}}
> and therefore the powermock dependency for the tests dropped.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-522) Decompression fails with IllegalStateException(Illegal LEN / NLEN values)
[
https://issues.apache.org/jira/browse/COMPRESS-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-522.
-
Resolution: Fixed
fixed with commit aa1bc872
> Decompression fails with IllegalStateException(Illegal LEN / NLEN values)
> -
>
> Key: COMPRESS-522
> URL: https://issues.apache.org/jira/browse/COMPRESS-522
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.IllegalStateException: Illegal LEN /
> NLEN values
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.switchToUncompressedState(HuffmanDecoder.java:174)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:139)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:17)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x08, 0x00, 0x09, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00,
> 0x61, 0x4a, 0x84, 0x02, 0x40, 0x00, 0x01, 0x00, 0xff, 0xff
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-521) Decompression fails with IllegalStateException
[
https://issues.apache.org/jira/browse/COMPRESS-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-521.
-
Resolution: Fixed
fixed with commit aa1bc872
> Decompression fails with IllegalStateException
> --
>
> Key: COMPRESS-521
> URL: https://issues.apache.org/jira/browse/COMPRESS-521
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.IllegalStateException: Attempt to read
> beyond memory: dist=9321
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$DecodingMemory.recordToBuffer(HuffmanDecoder.java:526)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.decodeNext(HuffmanDecoder.java:338)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder$HuffmanCodes.read(HuffmanDecoder.java:307)
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:152)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt:76)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x2e, 0x00, 0xb6, 0x00, 0x09, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0x00, 0xfe, 0x77, 0x63, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x55, 0x54, 0x0c, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x65, 0x55, 0x75, 0x78, 0x0b, 0x00,
> 0x01, 0x04, 0x5c, 0xf9, 0x01, 0x00, 0x04, 0x88, 0x13, 0x00,
> 0x00, 0x42, 0x5a, 0x68, 0x34
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (COMPRESS-525) Decompression fails with IllegalStateException: Unsupported compression: 3
[
https://issues.apache.org/jira/browse/COMPRESS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-525:
Fix Version/s: 1.21
> Decompression fails with IllegalStateException: Unsupported compression: 3
> --
>
> Key: COMPRESS-525
> URL: https://issues.apache.org/jira/browse/COMPRESS-525
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.IllegalStateException: Unsupported
> compression: 3
> at
> org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:149)
> at
> org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:84)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:493)
> at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
> at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:29)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x08, 0x00, 0x09, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x78, 0x00,
> 0x61, 0x4a, 0x04, 0x04, 0x00, 0x00, 0xff, 0xff, 0x50, 0x53,
> 0x07, 0x08, 0x43, 0xbe, 0xb7, 0xe8, 0x07, 0x00, 0x00, 0x00,
> 0x01, 0x00, 0x00, 0x00, 0x50, 0x4b, 0x03, 0x04, 0x14, 0x00,
> 0x08, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x01, 0x00, 0x00, 0x00, 0x62, 0x4a, 0x02, 0x04, 0x00, 0x00,
> 0xff, 0xff, 0x50, 0x4b, 0x7f, 0x08, 0xf9, 0xef, 0xbe, 0x71,
> 0x07, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x50, 0x4b,
> 0x03, 0x04, 0x14, 0x00, 0x08, 0x00, 0x08, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x63, 0x4a,
> 0x06, 0x04, 0x00, 0x00, 0xff, 0xff, 0x50, 0x4b, 0x07, 0x08,
> 0x6f, 0xdf
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-519) Decompression fails with IllegalArgumentException: fromIndex(50) > toIndex(49)
[
https://issues.apache.org/jira/browse/COMPRESS-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-519.
-
Resolution: Fixed
fixed with commit ea37bc16
> Decompression fails with IllegalArgumentException: fromIndex(50) > toIndex(49)
> --
>
> Key: COMPRESS-519
> URL: https://issues.apache.org/jira/browse/COMPRESS-519
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.IllegalArgumentException: fromIndex(50)
> > toIndex(49)
> at java.base/java.util.Arrays.rangeCheck(Arrays.java:116)
> at java.base/java.util.Arrays.fill(Arrays.java:3516)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.getAndMoveToFrontDecode(BZip2CompressorInputStream.java:670)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.initBlock(BZip2CompressorInputStream.java:332)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.(BZip2CompressorInputStream.java:136)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.(BZip2CompressorInputStream.java:113)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextZipEntry(ZipArchiveInputStream.java:368)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextEntry(ZipArchiveInputStream.java:435)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt:93)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x2e, 0x00, 0x00, 0x00, 0x0c, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0xb6, 0xfe, 0x77, 0x63, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x55, 0x54, 0x09, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x64, 0x55, 0x75, 0x78, 0x0b, 0x00,
> 0x01, 0x04, 0x5c, 0xf9, 0x01, 0x00, 0x04, 0x88, 0x13, 0x00,
> 0x00, 0x42, 0x5a, 0x68, 0x34, 0x31, 0x41, 0x59, 0x26, 0x53,
> 0x59, 0x62, 0xe4, 0x4f, 0x51, 0x80, 0x00, 0x0d, 0xd1, 0x80,
> 0x00, 0x10, 0x40, 0x00, 0x35, 0xf9, 0x8b, 0x00, 0x20, 0x00,
> 0x48, 0x89, 0xfa, 0x94, 0xf2, 0x9e, 0x29, 0xe8, 0xd2, 0x00,
> 0x00, 0x22, 0x00, 0x00, 0x00, 0x50, 0x4b, 0x03, 0x04, 0x14,
> 0x00, 0x08, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-523) Decompression fails with IllegalArgumentException
[
https://issues.apache.org/jira/browse/COMPRESS-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-523.
-
Resolution: Fixed
fixed with commit 2ea0e464
> Decompression fails with IllegalArgumentException
> -
>
> Key: COMPRESS-523
> URL: https://issues.apache.org/jira/browse/COMPRESS-523
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.IllegalArgumentException
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.realSkip(ZipArchiveInputStream.java:1120)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.skipRemainderOfArchive(ZipArchiveInputStream.java:1054)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextZipEntry(ZipArchiveInputStream.java:283)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextEntry(ZipArchiveInputStream.java:435)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt:15)
> at ru.example.kotlinfuzzer.tests.MainKt.main(main.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x01, 0x02, 0x14, 0x00, 0x14, 0x00, 0x08, 0x00,
> 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x43, 0xbe, 0x00, 0x00,
> 0x00, 0xb7, 0xe8, 0x07, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-518) Decompression fails with ClassCastException
[
https://issues.apache.org/jira/browse/COMPRESS-518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-518.
-
Resolution: Fixed
fixed with commit 1793167c
I've also fixed similar bugs (assuming a certain extra field must be of a given
type) in {{ZipFile}} and {{ZipArchiveOutputStream}} as well.
> Decompression fails with ClassCastException
> ---
>
> Key: COMPRESS-518
> URL: https://issues.apache.org/jira/browse/COMPRESS-518
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.ClassCastException: class
> org.apache.commons.compress.archivers.zip.UnrecognizedExtraField cannot be
> cast to class
> org.apache.commons.compress.archivers.zip.Zip64ExtendedInformationExtraField
> (org.apache.commons.compress.archivers.zip.UnrecognizedExtraField and
> org.apache.commons.compress.archivers.zip.Zip64ExtendedInformationExtraField
> are in unnamed module of loader 'app')
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.processZip64Extra(ZipArchiveInputStream.java:419)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextZipEntry(ZipArchiveInputStream.java:347)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextEntry(ZipArchiveInputStream.java:435)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt:88)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x2e, 0x00, 0x00, 0x00, 0x0c, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0xb6, 0xfe, 0x77, 0x63, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x01, 0x00, 0x09, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x64, 0x55, 0x75, 0x78, 0x0b, 0x00,
> 0x01, 0x04, 0x5c, 0xf9, 0x01, 0x00, 0x04, 0x88, 0x13, 0x00,
> 0x00
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-517) Decompression fails with NullPointerException
[
https://issues.apache.org/jira/browse/COMPRESS-517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-517.
-
Resolution: Fixed
fixed with commit f71d3ddd
> Decompression fails with NullPointerException
> -
>
> Key: COMPRESS-517
> URL: https://issues.apache.org/jira/browse/COMPRESS-517
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.commons.compress.archivers.zip.X5455_ExtendedTimestamp.getLocalFileDataData(X5455_ExtendedTimestamp.java:180)
> at
> org.apache.commons.compress.archivers.zip.ExtraFieldUtils.mergeLocalFileDataData(ExtraFieldUtils.java:250)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveEntry.setExtra(ZipArchiveEntry.java:691)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveEntry.setExtraFields(ZipArchiveEntry.java:415)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveEntry.mergeExtraFields(ZipArchiveEntry.java:893)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveEntry.setExtra(ZipArchiveEntry.java:676)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextZipEntry(ZipArchiveInputStream.java:341)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextEntry(ZipArchiveInputStream.java:435)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt:88)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt)
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x2e, 0x00, 0x00, 0x00, 0x0c, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0xb6, 0xfe, 0x77, 0x63, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x55, 0x54, 0x01, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x64, 0x55, 0x75, 0x78, 0x0b, 0x00,
> 0x01, 0x04, 0x5c, 0xf9, 0x01, 0x00, 0x04, 0x88, 0x13, 0x00,
> 0x00
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-516) Decompression fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/COMPRESS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig resolved COMPRESS-516.
-
Resolution: Fixed
fixed with commit 3497dcbc
> Decompression fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: COMPRESS-516
> URL: https://issues.apache.org/jira/browse/COMPRESS-516
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Array
> index out of range: 16777295
> at java.base/java.util.Arrays.rangeCheck(Arrays.java:123)
> at java.base/java.util.Arrays.fill(Arrays.java:3516)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.getAndMoveToFrontDecode(BZip2CompressorInputStream.java:670)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.initBlock(BZip2CompressorInputStream.java:332)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.(BZip2CompressorInputStream.java:136)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.(BZip2CompressorInputStream.java:113)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextZipEntry(ZipArchiveInputStream.java:368)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextEntry(ZipArchiveInputStream.java:435)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt:97)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt)
>
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x2e, 0x00, 0x00, 0x00, 0x0c, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0xb6, 0xfe, 0x77, 0x63, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x55, 0x54, 0x09, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x64, 0x55, 0x75, 0x78, 0x0b, 0x00,
> 0x01, 0x04, 0x5c, 0xf9, 0x01, 0x00, 0x04, 0x88, 0x13, 0x00,
> 0x00, 0x42, 0x5a, 0x68, 0x34, 0x31, 0x41, 0x59, 0x26, 0x53,
> 0x59, 0x62, 0xe4, 0x4f, 0x51, 0x00, 0x00, 0x0d, 0xd1, 0x80,
> 0x00, 0x10, 0x40, 0x00, 0x35, 0xf9, 0x8b, 0x00, 0x20, 0x00,
> 0x48, 0x89, 0xfa, 0x94, 0xf2, 0x9e, 0x29, 0xe8, 0xd2, 0x11,
> 0x8a, 0x4f, 0x53, 0x34, 0x0f, 0x51, 0x7a, 0xed, 0x86, 0x65,
> 0xd6, 0xed, 0x61, 0xee, 0x68, 0x89, 0x48, 0x7d, 0x07, 0x71,
> 0x92, 0x2a, 0x50, 0x60, 0x04, 0x95, 0x61, 0x35, 0x47, 0x73,
> 0x31, 0x29, 0xc2, 0xdd, 0x5e, 0xc7, 0x4a, 0x15, 0x14, 0x32,
> 0x4c, 0xda, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-516) Decompression fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/COMPRESS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114870#comment-17114870
]
Stefan Bodewig commented on COMPRESS-516:
-
Thanks, this is very similar to COMPRESS-424 and back then I only looked for
indexed access to the internal arrays and overlooked the {{fill}} case.
> Decompression fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: COMPRESS-516
> URL: https://issues.apache.org/jira/browse/COMPRESS-516
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.20
>Reporter: Maksim Zuev
>Priority: Major
> Fix For: 1.21
>
>
> This Kotlin code fails with exception
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Array
> index out of range: 16777295
> at java.base/java.util.Arrays.rangeCheck(Arrays.java:123)
> at java.base/java.util.Arrays.fill(Arrays.java:3516)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.getAndMoveToFrontDecode(BZip2CompressorInputStream.java:670)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.initBlock(BZip2CompressorInputStream.java:332)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.(BZip2CompressorInputStream.java:136)
> at
> org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.(BZip2CompressorInputStream.java:113)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextZipEntry(ZipArchiveInputStream.java:368)
> at
> org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.getNextEntry(ZipArchiveInputStream.java:435)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt:97)
> at ru.example.kotlinfuzzer.tests.CompressTestKt.main(CompressTest.kt)
>
> {code:java}
> import org.apache.commons.compress.archivers.ArchiveStreamFactory
> import java.io.ByteArrayInputStream
> fun main() {
> val bytes = listOf(
> 0x50, 0x4b, 0x03, 0x04, 0x2e, 0x00, 0x00, 0x00, 0x0c, 0x00,
> 0x84, 0xb6, 0xba, 0x46, 0x72, 0xb6, 0xfe, 0x77, 0x63, 0x00,
> 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x03, 0x00, 0x1c, 0x00,
> 0x62, 0x62, 0x62, 0x55, 0x54, 0x09, 0x00, 0x03, 0xe7, 0xce,
> 0x64, 0x55, 0xf3, 0xce, 0x64, 0x55, 0x75, 0x78, 0x0b, 0x00,
> 0x01, 0x04, 0x5c, 0xf9, 0x01, 0x00, 0x04, 0x88, 0x13, 0x00,
> 0x00, 0x42, 0x5a, 0x68, 0x34, 0x31, 0x41, 0x59, 0x26, 0x53,
> 0x59, 0x62, 0xe4, 0x4f, 0x51, 0x00, 0x00, 0x0d, 0xd1, 0x80,
> 0x00, 0x10, 0x40, 0x00, 0x35, 0xf9, 0x8b, 0x00, 0x20, 0x00,
> 0x48, 0x89, 0xfa, 0x94, 0xf2, 0x9e, 0x29, 0xe8, 0xd2, 0x11,
> 0x8a, 0x4f, 0x53, 0x34, 0x0f, 0x51, 0x7a, 0xed, 0x86, 0x65,
> 0xd6, 0xed, 0x61, 0xee, 0x68, 0x89, 0x48, 0x7d, 0x07, 0x71,
> 0x92, 0x2a, 0x50, 0x60, 0x04, 0x95, 0x61, 0x35, 0x47, 0x73,
> 0x31, 0x29, 0xc2, 0xdd, 0x5e, 0xc7, 0x4a, 0x15, 0x14, 0x32,
> 0x4c, 0xda, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> 0x00
> ).map { it.toByte() }.toByteArray()
> val input = ByteArrayInputStream(bytes)
> ArchiveStreamFactory().createArchiveInputStream("zip", input).use { ais ->
> ais.nextEntry
> ais.readAllBytes()
> }
> }
> {code}
> IOException expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-514) SevenZFile fails with encoded header over 2GiB
[
https://issues.apache.org/jira/browse/COMPRESS-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114860#comment-17114860
]
Stefan Bodewig commented on COMPRESS-514:
-
There are a few places where the 7z format says a certain value is a UINT64 and
we store it inside of a Java long at best. Even if we fix this particular case
in some way there will be more problems lurking (that I hope we all catch
before they cause ArrayIndexOutOufBoundsExceptions or similar things). Because
of this I'd be fine with listing the known limitations.
If we decide to deal with this specific problem then I'd prefer a solution that
doesn't penalize the normal case too much and doesn't hide problems we can find
early with the existing code. As long as we detect a bad CRC inside of
SevenZFile's constructor, your option 3 sounds reasonable. Nobody would see and
use the "bad" data, or am I overlooking something?
> SevenZFile fails with encoded header over 2GiB
> --
>
> Key: COMPRESS-514
> URL: https://issues.apache.org/jira/browse/COMPRESS-514
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.20
>Reporter: A Kelday
>Priority: Minor
> Attachments: HeaderChannelBuffer.java
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When reading what some may call a large encrypted 7zip file (1.2TB with 22
> million files), the read fails at the header stage with the trace below. Is
> this within the spec? I've written some code to handle it, because I did
> actually need to extract the file in java. If that's of any use I can provide
> it (it's a naive wrapper that just pages in a buffer at a time).
>
> {code:java}
> Exception in thread "main" java.io.IOException: Cannot handle
> unpackSize241696
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.assertFitsIntoInt(SevenZFile.java:1523)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.readEncodedHeader(SevenZFile.java:622)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.initializeArchive(SevenZFile.java:532)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:468)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:337)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:129)
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.(SevenZFile.java:116)
> {code}
> 7zip itself can also open it (and display/extract etc.), here are the stats:
>
>
> {code:java}
> Size: 2 489 903 580 875
> Packed Size: 1 349 110 308 832
> Folders: 40 005
> Files: 22 073 957
> CRC: E26F6A96
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-524) Mailing list addresses are out of date
[ https://issues.apache.org/jira/browse/COMPRESS-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114808#comment-17114808 ] Stefan Bodewig commented on COMPRESS-524: - I believe we inherit this from the parent POM and I really don't know enough about the site plugin to know whether this has been introduced via a changed version of the site-plugin or the parent POM's configuration should be any different. If there is something that needs to be changed in the parent POM then the correct Jira component would be COMMONSSITE. > Mailing list addresses are out of date > -- > > Key: COMPRESS-524 > URL: https://issues.apache.org/jira/browse/COMPRESS-524 > Project: Commons Compress > Issue Type: Bug >Reporter: Elliotte Rusty Harold >Priority: Major > > Spot checking it looks like the links on > [https://commons.apache.org/proper/commons-compress/mailing-lists.html] are > all 404 -- This message was sent by Atlassian Jira (v8.3.4#803005)
