Re: [PR] CLI-326: make option.getKey() public [commons-cli]
Claudenw commented on PR #239: URL: https://github.com/apache/commons-cli/pull/239#issuecomment-1970524259 at a conference today. Will try to get to this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Split Commons Compress into multiple artifacts [commons-compress]
ebourg commented on PR #490: URL: https://github.com/apache/commons-compress/pull/490#issuecomment-1970172265 I like this idea. If we follow this path I'd push further and fully modularize commons-compress, with a separate module for each compression and archive format. That may be easier to split the component in two groups with new names, for example commons-archive-* and commons-compression-*. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Split Commons Compress into multiple artifacts [commons-compress]
ppkarwasz opened a new pull request, #490:
URL: https://github.com/apache/commons-compress/pull/490
This is proof-of-concept of the possibility of splitting Commons Compress
into artifacts **without** optional dependencies, while maintaining backward
compatibility.
The Commons Compress code is split into:
- `commons-compress-brotli`: an artifact that includes Brotli support and
with a non-optional dependency on `org.brotli:dec`,
- `commons-compress-core`: the old Commons Compress with one entire package
and the dependency on `org.brotli:dec` removed,
- `common-compress`: an almost empty artifact, whose purpose is to provide
backward compatibility. It depends on the other artifacts and contains a module
descriptor that allows JPMS users that declares `requires
org.apache.commons.compress;` to also read the other modules:
```java
module org.apache.commons.compress {
requires transitive org.apache.commons.compress.brotli;
requires transitive org.apache.commons.compress.core;
}
```
- `commons-compress-bom`: a BOM artifact to help users prevent having
multiple copies of the same classes on the classpath (or module path). Without
using a BOM for dependency management, users could end up having
`commons-compress-core` version 1.27.0 and an older `commons-compress` as
transitive dependency. This would lead to a duplication of classes on the
classpath.
- `commons-compress-parent`: a technical POM to manage dependency versions
and plugin configurations in a single place. We might use Maven Flatten Plugin
to remove this artifact from the published ones.
The PoC is fully functional, although the POMs might need some small
adjustments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821892#comment-17821892
]
Gary D. Gregory commented on COMPRESS-666:
--
BTW how big is the new file that has an issue?
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although TarArchiveInputStream is marked as not thread safe, I am not reusing
> objects here. Those are in fact separate objects, presumably all with their
> own position tracking info.
>
> The stacktrace here looks like:
> {code:java}
> Caused by: java.io.IOException: Corrupted TAR archive.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1480)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:534)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:431)
> at
> Caused by: java.lang.IllegalArgumentException: Invalid byte 100 at offset 0
> in '' len=12
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:516)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:540)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1496)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1478)
> ... 7 more
> {code}
> That code shows that occasionally the header is wrong (the tar entry name
> contains gibberish bits) which makes me think that `getNextTarEntry()` can be
> faulty.
>
> Running that code with commons compress 1.25.0 works as expected. So it's
> probably something added since November. Note that this is something related
> to parallelism - using an executor service with a single thread doesn't
> suffer from the same error. The tgz to decompress doesn't really matter - you
> can use a manually created one worth a few KBs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821879#comment-17821879
]
Gary D. Gregory commented on COMPRESS-666:
--
Hello [~cosmin79]
What happens if you change the {{TarArchiveInputStream}} constructor to use the
smallest buffers:
{code:java}
TarArchiveInputStream tarInputStream = new TarArchiveInputStream(new
BufferedInputStream(new GZIPInputStream(inputStream)),
TarConstants.DEFAULT_RCDSIZE, TarConstants.DEFAULT_RCDSIZE)
{code}
?
Note that our constants are misnamed. If you look at the tar.h file, you'll see
a constant called {{BLOCKSIZE}} defined as {{{}512{}}}, but in our code we call
this {{DEFAULT_RCDSIZE}} and we have a {{DEFAULT_BLKSIZE}} defined as
{{{}DEFAULT_RCDSIZE * 20{}}}. See {{{}TarConstants{}}}.
What might be happening is that this test surfaces (maybe) an issue that's been
there all along and that we are only now seeing. Either due to the above
confusion or perhaps something like this: Under more load, the underlying input
stream serves fewer (or more) bytes into buffers and this issue only happens
when input is "chunked" into unlucky groups. Recall that when you ask a given
number of bytes from an input stream, you are not guaranteed to get all bytes
in one call. All this code is very much dependent on reading groups of bytes of
certain size, skipping some regions, and then expecting data to be layed out a
specific way.
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although TarArchiveInputStream is marked as not thread safe, I am not reusing
> objects here. Those are in fact separate objects, presumably all with their
> own position tracking info.
>
> The stacktrace here looks like:
> {code:java}
> Caused by: java.io.IOException: Corrupted TAR archive.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1480)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:534)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:431)
> at
> Caused by: java.lang.IllegalArgumentException: Invalid byte 100 at offset 0
> in '' len=12
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:516)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:540)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1496)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1478)
> ... 7 more
> {code}
> That code shows that occasionally the header is wrong (the tar entry name
> contains gibberish bits) which makes me think that `getNextTarEntry()` can be
> faulty.
>
>
[jira] [Commented] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821865#comment-17821865
]
Cosmin Carabet commented on COMPRESS-666:
-
I did actually make the error happen again on a much larger tar. It happens
more intermittently, but still happens. I don't think this fixes it.
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although TarArchiveInputStream is marked as not thread safe, I am not reusing
> objects here. Those are in fact separate objects, presumably all with their
> own position tracking info.
>
> The stacktrace here looks like:
> {code:java}
> Caused by: java.io.IOException: Corrupted TAR archive.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1480)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:534)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:431)
> at
> Caused by: java.lang.IllegalArgumentException: Invalid byte 100 at offset 0
> in '' len=12
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:516)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:540)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1496)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1478)
> ... 7 more
> {code}
> That code shows that occasionally the header is wrong (the tar entry name
> contains gibberish bits) which makes me think that `getNextTarEntry()` can be
> faulty.
>
> Running that code with commons compress 1.25.0 works as expected. So it's
> probably something added since November. Note that this is something related
> to parallelism - using an executor service with a single thread doesn't
> suffer from the same error. The tgz to decompress doesn't really matter - you
> can use a manually created one worth a few KBs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821864#comment-17821864
]
Cosmin Carabet commented on COMPRESS-666:
-
Hey Gary,
Thanks for looking into this. That does indeed seem to fix that test for me. Do
you have a feel for what changes? Wondering whether the buffering behaviour
fixes it vs making some sort of race condition less likely to occur.
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although TarArchiveInputStream is marked as not thread safe, I am not reusing
> objects here. Those are in fact separate objects, presumably all with their
> own position tracking info.
>
> The stacktrace here looks like:
> {code:java}
> Caused by: java.io.IOException: Corrupted TAR archive.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1480)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:534)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:431)
> at
> Caused by: java.lang.IllegalArgumentException: Invalid byte 100 at offset 0
> in '' len=12
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:516)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:540)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1496)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1478)
> ... 7 more
> {code}
> That code shows that occasionally the header is wrong (the tar entry name
> contains gibberish bits) which makes me think that `getNextTarEntry()` can be
> faulty.
>
> Running that code with commons compress 1.25.0 works as expected. So it's
> probably something added since November. Note that this is something related
> to parallelism - using an executor service with a single thread doesn't
> suffer from the same error. The tgz to decompress doesn't really matter - you
> can use a manually created one worth a few KBs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (COMPRESS-667) ZipFile Builder lacks convenience constructors for File/Path
[
https://issues.apache.org/jira/browse/COMPRESS-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821859#comment-17821859
]
Gary D. Gregory commented on COMPRESS-667:
--
{quote}The current Builder implementation appears to not have convenience
methods for the most common construction scenarios e.g. construction from a
File or Path.
{quote}
It does, look for all callers of {{ZipFile.builder()}} and you'll find plenty
of tests using different input types. For example:
ZipFileTest.testZipWithShortBeginningGarbage():
{code:java}
try (ZipFile zipFile = ZipFile.builder().setPath(path).get()) {
...
}
{code}
Constructors on builders just duplicates part of the problems of constructors
on a domain class; there is always one more "convenience" constructor to add,
and then you're back to ZipFile's fourteen constructors with no end in sight.
> ZipFile Builder lacks convenience constructors for File/Path
>
>
> Key: COMPRESS-667
> URL: https://issues.apache.org/jira/browse/COMPRESS-667
> Project: Commons Compress
> Issue Type: Improvement
>Affects Versions: 1.26.0
>Reporter: Daniel Lowe
>Priority: Major
>
> The current Builder implementation appears to not have convenience methods
> for the most common construction scenarios e.g. construction from a File or
> Path.
> As
> {color:#00}{color:#00}ZipFile.{color}{color:#00}builder{color}{color:#00}().get()
> {color}{color}isn't valid, personally I think the Builder constructor itself
> should require a
> {color:#00}{color:#00}SeekableByteChannel/{color}{color}File/Path{color:#00}{color:#00}
> {color}{color}
> I tend to agree with Elliotte's comments
> ([https://lists.apache.org/thread/5vtbbs18d4rntn8tpnpsvo0g9h6svc30),] and
> would also see de-deprecating the one argument constructors as a valid
> solution
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (COMPRESS-667) ZipFile Builder lacks convenience constructors for File/Path
[
https://issues.apache.org/jira/browse/COMPRESS-667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary D. Gregory resolved COMPRESS-667.
--
Resolution: Information Provided
> ZipFile Builder lacks convenience constructors for File/Path
>
>
> Key: COMPRESS-667
> URL: https://issues.apache.org/jira/browse/COMPRESS-667
> Project: Commons Compress
> Issue Type: Improvement
>Affects Versions: 1.26.0
>Reporter: Daniel Lowe
>Priority: Major
>
> The current Builder implementation appears to not have convenience methods
> for the most common construction scenarios e.g. construction from a File or
> Path.
> As
> {color:#00}{color:#00}ZipFile.{color}{color:#00}builder{color}{color:#00}().get()
> {color}{color}isn't valid, personally I think the Builder constructor itself
> should require a
> {color:#00}{color:#00}SeekableByteChannel/{color}{color}File/Path{color:#00}{color:#00}
> {color}{color}
> I tend to agree with Elliotte's comments
> ([https://lists.apache.org/thread/5vtbbs18d4rntn8tpnpsvo0g9h6svc30),] and
> would also see de-deprecating the one argument constructors as a valid
> solution
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (COMPRESS-667) ZipFile Builder lacks convenience constructors for File/Path
Daniel Lowe created COMPRESS-667:
Summary: ZipFile Builder lacks convenience constructors for
File/Path
Key: COMPRESS-667
URL: https://issues.apache.org/jira/browse/COMPRESS-667
Project: Commons Compress
Issue Type: Improvement
Affects Versions: 1.26.0
Reporter: Daniel Lowe
The current Builder implementation appears to not have convenience methods for
the most common construction scenarios e.g. construction from a File or Path.
As
{color:#00}{color:#00}ZipFile.{color}{color:#00}builder{color}{color:#00}().get()
{color}{color}isn't valid, personally I think the Builder constructor itself
should require a
{color:#00}{color:#00}SeekableByteChannel/{color}{color}File/Path{color:#00}{color:#00}
{color}{color}
I tend to agree with Elliotte's comments
([https://lists.apache.org/thread/5vtbbs18d4rntn8tpnpsvo0g9h6svc30),] and would
also see de-deprecating the one argument constructors as a valid solution
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MATH-1667) none of the documentation api links work
[ https://issues.apache.org/jira/browse/MATH-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821821#comment-17821821 ] robert engels commented on MATH-1667: - I've struggled with it for a while and figured it was something temporary - but after enough time I figured I better report it :) > none of the documentation api links work > > > Key: MATH-1667 > URL: https://issues.apache.org/jira/browse/MATH-1667 > Project: Commons Math > Issue Type: Bug >Reporter: robert engels >Priority: Major > > Visit [https://commons.apache.org/proper/commons-math/userguide/fitting.html] > and click on any of the links in the documentation on the right. None work. > Makes it extremely difficult to use. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MATH-1667) none of the documentation api links work
[ https://issues.apache.org/jira/browse/MATH-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821819#comment-17821819 ] Gilles Sadowski commented on MATH-1667: --- Thanks for the report and sorry about that. I think that the issue is related to the modularization (and the links still point to where the targets were before the modularization). > none of the documentation api links work > > > Key: MATH-1667 > URL: https://issues.apache.org/jira/browse/MATH-1667 > Project: Commons Math > Issue Type: Bug >Reporter: robert engels >Priority: Major > > Visit [https://commons.apache.org/proper/commons-math/userguide/fitting.html] > and click on any of the links in the documentation on the right. None work. > Makes it extremely difficult to use. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821807#comment-17821807
]
Gary D. Gregory edited comment on COMPRESS-666 at 2/28/24 6:29 PM:
---
[~cosmin79]
The simplest solution is to buffer the GZIP input stream:
{code:java}
TarArchiveInputStream tarInputStream = new TarArchiveInputStream(new
BufferedInputStream(new GZIPInputStream(inputStream)),
{code}
See {{GZipTest.testCompress666()}}
was (Author: garydgregory):
[~cosmin79]
The simplest solution is to buffer the GZIP input stream:
{code:java}
TarArchiveInputStream tarInputStream = new TarArchiveInputStream(new
BufferedInputStream(new GZIPInputStream(inputStream)),
{code}
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although TarArchiveInputStream is marked as not thread safe, I am not reusing
> objects here. Those are in fact separate objects, presumably all with their
> own position tracking info.
>
> The stacktrace here looks like:
> {code:java}
> Caused by: java.io.IOException: Corrupted TAR archive.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1480)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:534)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:431)
> at
> Caused by: java.lang.IllegalArgumentException: Invalid byte 100 at offset 0
> in '' len=12
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:516)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:540)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1496)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1478)
> ... 7 more
> {code}
> That code shows that occasionally the header is wrong (the tar entry name
> contains gibberish bits) which makes me think that `getNextTarEntry()` can be
> faulty.
>
> Running that code with commons compress 1.25.0 works as expected. So it's
> probably something added since November. Note that this is something related
> to parallelism - using an executor service with a single thread doesn't
> suffer from the same error. The tgz to decompress doesn't really matter - you
> can use a manually created one worth a few KBs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821807#comment-17821807
]
Gary D. Gregory commented on COMPRESS-666:
--
[~cosmin79]
The simplest solution is to buffer the GZIP input stream:
{code:java}
TarArchiveInputStream tarInputStream = new TarArchiveInputStream(new
BufferedInputStream(new GZIPInputStream(inputStream)),
{code}
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although TarArchiveInputStream is marked as not thread safe, I am not reusing
> objects here. Those are in fact separate objects, presumably all with their
> own position tracking info.
>
> The stacktrace here looks like:
> {code:java}
> Caused by: java.io.IOException: Corrupted TAR archive.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1480)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:534)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:431)
> at
> Caused by: java.lang.IllegalArgumentException: Invalid byte 100 at offset 0
> in '' len=12
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:516)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:540)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1496)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1478)
> ... 7 more
> {code}
> That code shows that occasionally the header is wrong (the tar entry name
> contains gibberish bits) which makes me think that `getNextTarEntry()` can be
> faulty.
>
> Running that code with commons compress 1.25.0 works as expected. So it's
> probably something added since November. Note that this is something related
> to parallelism - using an executor service with a single thread doesn't
> suffer from the same error. The tgz to decompress doesn't really matter - you
> can use a manually created one worth a few KBs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (MATH-1667) none of the documentation api links work
robert engels created MATH-1667: --- Summary: none of the documentation api links work Key: MATH-1667 URL: https://issues.apache.org/jira/browse/MATH-1667 Project: Commons Math Issue Type: Bug Reporter: robert engels Visit [https://commons.apache.org/proper/commons-math/userguide/fitting.html] and click on any of the links in the documentation on the right. None work. Makes it extremely difficult to use. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] CLI-326: make option.getKey() public [commons-cli]
garydgregory commented on PR #239: URL: https://github.com/apache/commons-cli/pull/239#issuecomment-1969485725 @Claudenw ping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] ExifRewriterRoundtripTest fixed, completed & enabled [commons-imaging]
StefanOltmann commented on code in PR #366:
URL: https://github.com/apache/commons-imaging/pull/366#discussion_r1506190234
##
src/test/java/org/apache/commons/imaging/formats/jpeg/exif/ExifRewriterRoundtripTest.java:
##
@@ -56,124 +61,160 @@ public static Stream data() throws Exception {
private File duplicateFile;
-private void assertTiffEquals(final TiffOutputSet tiffOutputSet, final
TiffOutputSet tiffOutputSet1) {
-final List directories =
tiffOutputSet.getDirectories();
-final List directories1 =
tiffOutputSet1.getDirectories();
-assertEquals(directories.size(), directories1.size(), "The
TiffOutputSets have different numbers of directories.");
-
-for (int i = 0; i < directories.size(); i++) {
-final List fields =
directories.get(i).getFields();
-final List fields1 =
directories1.get(i).getFields();
-assertEquals(fields.size(), fields1.size(), "The
TiffOutputDirectories have different numbers of fields.");
-
-for (int j = 0; j < fields.size(); j++) {
-final TiffOutputField field = fields.get(j);
-final TiffOutputField field1 = fields1.get(j);
-assertEquals(field.tag, field1.tag, "TiffOutputField tag
mismatch.");
-assertEquals(field.tagInfo, field1.tagInfo, "TiffOutputField
tagInfo mismatch.");
-assertEquals(field.abstractFieldType,
field1.abstractFieldType, "TiffOutputField fieldType mismatch.");
-assertEquals(field.count, field1.count, "TiffOutputField count
mismatch.");
+private void assertTiffEquals(final TiffOutputSet sourceTiffOutputSet,
final TiffOutputSet duplicateTiffOutputSet) {
+
+final List sourceDirectories =
sourceTiffOutputSet.getDirectories();
+final List duplicatedDirectories =
duplicateTiffOutputSet.getDirectories();
+
+assertEquals(sourceDirectories.size(), duplicatedDirectories.size(),
"The TiffOutputSets have different numbers of directories.");
+
+for (int i = 0; i < sourceDirectories.size(); i++) {
+
+final TiffOutputDirectory sourceDirectory =
sourceDirectories.get(i);
+final TiffOutputDirectory duplicateDirectory =
duplicatedDirectories.get(i);
+
+assertEquals(sourceDirectory.getType(),
duplicateDirectory.getType(), "Directory type mismatch.");
+
+final List sourceFields =
sourceDirectory.getFields();
+final List duplicateFields =
duplicateDirectory.getFields();
+
+final boolean fieldCountMatches = sourceFields.size() ==
duplicateFields.size();
+
+if (!fieldCountMatches) {
+
+/*
+ * Note that offset fields are not written again if a re-order
makes
+ * them unnecessary. The TiffWriter does not write in the same
order,
+ * but tries to optimize it.
+ */
+
+final List sourceTags = new ArrayList<>();
+final List duplicatedTags = new ArrayList<>();
+
+for (TiffOutputField field : sourceFields)
+sourceTags.add(field.tag);
+
+for (TiffOutputField field : duplicateFields)
+duplicatedTags.add(field.tag);
+
+final List missingTags = new ArrayList<>(sourceTags);
+missingTags.removeAll(duplicatedTags);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_EXIF_OFFSET.tag);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_GPSINFO.tag);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_INTEROP_OFFSET.tag);
+missingTags.remove((Integer)
TiffTagConstants.TIFF_TAG_JPEG_INTERCHANGE_FORMAT.tag);
+
+assertTrue(missingTags.isEmpty(), "Missing tags: " +
missingTags);
+}
+
+for (TiffOutputField sourceField : sourceFields) {
+
+final boolean isOffsetField =
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_EXIF_OFFSET.tag ||
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_GPSINFO.tag ||
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_INTEROP_OFFSET.tag ||
+sourceField.tag ==
TiffTagConstants.TIFF_TAG_JPEG_INTERCHANGE_FORMAT.tag;
+
+/*
+ * Ignore offset fields. They may not be needed after rewrite
+ * and their value changes anyway.
+ */
+if (isOffsetField)
+continue;
+
+TiffOutputField duplicateField =
duplicateDirectory.findField(sourceField.tag);
+
+assertNotNull(duplicateField, "Field is missing: " +
sourceField.tagInfo);
+
+assertEquals(sourceField.tag, duplicateField.tag,
"TiffOutputField tag mismatch.");
+
Re: [PR] [IMAGING-319] Fix for lost first character on String [commons-imaging]
StefanOltmann commented on PR #359: URL: https://github.com/apache/commons-imaging/pull/359#issuecomment-1969281785 > A PR should include main and test changes, not one PR for test changes, and a different PR for main changes. Yes, but the bugfix and the test are unrelated. One PR fixes a bug without a test. The other PR completes a test that’s right now disabled, because it wasn’t working. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] ExifRewriterRoundtripTest fixed, completed & enabled [commons-imaging]
StefanOltmann commented on PR #366: URL: https://github.com/apache/commons-imaging/pull/366#issuecomment-1969280669 > A PR should include main and test changes, not one PR for test changes, and a different PR for main changes. Yes, but the bugfix and the test are unrelated. One PR fixes a bug without a test. The other PR completes a test that’s right now disabled, because it wasn’t working. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IMAGING-319] Fix for lost first character on String [commons-imaging]
garydgregory commented on PR #359: URL: https://github.com/apache/commons-imaging/pull/359#issuecomment-1969247792 A PR should include main and test changes, not one PR for test changes, and a different PR for main changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] ExifRewriterRoundtripTest fixed, completed & enabled [commons-imaging]
garydgregory commented on PR #366: URL: https://github.com/apache/commons-imaging/pull/366#issuecomment-1969247182 A PR should include main and test changes, not one PR for test changes, and a different PR for main changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] ExifRewriterRoundtripTest fixed, completed & enabled [commons-imaging]
garydgregory commented on code in PR #366:
URL: https://github.com/apache/commons-imaging/pull/366#discussion_r1506153139
##
src/test/java/org/apache/commons/imaging/formats/jpeg/exif/ExifRewriterRoundtripTest.java:
##
@@ -56,124 +61,160 @@ public static Stream data() throws Exception {
private File duplicateFile;
-private void assertTiffEquals(final TiffOutputSet tiffOutputSet, final
TiffOutputSet tiffOutputSet1) {
-final List directories =
tiffOutputSet.getDirectories();
-final List directories1 =
tiffOutputSet1.getDirectories();
-assertEquals(directories.size(), directories1.size(), "The
TiffOutputSets have different numbers of directories.");
-
-for (int i = 0; i < directories.size(); i++) {
-final List fields =
directories.get(i).getFields();
-final List fields1 =
directories1.get(i).getFields();
-assertEquals(fields.size(), fields1.size(), "The
TiffOutputDirectories have different numbers of fields.");
-
-for (int j = 0; j < fields.size(); j++) {
-final TiffOutputField field = fields.get(j);
-final TiffOutputField field1 = fields1.get(j);
-assertEquals(field.tag, field1.tag, "TiffOutputField tag
mismatch.");
-assertEquals(field.tagInfo, field1.tagInfo, "TiffOutputField
tagInfo mismatch.");
-assertEquals(field.abstractFieldType,
field1.abstractFieldType, "TiffOutputField fieldType mismatch.");
-assertEquals(field.count, field1.count, "TiffOutputField count
mismatch.");
+private void assertTiffEquals(final TiffOutputSet sourceTiffOutputSet,
final TiffOutputSet duplicateTiffOutputSet) {
+
+final List sourceDirectories =
sourceTiffOutputSet.getDirectories();
+final List duplicatedDirectories =
duplicateTiffOutputSet.getDirectories();
+
+assertEquals(sourceDirectories.size(), duplicatedDirectories.size(),
"The TiffOutputSets have different numbers of directories.");
+
+for (int i = 0; i < sourceDirectories.size(); i++) {
+
+final TiffOutputDirectory sourceDirectory =
sourceDirectories.get(i);
+final TiffOutputDirectory duplicateDirectory =
duplicatedDirectories.get(i);
+
+assertEquals(sourceDirectory.getType(),
duplicateDirectory.getType(), "Directory type mismatch.");
+
+final List sourceFields =
sourceDirectory.getFields();
+final List duplicateFields =
duplicateDirectory.getFields();
+
+final boolean fieldCountMatches = sourceFields.size() ==
duplicateFields.size();
+
+if (!fieldCountMatches) {
+
+/*
+ * Note that offset fields are not written again if a re-order
makes
+ * them unnecessary. The TiffWriter does not write in the same
order,
+ * but tries to optimize it.
+ */
+
+final List sourceTags = new ArrayList<>();
+final List duplicatedTags = new ArrayList<>();
+
+for (TiffOutputField field : sourceFields)
+sourceTags.add(field.tag);
+
+for (TiffOutputField field : duplicateFields)
+duplicatedTags.add(field.tag);
+
+final List missingTags = new ArrayList<>(sourceTags);
+missingTags.removeAll(duplicatedTags);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_EXIF_OFFSET.tag);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_GPSINFO.tag);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_INTEROP_OFFSET.tag);
+missingTags.remove((Integer)
TiffTagConstants.TIFF_TAG_JPEG_INTERCHANGE_FORMAT.tag);
+
+assertTrue(missingTags.isEmpty(), "Missing tags: " +
missingTags);
+}
+
+for (TiffOutputField sourceField : sourceFields) {
+
+final boolean isOffsetField =
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_EXIF_OFFSET.tag ||
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_GPSINFO.tag ||
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_INTEROP_OFFSET.tag ||
+sourceField.tag ==
TiffTagConstants.TIFF_TAG_JPEG_INTERCHANGE_FORMAT.tag;
+
+/*
+ * Ignore offset fields. They may not be needed after rewrite
+ * and their value changes anyway.
+ */
+if (isOffsetField)
+continue;
+
+TiffOutputField duplicateField =
duplicateDirectory.findField(sourceField.tag);
+
+assertNotNull(duplicateField, "Field is missing: " +
sourceField.tagInfo);
+
+assertEquals(sourceField.tag, duplicateField.tag,
"TiffOutputField tag mismatch.");
+
Re: [PR] ExifRewriterRoundtripTest fixed, completed & enabled [commons-imaging]
garydgregory commented on code in PR #366:
URL: https://github.com/apache/commons-imaging/pull/366#discussion_r1506153139
##
src/test/java/org/apache/commons/imaging/formats/jpeg/exif/ExifRewriterRoundtripTest.java:
##
@@ -56,124 +61,160 @@ public static Stream data() throws Exception {
private File duplicateFile;
-private void assertTiffEquals(final TiffOutputSet tiffOutputSet, final
TiffOutputSet tiffOutputSet1) {
-final List directories =
tiffOutputSet.getDirectories();
-final List directories1 =
tiffOutputSet1.getDirectories();
-assertEquals(directories.size(), directories1.size(), "The
TiffOutputSets have different numbers of directories.");
-
-for (int i = 0; i < directories.size(); i++) {
-final List fields =
directories.get(i).getFields();
-final List fields1 =
directories1.get(i).getFields();
-assertEquals(fields.size(), fields1.size(), "The
TiffOutputDirectories have different numbers of fields.");
-
-for (int j = 0; j < fields.size(); j++) {
-final TiffOutputField field = fields.get(j);
-final TiffOutputField field1 = fields1.get(j);
-assertEquals(field.tag, field1.tag, "TiffOutputField tag
mismatch.");
-assertEquals(field.tagInfo, field1.tagInfo, "TiffOutputField
tagInfo mismatch.");
-assertEquals(field.abstractFieldType,
field1.abstractFieldType, "TiffOutputField fieldType mismatch.");
-assertEquals(field.count, field1.count, "TiffOutputField count
mismatch.");
+private void assertTiffEquals(final TiffOutputSet sourceTiffOutputSet,
final TiffOutputSet duplicateTiffOutputSet) {
+
+final List sourceDirectories =
sourceTiffOutputSet.getDirectories();
+final List duplicatedDirectories =
duplicateTiffOutputSet.getDirectories();
+
+assertEquals(sourceDirectories.size(), duplicatedDirectories.size(),
"The TiffOutputSets have different numbers of directories.");
+
+for (int i = 0; i < sourceDirectories.size(); i++) {
+
+final TiffOutputDirectory sourceDirectory =
sourceDirectories.get(i);
+final TiffOutputDirectory duplicateDirectory =
duplicatedDirectories.get(i);
+
+assertEquals(sourceDirectory.getType(),
duplicateDirectory.getType(), "Directory type mismatch.");
+
+final List sourceFields =
sourceDirectory.getFields();
+final List duplicateFields =
duplicateDirectory.getFields();
+
+final boolean fieldCountMatches = sourceFields.size() ==
duplicateFields.size();
+
+if (!fieldCountMatches) {
+
+/*
+ * Note that offset fields are not written again if a re-order
makes
+ * them unnecessary. The TiffWriter does not write in the same
order,
+ * but tries to optimize it.
+ */
+
+final List sourceTags = new ArrayList<>();
+final List duplicatedTags = new ArrayList<>();
+
+for (TiffOutputField field : sourceFields)
+sourceTags.add(field.tag);
+
+for (TiffOutputField field : duplicateFields)
+duplicatedTags.add(field.tag);
+
+final List missingTags = new ArrayList<>(sourceTags);
+missingTags.removeAll(duplicatedTags);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_EXIF_OFFSET.tag);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_GPSINFO.tag);
+missingTags.remove((Integer)
ExifTagConstants.EXIF_TAG_INTEROP_OFFSET.tag);
+missingTags.remove((Integer)
TiffTagConstants.TIFF_TAG_JPEG_INTERCHANGE_FORMAT.tag);
+
+assertTrue(missingTags.isEmpty(), "Missing tags: " +
missingTags);
+}
+
+for (TiffOutputField sourceField : sourceFields) {
+
+final boolean isOffsetField =
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_EXIF_OFFSET.tag ||
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_GPSINFO.tag ||
+sourceField.tag ==
ExifTagConstants.EXIF_TAG_INTEROP_OFFSET.tag ||
+sourceField.tag ==
TiffTagConstants.TIFF_TAG_JPEG_INTERCHANGE_FORMAT.tag;
+
+/*
+ * Ignore offset fields. They may not be needed after rewrite
+ * and their value changes anyway.
+ */
+if (isOffsetField)
+continue;
+
+TiffOutputField duplicateField =
duplicateDirectory.findField(sourceField.tag);
+
+assertNotNull(duplicateField, "Field is missing: " +
sourceField.tagInfo);
+
+assertEquals(sourceField.tag, duplicateField.tag,
"TiffOutputField tag mismatch.");
+
[jira] [Comment Edited] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821721#comment-17821721
]
Cosmin Carabet edited comment on COMPRESS-666 at 2/28/24 3:27 PM:
--
Ah, I see. You're still getting the "Corrupt GZIP trailer" behaviour though,
which is good. I'm getting the "Invalid byte 100 at offset 0 in ''
len=12" error on MacOS for what it's worth. I believe what you're getting is
closer to what I initially saw on one of my services running in Linux (I first
it observed in there and then wrote a test to reproduce it locally). In any
case, I wouldn't over-index on the "invalid byte" part, but rather on the
`readTrailer() -> Corrupt GZIP trailer` behaviour, which is consistent.
For what it's worth, if you cherry pick that test on top of 1.25.0, it works
correctly. That's why I basically suggested it's likely something which came in
between those 2 versions (1.25.0 - 1.26.0)
was (Author: JIRAUSER304399):
Ah, I see. You're still getting the "Corrupt GZIP trailer" behaviour though,
which is good. I'm getting the "Invalid byte 100 at offset 0 in ''
len=12" error on MacOS for what it's worth. I believe what you're getting is
closer to what I initially saw on one of my services running in Linux (I first
observed in there and then wrote a test to reproduce it locally). In any case,
I wouldn't over-index on the "invalid byte" part, but rather on the
`readTrailer() -> Corrupt GZIP trailer` behaviour, which is consistent.
For what it's worth, if you cherry pick that test on top of 1.25.0, it works
correctly. That's why I basically suggested it's likely something which came in
between those 2 versions (1.25.0 - 1.26.0)
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although TarArchiveInputStream is marked as not thread safe, I am not reusing
> objects here. Those are in fact separate objects, presumably all with their
> own position tracking info.
>
> The stacktrace here looks like:
> {code:java}
> Caused by: java.io.IOException: Corrupted TAR archive.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1480)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:534)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:431)
> at
> Caused by: java.lang.IllegalArgumentException: Invalid byte 100 at offset 0
> in '' len=12
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:516)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:540)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1496)
> at
>
[jira] [Commented] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821721#comment-17821721
]
Cosmin Carabet commented on COMPRESS-666:
-
Ah, I see. You're still getting the "Corrupt GZIP trailer" behaviour though,
which is good. I'm getting the "Invalid byte 100 at offset 0 in ''
len=12" error on MacOS for what it's worth. I believe what you're getting is
closer to what I initially saw on one of my services running in Linux (I first
observed in there and then wrote a test to reproduce it locally). In any case,
I wouldn't over-index on the "invalid byte" part, but rather on the
`readTrailer() -> Corrupt GZIP trailer` behaviour, which is consistent.
For what it's worth, if you cherry pick that test on top of 1.25.0, it works
correctly. That's why I basically suggested it's likely something which came in
between those 2 versions (1.25.0 - 1.26.0)
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although TarArchiveInputStream is marked as not thread safe, I am not reusing
> objects here. Those are in fact separate objects, presumably all with their
> own position tracking info.
>
> The stacktrace here looks like:
> {code:java}
> Caused by: java.io.IOException: Corrupted TAR archive.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1480)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.(TarArchiveEntry.java:534)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:431)
> at
> Caused by: java.lang.IllegalArgumentException: Invalid byte 100 at offset 0
> in '' len=12
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:516)
> at
> org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:540)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1496)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1478)
> ... 7 more
> {code}
> That code shows that occasionally the header is wrong (the tar entry name
> contains gibberish bits) which makes me think that `getNextTarEntry()` can be
> faulty.
>
> Running that code with commons compress 1.25.0 works as expected. So it's
> probably something added since November. Note that this is something related
> to parallelism - using an executor service with a single thread doesn't
> suffer from the same error. The tgz to decompress doesn't really matter - you
> can use a manually created one worth a few KBs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (IMAGING-373) OutOfMemory with invalid ICO input file
[ https://issues.apache.org/jira/browse/IMAGING-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary D. Gregory resolved IMAGING-373. - Fix Version/s: 1.0 Resolution: Fixed > OutOfMemory with invalid ICO input file > --- > > Key: IMAGING-373 > URL: https://issues.apache.org/jira/browse/IMAGING-373 > Project: Commons Imaging > Issue Type: Bug > Components: Format: BMP, Format: ICO >Affects Versions: 1.0-alpha3 >Reporter: Milan Nikl >Priority: Major > Fix For: 1.0 > > Attachments: epine.ico > > > While trying to _use org.apache.commons.imaging.formats.ico.IcoImageParser_ > to read an icon file, I'm getting OutOfMemory Error. The file is corrupted, > but that should not lead to an Error. > Icon is downloaded from [https://epine.es/assets/icos/epine.ico] and > hopefully attached to this issue. > [^epine.ico] > > Trying to debug the problem, I found out that the cause is in misinterpreting > some values in > _org.apache.commons.imaging.formats.bmp.BmpImageParser#readBmpHeaderInfo(java.io.InputStream, > org.apache.commons.imaging.FormatCompliance)_ method while trying to load > data of the single image bitmap in given ico file. > While _fileSize_ value is 5990, the _bitmapDataOffset_ value parsed is > 989265922. At the same time _expectedDataOffset_ value is 70, which leads to > _extraBytes_ value of 989265852. > Then > _org.apache.commons.imaging.common.BinaryFunctions#readBytes(java.lang.String, > java.io.InputStream, int, java.lang.String)_ tries to allocate byte array of > size 989265852. Which exhausts available memory of the JVM. > Maybe just a simple check of _extraBytes > bhi.fileSize_ could help? -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] IMAGING-373: Throw exception on invalid bitmap offset [commons-imaging]
garydgregory commented on PR #368: URL: https://github.com/apache/commons-imaging/pull/368#issuecomment-1969224775 @Draczech TY ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] IMAGING-373: Throw exception on invalid bitmap offset [commons-imaging]
garydgregory merged PR #368: URL: https://github.com/apache/commons-imaging/pull/368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (IMAGING-373) OutOfMemory with invalid ICO input file
[
https://issues.apache.org/jira/browse/IMAGING-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821717#comment-17821717
]
Gary D. Gregory commented on IMAGING-373:
-
For me locally, without the change to {{main}}, I do not get an OOME, I get a
{{IOException}}:
{noformat}
org.opentest4j.AssertionFailedError: Unexpected exception type thrown,
expected: but was:
at
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:67)
at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:35)
at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3115)
at
org.apache.commons.imaging.formats.ico.IcoImageParserTest.testImageWithInvalidBmpHeaders(IcoImageParserTest.java:37)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: java.io.IOException: Not a Valid BMP File, name: BitmapDataOffset,
length: 989265852
at
org.apache.commons.imaging.common.BinaryFunctions.readBytes(BinaryFunctions.java:255)
at
org.apache.commons.imaging.formats.bmp.BmpImageParser.readImageContents(BmpImageParser.java:577)
at
org.apache.commons.imaging.formats.bmp.BmpImageParser.getBufferedImage(BmpImageParser.java:123)
at
org.apache.commons.imaging.formats.ico.IcoImageParser.readBitmapIconData(IcoImageParser.java:450)
at
org.apache.commons.imaging.formats.ico.IcoImageParser.readIconData(IcoImageParser.java:527)
at
org.apache.commons.imaging.formats.ico.IcoImageParser.readImage(IcoImageParser.java:567)
at
org.apache.commons.imaging.formats.ico.IcoImageParser.getAllBufferedImages(IcoImageParser.java:258)
at
org.apache.commons.imaging.AbstractImageParser.getAllBufferedImages(AbstractImageParser.java:282)
at
org.apache.commons.imaging.formats.ico.IcoImageParserTest.lambda$0(IcoImageParserTest.java:37)
at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:53)
... 6 more
{noformat}
Please check your use case with git master or a snapshot build from
https://repository.apache.org/content/repositories/snapshots/org/apache/commons/commons-imaging/1.0-M1-SNAPSHOT/
We do different internal checking than in alpha3 now, but the suggested fix
still looks valid. I'll see if we can fail faster with the suggestion.
> OutOfMemory with invalid ICO input file
> ---
>
> Key: IMAGING-373
> URL: https://issues.apache.org/jira/browse/IMAGING-373
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: BMP, Format: ICO
>Affects Versions: 1.0-alpha3
>Reporter: Milan Nikl
>Priority: Major
> Attachments: epine.ico
>
>
> While trying to _use org.apache.commons.imaging.formats.ico.IcoImageParser_
> to read an icon file, I'm getting OutOfMemory Error. The file is corrupted,
> but that should not lead to an Error.
> Icon is downloaded from [https://epine.es/assets/icos/epine.ico] and
> hopefully attached to this issue.
> [^epine.ico]
>
> Trying to debug the problem, I found out that the cause is in misinterpreting
> some values in
> _org.apache.commons.imaging.formats.bmp.BmpImageParser#readBmpHeaderInfo(java.io.InputStream,
> org.apache.commons.imaging.FormatCompliance)_ method while trying to load
> data of the single image bitmap in given ico file.
> While _fileSize_ value is 5990, the _bitmapDataOffset_ value parsed is
> 989265922. At the same time _expectedDataOffset_ value is 70, which leads to
> _extraBytes_ value of 989265852.
> Then
> _org.apache.commons.imaging.common.BinaryFunctions#readBytes(java.lang.String,
> java.io.InputStream, int, java.lang.String)_ tries to allocate byte array of
> size 989265852. Which exhausts available memory of the JVM.
> Maybe just a simple check of _extraBytes > bhi.fileSize_ could help?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (COMPRESS-666) Multithreaded access to Tar archive throws java.util.zip.ZipException: Corrupt GZIP trailer
[
https://issues.apache.org/jira/browse/COMPRESS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821707#comment-17821707
]
Gary D. Gregory commented on COMPRESS-666:
--
Hello [~cosmin79],
Yes, this is what I get locally:
{noformat}
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:46)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:161)
at
org.apache.commons.compress.compressors.GZipTest.testCompress666(GZipTest.java:88)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: java.util.concurrent.ExecutionException:
org.opentest4j.AssertionFailedError:
org.apache.commons.compress.archivers.tar.TarArchiveEntry@101289c2
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.commons.compress.compressors.GZipTest.testCompress666(GZipTest.java:81)
... 3 more
Caused by: org.opentest4j.AssertionFailedError:
org.apache.commons.compress.archivers.tar.TarArchiveEntry@101289c2
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:42)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:150)
at
org.apache.commons.compress.compressors.GZipTest.lambda$1(GZipTest.java:75)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.zip.ZipException: Corrupt GZIP trailer
at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:225)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:119)
at org.apache.commons.io.IOUtils.skip(IOUtils.java:2422)
at org.apache.commons.io.IOUtils.skip(IOUtils.java:2380)
at
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.consumeRemainderOfLastBlock(TarArchiveInputStream.java:320)
at
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getRecord(TarArchiveInputStream.java:505)
at
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:418)
at
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextEntry(TarArchiveInputStream.java:392)
at
org.apache.commons.compress.compressors.GZipTest.lambda$1(GZipTest.java:71)
... 5 more
{noformat}
> Multithreaded access to Tar archive throws java.util.zip.ZipException:
> Corrupt GZIP trailer
> ---
>
> Key: COMPRESS-666
> URL: https://issues.apache.org/jira/browse/COMPRESS-666
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.26.0
> Environment: Commons compress 1.26.0 to get a failure. Any tar tgz.
>Reporter: Cosmin Carabet
>Priority: Major
>
> Something in
> [https://github.com/apache/commons-compress/compare/rel/commons-compress-1.25.0...master]
> seems to make iterating through the tar entries of multiple
> TarArchiveInputStreams throw Corrupted TAR archive:
>
> {code:java}
> @Test
> void bla() {
> ExecutorService executorService = Executors.newFixedThreadPool(10);
> List> tasks = IntStream.range(0, 200)
> .mapToObj(_idx -> CompletableFuture.runAsync(
> () -> {
> try (InputStream inputStream = this.getClass()
> .getResourceAsStream(
> "/");
> TarArchiveInputStream tarInputStream =
> new TarArchiveInputStream(new
> GZIPInputStream(inputStream))) {
> TarArchiveEntry tarEntry;
> while ((tarEntry =
> tarInputStream.getNextTarEntry()) != null) {
> System.out.println("Reading entry %s with
> size %d"
> .formatted(tarEntry.getName(),
> tarEntry.getSize()));
> }
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> },
> executorService))
> .toList();
> Futures.getUnchecked(CompletableFuture.allOf(tasks.toArray(new
> CompletableFuture[0])));
> } {code}
> Although
Re: [PR] Bump jakarta.mail from 1.6.5 to 2.0.1 [commons-email]
garydgregory commented on PR #43: URL: https://github.com/apache/commons-email/pull/43#issuecomment-1969086156 @mkurz I can try for an M1 at the end of the week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] IMAGING-373: Throw exception on invalid bitmap offset [commons-imaging]
Draczech commented on code in PR #368:
URL: https://github.com/apache/commons-imaging/pull/368#discussion_r1505992407
##
src/test/java/org/apache/commons/imaging/formats/ico/IcoImageParserTest.java:
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.imaging.formats.ico;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import org.apache.commons.imaging.ImagingException;
+import org.apache.commons.imaging.test.TestResources;
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+
+public class IcoImageParserTest {
+
+/**
+ * For https://issues.apache.org/jira/browse/IMAGING-373.
+ * There is a problem with loading bitmap stored in given ICO file, so
the exception is originally thrown by BmpImageParser.
Review Comment:
Fixed.
I haven't noticed any plugin like https://github.com/diffplug/spotless in
the project, I might just missed it. Apparently I'm relying too much on such
tools :)
##
src/main/java/org/apache/commons/imaging/formats/bmp/BmpImageParser.java:
##
@@ -569,7 +569,7 @@ private BmpImageContents readImageContents(final
InputStream is, final FormatCom
debugNumber("expectedDataOffset", expectedDataOffset, 4);
}
final int extraBytes = bhi.bitmapDataOffset - expectedDataOffset;
-if (extraBytes < 0) {
+if ((extraBytes < 0) || (extraBytes > bhi.fileSize)) {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Bump jakarta.mail from 1.6.5 to 2.0.1 [commons-email]
mkurz commented on PR #43: URL: https://github.com/apache/commons-email/pull/43#issuecomment-1968996435 @garydgregory Any ETA when you want to work on `2.0.0-M1`? Like do you have something in mind when a first 2.x release (milestone or rc) will be likely to happen? (I am aware of https://issues.apache.org/jira/projects/EMAIL/issues/EMAIL-203 and https://repository.apache.org/content/repositories/snapshots/ and https://repository.apache.org/content/repositories/snapshots/org/apache/commons/commons-email/) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] IMAGING-373: Throw exception on invalid bitmap offset [commons-imaging]
garydgregory commented on code in PR #368:
URL: https://github.com/apache/commons-imaging/pull/368#discussion_r1505963612
##
src/main/java/org/apache/commons/imaging/formats/bmp/BmpImageParser.java:
##
@@ -569,7 +569,7 @@ private BmpImageContents readImageContents(final
InputStream is, final FormatCom
debugNumber("expectedDataOffset", expectedDataOffset, 4);
}
final int extraBytes = bhi.bitmapDataOffset - expectedDataOffset;
-if (extraBytes < 0) {
+if ((extraBytes < 0) || (extraBytes > bhi.fileSize)) {
Review Comment:
No need for extra parens.
##
src/test/java/org/apache/commons/imaging/formats/ico/IcoImageParserTest.java:
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.imaging.formats.ico;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import org.apache.commons.imaging.ImagingException;
+import org.apache.commons.imaging.test.TestResources;
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+
+public class IcoImageParserTest {
+
+/**
+ * For https://issues.apache.org/jira/browse/IMAGING-373.
+ * There is a problem with loading bitmap stored in given ICO file, so
the exception is originally thrown by BmpImageParser.
Review Comment:
Close HTML tags. No need for a space after ``.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Update the tests to JUnit 5 [commons-cli]
garydgregory merged PR #238: URL: https://github.com/apache/commons-cli/pull/238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Update the tests to JUnit 5 [commons-cli]
garydgregory commented on PR #238: URL: https://github.com/apache/commons-cli/pull/238#issuecomment-1968972973 TY @DamnedElric ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (LANG-1729) NumberUtils.isParsable() returns true for Fullwidth Unicode digits
Andrei Anischevici created LANG-1729:
Summary: NumberUtils.isParsable() returns true for Fullwidth
Unicode digits
Key: LANG-1729
URL: https://issues.apache.org/jira/browse/LANG-1729
Project: Commons Lang
Issue Type: Bug
Components: lang.math.*
Affects Versions: 3.14.0
Reporter: Andrei Anischevici
*NumberUtils.isParsable()* returns *true* for Fullwidth Unicode digits (U+FF10
to U+FF19, see
[https://en.wikipedia.org/wiki/Halfwidth_and_Fullwidth_Forms_(Unicode_block)]),
even though these cannot be parsed by *Double.parseDouble()* and
{*}Float.parseFloat(){*}, as {*}NumberFormatException{*}s are thrown when
trying to parse these.
I've also opened a JDK bug, see
[https://bugs.java.com/bugdatabase/view_bug?bug_id=JDK-8326627] and
[https://bugs.openjdk.org/projects/JDK/issues/JDK-8326627]
Since the underlying issue is that *Character.isDigit()* returns *true* for
fullwidth digits and until this is fixed in the JDK, it would be great to fix
this in *NumberUtils.isParsable()* so that it would return *false* for
Fullwidth Unicode digits.
Gist reproducing the issue:
[https://gist.github.com/XDex/2e300189a07e8224524100f26b5f6b46]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (CLI-312) Inconsistent behaviour in key/value pairs (Java property style)
[
https://issues.apache.org/jira/browse/CLI-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maarten Mulders closed CLI-312.
---
LGTM, thanks for the fix!
> Inconsistent behaviour in key/value pairs (Java property style)
> ---
>
> Key: CLI-312
> URL: https://issues.apache.org/jira/browse/CLI-312
> Project: Commons CLI
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.5.0
>Reporter: Maarten Mulders
>Priority: Minor
> Fix For: 1.7.0
>
>
> In the Apache Maven project, we used to have an {{Option}} defined like this:
> {code:java}
> Option.builder("D").longOpt("define").hasArg().build()
> {code}
> This allowed our users to define properties using all of these:
> * {{-Dx=1}}
> * {{-Dx}}
> * {{-D x}}
> * {{--define x=1}}
> * {{--define x}}
> Splitting the key/value pairs was something that we did inside Maven
> ([{{MavenCli.java
> #1733}}|https://github.com/apache/maven/blob/master/maven-embedder/src/main/java/org/apache/maven/cli/MavenCli.java#L1733]
> and [{{MavenCli.java
> #1762}}|https://github.com/apache/maven/blob/master/maven-embedder/src/main/java/org/apache/maven/cli/MavenCli.java#L1762]).
> In the process of upgrading to Commons CLI 1.5 and moving away from the
> deprecated {{GnuParser}}, we found out that Commons CLI can split argument
> values thanks to the {{valueSeparator}}. We've changed our option declaration
> accordingly:
> {code:java}
> Option.builder("D").longOpt("define").numberOfArgs(2).valueSeparator('=').build()
> );
> {code}
> Unfortunately, this breaks our accepted inputs: {{-Dv -Dw=1 -D x=2 -D y -D
> z=3}} breaks the parsing in Commons CLI:
> {code}
> org.apache.commons.cli.MissingArgumentException: Missing argument for option:
> D
> at
> org.apache.commons.cli.DefaultParser.checkRequiredArgs(DefaultParser.java:231)
> at
> org.apache.commons.cli.DefaultParser.handleOption(DefaultParser.java:394)
> at
> org.apache.commons.cli.DefaultParser.handleShortAndLongOption(DefaultParser.java:467)
> at
> org.apache.commons.cli.DefaultParser.handleToken(DefaultParser.java:542)
> at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:714)
> at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:681)
> at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:662)
> {code}
> From debugging, I've learned that it breaks at the *fifth* occurrence of
> {{-D}} (the one followed by {{z=3}}). It considers the *fourth* occurrence of
> {{-D}} incomplete (having only one value and not two). The strange thing is
> that the *first* occurrence of {{-D}} also has just one value ({{v}}) but
> parsing {{-Dw=1}} does not break over that.
> I can submit a pull request for a bug that demonstrates this, but I have no
> clue how to address it. Also, I don't know if this is actually supposed to
> work. Looking forward to your reaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-373) OutOfMemory with invalid ICO input file
[
https://issues.apache.org/jira/browse/IMAGING-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821583#comment-17821583
]
Milan Nikl commented on IMAGING-373:
Using [https://github.com/drewnoakes/metadata-extractor] to check given file's
metadata, I can see:
* [ICO] Image Size Bytes - 5960
* [ICO] Image Offset Bytes - 22
Using org.apache.commons.imaging.formats.ico.IcoImageParser#readIconInfo I can
see the same:
* imageSize = 5960
* imageOffset = 22
So it looks like the problem is in transition to bitmap in
_org.apache.commons.imaging.formats.ico.IcoImageParser#readBitmapIconData_
where:
* _colorsUsed = 247316463_ which leads to {_}bitmapPixelsOffset =
989265922{_}, while _bitmapSize = 5990_ and _restOfFile.length = 5920_
> OutOfMemory with invalid ICO input file
> ---
>
> Key: IMAGING-373
> URL: https://issues.apache.org/jira/browse/IMAGING-373
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: BMP, Format: ICO
>Affects Versions: 1.0-alpha3
>Reporter: Milan Nikl
>Priority: Major
> Attachments: epine.ico
>
>
> While trying to _use org.apache.commons.imaging.formats.ico.IcoImageParser_
> to read an icon file, I'm getting OutOfMemory Error. The file is corrupted,
> but that should not lead to an Error.
> Icon is downloaded from [https://epine.es/assets/icos/epine.ico] and
> hopefully attached to this issue.
> [^epine.ico]
>
> Trying to debug the problem, I found out that the cause is in misinterpreting
> some values in
> _org.apache.commons.imaging.formats.bmp.BmpImageParser#readBmpHeaderInfo(java.io.InputStream,
> org.apache.commons.imaging.FormatCompliance)_ method while trying to load
> data of the single image bitmap in given ico file.
> While _fileSize_ value is 5990, the _bitmapDataOffset_ value parsed is
> 989265922. At the same time _expectedDataOffset_ value is 70, which leads to
> _extraBytes_ value of 989265852.
> Then
> _org.apache.commons.imaging.common.BinaryFunctions#readBytes(java.lang.String,
> java.io.InputStream, int, java.lang.String)_ tries to allocate byte array of
> size 989265852. Which exhausts available memory of the JVM.
> Maybe just a simple check of _extraBytes > bhi.fileSize_ could help?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] Update the tests to JUnit 5 [commons-cli]
DamnedElric commented on PR #238: URL: https://github.com/apache/commons-cli/pull/238#issuecomment-1968584469 > @DamnedElric Yes, let's please migrate _away_ from the vintage layer. I want to avoid anything non-version 5 from sneaking back in. @garydgregory Done :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
