[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=598072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-598072 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 17/May/21 18:29 Start Date: 17/May/21 18:29 Worklog Time Spent: 10m Work Description: garydgregory closed pull request #175: URL: https://github.com/apache/commons-io/pull/175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 598072) Time Spent: 3h 20m (was: 3h 10m) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Fix For: 2.9.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=598018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-598018 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 17/May/21 18:24 Start Date: 17/May/21 18:24 Worklog Time Spent: 10m Work Description: leskin-in commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-841833386 @garydgregory , [that](https://github.com/apache/commons-io/commit/7e0d317ac61427e6004372622df4014c1e602487) is a nice solution. As `ThresholdingOutputStream.written` is `long`, [`ThresholdingOutputStream.checkThreshold()`](https://github.com/apache/commons-io/blob/587a0354f0581fb6e32dda0ef14ca8f07c945c2d/src/main/java/org/apache/commons/io/output/ThresholdingOutputStream.java#L94-L107) works fine for `Integer.MAX_VALUE`. Should this PR be closed then? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 598018) Time Spent: 3h 10m (was: 3h) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Fix For: 2.9.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=597837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597837 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 17/May/21 18:01 Start Date: 17/May/21 18:01 Worklog Time Spent: 10m Work Description: garydgregory commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-841824967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597837) Time Spent: 3h (was: 2h 50m) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Fix For: 2.9.0 > > Time Spent: 3h > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=597418=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597418 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 16/May/21 18:54 Start Date: 16/May/21 18:54 Worklog Time Spent: 10m Work Description: garydgregory closed pull request #175: URL: https://github.com/apache/commons-io/pull/175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597418) Time Spent: 2h 50m (was: 2h 40m) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Fix For: 2.9.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=597417=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597417 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 16/May/21 18:54 Start Date: 16/May/21 18:54 Worklog Time Spent: 10m Work Description: garydgregory commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-841860619 A different solution is in git master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597417) Time Spent: 2h 40m (was: 2.5h) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Fix For: 2.9.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=597392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597392 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 16/May/21 15:33 Start Date: 16/May/21 15:33 Worklog Time Spent: 10m Work Description: leskin-in commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-841833386 @garydgregory , [that](https://github.com/apache/commons-io/commit/7e0d317ac61427e6004372622df4014c1e602487) is a nice solution. As `ThresholdingOutputStream.written` is `long`, [`ThresholdingOutputStream.checkThreshold()`](https://github.com/apache/commons-io/blob/587a0354f0581fb6e32dda0ef14ca8f07c945c2d/src/main/java/org/apache/commons/io/output/ThresholdingOutputStream.java#L94-L107) works fine for `Integer.MAX_VALUE`. Should this PR be closed then? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597392) Time Spent: 2.5h (was: 2h 20m) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Fix For: 2.9.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=597387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597387 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 16/May/21 14:29 Start Date: 16/May/21 14:29 Worklog Time Spent: 10m Work Description: garydgregory commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-841825015 @leskin-in Note that GitHub Actions are currently down. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597387) Time Spent: 2h 20m (was: 2h 10m) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=597386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597386 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 16/May/21 14:28 Start Date: 16/May/21 14:28 Worklog Time Spent: 10m Work Description: garydgregory commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-841824967 @leskin-in Please see git master. I've solved this differently which also has the benefit of NOT consuming the whole input stream when the next read would result in the byte array being larger than `Integer.MAX_VALUE`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597386) Time Spent: 2h 10m (was: 2h) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[
https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=596514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-596514
]
ASF GitHub Bot logged work on IO-429:
-
Author: ASF GitHub Bot
Created on: 14/May/21 06:08
Start Date: 14/May/21 06:08
Worklog Time Spent: 10m
Work Description: leskin-in commented on a change in pull request #175:
URL: https://github.com/apache/commons-io/pull/175#discussion_r632306989
##
File path: pom.xml
##
@@ -384,7 +384,7 @@ file comparators, endian transformation classes, and much
more.
1
false
- ${argLine} -Xmx25M
+ ${argLine} -Xmx4223M
Review comment:
[`IOUtilsTestCase` fails on lack of
memory](https://travis-ci.com/github/apache/commons-io/jobs/505247713#L549).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 596514)
Time Spent: 2h (was: 1h 50m)
> ByteArrayOutputStream can overflow
> --
>
> Key: IO-429
> URL: https://issues.apache.org/jira/browse/IO-429
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Reporter: Fabian Lange
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> There are many places involved in the problem, and a good fix might be
> problematic performance wise.
> For example:
> IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more
> than Integer.MAX_VALUE bytes will either crash with
> NegativeArraySizeException or maybe worse overflow in such a way that it
> returns fine (but only with partial data)
> The ByteArrayOutputStream will happily consume the full stream but "int
> count" will overflow. At some point then toByteArray is invoked which will do
> like new byte[count].
> maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the
> count and could check for the overflow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=596331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-596331 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 13/May/21 19:46 Start Date: 13/May/21 19:46 Worklog Time Spent: 10m Work Description: leskin-in commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-840790996 > @leskin-in > May you please rebase on master and see if you can get green builds? @garydgregory, thank you for reminding about this PR. I have rebased it against [the current `master`](https://github.com/apache/commons-io/tree/bae40a557de13b3f0924a13a24715ee78b65de1b) and took into account the notes from your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 596331) Time Spent: 1h 50m (was: 1h 40m) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[
https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=596330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-596330
]
ASF GitHub Bot logged work on IO-429:
-
Author: ASF GitHub Bot
Created on: 13/May/21 19:34
Start Date: 13/May/21 19:34
Worklog Time Spent: 10m
Work Description: leskin-in commented on a change in pull request #175:
URL: https://github.com/apache/commons-io/pull/175#discussion_r632054682
##
File path: pom.xml
##
@@ -384,7 +384,7 @@ file comparators, endian transformation classes, and much
more.
1
false
- ${argLine} -Xmx25M
+ ${argLine} -Xmx4223M
Review comment:
Left unchanged in
https://github.com/apache/commons-io/pull/175/commits/cc28d3daccafad190c3002f0a74baabd4c234edc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 596330)
Time Spent: 1h 40m (was: 1.5h)
> ByteArrayOutputStream can overflow
> --
>
> Key: IO-429
> URL: https://issues.apache.org/jira/browse/IO-429
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Reporter: Fabian Lange
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> There are many places involved in the problem, and a good fix might be
> problematic performance wise.
> For example:
> IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more
> than Integer.MAX_VALUE bytes will either crash with
> NegativeArraySizeException or maybe worse overflow in such a way that it
> returns fine (but only with partial data)
> The ByteArrayOutputStream will happily consume the full stream but "int
> count" will overflow. At some point then toByteArray is invoked which will do
> like new byte[count].
> maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the
> count and could check for the overflow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[
https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=596329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-596329
]
ASF GitHub Bot logged work on IO-429:
-
Author: ASF GitHub Bot
Created on: 13/May/21 19:34
Start Date: 13/May/21 19:34
Worklog Time Spent: 10m
Work Description: leskin-in commented on a change in pull request #175:
URL: https://github.com/apache/commons-io/pull/175#discussion_r632054513
##
File path: src/test/java/org/apache/commons/io/IOUtilsTestCase.java
##
@@ -1331,6 +1332,15 @@ public void testCopyLarge_SkipWithInvalidOffset() throws
IOException {
}
}
+@Test public void testToByteArray_InputStreamTooLong() throws Exception {
+try (CircularInputStream cin = new CircularInputStream(new byte[]{65,
65, 65}, ((long)Integer.MAX_VALUE) + 1L)) {
+IOUtils.toByteArray(cin);
+fail("IllegalArgumentException expected");
+} catch (final IllegalArgumentException exc) {
Review comment:
Thanks. Fixed in
https://github.com/apache/commons-io/pull/175/commits/cc28d3daccafad190c3002f0a74baabd4c234edc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 596329)
Time Spent: 1.5h (was: 1h 20m)
> ByteArrayOutputStream can overflow
> --
>
> Key: IO-429
> URL: https://issues.apache.org/jira/browse/IO-429
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Reporter: Fabian Lange
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> There are many places involved in the problem, and a good fix might be
> problematic performance wise.
> For example:
> IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more
> than Integer.MAX_VALUE bytes will either crash with
> NegativeArraySizeException or maybe worse overflow in such a way that it
> returns fine (but only with partial data)
> The ByteArrayOutputStream will happily consume the full stream but "int
> count" will overflow. At some point then toByteArray is invoked which will do
> like new byte[count].
> maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the
> count and could check for the overflow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[
https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=596327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-596327
]
ASF GitHub Bot logged work on IO-429:
-
Author: ASF GitHub Bot
Created on: 13/May/21 19:32
Start Date: 13/May/21 19:32
Worklog Time Spent: 10m
Work Description: leskin-in commented on a change in pull request #175:
URL: https://github.com/apache/commons-io/pull/175#discussion_r632053610
##
File path: src/main/java/org/apache/commons/io/IOUtils.java
##
@@ -2243,10 +2243,13 @@ public static BufferedReader toBufferedReader(final
Reader reader, final int siz
* @param input the InputStream to read from
* @return the requested byte array
* @throws IOException if an I/O error occurs
+ * @throws IllegalArgumentException if input is longer than the maximum
Java array length
*/
public static byte[] toByteArray(final InputStream input) throws
IOException {
try (final ByteArrayOutputStream output = new ByteArrayOutputStream())
{
-copy(input, output);
+if (copy(input, output) == -1) {
+throw new IllegalArgumentException("Stream cannot be longer
than Integer max value bytes");
Review comment:
That is correct if an `InputStream` is a `ByteArrayInputStream`.
However, this method accepts generic `InputStream`, which may wrap more data.
In that case, the implementation of
[`copy()`](https://github.com/apache/commons-io/blob/4dc7b2462ef0b6345828a13d358e34bfc9309ce2/src/main/java/org/apache/commons/io/IOUtils.java#L842-L869)
would return `-1`.
However, `ByteArrayOutputStream`, which is created in this method as an
intermediate buffer, does not check for its *underlying* buffer overflow at
[`write()`](https://github.com/apache/commons-io/blob/b2165b7bbe8500768b6e27e090f89a621510/src/main/java/org/apache/commons/io/output/ByteArrayOutputStream.java#L54-L68).
The checks in `write()` only ensure sanity of arguments passed to it; they are
valid in case of
[`copyLarge()`](https://github.com/apache/commons-io/blob/4dc7b2462ef0b6345828a13d358e34bfc9309ce2/src/main/java/org/apache/commons/io/IOUtils.java#L1148-L1174)
(ultimately called by `copy()` mentioned above).
The `ByteArrayOutputStream`, however, can store more than
`Integer.MAX_VALUE` bytes because it [can use multiple underlying byte
arrays](https://github.com/apache/commons-io/blob/401d17349e7ec52d8fa866c35efd24103f332c29/src/main/java/org/apache/commons/io/output/AbstractByteArrayOutputStream.java#L79-L109).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 596327)
Time Spent: 1h 20m (was: 1h 10m)
> ByteArrayOutputStream can overflow
> --
>
> Key: IO-429
> URL: https://issues.apache.org/jira/browse/IO-429
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Reporter: Fabian Lange
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> There are many places involved in the problem, and a good fix might be
> problematic performance wise.
> For example:
> IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more
> than Integer.MAX_VALUE bytes will either crash with
> NegativeArraySizeException or maybe worse overflow in such a way that it
> returns fine (but only with partial data)
> The ByteArrayOutputStream will happily consume the full stream but "int
> count" will overflow. At some point then toByteArray is invoked which will do
> like new byte[count].
> maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the
> count and could check for the overflow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=596089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-596089 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 13/May/21 12:52 Start Date: 13/May/21 12:52 Worklog Time Spent: 10m Work Description: garydgregory commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-840534277 @leskin-in May you please rebase on master and see if you can get green builds? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 596089) Time Spent: 1h 10m (was: 1h) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[
https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=533682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533682
]
ASF GitHub Bot logged work on IO-429:
-
Author: ASF GitHub Bot
Created on: 10/Jan/21 04:08
Start Date: 10/Jan/21 04:08
Worklog Time Spent: 10m
Work Description: garydgregory commented on a change in pull request #175:
URL: https://github.com/apache/commons-io/pull/175#discussion_r544754329
##
File path: pom.xml
##
@@ -384,7 +384,7 @@ file comparators, endian transformation classes, and much
more.
1
false
- ${argLine} -Xmx25M
+ ${argLine} -Xmx4223M
Review comment:
Not acceptable, leave as is for now.
##
File path: src/main/java/org/apache/commons/io/IOUtils.java
##
@@ -2243,10 +2243,13 @@ public static BufferedReader toBufferedReader(final
Reader reader, final int siz
* @param input the InputStream to read from
* @return the requested byte array
* @throws IOException if an I/O error occurs
+ * @throws IllegalArgumentException if input is longer than the maximum
Java array length
*/
public static byte[] toByteArray(final InputStream input) throws
IOException {
try (final ByteArrayOutputStream output = new ByteArrayOutputStream())
{
-copy(input, output);
+if (copy(input, output) == -1) {
+throw new IllegalArgumentException("Stream cannot be longer
than Integer max value bytes");
Review comment:
@leskin-in
This condition will not happen for a `ByteArrayInputStream`, instead of
you'll get a `IndexOutOfBoundsException` or am I missing something?
##
File path: src/test/java/org/apache/commons/io/IOUtilsTestCase.java
##
@@ -1331,6 +1332,15 @@ public void testCopyLarge_SkipWithInvalidOffset() throws
IOException {
}
}
+@Test public void testToByteArray_InputStreamTooLong() throws Exception {
+try (CircularInputStream cin = new CircularInputStream(new byte[]{65,
65, 65}, ((long)Integer.MAX_VALUE) + 1L)) {
+IOUtils.toByteArray(cin);
+fail("IllegalArgumentException expected");
+} catch (final IllegalArgumentException exc) {
Review comment:
Use assertThrows()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 533682)
Time Spent: 1h (was: 50m)
> ByteArrayOutputStream can overflow
> --
>
> Key: IO-429
> URL: https://issues.apache.org/jira/browse/IO-429
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Reporter: Fabian Lange
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There are many places involved in the problem, and a good fix might be
> problematic performance wise.
> For example:
> IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more
> than Integer.MAX_VALUE bytes will either crash with
> NegativeArraySizeException or maybe worse overflow in such a way that it
> returns fine (but only with partial data)
> The ByteArrayOutputStream will happily consume the full stream but "int
> count" will overflow. At some point then toByteArray is invoked which will do
> like new byte[count].
> maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the
> count and could check for the overflow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[
https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=533113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533113
]
ASF GitHub Bot logged work on IO-429:
-
Author: ASF GitHub Bot
Created on: 08/Jan/21 16:39
Start Date: 08/Jan/21 16:39
Worklog Time Spent: 10m
Work Description: garydgregory commented on a change in pull request #175:
URL: https://github.com/apache/commons-io/pull/175#discussion_r544754329
##
File path: pom.xml
##
@@ -384,7 +384,7 @@ file comparators, endian transformation classes, and much
more.
1
false
- ${argLine} -Xmx25M
+ ${argLine} -Xmx4223M
Review comment:
Not acceptable, leave as is for now.
##
File path: src/main/java/org/apache/commons/io/IOUtils.java
##
@@ -2243,10 +2243,13 @@ public static BufferedReader toBufferedReader(final
Reader reader, final int siz
* @param input the InputStream to read from
* @return the requested byte array
* @throws IOException if an I/O error occurs
+ * @throws IllegalArgumentException if input is longer than the maximum
Java array length
*/
public static byte[] toByteArray(final InputStream input) throws
IOException {
try (final ByteArrayOutputStream output = new ByteArrayOutputStream())
{
-copy(input, output);
+if (copy(input, output) == -1) {
+throw new IllegalArgumentException("Stream cannot be longer
than Integer max value bytes");
Review comment:
@leskin-in
This condition will not happen for a `ByteArrayInputStream`, instead of
you'll get a `IndexOutOfBoundsException` or am I missing something?
##
File path: src/test/java/org/apache/commons/io/IOUtilsTestCase.java
##
@@ -1331,6 +1332,15 @@ public void testCopyLarge_SkipWithInvalidOffset() throws
IOException {
}
}
+@Test public void testToByteArray_InputStreamTooLong() throws Exception {
+try (CircularInputStream cin = new CircularInputStream(new byte[]{65,
65, 65}, ((long)Integer.MAX_VALUE) + 1L)) {
+IOUtils.toByteArray(cin);
+fail("IllegalArgumentException expected");
+} catch (final IllegalArgumentException exc) {
Review comment:
Use assertThrows()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 533113)
Time Spent: 50m (was: 40m)
> ByteArrayOutputStream can overflow
> --
>
> Key: IO-429
> URL: https://issues.apache.org/jira/browse/IO-429
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Reporter: Fabian Lange
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> There are many places involved in the problem, and a good fix might be
> problematic performance wise.
> For example:
> IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more
> than Integer.MAX_VALUE bytes will either crash with
> NegativeArraySizeException or maybe worse overflow in such a way that it
> returns fine (but only with partial data)
> The ByteArrayOutputStream will happily consume the full stream but "int
> count" will overflow. At some point then toByteArray is invoked which will do
> like new byte[count].
> maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the
> count and could check for the overflow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=525244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-525244 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 16/Dec/20 20:28 Start Date: 16/Dec/20 20:28 Worklog Time Spent: 10m Work Description: leskin-in commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-746962320 @garydgregory thank you for your [comment](https://github.com/apache/commons-io/pull/175#issuecomment-745740037). I have [implemented](https://github.com/apache/commons-io/pull/175/commits/361765a7fecae0d419828f84de3b825fd77dfa50) a test using [`CircularInputStream`](https://github.com/apache/commons-io/blob/d4e09c7b4ba4a1433feb7e5d4895cd1071e51bb1/src/main/java/org/apache/commons/io/input/CircularInputStream.java); [`IOUtils.copyLarge()`](https://github.com/apache/commons-io/blob/9e71df2a6d879e76f69ffcc2d956dfd5d42f0ba9/src/main/java/org/apache/commons/io/IOUtils.java#L1163) does not return if provided `InifiniteCircularInputStream` as input. Note the new test case requires lots of memory, so heap size limit is [increased](https://github.com/apache/commons-io/pull/175/commits/361765a7fecae0d419828f84de3b825fd77dfa50#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8R387). I do not know whether this is appropriate. The chosen value is the lowest possible, which I obtained empirically on my laptop. [IO-161](https://issues.apache.org/jira/browse/IO-161) introduced heap size limit, and it has remained unchanged since then. But the reasons for this (and why `25M` was chosen) are not clear to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 525244) Time Spent: 40m (was: 0.5h) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=524811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-524811 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 16/Dec/20 03:28 Start Date: 16/Dec/20 03:28 Worklog Time Spent: 10m Work Description: garydgregory commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-745740037 Why would you use a real file? Just test with a `InfiniteCircularInputStream`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 524811) Time Spent: 0.5h (was: 20m) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=524660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-524660 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 15/Dec/20 19:27 Start Date: 15/Dec/20 19:27 Worklog Time Spent: 10m Work Description: leskin-in commented on pull request #175: URL: https://github.com/apache/commons-io/pull/175#issuecomment-745514116 Adding a test for this change (e.g. to [`IOUtilsTestCase`](https://github.com/apache/commons-io/blob/5dd0f87c8e37c7838ef336e20307caf24b61c329/src/test/java/org/apache/commons/io/IOUtilsTestCase.java#L1324)) seems quite complicated: `ByteArrayInputStream`, and `BufferedInputStream` both use common Java arrays, whose length is not enough for the test. A solution I can think of is to use `FileInputStream`; but this would require the test case to create a 2Gb file, which IMHO does not look good. It is also possible to implement a custom `InputStream`. It may turn to be useful in other tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 524660) Time Spent: 20m (was: 10m) > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (IO-429) ByteArrayOutputStream can overflow
[ https://issues.apache.org/jira/browse/IO-429?focusedWorklogId=524659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-524659 ] ASF GitHub Bot logged work on IO-429: - Author: ASF GitHub Bot Created on: 15/Dec/20 19:24 Start Date: 15/Dec/20 19:24 Worklog Time Spent: 10m Work Description: leskin-in opened a new pull request #175: URL: https://github.com/apache/commons-io/pull/175 Throw an `IllegalArgumentException` when an `InputStream` provided to `IOUtils.toByteArray()` is longer than `Integer.MAX_VALUE` bytes. Processing of such long arrays is not possible, as arrays with `long` indices are [forbidden](https://docs.oracle.com/javase/specs/jls/se14/html/jls-10.html#jls-10.4) by Java language specification. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 524659) Remaining Estimate: 0h Time Spent: 10m > ByteArrayOutputStream can overflow > -- > > Key: IO-429 > URL: https://issues.apache.org/jira/browse/IO-429 > Project: Commons IO > Issue Type: Bug > Components: Utilities >Reporter: Fabian Lange >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > There are many places involved in the problem, and a good fix might be > problematic performance wise. > For example: > IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more > than Integer.MAX_VALUE bytes will either crash with > NegativeArraySizeException or maybe worse overflow in such a way that it > returns fine (but only with partial data) > The ByteArrayOutputStream will happily consume the full stream but "int > count" will overflow. At some point then toByteArray is invoked which will do > like new byte[count]. > maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets the > count and could check for the overflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
