[ 
https://issues.apache.org/jira/browse/COMPRESS-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Karwasz updated COMPRESS-731:
-----------------------------------
    Description: 
LZMA and XZ decompression allocate a dictionary whose size is declared in the 
stream: up to about 2 GiB (xz-java's {{DICT_SIZE_MAX}}). xz-java's 
{{LZMAInputStream}} allocates it when the stream is opened, unless the header 
declares a smaller uncompressed size; the XZ decoder allocates it when the 
first block is read. Commons Compress lets applications bound this through:
* {{CompressorStreamFactory(boolean, int)}}
* {{LZMACompressorInputStream.Builder.setMemoryLimitKiB(int)}}
* {{XZCompressorInputStream.Builder.setMemoryLimitKiB(int)}}

With a limit set, xz-java throws a {{MemoryLimitException}} before allocating 
the dictionary.

The documentation does not say that the defaults are unbounded, and the 
examples page is out of date:
* {{new CompressorStreamFactory()}} and {{new 
CompressorStreamFactory(boolean)}} use no memory limit ({{-1}}). Their Javadoc 
doesn't say so, and the {{memoryLimitInKb}} parameter doesn't explain what 
{{-1}} means.
* The builders of {{LZMACompressorInputStream}} and {{XZCompressorInputStream}} 
also default to {{-1}}. The Javadoc examples for {{XZCompressorInputStream}} 
show {{.setMemoryLimitKiB(-1)}} without explaining that it disables the check.
* Only {{-1}} means "no limit" in xz-java, and other negative values behave 
differently between the formats: {{LZMAInputStream}} rejects every stream, 
while the XZ decoder treats any negative value as no limit.
* {{src/site/xdoc/examples.xml}} ("Compressors" section) says: "Note that there 
is no way to detect the LZMA or Brotli formats so only the two-arg version of 
{{createCompressorInputStream}} can be used." LZMA has been auto-detected since 
1.10 (COMPRESS-313), so a {{.lzma}} stream passed to the one-argument method is 
opened with the factory's limit, which is none by default.
* The "Restricting Memory Usage" subsection describes the 1.14 state and 
doesn't recommend setting a limit when reading untrusted input.

Proposed changes:
# State in the Javadoc of the {{CompressorStreamFactory}} constructors, and of 
the LZMA and XZ builders' {{setMemoryLimitKiB}}, that the default is unbounded 
and that {{-1}} disables the check.
# Fix the LZMA auto-detection note in {{examples.xml}}.
# In "Restricting Memory Usage", recommend setting a limit that fits the 
available heap when decompressing untrusted data, with an example.

  was:
LZMA and XZ decompression allocate a dictionary whose size is declared in the 
stream: up to about 2 GiB (xz-java's {{DICT_SIZE_MAX}}). xz-java's 
{{LZMAInputStream}} allocates it when the stream is opened, unless the header 
declares a smaller uncompressed size; the XZ decoder allocates it when the 
first block is read. Commons Compress lets applications bound this through:
* {{CompressorStreamFactory(boolean, int)}}
* {{LZMACompressorInputStream.Builder.setMemoryLimitKiB(int)}}
* {{XZCompressorInputStream.Builder.setMemoryLimitKiB(int)}}

With a limit set, xz-java throws a {{MemoryLimitException}} before allocating 
the dictionary.

The documentation does not say that the defaults are unbounded, and the 
examples page is out of date:
* {{new CompressorStreamFactory()}} and {{new 
CompressorStreamFactory(boolean)}} use no memory limit ({{-1}}). Their Javadoc 
doesn't say so, and the {{memoryLimitInKb}} parameter doesn't explain what 
{{-1}} means.
* The builders of {{LZMACompressorInputStream}} and {{XZCompressorInputStream}} 
also default to {{-1}}. The Javadoc examples for {{XZCompressorInputStream}} 
show {{.setMemoryLimitKiB(-1)}} without explaining that it disables the check.
* Only {{-1}} means "no limit" in xz-java, and other negative values behave 
differently between the formats: {{LZMAInputStream}} rejects every stream, 
while the XZ decoder treats any negative value as no limit.
* {{src/site/xdoc/examples.xml}} ("Compressors" section) says: "Note that there 
is no way to detect the LZMA or Brotli formats so only the two-arg version of 
{{createCompressorInputStream}} can be used." LZMA has been auto-detected since 
1.10 (COMPRESS-313), so a {{.lzma}} stream passed to the one-argument method is 
opened with the factory's limit, which is none by default.
* The "Restricting Memory Usage" subsection describes the 1.14 state and 
doesn't recommend setting a limit when reading untrusted input.

Proposed changes:

h1. State in the Javadoc of the {{CompressorStreamFactory}} constructors, and 
of the LZMA and XZ builders' {{setMemoryLimitKiB}}, that the default is 
unbounded and that {{-1}} disables the check.
h1. Fix the LZMA auto-detection note in {{examples.xml}}.
h1. In "Restricting Memory Usage", recommend setting a limit that fits the 
available heap when decompressing untrusted data, with an example.


> Document that CompressorStreamFactory and the LZMA/XZ input streams have no 
> memory limit by default
> ---------------------------------------------------------------------------------------------------
>
>                 Key: COMPRESS-731
>                 URL: https://issues.apache.org/jira/browse/COMPRESS-731
>             Project: Commons Compress
>          Issue Type: Improvement
>          Components: Documentation
>            Reporter: Piotr Karwasz
>            Priority: Major
>
> LZMA and XZ decompression allocate a dictionary whose size is declared in the 
> stream: up to about 2 GiB (xz-java's {{DICT_SIZE_MAX}}). xz-java's 
> {{LZMAInputStream}} allocates it when the stream is opened, unless the header 
> declares a smaller uncompressed size; the XZ decoder allocates it when the 
> first block is read. Commons Compress lets applications bound this through:
> * {{CompressorStreamFactory(boolean, int)}}
> * {{LZMACompressorInputStream.Builder.setMemoryLimitKiB(int)}}
> * {{XZCompressorInputStream.Builder.setMemoryLimitKiB(int)}}
> With a limit set, xz-java throws a {{MemoryLimitException}} before allocating 
> the dictionary.
> The documentation does not say that the defaults are unbounded, and the 
> examples page is out of date:
> * {{new CompressorStreamFactory()}} and {{new 
> CompressorStreamFactory(boolean)}} use no memory limit ({{-1}}). Their 
> Javadoc doesn't say so, and the {{memoryLimitInKb}} parameter doesn't explain 
> what {{-1}} means.
> * The builders of {{LZMACompressorInputStream}} and 
> {{XZCompressorInputStream}} also default to {{-1}}. The Javadoc examples for 
> {{XZCompressorInputStream}} show {{.setMemoryLimitKiB(-1)}} without 
> explaining that it disables the check.
> * Only {{-1}} means "no limit" in xz-java, and other negative values behave 
> differently between the formats: {{LZMAInputStream}} rejects every stream, 
> while the XZ decoder treats any negative value as no limit.
> * {{src/site/xdoc/examples.xml}} ("Compressors" section) says: "Note that 
> there is no way to detect the LZMA or Brotli formats so only the two-arg 
> version of {{createCompressorInputStream}} can be used." LZMA has been 
> auto-detected since 1.10 (COMPRESS-313), so a {{.lzma}} stream passed to the 
> one-argument method is opened with the factory's limit, which is none by 
> default.
> * The "Restricting Memory Usage" subsection describes the 1.14 state and 
> doesn't recommend setting a limit when reading untrusted input.
> Proposed changes:
> # State in the Javadoc of the {{CompressorStreamFactory}} constructors, and 
> of the LZMA and XZ builders' {{setMemoryLimitKiB}}, that the default is 
> unbounded and that {{-1}} disables the check.
> # Fix the LZMA auto-detection note in {{examples.xml}}.
> # In "Restricting Memory Usage", recommend setting a limit that fits the 
> available heap when decompressing untrusted data, with an example.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to