Piotr Karwasz created COMPRESS-731:
--------------------------------------

             Summary: Document that CompressorStreamFactory and the LZMA/XZ 
input streams have no memory limit by default
                 Key: COMPRESS-731
                 URL: https://issues.apache.org/jira/browse/COMPRESS-731
             Project: Commons Compress
          Issue Type: Improvement
          Components: Documentation
            Reporter: Piotr Karwasz


LZMA and XZ decompression allocate a dictionary whose size is declared in the 
stream: up to about 2 GiB (xz-java's {{DICT_SIZE_MAX}}). xz-java's 
{{LZMAInputStream}} allocates it when the stream is opened, unless the header 
declares a smaller uncompressed size; the XZ decoder allocates it when the 
first block is read. Commons Compress lets applications bound this through:
* {{CompressorStreamFactory(boolean, int)}}
* {{LZMACompressorInputStream.Builder.setMemoryLimitKiB(int)}}
* {{XZCompressorInputStream.Builder.setMemoryLimitKiB(int)}}

With a limit set, xz-java throws a {{MemoryLimitException}} before allocating 
the dictionary.

The documentation does not say that the defaults are unbounded, and the 
examples page is out of date:
* {{new CompressorStreamFactory()}} and {{new 
CompressorStreamFactory(boolean)}} use no memory limit ({{-1}}). Their Javadoc 
doesn't say so, and the {{memoryLimitInKb}} parameter doesn't explain what 
{{-1}} means.
* The builders of {{LZMACompressorInputStream}} and {{XZCompressorInputStream}} 
also default to {{-1}}. The Javadoc examples for {{XZCompressorInputStream}} 
show {{.setMemoryLimitKiB(-1)}} without explaining that it disables the check.
* Only {{-1}} means "no limit" in xz-java, and other negative values behave 
differently between the formats: {{LZMAInputStream}} rejects every stream, 
while the XZ decoder treats any negative value as no limit.
* {{src/site/xdoc/examples.xml}} ("Compressors" section) says: "Note that there 
is no way to detect the LZMA or Brotli formats so only the two-arg version of 
{{createCompressorInputStream}} can be used." LZMA has been auto-detected since 
1.10 (COMPRESS-313), so a {{.lzma}} stream passed to the one-argument method is 
opened with the factory's limit, which is none by default.
* The "Restricting Memory Usage" subsection describes the 1.14 state and 
doesn't recommend setting a limit when reading untrusted input.

Proposed changes:

h1. State in the Javadoc of the {{CompressorStreamFactory}} constructors, and 
of the LZMA and XZ builders' {{setMemoryLimitKiB}}, that the default is 
unbounded and that {{-1}} disables the check.
h1. Fix the LZMA auto-detection note in {{examples.xml}}.
h1. In "Restricting Memory Usage", recommend setting a limit that fits the 
available heap when decompressing untrusted data, with an example.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to