[issue21872] LZMA library sometimes fails to decompress a file

Ma Lin added the comment: Some memos: 1, In liblzma, these missing bytes were copied inside `dict_repeat` function: 788 case SEQ_COPY: 789 // Repeat len bytes from distance of rep0. 790 if (unlikely(dict_repeat(, rep0, ))) { See liblzma's source

[issue21872] LZMA library sometimes fails to decompress a file

Gregory P. Smith added the comment: thanks! -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker ___

[issue21872] LZMA library sometimes fails to decompress a file

miss-islington added the comment: New changeset a3c53a1b45b05bcb69660eac5a271443b37ecc42 by Miss Islington (bot) in branch '3.7': bpo-21872: fix lzma library decompresses data incompletely (GH-14048) https://github.com/python/cpython/commit/a3c53a1b45b05bcb69660eac5a271443b37ecc42

[issue21872] LZMA library sometimes fails to decompress a file

miss-islington added the comment: New changeset 824407f76e211a2a19c94d5cb1f39fc422ab62ee by Miss Islington (bot) in branch '3.8': bpo-21872: fix lzma library decompresses data incompletely (GH-14048) https://github.com/python/cpython/commit/824407f76e211a2a19c94d5cb1f39fc422ab62ee

[issue21872] LZMA library sometimes fails to decompress a file

Gregory P. Smith added the comment: New changeset 4ffd05d7ec47cfd0d7fc95dce851633be9663255 by Gregory P. Smith (animalize) in branch 'master': bpo-21872: fix lzma library decompresses data incompletely (GH-14048)

[issue21872] LZMA library sometimes fails to decompress a file

Change by miss-islington : -- pull_requests: +15678 pull_request: https://github.com/python/cpython/pull/16055 ___ Python tracker ___

[issue21872] LZMA library sometimes fails to decompress a file

Change by miss-islington : -- pull_requests: +15677 pull_request: https://github.com/python/cpython/pull/16054 ___ Python tracker ___

[issue21872] LZMA library sometimes fails to decompress a file

Ma Lin added the comment: toggle conditions -> trigger conditions -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue21872] LZMA library sometimes fails to decompress a file

Ma Lin added the comment: I investigated this problem. Here is the toggle conditions: - The format is FORMAT_ALONE, this is the legacy .lzma container format. - The file's header recorded "Uncompressed Size". - The file doesn't have "End of Payload Marker" or "End of Stream Marker".

[issue21872] LZMA library sometimes fails to decompress a file

Ma Lin added the comment: I wrote a review guide in PR 14048. -- versions: +Python 3.8, Python 3.9 -Python 2.7, Python 3.4, Python 3.5, Python 3.6 ___ Python tracker ___

[issue21872] LZMA library sometimes fails to decompress a file

Change by Ma Lin : -- pull_requests: +13910 stage: -> patch review pull_request: https://github.com/python/cpython/pull/14048 ___ Python tracker ___

[issue21872] LZMA library sometimes fails to decompress a file

Ma Lin added the comment: fix-bug.diff fixes this bug, I will submit a PR after thoroughly understanding the problem. -- keywords: +patch Added file: https://bugs.python.org/file48391/fix-bug.diff ___ Python tracker

[issue21872] LZMA library sometimes fails to decompress a file

Jeffrey Kintscher added the comment: I adapted the example in msg221784: with open('22h_ticks_bad.bi5', 'rb') as f: infile = f.read() for i in range(1, 9000): decompressor = _lzma.LZMADecompressor() first_out = decompressor.decompress(infile[:i]) first_len = len(first_out)

[issue21872] LZMA library sometimes fails to decompress a file

Change by Ma Lin : -- nosy: +Ma Lin ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue21872] LZMA library sometimes fails to decompress a file

Change by Jeffrey Kintscher : -- nosy: +Jeffrey.Kintscher ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue21872] LZMA library sometimes fails to decompress a file

Shinjo Park added the comment: Hi, I think I encountered this bug with Ubuntu 17.10 / Python 3.6.3. The same error was triggered by Python's LZMA library, while the xz command line tool can extract the problematic file. Not sure whether there is the bug in 3.7/3.8. I am

[issue21872] LZMA library sometimes fails to decompress a file

kenorb added the comment: The same with this attached file. It fails with Python 3.5 (small buffers like 128, 255, 1023, etc.) , but it seems to work in Python 3.4 with lzma._BUFFER_SIZE = 1023. So it looks like something regressed. -- nosy: +kenorb Added file:

[issue21872] LZMA library sometimes fails to decompress a file

Akira Li added the comment: If lzma._BUFFER_SIZE is less than 2048 then all example files are decompressed successfully (at least lzma module produces the same results as xz utility) -- Added file: http://bugs.python.org/file37241/decompress-example-files.py

[issue21872] LZMA library sometimes fails to decompress a file

Changes by Akira Li 4kir4...@gmail.com: Removed file: http://bugs.python.org/file37240/decompress-example-files.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21872 ___

[issue21872] LZMA library sometimes fails to decompress a file

Akira Li added the comment: @Esa changing the buffer size helps with some bad files but lzma module still fails on some files. I've uploaded decompress-example-files.py script that demonstrates it. -- nosy: +akira Added file: http://bugs.python.org/file37239/decompress-example-files.py

[issue21872] LZMA library sometimes fails to decompress a file

Changes by Akira Li 4kir4...@gmail.com: Added file: http://bugs.python.org/file37240/decompress-example-files.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21872 ___

[issue21872] LZMA library sometimes fails to decompress a file

Changes by Akira Li 4kir4...@gmail.com: Removed file: http://bugs.python.org/file37239/decompress-example-files.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21872 ___

[issue21872] LZMA library sometimes fails to decompress a file

Changes by Peter p.j.a.c...@googlemail.com: -- nosy: +maubp ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21872 ___ ___ Python-bugs-list mailing

[issue21872] LZMA library sometimes fails to decompress a file

Ville Nummela added the comment: Uploading a few more 'bad' lzma files for testing. -- Added file: http://bugs.python.org/file35822/more_bad_lzma_files.zip ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21872

[issue21872] LZMA library sometimes fails to decompress a file

Esa Peuha added the comment: This code import _lzma with open('22h_ticks_bad.bi5', 'rb') as f: infile = f.read() for i in range(8191, 8195): decompressor = _lzma.LZMADecompressor() first_out = decompressor.decompress(infile[:i]) first_len = len(first_out) last_out =

[issue21872] LZMA library sometimes fails to decompress a file

Serhiy Storchaka added the comment: import lzma f = lzma.open('22h_ticks_bad.bi5') len(f.read()) Traceback (most recent call last): File stdin, line 1, in module File /home/serhiy/py/cpython/Lib/lzma.py, line 310, in read return self._read_all() File

[issue21872] LZMA library sometimes fails to decompress a file

Ville Nummela added the comment: My stats so far: As of writing this, I have attempted to decompress about 5000 downloaded files (two years of tick data). 25 'bad' files were found within this lot. I re-downloaded all of them, plus about 500 other files as the minimum lot the server supplies

[issue21872] LZMA library sometimes fails to decompress a file

New submission from Ville Nummela: Python lzma library sometimes fails to decompress a file, even though the file does not appear to be corrupt. Originally discovered with OS X 10.9 / Python 2.7.7 / bacports.lzma Now also reproduced on OS X / Python 3.4 / lzma, please see

[issue21872] LZMA library sometimes fails to decompress a file

Josh Rosenberg added the comment: Just to be clear, when you say 1 - 5 times per 1000 downloaded files, have you confirmed that redownloading the same file a second time produces the same error? Just making sure we've ruled out corruption during transfer over the network; small errors might