> > Testing is welcome. It would be nice if someone who has 12-24 hardware > > threads could test if it scales well. One needs a file with like a > > hundred blocks, so with the default xz -6 that means a 2.5 gigabyte > > uncompressed file, smaller if one uses, for example, --block-size=8MiB > > when compressing. > > I made > Stream Blocks CompOffset UncompOffset CompSize > UncompSize Ratio Check Padding > 1 777 0 0 2.386.777.028 > 19.540.326.400 0,122 CRC64 0
Thanks for doing this testing! Looks like the threading scales fairly well on large files like this. I don't have a machine with resources like this, so I am glad you were able to do this. > > If the input is broken, it should produce as much output as the > > single-threaded stable version does. That is, if one thread detects an > > error, the data before that point is first flushed out before the error > > is reported. This has pros and cons. It would be easy to add a flag to > > allow switching to fast error reporting for applications that don't > > care about partial output from broken files. I started writing tests in the new framework and I found one bug and one issue. If you want to check out the tests I have so far, here is a link to check out my progress: https://github.com/JiaT75/XZ_Utils_Unofficial/tree/test_multithreaded_decoder The bug is with truncated xz files. In multithreaded mode, if a file has been corrupted and is missing the end, deadlock occurs. An easy way to recreate this is by using the truncate command: truncate -s 30000 some_multiblock_file.xz And then: xz -dk --verbose some_multiblock_file.xz --threads=2 This will result in a deadlock in multithreaded decoding, but not an error in single threaded decoding. The issue is with updating the memlimit with lzma_memlimit_set. As you note in your comment in stream_decoder_mt_memconfig there is no way to update memlimit_threading. If the memlimit_stop is set very low to start, it will force memlimit_threading to be that same small value. I could see users wanting to keep memlimit_threading and memlimit_stop in sync or have memlimit_threading always be some function of memlimit_stop (maybe memlimit_stop / 2 or something). I am not sure what the best fix is for this at the moment, but I don't think having only one of these values be configurable at runtime is the best idea. Especially when the initialization forces memlimit_threading to be at most memlimit_stop (which makes sense for almost every situation). I will continue to write more tests and then review the code itself. Nice job to both of you for getting this feature as far as it is already. Jia Tan
