[xz-devel] XZ Utils 5.0.7

2014-09-20 Thread Lasse Collin
zma if the linker flags were taken from pkg-config. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

[xz-devel] Optimizing lzma_memcmplen for non-x86 processors

2014-10-13 Thread Lasse Collin
test different lzma_memcmplen implementations. The reason for using -e is that the relative improvement tends to be bigger when that option is used. On x86-64 I've seen even 25 % faster compression with some files compared to the byte-by-byte method. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] How to compress a folder with xz ?

2014-11-11 Thread Lasse Collin
you have used .zip or .7z files. With such file formats the archiving, compression, and possibly also encryption are done by a single file format and tool. When using tar with gzip/bzip2/xz/whatever the tasks are done by separate file formats and tools. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] How to compress a folder with xz ?

2014-11-11 Thread Lasse Collin
On 2014-11-11 Kevin Wilson wrote: > Is the following also good fo > tar cfJ new.xz /etc > > because: > > file new.xz > returns: > new.xz: XZ compressed data Yes although you probably want to use new.tar.xz as the file name, not new.xz. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

[xz-devel] XZ Utils 5.0.8 and 5.2.0

2014-12-21 Thread Lasse Collin
Currently it is 5.0.0 for all supported .xz files except files with empty LZMA2 streams require 5.0.2. * xzdiff and xzgrep now support .lzo files if lzop is installed. The .tzo suffix is also recognized as a shorthand for .tar.lzo. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Fairly Complete MSVC 2013 Solution/Project

2015-01-28 Thread Lasse Collin
./lib/getopt.in.h"? - I don't have an opinion if it is useful to use CMake for this. Note that there already are resource files so creating one with CMake seems redundant. I don't know if the included resource files can be used as is with MSVC though. Thanks for your help. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Fairly Complete MSVC 2013 Solution/Project

2015-01-30 Thread Lasse Collin
, although for some unfortunate reason GCC and Clang only give a warning about them instead of an error even when in C99/C11 mode. XZ Utils doesn't compile with C89 compilers which limits the possible age of the compilers. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Fairly Complete MSVC 2013 Solution/Project

2015-02-03 Thread Lasse Collin
etopt.in.h to > getopt.h? The idea of .in.h is that if configure detects that the system getopt.h isn't good enough, a replacement getopt.h will be created. The "lib" directory, which might contain getopt.h, is always in the compiler include path. My understanding is that thi

[xz-devel] XZ Utils 5.2.1

2015-02-26 Thread Lasse Collin
penBSD. * Fixed xzdiff to be compatible with FreeBSD's mktemp which differs from most other mktemp implementations. * Changed CPU core count detection to use cpuset_getaffinity() on FreeBSD. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Fairly Complete MSVC 2013 Solution/Project

2015-03-08 Thread Lasse Collin
most warnings probably won't be fixed. I'll get back to you when I have more to say, hopefully in 1-3 weeks. Sorry that this is progressing slowly. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] xz 5.2.1 multi-threaded decompress problems (mutli-threaded compress works file)

2015-05-11 Thread Lasse Collin
ible in threaded mode (it's possible to do it if the input is seekable but it's likely that it won't be implemented). -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Fairly Complete MSVC 2013 Solution/Project

2015-05-11 Thread Lasse Collin
On 2015-03-08 Lasse Collin wrote: > On 2015-02-27 Gabi Davar wrote: > > I've addressed some(all?) of the comments above and rebased on > > 5.2.1. The relevant changes are available > > https://github.com/mindw/xz in the cmake branch. > > Thanks! I finally test-buil

Re: [xz-devel] xz 5.2.1 multi-threaded decompress problems (mutli-threaded compress works file)

2015-05-12 Thread Lasse Collin
ob;f=NEWS;hb=HEAD > > 69 - Multi-threaded decompression can be enabled with the > 70 --threads (-T) option. Good catch, thanks! I have fixed it. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] minimal liblzma MSVC2013 solution and project

2015-05-26 Thread Lasse Collin
something. With the above issues fixed, I could like to commit the project files and include them in XZ Utils 5.2.2. This doesn't make the other MSVC thread deprecated yet: I should fix a few more warnings, and perhaps xz.exe could be made MSVC compatible some day too. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] minimal liblzma MSVC2013 solution and project

2015-06-11 Thread Lasse Collin
On 2015-05-28 Adam Walling wrote: > On Tue, May 26, 2015 at 2:57 PM, Lasse Collin > wrote: > > With the above issues fixed, I could like to commit the project > > files and include them in XZ Utils 5.2.2. This doesn't make the > > other MSVC thread deprecated yet: I s

Re: [xz-devel] minimal liblzma MSVC2013 solution and project

2015-06-19 Thread Lasse Collin
On 2015-06-11 Lasse Collin wrote: > On 2015-05-28 Adam Walling wrote: > > Everything is updated at https://github.com/adzm/xz_win now > > > > You can also grab them directly at http://adzm.net/xz_win/ > > Thanks! It looks good now. There are still a few unneeded

Re: [xz-devel] minimal liblzma MSVC2013 solution and project

2015-06-30 Thread Lasse Collin
unctions. So yeah, mixing runtimes on Windows is bad. It's hard to do it correctly for every detail. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] minimal liblzma MSVC2013 solution and project

2015-07-12 Thread Lasse Collin
I'm sorry that it takes so long for me to reply. :-( On 2015-06-30 Adam Walling wrote: > On Fri, Jun 19, 2015 at 1:42 PM, Lasse Collin > wrote: > > ... > > My understanding is that the symbols exported from a DLL should be > > marked with __declspec(dllexport) when

Re: [xz-devel] minimal liblzma MSVC2013 solution and project

2015-08-09 Thread Lasse Collin
On 2015-07-12 Lasse Collin wrote: > On 2015-06-30 Adam Walling wrote: > > My apologies, DLL_EXPORT must indeed be set for the dll to generate > > the lib. > > > > I attached a git diff that does this. > > Thanks. Was the "xzWinTest" thing in the patch i

[xz-devel] XZ Utils 5.2.2

2015-09-29 Thread Lasse Collin
ompatible with 5.0.x so it should be straightforward to upgrade from 5.0.x to 5.2.x. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] valgrind conditional jump depends on uninitialized value

2015-11-04 Thread Lasse Collin
k then even though the reason was known. I committed the fix to to the master branch (it will be in v5.2 before 5.2.3). -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Only single core used when trying multi-thread compression

2016-04-12 Thread Lasse Collin
v/null > /dev/null If it is a SELinux issue (or similar), it would be good to report it to the distribution maintainers. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] Remove AC_CONFIG_AUX_DIR

2016-04-21 Thread Lasse Collin
nation of the Autotools package versions you have installed. I need to understand the problem better before I will consider the patch. I suspect that AC_CONFIG_AUX_DIR should still work because it's not a deprecated feature. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] Remove AC_CONFIG_AUX_DIR

2016-04-21 Thread Lasse Collin
is patch. If there's a problem with a stable release of Autotools, then I may need to commit the patch. When the issue is only in a development snapshot, there's a chance it will get fixed before the next release. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Preserve file time on MinGW

2016-05-12 Thread Lasse Collin
would complicate a little but it would be OK too if there were no other way. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

2016-06-15 Thread Lasse Collin
e --help output but it's mentioned in INSTALL section 3.1. That is an alternative to patching to get reproducible builds. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] XZ and C99 multiple definition issue

2016-10-05 Thread Lasse Collin
k of warnings doesn't mean that there aren't any aliasing issues. src/liblzma/common/memcmplen.h is another place where similar casts are done. It's a new file in 5.2.x (not present in 5.0.x) and was added to get a bit better performance for common buffer comparisons. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

[xz-devel] XZ for Java 1.6

2016-11-27 Thread Lasse Collin
w LZMA (i.e. LZMA1) streams and to the legacy .lzma format. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

[xz-devel] xz and fsync()

2016-12-18 Thread Lasse Collin
#x27;t call fsync(). If I change it, I'm sure that some other people won't like it because their xz use case got horribly slow. So I cannot make everyone happy. Currently I'm slightly in favor of keeping the existing non-fsync() behavior unchanged. It would be nice to hear other peopl

Re: [xz-devel] xz and fsync()

2016-12-28 Thread Lasse Collin
Thanks for your comment! On 2016-12-18 John Reiser wrote: > On 12/18/2016 11:30 AM, Lasse Collin wrote: > > There's a bug report about data loss due computer losing power. > > fsync() or fdatasync() in xz would quite likely have avoided the > > data loss. > >

[xz-devel] XZ Utils 5.2.3

2016-12-30 Thread Lasse Collin
ODERS and HAVE_DECODERS. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

[xz-devel] XZ for Java 1.7

2017-12-29 Thread Lasse Collin
g many tiny files in a row, taking advantage of this API can improve performance significantly. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Automatic-Module-Name for XZ for Java?

2017-12-31 Thread Lasse Collin
nfo.java file. > As XZ uses org.tukaani.xz as its OSGi bundle name, it seems logical to > use the same name for a module. The simple change below would achieve > that. Thanks. I will add that. There needs to be a new release soon anyway to fix the issue of XZ for Java 1.7 binaries in th

Re: [xz-devel] Automatic-Module-Name for XZ for Java?

2018-01-02 Thread Lasse Collin
On 2018-01-01 Stefan Bodewig wrote: > On 2017-12-31, Lasse Collin wrote: > > Would it be too complicated to turn XZ into a proper module? How > > useful is that? > > As XZ hasn't got any dependencies you'd only benefit from an explicit > module info if you want

[xz-devel] XZ for Java 1.8

2018-01-04 Thread Lasse Collin
". Set it to 1.6 or higher. The default value 1.5 isn't supported by OpenJDK 9 or later. * Add "Automatic-Module-Name" = "org.tukaani.xz". -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] patch for Intel compiler - use of Intel compiler intrinsics e.g. bit-scan-reverse

2018-01-10 Thread Lasse Collin
sages. Thanks, but no thanks, at least for now. I know it could be useful for testing compiler-specific things, but since I already have trouble getting things done, I probably am not going spend time on setting it up etc. in the foreseeable future. > In any source file that refers to I

[xz-devel] XZ Utils 5.2.4 and 5.3.1alpha

2018-04-29 Thread Lasse Collin
Add lzma_file_info_decoder() into liblzma and use it in xz to implement the --list feature. * Capsicum sandbox support is enabled by default where available (FreeBSD >= 10). -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Can "xz -T0 SMALLFILE1 SMALLFILE2 ... FILE99" be made multi-threaded?

2018-06-15 Thread Lasse Collin
ds or such which would print the value that is internally available for --threads=0. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] silence some static analysis warnings

2018-07-27 Thread Lasse Collin
s (I believe) and tukaani.org has SPF records. I'm not going to spend time to figure out how to get the list blessed by Google and I'm not really asking anyone to help either; I'm writing this mainly to inform people. If the problem repeats, I will put a notice about gmail.com ad

[xz-devel] Adding fuzz testing support to liblzma

2018-10-25 Thread Lasse Collin
will do that. Hopefully the above is a correct enough summary of what has happened so far. :-) > This patch addresses the following comments by Lasse Collin: > - Disable CRC check and add memory limit during decoder > initialization > - Check for LZMA_PROG_ERROR > - Remove super

Re: [xz-devel] Adding fuzz testing support to liblzma

2018-10-28 Thread Lasse Collin
On 2018-10-26 Bhargava Shastry wrote: > On 10/25/18 11:34 PM, Lasse Collin wrote: > >> +// Decode BUFSIZ==8192 bytes of inbuf at a time > > > > The value of BUFSIZ depends on libc and isn't always 8192. > > Agreed. What would you like it to be? Af

Re: [xz-devel] Adding fuzz testing support to liblzma

2018-10-29 Thread Lasse Collin
ing the input and output chunk sizes is probably needed to make the second version faster. You could try some odd values between 100 and 250, or maybe even up to 500. On the other hand, it's possible that I'm putting too much weight on the importance of fuzzing the stop & continue code paths. Thanks again! -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Adding fuzz testing support to liblzma

2018-10-30 Thread Lasse Collin
tput buffers aren't as valuable for fuzzing as I thought and we should just use the simple fast version. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Adding fuzz testing support to liblzma

2018-11-02 Thread Lasse Collin
On 2018-10-31 Bhargava Shastry wrote: > On 10/30/18 6:26 PM, Lasse Collin wrote: > > On 2018-10-30 Bhargava Shastry wrote: > >> - oss-fuzz requires a Google linked email address of the > >> maintainer. Could you please provide me one? > > > > No, I'

Re: [xz-devel] [PATCH] 'have have' typos

2018-11-22 Thread Lasse Collin
On 2018-11-22 Pavel Raiskup wrote: > --- > src/xz/signals.c | 2 +- > src/xz/xz.1 | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) Thanks! Committed. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Translation platform for XZ ?

2019-02-21 Thread Lasse Collin
ople will be handled and then submit 5.2.4 to be translated (or submit to a small subgroup to find out if something needs to be fixed at xz side first)? -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Translation platform for XZ ?

2019-03-03 Thread Lasse Collin
Hello! I'm sorry for the delay. On 2019-02-22 Mario Blättermann wrote: > Am Donnerstag, 21. Februar 2019, 18:38:06 CET schrieb Lasse Collin: > > On 2019-02-17 Mario Blättermann wrote: > > > It would be nice if xz would be integrated into a global > > > transl

Re: [xz-devel] Internationalization of man pages

2019-06-22 Thread Lasse Collin
xzcat.1 (as symlinks) too like there are for English versions. > Once it works, xz needs a second translation domain at GNU TP. OK. The translations of the command line tools are temporarily stuck due to me at the moment. :-( -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Internationalization of man pages

2019-06-26 Thread Lasse Collin
On 2019-06-23 Mario Blättermann wrote: > Am Sa., 22. Juni 2019 um 23:53 Uhr schrieb Lasse Collin > : > > - Is xzdec-man.pot intentionally there or should it be part of > > xz-man.pot? > > > It's an artifact from a separate creation of a pot file for xzdec.1

Re: [xz-devel] Internationalization of man pages

2019-06-28 Thread Lasse Collin
On 2019-06-26 Lasse Collin wrote: > The overlong lines in --help simply should be fixed by the > translators. In the near future xz won't have auto-wrap support for > --help strings. I will reconsider this. It might be less painful to wrap multibyte strings in portable C than

Re: [xz-devel] Internationalization of man pages

2019-07-14 Thread Lasse Collin
other hand it would be really nice if this could solve the line length and alignment problems for good. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Internationalization of man pages

2020-02-07 Thread Lasse Collin
se than GNU/Linux (the scripting for those details is ugly). The test should be done so that there are translated man pages available in the po4a/man directory, e.g. by using de.po from the first post of this thread. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Internationalization of man pages

2020-02-12 Thread Lasse Collin
On 2020-02-12 Mario Blättermann wrote: > Lasse Collin schrieb am Fr., 7. Feb. 2020, > 15:32: > > - po4a is never needed when building from a release tarball. > > This means, distribution packagers don't have to bother with po4a, > and the translated man pages will

Re: [xz-devel] Internationalization of man pages

2020-02-14 Thread Lasse Collin
On 2020-02-14 Mario Blättermann wrote: > Am Mi., 12. Feb. 2020 um 19:05 Uhr schrieb Lasse Collin > : > > On 2020-02-12 Mario Blättermann wrote: > > > Lasse Collin schrieb am Fr., 7. Feb. > > > 2020, 15:32: > > > > The extra po4a options like unkn

Re: [xz-devel] cmake support?

2020-02-15 Thread Lasse Collin
d start, i.e. from vcpkg. A link to a known good CMakeLists.txt (and possible other files) would be nice along with an explanation what it builds (e.g. both static and shared liblzma?) and which targets are supported (e.g. is it MSVC only?). Thanks! -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] cmake support?

2020-02-16 Thread Lasse Collin
ate or use config.h. The number of #defines isn't huge so it the command line lengths don't get too long, but is it bad/ugly on Windows? - NDEBUG shouldn't be #defined for debug builds. Thoughts, fixes, suggestions etc. are welcome. -- Lasse Collin | IRC: Larhzu @ IRCnet &

Re: [xz-devel] cmake support?

2020-02-16 Thread Lasse Collin
ess). I also added find_dependency(Threads) to liblzmaConfig.cmake since I got an impression from other sources that it is required. Thanks for your help! -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode # # # Thi

Re: [xz-devel] cmake support?

2020-02-16 Thread Lasse Collin
On 2020-02-16 Lasse Collin wrote: > I quickly checked a few random things and didn't find anything else > that wouldn't be OK with 3.1. target_link_options needs 3.13. Seems that it can be avoided by using set_target_properties(liblzma PROPERTIES LINK_FLAGS -Wl,

Re: [xz-devel] cmake support?

2020-02-16 Thread Lasse Collin
ess the importance of supporting old versions largely depends on if the CMake support will become usable for more than building liblzma on Windows. > On Sun, Feb 16, 2020 at 4:48 PM Lasse Collin > wrote: > > If I use add_library(lzma ...) and PREFIX "lib", isn't the end

Re: [xz-devel] cmake support?

2020-02-17 Thread Lasse Collin
s support too, having proper checks for the defines under "These are specific to GNU/Linux" would be important (and a check if librt is needed for clock_gettime). The M4 macros are in m4/tuklib_cpucores.m4 and m4/tuklib_physmem.m4. These aren&

Re: [xz-devel] cmake support?

2020-02-24 Thread Lasse Collin
tions in that email are still valid. In addition to testing with MSVC, it can now be tested on other platforms too. At least I would like to know if it works with MSVC to build static and shared liblzma, and does it name the shared liblzma file "liblzma.dll" as I hope it does. Thanks! -

Re: [xz-devel] cmake support?

2020-02-27 Thread Lasse Collin
*). Anyway, more feedback from CMake users is welcome. The liblzma part should work on multiple operating systems; it's not Windows-only anymore. See the comment in the beginning of CMakeLists.txt in xz.git. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] cmake support?

2020-02-28 Thread Lasse Collin
LTIN_ASSUME_ALIGNED This sounds like a bug in the CMake files unless MSVC really supports __builtin_assume_aligned that GCC and Clang support. xz 5.2.4 doesn't recognize that define so it's harmless there. So this must be fixed too if it is broken. The above snapshot is good for testing this with CMake. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

[xz-devel] XZ Utils 5.2.5

2020-03-17 Thread Lasse Collin
--help output or misaligned column headings in tables. In the future, many of these strings will be split and e.g. the table column alignment will be handled in software. This should make the strings easier to translate. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Parallel decompression support

2020-04-18 Thread Lasse Collin
threaded decompression (and also some other features) should have been implemented years ago. However, I haven't had energy to do it. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Compress thousands of similar tarballs by xz

2020-05-11 Thread Lasse Collin
a quick try I didn't get good results with with megabyte-sized files but perhaps I don't know how to use it correctly. Using a single .tar as a dictionary worked great though. In both cases you obviously need latest.tar to decompress the old*.tar{.delta,.zst} files. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Parallel decompression support

2020-05-11 Thread Lasse Collin
usy all the time. With decompression one has to decide how much memory can be used by default. If there is no limit, in the extreme case a decoder could read the whole input file in RAM and allocate output buffer for the whole uncompressed file. This problem doesn't exist in your mmap (or pread)

Re: [xz-devel] LZMA_BUF_ERROR

2020-06-04 Thread Lasse Collin
es bytes in the .zip header. So initialize lzma_alone_decoder(), pass the five bytes, then pass either the 64-bit uncompressed size (if EOS bit in .zip headers is unset) or eight 0xFF bytes (if EOS bit in .zip headers is set), and finally pass the actual LZMA data. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] CMake: Add missing include directory when building the shared library

2020-06-04 Thread Lasse Collin
mon/common_w32res.rc +++ b/src/common/common_w32res.rc @@ -6,7 +6,9 @@ */ #include -#include "config.h" +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif #define LZMA_H_INTERNAL #define LZMA_H_INTERNAL_RC #include "lzma/version.h" -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] CMake: Add missing include directory when building the shared library

2020-07-12 Thread Lasse Collin
t;https://tukaani.org/xz/> > Product version: 5.3.1alpha Looks good. I have finally committed it. Thanks! -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] BUG: liblzma: LZMA+BCJ raw decode: output truncated last word

2020-07-12 Thread Lasse Collin
32_t 8 bytes uncompressed size as little endian uint64_t; UINT64_MAX means unknown and then (and only then) EOPM must be present - LZMA2, possibly together with a BCJ or Delta filter, with lzma_raw_decoder() since LZMA2 always includes the end marker. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] parallelism-dependent output

2020-11-03 Thread Lasse Collin
compressors. However, this doesn't change the big picture much and the above paragraph is still true. Implementing threaded decompression would help xz but only with big packages. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [RFC WIP] liblzma: Add multi-threaded decoder

2020-12-13 Thread Lasse Collin
erformance (to prevent idle threads) and with a quick try it seems it might help with decoding too. The significance depends a lot on the data, of course. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [RFC WIP] liblzma: Add multi-threaded decoder

2020-12-16 Thread Lasse Collin
On 2020-12-14 Sebastian Andrzej Siewior wrote: > On 2020-12-13 23:19:25 [+0200], Lasse Collin wrote: > > Yes, reusing buffers and encoder/decoder states can be useful (fewer > > page faults). Perhaps even the input buffer could be reused if it > > is OK to waste some

Re: [xz-devel] [RFC PATCH] liblzma: Test memory availability for lzma_stream_encoder_mt

2020-12-26 Thread Lasse Collin
lerate allocation failures and be able to continue with fewer threads. However, I haven't heard complaints about this outside xz -T0 context and I think the existing hack isn't horrible. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] xz: Fix setting memory limit on 32-bit systems

2020-12-26 Thread Lasse Collin
ed at the same time. I cannot make everyone happy. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] xz: Fix setting memory limit on 32-bit systems

2021-01-08 Thread Lasse Collin
ecially if it is supposed to work with kernels other than Linux. Simply trying to allocate a lot of memory (to test if it works) is more realistic but I think it's still dumb. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] xz: Fix setting memory limit on 32-bit systems

2021-01-08 Thread Lasse Collin
On 2020-12-27 Vitaly Chikunov wrote: > On Sat, Dec 26, 2020 at 05:04:02PM +0200, Lasse Collin wrote: > > I cannot make everyone happy. > > Wow, that's philosophical! I think, we should solve this fundamental > problem first. -- Even if we cannot satisfy everybody, better

Re: [xz-devel] xz-java and newer java

2021-01-08 Thread Lasse Collin
ge too, and perhaps a few variations of it. Thanks! There are multiple things in XZ Utils that I try to look at in the near future so it will be a while until I will play with the Java code. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] xzdiff: Trap SIGPIPE

2021-01-08 Thread Lasse Collin
ipt already uses "trap" to delete temporary files. I think that code path is never used if the shell is fancy enough to do complex redirections. However, it traps also SIGPIPE and I think it can in such situations run rm -rf twice on the same path name. That feels suspicious but perhaps it's not too bad, especially since the code likely is never used on most systems. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] xzdiff: Trap SIGPIPE

2021-01-09 Thread Lasse Collin
On 2021-01-08 Lasse Collin wrote: > It's tempting to ignore exit statuses >= 128 at the end of the script > where it current checks for "$xz_status" -eq 0 but that doesn't work > because in the middle of the script there is also this: > > ca

Re: [xz-devel] [PATCH v2] xz: Ignore hard link count if not deleting

2021-01-09 Thread Lasse Collin
, and sticky bits are not copied +to the target file. +In earlier versions this was only done with +.BR \-\-force . .TP .BR \-f ", " \-\-force This option has several effects: -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH v2] liblzma: Add multi-threaded decoder

2021-01-09 Thread Lasse Collin
27;m able to focus on it. Thanks a lot for your help! -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [PATCH] xz: Fix setting memory limit on 32-bit systems

2021-01-09 Thread Lasse Collin
-bit kernel because it would set the limit to 4020 MiB if there's a lot of RAM. But with this patch it would happen without any extra arguments; using -T0 would be enough. So it's a hack but a hack that has minimal effect on existing behavior outside -T0. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] xz-java and newer java

2021-01-11 Thread Lasse Collin
on but I have so little Java experience that I don't have a clue about this. If you have time and interest, it would be valuable to know which tricks provide the largest performance improvements. However, I repeat that I cannot spend much time on this in the near future even though I thin

Re: [xz-devel] [PATCH] xzdiff: Trap SIGPIPE

2021-01-11 Thread Lasse Collin
Hello! Since xz-devel is subscribers only, I quote your message in full and also include your test scripts as an attachment for others to see. On 2021-01-09 Étienne Mollier wrote: > Lasse Collin, on 2021-01-09 17:38:20 +0200: > > The following patch replace -cdfq with -cdf an

Re: [xz-devel] [PATCH] xzdiff: Trap SIGPIPE

2021-01-11 Thread Lasse Collin
On 2021-01-11 Étienne Mollier wrote: > Lasse Collin, on 2021-01-11 19:19:09 +0200: > > I understand from your message that you got a different result. I > > wonder what would explain the difference. Your results are close to > > what I would expect with the "trap '

Re: [xz-devel] [PATCH v2] xz: Ignore hard link count if not deleting

2021-01-11 Thread Lasse Collin
On 2021-01-10 Sebastian Andrzej Siewior wrote: > On 2021-01-09 18:21:45 [+0200], Lasse Collin wrote: > > Any thoughts on this patch? > > Yes, I think it makes sense. Following the symlink makes sense but > removing the symlink is different from removing a file and since it i

Re: [xz-devel] [PATCH] xz: Fix setting memory limit on 32-bit systems

2021-01-18 Thread Lasse Collin
a mess. In any case, let's at least try to find some solution to the "xz -T0" case. It would be nice to hear if my suggestion makes any sense. Thanks. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] xz-java and newer java

2021-01-18 Thread Lasse Collin
s can be overlong instead of messing up the indentation). I think your patch will find its way into XZ for Java in some form but once again I repeat that it will take some time. These XZ projects are only a hobby for me and currently I don't even turn on my computer every day. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] java crc64 implementation

2021-01-19 Thread Lasse Collin
the C version in [1], the Java version in [2] reads the input byte[] array byte-by-byte. Using a fast method to read 8 *aligned* bytes at a time in native byte order should give more speed; after all, it's one of the benefits of this method that one can read multiple input bytes at a time

Re: [xz-devel] [PATCH v2] liblzma: Add multi-threaded decoder

2021-01-24 Thread Lasse Collin
rtial_update); If outq.head is something else, it is either already finished or partial output has already been enabled. In both cases lzma_outq_enable_partial_output() will do nothing. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] [RFC 2/2] Add xxHash, XX3 (128bit) for hashing.

2021-01-29 Thread Lasse Collin
ch an algorithm was in the early plans but so were a few other nice things but many never materialized. I will look at the SHA-256 patch later. There are unusually many things in the queue of XZ-related things. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Compatibility between CMake config file and FindLibLZMA.cmake

2021-01-30 Thread Lasse Collin
t was to use liblzma::liblzma in the config file. I guess it's too late to change it now. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] java crc64 implementation

2021-02-02 Thread Lasse Collin
e byte smaller at the end. The same thing happens too if the buffer size is kept at 8192 but first byte isn't used (making the beginning of the buffer misaligned). Moving the "(crc32 >> 32)" to a different position in the xor sequence can affect things too... it's almost spooky. ;-) It would be nice if you could compare these too and suggest what should be committed. Maybe you can figure out an even better version. Different CPU or 32-bit Java or other things may give quite different results. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] xz-java minor read improvements

2021-02-02 Thread Lasse Collin
ava support as multi-release JAR, so multi-release can be used for other things too. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] java crc64 implementation

2021-02-02 Thread Lasse Collin
> > version. Different CPU or 32-bit Java or other things may give > > quite different results. > > Truncating the crc to an int 1 time in the loop seems like a clear > winner. I will play with this in my benchmark. > My benchmark is calculating the crc64 of 8k of random bytes. I will > change it to include misaligned read as well. Thanks. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Re: java LZDecoder small improvement

2021-02-03 Thread Lasse Collin
put. +do { +buf[pos++] = buf[back++]; +} while (--left > 0); +} else { +System.arraycopy(buf, back, buf, pos, left); +pos += left; +} if (full < pos) full = pos; -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] java buffer writes

2021-02-05 Thread Lasse Collin
twork; such overheads can be much larger than locking. I put these optimizations in the "nice to have" category. Something could be done to make the code better but it's not urgent and so these won't be in the next release. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Re: [xz-devel] Re: java LZDecoder small improvement

2021-02-05 Thread Lasse Collin
On 2021-02-03 Brett Okken wrote: > On Wed, Feb 3, 2021 at 2:56 PM Lasse Collin > wrote: > > It seems to regress horribly if dist is zero. A file with a very > > long sequence of the same byte is good for testing. > > Would this be a valid test of what you are describin

<    1   2   3   >