way. The Java < 9 code is
completely separate so it cannot be chosen. The property needs to be
documented somewhere too.
I suppose the ARM64 speed is still to be determined by you or someone
else.
--
Lasse Collin
r JDK look bad if newer JDK used VarHandle
instead of Unsafe. Perhaps that worry was overblown. But the other
reasons and keeping the code simpler make me want to avoid Unsafe.
(C code via JNI wouldn't be memory safe but then the speed benefits
should be much more significant too.)
--
Lasse Collin
sh a more generic one?
--
Lasse Collin
ished yet but
now it's ready for testing and feedback. For example, some tweaks from
your array_comp_incremental could be considered after testing.
--
Lasse Collin
e it resulted in a
> smaller file?
They should be very close in practice. You need to compare to XZ Utils
in single-threaded mode: xz -T1
--
Lasse Collin
ad of requiring the array
comparison to support it (the C code in liblzma does it like that).
--
Lasse Collin
d. Different releases can in theory have different
output. XZ Utils output might change in future versions too.
--
Lasse Collin
ngs
are waiting in the queue from the past three years.
--
Lasse Collin
to 50 %) and even electricity. java.util.zip
uses native zlib for the performance-critical code.
In the long run both faster Java code and JNI might be worth doing.
There's more than enough pure Java stuff to do for now so any JNI
thoughts have to wait.
--
Lasse Collin
On 2024-02-28 Sebastian Andrzej Siewior wrote:
> On 2024-02-28 18:45:03 [+0200], Lasse Collin wrote:
> > V_DEBUG was commited to the master and v5.6 branches a few moments
> > ago, so yes, your plan sounds good. :-) Feel free to do it as you
> > prefer, either just making
ell.
Hopefully the already-added workarounds in other packages don't cause
any unwanted side effects in the future.
Thanks!
--
Lasse Collin
On 2024-02-27 Sebastian Andrzej Siewior wrote:
> On 2024-02-27 19:17:48 [+0200], Lasse Collin wrote:
> > - The silencing could be done with -q as well though.
>
> Wouldn't -q also shut some legitime warnings?
Yes. When compressing from stdin to stdout, there aren't many po
and LZMA2
dictionary size adjustment). The verbosity requirement of these messages
isn't being changed now.
--
Lasse Collin
the long term. Something has to be chosen for 5.6.0;
further tweaks can be made later.
By the way, the "time" command gives more precise results than "xz -v".
I use
TIMEFORMAT=$'\nreal\t%3R\nuser\t%3U\nsys\t%3S\ncpu%%\t%P'
in bash to keep the output as seconds instead of minutes and seconds.
--
Lasse Collin
e Java code can
help a bit but I suspect that it still won't be very fast. So far it has
been nice that the Java code is quite readable and I would like keep it
that way in the future too.
--
Lasse Collin
/download
--
Lasse Collin
ulness of =1 or =3 is clear.)
If the branchless C code is not consistent outside x86-64, then 5.6.0
likely should stick to =0. From your results it seems that the other
tweaks to the code provided a minor improvement on non-x86-64 still.
(The tweaks that LZMA_RANGE_DECODER_CONFIG doesn't affect.)
Thanks!
--
Lasse Collin
!
PS. XZ for Java has been idle longer than expected but it should
finally get at least some attention in the coming months.
--
Lasse Collin
64 filter still hasn't been submitted to Linux but it's on the
to-do list.
--
Lasse Collin
refactored a few tests.
* Translations:
- Updated the Brazilian Portuguese translation.
- Added Brazilian Portuguese man page translation.
--
Lasse Collin
format using xz.
* A few new tests were added.
* The liblzma-specific tests are now supported in CMake-based
builds too ("make test").
--
Lasse Collin
nslations: Chinese (simplified), Korean, and Turkish.
--
Lasse Collin
is which
forwards messages to Lasse Collin and Jia Tan.
--
Lasse Collin
On 2022-11-30 Lasse Collin wrote:
> Are there other good library options?
If the goal is to use SHA instructions on x86 then intrinsics in the C
code with runtime CPU detection are an option too. It's done in
crc64_fast.c in 5.3.4alpha already.
--
Lasse Collin
suppose such things happen all the time but when a library is
needed by a package manager it might perhaps have extra worries.
--
Lasse Collin
-pic or --without-pic requires that also
--disable-shared or --disable-static is used on GNU/Linux.
It's in xz.git now and will be in the next releases (5.2.9 is needed to
fix other bugs) so I hope any workarounds can be removed from distros
after that.
Thanks to Adrian for reporting the bug!
--
Lasse Collin
n the long term then
something is broken.
--
Lasse Collin
ideas.
Thoughts?
--
Lasse Collin
On 2022-11-23 John Paul Adrian Glaubitz wrote:
> On 11/23/22 12:31, Lasse Collin wrote:
> > (1) Does this make the problem go away?
>
> Yes, that fixes the linker problem for me. At least in the case of
> mariadb-10.6.
Why does it want static liblzma.a in the first plac
ions into a static liblzma. :-(
> I think we can waive for CentOS 7 compatibility on Debian unstable
> ia64 .
There is no official CentOS 7 for ia64 but that isn't the whole story
as the broken patch has been used elsewhere too. Not having those extra
symbols would still be fine in practice. :-)
--
Lasse Collin
show the same special behavior in ia64 liblzma.a
even though lzma_cputhreads is handled exactly like lzma_get_progress in
the liblzma C code and linker script.
--
Lasse Collin
he symbol versions. It should work fine with GCC >= 10 or Clang.
For what it is worth, when I wrote the patch I tested it on on
Slackware 10.1 (32-bit x86) that has GCC 3.3.4 and it worked perfectly
there. This symbol version stuff isn't a new thing so it really should
work.
--
Lasse Collin
Hello!
On 2021-09-02 Liao Hua wrote:
> +#define LZMA_FILTER_ARM64 LZMA_VLI_C(0x0a)
Is this ID 0x0A in actual use somewhere? Can it be used in the official
.xz format for something else than the filter you submitted?
On 2021-09-08 Lasse Collin wrote:
> On 2021-09-02 Liao Hua
erfect.
- Other improvements to tests.
* Updated translations: Croatian, Finnish, Hungarian, Polish,
Romanian, Spanish, Swedish, and Ukrainian.
--
Lasse Collin
n page translation file from
fr_FR.po to fr.po and thus also its install directory
(like /usr/share/man/fr_FR -> .../fr).
- Man page translations for upcoming 5.4.0 are now handled
in the Translation Project.
* Update doc/faq.txt a little so it's less out-of-date.
--
Lasse Collin
it can
affect critical tools in the operating system. :-)
--
Lasse Collin
currently planned.
Translations need to be updated too once the strings and man pages are
close to final. A development release needs to be sent to the
Translation Project at some point. If people want to translate the man
pages too, they will need quite a bit of time.
--
Lasse Collin
CMake files are now actually included in the release tarball.
They should have been in 5.2.5 already.
- Minor CMake fixes and improvements.
* Added a new translation: Turkish
--
Lasse Collin
cluded
Visual Studio project files.
--
Lasse Collin
likely is a way to enable it. I cannot test
MSVC builds now so I won't make many blind guesses.
--
Lasse Collin
v5.2 branch. Thanks!
--
Lasse Collin
nk it's good.
Does it work with CMake for you? I'm hoping that the VS project files
can be removed in the near-future and CMake used for building with VS.
That way there are fewer build files to maintain.
--
Lasse Collin
in most cases. In contrast, building
the command line tools with CMake is still clearly incomplete
and experimental and should be used for testing only.
--
Lasse Collin
ight even rely on the current
behavior. Trying to change this could cause problems in rare cases and,
if not done carefully enough, introduce new bugs. So I thank you for
the patch but it won't be included.
--
Lasse Collin
same time since I got one new feedback
wishing for xz-man in the TP. Clearly there are people who wish to
translate man pages. :-)
I won't be on my computer for about two weeks so I won't be able to
reply to emails before that.
Thanks!
--
Lasse Collin
sors might be worth trying too. Zstandard typically
compresses only slightly worse than XZ/LZMA but it is *a lot* faster to
decompress.
--
Lasse Collin
p is already in progress at least
for XZ Utils.
--
Lasse Collin
developer Igor Pavlov
informed about format changes (including new filters) is important too.
--
Lasse Collin
ff-list with XZ Utils and he might have a bigger
role in the future at least with XZ Utils. It's clear that my resources
are too limited (thus the many emails waiting for replies) so something
has to change in the long term.
--
Lasse Collin
On 2021-01-20 Sebastian Andrzej Siewior wrote:
> On 2021-01-18 23:52:50 [+0200], Lasse Collin wrote:
> > I have understood that *in practice* the problem with the xz command
> > line tool is limited to "xz -T0" usage so fixing this use case is
> > enough for most
-CAN-16587.patch.sig
It is also linked from the XZ Utils home page <https://tukaani.org/xz/>.
--
Lasse Collin
framework patches
I suppose then the next alpha release is close to ready.
--
Lasse Collin
ple but I won't look at it now.
For the ArrayUtil patch, it's a complex one and I'm not able to look at
it for now.
--
Lasse Collin
yet how much is worth the trouble. stream_decoder_mt_memconfig() has a
few FIXMEs too, maybe they don't need to be changed but it needs to be
decided.
I'm in a hurry now but I should have time for xz next week. :-)
--
Lasse Collin
corrupt input because, unless
LZMA_FINISH is used, liblzma cannot know if the input is truncated or
if there is merely a pause in the input for some application-specific
reason. That can result in LZMA_BUF_ERROR but if the application knows
that such pauses are possible then it can handle LZMA_BUF_ERROR
specially and continue decoding when more input is available.
--
Lasse Collin
that don't
care about partial output from broken files.
--
Lasse Collin
On 2022-02-11 Ed Maste wrote:
> src/liblzma/check/crc32_x86.S | 4 ++--
> src/liblzma/check/crc64_x86.S | 4 ++--
> 2 files changed, 4 insertions(+), 4 deletions(-)
I have committed (but not tested) this. Thanks!
--
Lasse Collin
l big FIXMEs are solved, only a few small ones to do. :-)
Gitweb is working again.
--
Lasse Collin
On 2022-01-08 Jean-Pierre Giraud wrote:
> Package xz-utils
> version 5.2.5-2
>
> Hi,
> Please find attached the french translation of the xz-utils manpage
> done by "bubu" and proofread by the debian-l10n-french mailing list
> contributors.
Thanks! Committed.
--
Lasse Collin
On 2021-12-31 Sebastian Andrzej Siewior wrote:
> On 2021-12-15 23:33:58 [+0200], Lasse Collin wrote:
> > Yes. It's fairly simple from implementation point of view but is it
> > clear enough for the users, I'm not sure.
> >
> > I suppose the alternative is having just
On 2021-12-04 Sebastian Andrzej Siewior wrote:
> On 2021-11-30 00:25:11 [+0200], Lasse Collin wrote:
> > Separate soft and hard limits might be convenient from
> > implementation point of view though. xz would need --memlimit-soft
> > (or some better name) which would alw
are
still 2-3 fewer than needed for the encoder.
I've made some other minor edits locally already so I would prefer to
*not* get new patch revisions until I have committed something.
Comments are very welcome. :-)
Thanks!
--
Lasse Collin
ds",
> - optarg, 0, 16384));
> + optarg, 0, LZMA_THREADS_MAX));
common.h is internal to liblzma and must not be used from xz. Maybe
LZMA_THREADS_MAX could be moved to the public API, I don't know right
now.
--
Lasse Collin
pts/{xzless,xzmore}.in]
indent_style = tab
indent_size = 8
[/src/scripts/{xzdiff,xzgrep}.in]
indent_style = space
indent_size = 2
[CMakeLists.txt,*.cmake]
indent_style = space
indent_size = 4
[*.vcxproj,xz_win.sln]
charset = utf-8-bom
---
Is it good enough or did I add bad bugs? :-)
--
Lasse Collin
a new alpha package.
--
Lasse Collin
It might still need a few
more fixes even for liblzma-only builds.
--
Lasse Collin
thing anyway.
Thanks! Committed.
--
Lasse Collin
0x3FC) != 0);
dest = is_encoder ? pc : 0U - pc;
}
dest >>= 2;
The "start=offset" option probably could be omitted. It's quite useless
inside .xz. XZ Embedded doesn't support it anyway.
Once a filter is ready, I will need to discuss it with Igor Pavlov (the
7-Zip's developer) too, and add the new filter ID to the official .xz
specification.
--
Lasse Collin
have any news. :-(
--
Lasse Collin
? I have no Go
experience so I have no idea which are good or already popular.
Thanks!
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
On 2021-04-15 Mario Blättermann wrote:
> Am So., 11. Apr. 2021 um 20:48 Uhr schrieb Lasse Collin
> :
> > I suppose I can just submit a snapshot from the master branch.
I have done this.
> I am curious to see when the first new translations will arrive :)
Me too. It's a lot of wo
e corrections,
some didn't. Thus multiple translations were omitted from 5.2.5. With
this background I feel that if 5.2.6 is needed I won't consider any
*new* xz.po files for it anyway; new xz-man.po languages would be fine.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
s with 32-bit
> userspace, or systems that use XPA (an extension similar to x86's
> PAE).
>
> So, for MIPS32, we have to impose stronger memory limits. I've chosen
> 2000MiB to give the process some headroom.
Thanks! Committed.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
Java 9
for module-info support but otherwise the code should still be
Java 5 compatible (see README and comments in build.properties).
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
as a fairly small effect.
Omitting it keeps the code a tiny bit simpler.
I have committed the change. I think xz-java.git should now be almost
ready for a release. I just need to add NEWS and bump the version
number.
Thanks for your help!
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
On 2021-02-13 Brett Okken wrote:
> On Thu, Feb 11, 2021 at 12:51 PM Lasse Collin
> wrote:
> > I still worry about short copies. If the file is full of tiny
> > matches/repeats of 1-3 bytes or so, arraycopy can be slower. Such
> > files aren't typical at all but I don't w
de, please make sure that word-wrapping
is disabled in the email client or use attachments. Thanks!
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
tem.arraycopy(buf, 0, buf, toCopy, remaining);
}
}
}
}
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
differences are smaller.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
d_library(liblzma ) to add_library(LibLZMA
) then the filename will be LibLZMA.something too. That isn't
good because then one cannot replace a CMake-built shared liblzma with
an Autotools-built one on operating systems where file and library
names are case sensitive.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
(buf, back, buf, pos, copySize);
}
pos += copySize;
left -= copySize;
} while (left > 0);
if (full < pos)
full = pos;
}
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
that
it's worth the extra effort and complexity for such a small speed gain.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
ook like
> > that writeChunk() could be overwriting the input.
>
> I assumed that lz.fillWindow(buf, off, len); would always process the
> 1 byte.
Yes, but it's not immediately obvious to a new reader. Also, many other
classes have tempBuf for identical use so it's good to keep that pa
On 2021-02-05 Brett Okken wrote:
> On Fri, Feb 5, 2021 at 11:07 AM Lasse Collin
> wrote:
> > Also, does it really help to unroll the loop? With 8191-byte
> > buffers I see no significant difference (in a quick
> > not-very-accurate test) if the switch-statement is replac
it. It would be confusing to use the same buffer in
write(int) and writeChunk(). At glance it would look like that
writeChunk() could be overwriting the input.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
t an OK
version to commit?
Is the following fine to you as the file header? Your email address can
be omitted if you prefer that. I will mention in the commit message
that you adapted the code from XZ Utils and benchmarked it.
/*
* CRC64
*
* Authors: Brett Okken
* Lasse Collin
*
* This file has been put into the public domain.
* You can do whatever you want with this file.
*/
Thanks!
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
m even if one uses BufferedInputStream. BufferedInputStream
has synchronized read(). I don't know how much locking matters in this
case. I'm not curious enough to try with a non-synchronized buffered
input stream now.
There are related comments in the "java buffer writes" thread.
--
Lasse Col
rheads can
be much larger than locking.
I put these optimizations in the "nice to have" category. Something
could be done to make the code better but it's not urgent and so these
won't be in the next release.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
On 2021-02-03 Brett Okken wrote:
> On Wed, Feb 3, 2021 at 2:56 PM Lasse Collin
> wrote:
> > It seems to regress horribly if dist is zero. A file with a very
> > long sequence of the same byte is good for testing.
>
> Would this be a valid test of what you are descri
{
+buf[pos++] = buf[back++];
+} while (--left > 0);
+} else {
+System.arraycopy(buf, back, buf, pos, left);
+pos += left;
+}
if (full < pos)
full = pos;
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
r 32-bit Java or other things may give
> > quite different results.
>
> Truncating the crc to an int 1 time in the loop seems like a clear
> winner. I will play with this in my benchmark.
> My benchmark is calculating the crc64 of 8k of random bytes. I will
> change it to include misaligned read as well.
Thanks.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
t as multi-release JAR,
so multi-release can be used for other things too.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
.
The same thing happens too if the buffer size is kept at 8192 but first
byte isn't used (making the beginning of the buffer misaligned).
Moving the "(crc32 >> 32)" to a different position in the xor sequence
can affect things too... it's almost spooky. ;-)
It would be nice if you could compare these too and suggest what should
be committed. Maybe you can figure out an even better version.
Different CPU or 32-bit Java or other things may give quite different
results.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
use liblzma::liblzma in the config file. I
guess it's too late to change it now.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
ch an algorithm was in the early plans but so were a
few other nice things but many never materialized.
I will look at the SHA-256 patch later. There are unusually many things
in the queue of XZ-related things.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
se, it is either already finished or
partial output has already been enabled. In both cases
lzma_outq_enable_partial_output() will do nothing.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
version in [2] reads the input
byte[] array byte-by-byte. Using a fast method to read 8 *aligned*
bytes at a time in native byte order should give more speed; after all,
it's one of the benefits of this method that one can read multiple
input bytes at a time.
A public domain patch for a faster CR
r patch will find its way into XZ for Java in some form
but once again I repeat that it will take some time. These XZ projects
are only a hobby for me and currently I don't even turn on my computer
every day.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
let's at least try to find some solution to the "xz -T0" case. It would
be nice to hear if my suggestion makes any sense. Thanks.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
On 2021-01-10 Sebastian Andrzej Siewior wrote:
> On 2021-01-09 18:21:45 [+0200], Lasse Collin wrote:
> > Any thoughts on this patch?
>
> Yes, I think it makes sense. Following the symlink makes sense but
> removing the symlink is different from removing a file and since i
1 - 100 of 172 matches
Mail list logo
