on in the source tree to
Markdown (CommonMark).
* The binaries of 1.10 in the Maven Central require Java 8 and
contain optimized classes for Java >= 9 as multi-release JAR.
They were built with OpenJDK 21.0.4 on GNU/Linux using the
following command:
SOURCE_DATE_EPOCH=1722262226 TZ=UTC0 ant maven
--
Lasse Collin
t RangeDecoder was the only performance-critical
place with unsigned integers.
--
Lasse Collin
ned integers.
Integer.compareUnsigned seems to give the same performance as the
"long" method on x86-64. So compareUnsigned is the obvious choice. I
have committed it to master.
Thanks!
--
Lasse Collin
On 2024-03-07 Lasse Collin wrote:
> On 2024-03-05 Dennis Ens wrote:
> > > I hope 1.10 could be done in a month or two but I don't want to
> > > make any promises or serious predictions. Historically those
> > > haven't been accurate at all.
> &g
On 2024-07-11 Christoph Biedl wrote:
> Lasse Collin wrote...
>
> > There weren't any comments in the CMake thread on GitHub[3] and no
> > one has objected on xz-devel so far either. Considering that
> > Autotools support isn't going away anytime soon, I thin
On 2024-03-26 Lasse Collin wrote:
> I suppose the ARM64 speed is still to be determined by you or someone
> else.
I was given results from one ARM64 system with Java 23:
- Java 8 code: 5.6 s
- Basic: 3.8 s
- UnalignedLongLE: 2.4 s
The test command:
time head -c100
On 2024-07-07 Sebastian Andrzej Siewior wrote:
> On 2024-07-06 15:42:41 [+0300], Lasse Collin wrote:
> > Does anyone think it's too early to require CMake >= 3.20?
>
> Debian Bookworm (stable) has cmake 3.25 and the release before it has
> it in backports. So if o
should be released at some point this year.
The Autotools support won't go away in 5.8.0, thus the CMake version
requirement matters only if one cannot or doesn't want to use the
Autotools-based build.
--
Lasse Collin
L39Z@pinacolada/
https://bugs.gentoo.org/934370
--
Lasse Collin
non-glibc Linux to match
what the Autotools build does. For example, symbol
versioning isn't enabled with musl.
* Symbol versioning variant can now be overridden by
setting SYMBOL_VERSIONING to "OFF", "generic", or
"linux".
* Documentation:
- Clarify the description of --disable-assembler in INSTALL.
The option only affects 32-bit x86 assembly usage.
- Don't install the TODO file as part of the documentation.
The file is out of date.
- Update home page URLs back to their old locations on
tukaani.org.
- Update maintainer info.
--
Lasse Collin
7;s done in a sensible enough way. The Java < 9 code is
completely separate so it cannot be chosen. The property needs to be
documented somewhere too.
I suppose the ARM64 speed is still to be determined by you or someone
else.
--
Lasse Collin
8 would
make upgrading to newer JDK look bad if newer JDK used VarHandle
instead of Unsafe. Perhaps that worry was overblown. But the other
reasons and keeping the code simpler make me want to avoid Unsafe.
(C code via JNI wouldn't be memory safe but then the speed benefits
should be much more significant too.)
--
Lasse Collin
sh a more generic one?
--
Lasse Collin
's not fully finished yet but
now it's ready for testing and feedback. For example, some tweaks from
your array_comp_incremental could be considered after testing.
--
Lasse Collin
t; Does xz-java always do a better job compressing since it resulted in a
> smaller file?
They should be very close in practice. You need to compare to XZ Utils
in single-threaded mode: xz -T1
--
Lasse Collin
ld be handled in those places instead of requiring the array
comparison to support it (the C code in liblzma does it like that).
--
Lasse Collin
ences which affects
both output and speed. Different releases can in theory have different
output. XZ Utils output might change in future versions too.
--
Lasse Collin
r things
are waiting in the queue from the past three years.
--
Lasse Collin
n JNI
could save both time (up to 50 %) and even electricity. java.util.zip
uses native zlib for the performance-critical code.
In the long run both faster Java code and JNI might be worth doing.
There's more than enough pure Java stuff to do for now so any JNI
thoughts have to wait.
--
Lasse Collin
On 2024-02-28 Sebastian Andrzej Siewior wrote:
> On 2024-02-28 18:45:03 [+0200], Lasse Collin wrote:
> > V_DEBUG was commited to the master and v5.6 branches a few moments
> > ago, so yes, your plan sounds good. :-) Feel free to do it as you
> > prefer, either just making the
Hopefully the already-added workarounds in other packages don't cause
any unwanted side effects in the future.
Thanks!
--
Lasse Collin
On 2024-02-27 Sebastian Andrzej Siewior wrote:
> On 2024-02-27 19:17:48 [+0200], Lasse Collin wrote:
> > - The silencing could be done with -q as well though.
>
> Wouldn't -q also shut some legitime warnings?
Yes. When compressing from stdin to stdout, there aren't m
tput (switching to single-threaded mode and LZMA2
dictionary size adjustment). The verbosity requirement of these messages
isn't being changed now.
--
Lasse Collin
perhaps
it's not great in the long term. Something has to be chosen for 5.6.0;
further tweaks can be made later.
By the way, the "time" command gives more precise results than "xz -v".
I use
TIMEFORMAT=$'\nreal\t%3R\nuser\t%3U\nsys\t%3S\ncpu%%\t%P'
in bash to keep the output as seconds instead of minutes and seconds.
--
Lasse Collin
help. Optimizing the Java code can
help a bit but I suspect that it still won't be very fast. So far it has
been nice that the Java code is quite readable and I would like keep it
that way in the future too.
--
Lasse Collin
all/warzone2100-data/download
--
Lasse Collin
ess of =1 or =3 is clear.)
If the branchless C code is not consistent outside x86-64, then 5.6.0
likely should stick to =0. From your results it seems that the other
tweaks to the code provided a minor improvement on non-x86-64 still.
(The tweaks that LZMA_RANGE_DECODER_CONFIG doesn't affect.)
Thanks!
--
Lasse Collin
!
PS. XZ for Java has been idle longer than expected but it should
finally get at least some attention in the coming months.
--
Lasse Collin
64 filter still hasn't been submitted to Linux but it's on the
to-do list.
--
Lasse Collin
Added and refactored a few tests.
* Translations:
- Updated the Brazilian Portuguese translation.
- Added Brazilian Portuguese man page translation.
--
Lasse Collin
ge(2) (OpenBSD)
* Scripts now support the .lz format using xz.
* A few new tests were added.
* The liblzma-specific tests are now supported in CMake-based
builds too ("make test").
--
Lasse Collin
d translations: Chinese (simplified), Korean, and Turkish.
--
Lasse Collin
or bug reports is which
forwards messages to Lasse Collin and Jia Tan.
--
Lasse Collin
On 2022-11-30 Lasse Collin wrote:
> Are there other good library options?
If the goal is to use SHA instructions on x86 then intrinsics in the C
code with runtime CPU detection are an option too. It's done in
crc64_fast.c in 5.3.4alpha already.
--
Lasse Collin
lzma needs to be rebuilt without changing its
soname? I suppose such things happen all the time but when a library is
needed by a package manager it might perhaps have extra worries.
--
Lasse Collin
is, --with-pic or --without-pic requires that also
--disable-shared or --disable-static is used on GNU/Linux.
It's in xz.git now and will be in the next releases (5.2.9 is needed to
fix other bugs) so I hope any workarounds can be removed from distros
after that.
Thanks to Adrian for reporting the bug!
--
Lasse Collin
s needed in the long term then
something is broken.
--
Lasse Collin
but I don't have better ideas.
Thoughts?
--
Lasse Collin
On 2022-11-23 John Paul Adrian Glaubitz wrote:
> On 11/23/22 12:31, Lasse Collin wrote:
> > (1) Does this make the problem go away?
>
> Yes, that fixes the linker problem for me. At least in the case of
> mariadb-10.6.
Why does it want static liblzma.a in the first place?
is that
XZ Utils build puts symbol versions into a static liblzma. :-(
> I think we can waive for CentOS 7 compatibility on Debian unstable
> ia64 😉.
There is no official CentOS 7 for ia64 but that isn't the whole story
as the broken patch has been used elsewhere too. Not having those extra
symbols would still be fine in practice. :-)
--
Lasse Collin
ity.
lzma_cputhreads doesn't show the same special behavior in ia64 liblzma.a
even though lzma_cputhreads is handled exactly like lzma_get_progress in
the liblzma C code and linker script.
--
Lasse Collin
to the symbol versions. It should work fine with GCC >= 10 or Clang.
For what it is worth, when I wrote the patch I tested it on on
Slackware 10.1 (32-bit x86) that has GCC 3.3.4 and it worked perfectly
there. This symbol version stuff isn't a new thing so it really should
work.
--
Lasse Collin
Hello!
On 2021-09-02 Liao Hua wrote:
> +#define LZMA_FILTER_ARM64 LZMA_VLI_C(0x0a)
Is this ID 0x0A in actual use somewhere? Can it be used in the official
.xz format for something else than the filter you submitted?
On 2021-09-08 Lasse Collin wrote:
> On 2021-09-02 Liao Hua
gure options.
It's still not perfect.
- Other improvements to tests.
* Updated translations: Croatian, Finnish, Hungarian, Polish,
Romanian, Spanish, Swedish, and Ukrainian.
--
Lasse Collin
- Renamed the French man page translation file from
fr_FR.po to fr.po and thus also its install directory
(like /usr/share/man/fr_FR -> .../fr).
- Man page translations for upcoming 5.4.0 are now handled
in the Translation Project.
* Update doc/faq.txt a little so it's less out-of-date.
--
Lasse Collin
I encourage testing, one needs to be careful when it can
affect critical tools in the operating system. :-)
--
Lasse Collin
.ac.
> > > This is everything currently planned.
Translations need to be updated too once the strings and man pages are
close to final. A development release needs to be sent to the
Translation Project at some point. If people want to translate the man
pages too, they will need quite a bit of time.
--
Lasse Collin
work.
- CMake files are now actually included in the release tarball.
They should have been in 5.2.5 already.
- Minor CMake fixes and improvements.
* Added a new translation: Turkish
--
Lasse Collin
Fix building of liblzma.dll with the included
Visual Studio project files.
--
Lasse Collin
default, there likely is a way to enable it. I cannot test
MSVC builds now so I won't make many blind guesses.
--
Lasse Collin
lso in the v5.2 branch. Thanks!
--
Lasse Collin
nk it's good.
Does it work with CMake for you? I'm hoping that the VS project files
can be removed in the near-future and CMake used for building with VS.
That way there are fewer build files to maintain.
--
Lasse Collin
Building static or shared
liblzma should work fine in most cases. In contrast, building
the command line tools with CMake is still clearly incomplete
and experimental and should be used for testing only.
--
Lasse Collin
applications doing single-shot decoding might even rely on the current
behavior. Trying to change this could cause problems in rare cases and,
if not done carefully enough, introduce new bugs. So I thank you for
the patch but it won't be included.
--
Lasse Collin
t the same time since I got one new feedback
wishing for xz-man in the TP. Clearly there are people who wish to
translate man pages. :-)
I won't be on my computer for about two weeks so I won't be able to
reply to emails before that.
Thanks!
--
Lasse Collin
e of LZMA/LZMA2.
Other compressors might be worth trying too. Zstandard typically
compresses only slightly worse than XZ/LZMA but it is *a lot* faster to
decompress.
--
Lasse Collin
hip is already in progress at least
for XZ Utils.
--
Lasse Collin
ports .xz and keeping its developer Igor Pavlov
informed about format changes (including new filters) is important too.
--
Lasse Collin
itory.
Jia Tan has helped me off-list with XZ Utils and he might have a bigger
role in the future at least with XZ Utils. It's clear that my resources
are too limited (thus the many emails waiting for replies) so something
has to change in the long term.
--
Lasse Collin
On 2021-01-20 Sebastian Andrzej Siewior wrote:
> On 2021-01-18 23:52:50 [+0200], Lasse Collin wrote:
> > I have understood that *in practice* the problem with the xz command
> > line tool is limited to "xz -T0" usage so fixing this use case is
> > enough for most
p-ZDI-CAN-16587.patch.sig
It is also linked from the XZ Utils home page <https://tukaani.org/xz/>.
--
Lasse Collin
reports.)
- Your test framework patches
I suppose then the next alpha release is close to ready.
--
Lasse Collin
s simple but I won't look at it now.
For the ArrayUtil patch, it's a complex one and I'm not able to look at
it for now.
--
Lasse Collin
know
yet how much is worth the trouble. stream_decoder_mt_memconfig() has a
few FIXMEs too, maybe they don't need to be changed but it needs to be
decided.
I'm in a hurry now but I should have time for xz next week. :-)
--
Lasse Collin
ncated files are a special case of corrupt input because, unless
LZMA_FINISH is used, liblzma cannot know if the input is truncated or
if there is merely a pause in the input for some application-specific
reason. That can result in LZMA_BUF_ERROR but if the application knows
that such pauses are possible then it can handle LZMA_BUF_ERROR
specially and continue decoding when more input is available.
--
Lasse Collin
plications that don't
care about partial output from broken files.
--
Lasse Collin
On 2022-02-11 Ed Maste wrote:
> src/liblzma/check/crc32_x86.S | 4 ++--
> src/liblzma/check/crc64_x86.S | 4 ++--
> 2 files changed, 4 insertions(+), 4 deletions(-)
I have committed (but not tested) this. Thanks!
--
Lasse Collin
ion of the threaded decoder in a few days.
All big FIXMEs are solved, only a few small ones to do. :-)
Gitweb is working again.
--
Lasse Collin
On 2022-01-08 Jean-Pierre Giraud wrote:
> Package xz-utils
> version 5.2.5-2
>
> Hi,
> Please find attached the french translation of the xz-utils manpage
> done by "bubu" and proofread by the debian-l10n-french mailing list
> contributors.
Thanks! Committed.
--
Lasse Collin
On 2021-12-31 Sebastian Andrzej Siewior wrote:
> On 2021-12-15 23:33:58 [+0200], Lasse Collin wrote:
> > Yes. It's fairly simple from implementation point of view but is it
> > clear enough for the users, I'm not sure.
> >
> > I suppose the alternative is h
On 2021-12-04 Sebastian Andrzej Siewior wrote:
> On 2021-11-30 00:25:11 [+0200], Lasse Collin wrote:
> > Separate soft and hard limits might be convenient from
> > implementation point of view though. xz would need --memlimit-soft
> > (or some better name) which would alw
directly as arguments as there are
still 2-3 fewer than needed for the encoder.
I've made some other minor edits locally already so I would prefer to
*not* get new patch revisions until I have committed something.
Comments are very welcome. :-)
Thanks!
--
Lasse Collin
quot;threads",
> - optarg, 0, 16384));
> + optarg, 0, LZMA_THREADS_MAX));
common.h is internal to liblzma and must not be used from xz. Maybe
LZMA_THREADS_MAX could be moved to the public API, I don't know right
now.
--
Lasse Collin
ate-po,/src/scripts/{xzless,xzmore}.in]
indent_style = tab
indent_size = 8
[/src/scripts/{xzdiff,xzgrep}.in]
indent_style = space
indent_size = 2
[CMakeLists.txt,*.cmake]
indent_style = space
indent_size = 4
[*.vcxproj,xz_win.sln]
charset = utf-8-bom
---
Is it good enough or did I add bad bugs? :-)
--
Lasse Collin
an deliver a new alpha package.
--
Lasse Collin
o the CMake support. It might still need a few
more fixes even for liblzma-only builds.
--
Lasse Collin
thing anyway.
Thanks! Committed.
--
Lasse Collin
dest & 0x3FC) == 0) {
assert((pc & 0x3FC) != 0);
dest = is_encoder ? pc : 0U - pc;
}
dest >>= 2;
The "start=offset" option probably could be omitted. It's quite useless
inside .xz. XZ Embedded doesn't support it anyway.
Once a filter is ready, I will need to discuss it with Igor Pavlov (the
7-Zip's developer) too, and add the new filter ID to the official .xz
specification.
--
Lasse Collin
ly I don't have any news. :-(
--
Lasse Collin
ed too. What do you think? I have no Go
experience so I have no idea which are good or already popular.
Thanks!
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
On 2021-04-15 Mario Blättermann wrote:
> Am So., 11. Apr. 2021 um 20:48 Uhr schrieb Lasse Collin
> :
> > I suppose I can just submit a snapshot from the master branch.
I have done this.
> I am curious to see when the first new translations will arrive :)
Me too. It's a lot
sted white-space corrections,
some didn't. Thus multiple translations were omitted from 5.2.5. With
this background I feel that if 5.2.6 is needed I won't consider any
*new* xz.po files for it anyway; new xz-man.po languages would be fine.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
On 2021-01-28 Sebastian Andrzej Siewior wrote:
> On 2021-01-24 23:56:15 [+0200], Lasse Collin wrote:
> > This way only one worker will be frequently locking the main mutex.
> > However, I haven't tried it and thus don't know how much this
> > affects performance in
th 32-bit
> userspace, or systems that use XPA (an extension similar to x86's
> PAE).
>
> So, for MIPS32, we have to impose stronger memory limits. I've chosen
> 2000MiB to give the process some headroom.
Thanks! Committed.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
Java 9
for module-info support but otherwise the code should still be
Java 5 compatible (see README and comments in build.properties).
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
ion has a fairly small effect.
Omitting it keeps the code a tiny bit simpler.
I have committed the change. I think xz-java.git should now be almost
ready for a release. I just need to add NEWS and bump the version
number.
Thanks for your help!
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
On 2021-02-13 Brett Okken wrote:
> On Thu, Feb 11, 2021 at 12:51 PM Lasse Collin
> wrote:
> > I still worry about short copies. If the file is full of tiny
> > matches/repeats of 1-3 bytes or so, arraycopy can be slower. Such
> > files aren't typical at all but I do
post a patch or other code, please make sure that word-wrapping
is disabled in the email client or use attachments. Thanks!
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
tem.arraycopy(buf, 0, buf, toCopy, remaining);
}
}
}
}
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
With real files the differences are smaller.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
MA)
> add_library(liblzma::liblzma ALIAS LibLZMA::LibLZMA)
If I change the main add_library(liblzma ) to add_library(LibLZMA
) then the filename will be LibLZMA.something too. That isn't
good because then one cannot replace a CMake-built shared liblzma with
an Autotools-built one on operating systems where file and library
names are case sensitive.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
else {
System.arraycopy(buf, back, buf, pos, copySize);
}
pos += copySize;
left -= copySize;
} while (left > 0);
if (full < pos)
full = pos;
}
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
yet that
it's worth the extra effort and complexity for such a small speed gain.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
look like
> > that writeChunk() could be overwriting the input.
>
> I assumed that lz.fillWindow(buf, off, len); would always process the
> 1 byte.
Yes, but it's not immediately obvious to a new reader. Also, many other
classes have tempBuf for identical use so it's good
On 2021-02-05 Brett Okken wrote:
> On Fri, Feb 5, 2021 at 11:07 AM Lasse Collin
> wrote:
> > Also, does it really help to unroll the loop? With 8191-byte
> > buffers I see no significant difference (in a quick
> > not-very-accurate test) if the switch-statement is replac
p it. It would be confusing to use the same buffer in
write(int) and writeChunk(). At glance it would look like that
writeChunk() could be overwriting the input.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
Is that an OK
version to commit?
Is the following fine to you as the file header? Your email address can
be omitted if you prefer that. I will mention in the commit message
that you adapted the code from XZ Utils and benchmarked it.
/*
* CRC64
*
* Authors: Brett Okken
* Lasse Collin
*
* This file has been put into the public domain.
* You can do whatever you want with this file.
*/
Thanks!
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
Stream even if one uses BufferedInputStream. BufferedInputStream
has synchronized read(). I don't know how much locking matters in this
case. I'm not curious enough to try with a non-synchronized buffered
input stream now.
There are related comments in the "java buffer writes" thread.
twork; such overheads can
be much larger than locking.
I put these optimizations in the "nice to have" category. Something
could be done to make the code better but it's not urgent and so these
won't be in the next release.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
On 2021-02-03 Brett Okken wrote:
> On Wed, Feb 3, 2021 at 2:56 PM Lasse Collin
> wrote:
> > It seems to regress horribly if dist is zero. A file with a very
> > long sequence of the same byte is good for testing.
>
> Would this be a valid test of what you are describin
1 - 100 of 298 matches
Mail list logo