Re: [PATCH 0/3] Update LZO compression
Thanks Arnd, On 2012-10-11 13:41, Arnd Bergmann wrote: On Tuesday 09 October 2012, Markus F.X.J. Oberhumer wrote: : This commit updates the kernel LZO code to the current upsteam version : which features a significant speed improvement - benchmarking the Calgary : and Silesia test corpora typically shows a doubled performance in : both compression and decompression on modern i386/x86_64/powerpc machines. There are significant clients of the LZO library - crypto, btrfs, jffs2, ubifs, squashfs and zcache. So let's give all those people a cc and ask that they test the LZO changes once they land in linux-next. For correctness and performance, please. The core compression and decompression code has been thoroughly tested, so I do not expect major problems. Good testing after the merge and feedback about build or performance issues (and improvements!) is highly appreciated. The addition of the lzo tree to linux-next caused this problem for ARM imx_v6_v7_defconfig: In file included from /home/arnd/linux-arm/arch/arm/boot/compressed/decompress.c:40:0: /home/arnd/linux-arm/arch/arm/boot/compressed/../../../../lib/decompress_unlzo.c:34:34: fatal error: lzo/lzo1x_decompress.c: No such file or directory Since the file was renamed, anything including it needs to be updated to the new file name. I will add that patch to my tree. Cheers, Markus Signed-off-by: Arnd Bergmann a...@arndb.de diff --git a/lib/decompress_unlzo.c b/lib/decompress_unlzo.c index 4531294..960183d 100644 --- a/lib/decompress_unlzo.c +++ b/lib/decompress_unlzo.c @@ -31,7 +31,7 @@ */ #ifdef STATIC -#include lzo/lzo1x_decompress.c +#include lzo/lzo1x_decompress_safe.c #else #include linux/decompress/unlzo.h #endif -- Markus Oberhumer, mar...@oberhumer.com, http://www.oberhumer.com/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/3] Update LZO compression
Hi Stephen, On 2012-10-09 21:26, Andrew Morton wrote: On Sun, 7 Oct 2012 17:07:55 +0200 Markus F.X.J. Oberhumer mar...@oberhumer.com wrote: As requested by akpm I am sending my lzo-update branch at git://github.com/markus-oberhumer/linux.git lzo-update to lkml as a patch series created by git format-patch -M v3.5..lzo-update. You can also browse the branch at https://github.com/markus-oberhumer/linux/compare/lzo-update and review the three patches at https://github.com/markus-oberhumer/linux/commit/7c979cebc0f93dc692b734c12665a6824d219c20 https://github.com/markus-oberhumer/linux/commit/10f6781c8591fe5fe4c8c733131915e5ae057826 https://github.com/markus-oberhumer/linux/commit/5f702781f158cb59075cfa97e5c21f52275057f1 The changes look OK to me. Please ask Stephen to include the tree in linux-next, for a 3.7 merge. I'd ask you to include my lzo-update branch in linux-next: git://github.com/markus-oberhumer/linux.git lzo-update The changelog for patch 2/3 says: : This commit updates the kernel LZO code to the current upsteam version : which features a significant speed improvement - benchmarking the Calgary : and Silesia test corpora typically shows a doubled performance in : both compression and decompression on modern i386/x86_64/powerpc machines. There are significant clients of the LZO library - crypto, btrfs, jffs2, ubifs, squashfs and zcache. So let's give all those people a cc and ask that they test the LZO changes once they land in linux-next. For correctness and performance, please. The core compression and decompression code has been thoroughly tested, so I do not expect major problems. Good testing after the merge and feedback about build or performance issues (and improvements!) is highly appreciated. Many thanks, Markus -- Markus Oberhumer, mar...@oberhumer.com, http://www.oberhumer.com/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL v2] Update LZO compression
Hi all, is there some official maintainer who feels responsible for pushing the LZO update to Linus? Otherwise I also could issue a pull-request on github, but somehow I don't think that this is the preferred workflow. Cheers, Markus On 2012-08-21 17:21, Markus F.X.J. Oberhumer wrote: Hi all, as suggested on the mailing list I have converted the updated LZO code into git, so please pull my lzo-update branch from git://github.com/markus-oberhumer/linux.git lzo-update You can browse the branch at https://github.com/markus-oberhumer/linux/compare/lzo-update I'd ask some official kernel maintainer for review and to push this into linux-next so that it will hopefully land in the 3.7 release. Share and enjoy, Markus Signed-off-by: Markus F.X.J. Oberhumer mar...@oberhumer.com [ Changes in v2: Optimize code for CPUs with inefficient unaligned access = significant speed increase on ARM ] On 2012-07-16 20:30, Markus F.X.J. Oberhumer wrote: Hi all, I finally have prepared a small package that updates the LZO version in the Linux kernel. Please get it from: http://www.oberhumer.com/opensource/lzo/download/Testing/linux-kernel-lzo-20120716.tar.gz As stated in the README this version is significantly faster (typically more than 2 times faster!) than the current version, has been thoroughly tested on x86_64/i386/powerpc platforms and is intended to get included into the official Linux 3.6 or 3.7 release. I encourage all compression users to test and benchmark this new version, and I also would ask some official LZO maintainer to convert the updated source files into a GIT commit and possibly push it to Linus or linux-next. Share and enjoy, Markus Signed-off-by: Markus F.X.J. Oberhumer mar...@oberhumer.com -- Markus Oberhumer, mar...@oberhumer.com, http://www.oberhumer.com/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL v2] Update LZO compression
Hi all, as suggested on the mailing list I have converted the updated LZO code into git, so please pull my lzo-update branch from git://github.com/markus-oberhumer/linux.git lzo-update You can browse the branch at https://github.com/markus-oberhumer/linux/compare/lzo-update I'd ask some official kernel maintainer for review and to push this into linux-next so that it will hopefully land in the 3.7 release. Share and enjoy, Markus Signed-off-by: Markus F.X.J. Oberhumer mar...@oberhumer.com [ Changes in v2: Optimize code for CPUs with inefficient unaligned access = significant speed increase on ARM ] On 2012-07-16 20:30, Markus F.X.J. Oberhumer wrote: Hi all, I finally have prepared a small package that updates the LZO version in the Linux kernel. Please get it from: http://www.oberhumer.com/opensource/lzo/download/Testing/linux-kernel-lzo-20120716.tar.gz As stated in the README this version is significantly faster (typically more than 2 times faster!) than the current version, has been thoroughly tested on x86_64/i386/powerpc platforms and is intended to get included into the official Linux 3.6 or 3.7 release. I encourage all compression users to test and benchmark this new version, and I also would ask some official LZO maintainer to convert the updated source files into a GIT commit and possibly push it to Linus or linux-next. Share and enjoy, Markus Signed-off-by: Markus F.X.J. Oberhumer mar...@oberhumer.com -- Markus Oberhumer, mar...@oberhumer.com, http://www.oberhumer.com/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] Update LZO compression
On 2012-08-15 16:45, Johannes Stezenbach wrote:
On Wed, Aug 15, 2012 at 02:02:43PM +0200, Markus F.X.J. Oberhumer wrote:
On 2012-08-14 14:39, Johannes Stezenbach wrote:
On Tue, Aug 14, 2012 at 01:44:02AM +0200, Markus F.X.J. Oberhumer wrote:
On 2012-07-16 20:30, Markus F.X.J. Oberhumer wrote:
As stated in the README this version is significantly faster (typically
more
than 2 times faster!) than the current version, has been thoroughly
tested on
x86_64/i386/powerpc platforms and is intended to get included into the
official Linux 3.6 or 3.7 release.
I encourage all compression users to test and benchmark this new version,
and I also would ask some official LZO maintainer to convert the updated
source files into a GIT commit and possibly push it to Linus or
linux-next.
Sorry for not reporting earlier, but I didn't have time to do real
benchmarks, just a quick test on ARM926EJ-S using barebox,
and found in the new version decompression is slower:
http://lists.infradead.org/pipermail/barebox/2012-July/008268.html
I can only guess, but maybe your ARM cpu does not have an efficient
implementation of {get,put}_unaligned().
Yes, ARMv5 cannot do unaligned access. ARMv6+ could, but
I think the Linux kernel normally traps it for debug,
all ARM seem to use generic {get,put}_unaligned() implementation
which use byte access and shift.
Hmm - I could imagine that we're wasting a lot of possible speed gain
by not exploiting that feature on ARMv6+.
Could you please try the following patch and test if you can see
any significant speed difference?
It isn't. I made the attached quick hack userspace code
using ARM kernel headers and barebox unlzop code.
(new == your new code, old == linux-3.5 git, test == new + your suggested
change)
(sorry I had no time to clean it up)
My suggested COPY4 replacement probably has a lot of load stalls - maybe some
ARM expert could have a look and suggest a more efficient implementation.
In any case, I still would like to see the new code in linux-next because
of the huge improvements on other modern CPUs.
Cheers,
Markus
I compressed a Linux Image with lzop (lzop arch/arm/boot/Image lzoimage)
and timed uncompression:
# time ./unlzopold lzoimage /dev/null
real0m 0.29s
user0m 0.19s
sys 0m 0.10s
# time ./unlzopold lzoimage /dev/null
real0m 0.29s
user0m 0.20s
sys 0m 0.09s
# time ./unlzopnew lzoimage /dev/null
real0m 0.41s
user0m 0.30s
sys 0m 0.10s
# time ./unlzopnew lzoimage /dev/null
real0m 0.40s
user0m 0.30s
sys 0m 0.10s
# time ./unlzopnew lzoimage /dev/null
real0m 0.40s
user0m 0.29s
sys 0m 0.11s
# time ./unlzoptest lzoimage /dev/null
real0m 0.39s
user0m 0.28s
sys 0m 0.11s
# time ./unlzoptest lzoimage /dev/null
real0m 0.39s
user0m 0.27s
sys 0m 0.11s
# time ./unlzoptest lzoimage /dev/null
real0m 0.39s
user0m 0.27s
sys 0m 0.11s
FWIW I also checked the sha1sum to confirm the Image uncompressed OK.
Johannes
--
Markus Oberhumer, mar...@oberhumer.com, http://www.oberhumer.com/
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] Update LZO compression
Hi Johannes,
On 2012-08-14 14:39, Johannes Stezenbach wrote:
On Tue, Aug 14, 2012 at 01:44:02AM +0200, Markus F.X.J. Oberhumer wrote:
On 2012-07-16 20:30, Markus F.X.J. Oberhumer wrote:
As stated in the README this version is significantly faster (typically more
than 2 times faster!) than the current version, has been thoroughly tested
on
x86_64/i386/powerpc platforms and is intended to get included into the
official Linux 3.6 or 3.7 release.
I encourage all compression users to test and benchmark this new version,
and I also would ask some official LZO maintainer to convert the updated
source files into a GIT commit and possibly push it to Linus or linux-next.
Sorry for not reporting earlier, but I didn't have time to do real
benchmarks, just a quick test on ARM926EJ-S using barebox,
and found in the new version decompression is slower:
http://lists.infradead.org/pipermail/barebox/2012-July/008268.html
I can only guess, but maybe your ARM cpu does not have an efficient
implementation of {get,put}_unaligned().
Could you please try the following patch and test if you can see
any significant speed difference?
Thanks,
Markus
diff --git a/lib/lzo/lzodefs.h b/lib/lzo/lzodefs.h
index ddc8db5..efc5714 100644
--- a/lib/lzo/lzodefs.h
+++ b/lib/lzo/lzodefs.h
@@ -12,8 +12,15 @@
*/
+#if defined(__arm__)
+#define COPY4(dst, src)\
+ (dst)[0] = (src)[0]; (dst)[1] = (src)[1]; \
+ (dst)[2] = (src)[2]; (dst)[3] = (src)[3]
+#endif
+#ifndef COPY4
#define COPY4(dst, src)\
put_unaligned(get_unaligned((const u32 *)(src)), (u32 *)(dst))
+#endif
#if defined(__x86_64__)
#define COPY8(dst, src)\
put_unaligned(get_unaligned((const u64 *)(src)), (u64 *)(dst))
BTW, do you have userspace code matching the old and new
lzo versions? It would be easier to benchmark.
Unfortunately I cannot claim high confidence in my benchmark results
due to missing time to do it properly, it would be useful if
someone else could do some benchmarks on ARM before merging this.
Johannes
--
Markus Oberhumer, mar...@oberhumer.com, http://www.oberhumer.com/
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] Update LZO compression
On 2012-08-14 05:15, Andi Kleen wrote: On Tue, Aug 14, 2012 at 01:44:02AM +0200, Markus F.X.J. Oberhumer wrote: Hi all, as suggested on the mailing list I have converted the updated LZO code into git, so please pull my lzo-update branch from git://github.com/markus-oberhumer/linux.git lzo-update You can browse the branch at https://github.com/markus-oberhumer/linux/compare/lzo-update Looks ok to me from a quick look. Since kernel lzo is security relevant, I assume the new version has been fuzz tested? Of course! I couldn't tell from the github view, but I assume you follow standard coding style. I've tried my best to follow the style guidelines. Cheers, Markus -Andi -- Markus Oberhumer, mar...@oberhumer.com, http://www.oberhumer.com/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] Update LZO compression
Hi all, as suggested on the mailing list I have converted the updated LZO code into git, so please pull my lzo-update branch from git://github.com/markus-oberhumer/linux.git lzo-update You can browse the branch at https://github.com/markus-oberhumer/linux/compare/lzo-update I'd ask some official kernel maintainer for review and to push this into linux-next so that it will hopefully land in the 3.7 release. Share and enjoy, Markus Signed-off-by: Markus F.X.J. Oberhumer mar...@oberhumer.com On 2012-07-16 20:30, Markus F.X.J. Oberhumer wrote: Hi all, I finally have prepared a small package that updates the LZO version in the Linux kernel. Please get it from: http://www.oberhumer.com/opensource/lzo/download/Testing/linux-kernel-lzo-20120716.tar.gz As stated in the README this version is significantly faster (typically more than 2 times faster!) than the current version, has been thoroughly tested on x86_64/i386/powerpc platforms and is intended to get included into the official Linux 3.6 or 3.7 release. I encourage all compression users to test and benchmark this new version, and I also would ask some official LZO maintainer to convert the updated source files into a GIT commit and possibly push it to Linus or linux-next. Share and enjoy, Markus Signed-off-by: Markus F.X.J. Oberhumer mar...@oberhumer.com -- Markus Oberhumer, mar...@oberhumer.com, http://www.oberhumer.com/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
