Re: Linus' sha1 is much faster!
Pádraig Brady p...@draigbrady.com writes: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i586 -mtune=generic -fasynchronous-unwind-tables -D_GNU_SOURCE=1 thanks. I did again all tests on my machine using these same options. I repeated each test 6 times and I took the median without consider the first result. Except the first run that it is not considered, I didn't report a big variance on results of the same test. gcc 4.3.3 gnulib sha1:real0m2.543s gnulib sha1 lookup: real0m1.906s (-25%) linus's sha1: real0m2.468s (-3%) linus's sha1 no asm:real0m2.289s (-9%) gcc 4.4.1 gnulib sha1:real0m3.386s gnulib sha1 lookup: real0m3.110s (-8%) linus's sha1: real0m1.701s (-49%) linus's sha1 no asm:real0m1.284s (-62%) I don't see such big differences in asm generated by gcc 4.4.1 and gcc 4.3.3 to explain this performance difference, what I noticed immediately is that in the gcc-4.4 generated asm there are more lea instructions (+30%), but I doubt this is the reason of these poor results. Anyway, I haven't yet looked much in details. Cheers, Giuseppe
Re: Linus' sha1 is much faster!
Linus Torvalds wrote: On Sat, 15 Aug 2009, Linus Torvalds wrote: That said, I don't know if the MPL is ok for X11. I've not looked at compatibility issues with MPL. For git, we could just ignore the MPL, since the GPLv2 was acceptable regardless of it. If MPL isn't ok for X11, then we'd need to make sure that even the silliest Mozilla crud has been rewritten. There really isn't much, but hey, the _history_ is based on the mozilla code, and who knows - the 'blk_SHA_CTX' struct has things like the fields in the same order as the Mozilla equivalent, for all those historical reasons. (Heh. Looking at that, I probably should move the 'size' field first, since that would have different alignment rules, and the struct would be more tightly packed that way, and initialize better). Afaik, none of the actual code remains (the mozilla SHA1 thing did the wrong thing for performance even for just the final bytes, and did those a byte at a time etc, so I rewrote even the trivial SHA1_Final parts). Of course, maybe the Mozilla people would be interested in taking my faster version, and say that the new-BSD license is ok, and make everybody happy. The only listed author for the Mozilla SHA1 is Paul Kocher. I added him to the Cc. Paul, for your information, we're talking about a faster rewritten mostly portable SHA1 routines that you can find at http://git.kernel.org/?p=git/git.git;a=tree;f=block-sha1;hb=pu (follow the blob pointers to see sha1.c and sha1.h). I don't know if you're active with Mozilla/Firefox or whether you even care, but you seem to be the logical choice of person to ask. I contacted Paul in february this year to get permission to use the mozilla sha1 code for libgit2. His reply then was: I'm not sure which version the diffs are relative to, so I haven't reviewed them. It's fine to distribute under BSD, GPL, or LGPL, however. I also got explicit permission to relicense it under GPLv2 with the gcc exception. I added the mail-address I used to contact him to CC as well. Sorry if you get this twice, Paul. Naturally, I'd like to use the faster version for libgit2 as well. The people who Linus listed as contributors earlier (Brandon Casy, Linus, Junio and Nicolas Pitre) have already consented to relicense their git contributions for libgit2 use. If anyone would like to revoke that consent for this code, speak now please, or I'll patch it into libgit2 as well. -- Andreas Ericsson andreas.erics...@op5.se OP5 AB www.op5.se Tel: +46 8-230225 Fax: +46 8-230231 Considering the successes of the wars on alcohol, poverty, drugs and terror, I think we should give some serious thought to declaring war on peace.
chroot diff. errors?
Hi; Sometimes the command chroot does not work, for instance ,earlier the same command (below) worked in the same directory! But now I get these: r...@ubuntu9:diskchroot . /bin/bash: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory r...@ubuntu9:diskpwd /media/disk Then I thought may be since I was running rsync it gave the above error, I exited rsync then: r...@ubuntu9:diskchroot . chroot: cannot run command `/bin/bash': Exec format error I checked /bin and bash exists (dates in March 2nd 2009/ no changes)) The /media/disk is a hard drive not on obtical but I changed directories anyway but then I get this other error: r...@ubuntu9:xenchroot /mnt/xen/ chroot: cannot run command `/bin/bash': No such file or directory r...@ubuntu9:xenpwd /mnt/xen r...@ubuntu9:xenls /bin/bash -l -rwxr-xr-x 1 root root 729040 2009-03-02 14:22 /bin/bash Sorry for the long email, I am not looking for answer or reply, it is probably something I am running/doing?! Though as I said the chroot worked earlier today on the same machine!? Thank You Very Much! Mehdi B. Oh.. BTW I forgot: uname -a Linux Ubuntu9 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC 2009 i686 GNU/Linux r...@ubuntu9:disklsb_release -a No LSB modules are available. Distributor ID:Ubuntu Description:Ubuntu 9.04 Release:9.04 Codename:jaunty Best Regards; Mehdi Bagheri Austin Texas USA Southampton England (UK) _ Get back to school stuff for them and cashback for you. http://www.bing.com/cashback?form=MSHYCBpubl=WLHMTAGcrea=TEXT_MSHYCB_BackToSchool_Cashback_BTSCashback_1x1
Re: Linus' sha1 is much faster!
On Mon, Aug 17, 2009 at 3:51 AM, Giuseppe Scrivanogscriv...@gnu.org wrote: Pádraig Brady p...@draigbrady.com writes: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i586 -mtune=generic -fasynchronous-unwind-tables -D_GNU_SOURCE=1 thanks. I did again all tests on my machine using these same options. I repeated each test 6 times and I took the median without consider the first result. Except the first run that it is not considered, I didn't report a big variance on results of the same test. gcc 4.3.3 gnulib sha1: real 0m2.543s gnulib sha1 lookup: real 0m1.906s (-25%) linus's sha1: real 0m2.468s (-3%) linus's sha1 no asm: real 0m2.289s (-9%) gcc 4.4.1 gnulib sha1: real 0m3.386s gnulib sha1 lookup: real 0m3.110s (-8%) linus's sha1: real 0m1.701s (-49%) linus's sha1 no asm: real 0m1.284s (-62%) I don't see such big differences in asm generated by gcc 4.4.1 and gcc 4.3.3 to explain this performance difference, what I noticed immediately is that in the gcc-4.4 generated asm there are more lea instructions (+30%), but I doubt this is the reason of these poor results. Anyway, I haven't yet looked much in details. Cheers, Giuseppe Interesting. I compared Linus' implementation to the public domain one by Steve Reid[1], which is used in OpenLDAP and a few other projects. Anyone with some experience testing these kinds of things in a statistically sound manner want to try it out? In my tests, I got this: (average of 5 runs) Linus' sha1: 283MB/s Steve Reid's sha1: 305MB/s - Steven [1] http://gpl.nas-central.org/SYNOLOGY/x07-series/514_UNTARED/source/openldap-2.3.11/libraries/liblutil/sha1.c
Re: Linus' sha1 is much faster!
On Mon, 17 Aug 2009, Steven Noonan wrote: Interesting. I compared Linus' implementation to the public domain one by Steve Reid[1] You _really_ need to talk about what kind of environment you have. There are three major issues: - Netburst vs non-netburst - 32-bit vs 64-bit - compiler version Steve Reid's code looks great, but the way it is coded, gcc makes a mess of it, which is exactly what my SHA1 tries to avoid. [ In contrast, gcc does very well on just about _any_ straightforward unrolled SHA1 C code if the target architecture is something like PPC or ia64 that has enough registers to keep it all in registers. I haven't really tested other compilers - a less aggressive compiler would actually do _better_ on SHA1, because the problem with gcc is that it turns the whole temporary 16-entry word array into register accesses, and tries to do register allocation on that _array_. That is wonderful for the above-mentioned PPC and IA64, but it makes gcc create totally crazy code when there aren't enough registers, and then gcc starts spilling randomly (ie it starts spilling a-e etc). This is why the compiler and version matters so much. ] (average of 5 runs) Linus' sha1: 283MB/s Steve Reid's sha1: 305MB/s So I get very different results: # TIME[s] SPEED[MB/s] Reid2.742 222.6 linus 1.464 417 this is Intel Nehalem, but compiled for 32-bit mode (which is the more challenging one because x86-32 only has 7 general-purpose registers), and with gcc-4.4.0. Linus
Re: Linus' sha1 is much faster!
Hi, These are the results I reported (median of 5 plus an additional not considered first run) on the Steve Reid's SHA1 implementation using the same flags to the compiler that I used for previous tests. GCC 4.3.3: real0m2.627s GCC 4.4.1: real0m3.742s In both cases it showed to be slower than other implementations I have already tried. Additional note: as for gnulib SHA1, GCC 4.4.1 produced slower code than GCC 4.3.3. Cheers, Giuseppe Steven Noonan ste...@uplinklabs.net writes: Interesting. I compared Linus' implementation to the public domain one by Steve Reid[1], which is used in OpenLDAP and a few other projects. Anyone with some experience testing these kinds of things in a statistically sound manner want to try it out? In my tests, I got this: (average of 5 runs) Linus' sha1: 283MB/s Steve Reid's sha1: 305MB/s - Steven [1] http://gpl.nas-central.org/SYNOLOGY/x07-series/514_UNTARED/source/openldap-2.3.11/libraries/liblutil/sha1.c
[patch #2565] added option to sort: sort human readable file sizes
Update of patch #2565 (project coreutils): Status:None = In Progress ___ Follow-up Comment #2: --human-numeric-sort has been added in upstream git, for the imminent coreutils-7.5 release ___ Reply to this item at: http://savannah.gnu.org/patch/?2565 ___ Message sent via/by Savannah http://savannah.gnu.org/
Re: new snapshot available: coreutils-7.4.125-eca6
Mike Frysinger wrote: Changes in coreutils since 7.4.115-c9c92: `make make check` passes for me: - non-root user - glibc-2.10.1 - gcc-4.4.1 - linux-2.6.30.4 - x86_64 system Good to hear. Thanks for the speedy feedback!
Re: Linus' sha1 is much faster!
On Mon, Aug 17, 2009 at 9:22 AM, Linus Torvaldstorva...@linux-foundation.org wrote: On Mon, 17 Aug 2009, Steven Noonan wrote: Interesting. I compared Linus' implementation to the public domain one by Steve Reid[1] You _really_ need to talk about what kind of environment you have. There are three major issues: - Netburst vs non-netburst - 32-bit vs 64-bit - compiler version Right. I'm running a Core 2 Merom 2.33GHz. The code was compiled for x86_64 with GCC 4.2.1. I didn't _expect_ it to compile for x86_64, but apparently the version of GCC that ships with Xcode 3.2 defaults to compiling 64-bit code on machines that are capable of running it. Steve Reid's code looks great, but the way it is coded, gcc makes a mess of it, which is exactly what my SHA1 tries to avoid. [ In contrast, gcc does very well on just about _any_ straightforward unrolled SHA1 C code if the target architecture is something like PPC or ia64 that has enough registers to keep it all in registers. I haven't really tested other compilers - a less aggressive compiler would actually do _better_ on SHA1, because the problem with gcc is that it turns the whole temporary 16-entry word array into register accesses, and tries to do register allocation on that _array_. That is wonderful for the above-mentioned PPC and IA64, but it makes gcc create totally crazy code when there aren't enough registers, and then gcc starts spilling randomly (ie it starts spilling a-e etc). This is why the compiler and version matters so much. ] (average of 5 runs) Linus' sha1: 283MB/s Steve Reid's sha1: 305MB/s So I get very different results: # TIME[s] SPEED[MB/s] Reid 2.742 222.6 linus 1.464 417 Added -m32: Steve Reid: 156MB/s Linus: 209MB/s So on x86, your code really kicks butt. this is Intel Nehalem, but compiled for 32-bit mode (which is the more challenging one because x86-32 only has 7 general-purpose registers), and with gcc-4.4.0. Linus
Re: new snapshot available: coreutils-7.4.125-eca6
On Mon, Aug 17, 2009 at 10:51:54PM +0200, Jim Meyering wrote: Mike Frysinger wrote: Changes in coreutils since 7.4.115-c9c92: `make make check` passes for me: - non-root user - glibc-2.10.1 - gcc-4.4.1 - linux-2.6.30.4 - x86_64 system Good to hear. Thanks for the speedy feedback! make make check passes for me too: - non-root - Debian/sid (linux 2.6.30, gcc 4.3.4, glibc 2.9) - x86_32 system Erik
Re: chroot diff. errors?
Mehdi _1 wrote: Sometimes the command chroot does not work, for instance ,earlier the same command (below) worked in the same directory! But now I get these: r...@ubuntu9:diskchroot . /bin/bash: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory r...@ubuntu9:diskpwd /media/disk This means that loading libncurses.so.5 needed by bash in the chroot failed. In the chroot probably means /media/disk/lib/libncurses.so.5 inside the chroot but also means any dependent libraries too. Do all of those libraries exist in the chroot? The /media/disk is a hard drive not on obtical but I changed directories anyway but then I get this other error: r...@ubuntu9:xenchroot /mnt/xen/ chroot: cannot run command `/bin/bash': No such file or directory That means that /mnt/xen/bin/bash could not be invoked in the chroot. Does it exist there? Sorry for the long email, I am not looking for answer or reply, it is probably something I am running/doing?! Though as I said the chroot worked earlier today on the same machine!? The most typical problem people have with creating chroots is not installing all of the necessary executables and dependent libraries inside the chroot. The documentation for chroot says: If you want to use a dynamically linked executable, say `bash', then first run `ldd bash' to see what shared objects it needs. Then, in addition to copying the actual binary, also copy the listed files to the required positions under your intended new root directory. Finally, if the executable requires any other files (e.g., data, state, device files), copy them into place, too. Bob
Re: new snapshot available: coreutils-7.4.125-eca6
Erik Auerswald wrote: make make check passes for me too: - non-root - Debian/sid (linux 2.6.30, gcc 4.3.4, glibc 2.9) - x86_32 system Thank you, too!
