Thanks Sašo! Comments below... On Jul 10, 2012, at 4:56 PM, Sašo Kiselkov wrote:
> Hi guys, > > I'm contemplating implementing a new fast hash algorithm in Illumos' ZFS > implementation to supplant the currently utilized sha256. No need to supplant, there are 8 bits for enumerating hash algorithms, so adding another is simply a matter of coding. With the new feature flags, it is almost trivial to add new algorithms without causing major compatibility headaches. Darren points out that Oracle is considering doing the same, though I do not expect Oracle to pick up the feature flags. > On modern > 64-bit CPUs SHA-256 is actually much slower than SHA-512 and indeed much > slower than many of the SHA-3 candidates, so I went out and did some > testing (details attached) on a possible new hash algorithm that might > improve on this situation. > > However, before I start out on a pointless endeavor, I wanted to probe > the field of ZFS users, especially those using dedup, on whether their > workloads would benefit from a faster hash algorithm (and hence, lower > CPU utilization). Developments of late have suggested to me three > possible candidates: > > * SHA-512: simplest to implement (since the code is already in the > kernel) and provides a modest performance boost of around 60%. > > * Skein-512: overall fastest of the SHA-3 finalists and much faster > than SHA-512 (around 120-150% faster than the current sha256). > > * Edon-R-512: probably the fastest general purpose hash algorithm I've > ever seen (upward of 300% speedup over sha256) , but might have > potential security problems (though I don't think this is of any > relevance to ZFS, as it doesn't use the hash for any kind of security > purposes, but only for data integrity & dedup). > > My testing procedure: nothing sophisticated, I took the implementation > of sha256 from the Illumos kernel and simply ran it on a dedicated > psrset (where possible with a whole CPU dedicated, even if only to a > single thread) - I tested both the generic C implementation and the > Intel assembly implementation. The Skein and Edon-R implementations are > in C optimized for 64-bit architectures from the respective authors (the > most up to date versions I could find). All code has been compiled using > GCC 3.4.3 from the repos (the same that can be used for building > Illumos). Sadly, I don't have access to Sun Studio. The last studio release suitable for building OpenSolaris is available in the repo. See the instructions at http://wiki.illumos.org/display/illumos/How+To+Build+illumos I'd be curious about whether you see much difference based on studio 12.1, gcc 3.4.3 and gcc 4.4 (or even 4.7) -- richard -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
