looks like i never sent my previous patch to the list, and looks like i lost it on disk too. (it's probably still *somewhere*, but at some point it's cheaper to stop looking and just do it again...)
attached is a rewrite that shows the basic idea, but misses the step of refactoring your portable implementation to look like the libcrypto one so we can remove the 10 lines of duplication. (the majority of the lines added to this patch is just the same help text duplicated over and over.) this actually makes toybox faster than the version in ubuntu 14.04. [PATCH] Add support for libcrypto for MD5/SHA. Orders of magnitude faster (for architectures where OpenSSL/BoringSSL has optimized assembler). Also adds sha224sum, sha256sum, sha384sum, and sha512sum for folks building with libcrypto. The fallback portable C implementations could easily be refactored to be API-compatible, but I don't know whether they'd stay here or move to lib/ so I've left that part alone for now. --- Config.in | 6 +++ scripts/make.sh | 2 +- toys/lsb/md5sum.c | 133 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 135 insertions(+), 6 deletions(-) On Mon, Jul 11, 2016 at 2:21 PM, Rob Landley <[email protected]> wrote: > On 07/11/2016 03:10 PM, enh wrote: > > On Mon, Jul 11, 2016 at 12:45 PM, Rob Landley <[email protected]> wrote: > >> On 07/11/2016 11:46 AM, enh wrote: > >>> On Sun, Jul 10, 2016 at 10:28 AM, Rob Landley <[email protected]> wrote: > >>> another plug for supporting a libcrypto dependency: the *sum utilities > >>> are orders of magnitude faster with libcrypto, the SSL support in > >>> things like netcat/wget would be something i could actually use on > >>> Android (there's no way i'll be able to ship an alternative SSL > >>> implementation) and you'd get arbitrary precision integers too. > >> > >> What I want to do is take the approach Isaac Dunham suggested, of using > >> "openssl s_client -quiet -connect" as an alternative to netcat. So > >> toybox wget should call out to that to get https support, and that would > >> be provided by something external. > >> > >> Lipi Lee did a first pass at this already, which I didn't immediately > >> apply because for some reason the patch he sent me didn't apply to the > >> wget he sent me (I don't think I'd modified it). When I tested the wget > >> it was corrupting the files it downloaded (outputting numbers in the > >> middle of the data), and that sent me down the road of rewriting the > >> thing... > >> > >> I'm balancing some competing design goals here: 'self-contained' vs > >> 'people use this and need speed out of some tightly optimized > >> algorithms'. The way busybox dealt with this was by having multiple > >> implementations (CONFIG_MD5_SMALL has 4 settings), which I very much > >> don't want to do... > >> > >> The problem with having an internal an external implementation is the > >> internal one gets much less testing that way. I suspect the right answer > >> is to just lump it and have the actual unrolled fast version in toybox, > >> because the simple one isn't good enough for the userbase. That said, a > >> lot of these external libraries have assembly optimized versions for > >> various platforms, and I KNOW I'm not going there... > >> > >> Hmmm. Which lib is "libcrypto", by the way? > > > > any flavor of openssl/libressl/boringssl. they all share the same API > > for this subset. > > > > boringssl -- which Google uses -- is basically just openssl cut down > > to "what you actually need in the modern world" anyway. boringssl has > > an equivalent that would make the s_client approach work, so that's > > fine by me for netcat/wget. > > > >>> i don't think you're likely to go this route, but i do like to keep > >>> bringing it up so the idea of being API-compatible enough that it's > >>> possible to use toybox with either your backend or *ssl is in the back > >>> of your mind... > >>> > >>> (no one's complained about the slow *sum commands yet, but if you're > >>> interested i'm happy to send a patch.) > >> > >> People have sent patches to speed up md5sum and sha1sum and it boils > >> down to lots of loop unrolling that makes the algorithm harder to > >> understand. It was back around here: > >> > http://lists.landley.net/pipermail/toybox-landley.net/2014-May/006638.html > >> > >> I applied the first few, but the code got very large and very unreadable > >> and I kept hoping there was a way the compiler's darn optimizer could do > >> that for me. I should go back and look at those patches again, but it's > >> competing with 60 other todo items... > > > > oh, no, my patch just left your portable-but-slow implementations in > > place and called *ssl if configured that way. so it's basically one or > > two extra lines per toy (an if and a function call). > > Ok, I'll bite. What does your patch look like? > > (If I can shove it in portability.c, retain the option of _not_ doing it > via menuconfig, and probably only support static linking of this extra > library... I'm still mildly concerned about the built-in version getting > less testing, but hashes aren't likely to fail in a _subtle_ manner...) > > Rob > -- Elliott Hughes - http://who/enh - http://jessies.org/~enh/ Android native code/tools questions? Mail me/drop by/add me as a reviewer.
From bce8929997bcf8f43bbb03b46d0fe92ea68601a5 Mon Sep 17 00:00:00 2001 From: Elliott Hughes <[email protected]> Date: Wed, 13 Jul 2016 15:15:06 -0700 Subject: [PATCH] Add support for libcrypto for MD5/SHA. Orders of magnitude faster (for architectures where OpenSSL/BoringSSL has optimized assembler). Also adds sha224sum, sha256sum, sha384sum, and sha512sum for folks building with libcrypto. The fallback portable C implementations could easily be refactored to be API-compatible, but I don't know whether they'd stay here or move to lib/ so I've left that part alone for now. --- Config.in | 6 +++ scripts/make.sh | 2 +- toys/lsb/md5sum.c | 133 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 135 insertions(+), 6 deletions(-) diff --git a/Config.in b/Config.in index 3c560f8..da27e2e 100644 --- a/Config.in +++ b/Config.in @@ -62,6 +62,12 @@ config TOYBOX_SMACK endchoice +config TOYBOX_LIBCRYPTO + bool "Use libcrypto (OpenSSL/BoringSSL)" + default n + help + Use libcrypto to implement hashing. + config TOYBOX_FLOAT bool "Floating point support" default y diff --git a/scripts/make.sh b/scripts/make.sh index 07bf2ac..50415e5 100755 --- a/scripts/make.sh +++ b/scripts/make.sh @@ -103,7 +103,7 @@ then # for it. > generated/optlibs.dat - for i in util crypt m resolv selinux smack attr rt + for i in util crypt m resolv selinux smack attr rt crypto do echo "int main(int argc, char *argv[]) {return 0;}" | \ ${CROSS_COMPILE}${CC} $CFLAGS -xc - -o generated/libprobe -Wl,--as-needed -l$i > /dev/null 2>/dev/null && diff --git a/toys/lsb/md5sum.c b/toys/lsb/md5sum.c index 12f0450..e1cc3a4 100644 --- a/toys/lsb/md5sum.c +++ b/toys/lsb/md5sum.c @@ -1,4 +1,4 @@ -/* md5sum.c - Calculate RFC 1321 md5 hash and sha1 hash. +/* md5sum.c - Calculate RFC 1321 md5 hash and various sha hashes. * * Copyright 2012 Rob Landley <[email protected]> * @@ -10,6 +10,10 @@ USE_MD5SUM(NEWTOY(md5sum, "b", TOYFLAG_USR|TOYFLAG_BIN)) USE_SHA1SUM(NEWTOY(sha1sum, "b", TOYFLAG_USR|TOYFLAG_BIN)) +USE_SHA224SUM(NEWTOY(sha224sum, "b", TOYFLAG_USR|TOYFLAG_BIN)) +USE_SHA256SUM(NEWTOY(sha256sum, "b", TOYFLAG_USR|TOYFLAG_BIN)) +USE_SHA384SUM(NEWTOY(sha384sum, "b", TOYFLAG_USR|TOYFLAG_BIN)) +USE_SHA512SUM(NEWTOY(sha512sum, "b", TOYFLAG_USR|TOYFLAG_BIN)) config MD5SUM bool "md5sum" @@ -18,7 +22,7 @@ config MD5SUM usage: md5sum [FILE]... Calculate md5 hash for each input file, reading from stdin if none. - Output one hash (16 hex digits) for each input file, followed by + Output one hash (32 hex digits) for each input file, followed by filename. -b brief (hash only, no filename) @@ -29,8 +33,60 @@ config SHA1SUM help usage: sha1sum [FILE]... - calculate sha1 hash for each input file, reading from stdin if none. - Output one hash (20 hex digits) for each input file, followed by + Calculate sha1 hash for each input file, reading from stdin if none. + Output one hash (40 hex digits) for each input file, followed by + filename. + + -b brief (hash only, no filename) + +config SHA224SUM + bool "sha224sum" + default y + depends on TOYBOX_LIBCRYPTO + help + usage: sha224sum [FILE]... + + Calculate sha224 hash for each input file, reading from stdin if none. + Output one hash (56 hex digits) for each input file, followed by + filename. + + -b brief (hash only, no filename) + +config SHA256SUM + bool "sha256sum" + default y + depends on TOYBOX_LIBCRYPTO + help + usage: sha256sum [FILE]... + + Calculate sha256 hash for each input file, reading from stdin if none. + Output one hash (64 hex digits) for each input file, followed by + filename. + + -b brief (hash only, no filename) + +config SHA384SUM + bool "sha384sum" + default y + depends on TOYBOX_LIBCRYPTO + help + usage: sha384sum [FILE]... + + Calculate sha384 hash for each input file, reading from stdin if none. + Output one hash (96 hex digits) for each input file, followed by + filename. + + -b brief (hash only, no filename) + +config SHA512SUM + bool "sha512sum" + default y + depends on TOYBOX_LIBCRYPTO + help + usage: sha512sum [FILE]... + + Calculate sha512 hash for each input file, reading from stdin if none. + Output one hash (128 hex digits) for each input file, followed by filename. -b brief (hash only, no filename) @@ -39,7 +95,13 @@ config SHA1SUM #define FOR_md5sum #include "toys.h" +#if CFG_TOYBOX_LIBCRYPTO +#include <openssl/md5.h> +#include <openssl/sha.h> +#endif + GLOBALS( + // TODO: only used for non-libcrypto. unsigned state[5]; unsigned oldstate[5]; uint64_t count; @@ -189,8 +251,48 @@ static void hash_update(char *data, unsigned int len, void (*transform)(void)) static void do_hash(int fd, char *name) { +#if CFG_TOYBOX_LIBCRYPTO + +#define HASH_INIT(name, prefix) { name, (void *) prefix ## _Init, \ + (void *) prefix ## _Update, (void *) prefix ## _Final, \ + prefix ## _DIGEST_LENGTH, } +#define SHA1_DIGEST_LENGTH SHA_DIGEST_LENGTH + + union { + MD5_CTX md5; + SHA_CTX sha; + } ctx; + struct hash { + const char *name; + int (*init)(void *); + int (*update)(void *, const void *, size_t); + int (*final)(void *, void *); + int digest_length; + } algorithms[] = { + HASH_INIT("md5sum", MD5), HASH_INIT("sha1sum", SHA1), + HASH_INIT("sha224sum", SHA224), HASH_INIT("sha256sum", SHA256), + HASH_INIT("sha384sum", SHA384), HASH_INIT("sha512sum", SHA512), + }, * hash; + int i; + + for (i = 0; i < ARRAY_LEN(algorithms); i++) + if (!strcmp(toys.which->name, algorithms[i].name)) break; + hash = &algorithms[i]; + + hash->init(&ctx); + + for (;;) { + i = read(fd, toybuf, sizeof(toybuf)); + if (i<1) break; + hash->update(&ctx, toybuf, i); + } + hash->final(toybuf, &ctx); + + for (i = 0; i < hash->digest_length; i++) + printf("%02x", toybuf[i]); +#else uint64_t count; - int i, sha1=toys.which->name[0]=='s';; + int i, sha1=toys.which->name[0]=='s'; char buf; void (*transform)(void); @@ -233,6 +335,7 @@ static void do_hash(int fd, char *name) // Wipe variables. Cryptographer paranoia. memset(&TT, 0, sizeof(TT)); +#endif printf((toys.optflags & FLAG_b) ? "\n" : " %s\n", name); } @@ -246,3 +349,23 @@ void sha1sum_main(void) { md5sum_main(); } + +void sha224sum_main(void) +{ + md5sum_main(); +} + +void sha256sum_main(void) +{ + md5sum_main(); +} + +void sha384sum_main(void) +{ + md5sum_main(); +} + +void sha512sum_main(void) +{ + md5sum_main(); +} -- 2.8.0.rc3.226.g39d4020
_______________________________________________ Toybox mailing list [email protected] http://lists.landley.net/listinfo.cgi/toybox-landley.net
