On Mon, Jul 11, 2016 at 12:45 PM, Rob Landley <[email protected]> wrote: > On 07/11/2016 11:46 AM, enh wrote: >> On Sun, Jul 10, 2016 at 10:28 AM, Rob Landley <[email protected]> wrote: >> another plug for supporting a libcrypto dependency: the *sum utilities >> are orders of magnitude faster with libcrypto, the SSL support in >> things like netcat/wget would be something i could actually use on >> Android (there's no way i'll be able to ship an alternative SSL >> implementation) and you'd get arbitrary precision integers too. > > What I want to do is take the approach Isaac Dunham suggested, of using > "openssl s_client -quiet -connect" as an alternative to netcat. So > toybox wget should call out to that to get https support, and that would > be provided by something external. > > Lipi Lee did a first pass at this already, which I didn't immediately > apply because for some reason the patch he sent me didn't apply to the > wget he sent me (I don't think I'd modified it). When I tested the wget > it was corrupting the files it downloaded (outputting numbers in the > middle of the data), and that sent me down the road of rewriting the > thing... > > I'm balancing some competing design goals here: 'self-contained' vs > 'people use this and need speed out of some tightly optimized > algorithms'. The way busybox dealt with this was by having multiple > implementations (CONFIG_MD5_SMALL has 4 settings), which I very much > don't want to do... > > The problem with having an internal an external implementation is the > internal one gets much less testing that way. I suspect the right answer > is to just lump it and have the actual unrolled fast version in toybox, > because the simple one isn't good enough for the userbase. That said, a > lot of these external libraries have assembly optimized versions for > various platforms, and I KNOW I'm not going there... > > Hmmm. Which lib is "libcrypto", by the way?
any flavor of openssl/libressl/boringssl. they all share the same API for this subset. boringssl -- which Google uses -- is basically just openssl cut down to "what you actually need in the modern world" anyway. boringssl has an equivalent that would make the s_client approach work, so that's fine by me for netcat/wget. >> i don't think you're likely to go this route, but i do like to keep >> bringing it up so the idea of being API-compatible enough that it's >> possible to use toybox with either your backend or *ssl is in the back >> of your mind... >> >> (no one's complained about the slow *sum commands yet, but if you're >> interested i'm happy to send a patch.) > > People have sent patches to speed up md5sum and sha1sum and it boils > down to lots of loop unrolling that makes the algorithm harder to > understand. It was back around here: > http://lists.landley.net/pipermail/toybox-landley.net/2014-May/006638.html > > I applied the first few, but the code got very large and very unreadable > and I kept hoping there was a way the compiler's darn optimizer could do > that for me. I should go back and look at those patches again, but it's > competing with 60 other todo items... oh, no, my patch just left your portable-but-slow implementations in place and called *ssl if configured that way. so it's basically one or two extra lines per toy (an if and a function call). > According to http://valerieaurora.org/hash.html both md5sum and sha1sum > are semi-obsolete. I need to do sha256 and sha3 and so on, and adding well, i did sha256, sha384, and sha512 too. no reason not to when they're all there for free. plus being able to say "if you care about performance, link against *ssl and get hand-optimized assembler for your specific target platform" gets all the loop unrollers off your lawn forever :-) > those is a higher priority todo item for me than speeding up the old > ones. But despite being obsolete as _cryopto_, rsync moved from md4sum > to md5sum a couple years back, and git is based on sha1, and neither is > moving off those because you wrap the transport in https and sign the > commits if you care about security... > > My deflate implementation is also half the speed it should be, although > the first pass was focusing on correctness rather than any kind of > optimization. (In theory zlib started life as a public domain > implementation, but that version seems to have fallen off the net and > doesn't have a lot of modern optimizations anyway.) > > One thing I've been meaning to do with deflate/inflate is add the SMP > mode (where it outputs zero byte packets at each dictionary reset so you > can scan ahead for them and do blocks in parallel). > > So many todo items... > > Rob -- Elliott Hughes - http://who/enh - http://jessies.org/~enh/ Android native code/tools questions? Mail me/drop by/add me as a reviewer. _______________________________________________ Toybox mailing list [email protected] http://lists.landley.net/listinfo.cgi/toybox-landley.net
