Hi Frank, > On Jul 14, 2016, at 12:23 AM, Frank Siebenlist <frank.siebenl...@gmail.com> > wrote: > > Python's native hashing module (hashlib), shows similar results: > - about the same time when passed the 8MB blob in one go > (probably expected as both use openssl) > - substantial overhead when looping over small chunks (up to 100 times) > - except that it's about 6 times faster per single byte..
The perf by chunk is a consequence of how SHA256 works. The higher perf for many calls is a consequence of extension modules vs cffi. lvh > n: 8388608 > b: 1.958238124847412 > b2: 1.0818939208984375 > b8: 0.2987058162689209 > b32: 0.10640311241149902 > b64: 0.06242084503173828 > b128: 0.04123806953430176 > b256: 0.03258681297302246 > 8388608 bs: 0.02389383316040039 > > Guess hashlib used some better optimization on the C-calls (?). > > This is my last update on this observation. > Conclusion is "so be it", and using bigger chunks for hashing gives > (much) better performance. > > -Frank. > > On Tue, Jul 12, 2016 at 10:49 AM, Frank Siebenlist > <frank.siebenl...@gmail.com> wrote: >> After I sent my message yesterday evening, I was also wondering about >> that 512bit (64byte) block-size of sha256, and if that would add to >> the observed slowness. >> The following output shows time as a function of byte-chunk size >> (1,2,8,32,64,128,256 bytes) >> >> b: 12.111763954162598 >> b2: 5.806451082229614 >> b8: 1.4664850234985352 >> b32: 0.37551307678222656 >> b64: 0.20229697227478027 >> b128: 0.11141395568847656 >> b256: 0.06758689880371094 >> 8388608 bs: 0.020879030227661133 >> >> Time seems to go down linearly with increase of chunk size, and there >> is no perceived "speed boost" when we go through the 64byte >> thresh-hold. >> Time seems to be only linearly related to the number of python-to-C calls. >> >> And again, I can understand that the overhead is proportional to the >> number of python-to-C calls, but it's just the factor of 500 (2-3 >> order of magnitude) that (unpleasantly) surprised me. It requires one >> to optimize on byte-string size to pass in the update(), when you have >> many bytes to hash. For example, if you read from a file or socket, >> don't update() 1 byte at the time while you read from the stream, but >> fill-up a (big) buffer first and pass that buffer. >> >> -Frank. >> >> PS. I haven't looked at the sha256 C-code, but I can imagine that when >> you pass the update() one byte at the time, it will fill-up some >> 64byte-buffer, and if that buffer is filled, it will churn/hash that >> block. The adding a byte to the buffer is all low-level fast code in >> C, while the churning would use significantly more CPU cycles... hard >> to phantom that you would see much slower performance when you pass a >> single byte at the time in C... >> >> >> On Tue, Jul 12, 2016 at 8:07 AM, lvh <_...@lvh.io> wrote: >>> Hi, >>> >>>> On Jul 11, 2016, at 10:42 PM, Frank Siebenlist >>>> <frank.siebenl...@gmail.com> wrote: >>> >>> <snipsnip> >>> >>>> I understand that there may be a few more object-creations and casts >>>> involved in the looping, but 500 times slower… that was un unexpected >>>> surprise. >>> >>> As expected. You both get massively increased C call overhead and the worst >>> case because you don’t get to hit a block until every 512/8 == 64 updates. >>> Alas, openssl speed doesn’t distinguish between the same message sizes but >>> in different chunk sizes, but you can at least clearly see the performance >>> multiplier for larger messages. >>> >>> lvh >>> _______________________________________________ >>> Cryptography-dev mailing list >>> Cryptography-dev@python.org >>> https://mail.python.org/mailman/listinfo/cryptography-dev > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev@python.org > https://mail.python.org/mailman/listinfo/cryptography-dev
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Cryptography-dev mailing list Cryptography-dev@python.org https://mail.python.org/mailman/listinfo/cryptography-dev
