Yeah, just sent it there, sorry
On 13 August 2018 at 19:03, akuster808 <[email protected]> wrote: > > > On 08/13/2018 10:20 AM, Ross Burton wrote: >> Opening a file in binary mode and iterating it seems like the simple solution >> but will still break on newlines, which for binary files isn't really useful >> as >> the size of the chunks could be huge or tiny. >> >> Instead, let's be a bit more clever: we'll be MD5ing lots of files, but we >> don't >> want to fill up memory: use mmap() to open the file and read the file in 8k >> blocks. >> >> Signed-off-by: Ross Burton <[email protected]> > > shouldn't this go to the bitbake mailing list ? >> --- >> bitbake/lib/bb/utils.py | 13 +++++++++---- >> 1 file changed, 9 insertions(+), 4 deletions(-) >> >> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py >> index 9903183213b..b20cdabcf01 100644 >> --- a/bitbake/lib/bb/utils.py >> +++ b/bitbake/lib/bb/utils.py >> @@ -524,12 +524,17 @@ def md5_file(filename): >> """ >> Return the hex string representation of the MD5 checksum of filename. >> """ >> - import hashlib >> - m = hashlib.md5() >> + import hashlib, mmap >> >> with open(filename, "rb") as f: >> - for line in f: >> - m.update(line) >> + m = hashlib.md5() >> + try: >> + with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm: >> + for chunk in iter(lambda: mm.read(8192), b''): >> + m.update(chunk) >> + except ValueError: >> + # You can't mmap() an empty file so silence this exception >> + pass >> return m.hexdigest() >> >> def sha256_file(filename): > -- _______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
