On Tue, Mar 09, 2010 at 06:39:01PM -0600, Shawn Walker wrote: > > - Does it save us time to incrementally update the sha_1 hash by > > calling update() instead of generating it in one pass when the file > > is written, or is this to cope with the case where the file is > > written incrementally? > > So this means that roughly 4.4 million+ iterations are done for the > /dev catalog currently, so it's much faster to do the sha-1 calc in > a single pass in filesystem-sized chunks.
Ok, that's what I suspected, but I guess I asked the question in a weird way. > At some point in the future, I Hope to have enough time to either > write a new serialiser (in C), or create a patch for simplejson that > makes it suck less. > > In particular, I could reduce memory usage and possibly reduce > serialisation time further if I could have the iterative encoder > write to an fd wrapped in a stdio stream with fdopen() instead of > yielding each value or creating a giant python list object with > everything inside. If you re-write in C, it might even be possible to eschew stdio altogether and issue block-sized writes as you go. I guess that's implementation minutia, though. > However, I felt that this changeset was a big win for a relatively > small amount of work. Completely agree. Thanks for doing this. -j _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
