Thanks. Made the go code similar to python using CopyBuffer with a block 
size of 65536. 

    buf := make([]byte, 65536)
    if _, err := io.CopyBuffer(hash, file, buf); err != nil {

Didn't make too much of a difference, was slightly faster.

What got it to the same place was running ComputeHash in the same goroutine 
as the Walk function vs its own go routine for each file

+    ComputeHash(path, info, queue, wg)
-    go ComputeHash(path, info, queue, wg)

*2.88s user 0.98s system 23% cpu 16.086 total, memory usage ~ 7MB*

Here's the before and after pprof webs:

BEFORE with 'go ComputeHash(...):


AFTER with 'ComputeHash(...):


Since disk read are SOO much slower, computing the hash for each file in 
its own goroutine caused a huge slowdown.. 

btw this is on a RAID10, with SSD: 

Old code SSD: 3.31s user 17.87s system 244% cpu 8.667 total

New code SDD:* 2.88s user 0.84s system 69% cpu 5.369 total*

Shows you can throw hardware at a problem BUT the old code locks up my 
system momentarily..

On Saturday, October 15, 2016 at 3:27:38 PM UTC-4, Kevin Malachowski wrote:
> Sorry, I meant that calling Write on the hash type might be slower if it's 
> called more often.
> (I'm on mobile right now. When I get back to a keyboard I'll try to come 
> up with an example)

You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
For more options, visit

Reply via email to