I wrote a multi-threaded duplicate file checker using md5, here is the 
source: https://github.com/hbfs/dupe_check/blob/master/dupe_check.go

Benched two variants on the same machine, on the same set of files (~1.7GB 
folder with  ~600 files, each avg 3MB), multiple times, purging disk cache 
in between each run.

With this code:

    hash := md5.New()

    if _, err := io.Copy(hash, file); err != nil {

    var md5sum [md5.Size]byte
    copy(md5sum[:], hash.Sum(nil)[:16])

*// 3.35s user 105.20s system 213% cpu 50.848 total, memory usage is ~ 30MB*

With this code:

  data, err := ioutil.ReadFile(path)
  if err != nil {

  md5sum := md5.Sum(data)

* // 3.10s user 31.75s system 104% cpu 33.210 total, memory usage is ~ 

The memory usage make sense, but why is the streaming version ~3x slower 
than the read the entire file into memory version? This trade off doesn't 
make sense to me since the file is being read from disk in both situations 
which should be the limiting factor. Then the md5sum is being computed. 

In the streaming version, there is an extra copy from []byte to [16]byte 
but that should be negligible.

My only theory I can think of is context switching

streaming version:
disk -> processor
processor waiting for disk read so it switch to read another file, sleeping 
the thread.

entire file:
disk -> memory -> processor
file is in memory so not as much context switching.

What do you think? Thanks!

You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to