I don't have specific answers, but I do have some thoughts.
First, the externally parallel has no overhead related to coordination. I
would expect you to get the result you have of full core utilization and
nearly perfect scaling.
As soon as you have a common resource, you will have the related overhead
due to management of the concurrent actions. The actual cost will be
determined by how much contention you have for that resource.
Have you looked at your program using the Go Profiler? I found the output
quite useful in determining where my program was spending its time.
Another thing to check is CPU load. Are you saturating the CPU with 6
threads. If not, you are likely waiting somewhere.
There are good resources online about using the Profiler to debug
performance issues. I apologize for not including them here.
On Saturday, October 29, 2016 at 3:42:02 PM UTC-4, Florian Weimer wrote:
>
> I'm trying to parse OpenPGP key server dumps. Throughput is not too
> bad so far, but I'd like to speed things up by introducing
> parallelism.
>
> The dumps are split into several files (each about 40 MB large, with a
> few thousand keys). The toy version I have so far does not need any
> coordination between the processing of individual files
>
> This works quite well when running the sequential version in parallel,
> using GNU parallel. With GOMAXPROCS=1, serial execution time is
> reduced from 141 seconds to 31 seconds. This is roughly what I would
> expect from six cores with hyperthreading and use of GNU parallel
> (which has some overhead of its own in this scenario).
>
> However, the version with built-in parallelism runs in 55 seconds, so
> only half as fast as the GNU parallel approach. I would have expect
> it to fare better compared to that. The parallel version is my second
> one which has decent performance. I also tried a variant which has a
> per-thread buffer which is occasionally written to standard output,
> synchronized by a sync.Mutex. (It would likely have benefited from a
> sync.Mutex.TryLock() function, delaying the buffer flush if there was
> contention.) This was still significantly slower than the external
> parallelization, but perhaps a little faster than parallel version
> attached below. I think both parallel approaches produce more garbage
> than the sequential version, but likely not as much to explain the
> speed difference compared to the sequential version with external
> parallelization.
>
> “perf top” suggests that a lot of time is spent in GC-related Go
> functions (which is expected, considering what the program does). But
> vmstat shows a large number of context switches, which I find
> surprising. It is much higher than the number of context switching
> during a GNU parallel run.
>
> Most tests were run with the current master branch.
>
> Is there anything else I could try to make the internally parallized
> version as fast the externally parallized one?
>
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
