Hi, with https://github.com/vimeo/statsdaemon, our statsd replacement in go (1.3) we are processing between 40~70k messages a second (10k timers/s, 500 gauges/s and the rest is counters), and: * cpu (Xeon E5-2697 v2 @ 2.70GHz) is typically 70~90% used with the occasional 100% (increasing GOMAXPROCS makes it go up to 130% but doesn't change the rest of this story, statsdaemon is not very optimized for multi-core) * once in a while the udp buffer runs full and the kernel drops incoming messages (presumably when cpu maxes out) (the udf buffer is as big as the kernel lets me)
I've been profiling and been able to make some small improvements, though interestingly, pprof cpu profile tells me net.readFromUDP is the biggest cpu consumer. perhaps i could make more improvements and debug and rearchitect some things but before I go there I want to familiarize myself more with heka. After all heka looks like a fantastic tool and i want to use it. (i would need to patch in metrics2.0 support but that's another story) So I plan to do some experimenting with heka, but meanwhile I'ld like to hear from you. 1) does anyone have some empirical numbers on heka-as-statsd performance? in particular, when does it start dropping udp packets? what cpu % do you see (in particular can you spot spikes), and how many msg/s is it doing? 2) have you also noticed reading messages from the udp connection going slowly/maxing out cpu? i see heka calls read instead of readFromUDP like I do. I should look into that.. 3) does heka leverage multi-core properly for the statsd workload (receiving the packets, and doing the aggregations)? any other tips on replacing statsd with heka? is there a good tool to catch udp traffic and send it to both stats[daemon] and heka (so i can try the same production load on both)? there's xtables-addons kernel modules to do this, but i think our version of centos is too old. I quickly wrote up a go program to try to do this in userspace but i was hitting the same perf problem just calling net.UDPConn.readFromUDP in a loop. thanks, Dieter _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
