BTW, I've measured throughput of Redis on my machine, and it resulted in only 30K/sec, which is 10 times slower than ZMQ. Possibly, some hybrid solution of several alternative backends will work.
On Saturday, May 23, 2015 at 2:04:02 AM UTC+3, Andrei Zh wrote: > > > It still helps a lot, because you can have many reporting programs, each >> talking to different processes on the server, and those processes are able >> to get the transactions done very quickly, with multiple write daemons and >> journaling daemons doing the actual I/O from the shared buffer pool. (and >> yes, this has been used precisely for collecting metrics, on a very large >> scale, across all of Germany in one case, IIRC) >> > > I've got curious about this use case, are you permitted to share more > details about it? Were they technical or business metrics? What volume of > data passed through the system and what were parameters or the server? > > In fact, my initial question in this thread was purposely very broad, > because I was looking for any related projects. Now I see 2 different kinds > of metric collection systems. One is for reporting and monitoring purposes, > e.g. measuring operation run time, memory usage, number of simultaneously > connected users, etc. Normally, in such systems metrics are not stored on > collector side for a long time, but instead are aggregated and sent to > something like graphite almost immediately. Also it's ok to lose some part > of this information or delete old metrics. Another kind is for collecting > and further analysis of business metrics. And in this case we need reliable > storage first of all. > > My primary goal in this little project is to create system for first kind > of metrics, but if there's interest in second kind, I'll be glad to spend > some time on making something for broader range of users. >
