I'm not sure a debate about the relative merits of RDBMS vs other kinds of stores is going to be productive in this list.. Its a question that hasn't been settled in the lifetime of computers.....
Andrei building a fast persistent queue using ZMQ is something that I've been interested in, but there are only so many hours a day. You mention the number of lines of code in your current trial.. but you are building a concurrent store, so not sure it can get much simpler than that. ZMQ will give you quite a bit of leverage in the networking layer, but you'll still need to build a user level api over it, and handle storage. One option, if you care to go down that route, is to use Redis as a store. There are a couple of julia Redis drivers around. Regards - Avik On Tuesday, 19 May 2015 17:23:23 UTC+1, Scott Jones wrote: > > > > On Tuesday, May 19, 2015 at 11:39:24 AM UTC-4, Andrei Zh wrote: >> >> Not quite, I'm more interested in systems for collecting metrics (think >> of reporting server or application health, for example) rather than storage >> with random reads and writes. It is possible to use databases for this, but >> it will be pretty inefficient. For example: >> >> 1. If we tale traditional RDBMS like MySQL or PostgreSQL, each insert >> into DB will result in writing to disk (slow!), triggering indexes and >> other maintenance activities. So these systems simply cannot handle the >> load and will shortly become a bottleneck. >> > > ??? Not on any real RDBMS like MySQL, PostgreSQL... > You can also look at systems like InterSystem's Caché, which can be used > as a fast key-value store, but also has SQL and object capabilities... > It is fast enough to be used to store huge amounts of data from the > European Space Agency... > It has been used for storing metrics... > > 2. Some NoSQL databases like Couchbase or Aerospike may give pretty good >> performance, but since they are key-value stores, scanning them for results >> may be terribly slow. >> > > Why would being a key-value store make getting results slow? > I think really all depends on the indices that you build... > > >> 3. Persistent queues like Apache Kafka provide very fast sequential >> writes and high-performance reads. But given their terrible API and >> permanent breaking changes even in minor versions I don't even want to try >> to connect to it from Julia. Well, not for the purpose of this little >> project, at least. >> 4. In-memory queues with many producers and single consumer / collector >> are pretty much close. So here we come to fast RabbitMQ and even faster >> ZMQ. Still, they require some effort to build metric collection on top of >> them. >> 5. There is also StatsD which is close in spirit, but hard to integrate >> with Julia. It also requires too many unrelated dependencies (e.g. >> node.js), especially compared to integrated ZMQ. >> >> >> I don't expect ready-to-use library or even well defined approach, but >> maybe someone in Julia community has already encountered problem of metric >> collection? >> >
