@Avik, I forgot about Redis completely, thanks! It's definitely worth trying, especially given that it has special queue-oriented capabilities.
On Tuesday, May 19, 2015 at 7:37:02 PM UTC+3, Avik Sengupta wrote: > > I'm not sure a debate about the relative merits of RDBMS vs other kinds of > stores is going to be productive in this list.. Its a question that hasn't > been settled in the lifetime of computers..... > > Andrei building a fast persistent queue using ZMQ is something that I've > been interested in, but there are only so many hours a day. You mention the > number of lines of code in your current trial.. but you are building a > concurrent store, so not sure it can get much simpler than that. ZMQ will > give you quite a bit of leverage in the networking layer, but you'll still > need to build a user level api over it, and handle storage. > > One option, if you care to go down that route, is to use Redis as a store. > There are a couple of julia Redis drivers around. > > Regards > - > Avik > > On Tuesday, 19 May 2015 17:23:23 UTC+1, Scott Jones wrote: >> >> >> >> On Tuesday, May 19, 2015 at 11:39:24 AM UTC-4, Andrei Zh wrote: >>> >>> Not quite, I'm more interested in systems for collecting metrics (think >>> of reporting server or application health, for example) rather than storage >>> with random reads and writes. It is possible to use databases for this, but >>> it will be pretty inefficient. For example: >>> >>> 1. If we tale traditional RDBMS like MySQL or PostgreSQL, each insert >>> into DB will result in writing to disk (slow!), triggering indexes and >>> other maintenance activities. So these systems simply cannot handle the >>> load and will shortly become a bottleneck. >>> >> >> ??? Not on any real RDBMS like MySQL, PostgreSQL... >> You can also look at systems like InterSystem's Caché, which can be used >> as a fast key-value store, but also has SQL and object capabilities... >> It is fast enough to be used to store huge amounts of data from the >> European Space Agency... >> It has been used for storing metrics... >> >> 2. Some NoSQL databases like Couchbase or Aerospike may give pretty good >>> performance, but since they are key-value stores, scanning them for results >>> may be terribly slow. >>> >> >> Why would being a key-value store make getting results slow? >> I think really all depends on the indices that you build... >> >> >>> 3. Persistent queues like Apache Kafka provide very fast sequential >>> writes and high-performance reads. But given their terrible API and >>> permanent breaking changes even in minor versions I don't even want to try >>> to connect to it from Julia. Well, not for the purpose of this little >>> project, at least. >>> 4. In-memory queues with many producers and single consumer / collector >>> are pretty much close. So here we come to fast RabbitMQ and even faster >>> ZMQ. Still, they require some effort to build metric collection on top of >>> them. >>> 5. There is also StatsD which is close in spirit, but hard to integrate >>> with Julia. It also requires too many unrelated dependencies (e.g. >>> node.js), especially compared to integrated ZMQ. >>> >>> >>> I don't expect ready-to-use library or even well defined approach, but >>> maybe someone in Julia community has already encountered problem of metric >>> collection? >>> >>
