> But, ZMQ isn't storing it anywhere... that's just a messaging protocol, > isn't it? >
Yes, and in fact, it may be even enough if you have sufficiently smart consumer on other end. For example, if the goal is to keep only statistics, you can use running max, average, histogram, etc. In this case only very little memory is needed. I'm not saying that persistence is not needed in this case - it would be nice to have it too - but high throughput and low latency are much more desirable. I was able to get 2M/sec without transactions, and 909K/sec with > transactions (so it's durable), on my laptop, using Caché... (from > InterSystems... I used to consult for them) > They do have a free single user database engine, Globals, that you might > be able to use... I don't recall what's available with that version... > I suppose you got 2M/sec on some server with pretty high resources - I cannot imagine 2M network operations on my local machine. Anyway, I think I will start with Redis, which is more accessible from Julia both in terms of programming and openness. >> Indices also have their cost. And, most important, they are not really >> needed here. In Kafka, for example, new records are simply added to the end >> of the queue, and pointer to the end is moved further. In our tests it >> gives about 50K/sec, which is much higher than for RBMSs mentioned above, >> and unlike most key-value storages provides easy and fast way to read all >> messages down the stream. >> > > That seems *really* slow... but, if you are just adding records to the end > of a queue, and you have your key simply be the counter, why would that be > slow? > 50K/sec is just our result from quick test, Kakfa's own test show several times higher results. Anyway, the point was that Kafka out of the box is *much* faster than things like MySQL, so there's really no point to mess up with this kind of databases.
