On Tuesday, May 19, 2015 at 4:05:29 PM UTC-4, Andrei Zh wrote: > > @Scott, thanks for your comment, but I think we are talking about > different scenarios and different speed levels :) > > ??? Not on any real RDBMS like MySQL, PostgreSQL... > > > If you want reported message to be durable, you need either to replicate > it to another server, or write it to disk. In-memory log doesn't help here, > since transaction is still flushed on commit. You can tune up your DB, for > example, switching off committing after each transaction (InnoDB in MySQL > has this possibility, as far as I remember), but even in this case I doubt > MySQL or PostgreSQL will be able to beat 10K/sec. With ZMQ I've got > 300K/sec from the very first time on my dev machine. >
But, ZMQ isn't storing it anywhere... that's just a messaging protocol, isn't it? I was able to get 2M/sec without transactions, and 909K/sec with transactions (so it's durable), on my laptop, using Caché... (from InterSystems... I used to consult for them) They do have a free single user database engine, Globals, that you might be able to use... I don't recall what's available with that version... -Scott > > >> Why would being a key-value store make getting results slow? >> I think really all depends on the indices that you build... >> > > Indices also have their cost. And, most important, they are not really > needed here. In Kafka, for example, new records are simply added to the end > of the queue, and pointer to the end is moved further. In our tests it > gives about 50K/sec, which is much higher than for RBMSs mentioned above, > and unlike most key-value storages provides easy and fast way to read all > messages down the stream. > That seems *really* slow... but, if you are just adding records to the end of a queue, and you have your key simply be the counter, why would that be slow?
