On Tuesday, May 19, 2015 at 7:23:13 PM UTC-4, Andrei Zh wrote: > > > >> But, ZMQ isn't storing it anywhere... that's just a messaging protocol, >> isn't it? >> > > Yes, and in fact, it may be even enough if you have sufficiently smart > consumer on other end. For example, if the goal is to keep only statistics, > you can use running max, average, histogram, etc. In this case only very > little memory is needed. > > I'm not saying that persistence is not needed in this case - it would be > nice to have it too - but high throughput and low latency are much more > desirable. > > > I was able to get 2M/sec without transactions, and 909K/sec with >> transactions (so it's durable), on my laptop, using Caché... (from >> InterSystems... I used to consult for them) >> They do have a free single user database engine, Globals, that you might >> be able to use... I don't recall what's available with that version... >> > > I suppose you got 2M/sec on some server with pretty high resources - I > cannot imagine 2M network operations on my local machine. Anyway, I think I > will start with Redis, which is more accessible from Julia both in terms of > programming and openness. >
No, that's on my 1 year old MacBook Pro... with just the default database settings: Platform Info: System: Darwin (x86_64-apple-darwin14.4.0) CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz Starting Control Process Automatically configuring buffers Allocated 374MB shared memory: 256MB global buffers, 35MB routine buffers My point was mainly that a good KVS should be able to achieve those kinds of speeds, even on a laptop... unfortunately, Caché is not open source software, so you'd probably need some other solution... Redis may be a good bet... I'd be interested in what performance numbers you get with Julia/Redis!
