Thinking about it maybe what you want is storing a monotonic timestamp as
the key and having a seperate field that says "last timestamp that server
has successfully received". That way you can send a tailable readStream({
greater then that last timestamp }) on every connection to the server.You probably want to append a uuid to your keys so that they stay sortable by time but are unique and don't cause clashes when they arrive at the single read server. The only question is whether or not Tim's adhoc in memory store may be better suited. It should also be noted that the levelidb caveat is Chrome/FF/IE10 only support, which probably isn't an issue in your case. On Wed, Dec 19, 2012 at 12:10 AM, Jake Verbaten <[email protected]> wrote: > Another solution might be leveldb (using the [levelup][2] driver). > > /cc dominic, he should have an opinion on whether leveldb is suited. > > Each client (browser or node.js) that is a writer has a leveldb (or > [levelidb][1] for browsers) > > You can then trivially take a tailable live stream of all the data in a > client (using [level-livefeed][3] or [level-live-stream][4]) and send it > down any stream (like [engine.io-stream][5] or TCP) and pipe it into > [levelup.writeStream][6] > > levelup has an example of this ["copy" functionality][7] > > 0) Installable with npm > > very npm friendly > > 1) Very high bandwidth, my app can use as much as you give it > > leveldb is in process. so it handles as much bandwidth as node does. > > 2) Multi-writers, including browsers and other servers > > if multiple people write to the same key the last one wins. > > 3) One reader on a single server > > Nothing enforces this. But the way it would be set up above is all writers > send all data to the single reader server > > 5) Ordered data delivery would be nice but I could handle order > problems in my code > > if your transport is ordered then data is ordered. However leveldb doesnt > store a timestamp anywhere. You can trivially store them in the data or keys > > 6) Reliable delivery would be nice not not critical. End use of data > is data-mining to be read by humans and I can show the human an > indication data is missing. It would make my app look bad though. > > Delivery is as reliable as your transport. > > Advantages are that it works with any streaming transport. and that > leveldb is very lightweight and minimalist. > > Disadvantages is how do you want to handle getting the data from the > client to server. Do you want to flush the entire client db to the server > which means there is lots of wasted bandwidth and excessive setting of keys > that already exist. Or do you want to just want to stream only changes to > the client db (not implemented in libraries but its a 3 line addition). > > The problem with streaming changes is that if your transport goes down > changes get buffer in memory and if the client disconnects before the > transport comes up those are lost. > A second issues is how do you handle the last few messages in flight > before the client disconnects, what if they get lost. > > These could be handled by replication but then you have to implement > replication for your needs without flushing the entire db. > > [1]: https://github.com/Raynos/levelidb > [2]: https://github.com/rvagg/node-levelup > [3]: https://github.com/Raynos/level-livefeed > [4]: https://github.com/dominictarr/level-live-stream > [5]: https://github.com/Raynos/engine.io-stream > [6]: https://github.com/rvagg/node-levelup#writeStream > [7]: > https://github.com/rvagg/node-levelup#pipes-and-node-stream-compatibility > > > On Tue, Dec 18, 2012 at 1:57 PM, Mark Hahn <[email protected]> wrote: > >> @Ciprian >> >> Thx. I will definitely check out zeromq. >> >> On Tue, Dec 18, 2012 at 1:54 PM, Mark Hahn <[email protected]> wrote: >> > You analysis is spot-on. Thx. >> > >> >> If your browser clients are new enough browsers, >> > >> > Yes. Only developers will use the tool so chrome will be preferred. >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
