it's an interesting thing to try indeed. It's hard to tell exactly how much, because we're pretty at the beginning of this project and the load will depend a lot on "who" (as in authorities/companies) that are ready to share with us some (all?) real-time sensed data.
I do believe we could have easily scenarios where we would have hundreds of data points/sec. and the fan-out could vary between dozens up to thousands. We have two parts in the work we do. The "public" part, where interactions are mainly web-based to read/write raw data, and streaming processed feeds of the raw data (e.g. real-time traffic information at a sub-second scale for example), and the backend, where we have to use a more efficient messaging system to be able to process (& scale) such computation-heavy filtering, e.g. we're looking at zeromq for that. The challenge is how to bind more general/efficient pub/sub systems with the web-equivalent, and be able to efficiently bind the both worlds, and combine a Web-based front-end app, with an efficient zeromq/amqp backend. Seems like RestMS was going in this direction to an extent, but didn't hear much about them. just a couple of thoughts I'm launching here. Could share as things are built up, but feedback/ideas/links on that more than welcome!! On 05-Nov-2010, at 4:43 PM, Brett Slatkin wrote: > Sounds like a great use-case. What's the total updates/second you > think you'll need for incoming events, and what's your average fan-out > factor (eg, 1 incoming event goes to 10 subscribers)? > > On Wed, Nov 3, 2010 at 2:20 PM, Vlad Trifa <[email protected]> wrote: >> Yep, didn't look much at the server-side code, but the example is clearly >> promising. I would be very interested how this could be implemented on a >> massively scalable server back-end optimized for many concurrent RTW >> interactions (seen stuff for tornado which is kinda promising), jetty also >> has websockets support so could also be an interesting starting point. >> >> My main goal is to be able to stream LOTS of REAL-TIME (sensor) data (for >> example the real-time location of all the buses in boston and track/monitor >> them as soon as they emit their position to detect accidents for example), >> fast (sub-second publisher to subscriber notification latency), and >> especially various "streams" to LOTS of people (so not just 1"feed/stream" >> pushed to thousands of people, but dozens of feeds to thousands of people). >> I will be working on this the next few weeks and will share my thoughts >> here, but any ideas on what you guys have tried or what platform should be a >> good start to build this is more than appreciated (I'm most familiar with >> java/php, so maybe something in this direction will be ever more exciting). >> >> Using websockets for displaying thousands of locations per second on a map >> (from all 16'000 taxis in singapore) is impressively fast with a simple >> hacked and totally non-optimized prototype php script as server, so I can't >> wait to see this brought a the next level, which is a much more optimized >> server to scale this with much more data! >> >> ideas to where to look for this totally welcome (just like any questions >> comments :) >> >> Cheers! >> >> Vlad >> >> >> >> On 03-Nov-2010, at 11:52 AM, Brett Slatkin wrote: >> >>> Yes, Julien's example is great. I hope that we can being to >>> standardize this approach in the coming months as WebSockets gets more >>> widely deployed. >>> >>> On Wed, Nov 3, 2010 at 4:33 AM, Julien Genestoux >>> <[email protected]> wrote: >>>> Vlad, >>>> We have a small example with this : http://julien51.github.com/socket-sub/ >>>> and this http://julien51.github.com/socket-sub/maps >>>> It's all on github. >>>> Cheers, >>>> >>>> >>>> On Tue, Nov 2, 2010 at 10:59 PM, Vlad Trifa <[email protected]> wrote: >>>>> >>>>> hi all, >>>>> >>>>> just saw the new "push" api for google with pubsubhubbub support, great >>>>> thing! >>>>> >>>>> I was wondering if you guys have tried any implementation based on html5 >>>>> websockets? I've started playing with this recently and I do believe >>>>> there's >>>>> definitely much that could come of a pubsubhubbub implemented on top of >>>>> websockets and/or classic web hooks, especially as it seems pretty >>>>> efficient, so I just want to make sure if any one of you has been playing >>>>> with this, just so that we can go on in this direction. >>>>> >>>>> thanks a bunch for thoughts! >>>>> >>>>> Cheers >>>>> >>>>> Vlad >>>>> >>>>> >>>> >>>> >> >>
