Given your requirements, I think the most important question here is *volume*. How many clients/events per day to you expect? Unless you expect huge amount of events right away, I would suggest that you start with a minimum viable product (including pub/sub), since it wouldn't be too hard to later replace whatever you choose to use with Kafka, once such a need arises. It's just that Kafka, being open source, requires a bit more engineering effort than some other products. But if you're willing to tackle a bug or two and don't mind sparing a bit more engineering time, go ahead and use Kafka regardless of the load.
Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext <http://sematext.com/> | Contact <http://sematext.com/about/contact.html> On Mon, Mar 21, 2016 at 6:25 PM, Ben Stopford <b...@confluent.io> wrote: > It sounds like a fairly typical pub-sub use case where you’d likely be > choosing Kafka because of its scalable data retention and built in fault > tolerance. As such it’s a reasonable choice. > > On 21 Mar 2016, at 17:07, Mark van Leeuwen <m...@vl.id.au> wrote: > > > > Hi Sandesh, > > > > Thanks for the suggestions. I've looked at them now :-) > > > > The core problem that needs to be solved with my app is keeping a full > replayable history of changes, transmitting latest state to web apps when > they start, then keeping them in sync with latest state as changes are made > by all current clients, preferably without polling. That's why keeping > track of offsets with each client seemed the way to go. > > > > Not sure how stream processing engines help with that - but happy to be > advised otherwise. > > > > Cheers. > > > > On 22/03/16 02:35, Sandesh Hegde wrote: > >> Hello Mark, > >> > >> Have you looked at one of the streaming engines like Apache Apex, Flink? > >> > >> Thanks > >> > >> On Mon, Mar 21, 2016 at 7:56 AM Gerard Klijs <gerard.kl...@dizzit.com> > >> wrote: > >> > >>> Hi Mark, > >>> > >>> I don't think it would be a good solution with the latencies to and > from > >>> the server your running from in mind. This is less of a problem is > your app > >>> is only mainly used in one region. > >>> > >>> I recently went to a Firebase event, and it seems a lot more fitting. > It > >>> also allows the user to see it's own changes real-time, and provides > >>> several authentication options, and has servers world-wide. > >>> > >>> On Mon, Mar 21, 2016 at 7:53 AM Mark van Leeuwen <m...@vl.id.au> > wrote: > >>> > >>>> Hi, > >>>> > >>>> I'm soon to begin design and dev of a collaborative web app where > >>>> changes made by one user should appear to other users in near real > time. > >>>> > >>>> I'm new to Kafka, but having read a bit about Kafka streams I'm > >>>> wondering if it would be a good solution. Change events produced by > one > >>>> user would be published to multiple consumer clients over a websocket, > >>>> each having their own offset. > >>>> > >>>> Would this be viable? > >>>> > >>>> Are there any considerations I should be aware of? > >>>> > >>>> Thanks, > >>>> Mark > >>>> > >>>> > > > >
