Since there is some confusion on the thread, I would like to clarify that I am using EventStreams in the labs version. There is no way too use Kafka outside production and no way to replay historical events (which was what made this so much better in production).
(code for this is mostly in *https://github.com/jdlrobson/wikitrender/blob/master/index.js <https://github.com/jdlrobson/wikitrender/blob/master/index.js> *for those interested) On Tue, 12 Dec 2017 at 13:33 Andrew Otto <[email protected]> wrote: > > This is a little inferior to the production version as it is unable to > use > production kafka and if it has any outages it will lose data. > > EventStreams isn’t as good as using Kafka, but an outage shouldn’t be a > reason to lose data. Store the Last-Event-ID > <https://wikitech.wikimedia.org/wiki/EventStreams#Format> that > EventStreams > gives you, and you can use it when the service starts back up to start from > where you left off. > > > and maybe, as others have mentioned, a good reason to get production > Kafka events flowing into Cloud VPS backed projects. > > Def not opposed to a Kafka cluster in Cloud mirroring from Prod. :) > > BTW, this is a wee relevant: > https://wikitech.wikimedia.org/wiki/User:Ottomata/Stream_Data_Platform > > This is a draft! I’m shopping this around as a program for next FY. We > will see! > > > > > > > On Tue, Dec 12, 2017 at 4:16 PM, Gergo Tisza <[email protected]> wrote: > > > On Tue, Dec 12, 2017 at 12:12 PM, Jon Robson <[email protected]> > wrote: > > > > > This is a little inferior to the production version as it is unable to > > use > > > production kafka and if it has any outages it will lose data. > > > > > > > Hopefully that gets fixed soon, Cloud VPS / Toolforge is the foundation > for > > out volunteer tool developer community who really shouldn't be treated as > > second-class citizens. > > > > Other than that, moving to the Cloud is not a bad thing for an > experimental > > project IMO. It makes it easier to experiment with minimal risk, and it > > makes it easy to add co-collaborators to your project without having to > get > > prod access for them. > > > > I'm hoping to get this onto IFTTT <https://ifttt.com/wikipedia> with > help > > > from Stephen Laporte in my volunteer time, as I think this feature is a > > > pretty powerful one which has failed to find its use case in the wiki > > > world. As Kaldari points out it's incredibly good at detecting edit > wars > > > and I personally have learned a lot about what our editors see as > > important > > > and notable in the world (our editors really seem to like wrestling). I > > > think there are ample and exciting things people could build on top of > > this > > > api. > > > > > > > Yeah a "give me push notifications about ongoing edit wars" tool for > admins > > sounds really cool. Although you'd probably want to look at revert trends > > (or both edit and revert trends) for that. > > _______________________________________________ > > Wikitech-l mailing list > > [email protected] > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > _______________________________________________ > Wikitech-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
