On Friday, March 1, 2013, Tyler Romeo wrote: > On Fri, Mar 1, 2013 at 11:46 AM, Asher Feldman > <[email protected]<javascript:;> > >wrote: > > > don't think a custom daemon would actually be needed. > > > > http://redis.io/topics/pubsub > > > > > > > > While I was at flickr, we implemented a pubsub based system to push > > notifications of all photo uploads and metadata changes to google using > > redis as the backend. The rate of uploads and edits at flickr in 2010 was > > orders of magnitude greater than the rate of edits across all wmf > projects. > > Publishing to a redis pubsub channel does grow in cost as the number of > > subscribers increases but I don't see a problem at our scale. If so, > there > > are ways around it. > > > > We are planning on migrating the wiki job queues from mysql to redis in > the > > next few weeks, so it's already a growing piece of our infrastructure. I > > think the bulk of the work here would actually just be in building > > a frontend webservice that supports websockets / long polling, provides a > > clean api, and preferably uses oauth or some form of registration to ward > > off abuse and allow us to limit the growth of subscribers as we scale. > > > > Interesting. Didn't know Redis had something like this. I'm not too > knowledgeable about Redis, but would clients be able to subscribe directly > to Redis queues? Or would that be a security issue (like allowing people to > access Memcached would be) and we would have to implement our own > notification service anyway?
I think a very light weight proxy that only passes subscribe commands to redis would work. A read only redis slave could be provided but I don't think it includes a way to limit what commands clients can run, including administrative ones. I think we'd want a thin proxy layer in front anyways, to track and if necessary, selectively limit access. It could be very simple though. > 0mq? RabbitMQ? Seem to fit the use case pretty well / closely. > > > Hmm, I've always only thought of RabbitMQ as a messaging service between > linked applications, but I guess it could be used as a type of push > notification service as well. > *--* > *Tyler Romeo* > Stevens Institute of Technology, Class of 2015 > Major in Computer Science > www.whizkidztech.com | [email protected] <javascript:;> > _______________________________________________ > Wikitech-l mailing list > [email protected] <javascript:;> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
