I would like to broadcast one message to a 'channel' and have many subscribers get updated in real-time using the App Engine Channel Service. How can I do this?
The Channel Service<https://developers.google.com/appengine/docs/python/channel/functions> send_message(client_id, message) is only intended to be used to send *one* message to *one* client at a time. My current implementation uses memcache to store a key which is my channel_id and list of values which are Channel Service client_ids. When I want to broadcast to a channel_id, I get the list of client_ids from memcache and call send_message(client_id, message) for each of them. The implementation is currently written in Python and to optimise I do the following: - Use marshal instead of pickle to store the list of client_ids in memcache. - Use a frontend request (not a Task Queue) to call send_message(client_id, message) because Task Queues often seem to have large delays between being called and execution. - Call each send_message(client_id, message) on a separate thread. This works quite well when broadcasting to 10 - 20 client_ids and I don't mind too much that the memcache mapping from channel_id to client_ids could be evicted at any point (so long as the memcache service does not go down completely). However the multiple calls to send_message(client_id, message) start to introduce too much latency for more client_ids. I don't mind average latency up to 1.5 seconds. How can I scale to 100 - 500 client_ids listening to one channel_id? My initial thought was to use mapreduce map to fan-out but that is way too heavy and therefore expensive, plus it uses Task Queues so is prone to latency spikes when the tasks don't get executed in a timely manner. My next thought was: Have an entity type for each channel_id. class Channel[channel_id](ndb.Model): pass Use client_id as the key for each client that is subscribed to Channel[channel_id]. Use the special __scatter__ property<https://code.google.com/p/appengine-mapreduce/wiki/ScatterPropertyImplementation>of each entity to shard groups of client_ids into separate memcache entities. (I realise for my scale __scatter__ is unlikely to be populated.) Have one memcache entity containing all the __scatter__ ids. Create a publish(channel_id, message) function that fans out URLFetch to frontend instances for each __scatter__ value. Each frontend request will in turn look up the memcache entity for their __scatter__ id and call send_message(client_id, message) for each client_id. If everything worked smoothly with no memcache evictions, the datastore would not need to be touched most of the time. Would the Go runtime be best for implementing this most efficiently? Alternatively does anyone know of a good push service that has a decent free quota and only charges what is used beyond that. I can't find one after looking for months. I see PubNub's free quota is too small (20 peak connections) and I am loathed to pay $14 per month for going over the quota by only a small amount probably 10 times per month. Or does anyone know of some code that does what I want already out in the public domain? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
