Have you looked into using pull queues combined with sharded counters? If you lease 1,000 tasks at a time you can probably condense writes considerably. There are several different sharding strategies you could take, even just one master set of shards each of which covers all items. 100 shards * 1,000 tasks should give you a minimum write rate limit of 100,000/sec, magnified by the amount of overlap (ie if those 1,000 tasks only updated 100 counts, your minimum write rate limit is 1 million/sec).
I haven't tried this. It presumes you can write 1k entities to a single entity group in a single transaction - maybe a googler can chime in. It would probably be expensive to store. Jeff On Mon, Sep 26, 2011 at 12:41 AM, Peter Dev <dev133...@gmail.com> wrote: > We are developing an application, where users can vote for many > objects. > (for example, voting the best music video of the week) > > - This means, we have millions of possible objects to vote for, and > millions of users > > To our best knowledge, after taking in consideration different > options, the best (or the only) voting system is: Memcache+Bulk DB > write in DB > - If number of objects in Memcache will achieve a specified limit (for > example 3000) then write in DB. > > The writing speed into DB is about 100/sec. This also means, if we > would set the above mentioned 3000 objects, the writing would last for > 30 sec... > > The problem; during save in DB, voting must be blocked. In other > words, if from many millions of objects, there are 3000 achieved voted > objects, we need to write it into DB and it can happen too many times, > and blocking the whole voting mechanism. > > If we do not block the voting whilst writing in DB, the result could > be wrong number of votes from cache (see Workflow: step 3). > > Workflow: > > 1. vote received > 2. find object in memcache > 3. if not found in memcache get from DB and put it in > 4. increment the number of votes of the object in memcache > 5. check object number in memcache > 6. if necessary, save in DB and empty memcache > ... > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
