Hi Tonny, I've done quite a bit with the approaches used in that talk. Some of the entities / markers are kept around and some are not. If you are simply counting, which is what it sounds like, your overhead will be minimal. The permanent markers will basicaly only contain the key plus two integers, and neither will be indexed. All db ops on these entities are fetches by key.
In the general case you will have a marker for each 'vote', plus another marker for each aggregation that vote is in, plus the aggregation entities themselves. Depending on the specifics of your application, you might be able to optimize to eliminate or reduce the number of vote markers needed. If your count is really a "one-way" operation (users can not un-express interest) you could potentially cleanup old markers. If you want to get some idea of how this might work, check out slagg (http://bitbucket.org/thebobert/slagg). You could have a working prototype up pretty fast to do some tests with. I've got a number of improvements i hope to get out soon, incuding better docs and some more examples (like counting votes). Let me know if you have any questions about it. Robert On Wednesday, November 24, 2010, Tonny <[email protected]> wrote: > Hi There > > I'm considering using the design "Transactional sequences" described > in the excellent "Building high-throughput data pipelines" talk by > Brett Slatkin (http://www.google.com/events/io/2010/sessions/high- > throughput-data-pipelines-appengine.html). > > The design uses markers for both the source of the update and the > targets which should eventually be in sync. > I'm just wondering if anyone has some experience with the storage > requirements of these markers. > > As far as I can tell, the are kept around - forever. So, if have 100k > users where each on an average is involved with about 1500 products > (let's say I was counting how many users have shown interest in a > product) that would leave me with 100k * 1500 + 100k ~= 150M markers, > and growing since new users and products will add markers, since > deleting old users and old products won't clear out the markers > (unless I add some referential integrity). > > Any thought anyone? > > Cheers > /Tonny > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- ------ Robert Kluin Ezox Systems, LLC -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
