My code logic works at whatever resolution you need it to. those could be 10,000/th of a second if you need.
Small and Big is really meant in terms of "Permanent" and Short Term. EveryTime you Commit the writes you would start over on a new "small Data Store" From: [email protected] [mailto:[email protected]] On Behalf Of Andrius A Sent: Tuesday, December 06, 2011 5:01 PM To: [email protected] Subject: Re: [google-appengine] DataStore Tiering Was Dealine Exceeded Errors Thanks Brandon, but in your example if I had 50 users bidding in 1 second it would take another 50 sec for the final result to appear. That's not going to work unfortunately because there will be another 20 users bidding on next second. or I am not catching something? What do you mean by small and big datastore? How do you implement it? and I can't keep data in backends instance memory or memcache, because backend can be restarted or memcache evicted anytime. I found that fastest way to have this working is having 50 sharded counters in MS, so I can really process the bids within a second! On 7 December 2011 00:40, Brandon Wirtz <[email protected]> wrote: You should be doing something like 12:00:00 AuctionID451 Brandons Dirty Socks $84 Andrius 12:00:01 AuctionID451 Brandons Dirty Socks $86 Greg 12:00:02 AuctionID451 Brandons Dirty Socks $94 Brian 12:00:02 AuctionID453 Adult Mermaid Costume $134 Brandon 12:00:03 AuctionID451 Brandons Dirty Socks $104 Andrius Every so often when this table gets too big, or too old you would commit it to the larger datastore When you ask for the data you would ask the Big data store for the price and then check if there was a more recent price in the small data store. MemCache and local instance memory could be used to check the Small data store without a Read to the data store as one level of cache, but could also contain the list of auctions that it contains the most recent data on so that you wouldn't have to hit the main data store. Over all this should be faster, cost less, and more reliable than MS. From: [email protected] [mailto:[email protected]] On Behalf Of Andrius A Sent: Tuesday, December 06, 2011 4:26 PM To: [email protected] Subject: Re: [google-appengine] DataStore Tiering Was Dealine Exceeded Errors thanks Brandon I am thinking about it all the time. I wonder if you could help me with this particular case. I am selling an item in the auction, which has a time to expire and the price changes with each bid received. Lets say I have 50 users bidding in the same time. How could I use "Cache" here to calculate the item price on demand within a second using HRD? With MS I can do it easily without using non-reliable memcache. Or would you say that GAE is not suitable for that? Thank you, Andrius On 7 December 2011 00:10, Brandon Wirtz <[email protected]> wrote: I posted this logic once before but people only listen when they are ranting. You can only change an entity once per second. You can however maintain a data store "Cache" that will save you money, writes, and prevent you from ever seeing write limits. By maintaining a Small DataStore of "Writes To commit" things you want to write to data store permanently can be put in to a data store which you increment the task on. When you do a Read from the datastore you check that the writes to commit don't supersede the data in the main data store. This will save you on indexing operations, and reduce your write throttling and if done correctly can even move most of your reads to Memory rather than DataStore. This will increase your number of reads. Which will increase your Costs slightly over MS, but likely is better than having MS down or MS Read OnlyTime. From: [email protected] [mailto:[email protected]] On Behalf Of Andrius A Sent: Tuesday, December 06, 2011 3:44 PM To: [email protected] Subject: Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then In docs it says that HRD is more costly "The High Replication Datastore costs more due to the additional replication (see billing page for pricing details). Due to the higher cost, we recommend High Replication Datastore primarily for those developers building critical App Engine applications that require the highest availability". It also says "queries on a single guestbook to be strongly consistent, but also limits changes to the guestbook to 1 write per second (the supported limit for entity groups). Therefore, writing to a single entity group per guestbook is not ideal when high usage is expected. If your app is likely to encounter heavy write usage, consider using another means. For example, you can put recent posts in memcache with an expiration, and then display a mix of recent posts from memcache and posts retrieved from the datastore." How could we trust by putting data to memcache if we know it can be evicted any time? To use HRD is not viable for applications which need strong consistency for high rate of puts. At the moment MS is perfect what it does but bloody thing keeps dying.. I can't believe there is no way to improve MS and you are switching to HRD. In long term HRD can cause more problems for applications parts were it wasn't aticipating to receive higher rate of inserts, and having a limit of 1 write per second is a disastrous. On 6 December 2011 23:29, Gregory D'alesandre <[email protected]> wrote: Hi Kenneth, We are planning to add blobstore migration at some point but it won't likely be until the middle of Q1 at the earliest. Greg On Tue, Dec 6, 2011 at 3:27 PM, Kenneth <[email protected]> wrote: Hi Greg, Can you give us some indication if the migration tool is ever going to include blobstore migration? That's the only thing holding me back. I know I can do it myself but I'd rather not. I see now that it has gone ga, have you stuck a fork in it and called it done? Thanks -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected] <mailto:google-appengine%[email protected]> . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected] <mailto:google-appengine%[email protected]> . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected] <mailto:google-appengine%[email protected]> . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected] <mailto:google-appengine%[email protected]> . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
