Thanks Brandon, but in your example if I had 50 users bidding in 1 second it would take another 50 sec for the final result to appear. That's not going to work unfortunately because there will be another 20 users bidding on next second. or I am not catching something?
What do you mean by small and big datastore? How do you implement it? and I can't keep data in backends instance memory or memcache, because backend can be restarted or memcache evicted anytime. I found that fastest way to have this working is having 50 sharded counters in MS, so I can really process the bids within a second! On 7 December 2011 00:40, Brandon Wirtz <[email protected]> wrote: > You should be doing something like **** > > ** ** > > 12:00:00 AuctionID451 Brandons Dirty Socks > $84 Andrius**** > > 12:00:01 AuctionID451 Brandons Dirty Socks > $86 Greg**** > > 12:00:02 AuctionID451 Brandons Dirty Socks > $94 Brian**** > > 12:00:02 AuctionID453 Adult Mermaid Costume > $134 Brandon**** > > 12:00:03 AuctionID451 Brandons Dirty Socks > $104 Andrius**** > > ** ** > > Every so often when this table gets too big, or too old you would commit > it to the larger datastore**** > > ** ** > > When you ask for the data you would ask the Big data store for the price > and then check if there was a more recent price in the small data store.** > ** > > MemCache and local instance memory could be used to check the Small data > store without a Read to the data store as one level of cache, but could > also contain the list of auctions that it contains the most recent data on > so that you wouldn’t have to hit the main data store.**** > > ** ** > > Over all this should be faster, cost less, and more reliable than MS.**** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Andrius A > *Sent:* Tuesday, December 06, 2011 4:26 PM > *To:* [email protected] > *Subject:* Re: [google-appengine] DataStore Tiering Was Dealine Exceeded > Errors**** > > ** ** > > thanks Brandon I am thinking about it all the time. I wonder if you could > help me with this particular case. I am selling an item in the auction, > which has a time to expire and the price changes with each bid received. > Lets say I have 50 users bidding in the same time.**** > > How could I use "Cache" here to calculate the item price on demand within > a second using HRD? With MS I can do it easily without using non-reliable > memcache. Or would you say that GAE is not suitable for that?**** > > ** ** > > Thank you,**** > > Andrius**** > > ** ** > > On 7 December 2011 00:10, Brandon Wirtz <[email protected]> wrote:**** > > I posted this logic once before but people only listen when they are > ranting.**** > > **** > > You can only change an entity once per second. You can however maintain a > data store “Cache” that will save you money, writes, and prevent you from > ever seeing write limits.**** > > **** > > By maintaining a Small DataStore of “Writes To commit” things you want to > write to data store permanently can be put in to a data store which you > increment the task on.**** > > **** > > When you do a Read from the datastore you check that the writes to commit > don’t supersede the data in the main data store.**** > > **** > > This will save you on indexing operations, and reduce your write > throttling and if done correctly can even move most of your reads to Memory > rather than DataStore.**** > > **** > > This will increase your number of reads. Which will increase your Costs > slightly over MS, but likely is better than having MS down or MS Read > OnlyTime.**** > > **** > > **** > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Andrius A > *Sent:* Tuesday, December 06, 2011 3:44 PM > *To:* [email protected] > *Subject:* Re: [google-appengine] Re: DeadlineExceeded errors haunting > every now and then**** > > **** > > In docs it says that HRD is more costly "*The High Replication Datastore > costs more due to the additional replication (see billing page for pricing > details). Due to the higher cost, we recommend High Replication Datastore > primarily for those developers building critical App Engine applications > that require the highest availability*". **** > > It also says "*queries on a single guestbook to be strongly consistent, > but also limits changes to the guestbook to 1 write per second (the > supported limit for entity groups). Therefore, writing to a single entity > group per guestbook is not ideal when high usage is expected. If your app > is likely to encounter heavy write usage, consider using another means. For > example, you can put recent posts in memcache with an expiration, and then > display a mix of recent posts from memcache and posts retrieved from the > datastore.*" **** > > How could we trust by putting data to memcache if we know it can be > evicted any time? To use HRD is not viable for applications which need > strong consistency for high rate of puts. At the moment MS is perfect what > it does but bloody thing keeps dying.. I can't believe there is no way to > improve MS and you are switching to HRD. In long term HRD can cause more > problems for applications parts were it wasn't aticipating to receive > higher rate of inserts, and having a limit of 1 write per second is > a disastrous.**** > > On 6 December 2011 23:29, Gregory D'alesandre <[email protected]> wrote:*** > * > > Hi Kenneth, **** > > **** > > We are planning to add blobstore migration at some point but it won't > likely be until the middle of Q1 at the earliest.**** > > **** > > Greg**** > > **** > > On Tue, Dec 6, 2011 at 3:27 PM, Kenneth <[email protected]> > wrote:**** > > Hi Greg, > > Can you give us some indication if the migration tool is ever going to > include blobstore migration? That's the only thing holding me back. I know > I can do it myself but I'd rather not. I see now that it has gone ga, have > you stuck a fork in it and called it done? > > Thanks**** > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group.**** > > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.**** > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.**** > > **** > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.**** > > **** > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.**** > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.**** > > ** ** > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.**** > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
