Hi Jon, I don't like sharded counter with the datastore and always preferred pull queue or memcache counters for the job. Only issue with memcache counters is that you dont control when the cached key will be flushed so you have to check the counters very often. i.e. frequency depends on how often you are ready to lose data : ex if you can bear to lose the last 1min of data just check+rest the counters in memcache every 1min. For memcache counters i check every 1-5s: i.e. read counter value and decrement by the value you just read. If you use the same key frequently enough it shouldnt disappear so often.
For pull queue i've been using that all along and so far it works great. It's like sharded counters without the datastore writes which is great for temporary data (the counters). Just make sure your pull queue is large enough. Also you may want to use multiple queues if your write frequency is very high. Or switch to memcache counters of your write frequency is super high (much faster than the queue api but risk of losing data). On Jan 2, 8:02 am, jon <[email protected]> wrote: > Yohan your suggestion regarding the use of pull queue for batch writes > is interesting. In the context of keeping counts, do you think this > technique works better than sharded counters? I'm using the latter but > it has what I think are non-trivial drawbacks: > 1. The code has too many moving parts: the sharded counter > implementation, a job that regularly calculates aggregate > 2. One count value requires writing and reading multiple pieces of > data: the shards and the aggregated value. More reads/writes = more > expensive bill. > > What are the pros and cons of using pull queue + batch writes instead > of sharded counters? > > On Jan 1, 3:59 pm, Yohan <[email protected]> wrote: > > > > > > > > > 10,000$ a month in gae that's a lot and then 10,000$ in consulting > > fees with brandon haha > > > anyway back to business: > > 1) are you using python or java > > 2) how many apps are generating that much traffic > > 3) check the admin panel maybe even post a > > screenshot here (or link to it or something) it will already give you > > good clues where you spend the most money on (bandwidth, data write, > > read, instances, urls that have higher traffic) whih should help > > identify where to start. > > 4) are you using third party frameworks like jdo, objectify, etc > > > Another problem will be if every url is optimized, cached etc but you > > just have a lot of hits that are generating small reasonable costs. > > Then you must start thinking about proxying and relaying the content > > from external systems and also batch+delayed processing of your data > > (i.e. instead pf writing to datastore at each request you push it to a > > pull queue that will write in bulk). > > > I have no doubt you already know that are are probably doing it > > already. Also i dont like appstats very hard to track properly and i > > believe it is adding more load on the requests. > > > GAE used to return the cost of each request in the http response > > header maybe it is still there. > > > Another thing easy to forget is that memcache sucks since you have > > zero control on data expiry (you can put in memcache and 2 ms later > > your data will be gone thus hitting your datastore at every request), > > dont hesitate to implement your own in memory caching solution (a > > simple java hashmap could do). > > > Keep us posted. > > > Cheers > > > On Jan 1, 5:40 am, "Brandon Wirtz" <[email protected]> wrote: > > > > I do consulting on this. You are likely in the price range where you > > > could > > > benefit from it. > > > > You should also check the archive for some of the optimizations I have > > > talked about publicly. But if you want to hit me up off list, I can > > > usually > > > give you an estimate of how much reduction can be made just by looking at > > > your Google Analytics, and a few hours of your apps logs. > > > > Most clients see a reduction of about 30% on an 8 Hour consult. During > > > that > > > consult we'd discuss code logic, your team would make the code changes. > > > If > > > my team makes the code changes it will cost a bit more. We have on rare > > > instances seen up to 80% reductions in hosting costs, but 30% is very > > > typical, and even companies that have done a lot of optimizations often > > > are > > > surprised to see 25%. If I can't fine enough to offset my consulting > > > price > > > in 90 days, I won't charge you for the consult. > > > > -Brandon > > > > Brandon Wirtz > > > BlackWaterOps: President / Lead Mercenary > > > > Description:http://www.linkedin.com/img/signature/bg_slate_385x42.jpg > > > > Work: 510-992-6548 > > > Toll Free: 866-400-4536 > > > > IM: [email protected] (Google Talk) > > > Skype: drakegreene > > > > <http://www.blackwaterops.com/> BlackWater Ops > > > > From: [email protected] > > > [mailto:[email protected]] On Behalf Of Andrin von > > > Rechenberg > > > Sent: Saturday, December 31, 2011 7:42 AM > > > To: [email protected] > > > Subject: [google-appengine] Where to start optimizing cost? > > > > Hey there > > > > I'm an absolute GAE-Lover! The only thing that is bothering me is cost. > > > > We spend about $10'000 a month in GAE cost. That's just too much for > > > > the traffic we serve. We are starting to look around for alternatives, but > > > > I'd really love to stay with GAE and just optimize the system. > > > > One of GAE's biggest flaws IMHO is that it is really hard to see where > > > > the costs are coming from. (Yes we have appstats in place, but our > > > > system is just too big to manually sample hundreds of requests). > > > > So here is a feature request that might help us optimize and stay with > > > GAE: > > > > Show cost per URI in the dashboard. That would be incredibly helpful. > > > > Does anyone know of an existing elegant way to figure out where the > > > > cost come from? (I'm looking more for a way to measure rather than > > > > software solutions like "use memcache"). I could do something with > > > > prodeagle.com and estimate the cost per request myself but I'd rather > > > > use an already existing solution... > > > > Cheers, > > > > -Andrin > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]. > > > For more options, visit this group > > > athttp://groups.google.com/group/google-appengine?hl=en. > > > > image003.jpg > > > < 1KViewDownload -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
