2 Main parts of the application. Font End Display StarCraft 2 Player 1v1 Ladder information from all Regions. Search options: Order by (6 categories) filter by (Region) filter by (Race). <-- This is the cause of my exploding index. Display 100 players per page. I have implemented some cache use for this part, so it is not much of an issue currently.
Back End (This is the big CPU Hog) Load Player information from battle.net http://us.battle.net/sc2/en/profile/388538/1/HuK/ladder/1794#current-rank This is done by a task that gets the data, parses it and saves any player changes. There is 100 players per page, but usually less than 30 need to be updated each scan. Each save/update takes almost 1 second. .5 seconds with a version that has NO INDEXES. Currently the speed is not an issue, but the cost of CPU time for Datastore operations is. Ideally, I would like to support the almost 1 million players on battle.net like sc2Ranks.com But that would not be practical cost wise using the Datastore. For now, I'm only supporting the top 10K players. Don't get me wrong. I really love GAE. I have created a few small apps that run fine. However, this app is looking like it won't be practical for GAE due to the Datastore? On Aug 13, 1:24 pm, "Ikai L (Google)" <[email protected]> wrote: > Saving 100 entities in a single request, especially if a TON of indexes will > be created is very expensive and will cause a lot of CPU to be consumed. > There probably is a design that will allow you to accomplish what you need > to do without this limitation. Can you post your use case? > > On Thu, Aug 12, 2010 at 11:02 PM, SC2PlayerFinder <[email protected] > > > > > > > wrote: > > > Used appstat and didn't notice anything out of the ordinary. Still > > thinking that the main issue here is exploding index and I'm not sure > > how to get around that other than to cut search features from the > > site, but that would more or less make the app not very useful. > > > On Aug 12, 10:19 pm, SC2PlayerFinder <[email protected]> > > wrote: > > > Thanks for the response. I wanted to get back to you sooner, but I > > > have been scrambling to optimize the application. I'm making some > > > progress, however, it is still a relative CPU hog due to an exploding > > > index and the frequent updates the application needs to make in order > > > to keep the data current. 71000 CPU to save 100 records seems rather > > > expensive?. > > > > My next step is to run appstat on it and see what is going on. > > > > Current version of the app is running a sc2pf6.appstat.com. The older > > > version is running at sc2pf2.appspot.com. > > > > On Aug 9, 3:51 pm, "Ikai L (Google)" <[email protected]> wrote: > > > > > I've got a few questions for you: > > > > > 1. What is your application ID? > > > > 2. What are you doing in the tasks? > > > > 3. Are you using transactions or entity groups? > > > > 4. Are you writing to the same datastore entities? > > > > > My guess is that you are running into write contention in the > > datastore. > > > > It'd be very helpful to understand what it is you're doing. > > > > > App Engine will scale up the number of instances if you are returning > > web > > > > requests in less than 800ms (thought 400ms is optimal), but if you are > > > > writing to a small number of entities over and over it won't be able to > > > > resolve that bottleneck. > > > > > On Mon, Aug 9, 2010 at 12:28 PM, SC2PlayerFinder > > > > <[email protected]>wrote: > > > > > > Further explanation. I had a large increase in users which caused a > > > > > chain reaction. Tasks that used to take about 10-20 seconds started > > > > > taking longer and using more CPU time. Interestingly, these tasks are > > > > > doing the exact same thing, but under a higher load, they take longer > > > > > and use more CPU time. I can understand the process taking longer > > > > > under load, but why would it use a significant amount more of CPU > > > > > time? I will admit that my code is not the most optimal, but then > > > > > again isn't the main focus of GAE built in scalability and speed? In > > > > > short, I needed on the fly scalability and speed with an influx of > > new > > > > > users and I don't feel like I got that with GAE? Worse, there was no > > > > > way for me to contact someone who could help me out. > > > > > > I'm left with the feeling that GAE does not do a good job of scaling > > > > > up and it is kind of slow as well. Creating and updating records > > > > > seems painfully expensive, time out due to new instance creation > > > > > happens way to often. Now this is fine for a free app, but I'm going > > > > > way over my daily free quota and it is not cheap. Maybe I should be > > > > > using Phython and not Java? > > > > > > On Aug 6, 10:52 pm, SC2PlayerFinder <[email protected]> > > wrote: > > > > > > Application was running ok yesturday. Still had about 10% > > > > > > HardDeadlineExceededError, but the same task that executed > > yesturday > > > > > > are not failing almost 100% of the time right now. I even cut back > > > > > > the amount of records it is processing from 100 to 10 and it is > > still > > > > > > failing. > > > > > > > How do I get ahold of someone at Google, to see if there is > > something > > > > > > wrong with the server my app is on or something. Why would the same > > > > > > proccess consume more CPU time one day than another? > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > Groups > > > > > "Google App Engine for Java" group. > > > > > To post to this group, send email to > > > > > [email protected]. > > > > > To unsubscribe from this group, send email to > > > > > [email protected]<google-appengine-java%2B > > > > > [email protected]><google-appengine-java%2B > > [email protected]> > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > -- > > > > Ikai Lan > > > > Developer Programs Engineer, Google App Engine > > > > Blog:http://googleappengine.blogspot.com > > > > Twitter:http://twitter.com/app_engine > > > > Reddit:http://www.reddit.com/r/appengine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine-java%2B > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
