I'm referring to Task Queues: http://code.google.com/appengine/docs/python/taskqueue/
On Sat, May 29, 2010 at 9:41 AM, Philip Tucker <[email protected]> wrote: > One more followup on this, when you say "you just pay your cost at > write time, possibly > asynchronously" are you referring to cron jobs? As far as I can tell > there's no other support for concurrent processing. > > On May 4, 3:28 am, "Ikai L (Google)" <[email protected]> wrote: > > Query performance is generally a function of how many objects have to be > > returned in the simple case, which seems to be what you are doing here. > How > > many PlayerGameStates exist per User? > > > > Queries work like this: > > > > - iterate through the index to find all PlayerGameStates with the > property > > given to find the keys of all PlayerGameStates matching this property > > - run the query for all objects > > > > Are PlayerGameStates child entities of a User? I have to double check, > but I > > believe this could reduce the App Engine datastore's ability to > parallelize > > the queries-by-key, though the effect of this should be negligible if you > > are only fetching 5 objects. > > > > The reason I mentioned "denormalizing" is because the concept of > > normalization doesn't make as much sense with App Engine's datastore. We > > don't provide referential integrity, for instance, and transactions exist > > around entity groups, not globally. You don't pay a penalty for storing > data > > that is sparse - some User entities can be huge, some can be tiny. You > > should optimize for reads. Doing multiple writes and the risk of data > > getting out of sync can suck, but as long as it is recoverable you should > be > > okay. Contrast the performance of the query above with storing > > PlayerGameStates inside User. You'd do this: > > > > - Get user by key > > > > That's it! This is really fast, so you don't have to go through doing an > > unnecessary query - you just pay your cost at write time, possibly > > asynchronously. > > > > > > > > On Tue, May 4, 2010 at 12:14 PM, Philip Tucker <[email protected]> > wrote: > > > I don't think duplicating the data won't work because then I'll be > > > doing 2 writes for every turn of the game, and I need that to be fast. > > > I don't mind if fetching a user's game list is a little bit slow, but > > > 5-20 seconds is too much. I can denormalize all the data such that > > > Game 2 fields for each entry in PlayerGameState if that helps, but I'd > > > still like to know why it's so slow now. > > > > > I do have appstat installed. For one particular misbehaving request, > > > the query fetches 5 PlayerGameState records. It iterates through the > > > first 4 taking ~20 ms each, then I see a RunQuery taking 8 seconds. > > > > > I don't see an easy way to export appstat data, but I can include more > > > details if necessary. > > > > > Here's the query code in question. > > > > > Set<Key> myUserCache = Sets.newHashSet(); > > > Query query = pm.newQuery(PlayerGameState.class); > > > query.declareParameters( > > > "com.google.appengine.api.datastore.Key userKeyParam"); > > > query.setFilter("userKey == userKeyParam"); > > > List<PlayerGameState> playerStates = > > > (List<PlayerGameState>) query.execute(playerKey); > > > List<Game> games = Lists.newArrayList(); > > > for (PlayerGameState playerState : playerStates) { > > > Game game = playerState.getGame(); > > > myUserCache.add(game.getPlayerState().getUserKey()); > > > myUserCache.add(game.getOpponentState().getUserKey()); > > > games.add(game); > > > } > > > > > On May 3, 1:16 am, "Ikai L (Google)" <[email protected]> wrote: > > > > Indexes don't work the same way they do in relational databases. If > you > > > can > > > > fetch a value with a query, that likely means an index exists on that > > > > property. In a relational database, when an index is not present, the > > > > database will do a full table scan. > > > > > > A fetch by Key is always fast. Have you considered storing multiple > games > > > > inside a single Game instance? Or simply storing the data you need > both > > > in > > > > the Game instances? If you want to cut down on request latency, do as > > > many > > > > by key gets as possible. Don't try to approach the modeling from a > > > > normalization perspective. How many entities are you retrieving? And > have > > > > you tried looking at what's happening via AppStats yet ( > > >http://code.google.com/appengine/docs/java/tools/appstats.html)? > > > > > > On Fri, Apr 30, 2010 at 1:54 AM, Philip Tucker <[email protected]> > > > wrote: > > > > > I've got a Game object that includes two PlayerGameState objects. > Both > > > > > are persistence-capable. My query fetches all PlayerGameState > objects > > > > > for a particular user, then I get the associated game and other > > > > > player. The initial fetch is always fast, and the entire operation > is > > > > > generally lass than a second. But periodically, maybe 5 of the > time, > > > > > it takes much longer, from 5-20 seconds. It's not fetching the > > > > > PlayerGameState objects that's slow, it's accessing > > > > > PlagerGameState.game and Game.playerStates. > > > > > > > I shouldn't need to add an index on the keys, should I? I'd like to > > > > > fetch the Game objects directly but as far as I can tell we can't > > > > > query on a joined table; ie, I can't do something like "select from > > > > > Game, PlayerGameState join on PlayerGameState.game where > > > > > PlayerGameState.userKey == userKeyParam". > > > > > > > @PersistenceCapable(identityType = IdentityType.APPLICATION) > > > > > public class Game { > > > > > @PrimaryKey > > > > > @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) > > > > > private Key key; > > > > > > > @Persistent(mappedBy = "game") > > > > > @Element(dependent = "true") > > > > > private List<PlayerGameState> playerStates; > > > > > > > ... > > > > > } > > > > > > > @PersistenceCapable(identityType = IdentityType.APPLICATION) > > > > > public class PlayerGameState { > > > > > @SuppressWarnings("unused") > > > > > @PrimaryKey > > > > > @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) > > > > > private Key key; > > > > > > > @Persistent > > > > > private Key userKey; > > > > > > > @Persistent > > > > > private Game game; > > > > > > > ... > > > > > } > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Google App Engine for Java" group. > > > > > To post to this group, send email to > > > > > [email protected]. > > > > > To unsubscribe from this group, send email to > > > > > [email protected]<google-appengine-java%[email protected]> > <google-appengine-java%[email protected]<google-appengine-java%[email protected]> > ><google-appengine-java%2B > > > [email protected]> > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > > -- > > > > Ikai Lan > > > > Developer Relations, Google App Engine > > > > Twitter:http://twitter.com/ikai > > > > Delicious:http://delicious.com/ikailan > > > > > > ---------------- > > > > Google App Engine links: > > > > Blog:http://googleappengine.blogspot.com > > > > Twitter:http://twitter.com/app_engine > > > > Reddit:http://www.reddit.com/r/appengine > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine for Java" group. > > > > To post to this group, send email to > > > [email protected]. > > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine-java%[email protected]> > <google-appengine-java%[email protected]<google-appengine-java%[email protected]> > > > > > . > > > > For more options, visit this group athttp:// > > > groups.google.com/group/google-appengine-java?hl=en. > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to > > > [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine-java%[email protected]> > <google-appengine-java%[email protected]<google-appengine-java%[email protected]> > > > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > -- > > Ikai Lan > > Developer Relations, Google App Engine > > Twitter:http://twitter.com/ikai > > Delicious:http://delicious.com/ikailan > > > > ---------------- > > Google App Engine links: > > Blog:http://googleappengine.blogspot.com > > Twitter:http://twitter.com/app_engine > > Reddit:http://www.reddit.com/r/appengine > > > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > > To post to this group, send email to > [email protected]. > > To unsubscribe from this group, send email to > [email protected]<google-appengine-java%[email protected]> > . > > For more options, visit this group athttp:// > groups.google.com/group/google-appengine-java?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine-java%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
