Re: GAE, DataView and performance
On 9/15/2011 9:38 AM, manuelbarzi wrote: > may you execute the query db in IDataProvider.iterator, instead of > populateItem. see IDataProvider javadoc. Wow...that was easy! I'm so glad I asked. You guys have thought of everything. I love Wicket! Chris -- - Chris Merrill | Web Performance, Inc. ch...@webperformance.com| http://webperformance.com 919-433-1762| 919-845-7601 Web Performance: Website Load Testing Software & Services - - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
Re: GAE, DataView and performance
may you execute the query db in IDataProvider.iterator, instead of populateItem. see IDataProvider javadoc. . On Thu, Sep 15, 2011 at 3:16 PM, Chris Merrill wrote: > I'm using Objectify for datastore access in GAE, though the problem would > be the same if I was using the datastore API directly. > > For optimal performance, Google recommends doing batch requests to the > datastore whenever possible - they say making the requests in parallel > is much faster than serial. They also recommend retrieving keys instead > of the entire entity whenever you may not need each entity right away - > such as for a paginated list. I've seen some posts indicating that the > performance improvement can be a factory of 10x - e.g. serially fetching > 20 items will be 20x a single fetch, but fetching them in parallel > might only be 2x a single fetch. > > I'm having difficulty seeing how I would follow those recommendations when > using a DataView with PagingNavigator. My initial query for all the entities > in the list should be for the keys, since only a fraction of the entities will > be displayed at a time. However, since populateItem() is called serially for > each list item, I cannot batch the requests that resolve the keys to their > entities. For best performance, I need to get a callback when a group of items > will be rendered so that I can fetch them all at once - and then call > populateItem() with the resulting items one at a time. > > I'm guessing I'll need to dig into DataView and extend it, or perhaps > roll my own implementation? > > > As an example, here is my current, non-optimal, implementation: > > List> keys = db.getMyEntityKeys(); > DataView> list = new DataView<>("entity_list", new > ListDataProvider>(keys) > { > protected void populateItem(final Item> item) > { > Key key = item.getModel().getObject(); > MyEntity entity = db.getMyEntity(key); > add components for the entity > } > } > list.setItemsPerPage(20); > add(list); > add(new PagingNavigator("paginator", list); > > > The optimized version would perhaps look something like the example below, > which resolves all the keys to their entities in one method and then the > populateItem() expects the entity, rather than the key: > > List> keys = db.getMyEntityKeys(); > DataView> list = new DataView<>("entity_list", new > ListDataProvider>(keys) > { > protected List getPageOfEntities(List> keys) > { > return db.getEntitiesForKeys(keys); > } > protected void populateItem(final Item item) > { > MyEntity entity = item.getModel().getObject(); > add components for the entity > } > } > list.setItemsPerPage(20); > add(list); > add(new PagingNavigator("paginator", list); > > > Anyone have any suggestions how I can do this without essentially > re-implementing > DataView? I probably should have looked further into the DataView code before > posting...but the Wicket community seems to have already solved every other > problem > I've come across, so I though I'd ask here first :> > > TIA! > Chris > > > -- > - > Chris Merrill | Web Performance, Inc. > ch...@webperformance.com | http://webperformance.com > 919-433-1762 | 919-845-7601 > > Web Performance: Website Load Testing Software & Services > - > > - > To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org > For additional commands, e-mail: users-h...@wicket.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
GAE, DataView and performance
I'm using Objectify for datastore access in GAE, though the problem would be the same if I was using the datastore API directly. For optimal performance, Google recommends doing batch requests to the datastore whenever possible - they say making the requests in parallel is much faster than serial. They also recommend retrieving keys instead of the entire entity whenever you may not need each entity right away - such as for a paginated list. I've seen some posts indicating that the performance improvement can be a factory of 10x - e.g. serially fetching 20 items will be 20x a single fetch, but fetching them in parallel might only be 2x a single fetch. I'm having difficulty seeing how I would follow those recommendations when using a DataView with PagingNavigator. My initial query for all the entities in the list should be for the keys, since only a fraction of the entities will be displayed at a time. However, since populateItem() is called serially for each list item, I cannot batch the requests that resolve the keys to their entities. For best performance, I need to get a callback when a group of items will be rendered so that I can fetch them all at once - and then call populateItem() with the resulting items one at a time. I'm guessing I'll need to dig into DataView and extend it, or perhaps roll my own implementation? As an example, here is my current, non-optimal, implementation: List> keys = db.getMyEntityKeys(); DataView> list = new DataView<>("entity_list", new ListDataProvider>(keys) { protected void populateItem(final Item> item) { Key key = item.getModel().getObject(); MyEntity entity = db.getMyEntity(key); add components for the entity } } list.setItemsPerPage(20); add(list); add(new PagingNavigator("paginator", list); The optimized version would perhaps look something like the example below, which resolves all the keys to their entities in one method and then the populateItem() expects the entity, rather than the key: List> keys = db.getMyEntityKeys(); DataView> list = new DataView<>("entity_list", new ListDataProvider>(keys) { protected List getPageOfEntities(List> keys) { return db.getEntitiesForKeys(keys); } protected void populateItem(final Item item) { MyEntity entity = item.getModel().getObject(); add components for the entity } } list.setItemsPerPage(20); add(list); add(new PagingNavigator("paginator", list); Anyone have any suggestions how I can do this without essentially re-implementing DataView? I probably should have looked further into the DataView code before posting...but the Wicket community seems to have already solved every other problem I've come across, so I though I'd ask here first :> TIA! Chris -- - Chris Merrill | Web Performance, Inc. ch...@webperformance.com| http://webperformance.com 919-433-1762| 919-845-7601 Web Performance: Website Load Testing Software & Services - - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org