Re: GAE, DataView and performance
On 9/15/2011 9:38 AM, manuelbarzi wrote: may you execute the query db in IDataProvider.iterator, instead of populateItem. see IDataProvider javadoc. Wow...that was easy! I'm so glad I asked. You guys have thought of everything. I love Wicket! Chris -- - Chris Merrill | Web Performance, Inc. ch...@webperformance.com| http://webperformance.com 919-433-1762| 919-845-7601 Web Performance: Website Load Testing Software Services - - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
GAE, DataView and performance
I'm using Objectify for datastore access in GAE, though the problem would
be the same if I was using the datastore API directly.
For optimal performance, Google recommends doing batch requests to the
datastore whenever possible - they say making the requests in parallel
is much faster than serial. They also recommend retrieving keys instead
of the entire entity whenever you may not need each entity right away -
such as for a paginated list. I've seen some posts indicating that the
performance improvement can be a factory of 10x - e.g. serially fetching
20 items will be 20x a single fetch, but fetching them in parallel
might only be 2x a single fetch.
I'm having difficulty seeing how I would follow those recommendations when
using a DataView with PagingNavigator. My initial query for all the entities
in the list should be for the keys, since only a fraction of the entities will
be displayed at a time. However, since populateItem() is called serially for
each list item, I cannot batch the requests that resolve the keys to their
entities. For best performance, I need to get a callback when a group of items
will be rendered so that I can fetch them all at once - and then call
populateItem() with the resulting items one at a time.
I'm guessing I'll need to dig into DataView and extend it, or perhaps
roll my own implementation?
As an example, here is my current, non-optimal, implementation:
ListKeyMyEntity keys = db.getMyEntityKeys();
DataViewKeyMyEntity list = new DataViewKeyMyEntity(entity_list, new
ListDataProviderKeyMyEntity(keys)
{
protected void populateItem(final ItemKeyMyEntity item)
{
KeyMyEntity key = item.getModel().getObject();
MyEntity entity = db.getMyEntity(key);
add components for the entity
}
}
list.setItemsPerPage(20);
add(list);
add(new PagingNavigator(paginator, list);
The optimized version would perhaps look something like the example below,
which resolves all the keys to their entities in one method and then the
populateItem() expects the entity, rather than the key:
ListKeyMyEntity keys = db.getMyEntityKeys();
DataViewKeyMyEntity list = new DataViewKeyMyEntity(entity_list, new
ListDataProviderKeyMyEntity(keys)
{
protected ListMyEntity getPageOfEntities(ListKeyMyEntity keys)
{
return db.getEntitiesForKeys(keys);
}
protected void populateItem(final ItemMyEntity item)
{
MyEntity entity = item.getModel().getObject();
add components for the entity
}
}
list.setItemsPerPage(20);
add(list);
add(new PagingNavigator(paginator, list);
Anyone have any suggestions how I can do this without essentially
re-implementing
DataView? I probably should have looked further into the DataView code before
posting...but the Wicket community seems to have already solved every other
problem
I've come across, so I though I'd ask here first :
TIA!
Chris
--
-
Chris Merrill | Web Performance, Inc.
ch...@webperformance.com| http://webperformance.com
919-433-1762| 919-845-7601
Web Performance: Website Load Testing Software Services
-
-
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org
Re: GAE, DataView and performance
may you execute the query db in IDataProvider.iterator, instead of
populateItem. see IDataProvider javadoc.
.
On Thu, Sep 15, 2011 at 3:16 PM, Chris Merrill ch...@webperformance.com wrote:
I'm using Objectify for datastore access in GAE, though the problem would
be the same if I was using the datastore API directly.
For optimal performance, Google recommends doing batch requests to the
datastore whenever possible - they say making the requests in parallel
is much faster than serial. They also recommend retrieving keys instead
of the entire entity whenever you may not need each entity right away -
such as for a paginated list. I've seen some posts indicating that the
performance improvement can be a factory of 10x - e.g. serially fetching
20 items will be 20x a single fetch, but fetching them in parallel
might only be 2x a single fetch.
I'm having difficulty seeing how I would follow those recommendations when
using a DataView with PagingNavigator. My initial query for all the entities
in the list should be for the keys, since only a fraction of the entities will
be displayed at a time. However, since populateItem() is called serially for
each list item, I cannot batch the requests that resolve the keys to their
entities. For best performance, I need to get a callback when a group of items
will be rendered so that I can fetch them all at once - and then call
populateItem() with the resulting items one at a time.
I'm guessing I'll need to dig into DataView and extend it, or perhaps
roll my own implementation?
As an example, here is my current, non-optimal, implementation:
ListKeyMyEntity keys = db.getMyEntityKeys();
DataViewKeyMyEntity list = new DataViewKeyMyEntity(entity_list, new
ListDataProviderKeyMyEntity(keys)
{
protected void populateItem(final ItemKeyMyEntity item)
{
KeyMyEntity key = item.getModel().getObject();
MyEntity entity = db.getMyEntity(key);
add components for the entity
}
}
list.setItemsPerPage(20);
add(list);
add(new PagingNavigator(paginator, list);
The optimized version would perhaps look something like the example below,
which resolves all the keys to their entities in one method and then the
populateItem() expects the entity, rather than the key:
ListKeyMyEntity keys = db.getMyEntityKeys();
DataViewKeyMyEntity list = new DataViewKeyMyEntity(entity_list, new
ListDataProviderKeyMyEntity(keys)
{
protected ListMyEntity getPageOfEntities(ListKeyMyEntity keys)
{
return db.getEntitiesForKeys(keys);
}
protected void populateItem(final ItemMyEntity item)
{
MyEntity entity = item.getModel().getObject();
add components for the entity
}
}
list.setItemsPerPage(20);
add(list);
add(new PagingNavigator(paginator, list);
Anyone have any suggestions how I can do this without essentially
re-implementing
DataView? I probably should have looked further into the DataView code before
posting...but the Wicket community seems to have already solved every other
problem
I've come across, so I though I'd ask here first :
TIA!
Chris
--
-
Chris Merrill | Web Performance, Inc.
ch...@webperformance.com | http://webperformance.com
919-433-1762 | 919-845-7601
Web Performance: Website Load Testing Software Services
-
-
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org
