It's still producing *wild* variance when I click those buttons. By a factor of 2.
If you want an accurate benchmark: 1) You need more iterations. 5 is not enough. At a minimum I would say 10. If this is too long for a single button click, you can halve the # of entities fetched at once - it shouldn't matter for what we are trying to measure. 2) You cannot use an arithmetic mean to measure average. GAE performance is too erratic and outliers overwhelm the mean with noise. Use a median value, this will be more stable and will more accurately represent the true performance of the system. Basically, you will know the measurement is proper when multiple executions produce fairly consistent results. Right now you're just measuring noise. Jeff On Thu, Jun 9, 2011 at 2:34 PM, Yasuo Higa <[email protected]> wrote: > I modified the samples to use Iterator for a fair benchmark: > > public Iterator<Entity> getBarsUsingLL() { > AsyncDatastoreService ds = > DatastoreServiceFactory.getAsyncDatastoreService(); > Query q = new Query("Bar"); > PreparedQuery pq = ds.prepare(q); > return pq.asIterator(FetchOptions.Builder.withDefaults().limit( > Integer.MAX_VALUE)); > } > > public Iterator<Bar> getBarsUsingSlim3() { > return Datastore.query(Bar.class).asIterator(); > } > > public Iterator<BarObjectify> getBarsUsingObjectify() { > Objectify ofy = ObjectifyService.begin(); > return ofy.query(BarObjectify.class).iterator(); > } > > @SuppressWarnings("unchecked") > public List<BarJDO> getBarsUsingJDO() { > List<BarJDO> list = null; > PersistenceManager pm = PMF.get().getPersistenceManager(); > try { > javax.jdo.Query q = pm.newQuery(BarJDO.class); > list = (List<BarJDO>) q.execute(); > list = (List<BarJDO>) pm.detachCopyAll(list); > } finally { > pm.close(); > } > return list; > } > > private static final int COUNT = 5; > > @Override > public Navigation run() throws Exception { > long start = System.currentTimeMillis(); > for (int i = 0; i < COUNT; i++) { > for (Iterator<Entity> ite = service.getBarsUsingLL(); > ite.hasNext();) { > Entity e = ite.next(); > e.getKey(); > e.getProperty("sortValue"); > } > } > sessionScope("getLL", (System.currentTimeMillis() - start) / COUNT); > return redirect(basePath); > } > > @Override > public Navigation run() throws Exception { > long start = System.currentTimeMillis(); > for (int i = 0; i < COUNT; i++) { > for (Iterator<Bar> ite = service.getBarsUsingSlim3(); > ite.hasNext();) { > Bar bar = ite.next(); > bar.getKey(); > bar.getSortValue(); > } > } > sessionScope("getSlim3", (System.currentTimeMillis() - start) / COUNT); > return redirect(basePath); > } > > @Override > public Navigation run() throws Exception { > long start = System.currentTimeMillis(); > for (int i = 0; i < COUNT; i++) { > for (Iterator<BarObjectify> ite = > service.getBarsUsingObjectify(); ite > .hasNext();) { > BarObjectify bar = ite.next(); > bar.getKey(); > bar.getSortValue(); > } > } > sessionScope("getObjectify", (System.currentTimeMillis() - start) > / COUNT); > return redirect(basePath); > } > > @Override > public Navigation run() throws Exception { > long start = System.currentTimeMillis(); > for (int i = 0; i < COUNT; i++) { > for (BarJDO bar : service.getBarsUsingJDO()) { > bar.getKey(); > bar.getSortValue(); > } > } > sessionScope("getJDO", (System.currentTimeMillis() - start) / COUNT); > return redirect(basePath); > } > > http://slim3demo.appspot.com/performance/ > > Slim3 is slightly faster than frameworks that use reflection, > but the difference is small. > > Yasuo Higa > > On Fri, Jun 10, 2011 at 2:29 AM, Alfred Fuller > <[email protected]> wrote: >> If you are going to just iterate through a list with out doing any work, >> fetching everything up front is always going to be faster. However, we >> expect that you are going to be doing something with the entities you fetch. >> The lazy list tries to hid the cost of fetching the entities by doing it >> asynchronously while you are 'processing' the previous batch of entities. In >> my experiments (in Python) where work was actually being done (namely >> thread.sleep(X)) it gave a 10-15% speed up (depending on how much work you >> are doing, as the fetch time is a constant value). >> I would not have expected the difference to be this significant in Java >> though. We never know the # of results ahead of time so the difference >> couldn't be related to growing the List organically (as it always grows >> organically). >> One thing that should be noted, a wrapper that proactively converts entities >> (instead of doing the conversion on demand) will negate any benefit from >> async prefetching. Also a realistic benchmark should tak into account that >> some amount of work will be done on the entities fetched. >> >> On Thu, Jun 9, 2011 at 9:51 AM, Alfred Fuller >> <[email protected]> wrote: >>> >>> It does uses a lazy list to do asynchronous prefetching: >>> >>> http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/LazyList.java >>> >>> On Tue, Jun 7, 2011 at 3:19 AM, Anders <[email protected]> wrote: >>>> >>>> I doubt that the difference can be that large. The performance test code >>>> uses the low-level PreparedQuery#asList call. The question is if the list >>>> (List<Entity>) contains entities loaded with data or if the list returned >>>> has a lazy loading implementation so that the actual data from the the >>>> datastore only gets loaded when entity properties are accessed. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine" group. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msg/google-appengine/-/WVhBRXBqMWFMZ3dK. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
