Thank you for fixing the benchmark. I am very curious. According to this new benchmark - it's hard to tell without pushing the buttons a lot of times, but there seems to be a trend - Slim3 is somewhat faster than the Low Level API.
Doesn't Slim3 use the Low Level API underneath? How can it possibly be faster? Jeff On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa <[email protected]> wrote: > What I want to provide is a fair and casual benchmark. > > As jeff advised, I modified samples as follows: > for (Entity e : service.getBarListUsingLL()) { > e.getKey(); > e.getProperty("sortValue"); > } > > for (Bar bar : service.getBarListUsingSlim3()) { > bar.getKey(); > bar.getSortValue(); > } > > for (BarObjectify bar : service.getBarListUsingObjectify()) { > bar.getKey(); > bar.getSortValue(); > } > > for (BarJDO bar : service.getBarListUsingJDO()) { > bar.getKey(); > bar.getSortValue(); > } > > LL API is much slower than before. > http://slim3demo.appspot.com/performance/ > > Yasuo Higa > > > On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer <[email protected]> wrote: >> Slim3 may be a nice piece of software, but it has not been >> demonstrated to be faster than anything (including JDO). It might or >> might not be faster - I don't know - but based on the sloppy >> benchmarking, I'm pretty certain that the people making this claim >> don't know either. >> >> There's another ill-conceived performance claim on the Slim3 website: >> "You may worry about the overhead of global transactions. Don't worry. >> It is not very expensive." There are three problems with their little >> performance test: >> >> 1) It only measures wall-clock time, not cost. >> 2) It does not measure what happens under contention. >> 3) The numbers are obviously wrong - they don't even pass a smoke test. >> >> Look at these numbers (from the Slim3 home page): >> >> Entity Groups Local Transaction(millis) Global Transaction(millis) >> 1 67 70 >> 2 450 415 >> 3 213 539 >> 4 1498 981 >> 5 447 781 >> >> Just like the 1ms low-level API query performance in the benchmark >> that spawned this thread, even a casual observer should be able to >> recognize the obvious flaw - the numbers don't show any expected >> relationship between # of entity groups or the use of global >> transactions. Interpreted literally, you would assume that local >> transactions are much faster for 5 entity groups, but global >> transactions are much faster for 4 entity groups. >> >> It's pretty obvious that the benchmark author just ran one pass and >> took the numbers as-is. If you actually run multiple passes, you'll >> find that there is enormous variance in the timing. The only way you >> can realistically measure results like this on appengine is to execute >> the test 100 times and take a median. >> >> FWIW, I deliberately haven't made any performance claims for Objectify >> because I haven't put the necessary amount of due diligence into >> creating a proper set of benchmarks. It is not easy to measure >> performance, especially in a dynamic environment like appengine. I >> only hope that the Slim3 authors have put more care and thought into >> crafting their library than they have their benchmarks. >> >> Jeff >> >> >> On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber <[email protected]> wrote: >>> Slim3 is not only fast, the api is completely awesome. It has been my choice >>> for a year now for all gae projects. >>> It includes "name safety" and and amazing querying utils. >>> Very recommendable! >>> >>> On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer <[email protected]> wrote: >>>> >>>> You are wrong. >>>> >>>> Try adding getProperty() calls to your LL performance test, and the >>>> speed advantage of the LL API goes away. I don't know what to say >>>> about Slim3, but here's my test case: >>>> >>>> >>>> http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test >>>> >>>> I created 10,000 entities in the datastore that have the same format >>>> as your test case - a single string property. Here's what happens >>>> (try it - and remember to reload the urls several times to get a >>>> realistic median value): >>>> >>>> ###### Low Level API with just .size() >>>> >>>> http://voodoodyne.appspot.com/fetchLLSize >>>> >>>> The code: >>>> >>>> List<Entity> things = >>>> DatastoreServiceFactory.getDatastoreService() >>>> .prepare(new >>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >>>> .asList(FetchOptions.Builder.withDefaults()); >>>> things.size(); >>>> >>>> Note that results are almost always under 2000ms. Wild guess I'd say >>>> the median elapsed is ~1900, just like your example. >>>> >>>> ###### Low Level API with actual fetch of the data >>>> >>>> http://voodoodyne.appspot.com/fetchLL >>>> >>>> The code: >>>> >>>> List<Entity> things = >>>> DatastoreServiceFactory.getDatastoreService() >>>> .prepare(new >>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >>>> .asList(FetchOptions.Builder.withDefaults()); >>>> for (Entity ent: things) >>>> { >>>> ent.getKey(); >>>> ent.getProperty("value"); >>>> } >>>> >>>> Note that the duration is now considerably longer. Eyeballing the >>>> median elapsed time, I'd say somewhere around 3000ms. >>>> >>>> ###### Objectify fetching from datastore >>>> >>>> http://voodoodyne.appspot.com/fetch >>>> >>>> Objectify ofy = ObjectifyService.begin(); >>>> List<Thing> things = ofy.query(Thing.class).list(); >>>> for (Thing thing: things) >>>> { >>>> thing.getId(); >>>> thing.getValue(); >>>> } >>>> >>>> Note that the timing is pretty much the same as the LL API when it >>>> includes actual fetches of the entity values. It is, no doubt, just a >>>> little higher. >>>> >>>> ###### A pure measurement of Objectify's overhead >>>> >>>> http://voodoodyne.appspot.com/fakeFetch >>>> >>>> This causes Objectify to translate 10,000 statically-created Entity >>>> objects to POJOs. You can see the code here: >>>> >>>> >>>> http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java >>>> >>>> You'll notice (after you hit the URL a couple times to warm up the >>>> JIT) that elapsed time converges to somewhere around 120ms. >>>> >>>> ----------- >>>> >>>> Conclusion: >>>> >>>> The numbers in the original benchmark are a result of improper >>>> measurements. The actual wall-clock overhead for Objectify in this >>>> test is ~4% (120ms out of 3000ms). >>>> >>>> Further speculation on my part, but probably correct: The overhead of >>>> reflection is unlikely to be a significant part of that 4%. >>>> >>>> Sloppy work. >>>> >>>> Jeff >>>> >>>> On Wed, Jun 8, 2011 at 7:55 AM, Yasuo Higa <[email protected]> wrote: >>>> > It is not bogus. >>>> > LazyList#size() fetches all data as follows: >>>> > public int size() { >>>> > resolveAllData(); >>>> > return results.size(); >>>> > } >>>> > >>>> > Yasuo Higa >>>> > >>>> > On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson >>>> > <[email protected]> wrote: >>>> >> It's not my benchmark, it's Slim3's :) ...but you're right, it's bogus. >>>> >> I >>>> >> asked on the main appengine group too, and it turns out the low-level >>>> >> benchmark is doing lazy loading. With that fixed, their numbers come >>>> >> out >>>> >> like yours. >>>> >> I found this one too, which also gets results like yours: >>>> >> http://gaejava.appspot.com/ >>>> >> >>>> >> On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur <[email protected]> >>>> >> wrote: >>>> >>> >>>> >>> Indeed Dennis's measurements are very suspicious. First you should do >>>> >>> a couple of warming ups on each of the implementations to prevent >>>> >>> pollution like the JDO classpath scan for enhanced classes (which is >>>> >>> one of the reasons for the high initial run). Then do a couple of run >>>> >>> to determine a range of measurements to spot outlyers. your low-level >>>> >>> API 2millis is definately one. >>>> >>> >>>> >>> When I did the measurements I got the following results >>>> >>> low-level: 1150-1550 >>>> >>> Slim3: 1150-1600 >>>> >>> Objectify: 1950-2400 >>>> >>> JDO: 2100-2700 >>>> >>> >>>> >>> These measurements confirm that GAE designed implementations are >>>> >>> faster then the GAE implementation of a generic data access layer >>>> >>> (JDO), but not so extrem as initially posted. >>>> >>> >>>> >>> The initial response using JDO is a known issue and especially low >>>> >>> trafic website should not use it or use the always on feature (maybe >>>> >>> this will change in the new pricing model) >>>> >>> >>>> >>> Regards, >>>> >>> >>>> >>> Erwin >>>> >>> >>>> >>> On Jun 7, 11:00 am, Ian Marshall <[email protected]> wrote: >>>> >>> > The low-level API does indeed look very fast. >>>> >>> > >>>> >>> > Just a comment on JDO: repeat runs roughly halve the JDO run time. I >>>> >>> > presume that this is because for repeat runs the JDO persistence >>>> >>> > manager factory has already been constructed. >>>> >>> > >>>> >>> > On Jun 6, 8:44 pm, DennisP <[email protected]> wrote: >>>> >>> > >>>> >>> > > I'm looking at this online >>>> >>> > > demo:http://slim3demo.appspot.com/performance/ >>>> >>> > >>>> >>> > > Sample run: >>>> >>> > > The number of entities: 10000 >>>> >>> > > low-level API:get: 2 millis >>>> >>> > > Slim3: 2490 millis >>>> >>> > > JDO: 6030 millis >>>> >>> > >>>> >>> > > Is the low-level API really that much faster? >>>> >>> >>>> >>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> >>> Groups >>>> >>> "Google App Engine for Java" group. >>>> >>> To post to this group, send email to >>>> >>> [email protected]. >>>> >>> To unsubscribe from this group, send email to >>>> >>> [email protected]. >>>> >>> For more options, visit this group at >>>> >>> http://groups.google.com/group/google-appengine-java?hl=en. >>>> >>> >>>> >> >>>> >> -- >>>> >> You received this message because you are subscribed to the Google >>>> >> Groups >>>> >> "Google App Engine for Java" group. >>>> >> To post to this group, send email to >>>> >> [email protected]. >>>> >> To unsubscribe from this group, send email to >>>> >> [email protected]. >>>> >> For more options, visit this group at >>>> >> http://groups.google.com/group/google-appengine-java?hl=en. >>>> >> >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> > Groups "Google App Engine for Java" group. >>>> > To post to this group, send email to >>>> > [email protected]. >>>> > To unsubscribe from this group, send email to >>>> > [email protected]. >>>> > For more options, visit this group at >>>> > http://groups.google.com/group/google-appengine-java?hl=en. >>>> > >>>> > >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine for Java" group. >>>> To post to this group, send email to >>>> [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>> >>> >>> >>> >>> -- >>> Guit: Elegant, beautiful, modular and *production ready* gwt applications. >>> >>> http://code.google.com/p/guit/ >>> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine for Java" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
