What I want to provide is a fair and casual benchmark.
As jeff advised, I modified samples as follows:
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty("sortValue");
}
for (Bar bar : service.getBarListUsingSlim3()) {
bar.getKey();
bar.getSortValue();
}
for (BarObjectify bar : service.getBarListUsingObjectify()) {
bar.getKey();
bar.getSortValue();
}
for (BarJDO bar : service.getBarListUsingJDO()) {
bar.getKey();
bar.getSortValue();
}
LL API is much slower than before.
http://slim3demo.appspot.com/performance/
Yasuo Higa
On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer <[email protected]> wrote:
> Slim3 may be a nice piece of software, but it has not been
> demonstrated to be faster than anything (including JDO). It might or
> might not be faster - I don't know - but based on the sloppy
> benchmarking, I'm pretty certain that the people making this claim
> don't know either.
>
> There's another ill-conceived performance claim on the Slim3 website:
> "You may worry about the overhead of global transactions. Don't worry.
> It is not very expensive." There are three problems with their little
> performance test:
>
> 1) It only measures wall-clock time, not cost.
> 2) It does not measure what happens under contention.
> 3) The numbers are obviously wrong - they don't even pass a smoke test.
>
> Look at these numbers (from the Slim3 home page):
>
> Entity Groups Local Transaction(millis) Global Transaction(millis)
> 1 67 70
> 2 450 415
> 3 213 539
> 4 1498 981
> 5 447 781
>
> Just like the 1ms low-level API query performance in the benchmark
> that spawned this thread, even a casual observer should be able to
> recognize the obvious flaw - the numbers don't show any expected
> relationship between # of entity groups or the use of global
> transactions. Interpreted literally, you would assume that local
> transactions are much faster for 5 entity groups, but global
> transactions are much faster for 4 entity groups.
>
> It's pretty obvious that the benchmark author just ran one pass and
> took the numbers as-is. If you actually run multiple passes, you'll
> find that there is enormous variance in the timing. The only way you
> can realistically measure results like this on appengine is to execute
> the test 100 times and take a median.
>
> FWIW, I deliberately haven't made any performance claims for Objectify
> because I haven't put the necessary amount of due diligence into
> creating a proper set of benchmarks. It is not easy to measure
> performance, especially in a dynamic environment like appengine. I
> only hope that the Slim3 authors have put more care and thought into
> crafting their library than they have their benchmarks.
>
> Jeff
>
>
> On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber <[email protected]> wrote:
>> Slim3 is not only fast, the api is completely awesome. It has been my choice
>> for a year now for all gae projects.
>> It includes "name safety" and and amazing querying utils.
>> Very recommendable!
>>
>> On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer <[email protected]> wrote:
>>>
>>> You are wrong.
>>>
>>> Try adding getProperty() calls to your LL performance test, and the
>>> speed advantage of the LL API goes away. I don't know what to say
>>> about Slim3, but here's my test case:
>>>
>>>
>>> http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test
>>>
>>> I created 10,000 entities in the datastore that have the same format
>>> as your test case - a single string property. Here's what happens
>>> (try it - and remember to reload the urls several times to get a
>>> realistic median value):
>>>
>>> ###### Low Level API with just .size()
>>>
>>> http://voodoodyne.appspot.com/fetchLLSize
>>>
>>> The code:
>>>
>>> List<Entity> things =
>>> DatastoreServiceFactory.getDatastoreService()
>>> .prepare(new
>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
>>> .asList(FetchOptions.Builder.withDefaults());
>>> things.size();
>>>
>>> Note that results are almost always under 2000ms. Wild guess I'd say
>>> the median elapsed is ~1900, just like your example.
>>>
>>> ###### Low Level API with actual fetch of the data
>>>
>>> http://voodoodyne.appspot.com/fetchLL
>>>
>>> The code:
>>>
>>> List<Entity> things =
>>> DatastoreServiceFactory.getDatastoreService()
>>> .prepare(new
>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
>>> .asList(FetchOptions.Builder.withDefaults());
>>> for (Entity ent: things)
>>> {
>>> ent.getKey();
>>> ent.getProperty("value");
>>> }
>>>
>>> Note that the duration is now considerably longer. Eyeballing the
>>> median elapsed time, I'd say somewhere around 3000ms.
>>>
>>> ###### Objectify fetching from datastore
>>>
>>> http://voodoodyne.appspot.com/fetch
>>>
>>> Objectify ofy = ObjectifyService.begin();
>>> List<Thing> things = ofy.query(Thing.class).list();
>>> for (Thing thing: things)
>>> {
>>> thing.getId();
>>> thing.getValue();
>>> }
>>>
>>> Note that the timing is pretty much the same as the LL API when it
>>> includes actual fetches of the entity values. It is, no doubt, just a
>>> little higher.
>>>
>>> ###### A pure measurement of Objectify's overhead
>>>
>>> http://voodoodyne.appspot.com/fakeFetch
>>>
>>> This causes Objectify to translate 10,000 statically-created Entity
>>> objects to POJOs. You can see the code here:
>>>
>>>
>>> http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java
>>>
>>> You'll notice (after you hit the URL a couple times to warm up the
>>> JIT) that elapsed time converges to somewhere around 120ms.
>>>
>>> -----------
>>>
>>> Conclusion:
>>>
>>> The numbers in the original benchmark are a result of improper
>>> measurements. The actual wall-clock overhead for Objectify in this
>>> test is ~4% (120ms out of 3000ms).
>>>
>>> Further speculation on my part, but probably correct: The overhead of
>>> reflection is unlikely to be a significant part of that 4%.
>>>
>>> Sloppy work.
>>>
>>> Jeff
>>>
>>> On Wed, Jun 8, 2011 at 7:55 AM, Yasuo Higa <[email protected]> wrote:
>>> > It is not bogus.
>>> > LazyList#size() fetches all data as follows:
>>> > public int size() {
>>> > resolveAllData();
>>> > return results.size();
>>> > }
>>> >
>>> > Yasuo Higa
>>> >
>>> > On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson
>>> > <[email protected]> wrote:
>>> >> It's not my benchmark, it's Slim3's :) ...but you're right, it's bogus.
>>> >> I
>>> >> asked on the main appengine group too, and it turns out the low-level
>>> >> benchmark is doing lazy loading. With that fixed, their numbers come
>>> >> out
>>> >> like yours.
>>> >> I found this one too, which also gets results like yours:
>>> >> http://gaejava.appspot.com/
>>> >>
>>> >> On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur <[email protected]>
>>> >> wrote:
>>> >>>
>>> >>> Indeed Dennis's measurements are very suspicious. First you should do
>>> >>> a couple of warming ups on each of the implementations to prevent
>>> >>> pollution like the JDO classpath scan for enhanced classes (which is
>>> >>> one of the reasons for the high initial run). Then do a couple of run
>>> >>> to determine a range of measurements to spot outlyers. your low-level
>>> >>> API 2millis is definately one.
>>> >>>
>>> >>> When I did the measurements I got the following results
>>> >>> low-level: 1150-1550
>>> >>> Slim3: 1150-1600
>>> >>> Objectify: 1950-2400
>>> >>> JDO: 2100-2700
>>> >>>
>>> >>> These measurements confirm that GAE designed implementations are
>>> >>> faster then the GAE implementation of a generic data access layer
>>> >>> (JDO), but not so extrem as initially posted.
>>> >>>
>>> >>> The initial response using JDO is a known issue and especially low
>>> >>> trafic website should not use it or use the always on feature (maybe
>>> >>> this will change in the new pricing model)
>>> >>>
>>> >>> Regards,
>>> >>>
>>> >>> Erwin
>>> >>>
>>> >>> On Jun 7, 11:00 am, Ian Marshall <[email protected]> wrote:
>>> >>> > The low-level API does indeed look very fast.
>>> >>> >
>>> >>> > Just a comment on JDO: repeat runs roughly halve the JDO run time. I
>>> >>> > presume that this is because for repeat runs the JDO persistence
>>> >>> > manager factory has already been constructed.
>>> >>> >
>>> >>> > On Jun 6, 8:44 pm, DennisP <[email protected]> wrote:
>>> >>> >
>>> >>> > > I'm looking at this online
>>> >>> > > demo:http://slim3demo.appspot.com/performance/
>>> >>> >
>>> >>> > > Sample run:
>>> >>> > > The number of entities: 10000
>>> >>> > > low-level API:get: 2 millis
>>> >>> > > Slim3: 2490 millis
>>> >>> > > JDO: 6030 millis
>>> >>> >
>>> >>> > > Is the low-level API really that much faster?
>>> >>>
>>> >>> --
>>> >>> You received this message because you are subscribed to the Google
>>> >>> Groups
>>> >>> "Google App Engine for Java" group.
>>> >>> To post to this group, send email to
>>> >>> [email protected].
>>> >>> To unsubscribe from this group, send email to
>>> >>> [email protected].
>>> >>> For more options, visit this group at
>>> >>> http://groups.google.com/group/google-appengine-java?hl=en.
>>> >>>
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> >> Groups
>>> >> "Google App Engine for Java" group.
>>> >> To post to this group, send email to
>>> >> [email protected].
>>> >> To unsubscribe from this group, send email to
>>> >> [email protected].
>>> >> For more options, visit this group at
>>> >> http://groups.google.com/group/google-appengine-java?hl=en.
>>> >>
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups "Google App Engine for Java" group.
>>> > To post to this group, send email to
>>> > [email protected].
>>> > To unsubscribe from this group, send email to
>>> > [email protected].
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/google-appengine-java?hl=en.
>>> >
>>> >
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine for Java" group.
>>> To post to this group, send email to
>>> [email protected].
>>> To unsubscribe from this group, send email to
>>> [email protected].
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine-java?hl=en.
>>>
>>
>>
>>
>> --
>> Guit: Elegant, beautiful, modular and *production ready* gwt applications.
>>
>> http://code.google.com/p/guit/
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine for Java" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine-java?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
