[
https://issues.apache.org/jira/browse/TINKERPOP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208333#comment-15208333
]
Ted Wilmes edited comment on TINKERPOP-1233 at 3/23/16 12:29 PM:
-----------------------------------------------------------------
We could add asserts on the benchmark results object. I think the
{{assertFaster}} could be useful because it operates on a delta in latencies vs
raw latencies. That should be more stable across systems. The {{assertTime}}
may be tricky because, like you're suggesting, that raw number could vary
greatly from system to system. Your {{assertSmaller}} is a neat idea. JMH has
a number of profilers that you can run while your benchmarks are running but
the closest they get to object creation is a GC profiler that gives you how
much time was spent GCing and how many happened. We could see if that gave us
meaningful results, if not, we'd have to fall back on using Yourkit or
something else with a more feature-rich memory profiler.
As far as the traversals go, I thought it might be good to hit two areas
broadly speaking; traversal preparation/compilation (strategy application,
requirements, any other pre-execution prep that happens) and traversal
execution (run time iteration/step execution). An example of step execution
could be taking the HasStep or specifically, the filtering portion
(hasContainers) at its interior, and adding benchmarks to measure the
performance. I think it'd be cool to be able to point providers and potential
providers to benchmarking results that show we are diligently tracking and
improving latencies in key areas of TinkerPop that play a part in the response
times that their end users see.
was (Author: twilmes):
We could add asserts on the benchmark results object. I think the
`assertFaster` could be useful because it operates on a delta in latencies vs
raw latencies. That should be more stable across systems. The `assertTime`
may be tricky because, like you're suggesting, that raw number could vary
greatly from system to system. Your `assertSmaller` is a neat idea. JMH has a
number of profilers that you can run while your benchmarks are running but the
closest they get to object creation is a GC profiler that gives you how much
time was spent GCing and how many happened. We could see if that gave us
meaningful results, if not, we'd have to fall back on using Yourkit or
something else with a more feature-rich memory profiler.
As far as the traversals go, I thought it might be good to hit two areas
broadly speaking; traversal preparation/compilation (strategy application,
requirements, any other pre-execution prep that happens) and traversal
execution (run time iteration/step execution). An example of step execution
could be taking the HasStep or specifically, the filtering portion
(hasContainers) at its interior, and adding benchmarks to measure the
performance. I think it'd be cool to be able to point providers and potential
providers to benchmarking results that show we are diligently tracking and
improving latencies in key areas of TinkerPop that play a part in the response
times that their end users see.
> Gremlin-Benchmark wish list.
> ----------------------------
>
> Key: TINKERPOP-1233
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1233
> Project: TinkerPop
> Issue Type: Improvement
> Components: benchmark
> Affects Versions: 3.2.0-incubating
> Reporter: Marko A. Rodriguez
>
> [~twilmes] has developed {{gremlin-benchmark}} which is slated for 3.2.0
> (TINKERPOP-1016). This is really good as now we can ensure the Gremlin
> traversal machine only speeds up with each release. Here is a collection of
> things I would like to be able to do with {{gremlin-benchmark}}.
> ----
> *Benchmarks in the Strategy Tests*
> {code}
> // ensure that traversalA is at least 1.5 times faster than traversalB
> assertTrue(Benchmark.compare(traversalA,traversalB) > 1.50d)
> {code}
> With this, I can have an {{OptimizationStrategy}} applied to {{traversalA}}
> and not to {{traversalB}} and prove via "mvn clean install" that the strategy
> is in fact "worth it." I bet there are other good static methods we could
> create. Hell, why not just have a {{BenchmarkAsserts}} that we can statically
> import like JUnit {{Asserts}}. Then its just:
> {code}
> assertFaster(traversalA,traversalB,1.50d)
> assertSmaller(traversalA,traversalB) // memory usage or object creation?
> assertTime(traversal, 1000, TimeUnits.MILLISECONDS) // has to complete in 1
> second?
> ... ?
> {code}
> Its a little scary as not all computers are the same, but it would be nice to
> know that we have tests for space and time costs.
> ----
> *Benchmarks saved locally over the course of a release*
> This is tricky, but it would be cool if local folders (not to GitHub) were
> created like this:
> {code}
> tinkerpop3/gremlin-benchmark/benchmarks/g_V_out_out_12:23:66UT23.txt
> {code}
> Then a test case could ensure that all newer runs of that benchmark are
> faster than older ones. If its, lets say, 10%+ slower, {{Exception}} and test
> fails. ??
> What else can we do? Can we know whether a certain area of code is faster?
> For instance, strategy application or requirements aggregation? If we can
> introspect like that, that would be stellar.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
