There are performance tests in the test suite and you can wire them up just as you do the normal tests - note how TinkerGraph does them:
https://github.com/apache/incubator-tinkerpop/blob/ad27fce579a182de3ebf886fdbd85d5960852bdd/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphStructurePerformanceTest.java https://github.com/apache/incubator-tinkerpop/blob/ad27fce579a182de3ebf886fdbd85d5960852bdd/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/groovy/TinkerGraphGroovyEnvironmentPerformanceTest.java There's not much meat to them I'm afraid but it's what we have right now. I think that expanding on these in the way Marko had said would make some sense. For TinkerPop core purposes we can focus on tracking TinkerGraph only, but if those who implement the core interfaces want to make use of the benchmarking tests I think it would be good if we allowed that. On Mon, Sep 28, 2015 at 7:40 AM, Ran Magen <[email protected]> wrote: > Hopping on the discussion, is there a standard way for vendors to test > performance? Do the performance suites give some kind of result? Can it be > used to compare different implementations? > On יום א׳, 27 בספט׳ 2015 at 17:26 Stephen Mallette <[email protected]> > wrote: > > > We already have a pattern for "performance tests" that uses some > > benchmarking libs so it wouldn't be hard to extend on that. We'd > discussed > > this kinda thing before on the list (or perhaps an old ticket) - Daniel > > Kuppitz had some reservations about it, though I don't remember what they > > were, so we never extended on it. Kinda like the idea of storing the > stats > > (labelled as you have it) in the repo itself - that seems like a good > idea. > > > > On Wed, Sep 23, 2015 at 2:35 PM, Marko Rodriguez <[email protected]> > > wrote: > > > > > Hello, > > > > > > I think it would be important to have have an integration test that > > > benchmarks Gremlin (over TinkerGraph). > > > > > > What does this look like? > > > > > > 1. We have a collection of traversals that span the various > uses > > > of Gremlin. (write, read, path, aggregates, etc.) > > > 2. We have a scale free graph (250k edges?) in TinkerGraph that > > we > > > run this traversal set against. > > > 3. We save the results of this benchmark a stats/ like > directory > > > that get pushed to the repository. > > > - ???/stats/marko-09-23-2015-macosx-3.0.1.txt > > > - ???/stats/marko-09-23-2015-macosx-3.1.0-SNAPSHOT.txt > > > - etc. > > > 4. We can then look at how queries become better or worse with > > > each release (and in SNAPSHOTS). > > > - A cross file visualization these benchmarks would be > > > great so we can see (easily) which aspects of Gremlin are getting > > > better/worse. > > > > > > Why only TinkerGraph? > > > > > > 1. We don't want to benchmark a database/disk. This is to > > > benchmark Gremlin with itself through time. > > > 2. TinkerGraph doesn't evolve. Its probably the most stable > piece > > > of code in TP3 -- thus, its a good baseline system. > > > > > > Is anyone interested in working on something like this? Note that I'm > not > > > versed in the best practices for doing this, so if there is a better > way > > to > > > accomplish the goal of benchmarking Gremlin over releases, lets do > that. > > > However, lets start simple and grow so we can get something working and > > > providing us insights ASAP. > > > > > > Finally, if this is generally a good idea, I can make a ticket in JIRA > > for > > > this. > > > > > > Thoughts?, > > > Marko. > > > > > > http://markorodriguez.com > > > > > > > > >
