We already have a pattern for "performance tests" that uses some benchmarking libs so it wouldn't be hard to extend on that. We'd discussed this kinda thing before on the list (or perhaps an old ticket) - Daniel Kuppitz had some reservations about it, though I don't remember what they were, so we never extended on it. Kinda like the idea of storing the stats (labelled as you have it) in the repo itself - that seems like a good idea.
On Wed, Sep 23, 2015 at 2:35 PM, Marko Rodriguez <[email protected]> wrote: > Hello, > > I think it would be important to have have an integration test that > benchmarks Gremlin (over TinkerGraph). > > What does this look like? > > 1. We have a collection of traversals that span the various uses > of Gremlin. (write, read, path, aggregates, etc.) > 2. We have a scale free graph (250k edges?) in TinkerGraph that we > run this traversal set against. > 3. We save the results of this benchmark a stats/ like directory > that get pushed to the repository. > - ???/stats/marko-09-23-2015-macosx-3.0.1.txt > - ???/stats/marko-09-23-2015-macosx-3.1.0-SNAPSHOT.txt > - etc. > 4. We can then look at how queries become better or worse with > each release (and in SNAPSHOTS). > - A cross file visualization these benchmarks would be > great so we can see (easily) which aspects of Gremlin are getting > better/worse. > > Why only TinkerGraph? > > 1. We don't want to benchmark a database/disk. This is to > benchmark Gremlin with itself through time. > 2. TinkerGraph doesn't evolve. Its probably the most stable piece > of code in TP3 -- thus, its a good baseline system. > > Is anyone interested in working on something like this? Note that I'm not > versed in the best practices for doing this, so if there is a better way to > accomplish the goal of benchmarking Gremlin over releases, lets do that. > However, lets start simple and grow so we can get something working and > providing us insights ASAP. > > Finally, if this is generally a good idea, I can make a ticket in JIRA for > this. > > Thoughts?, > Marko. > > http://markorodriguez.com > >
