Just another update on progress with the test suite. The language of the .feature files is getting slightly more complex as I try to translate more and more of our Java process suite tests into the language of the gherkin files. You can see here where I've added the ability to include parameters:
https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/gremlin-test/features/filter/Has.feature#L33 I also came up with a method of asserting edges (didn't want to rely on ids as it makes the gherkin harder to read, plus i didn't want to assume TinkerGraph identifiers in case these tests were every used with some other graph database that didn't use longs): https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/gremlin-test/features/map/Vertex.feature#L105-L111 for all those additions (and others) the logic required by the GLV to process these tests has stayed surprisingly simple: https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/gremlin-python/src/main/jython/radish/feature_steps.py There's a fair bit of regex/string manipulation involved there, but it's processing strings from the feature file so that's the nature of it I suppose. I think I'm of the mind that I want to port all of the tests to feature files, so I wrote this unit test to help validate that none were missed: https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/gremlin-test/src/test/java/org/apache/tinkerpop/gremlin/structure/FeatureCoverageTest.java I don't know that we have to get to 100% porting right away, but I think that once we have these gherkin files written they not only become the basis for our current GLV testing work, but we might be able to simply use them for testing the Java stuff as well - that would rid us of having the test code duplication. It also sets us up with a portable body of tests that can be re-used in TinkerPop 4.x. I'm open to suggestions if anyone has any. On Mon, Sep 25, 2017 at 11:23 AM, Jorge Bay Gondra <jorgebaygon...@gmail.com > wrote: > I was able to build a proof of concept for a Gherkin-based test runner in > C#, that takes the proposed count and select features > <https://github.com/apache/tinkerpop/tree/TINKERPOP-1784/ > gremlin-test/features/map> > and runs them using C# step definitions. > > It uses the Gherkin parser <https://github.com/cucumber/gherkin-dotnet> > from > cucumber, there isn't a release of the parser with .NET Core support so > I've > asked them to release one > <https://github.com/cucumber/gherkin-dotnet/issues/11> (there is no > limitation from their source files). If they are not able to release it in > the next few days, we can implement our own as it should be pretty straight > forward. > > On Fri, Sep 22, 2017 at 2:23 PM, Stephen Mallette <spmalle...@gmail.com> > wrote: > > > Thanks for the update. I'm trying to keep the test language as simple as > > possible so that we don't need an overly complicated test implementation. > > Hopefully that will help make the .NET approach as easy as possible. > > > > On Fri, Sep 22, 2017 at 8:20 AM, Jorge Bay Gondra < > > jorgebaygon...@gmail.com> > > wrote: > > > > > I've been looking into Gherkin support for .NET: SpecFlow, the cucumber > > > implementation for .NET <https://cucumber.io/docs# > > cucumber-implementations > > > >, > > > does not support .NET Core platform (we use .NET Core build tools for > the > > > .NET GLV) and only supports .NET Framework. > > > > > > From what I can see <https://github.com/techtalk/SpecFlow/projects/2>, > > > .NET > > > Core support on SpecFlow is coming at a very slow pace and we shouldn't > > > expect to land any time soon (there were some design decisions in > > SpecFlow > > > library that makes supporting other platforms non-trivial, like > requiring > > > code gen). > > > > > > The alternative would be to implement our own harness to support it: > > from a > > > xunit test, look for certain types and parse the annotations, and > execute > > > them using the Gherkin features. > > > There is a .NET cross-platform Gherkin parser: > > > https://github.com/cucumber/gherkin-dotnet > > > I'll continue looking into this option and try to understand the effort > > > required... > > > > > > On Tue, Sep 19, 2017 at 6:21 PM, Jorge Bay Gondra < > > > jorgebaygon...@gmail.com> > > > wrote: > > > > > > > Nice! Gherkin will make our lives easier with a growing number of > GLVs. > > > > > > > > We should find a way to define the different features supported by > each > > > > GLV, as it's reasonable to have different maturity levels per GLV > (ie: > > > > lambdas support, traversal strategy, ...). I don't know if it will be > > > > beneficial to do it in the Gherkin files or within each GLV > > > implementation. > > > > Also, we should consider the process of rolling out a new method / > > class > > > > in the java implementation, how that could affect each GLV. > > > > > > > > On Thu, Sep 14, 2017 at 11:12 PM, Stephen Mallette < > > spmalle...@gmail.com > > > > > > > > wrote: > > > > > > > >> that's what i meant by "reflection" or as you suggest eval(). I > guess > > > the > > > >> point is that if the language can support some way of taking the > > string > > > >> value and turning it automatically into a traversal in that GLVs > style > > > >> then > > > >> we should do that. > > > >> > > > >> On Thu, Sep 14, 2017 at 4:45 PM, Daniel Kuppitz <m...@gremlin.guru> > > > wrote: > > > >> > > > >> > For unparameterized queries it can probably be as easy as: > > > >> > > > > >> > @given("the traversal of") > > > >> > def translate_traversal(step): > > > >> > g = step.context.g > > > >> > step.context.traversal = eval(step.text) > > > >> > > > > >> > > > > >> > Cheers, > > > >> > Daniel > > > >> > > > > >> > > > > >> > On Thu, Sep 14, 2017 at 1:39 PM, Daniel Kuppitz <m...@gremlin.guru> > > > >> wrote: > > > >> > > > > >> > > That's great stuff. I haven't used Cucumber / Gherkin for years, > > > but I > > > >> > > really like the BDD approach. > > > >> > > > > > >> > > and then you can look at the GLV Gremlin translations > specifically > > > >> here: > > > >> > >> https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/grem > > > >> > >> lin-python/src/main/jython/radish/count_features_step.py# > L34-L46 > > > >> > > > > > >> > > > > > >> > > This part is the only thing that looks weird to me. You're > > basically > > > >> > > writing every query twice; is there really no easier way to do > > that? > > > >> > > > > > >> > > Cheers, > > > >> > > Daniel > > > >> > > > > > >> > > > > > >> > > On Thu, Sep 14, 2017 at 1:17 PM, Stephen Mallette < > > > >> spmalle...@gmail.com> > > > >> > > wrote: > > > >> > > > > > >> > >> I've brought this issue up in the past and had suggested some > > > >> options I > > > >> > >> had > > > >> > >> in mind but now I've finally put the basics of those ideas in > > place > > > >> so I > > > >> > >> figured I'd start a fresh thread. Recall that the issue at hand > > is > > > >> that > > > >> > we > > > >> > >> don't have a test suite for GLVs as gremlin-test is bound to > the > > > >> JVM. We > > > >> > >> have some tricks that let us test gremlin-python with it but > > those > > > >> > tricks > > > >> > >> won't work for every language and we now have the first > language > > in > > > >> > >> gremlin-dotnet and upcoming gremlin-javascript which won't > > support > > > it > > > >> > >> (yes, > > > >> > >> i know that gremlin-javascript can run on the jvm but there are > > > >> issues > > > >> > >> with > > > >> > >> getting it all to work with the test framework that make it > > unduly > > > >> > >> complicated). > > > >> > >> > > > >> > >> On other threads I offered the idea that we look to use Gherkin > > to > > > >> write > > > >> > >> general Gremlin test specifications, which then could be read > and > > > >> > >> processed > > > >> > >> by the wide variety of test frameworks that can read that > format > > - > > > >> there > > > >> > >> tend to be Gherkin processors in just about every language - > for > > > >> > example, > > > >> > >> see: > > > >> > >> > > > >> > >> https://cucumber.io/ > > > >> > >> > > > >> > >> I just created this issue: > > > >> > >> > > > >> > >> https://issues.apache.org/jira/browse/TINKERPOP-1784 > > > >> > >> > > > >> > >> and pushed this branch: > > > >> > >> > > > >> > >> https://github.com/apache/tinkerpop/tree/TINKERPOP-1784 > > > >> > >> > > > >> > >> which demonstrates how this works with gremlin-python. The > basic > > > >> anatomy > > > >> > >> of > > > >> > >> this setup involves this new directory in gremlin-test: > > > >> > >> > > > >> > >> https://github.com/apache/tinkerpop/tree/TINKERPOP-1784/grem > > > >> > >> lin-test/features > > > >> > >> > > > >> > >> It contains the Gherkin .features files. These are the test > > > >> > >> specifications. > > > >> > >> They are written using gremlin-java as the "model" language. > GLVs > > > >> will > > > >> > >> then > > > >> > >> need to write some infrastructure to process these Gherkin > files. > > > The > > > >> > key > > > >> > >> to making this "easy" to implement will lie in our abiilty to > > keep > > > >> the > > > >> > >> assertions we want to have relatively simple. The more > simplistic > > > the > > > >> > >> language in the Gherkin .feature files the easier the job it > will > > > be > > > >> for > > > >> > >> GLVs to build their infrastructure. Of course, once that > > > >> infrastructure > > > >> > is > > > >> > >> in place, the GLV developer just has to write the GLV version > of > > > the > > > >> > >> Gremlin specified in the .feature file. So you can look at all > > the > > > >> > >> "infrastructure" code here in this pair of files: > > > >> > >> > > > >> > >> https://github.com/apache/tinkerpop/tree/TINKERPOP-1784/grem > > > >> > >> lin-python/src/main/jython/radish > > > >> > >> > > > >> > >> and then you can look at the GLV Gremlin translations > > specifically > > > >> here: > > > >> > >> > > > >> > >> https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/grem > > > >> > >> lin-python/src/main/jython/radish/count_features_step.py# > L34-L46 > > > >> > >> > > > >> > >> I think this approach works pretty well and solves our general > > > >> problems > > > >> > >> for > > > >> > >> GLV testing. There is some pain up front in implementing the > > > >> > >> "infrastructure" but after that new Gremlin tests added to > > .feature > > > >> > files > > > >> > >> just need to translated in the GLV. I suppose we could > > "automate" a > > > >> good > > > >> > >> portion of the translation with reflection of some sort. > Anything > > > >> else > > > >> > >> could just be handled manually. > > > >> > >> > > > >> > >> Not sure if we need to use this new model to wholly replace the > > old > > > >> one. > > > >> > >> The process test suite has its place in helping graph database > > > >> providers > > > >> > >> test their stuff. I also imagine that introducing this approach > > in > > > >> that > > > >> > >> context would create a breaking change which we would then need > > to > > > >> push > > > >> > >> off > > > >> > >> to 3.4.0. I suppose that gives us time to think, but for now > it > > > >> might > > > >> > not > > > >> > >> be best to conflate the two and just treat them as separate > > aspects > > > >> of > > > >> > the > > > >> > >> test suite. > > > >> > >> > > > >> > >> Anyway - it's important we settle on an approach to testing as > we > > > >> really > > > >> > >> shouldn't do a GA release of the Gremlin .NET GLV without > getting > > > the > > > >> > test > > > >> > >> suite solid. please yell if you have any ideas or feedback on > > this > > > >> > >> approach. > > > >> > >> > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > >
