I've written this sort of post before with ideas around testing. Those past posts haven't really materialized into actual implementation in most cases as they've simply not panned out or some new realization developed which shipped them to the bin. In any case, this post is meant to supersede those.
As a quick reminder of the problems I routinely consider: 1. Gremlin integration tests are running and re-running in redundant configurations (e.g. test serialization directly but then re-test serialization again in each GLV run) 2. Integration tests are duplicated - must write them once in Java in gremlin-test and then again in Gherkin as a feature file. 3. There are under-tested areas like Translators which could hook into the wealth of Gremlin examples in the feature files (I consider this a "new" sort of problem because Translators have become more like first class citizens recently and they lack tests). 4. It's possible that we are over-testing with Structure and Process integration tests - the interface we want providers to adhere to is Gremlin and the Structure API really doesn't help there. It just adds a burden. (I haven't done a full analysis here so this is just a hunch, but I can't help but wonder if the Structure tests can't be retired) My latest thinking on these issues is mostly shaped by some recent work on: https://issues.apache.org/jira/browse/TINKERPOP-2443 where I've made it so that feature files are passed to through the PythonTranslator to generate a Python test file which gets referenced by radish when gremlin-python is built, which means that we fully test the output of the PythonTranslator in native python and in the process validate that gremlin-python and its serializers work as well. That body of work points to where testing in 3.5.0 should head to help eliminate those problems I previously listed. At a really high level, I think that means the following sorts of changes: 1. Implement this model from Python for Javascript and .NET (I assume .NET will be "hard" to write a Translator for, but we'll see how that goes) 2. Implement this model for gremlin-java (which would be a new module named to be consistent with other variants - will help remove a cyclic dependency with the server as well) and gremlin-groovy. 3. Restructure gremlin-test to use Gherkin exclusively and the feature files and remove the old junit tests and any other tests that are duplicative of functionality already tested fully under the GLV tests. 4. Look at what tests remain in gremlin-test and build out more gherkin functionality to support those cases. For providers they will continue to use gremlin-test as they normally have but then will also be able to easily run variant level tests to ensure that their implementations work in all languages. I think that the risks in getting this all working will be with spark/hadoop level testing, but that has its own shortcomings https://issues.apache.org/jira/browse/TINKERPOP-2387 and perhaps can be ignored in favor of trying to finally simplify our test suite and bring it up to date with the current span of thinking about what's important in the project. Anyway, kinda boring but I thought i'd try to express where my head was these days on this topic and help organize my own thoughts. Once TINKERPOP-2443 merges, I will keep heading down this path to see where it goes. Please let me know if there are any questions on what I'm suggesting or if there are other ideas that should be considered. Thanks!
