This is an automated email from the ASF dual-hosted git repository. spmallette pushed a commit to branch TINKERPOP-2601 in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit ffbb3ed6b5ad876f9c2964f3d47506b34caf2db5 Author: Stephen Mallette <[email protected]> AuthorDate: Fri Aug 13 10:58:42 2021 -0400 TINKERPOP-2601 Documentation for Gherkin for providers --- CHANGELOG.asciidoc | 1 + docs/src/dev/developer/for-committers.asciidoc | 17 ++- docs/src/dev/provider/index.asciidoc | 114 ++++++++++++++++++++- docs/src/upgrade/release-3.6.x.asciidoc | 17 +++ .../tinkerpop/gremlin/features/StepDefinition.java | 3 - .../tinkergraph/TinkerGraphFeatureTest.java | 2 - 6 files changed, 147 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index cdd2725..8f87718 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -31,6 +31,7 @@ limitations under the License. * Removed `groovy` and `groovy-json` dependencies from `gremlin-driver` as well as related `JsonBuilder` serialization support. * Replaced log4j usage with logback where builds rely on and packaged distributions now contain the latter. * Prevented metrics computation unless the traversal is in a locked state. +* Exposed Gherkin tests as part of the provider test suite. * Bumped to Apache Hadoop 3.3.1. * Bumped to Apache Spark 3.1.2. diff --git a/docs/src/dev/developer/for-committers.asciidoc b/docs/src/dev/developer/for-committers.asciidoc index dda05e0..e8935ce 100644 --- a/docs/src/dev/developer/for-committers.asciidoc +++ b/docs/src/dev/developer/for-committers.asciidoc @@ -198,6 +198,7 @@ call out this fact. The JIRA issues that track removal of deprecated methods should be periodically evaluated to determine if it is prudent to schedule them into a release. +[[developing-tests]] == Developing Tests TinkerPop has a wide variety of test types that help validate its internal code as well as external provider code. @@ -224,9 +225,10 @@ environments without a `GraphProvider` being initialized by a suite. These types `Check` instead. Please see link:https://github.com/apache/tinkerpop/blob/e32a4187e4f25e290aabe14007f9087c48a06521/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/structure/NativeNeo4jStructureCheck.java[NativeNeo4jStructureCheck] for an example. +[[gremlin-language-test-cases]] === Gremlin Language Test Cases -Test cases for the Gremlin Language currently requires that the newly developed test be added in three places: +Test cases for the Gremlin Language currently requires that the newly developed test be added in two places: 1. As a test written in Java in the `gremlin-test` module within the subpackages of `org.apache.tinkerpop.gremlin.process.traversal.step` @@ -464,6 +466,19 @@ Scenario: g_V_outE_drop And the graph should return 0 for count of "g.E()" ---- +[[gherkin-tags]] +==== Tags + +Scenarios have tags associated with them that help identify subsets of tests so that a test runner can selectively +include or ignore certain tests. The tags enable the practical and necessary ability for providers to ignore tests that +they simply cannot support. It is important to be aware of the following tags when writing tests as not including a +tag when one is necessary will cause provider tests to fail: + +* *@MultiMetaProperties* - The scenario uses "The Crew" graph as a dataset or otherwise utilizes multi-properties +or meta-properties for creating or querying data. +* *@RemoteOnly* - The scenario uses some Gremlin syntax that cannot be supported outside of remote test executions. The +best example of this sort of test would be one that uses the remote `Lambda` syntax. + == Developing Benchmarks Benchmarks are a useful tool to track performance between TinkerPop versions and also as tools to aid development diff --git a/docs/src/dev/provider/index.asciidoc b/docs/src/dev/provider/index.asciidoc index 1c928f6..480dee0 100644 --- a/docs/src/dev/provider/index.asciidoc +++ b/docs/src/dev/provider/index.asciidoc @@ -617,6 +617,29 @@ image:gremlin-edumacated.png[width=225] </dependency> ---- +Providers currently have two approaches to consider when validating their TinkerPop implementations. The first approach +comes from the wholly JVM oriented original test suite which was developed in the early days of TinkerPop 3.x design +and development. The second approach is available as of 3.6.0, is Gherkin-based and originates from the Gremlin +Language Variant test suite which is language agnostic. + +The first approach is more complete and more opinionated as to how an implementation should behave and in many ways +helpful in getting an implementation semantically correct from the ground up (i.e. first getting the `Graph` Structure +API implemented well by getting the Structure Suite to pass which will almost inevitably ensure that the most of the +Gremlin language oriented tests in the Process Suite pass early on). On the other hand, the fact that this test suite +is rigorous also can make it harder to implement especially if your graph already exists and behaves in a certain +fashion. + +The second approach only validates Gremlin semantics which is ultimately what users concern themselves with as that is +the method by which they will interact with a provider's `Graph`. This test suite is less concerned with how a +TinkerPop implementation does what it does, so long as it succeeds at processing Gremlin traversals. There is +significant overlap between this test suite and the aforementioned Process Suite. + +At this time, it would be wise for providers to implement both approaches as the goal for TinkerPop is to move away +from the rigors of the JVM Structure and Process Suites in favor of Gherkin. Over time, the Structure and Process +Suites will be deprecated and removed. + +==== JVM Test Suite + The operational semantics of any OLTP or OLAP implementation are validated by `gremlin-test`. To implement these tests, provide test case implementations as shown below, where `XXX` below denotes the name of the graph implementation (e.g. TinkerGraph, Neo4jGraph, HadoopGraph, etc.). @@ -758,7 +781,7 @@ environment variable, depending on your project layout. Some tests require this creating temporary files. The value is typically set to the project build directory. For example using the Maven SureFire Plugin, this is done via the configuration argLine with `-Dbuild.dir=${project.build.directory}`. -==== Checking resource leak +===== Checking Resource Leaks The TinkerPop query engine retrieves data by interfacing with the provider using iterators. These iterators (depending on the provider) may hold up resources in the underlying storage layer and hence, it is critical to close them after @@ -772,6 +795,95 @@ is provided with TinkerGraph as `TinkerGraphIterator.java`. Assertions for leak detection are enabled by default when running the test suite. They can be temporarily disabled by way of a system property - simply set `-DtestIteratorLeaks=false". +[[gherkin-tests-suite]] +==== Gherkin Test Suite + +The Gherkin Test Suite is a language agnostic set of tests that verify Gremlin semantics. It provides a unified set of +tests that validate many TinkerPop components internally. The tests themselves can be found in `gremlin-tests/features` +(link:https://github.com/apache/tinkerpop/tree/x.y.z/gremlin-test/features[here]) with their syntax described in the +link:https://tinkerpop.apache.org/docs/x.y.z/dev/developer/#gremlin-language-test-cases[TinkerPop Developer Documentation]. + +TinkerPop provides some infrastructure, for JVM based graphs, to help make it easier for providers to implement these +tests against their implementations. This infrastructure is built on `cucumber-java` which is a dependency of +`gremlin-test`. There are two main components to implementing the tests: + +1. A `org.apache.tinkerpop.gremlin.features.World` implementation which is a class in `gremlin-test`. +2. A JUnit test class that will act as the runner for the tests with the appropriate annotations + +TIP: It may be helpful to get familiar with link:https://cucumber.io/docs/installation/java/[Cucumber] before +proceeding with an implementation. + +The `World` implementation provides context to the tests and allows providers to intercept test events that might be +important to proper execution specific to their implementations. The most important part of implementing `World` is +properly implementing the `GraphTraversalSource getGraphTraversalSource(GraphData)` method which provides to the test +the `GraphTraversalSource` to execute the test against. + +The JUnit test class is really just the test runner. It is a simple class which must include some Cucumber annotations. +The following is just an example as taken from TinkerGraph: + +[source,java] +---- +@RunWith(Cucumber.class) +@CucumberOptions( + tags = "not @RemoteOnly", + glue = { "org.apache.tinkerpop.gremlin.features" }, + features = { "../gremlin-test/features" }, + plugin = {"pretty", "junit:target/cucumber.xml", + objectFactory = GuiceFactory.class}) +---- + +The `@CucumberOptions` that are used are mostly implementation specific, so it will be up to the provider to make some +choices as to what is right for their environment. For TinkerGraph, it needed to ignore Gherkin tests with the +`@RemoteOnly` tag (the full list of possible tags can be found link:https://tinkerpop.apache.org/docs/current/dev/developer/#gherkin-tags[here]), +as will most providers. The "glue" will be the same for all test implementers as it refers to a package containing +TinkerPop's test infrastructure in `gremlin-test` (unless of course, a provider needs to develop their own +infrastructure for some reason). The "features" is the path to the actual Gherkin test files that should be made +available locally. The "plugin" defines a JUnit style output, which happens to be understood by Maven. + +The "objectFactory" is the last component. Cucumber relies on dependency injection to get a `World` implementation into +the test infrastructure. Providers may choose from multiple available implementations, but TinkerPop chose to use +Guice. To follow this approach include the following module: + +[source,xml] +---- +<dependency> + <groupId>com.google.inject</groupId> + <artifactId>guice</artifactId> + <version>4.2.3</version> + <scope>test</scope> +</dependency> +---- + +Following the TinkerGraph implementation, there are two classes to construct: + +[source,java] +---- +public class ServiceModule extends AbstractModule { + @Override + protected void configure() { + bind(World.class).to(TinkerGraphWorld.class); + } +} + +public class WorldInjectorSource implements InjectorSource { + @Override + public Injector getInjector() { + return Guice.createInjector(Stage.PRODUCTION, CucumberModules.createScenarioModule(), new ServiceModule()); + } +} +---- + +The key here is that the `TinkerGraphWorld` implementation gets bound to `World` in the `ServiceModule` and there is +a `WorldInjectorSource` that specifies the `ServiceModule` to Cucumber. As a final step, the provider's test resources +needs a `cucumber.properties` file with an entry that specifies the `InjectorSource` so that Guice can find it. Here +is the example taken from TinkerGraph where the `WorldInjectorSource` is inner class of `TinkerGraphFeatureTest` +itself. + +[source,text] +---- +guice.injector-source=org.apache.tinkerpop.gremlin.tinkergraph.TinkerGraphFeatureTest$WorldInjectorSource +---- + === Accessibility via GremlinPlugin image:gremlin-plugin.png[width=100,float=left] The applications distributed with TinkerPop do not distribute with diff --git a/docs/src/upgrade/release-3.6.x.asciidoc b/docs/src/upgrade/release-3.6.x.asciidoc index 2d0c3df..b8286b3 100644 --- a/docs/src/upgrade/release-3.6.x.asciidoc +++ b/docs/src/upgrade/release-3.6.x.asciidoc @@ -77,6 +77,23 @@ See: link:https://issues.apache.org/jira/browse/TINKERPOP-2593[TINKERPOP-2593] ==== Graph System Providers +===== Gherkin Tests + +TinkerPop originally introduced Gherkin-based feature tests when GLVs were first introduced to help provide a language +agnostic test capability. The Gherkin tests were a near one-to-one copy of the tests of the Gremlin Process Test Suite +which focus on Gremlin semantics. Unfortunately, having both JVM tests and Gherkin tests meant maintaining two sets +of tests which were testing identical things. + +To simplify the ongoing maintenance of the test suite and to make it even easier to contribute to the enforcement of +Gremlin semantics, TinkerPop now provides infrastructure in the `gremlin-test` module to run the Gherkin-based tests. +For 3.6.0, the old test suite remains intact and is not deprecated, but providers are encouraged to implement the +Gherkin tests as they will include newer tests that may not be in the old test suite and it would be good to gather +feedback on the new test suite's usage so that when deprecation and removal of the old suite comes to pass the +transition will not carry as much friction. + +See: link:https://issues.apache.org/jira/browse/TINKERPOP-2601[TINKERPOP-2601], +link:https://tinkerpop.apache.org/docs/3.6.0/dev/provider/#gherkin-tests-suite[Provider Documentation] + ===== Filters with Mixed Id Types The requirement that "ids" passed to `Graph.vertices` and `Graph.edges` all be of a single type has been removed. This diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/features/StepDefinition.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/features/StepDefinition.java index 3bd136b..b769e21 100644 --- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/features/StepDefinition.java +++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/features/StepDefinition.java @@ -187,9 +187,6 @@ public final class StepDefinition { @Before public void beforeEachScenario(final Scenario scenario) throws Exception { - if (scenario.getName().equals("g_VX1_2_3_4X_name")) - System.out.println("stop!"); - world.beforeEachScenario(scenario); stringParameters.clear(); if (traversal != null) { diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphFeatureTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphFeatureTest.java index 478fee1..3cc834d 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphFeatureTest.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphFeatureTest.java @@ -25,7 +25,6 @@ import com.google.inject.Stage; import io.cucumber.guice.CucumberModules; import io.cucumber.guice.GuiceFactory; import io.cucumber.guice.InjectorSource; -import io.cucumber.java.Scenario; import io.cucumber.junit.Cucumber; import io.cucumber.junit.CucumberOptions; import org.apache.commons.configuration2.BaseConfiguration; @@ -34,7 +33,6 @@ import org.apache.tinkerpop.gremlin.features.World; import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerFactory; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; -import org.junit.AssumptionViolatedException; import org.junit.runner.RunWith; import static org.apache.tinkerpop.gremlin.LoadGraphWith.GraphData;
