TINKERPOP-1985 Changing position on bulk import/export Deprecated BulkLoaderVertexProgram and BulkDumperVertexProgram. BLVP is not replaced, but BDVP has been renamed to CloneVertexProgram which is more aptly named for what it does. Modified docs to discuss all these changes.
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/fe250834 Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/fe250834 Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/fe250834 Branch: refs/heads/TINKERPOP-1987 Commit: fe250834684202c0fbd947da816d172b07d5c9d4 Parents: 7c7dddb Author: Stephen Mallette <sp...@genoprime.com> Authored: Thu Jun 21 15:11:36 2018 -0400 Committer: Stephen Mallette <sp...@genoprime.com> Committed: Thu Jun 21 15:11:36 2018 -0400 ---------------------------------------------------------------------- CHANGELOG.asciidoc | 2 + docs/src/dev/provider/index.asciidoc | 13 +++ .../reference/implementations-spark.asciidoc | 64 ++----------- docs/src/reference/the-graphcomputer.asciidoc | 85 ++++------------- .../upgrade/release-3.2.x-incubating.asciidoc | 27 ++++++ .../tinkerpop/gremlin/jsr223/CoreImports.java | 2 + .../bulkdumping/BulkDumperVertexProgram.java | 3 + .../computer/bulkloading/BulkLoader.java | 2 + .../bulkloading/BulkLoaderVertexProgram.java | 2 + .../bulkloading/IncrementalBulkLoader.java | 2 + .../computer/bulkloading/OneTimeBulkLoader.java | 2 + .../computer/clone/CloneVertexProgram.java | 98 ++++++++++++++++++++ .../gremlin/process/remote/RemoteGraph.java | 4 + .../tinkerpop/gremlin/util/CoreImports.java | 2 + .../AbstractImportCustomizerProvider.java | 2 + .../gremlin/process/ProcessComputerSuite.java | 2 + .../computer/clone/CloneVertexProgramTest.java | 98 ++++++++++++++++++++ 17 files changed, 287 insertions(+), 123 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/CHANGELOG.asciidoc ---------------------------------------------------------------------- diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 05b74da..fb01fac 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -29,6 +29,8 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima * Fixed bug in `branch()` where reducing steps as options would produce incorrect results. * Removed recursive handling of streaming results from Gremlin-Python driver to avoid max recursion depth errors. * Improved performance of `TraversalVertexProgram` and related infrastructure. +* Deprecated `BulkLoaderVertexProgram` and related infrastructure. +* Deprecated `BulkDumperVertexProgram` with the more aptly named `CloneVertexProgram`. * Added `createGratefulDead()`to `TinkerFactory` to help make it easier to try to instantiate that toy graph. * Added identifiers to edges in the Kitchen Sink toy graph. * Refactored the Gremlin Server integration testing framework and streamlined that infrastructure. http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/docs/src/dev/provider/index.asciidoc ---------------------------------------------------------------------- diff --git a/docs/src/dev/provider/index.asciidoc b/docs/src/dev/provider/index.asciidoc index dddcb18..6cb1b22 100644 --- a/docs/src/dev/provider/index.asciidoc +++ b/docs/src/dev/provider/index.asciidoc @@ -620,6 +620,19 @@ something required if there is a need to exploit `withRemote()` in some way. If similar to Gremlin Server that can accept a serialized `Traversal` instance, then that would be one example of a reason to implement this interface. +[bulk-import-export]] +==== Bulk Import Export + +When it comes to doing "bulk" operations, the diverse nature of the available graph databases and their specific +capabilities, prevents TinkerPop from doing a good job of generalizing that capability well. TinkerPop refers users to +the bulk import/export facilities of specific graph providers as they tend to be more efficient and easier to use than +the options TinkerPop has tried to generalize in the past. + +That said, for graph providers that don't have a special bulk loading feature, they can get a basic bulk loader from +TinkerPop using the link:http://tinkerpop.apache.org/docs/x.y.z/reference/#clonevertexprogram[CloneVertexProgram]. +Simply provide a `InputFormat` and `OutputFormat` that can be referenced by a `HadoopGraph` instance as discussed +in the link:http://tinkerpop.apache.org/docs/x.y.z/reference/#clonevertexprogram[Reference Documentation]. + [[validating-with-gremlin-test]] === Validating with Gremlin-Test http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/docs/src/reference/implementations-spark.asciidoc ---------------------------------------------------------------------- diff --git a/docs/src/reference/implementations-spark.asciidoc b/docs/src/reference/implementations-spark.asciidoc index c7a7f24..e265d23 100644 --- a/docs/src/reference/implementations-spark.asciidoc +++ b/docs/src/reference/implementations-spark.asciidoc @@ -123,10 +123,12 @@ references to that Spark Context. The exception to this rule are those propertie Finally, there is a `spark` object that can be used to manage persisted RDDs (see <<interacting-with-spark, Interacting with Spark>>). [[bulkdumpervertexprogramusingspark]] -===== Exporting with BulkDumperVertexProgram +[[clonevertexprogramusingspark]] +===== Using CloneVertexProgram -The <<bulkdumpervertexprogram, BulkDumperVertexProgram>> exports a whole graph in any of the supported Hadoop GraphOutputFormats (`GraphSONOutputFormat`, -`GryoOutputFormat` or `ScriptOutputFormat`). The example below takes a Hadoop graph as the input (in `GryoInputFormat`) and exports it as a GraphSON file +The <<clonevertexprogram, CloneVertexProgram>> copies a whole graph from any graph `InputFormat` to any graph +`OutputFormat`. TinkerPop provides formats such as `GraphSONOutputFormat`, `GryoOutputFormat` or `ScriptOutputFormat`. +The example below takes a Hadoop graph as the input (in `GryoInputFormat`) and exports it as a GraphSON file (`GraphSONOutputFormat`). [gremlin-groovy] @@ -137,58 +139,4 @@ graph.configuration().setProperty('gremlin.hadoop.graphWriter', 'org.apache.tink graph.compute(SparkGraphComputer).program(BulkDumperVertexProgram.build().create()).submit().get() hdfs.ls('output') hdfs.head('output/~g') ----- - -===== Loading with BulkLoaderVertexProgram - -The <<bulkloadervertexprogram, BulkLoaderVertexProgram>> is a generalized bulk loader that can be used to load large -amounts of data to and from different `Graph` implementations. The following code demonstrates how to load the -Grateful Dead graph from HadoopGraph into TinkerGraph over Spark: - -[gremlin-groovy] ----- -hdfs.copyFromLocal('data/grateful-dead.kryo', 'grateful-dead.kryo') -readGraph = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties') -writeGraph = 'conf/tinkergraph-gryo.properties' -blvp = BulkLoaderVertexProgram.build(). - keepOriginalIds(false). - writeGraph(writeGraph).create(readGraph) -readGraph.compute(SparkGraphComputer).workers(1).program(blvp).submit().get() -:set max-iteration 10 -graph = GraphFactory.open(writeGraph) -g = graph.traversal() -g.V().valueMap() -graph.close() ----- - -[source,properties] ----- -# hadoop-grateful-gryo.properties - -# -# Hadoop Graph Configuration -# -gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph -gremlin.hadoop.graphReader=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat -gremlin.hadoop.inputLocation=grateful-dead.kryo -gremlin.hadoop.outputLocation=output -gremlin.hadoop.jarsInDistributedCache=true - -# -# SparkGraphComputer Configuration -# -spark.master=local[1] -spark.executor.memory=1g -spark.serializer=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer ----- - -[source,properties] ----- -# tinkergraph-gryo.properties - -gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph -gremlin.tinkergraph.graphFormat=gryo -gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo ----- - -IMPORTANT: The path to TinkerGraph jars needs to be included in the `HADOOP_GREMLIN_LIBS` for the above example to work. +---- \ No newline at end of file http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/docs/src/reference/the-graphcomputer.asciidoc ---------------------------------------------------------------------- diff --git a/docs/src/reference/the-graphcomputer.asciidoc b/docs/src/reference/the-graphcomputer.asciidoc index 4276f5d..7989c2f 100644 --- a/docs/src/reference/the-graphcomputer.asciidoc +++ b/docs/src/reference/the-graphcomputer.asciidoc @@ -368,71 +368,26 @@ g.V().peerPressure().by(outE('knows')).by('cluster').valueMap() ---- [[bulkdumpervertexprogram]] -=== BulkDumperVertexProgram - -The `BulkDumperVertexProgram` can be used to export a whole graph in any of the provided Hadoop GraphOutputFormats (e.g. -`GraphSONOutputFormat`, `GryoOutputFormat` or `ScriptOutputFormat`). The input can be any Hadoop GraphInputFormat -(e.g. `GraphSONInputFormat`, `GryoInputFormat` or `ScriptInputFormat`). An <<bulkdumpervertexprogramusingspark,example>> -is provided in the SparkGraphComputer section. - -[[bulkloadervertexprogram]] -=== BulkLoaderVertexProgram - -image:batch-graph.png[width=220,float=left] The `BulkLoaderVertexProgram` provides a generalized way for loading -graphs of any size into a persistent `Graph`. It is especially useful for large graphs (i.e. hundreds of millions -or billions of edges) as it can take advantage of the parallel processing offered by `GraphComputer` instances. The -input can be any existing `Graph` database supporting TinkerPop3 or any of the Hadoop GraphInputFormats (e.g. -`GraphSONInputFormat`, `GryoInputFormat` or `ScriptInputFormat`). The following example demonstrates how to load data -from one TinkerGraph to another: - -[gremlin-groovy] ----- -writeGraphConf = new BaseConfiguration() -writeGraphConf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph") -writeGraphConf.setProperty("gremlin.tinkergraph.graphFormat", "gryo") -writeGraphConf.setProperty("gremlin.tinkergraph.graphLocation", "/tmp/tinkergraph.kryo") -modern = TinkerFactory.createModern() -blvp = BulkLoaderVertexProgram.build(). - bulkLoader(OneTimeBulkLoader). - writeGraph(writeGraphConf).create(modern) -modern.compute().workers(1).program(blvp).submit().get() -graph = GraphFactory.open(writeGraphConf) -g = graph.traversal() -g.V().valueMap() -graph.close() ----- - -.Available configuration options -[width="800px",options="header"] -|======================================== -|Builder Method |Purpose | Default Value -|`bulkLoader(Class\|String)` | Sets the class of the bulk loader implementation. | `IncrementalBulkLoader` -|`vertexIdProperty(String)` | Sets the name of the property in the target graph that holds the vertex id from the -source graph. | `bulkLoader.vertex.id` -|`keepOriginalIds(boolean)` |Whether to keep the id's from the source graph in the target graph or not. It's -recommended to keep them if it's planned to do further bulk loads using the same datasources. | `true` -|`userSuppliedIds(boolean)` |Whether to use the id's from the source graph as id's in the target graph. If set to -`true`, `vertexIdProperty` will be ignored. Note, that the target graph must support user supplied identifiers. | `false` -|`intermediateBatchSize(int)` |Sets the batch size for intermediate transactions. This is per thread in a -multi-threaded environment. +0+ means that transactions will only be committed at the end of an iteration cycle. -It's recommended to tune this property for the target graph and not use the default value of +0+. | `0` -|`writeGraph(String)` | Sets the path to a `GraphFactory` compatible configuration file for the target graph. | _none_ -|======================================== - -NOTE: `BulkLoaderVertexProgram` uses the `IncrementalBulkLoader` by default. The other option is the `OneTimeBulkLoader`, -which doesn't store any temporary IDs in the `writeGraph` and thus should only be used for initial bulk loads. Both -implementations should cover the majority of use-cases, but have a limitation though: They don't support multi-valued -properties. `OneTimeBulkLoader` and `IncrementalBulkLoader` will handle every property as a single-valued property. A -custom `BulkLoader` implementation has to be used if the default behavior is not sufficient. - -NOTE: A custom `BulkLoader` implementation for incremental loading should use `GraphTraversal` methods to create/update -elements (e.g. `g.addV()` instead of `graph.addVertex()`). This way the `BulkLoaderVertexProgram` is able to efficiently -track changes in the underlying graph and can apply several optimization techniques. - -WARNING: Edges in the input graph must be present in both directions, e.g. from the source vertex to the target vertex -as an out-edge and from the target vertex to the source vertex as an in-edge. This is especially important if the input -graph is a `HadoopGraph`. `BulkLoaderVertexProgram` will likely fail with a `FastNoSuchElementException` if one of the -edges is missing. +[[clonevertexprogram]] +=== CloneVertexProgram + +The `CloneVertexProgram` (known in versions prior to 3.2.10 as `BulkDumperVertexProgram`) copies a whole graph from +any graph `InputFormat` to any graph `OutputFormat`. TinkerPop provides the following: + +* `OutputFormat` +** `GraphSONOutputFormat` +** `GryoOutputFormat` +** `ScriptOutputFormat` +* `InputFormat` +** `GraphSONInputFormat` +** `GryoInputFormat` +** `ScriptInputFormat`). + +An <<clonevertexprogramusingspark,example>> is provided in the SparkGraphComputer section. + +Graph Providers should consider writing their own `OutputFormat` and `InputFormat` which would allow bulk loading and +export capabilities through this `VertexProgram`. This topic is discussed further in the +link:http://tinkerpop.apache.org/docs/x.y.z/dev/provider/#bulk-import-export[Provider Documentation]. [[traversalvertexprogram]] === TraversalVertexProgram http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/docs/src/upgrade/release-3.2.x-incubating.asciidoc ---------------------------------------------------------------------- diff --git a/docs/src/upgrade/release-3.2.x-incubating.asciidoc b/docs/src/upgrade/release-3.2.x-incubating.asciidoc index 90dd568..af03937 100644 --- a/docs/src/upgrade/release-3.2.x-incubating.asciidoc +++ b/docs/src/upgrade/release-3.2.x-incubating.asciidoc @@ -29,6 +29,19 @@ Please see the link:https://github.com/apache/tinkerpop/blob/3.2.10/CHANGELOG.as === Upgrading for Users +==== Bulk Import and Export + +TinkerPop has provided some general methods for importing and exporting data, but more and more graph providers are +producing their own bulk import/export facilities and they are more efficient and easier to use than TinkerPop's +methods. As a result, TinkerPop will now refer users to the bulk import/export features of individual graph providers +and as such, has deprecated `BulkLoaderVertexProgram` as of this release. + +As part of this change, the `BulkDumperVertexProgram` has been renamed to `CloneVertexProgram` with the former being +deprecated. `CloneVertexProgram` is more aptly named, as it essentially copies a graph from a graph `InputFormat` +to a graph `OutputFormat`. + +link:https://issues.apache.org/jira/browse/TINKERPOP-1985[TINKERPOP-1985] + ==== Docker Images Docker images are now available on link:https://hub.docker.com/u/tinkerpop/[Docker Hub] for Gremlin Console and Gremlin @@ -59,6 +72,20 @@ packaged dataset that needed to be loaded through the various IO options availab benefit of `TinkerFactory` to help get them bootstrapped. For 3.2.10, Grateful Dead is now more conveniently loaded via that same method as the other toy graphs with `TinkerFactory.createGratefulDead()`. +=== Upgrading for Providers + +==== Graph Database Providers + +===== Bulk Import and Export + +As noted in the user section, TinkerPop has deprecated its bulk loading feature in `BulkLoaderVertexProgram` and will +refer TinkerPop users who need bulk import/export capabilities to the native tools of the graph database they have +chosen. If a graph database provider does not have any bulk loading tools it can choose to build graph `InputFormat` +and `OutputFormat` implementations which can be used by `CloneVertexProgram` (formerly `BulkDumperVertexProgram`) as +a easy way to get such a feature. + +link:https://issues.apache.org/jira/browse/TINKERPOP-1985[TINKERPOP-1985] + == TinkerPop 3.2.9 *Release Date: May 8, 2018* http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/jsr223/CoreImports.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/jsr223/CoreImports.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/jsr223/CoreImports.java index e6665e9..adbe766 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/jsr223/CoreImports.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/jsr223/CoreImports.java @@ -41,6 +41,7 @@ import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoader; import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.bulkloading.IncrementalBulkLoader; import org.apache.tinkerpop.gremlin.process.computer.bulkloading.OneTimeBulkLoader; +import org.apache.tinkerpop.gremlin.process.computer.clone.CloneVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.clustering.peerpressure.ClusterCountMapReduce; import org.apache.tinkerpop.gremlin.process.computer.clustering.peerpressure.ClusterPopulationMapReduce; import org.apache.tinkerpop.gremlin.process.computer.clustering.peerpressure.PeerPressureVertexProgram; @@ -246,6 +247,7 @@ public final class CoreImports { CLASS_IMPORTS.add(GraphComputer.class); CLASS_IMPORTS.add(Memory.class); CLASS_IMPORTS.add(VertexProgram.class); + CLASS_IMPORTS.add(CloneVertexProgram.class); CLASS_IMPORTS.add(BulkDumperVertexProgram.class); CLASS_IMPORTS.add(BulkLoader.class); CLASS_IMPORTS.add(BulkLoaderVertexProgram.class); http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkdumping/BulkDumperVertexProgram.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkdumping/BulkDumperVertexProgram.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkdumping/BulkDumperVertexProgram.java index fcbdc6c..f70fcc2 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkdumping/BulkDumperVertexProgram.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkdumping/BulkDumperVertexProgram.java @@ -23,6 +23,7 @@ import org.apache.tinkerpop.gremlin.process.computer.Memory; import org.apache.tinkerpop.gremlin.process.computer.MessageScope; import org.apache.tinkerpop.gremlin.process.computer.Messenger; import org.apache.tinkerpop.gremlin.process.computer.VertexProgram; +import org.apache.tinkerpop.gremlin.process.computer.clone.CloneVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.util.AbstractVertexProgramBuilder; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.Vertex; @@ -34,7 +35,9 @@ import java.util.Set; /** * @author Daniel Kuppitz (http://gremlin.guru) + * @deprecated As of release 3.2.10, replaced by {@link CloneVertexProgram}. */ +@Deprecated public class BulkDumperVertexProgram implements VertexProgram<Tuple> { private BulkDumperVertexProgram() { http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java index fe1fb77..3767f73 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java @@ -27,7 +27,9 @@ import org.apache.tinkerpop.gremlin.structure.VertexProperty; /** * @author Daniel Kuppitz (http://gremlin.guru) + * @deprecated As of release 3.2.10, not directly replaced - consider graph provider specific bulk loading methods */ +@Deprecated public interface BulkLoader { /** http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java index 508af3e..1e75ce6 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java @@ -56,7 +56,9 @@ import java.util.Set; /** * @author Daniel Kuppitz (http://gremlin.guru) + * @deprecated As of release 3.2.10, not directly replaced - consider graph provider specific bulk loading methods */ +@Deprecated public class BulkLoaderVertexProgram implements VertexProgram<Tuple> { private static final Logger LOGGER = LoggerFactory.getLogger(BulkLoaderVertexProgram.class); http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java index 1e8a3d1..22c6999 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java @@ -33,7 +33,9 @@ import java.util.Iterator; /** * @author Daniel Kuppitz (http://gremlin.guru) + * @deprecated As of release 3.2.10, not directly replaced - consider graph provider specific bulk loading methods */ +@Deprecated public class IncrementalBulkLoader implements BulkLoader { private String bulkLoaderVertexId = BulkLoaderVertexProgram.DEFAULT_BULK_LOADER_VERTEX_ID; http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/OneTimeBulkLoader.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/OneTimeBulkLoader.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/OneTimeBulkLoader.java index cf0528c..ef34fce 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/OneTimeBulkLoader.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/OneTimeBulkLoader.java @@ -35,7 +35,9 @@ import org.apache.tinkerpop.gremlin.structure.VertexProperty; * only clones each element from the source graph. * * @author Daniel Kuppitz (http://gremlin.guru) + * @deprecated As of release 3.2.10, not directly replaced - consider graph provider specific bulk loading methods */ +@Deprecated public class OneTimeBulkLoader implements BulkLoader { private boolean userSuppliedIds = false; http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/clone/CloneVertexProgram.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/clone/CloneVertexProgram.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/clone/CloneVertexProgram.java new file mode 100644 index 0000000..6afa0e5 --- /dev/null +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/clone/CloneVertexProgram.java @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.process.computer.clone; + +import org.apache.tinkerpop.gremlin.process.computer.GraphComputer; +import org.apache.tinkerpop.gremlin.process.computer.Memory; +import org.apache.tinkerpop.gremlin.process.computer.MessageScope; +import org.apache.tinkerpop.gremlin.process.computer.Messenger; +import org.apache.tinkerpop.gremlin.process.computer.VertexProgram; +import org.apache.tinkerpop.gremlin.process.computer.util.AbstractVertexProgramBuilder; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.util.StringFactory; +import org.javatuples.Tuple; + +import java.util.Collections; +import java.util.Set; + +/** + * @author Daniel Kuppitz (http://gremlin.guru) + */ +public class CloneVertexProgram implements VertexProgram<Tuple> { + + private CloneVertexProgram() { + } + + @Override + public void setup(final Memory memory) { + } + + @Override + public void execute(final Vertex sourceVertex, final Messenger<Tuple> messenger, final Memory memory) { + } + + @Override + public boolean terminate(final Memory memory) { + return true; + } + + @Override + public Set<MessageScope> getMessageScopes(final Memory memory) { + return Collections.emptySet(); + } + + @SuppressWarnings({"CloneDoesntDeclareCloneNotSupportedException", "CloneDoesntCallSuperClone"}) + @Override + public VertexProgram<Tuple> clone() { + return this; + } + + @Override + public GraphComputer.ResultGraph getPreferredResultGraph() { + return GraphComputer.ResultGraph.NEW; + } + + @Override + public GraphComputer.Persist getPreferredPersist() { + return GraphComputer.Persist.EDGES; + } + + @Override + public String toString() { + return StringFactory.vertexProgramString(this); + } + + public static Builder build() { + return new Builder(); + } + + public static class Builder extends AbstractVertexProgramBuilder<Builder> { + + private Builder() { + super(CloneVertexProgram.class); + } + + @SuppressWarnings("unchecked") + @Override + public CloneVertexProgram create(final Graph graph) { + return (CloneVertexProgram) VertexProgram.createVertexProgram(graph, configuration); + } + } +} http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/RemoteGraph.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/RemoteGraph.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/RemoteGraph.java index d69e12e..8a1ab02 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/RemoteGraph.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/RemoteGraph.java @@ -105,6 +105,10 @@ import java.util.Iterator; method = "*", reason = "RemoteGraph does not support direct Graph.compute() access") @Graph.OptOut( + test = "org.apache.tinkerpop.gremlin.process.computer.clone.CloneVertexProgramTest", + method = "*", + reason = "RemoteGraph does not support direct Graph.compute() access") +@Graph.OptOut( test = "org.apache.tinkerpop.gremlin.process.computer.GraphComputerTest", method = "*", reason = "RemoteGraph does not support direct Graph.compute() access") http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/util/CoreImports.java ---------------------------------------------------------------------- diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/util/CoreImports.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/util/CoreImports.java index 597facf..ef6f5e2 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/util/CoreImports.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/util/CoreImports.java @@ -41,6 +41,7 @@ import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoader; import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.bulkloading.IncrementalBulkLoader; import org.apache.tinkerpop.gremlin.process.computer.bulkloading.OneTimeBulkLoader; +import org.apache.tinkerpop.gremlin.process.computer.clone.CloneVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.clustering.peerpressure.ClusterCountMapReduce; import org.apache.tinkerpop.gremlin.process.computer.clustering.peerpressure.ClusterPopulationMapReduce; import org.apache.tinkerpop.gremlin.process.computer.clustering.peerpressure.PeerPressureVertexProgram; @@ -209,6 +210,7 @@ public final class CoreImports { CLASS_IMPORTS.add(GraphComputer.class); CLASS_IMPORTS.add(Memory.class); CLASS_IMPORTS.add(VertexProgram.class); + CLASS_IMPORTS.add(CloneVertexProgram.class); CLASS_IMPORTS.add(BulkDumperVertexProgram.class); CLASS_IMPORTS.add(BulkLoader.class); CLASS_IMPORTS.add(BulkLoaderVertexProgram.class); http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/AbstractImportCustomizerProvider.java ---------------------------------------------------------------------- diff --git a/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/AbstractImportCustomizerProvider.java b/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/AbstractImportCustomizerProvider.java index efc4d3d..a35df18 100644 --- a/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/AbstractImportCustomizerProvider.java +++ b/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/AbstractImportCustomizerProvider.java @@ -28,6 +28,7 @@ import org.apache.tinkerpop.gremlin.process.computer.Computer; import org.apache.tinkerpop.gremlin.process.computer.GraphComputer; import org.apache.tinkerpop.gremlin.process.computer.bulkdumping.BulkDumperVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram; +import org.apache.tinkerpop.gremlin.process.computer.clone.CloneVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.clustering.peerpressure.PeerPressureVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram; import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram; @@ -149,6 +150,7 @@ public abstract class AbstractImportCustomizerProvider implements ImportCustomiz imports.add(TraversalVertexProgram.class.getPackage().getName() + DOT_STAR); imports.add(BulkLoaderVertexProgram.class.getPackage().getName() + DOT_STAR); imports.add(BulkDumperVertexProgram.class.getPackage().getName() + DOT_STAR); + imports.add(CloneVertexProgram.class.getPackage().getName() + DOT_STAR); // groovy extras imports.add(Grape.class.getCanonicalName()); http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessComputerSuite.java ---------------------------------------------------------------------- diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessComputerSuite.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessComputerSuite.java index 1d69a76..204d038 100644 --- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessComputerSuite.java +++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessComputerSuite.java @@ -23,6 +23,7 @@ import org.apache.tinkerpop.gremlin.process.computer.GraphComputer; import org.apache.tinkerpop.gremlin.process.computer.GraphComputerTest; import org.apache.tinkerpop.gremlin.process.computer.bulkdumping.BulkDumperVertexProgramTest; import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgramTest; +import org.apache.tinkerpop.gremlin.process.computer.clone.CloneVertexProgramTest; import org.apache.tinkerpop.gremlin.process.computer.clustering.peerpressure.PeerPressureVertexProgramTest; import org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgramTest; import org.apache.tinkerpop.gremlin.process.traversal.TraversalEngine; @@ -194,6 +195,7 @@ public class ProcessComputerSuite extends AbstractGremlinSuite { PeerPressureVertexProgramTest.class, BulkLoaderVertexProgramTest.class, BulkDumperVertexProgramTest.class, + CloneVertexProgramTest.class, // creations TranslationStrategyProcessTest.class, http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/fe250834/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/clone/CloneVertexProgramTest.java ---------------------------------------------------------------------- diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/clone/CloneVertexProgramTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/clone/CloneVertexProgramTest.java new file mode 100644 index 0000000..760a465 --- /dev/null +++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/clone/CloneVertexProgramTest.java @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.process.computer.clone; + +import org.apache.tinkerpop.gremlin.LoadGraphWith; +import org.apache.tinkerpop.gremlin.process.AbstractGremlinProcessTest; +import org.apache.tinkerpop.gremlin.process.computer.ComputerResult; +import org.apache.tinkerpop.gremlin.process.computer.GraphComputer; +import org.apache.tinkerpop.gremlin.structure.Direction; +import org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils; +import org.junit.Test; + +import static org.apache.tinkerpop.gremlin.LoadGraphWith.GraphData.MODERN; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * @author Daniel Kuppitz (http://gremlin.guru) + */ +public class CloneVertexProgramTest extends AbstractGremlinProcessTest { + + @Test + @LoadGraphWith(MODERN) + public void shouldDumpWholeGraph() throws Exception { + if (graphProvider.getGraphComputer(graph).features().supportsResultGraphPersistCombination(GraphComputer.ResultGraph.NEW, GraphComputer.Persist.EDGES)) { + final ComputerResult result = graphProvider.getGraphComputer(graph).program(CloneVertexProgram.build().create(graph)).submit().get(); + result.graph().traversal().V().forEachRemaining(v -> { + assertEquals(2, v.keys().size()); + assertTrue(v.keys().contains("name")); + assertTrue(v.keys().contains("age") || v.keys().contains("lang")); + assertEquals(1, IteratorUtils.count(v.values("name"))); + assertEquals(1, IteratorUtils.count(v.values("age", "lang"))); + final String name = v.value("name"); + if (name.equals("marko")) { + assertEquals("person", v.label()); + assertEquals(Integer.valueOf(29), v.value("age")); + assertEquals(3, IteratorUtils.count(v.edges(Direction.OUT))); + assertEquals(2, IteratorUtils.count(v.edges(Direction.OUT, "knows"))); + assertEquals(1, IteratorUtils.count(v.edges(Direction.OUT, "created"))); + assertEquals(0, IteratorUtils.count(v.edges(Direction.IN))); + } else if (name.equals("vadas")) { + assertEquals("person", v.label()); + assertEquals(Integer.valueOf(27), v.value("age")); + assertEquals(0, IteratorUtils.count(v.edges(Direction.OUT))); + assertEquals(1, IteratorUtils.count(v.edges(Direction.IN))); + assertEquals(1, IteratorUtils.count(v.edges(Direction.IN, "knows"))); + } else if (name.equals("lop")) { + assertEquals("software", v.label()); + assertEquals("java", v.value("lang")); + assertEquals(0, IteratorUtils.count(v.edges(Direction.OUT))); + assertEquals(3, IteratorUtils.count(v.edges(Direction.IN))); + assertEquals(3, IteratorUtils.count(v.edges(Direction.IN, "created"))); + } else if (name.equals("josh")) { + assertEquals("person", v.label()); + assertEquals(Integer.valueOf(32), v.value("age")); + assertEquals(2, IteratorUtils.count(v.edges(Direction.OUT))); + assertEquals(2, IteratorUtils.count(v.edges(Direction.OUT, "created"))); + assertEquals(1, IteratorUtils.count(v.edges(Direction.IN))); + assertEquals(1, IteratorUtils.count(v.edges(Direction.IN, "knows"))); + } else if (name.equals("ripple")) { + assertEquals("software", v.label()); + assertEquals("java", v.value("lang")); + assertEquals(0, IteratorUtils.count(v.edges(Direction.OUT))); + assertEquals(1, IteratorUtils.count(v.edges(Direction.IN))); + assertEquals(1, IteratorUtils.count(v.edges(Direction.IN, "created"))); + } else if (name.equals("peter")) { + assertEquals("person", v.label()); + assertEquals(Integer.valueOf(35), v.value("age")); + assertEquals(1, IteratorUtils.count(v.edges(Direction.OUT))); + assertEquals(1, IteratorUtils.count(v.edges(Direction.OUT, "created"))); + assertEquals(0, IteratorUtils.count(v.edges(Direction.IN))); + } else + throw new IllegalStateException("The following vertex should not exist in the graph: " + name); + }); + assertEquals(3.5, (Double) result.graph().traversal().E().values("weight").sum().next(), 0.01); + assertEquals(1.5, (Double) result.graph().traversal().E().hasLabel("knows").values("weight").sum().next(), 0.01); + assertEquals(2.0, (Double) result.graph().traversal().E().hasLabel("created").values("weight").sum().next(), 0.01); + assertEquals(result.memory().getIteration(), 0); + assertEquals(result.memory().asMap().size(), 0); + } + } +} \ No newline at end of file
