[1/2] tinkerpop git commit: TINKERPOP-1958 Fixed a bug in TinkerGraphCountStrategy
Repository: tinkerpop Updated Branches: refs/heads/tp32 bd85e5feb -> 7c7001394 TINKERPOP-1958 Fixed a bug in TinkerGraphCountStrategy The strategy did not consider, that certain map steps may not emit an element. Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/2407739b Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/2407739b Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/2407739b Branch: refs/heads/tp32 Commit: 2407739b2e7b3624478bd9b5dbc55c9c28f1da28 Parents: 288b455 Author: Daniel Kuppitz Authored: Wed May 9 07:53:24 2018 -0700 Committer: Daniel Kuppitz Committed: Fri May 11 18:18:53 2018 -0700 -- CHANGELOG.asciidoc | 1 + .../traversal/step/map/GroovySelectTest.groovy | 5 gremlin-test/features/map/Select.feature| 22 - .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 6 files changed, 58 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/CHANGELOG.asciidoc -- diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 41d83c4..23e130e 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -23,6 +23,7 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima [[release-3-2-10]] === TinkerPop 3.2.10 (Release Date: NOT OFFICIALLY RELEASED YET) +* Fixed a bug in `TinkerGraphCountStrategy`, which didn't consider that certain map steps may not emit an element. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018) http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy -- diff --git a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy index 89fc691..104322d 100644 --- a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy +++ b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy @@ -104,6 +104,11 @@ public abstract class GroovySelectTest { new ScriptTraversal<>(g, "gremlin-groovy", "g.V.choose(__.outE().count().is(0L), __.as('a'), __.as('b')).choose(select('a'),select('a'),select('b'))") } +@Override +public Traversal get_g_V_selectXaX_count() { +new ScriptTraversal<>(g, "gremlin-groovy", "g.V.select('a').count") +} + // below are original back()-tests @Override http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-test/features/map/Select.feature -- diff --git a/gremlin-test/features/map/Select.feature b/gremlin-test/features/map/Select.feature index 35d9322..1e45a0e 100644 --- a/gremlin-test/features/map/Select.feature +++ b/gremlin-test/features/map/Select.feature @@ -514,4 +514,24 @@ Feature: Step - select() Then the result should be unordered | result | | d[2].l | - | d[2].l | \ No newline at end of file + | d[2].l | + + Scenario: g_V_selectXaX +Given the modern graph +And the traversal of + """ + g.V().select("a") + """ +When iterated to list +Then the result should be empty + + Scenario: g_V_selectXaX_count +Given the modern graph +And the traversal of + """ + g.V().select("a").count() + """ +When iterated to list +Then the result should be unordered + | result | + | d[0].l | http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java -- diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java index c0486d0..3d778e4 100644 --- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java +++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java @@ -83,6 +83,8 @@ public abstract class SelectTest extends Abstract
[2/2] tinkerpop git commit: Merge branch 'TINKERPOP-1958' into tp32
Merge branch 'TINKERPOP-1958' into tp32 Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/7c700139 Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/7c700139 Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/7c700139 Branch: refs/heads/tp32 Commit: 7c7001394a3dd5706c5ff38bcbf5c96dd6c611fc Parents: bd85e5f 2407739 Author: Daniel Kuppitz Authored: Tue May 22 08:04:24 2018 -0700 Committer: Daniel Kuppitz Committed: Tue May 22 08:04:24 2018 -0700 -- CHANGELOG.asciidoc | 1 + .../traversal/step/map/GroovySelectTest.groovy | 5 gremlin-test/features/map/Select.feature| 22 - .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 6 files changed, 58 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/7c700139/CHANGELOG.asciidoc -- diff --cc CHANGELOG.asciidoc index 395bb55,23e130e..3d33c78 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@@ -23,10 -23,7 +23,11 @@@ image::https://raw.githubusercontent.co [[release-3-2-10]] === TinkerPop 3.2.10 (Release Date: NOT OFFICIALLY RELEASED YET) +* Removed recursive handling of streaming results from Gremlin-Python driver to avoid max recursion depth errors. +* Improved performance of `TraversalVertexProgram` and related infrastructure. +* Fixed bug in `GroovyTranslator` that didn't properly handle empty `Map` objects. +* Added concrete configuration methods to `SparkGraphComputer` to make a more clear API for configuring it. + * Fixed a bug in `TinkerGraphCountStrategy`, which didn't consider that certain map steps may not emit an element. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018)
[2/4] tinkerpop git commit: Merge branch 'TINKERPOP-1958' into tp32
Merge branch 'TINKERPOP-1958' into tp32 Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/7c700139 Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/7c700139 Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/7c700139 Branch: refs/heads/master Commit: 7c7001394a3dd5706c5ff38bcbf5c96dd6c611fc Parents: bd85e5f 2407739 Author: Daniel Kuppitz Authored: Tue May 22 08:04:24 2018 -0700 Committer: Daniel Kuppitz Committed: Tue May 22 08:04:24 2018 -0700 -- CHANGELOG.asciidoc | 1 + .../traversal/step/map/GroovySelectTest.groovy | 5 gremlin-test/features/map/Select.feature| 22 - .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 6 files changed, 58 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/7c700139/CHANGELOG.asciidoc -- diff --cc CHANGELOG.asciidoc index 395bb55,23e130e..3d33c78 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@@ -23,10 -23,7 +23,11 @@@ image::https://raw.githubusercontent.co [[release-3-2-10]] === TinkerPop 3.2.10 (Release Date: NOT OFFICIALLY RELEASED YET) +* Removed recursive handling of streaming results from Gremlin-Python driver to avoid max recursion depth errors. +* Improved performance of `TraversalVertexProgram` and related infrastructure. +* Fixed bug in `GroovyTranslator` that didn't properly handle empty `Map` objects. +* Added concrete configuration methods to `SparkGraphComputer` to make a more clear API for configuring it. + * Fixed a bug in `TinkerGraphCountStrategy`, which didn't consider that certain map steps may not emit an element. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018)
[3/3] tinkerpop git commit: Merge branch 'tp32' into tp33
Merge branch 'tp32' into tp33 Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/d975e191 Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/d975e191 Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/d975e191 Branch: refs/heads/tp33 Commit: d975e1917272ea12c019b77f72953345fa59036f Parents: 3891777 7c70013 Author: Daniel Kuppitz Authored: Tue May 22 08:44:12 2018 -0700 Committer: Daniel Kuppitz Committed: Tue May 22 08:44:12 2018 -0700 -- CHANGELOG.asciidoc | 1 + gremlin-test/features/map/Select.feature| 20 .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 5 files changed, 52 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/d975e191/CHANGELOG.asciidoc -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/d975e191/gremlin-test/features/map/Select.feature -- diff --cc gremlin-test/features/map/Select.feature index 341a378,1e45a0e..788eda7 --- a/gremlin-test/features/map/Select.feature +++ b/gremlin-test/features/map/Select.feature @@@ -516,46 -516,22 +516,66 @@@ Feature: Step - select( | d[2].l | | d[2].l | + Scenario: g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX +Given the modern graph +And the traversal of + """ + g.V().as("a").group("m").by().by(__.bothE().count()).barrier().select("m").select(__.select("a")) + """ +When iterated to list +Then the result should be unordered + | result | + | d[3].l | + | d[1].l | + | d[3].l | + | d[3].l | + | d[1].l | + | d[1].l | + + Scenario: g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX_byXmathX_plus_XX +Given the modern graph +And the traversal of + """ + g.V().as("a").group("m").by().by(__.bothE().count()).barrier().select("m").select(__.select("a")).by(__.math("_+_")) + """ +When iterated to list +Then the result should be unordered + | result | + | d[6].d | + | d[2].d | + | d[6].d | + | d[6].d | + | d[2].d | + | d[2].d | + + Scenario: g_V_asXaX_outXknowsX_asXaX_selectXall_constantXaXX +Given the modern graph +And the traversal of + """ + g.V().as("a").out("knows").as("a").select(Pop.all, __.constant("a")) + """ +When iterated to list +Then the result should be unordered + | result | + | l[v[marko],v[vadas]] | + | l[v[marko],v[josh]] | ++ + Scenario: g_V_selectXaX + Given the modern graph + And the traversal of + """ + g.V().select("a") + """ + When iterated to list + Then the result should be empty + + Scenario: g_V_selectXaX_count + Given the modern graph + And the traversal of + """ + g.V().select("a").count() + """ + When iterated to list + Then the result should be unordered + | result | + | d[0].l | http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/d975e191/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java -- diff --cc gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java index 72a1872,3d778e4..20eee0f --- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java +++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java @@@ -83,12 -83,8 +83,14 @@@ public abstract class SelectTest extend public abstract Traversal get_g_V_chooseXoutE_count_isX0X__asXaX__asXbXX_chooseXselectXaX__selectXaX__selectXbXX(); +public abstract Traversal get_g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX(); + +public abstract Traversal get_g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX_byXmathX_plus_XX(); + +public abstract Traversal> get_g_V_asXaX_outXknowsX_asXaX_selectXall_constantXaXX(); + + public abstract Traversal get_g_V_selectXaX_count(); + // below are original back()-tests public abstract Traversal get_g_VX1X_asXhereX_out_selectXhereX(final Object v1Id); @@@ -349,31 -348,12 +351,39 @@@ @Test @LoadGraphWith(MODERN) +public void g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX()
[2/3] tinkerpop git commit: Merge branch 'TINKERPOP-1958' into tp32
Merge branch 'TINKERPOP-1958' into tp32 Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/7c700139 Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/7c700139 Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/7c700139 Branch: refs/heads/tp33 Commit: 7c7001394a3dd5706c5ff38bcbf5c96dd6c611fc Parents: bd85e5f 2407739 Author: Daniel Kuppitz Authored: Tue May 22 08:04:24 2018 -0700 Committer: Daniel Kuppitz Committed: Tue May 22 08:04:24 2018 -0700 -- CHANGELOG.asciidoc | 1 + .../traversal/step/map/GroovySelectTest.groovy | 5 gremlin-test/features/map/Select.feature| 22 - .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 6 files changed, 58 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/7c700139/CHANGELOG.asciidoc -- diff --cc CHANGELOG.asciidoc index 395bb55,23e130e..3d33c78 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@@ -23,10 -23,7 +23,11 @@@ image::https://raw.githubusercontent.co [[release-3-2-10]] === TinkerPop 3.2.10 (Release Date: NOT OFFICIALLY RELEASED YET) +* Removed recursive handling of streaming results from Gremlin-Python driver to avoid max recursion depth errors. +* Improved performance of `TraversalVertexProgram` and related infrastructure. +* Fixed bug in `GroovyTranslator` that didn't properly handle empty `Map` objects. +* Added concrete configuration methods to `SparkGraphComputer` to make a more clear API for configuring it. + * Fixed a bug in `TinkerGraphCountStrategy`, which didn't consider that certain map steps may not emit an element. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018)
[4/4] tinkerpop git commit: Merge branch 'tp33'
Merge branch 'tp33' Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/f3936334 Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/f3936334 Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/f3936334 Branch: refs/heads/master Commit: f39363347ec09d21475757b45b154b43d05a85d8 Parents: 4f46095 d975e19 Author: Daniel Kuppitz Authored: Tue May 22 08:44:31 2018 -0700 Committer: Daniel Kuppitz Committed: Tue May 22 08:44:31 2018 -0700 -- CHANGELOG.asciidoc | 1 + gremlin-test/features/map/Select.feature| 20 .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 5 files changed, 52 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/f3936334/CHANGELOG.asciidoc -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/f3936334/gremlin-test/features/map/Select.feature -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/f3936334/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java --
[1/3] tinkerpop git commit: TINKERPOP-1958 Fixed a bug in TinkerGraphCountStrategy
Repository: tinkerpop Updated Branches: refs/heads/tp33 3891777e4 -> d975e1917 TINKERPOP-1958 Fixed a bug in TinkerGraphCountStrategy The strategy did not consider, that certain map steps may not emit an element. Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/2407739b Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/2407739b Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/2407739b Branch: refs/heads/tp33 Commit: 2407739b2e7b3624478bd9b5dbc55c9c28f1da28 Parents: 288b455 Author: Daniel Kuppitz Authored: Wed May 9 07:53:24 2018 -0700 Committer: Daniel Kuppitz Committed: Fri May 11 18:18:53 2018 -0700 -- CHANGELOG.asciidoc | 1 + .../traversal/step/map/GroovySelectTest.groovy | 5 gremlin-test/features/map/Select.feature| 22 - .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 6 files changed, 58 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/CHANGELOG.asciidoc -- diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 41d83c4..23e130e 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -23,6 +23,7 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima [[release-3-2-10]] === TinkerPop 3.2.10 (Release Date: NOT OFFICIALLY RELEASED YET) +* Fixed a bug in `TinkerGraphCountStrategy`, which didn't consider that certain map steps may not emit an element. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018) http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy -- diff --git a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy index 89fc691..104322d 100644 --- a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy +++ b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy @@ -104,6 +104,11 @@ public abstract class GroovySelectTest { new ScriptTraversal<>(g, "gremlin-groovy", "g.V.choose(__.outE().count().is(0L), __.as('a'), __.as('b')).choose(select('a'),select('a'),select('b'))") } +@Override +public Traversal get_g_V_selectXaX_count() { +new ScriptTraversal<>(g, "gremlin-groovy", "g.V.select('a').count") +} + // below are original back()-tests @Override http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-test/features/map/Select.feature -- diff --git a/gremlin-test/features/map/Select.feature b/gremlin-test/features/map/Select.feature index 35d9322..1e45a0e 100644 --- a/gremlin-test/features/map/Select.feature +++ b/gremlin-test/features/map/Select.feature @@ -514,4 +514,24 @@ Feature: Step - select() Then the result should be unordered | result | | d[2].l | - | d[2].l | \ No newline at end of file + | d[2].l | + + Scenario: g_V_selectXaX +Given the modern graph +And the traversal of + """ + g.V().select("a") + """ +When iterated to list +Then the result should be empty + + Scenario: g_V_selectXaX_count +Given the modern graph +And the traversal of + """ + g.V().select("a").count() + """ +When iterated to list +Then the result should be unordered + | result | + | d[0].l | http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java -- diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java index c0486d0..3d778e4 100644 --- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java +++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java @@ -83,6 +83,8 @@ public abstract class SelectTest extends Abstract
[3/4] tinkerpop git commit: Merge branch 'tp32' into tp33
Merge branch 'tp32' into tp33 Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/d975e191 Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/d975e191 Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/d975e191 Branch: refs/heads/master Commit: d975e1917272ea12c019b77f72953345fa59036f Parents: 3891777 7c70013 Author: Daniel Kuppitz Authored: Tue May 22 08:44:12 2018 -0700 Committer: Daniel Kuppitz Committed: Tue May 22 08:44:12 2018 -0700 -- CHANGELOG.asciidoc | 1 + gremlin-test/features/map/Select.feature| 20 .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 5 files changed, 52 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/d975e191/CHANGELOG.asciidoc -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/d975e191/gremlin-test/features/map/Select.feature -- diff --cc gremlin-test/features/map/Select.feature index 341a378,1e45a0e..788eda7 --- a/gremlin-test/features/map/Select.feature +++ b/gremlin-test/features/map/Select.feature @@@ -516,46 -516,22 +516,66 @@@ Feature: Step - select( | d[2].l | | d[2].l | + Scenario: g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX +Given the modern graph +And the traversal of + """ + g.V().as("a").group("m").by().by(__.bothE().count()).barrier().select("m").select(__.select("a")) + """ +When iterated to list +Then the result should be unordered + | result | + | d[3].l | + | d[1].l | + | d[3].l | + | d[3].l | + | d[1].l | + | d[1].l | + + Scenario: g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX_byXmathX_plus_XX +Given the modern graph +And the traversal of + """ + g.V().as("a").group("m").by().by(__.bothE().count()).barrier().select("m").select(__.select("a")).by(__.math("_+_")) + """ +When iterated to list +Then the result should be unordered + | result | + | d[6].d | + | d[2].d | + | d[6].d | + | d[6].d | + | d[2].d | + | d[2].d | + + Scenario: g_V_asXaX_outXknowsX_asXaX_selectXall_constantXaXX +Given the modern graph +And the traversal of + """ + g.V().as("a").out("knows").as("a").select(Pop.all, __.constant("a")) + """ +When iterated to list +Then the result should be unordered + | result | + | l[v[marko],v[vadas]] | + | l[v[marko],v[josh]] | ++ + Scenario: g_V_selectXaX + Given the modern graph + And the traversal of + """ + g.V().select("a") + """ + When iterated to list + Then the result should be empty + + Scenario: g_V_selectXaX_count + Given the modern graph + And the traversal of + """ + g.V().select("a").count() + """ + When iterated to list + Then the result should be unordered + | result | + | d[0].l | http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/d975e191/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java -- diff --cc gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java index 72a1872,3d778e4..20eee0f --- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java +++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java @@@ -83,12 -83,8 +83,14 @@@ public abstract class SelectTest extend public abstract Traversal get_g_V_chooseXoutE_count_isX0X__asXaX__asXbXX_chooseXselectXaX__selectXaX__selectXbXX(); +public abstract Traversal get_g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX(); + +public abstract Traversal get_g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX_byXmathX_plus_XX(); + +public abstract Traversal> get_g_V_asXaX_outXknowsX_asXaX_selectXall_constantXaXX(); + + public abstract Traversal get_g_V_selectXaX_count(); + // below are original back()-tests public abstract Traversal get_g_VX1X_asXhereX_out_selectXhereX(final Object v1Id); @@@ -349,31 -348,12 +351,39 @@@ @Test @LoadGraphWith(MODERN) +public void g_V_asXaX_groupXmX_by_byXbothE_countX_barrier_selectXmX_selectXselectXaXX(
[1/4] tinkerpop git commit: TINKERPOP-1958 Fixed a bug in TinkerGraphCountStrategy
Repository: tinkerpop Updated Branches: refs/heads/master 4f46095ba -> f39363347 TINKERPOP-1958 Fixed a bug in TinkerGraphCountStrategy The strategy did not consider, that certain map steps may not emit an element. Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/2407739b Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/2407739b Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/2407739b Branch: refs/heads/master Commit: 2407739b2e7b3624478bd9b5dbc55c9c28f1da28 Parents: 288b455 Author: Daniel Kuppitz Authored: Wed May 9 07:53:24 2018 -0700 Committer: Daniel Kuppitz Committed: Fri May 11 18:18:53 2018 -0700 -- CHANGELOG.asciidoc | 1 + .../traversal/step/map/GroovySelectTest.groovy | 5 gremlin-test/features/map/Select.feature| 22 - .../process/traversal/step/map/SelectTest.java | 15 .../optimization/TinkerGraphCountStrategy.java | 2 +- .../TinkerGraphCountStrategyTest.java | 25 6 files changed, 58 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/CHANGELOG.asciidoc -- diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 41d83c4..23e130e 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -23,6 +23,7 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima [[release-3-2-10]] === TinkerPop 3.2.10 (Release Date: NOT OFFICIALLY RELEASED YET) +* Fixed a bug in `TinkerGraphCountStrategy`, which didn't consider that certain map steps may not emit an element. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018) http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy -- diff --git a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy index 89fc691..104322d 100644 --- a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy +++ b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/map/GroovySelectTest.groovy @@ -104,6 +104,11 @@ public abstract class GroovySelectTest { new ScriptTraversal<>(g, "gremlin-groovy", "g.V.choose(__.outE().count().is(0L), __.as('a'), __.as('b')).choose(select('a'),select('a'),select('b'))") } +@Override +public Traversal get_g_V_selectXaX_count() { +new ScriptTraversal<>(g, "gremlin-groovy", "g.V.select('a').count") +} + // below are original back()-tests @Override http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-test/features/map/Select.feature -- diff --git a/gremlin-test/features/map/Select.feature b/gremlin-test/features/map/Select.feature index 35d9322..1e45a0e 100644 --- a/gremlin-test/features/map/Select.feature +++ b/gremlin-test/features/map/Select.feature @@ -514,4 +514,24 @@ Feature: Step - select() Then the result should be unordered | result | | d[2].l | - | d[2].l | \ No newline at end of file + | d[2].l | + + Scenario: g_V_selectXaX +Given the modern graph +And the traversal of + """ + g.V().select("a") + """ +When iterated to list +Then the result should be empty + + Scenario: g_V_selectXaX_count +Given the modern graph +And the traversal of + """ + g.V().select("a").count() + """ +When iterated to list +Then the result should be unordered + | result | + | d[0].l | http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/2407739b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java -- diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java index c0486d0..3d778e4 100644 --- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java +++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SelectTest.java @@ -83,6 +83,8 @@ public abstract class SelectTest extends Abst
tinkerpop git commit: TINKERPOP-1113 Added spark configuration options as concrete methods CTR
Repository: tinkerpop Updated Branches: refs/heads/tp32 f36eb4f34 -> bd85e5feb TINKERPOP-1113 Added spark configuration options as concrete methods CTR Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/bd85e5fe Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/bd85e5fe Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/bd85e5fe Branch: refs/heads/tp32 Commit: bd85e5febee56434c4de4e7ab31e337a9f5e Parents: f36eb4f Author: Stephen Mallette Authored: Tue May 22 06:55:46 2018 -0400 Committer: Stephen Mallette Committed: Tue May 22 06:55:46 2018 -0400 -- CHANGELOG.asciidoc | 1 + .../process/computer/SparkGraphComputer.java| 90 +--- .../computer/SparkHadoopGraphProvider.java | 5 +- 3 files changed, 80 insertions(+), 16 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/bd85e5fe/CHANGELOG.asciidoc -- diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 21fde2c..395bb55 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -26,6 +26,7 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima * Removed recursive handling of streaming results from Gremlin-Python driver to avoid max recursion depth errors. * Improved performance of `TraversalVertexProgram` and related infrastructure. * Fixed bug in `GroovyTranslator` that didn't properly handle empty `Map` objects. +* Added concrete configuration methods to `SparkGraphComputer` to make a more clear API for configuring it. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018) http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/bd85e5fe/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java -- diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java index 00a2e46..4c896cd 100644 --- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java +++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java @@ -30,11 +30,10 @@ import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.spark.HashPartitioner; import org.apache.spark.Partitioner; -import org.apache.spark.SparkConf; -import org.apache.spark.SparkContext; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.launcher.SparkLauncher; +import org.apache.spark.serializer.Serializer; import org.apache.spark.storage.StorageLevel; import org.apache.tinkerpop.gremlin.hadoop.Constants; import org.apache.tinkerpop.gremlin.hadoop.process.computer.AbstractHadoopGraphComputer; @@ -79,7 +78,16 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; import java.util.concurrent.ThreadFactory; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_GRAPH_STORAGE_LEVEL; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_CONTEXT; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_STORAGE_LEVEL; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_GRAPH_CACHE; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_PARTITIONER; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.SPARK_SERIALIZER; + /** + * {@link GraphComputer} implementation for Apache Spark. + * * @author Marko A. Rodriguez (http://markorodriguez.com) */ public final class SparkGraphComputer extends AbstractHadoopGraphComputer { @@ -107,8 +115,12 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer { ConfigurationUtils.copy(this.hadoopGraph.configuration(), this.sparkConfiguration); } +/** + * Sets the number of workers. If the {@code spark.master} configuration is configured with "local" then it will + * change that configuration to use the specified number of worker threads. + */ @Override -public GraphComputer workers(final int workers) { +public SparkGraphComputer workers(final int workers) { super.workers(workers); if (this.sparkConfiguration.containsKey(SparkLauncher.SPARK_MASTER) && this.sparkConfiguration.getString(SparkLauncher.SPARK_MASTER).startsWith("local")) { this.sparkConfiguration.set
[1/2] tinkerpop git commit: TINKERPOP-1113 Added spark configuration options as concrete methods CTR
Repository: tinkerpop Updated Branches: refs/heads/tp33 a708cc3bd -> 3891777e4 TINKERPOP-1113 Added spark configuration options as concrete methods CTR Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/bd85e5fe Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/bd85e5fe Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/bd85e5fe Branch: refs/heads/tp33 Commit: bd85e5febee56434c4de4e7ab31e337a9f5e Parents: f36eb4f Author: Stephen Mallette Authored: Tue May 22 06:55:46 2018 -0400 Committer: Stephen Mallette Committed: Tue May 22 06:55:46 2018 -0400 -- CHANGELOG.asciidoc | 1 + .../process/computer/SparkGraphComputer.java| 90 +--- .../computer/SparkHadoopGraphProvider.java | 5 +- 3 files changed, 80 insertions(+), 16 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/bd85e5fe/CHANGELOG.asciidoc -- diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 21fde2c..395bb55 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -26,6 +26,7 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima * Removed recursive handling of streaming results from Gremlin-Python driver to avoid max recursion depth errors. * Improved performance of `TraversalVertexProgram` and related infrastructure. * Fixed bug in `GroovyTranslator` that didn't properly handle empty `Map` objects. +* Added concrete configuration methods to `SparkGraphComputer` to make a more clear API for configuring it. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018) http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/bd85e5fe/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java -- diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java index 00a2e46..4c896cd 100644 --- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java +++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java @@ -30,11 +30,10 @@ import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.spark.HashPartitioner; import org.apache.spark.Partitioner; -import org.apache.spark.SparkConf; -import org.apache.spark.SparkContext; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.launcher.SparkLauncher; +import org.apache.spark.serializer.Serializer; import org.apache.spark.storage.StorageLevel; import org.apache.tinkerpop.gremlin.hadoop.Constants; import org.apache.tinkerpop.gremlin.hadoop.process.computer.AbstractHadoopGraphComputer; @@ -79,7 +78,16 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; import java.util.concurrent.ThreadFactory; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_GRAPH_STORAGE_LEVEL; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_CONTEXT; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_STORAGE_LEVEL; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_GRAPH_CACHE; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_PARTITIONER; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.SPARK_SERIALIZER; + /** + * {@link GraphComputer} implementation for Apache Spark. + * * @author Marko A. Rodriguez (http://markorodriguez.com) */ public final class SparkGraphComputer extends AbstractHadoopGraphComputer { @@ -107,8 +115,12 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer { ConfigurationUtils.copy(this.hadoopGraph.configuration(), this.sparkConfiguration); } +/** + * Sets the number of workers. If the {@code spark.master} configuration is configured with "local" then it will + * change that configuration to use the specified number of worker threads. + */ @Override -public GraphComputer workers(final int workers) { +public SparkGraphComputer workers(final int workers) { super.workers(workers); if (this.sparkConfiguration.containsKey(SparkLauncher.SPARK_MASTER) && this.sparkConfiguration.getString(SparkLauncher.SPARK_MASTER).startsWith("local")) { this.sparkConfiguration.set
[3/3] tinkerpop git commit: Merge branch 'tp33'
Merge branch 'tp33' Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/4f46095b Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/4f46095b Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/4f46095b Branch: refs/heads/master Commit: 4f46095badec37d82d220751bcc450898df2609b Parents: 6a645c0 3891777 Author: Stephen Mallette Authored: Tue May 22 07:08:35 2018 -0400 Committer: Stephen Mallette Committed: Tue May 22 07:08:35 2018 -0400 -- CHANGELOG.asciidoc | 1 + .../process/computer/SparkGraphComputer.java| 104 --- 2 files changed, 93 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/4f46095b/CHANGELOG.asciidoc --
[2/2] tinkerpop git commit: Merge branch 'tp32' into tp33
Merge branch 'tp32' into tp33 Conflicts: spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkHadoopGraphProvider.java Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/3891777e Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/3891777e Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/3891777e Branch: refs/heads/tp33 Commit: 3891777e4b30665bd47a5ead9e50871f37f7e9d8 Parents: a708cc3 bd85e5f Author: Stephen Mallette Authored: Tue May 22 07:08:22 2018 -0400 Committer: Stephen Mallette Committed: Tue May 22 07:08:22 2018 -0400 -- CHANGELOG.asciidoc | 1 + .../process/computer/SparkGraphComputer.java| 104 --- 2 files changed, 93 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/3891777e/CHANGELOG.asciidoc -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/3891777e/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java -- diff --cc spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java index dafe613,4c896cd..5184db6 --- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java +++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java @@@ -33,9 -33,7 +33,9 @@@ import org.apache.spark.Partitioner import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.launcher.SparkLauncher; ++import org.apache.spark.serializer.KryoRegistrator; +import org.apache.spark.serializer.KryoSerializer; + import org.apache.spark.serializer.Serializer; import org.apache.spark.storage.StorageLevel; import org.apache.tinkerpop.gremlin.hadoop.Constants; import org.apache.tinkerpop.gremlin.hadoop.process.computer.AbstractHadoopGraphComputer; @@@ -87,7 -78,16 +87,17 @@@ import java.util.concurrent.Executors import java.util.concurrent.Future; import java.util.concurrent.ThreadFactory; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_GRAPH_STORAGE_LEVEL; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_CONTEXT; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_STORAGE_LEVEL; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_GRAPH_CACHE; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_PARTITIONER; ++import static org.apache.tinkerpop.gremlin.hadoop.Constants.SPARK_KRYO_REGISTRATION_REQUIRED; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.SPARK_SERIALIZER; + /** + * {@link GraphComputer} implementation for Apache Spark. + * * @author Marko A. Rodriguez (http://markorodriguez.com) */ public final class SparkGraphComputer extends AbstractHadoopGraphComputer { @@@ -116,10 -112,15 +126,14 @@@ public SparkGraphComputer(final HadoopGraph hadoopGraph) { super(hadoopGraph); this.sparkConfiguration = new HadoopConfiguration(); -ConfigurationUtils.copy(this.hadoopGraph.configuration(), this.sparkConfiguration); } + /** + * Sets the number of workers. If the {@code spark.master} configuration is configured with "local" then it will + * change that configuration to use the specified number of worker threads. + */ @Override - public GraphComputer workers(final int workers) { + public SparkGraphComputer workers(final int workers) { super.workers(workers); if (this.sparkConfiguration.containsKey(SparkLauncher.SPARK_MASTER) && this.sparkConfiguration.getString(SparkLauncher.SPARK_MASTER).startsWith("local")) { this.sparkConfiguration.setProperty(SparkLauncher.SPARK_MASTER, "local[" + this.workers + "]"); @@@ -134,6 -135,56 +148,72 @@@ return this; } + /** + * Sets the configuration option for {@code spark.master} which is the cluster manager to connect to which may be + * one of the https://spark.apache.org/docs/latest/submitting-applications.html#master-urls";>allowed master URLs. + */ + public SparkGraphComputer master(final String clusterManager) { + return configure(SparkLauncher.SPARK_MASTER, clusterManager); + } + + /** + * Determines if the Spark context should be left open preventing
[1/3] tinkerpop git commit: TINKERPOP-1113 Added spark configuration options as concrete methods CTR
Repository: tinkerpop Updated Branches: refs/heads/master 6a645c0ae -> 4f46095ba TINKERPOP-1113 Added spark configuration options as concrete methods CTR Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/bd85e5fe Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/bd85e5fe Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/bd85e5fe Branch: refs/heads/master Commit: bd85e5febee56434c4de4e7ab31e337a9f5e Parents: f36eb4f Author: Stephen Mallette Authored: Tue May 22 06:55:46 2018 -0400 Committer: Stephen Mallette Committed: Tue May 22 06:55:46 2018 -0400 -- CHANGELOG.asciidoc | 1 + .../process/computer/SparkGraphComputer.java| 90 +--- .../computer/SparkHadoopGraphProvider.java | 5 +- 3 files changed, 80 insertions(+), 16 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/bd85e5fe/CHANGELOG.asciidoc -- diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 21fde2c..395bb55 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -26,6 +26,7 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima * Removed recursive handling of streaming results from Gremlin-Python driver to avoid max recursion depth errors. * Improved performance of `TraversalVertexProgram` and related infrastructure. * Fixed bug in `GroovyTranslator` that didn't properly handle empty `Map` objects. +* Added concrete configuration methods to `SparkGraphComputer` to make a more clear API for configuring it. [[release-3-2-9]] === TinkerPop 3.2.9 (Release Date: May 8, 2018) http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/bd85e5fe/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java -- diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java index 00a2e46..4c896cd 100644 --- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java +++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java @@ -30,11 +30,10 @@ import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.spark.HashPartitioner; import org.apache.spark.Partitioner; -import org.apache.spark.SparkConf; -import org.apache.spark.SparkContext; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.launcher.SparkLauncher; +import org.apache.spark.serializer.Serializer; import org.apache.spark.storage.StorageLevel; import org.apache.tinkerpop.gremlin.hadoop.Constants; import org.apache.tinkerpop.gremlin.hadoop.process.computer.AbstractHadoopGraphComputer; @@ -79,7 +78,16 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; import java.util.concurrent.ThreadFactory; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_GRAPH_STORAGE_LEVEL; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_CONTEXT; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_STORAGE_LEVEL; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_GRAPH_CACHE; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_PARTITIONER; +import static org.apache.tinkerpop.gremlin.hadoop.Constants.SPARK_SERIALIZER; + /** + * {@link GraphComputer} implementation for Apache Spark. + * * @author Marko A. Rodriguez (http://markorodriguez.com) */ public final class SparkGraphComputer extends AbstractHadoopGraphComputer { @@ -107,8 +115,12 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer { ConfigurationUtils.copy(this.hadoopGraph.configuration(), this.sparkConfiguration); } +/** + * Sets the number of workers. If the {@code spark.master} configuration is configured with "local" then it will + * change that configuration to use the specified number of worker threads. + */ @Override -public GraphComputer workers(final int workers) { +public SparkGraphComputer workers(final int workers) { super.workers(workers); if (this.sparkConfiguration.containsKey(SparkLauncher.SPARK_MASTER) && this.sparkConfiguration.getString(SparkLauncher.SPARK_MASTER).startsWith("local")) { this.sparkConfiguration
[2/3] tinkerpop git commit: Merge branch 'tp32' into tp33
Merge branch 'tp32' into tp33 Conflicts: spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkHadoopGraphProvider.java Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/3891777e Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/3891777e Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/3891777e Branch: refs/heads/master Commit: 3891777e4b30665bd47a5ead9e50871f37f7e9d8 Parents: a708cc3 bd85e5f Author: Stephen Mallette Authored: Tue May 22 07:08:22 2018 -0400 Committer: Stephen Mallette Committed: Tue May 22 07:08:22 2018 -0400 -- CHANGELOG.asciidoc | 1 + .../process/computer/SparkGraphComputer.java| 104 --- 2 files changed, 93 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/3891777e/CHANGELOG.asciidoc -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/3891777e/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java -- diff --cc spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java index dafe613,4c896cd..5184db6 --- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java +++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java @@@ -33,9 -33,7 +33,9 @@@ import org.apache.spark.Partitioner import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.launcher.SparkLauncher; ++import org.apache.spark.serializer.KryoRegistrator; +import org.apache.spark.serializer.KryoSerializer; + import org.apache.spark.serializer.Serializer; import org.apache.spark.storage.StorageLevel; import org.apache.tinkerpop.gremlin.hadoop.Constants; import org.apache.tinkerpop.gremlin.hadoop.process.computer.AbstractHadoopGraphComputer; @@@ -87,7 -78,16 +87,17 @@@ import java.util.concurrent.Executors import java.util.concurrent.Future; import java.util.concurrent.ThreadFactory; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_GRAPH_STORAGE_LEVEL; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_CONTEXT; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_PERSIST_STORAGE_LEVEL; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_GRAPH_CACHE; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.GREMLIN_SPARK_SKIP_PARTITIONER; ++import static org.apache.tinkerpop.gremlin.hadoop.Constants.SPARK_KRYO_REGISTRATION_REQUIRED; + import static org.apache.tinkerpop.gremlin.hadoop.Constants.SPARK_SERIALIZER; + /** + * {@link GraphComputer} implementation for Apache Spark. + * * @author Marko A. Rodriguez (http://markorodriguez.com) */ public final class SparkGraphComputer extends AbstractHadoopGraphComputer { @@@ -116,10 -112,15 +126,14 @@@ public SparkGraphComputer(final HadoopGraph hadoopGraph) { super(hadoopGraph); this.sparkConfiguration = new HadoopConfiguration(); -ConfigurationUtils.copy(this.hadoopGraph.configuration(), this.sparkConfiguration); } + /** + * Sets the number of workers. If the {@code spark.master} configuration is configured with "local" then it will + * change that configuration to use the specified number of worker threads. + */ @Override - public GraphComputer workers(final int workers) { + public SparkGraphComputer workers(final int workers) { super.workers(workers); if (this.sparkConfiguration.containsKey(SparkLauncher.SPARK_MASTER) && this.sparkConfiguration.getString(SparkLauncher.SPARK_MASTER).startsWith("local")) { this.sparkConfiguration.setProperty(SparkLauncher.SPARK_MASTER, "local[" + this.workers + "]"); @@@ -134,6 -135,56 +148,72 @@@ return this; } + /** + * Sets the configuration option for {@code spark.master} which is the cluster manager to connect to which may be + * one of the https://spark.apache.org/docs/latest/submitting-applications.html#master-urls";>allowed master URLs. + */ + public SparkGraphComputer master(final String clusterManager) { + return configure(SparkLauncher.SPARK_MASTER, clusterManager); + } + + /** + * Determines if the Spark context should be left open preventin
tinkerpop git commit: TINKERPOP-1968 Changelog updates
Repository: tinkerpop Updated Branches: refs/heads/TINKERPOP-1968 b6a8f737c -> 7ce6f99b7 TINKERPOP-1968 Changelog updates Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/7ce6f99b Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/7ce6f99b Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/7ce6f99b Branch: refs/heads/TINKERPOP-1968 Commit: 7ce6f99b7d4e029a48c5788b56fe90d136e58501 Parents: b6a8f73 Author: Stephen Mallette Authored: Tue May 22 06:17:22 2018 -0400 Committer: Stephen Mallette Committed: Tue May 22 06:17:22 2018 -0400 -- CHANGELOG.asciidoc | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/7ce6f99b/CHANGELOG.asciidoc -- diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index b23657b..80a214c 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -26,6 +26,8 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima * Removed recursive handling of streaming results from Gremlin-Python driver to avoid max recursion depth errors. * Improved performance of `TraversalVertexProgram` and related infrastructure. * Added `createGratefulDead()`to `TinkerFactory` to help make it easier to try to instantiate that toy graph. +* Added identifiers to edges in the Kitchen Sink toy graph. +* Refactored the Gremlin Server integration testing framework and streamlined that infrastructure. * Fixed bug in `GroovyTranslator` that didn't properly handle empty `Map` objects. [[release-3-2-9]]