date:20160603

[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315071#comment-15315071
 ] 

ASF GitHub Bot commented on TINKERPOP-1301:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-tinkerpop/pull/327


> Provide Javadoc for ScriptInput/OutputFormat's
> --
>
> Key: TINKERPOP-1301
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1301
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 3.2.0-incubating
>Reporter: Lewis John McGibbney
>Priority: Minor
>
> Right now 
> [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html]
>  and 
> [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html]
>  are not documented. Descriptions are however present over on some old 
> [TitanDB 
> documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html]
>  which can be used to provide Javadoc level documentation for developers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-tinkerpop pull request #327: TINKERPOP-1301 Provide Javadoc for Sc...

2016-06-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-tinkerpop/pull/327


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1317) IoGraphTest throws error: URI is not hierarchical

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314949#comment-15314949
 ] 

ASF GitHub Bot commented on TINKERPOP-1317:
---

Github user twilmes commented on a diff in the pull request:

https://github.com/apache/incubator-tinkerpop/pull/328#discussion_r65783485
  
--- Diff: docs/src/dev/provider/index.asciidoc ---
@@ -505,6 +505,11 @@ off for the test suite (the Maven SureFire Plugin is 
configured this way by defa
 include this setting, `false`, in the SureFire 
configuration if tests are failing in an
 unexplainable way.
 
+TIP: When running the `gremlin-test` suite against your implementation, 
you may need to set `build.dir` as an
+envrionment variable, depending on your project layout. Some tests require 
this to find a writable directory for
--- End diff --

Tiny little typo here: "envrionment" -> "environment"


> IoGraphTest throws error: URI is not hierarchical
> -
>
> Key: TINKERPOP-1317
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1317
> Project: TinkerPop
>  Issue Type: Bug
>  Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> Via https://groups.google.com/d/msg/gremlin-users/aap3pxZtGyU/t-eOC6ZyAAAJ
> These methods will fail for graph providers that are trying to implement the 
> test suite:
> * {{IoGraphTest.shouldReadWriteModernToFileWithHelpers()}}
> * {{IoGraphTest.shouldReadWriteClassicToFileWithHelpers()}}
> Example stack trace:
> {noformat}
> shouldReadWriteModernToFileWithHelpers[gryo](org.apache.tinkerpop.gremlin.structure.io.IoGraphTest)
>   Time elapsed: 0.004 sec  <<< ERROR!
> java.lang.RuntimeException: Unable to computePath for class 
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest
>   at 
> org.apache.tinkerpop.gremlin.TestHelper.computePath(TestHelper.java:117)
>   at 
> org.apache.tinkerpop.gremlin.TestHelper.getRootOfBuildDirectory(TestHelper.java:105)
>   at 
> org.apache.tinkerpop.gremlin.TestHelper.makeTestDataPath(TestHelper.java:70)
>   at 
> org.apache.tinkerpop.gremlin.TestHelper.generateTempFile(TestHelper.java:127)
>   at 
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest.shouldReadWriteModernToFileWithHelpers(IoGraphTest.java:164)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at org.junit.runners.Suite.runChild(Suite.java:27)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at 
>

[GitHub] incubator-tinkerpop pull request #328: TINKERPOP-1317: use graph class as ro...

2016-06-03 Thread twilmes

Github user twilmes commented on a diff in the pull request:

https://github.com/apache/incubator-tinkerpop/pull/328#discussion_r65783485
  
--- Diff: docs/src/dev/provider/index.asciidoc ---
@@ -505,6 +505,11 @@ off for the test suite (the Maven SureFire Plugin is 
configured this way by defa
 include this setting, `false`, in the SureFire 
configuration if tests are failing in an
 unexplainable way.
 
+TIP: When running the `gremlin-test` suite against your implementation, 
you may need to set `build.dir` as an
+envrionment variable, depending on your project layout. Some tests require 
this to find a writable directory for
--- End diff --

Tiny little typo here: "envrionment" -> "environment"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop issue #328: TINKERPOP-1317: use graph class as root clas...

2016-06-03 Thread analytically

Github user analytically commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/328
  
Genius PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop issue #328: TINKERPOP-1317: use graph class as root clas...

2016-06-03 Thread dkuppitz

Github user dkuppitz commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/328
  
VOTE: +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP

2016-06-03 Thread stephen mallette (JIRA)


 [ 
https://issues.apache.org/jira/browse/TINKERPOP-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stephen mallette updated TINKERPOP-1325:

Affects Version/s: (was: 3.2.1)
   3.2.0-incubating
  Component/s: (was: tinkergraph)
   server
   hadoop

> Query results with lambda step not serializable in OLAP
> ---
>
> Key: TINKERPOP-1325
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1325
> Project: TinkerPop
>  Issue Type: Bug
>  Components: hadoop, server
>Affects Versions: 3.2.0-incubating
> Environment: OSX 10.11.5
> apache-gremlin-console-3.2.1
>Reporter: Sergey Gmyzov
>
> Query with lambda step fails in OLAP mode:
> {code}
> gremlin> :remote config alias g ds232.a
> ==>g=ds232.a
> gremlin> :> 
> schema.config().option('graph.traversal_sources.g.restrict_lambda').set(false);
> ==>null
> gremlin> :> 
> schema.config().option('graph.traversal_sources.a.restrict_lambda').set(false);
> ==>null
> gremlin> :> graph.tx().commit();
> ==>null
> gremlin> :> g.V().has("movieId","m267").map{it.get().value("title")};
> org.apache.spark.SparkException: Task not serializable
> Display stack trace? [yN] y
> org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException: 
> org.apache.spark.SparkException: Task not serializable
>   at 
> org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:156)
>   at 
> org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)
>   at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:104)
>   at 
> org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)
>   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>   at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
>   at 
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
>   at 
> org.codehaus.groovy.tools.shell.Groovysh.executeCommand(Groovysh.groovy:259)
>   at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:158)
>   at 
> org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
>   at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>   at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
>   at 
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
>   at 
> org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:63)
>   at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
>   at 
> org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
>   at 
> org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>   at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
>   at 
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
>   at 
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
>   at 
> org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
>   at 
> org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
>   at 
> org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>

[jira] [Updated] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP

2016-06-03 Thread Sergey Gmyzov (JIRA)


 [ 
https://issues.apache.org/jira/browse/TINKERPOP-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Gmyzov updated TINKERPOP-1325:
-
Description: 
Query with lambda step fails in OLAP mode:

{code}
gremlin> :remote config alias g ds232.a
==>g=ds232.a
gremlin> :> 
schema.config().option('graph.traversal_sources.g.restrict_lambda').set(false);
==>null
gremlin> :> 
schema.config().option('graph.traversal_sources.a.restrict_lambda').set(false);
==>null
gremlin> :> graph.tx().commit();
==>null
gremlin> :> g.V().has("movieId","m267").map{it.get().value("title")};
org.apache.spark.SparkException: Task not serializable
Display stack trace? [yN] y
org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException: 
org.apache.spark.SparkException: Task not serializable
at 
org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:156)
at 
org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)
at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:104)
at 
org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at 
org.codehaus.groovy.tools.shell.Groovysh.executeCommand(Groovysh.groovy:259)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:158)
at 
org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at 
org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:63)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at 
org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at 
org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at 
org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
at 
org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at 
org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at 
org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)

[jira] [Created] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP

2016-06-03 Thread Sergey Gmyzov (JIRA)

Sergey Gmyzov created TINKERPOP-1325:


 Summary: Query results with lambda step not serializable in OLAP
 Key: TINKERPOP-1325
 URL: https://issues.apache.org/jira/browse/TINKERPOP-1325
 Project: TinkerPop
  Issue Type: Bug
  Components: tinkergraph
Affects Versions: 3.2.1
 Environment: OSX 10.11.5
apache-gremlin-console-3.2.1
Reporter: Sergey Gmyzov


Query with lambda step fails in OLAP mode:

{panel}
gremlin> :remote config alias g ds232.a
==>g=ds232.a
gremlin> :> 
schema.config().option('graph.traversal_sources.g.restrict_lambda').set(false);
==>null
gremlin> :> 
schema.config().option('graph.traversal_sources.a.restrict_lambda').set(false);
==>null
gremlin> :> graph.tx().commit();
==>null
gremlin> :> g.V().has("movieId","m267").map{it.get().value("title")};
org.apache.spark.SparkException: Task not serializable
Display stack trace? [yN] y
org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException: 
org.apache.spark.SparkException: Task not serializable
at 
org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:156)
at 
org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)
at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:104)
at 
org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at 
org.codehaus.groovy.tools.shell.Groovysh.executeCommand(Groovysh.groovy:259)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:158)
at 
org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at 
org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:63)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at 
org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at 
org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at 
org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
at 
org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at 
org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314682#comment-15314682
 ] 

ASF GitHub Bot commented on TINKERPOP-1321:
---

Github user dalaro commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
Additional notes from Marko:

> okay, so as long as there is never a need for MyRegistrar, then can you 
please update your PR accordingly?
> Finally, call it GryoRegistrar (not TinkerPopRegistrar) as this has to do 
with Gryo IORegistry classes.


> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting 
> spark.serializer=GryoSerializer.  This makes GryoSerializer responsible for 
> serializing not just TinkerPop types but also scala runtime types and Spark 
> internals.  GryoSerializer doesn't extend either of the two serializers 
> provided by Spark.  It effectively assumes responsibility for reimplementing 
> them.
> This is problematic.  It is not totally trivial to replicate the 
> functionality of Spark's standard serializers.  It is also not easy to 
> empirically test all meaningful cases.  For instance, there is a conditional 
> within Spark that selects between two concrete Map implementations depending 
> on whether the current RDD partition count exceeds 2k 
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
>   The implementation used below this threshold serializes fine on 
> GryoSerializer.  The implementation used above the threshold does not.  Above 
> the partition threshold, I've started getting 
> {{org.apache.spark.SparkException: Job aborted due to stage failure: 
> Exception while getting task result: 
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed 
> to find one of the right cookies.}}  Google leads to 
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64.  However, just 
> switching to Spark's {{KryoSerializer}} without changing anything somehow 
> fixes the problem in my environment, implying that Spark has done something 
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.  
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph 
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.  
> These still serialize, but KryoSerializer uses its default behavior 
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's 
> StarGraphSerializer.  TinkerPop's reliance on its own internal shaded Kryo 
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just 
> by tweaking the configuration.  Besides spark.serializer, there is also a 
> setting spark.closure.serializer for which the only supported value is 
> JavaSerializer.  Key TP classes that make it into the object reference graphs 
> of Spark closures implement Serializable by resorting to TinkerPop's shaded 
> Kryo via HadoopPools (looking at Object/VertexWritable).  This leads to 
> surprises with custom property data types.  It doesn't matter if those types 
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is 
> configured to accept those types.  If  those types are reachable from 
> Object/VertexWritable, then they must be registered with TinkerPop's internal 
> shaded Kryo, or else it will choke on them (unless it was explicitly 
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice 
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if 
> they choose not to use GryoSerializer: introduce lightweight interfaces that 
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded 
> package doing the work.  TinkerPop's serialization logic would be written 
> against interfaces that replicate a minimal subset of Kryo, and then TP's 
> shaded Kryo or Spark's unshaded Kryo could be plugged in underneath without 
> having to touch the source, recompile any TinkerPop code, or munge bytecode 
> at runtime.
> This would not resolve all of the potential problems/complexity around 
> TinkerPop serialization, but it would make it possible for users to apply the 
> spark.serializer of their choice, switching

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

2016-06-03 Thread dalaro

Github user dalaro commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
Additional notes from Marko:

> okay, so as long as there is never a need for MyRegistrar, then can you 
please update your PR accordingly?
> Finally, call it GryoRegistrar (not TinkerPopRegistrar) as this has to do 
with Gryo IORegistry classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314655#comment-15314655
 ] 

ASF GitHub Bot commented on TINKERPOP-1321:
---

Github user dalaro commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
After discussion on Slack, I think @okram and I tentatively agreed to 
proceed with this PR after I do additional work to save users who have custom 
serializers the effort of maintaining separate `IoRegistry` and 
`spark.kryo.registrator` implementations with near-identical contents.

I may discover other complications during the process, but I think this 
involves two changes:

1. I will attempt to subclass KryoSerializer so that I can access the 
SparkConf passed to its constructior and check for 
`GryoPool.CONFIG_IO_REGISTRY` (similar to what GryoSerializer does now), then 
apply any registrations found therein so long as each registration either:

   * specifies no explicit serializer (using Kryo's internal default) or
   * specifies a shim serializer

   In particular, if the registry contains an old-style TP shaded Kryo 
serializer that hasn't been ported to the shim, then the KryoSerializer 
subclass will throw an exception.

   This change is necessary to support custom-serialized, 
`IoRegistry`-listed datatypes that appear in most Spark data outside of 
closures (say, in the RDD itself).

2. I will replace current callsites of `HadoopPools.initialize(conf)` with 
some other static method that calls `HadoopPools.initialize(conf)` and then 
calls some roughly equivalent `initialize(conf)` method for the static KryoPool 
backing `KryoPoolShimService`.

   This change is necessary to support custom-serialized, 
`IoRegistry`-listed datatypes that appear in closures.


> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting 
> spark.serializer=GryoSerializer.  This makes GryoSerializer responsible for 
> serializing not just TinkerPop types but also scala runtime types and Spark 
> internals.  GryoSerializer doesn't extend either of the two serializers 
> provided by Spark.  It effectively assumes responsibility for reimplementing 
> them.
> This is problematic.  It is not totally trivial to replicate the 
> functionality of Spark's standard serializers.  It is also not easy to 
> empirically test all meaningful cases.  For instance, there is a conditional 
> within Spark that selects between two concrete Map implementations depending 
> on whether the current RDD partition count exceeds 2k 
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
>   The implementation used below this threshold serializes fine on 
> GryoSerializer.  The implementation used above the threshold does not.  Above 
> the partition threshold, I've started getting 
> {{org.apache.spark.SparkException: Job aborted due to stage failure: 
> Exception while getting task result: 
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed 
> to find one of the right cookies.}}  Google leads to 
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64.  However, just 
> switching to Spark's {{KryoSerializer}} without changing anything somehow 
> fixes the problem in my environment, implying that Spark has done something 
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.  
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph 
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.  
> These still serialize, but KryoSerializer uses its default behavior 
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's 
> StarGraphSerializer.  TinkerPop's reliance on its own internal shaded Kryo 
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just 
> by tweaking the configuration.  Besides spark.serializer, there is also a 
> setting spark.closure.serializer for which the only supported value is 
> JavaSerializer.  Key TP classes that make it into the object reference graphs 
> of Spark closures implement Serializable by resorting to TinkerPop's shaded 
> Kryo via HadoopPools (looking at

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

2016-06-03 Thread okram

Github user okram commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
@dalaro -- what about it doesn't work? Meaning, is your problem just a 
"unregistered class" issue? If so, we just register it. If I have a "bad" 
serializer for one of the registered classes, then we can fix it?

See the classes that I've registered in `GryoSerializer` (The last 7 are 
TinkerPop specific.):

```java
builder.addCustom(Tuple2.class, new 
Tuple2Serializer())
.addCustom(Tuple2[].class)
.addCustom(Tuple3.class, new 
Tuple3Serializer())
.addCustom(Tuple3[].class)
.addCustom(CompactBuffer.class, new 
CompactBufferSerializer())
.addCustom(CompactBuffer[].class)
.addCustom(CompressedMapStatus.class)
.addCustom(BlockManagerId.class)
.addCustom(HighlyCompressedMapStatus.class, 
new ExternalizableSerializer()) 
.addCustom(HttpBroadcast.class)
.addCustom(PythonBroadcast.class)
.addCustom(BoxedUnit.class)

.addCustom(Class.forName("scala.reflect.ClassTag$$anon$1"), new 
JavaSerializer())

.addCustom(Class.forName("scala.reflect.ManifestFactory$$anon$1"), new 
JavaSerializer())
.addCustom(WrappedArray.ofRef.class, new 
WrappedArraySerializer())
.addCustom(MessagePayload.class)
.addCustom(ViewIncomingPayload.class)
.addCustom(ViewOutgoingPayload.class)
.addCustom(ViewPayload.class)
.addCustom(SerializableConfiguration.class, 
new JavaSerializer())
.addCustom(VertexWritable.class, new 
VertexWritableSerializer())
.addCustom(ObjectWritable.class, new 
ObjectWritableSerializer())
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314508#comment-15314508
 ] 

ASF GitHub Bot commented on TINKERPOP-1321:
---

Github user dalaro commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
The problem is that GryoSerializer doesn't actually work.  This is not the 
first time it has shown behavior on Spark or scala runtime classes which 
diverges from the behavior of Spark's KryoSerializer/JavaSerializer.

I realize that asking users to write custom serializers against the shim 
(to be compatible with everything) in the long term or to duplicate 
registrations in the short term (if they want to use Gryo in one place and 
Spark graph computation in another, using the same custom types) is not ideal, 
but I think the status quo is even worse.


> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting 
> spark.serializer=GryoSerializer.  This makes GryoSerializer responsible for 
> serializing not just TinkerPop types but also scala runtime types and Spark 
> internals.  GryoSerializer doesn't extend either of the two serializers 
> provided by Spark.  It effectively assumes responsibility for reimplementing 
> them.
> This is problematic.  It is not totally trivial to replicate the 
> functionality of Spark's standard serializers.  It is also not easy to 
> empirically test all meaningful cases.  For instance, there is a conditional 
> within Spark that selects between two concrete Map implementations depending 
> on whether the current RDD partition count exceeds 2k 
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
>   The implementation used below this threshold serializes fine on 
> GryoSerializer.  The implementation used above the threshold does not.  Above 
> the partition threshold, I've started getting 
> {{org.apache.spark.SparkException: Job aborted due to stage failure: 
> Exception while getting task result: 
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed 
> to find one of the right cookies.}}  Google leads to 
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64.  However, just 
> switching to Spark's {{KryoSerializer}} without changing anything somehow 
> fixes the problem in my environment, implying that Spark has done something 
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.  
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph 
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.  
> These still serialize, but KryoSerializer uses its default behavior 
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's 
> StarGraphSerializer.  TinkerPop's reliance on its own internal shaded Kryo 
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just 
> by tweaking the configuration.  Besides spark.serializer, there is also a 
> setting spark.closure.serializer for which the only supported value is 
> JavaSerializer.  Key TP classes that make it into the object reference graphs 
> of Spark closures implement Serializable by resorting to TinkerPop's shaded 
> Kryo via HadoopPools (looking at Object/VertexWritable).  This leads to 
> surprises with custom property data types.  It doesn't matter if those types 
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is 
> configured to accept those types.  If  those types are reachable from 
> Object/VertexWritable, then they must be registered with TinkerPop's internal 
> shaded Kryo, or else it will choke on them (unless it was explicitly 
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice 
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if 
> they choose not to use GryoSerializer: introduce lightweight interfaces that 
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded 
> package doing the work.  TinkerPop's serialization logic would be written 
> against interfaces that replicate a minimal subset of Kryo, and then TP's 
> shaded Kryo or Spark's unshaded Kryo could be plugged in

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

2016-06-03 Thread dalaro

Github user dalaro commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
The problem is that GryoSerializer doesn't actually work.  This is not the 
first time it has shown behavior on Spark or scala runtime classes which 
diverges from the behavior of Spark's KryoSerializer/JavaSerializer.

I realize that asking users to write custom serializers against the shim 
(to be compatible with everything) in the long term or to duplicate 
registrations in the short term (if they want to use Gryo in one place and 
Spark graph computation in another, using the same custom types) is not ideal, 
but I think the status quo is even worse.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314503#comment-15314503
 ] 

ASF GitHub Bot commented on TINKERPOP-1321:
---

Github user okram commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
So yea... sorta feels like we are where we were before in a way. And yea, 
all that Input, Output, Kryo stuff is what causes all the problems.

So now I'm wondering, why not just register the requisite classes with 
`GryoSerializer`? Yes, we miss one every now and then, but we add it. Sooner or 
later they will all be there. Moreover, if a person needs it now, they can just 
register it with `GryoMapper`. Did you try just registering your "high density 
map" w/ `GryoSerializer`?

I know that last paragraph sucks given all the work you did, but having two 
ways of doing something and one way being Spark-specific doesn't make me happy 
and this is why `GryoSerializer` came into being.. 


> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting 
> spark.serializer=GryoSerializer.  This makes GryoSerializer responsible for 
> serializing not just TinkerPop types but also scala runtime types and Spark 
> internals.  GryoSerializer doesn't extend either of the two serializers 
> provided by Spark.  It effectively assumes responsibility for reimplementing 
> them.
> This is problematic.  It is not totally trivial to replicate the 
> functionality of Spark's standard serializers.  It is also not easy to 
> empirically test all meaningful cases.  For instance, there is a conditional 
> within Spark that selects between two concrete Map implementations depending 
> on whether the current RDD partition count exceeds 2k 
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
>   The implementation used below this threshold serializes fine on 
> GryoSerializer.  The implementation used above the threshold does not.  Above 
> the partition threshold, I've started getting 
> {{org.apache.spark.SparkException: Job aborted due to stage failure: 
> Exception while getting task result: 
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed 
> to find one of the right cookies.}}  Google leads to 
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64.  However, just 
> switching to Spark's {{KryoSerializer}} without changing anything somehow 
> fixes the problem in my environment, implying that Spark has done something 
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.  
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph 
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.  
> These still serialize, but KryoSerializer uses its default behavior 
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's 
> StarGraphSerializer.  TinkerPop's reliance on its own internal shaded Kryo 
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just 
> by tweaking the configuration.  Besides spark.serializer, there is also a 
> setting spark.closure.serializer for which the only supported value is 
> JavaSerializer.  Key TP classes that make it into the object reference graphs 
> of Spark closures implement Serializable by resorting to TinkerPop's shaded 
> Kryo via HadoopPools (looking at Object/VertexWritable).  This leads to 
> surprises with custom property data types.  It doesn't matter if those types 
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is 
> configured to accept those types.  If  those types are reachable from 
> Object/VertexWritable, then they must be registered with TinkerPop's internal 
> shaded Kryo, or else it will choke on them (unless it was explicitly 
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice 
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if 
> they choose not to use GryoSerializer: introduce lightweight interfaces that 
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded 
> package doing the work.  TinkerPop's serialization logic would be

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

2016-06-03 Thread okram

Github user okram commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
So yea... sorta feels like we are where we were before in a way. And yea, 
all that Input, Output, Kryo stuff is what causes all the problems.

So now I'm wondering, why not just register the requisite classes with 
`GryoSerializer`? Yes, we miss one every now and then, but we add it. Sooner or 
later they will all be there. Moreover, if a person needs it now, they can just 
register it with `GryoMapper`. Did you try just registering your "high density 
map" w/ `GryoSerializer`?

I know that last paragraph sucks given all the work you did, but having two 
ways of doing something and one way being Spark-specific doesn't make me happy 
and this is why `GryoSerializer` came into being.. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314456#comment-15314456
 ] 

ASF GitHub Bot commented on TINKERPOP-1321:
---

Github user dalaro commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
BTW, in case you're wondering why I think it's hard to create proxies, 
recall that o.a.t.shaded.kryo.{Kryo,io.Input,io.Output} are all classes, not 
interfaces.  There are a lot of approaches that would make this work, but the 
ones I can think of suck for various reasons.


> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting 
> spark.serializer=GryoSerializer.  This makes GryoSerializer responsible for 
> serializing not just TinkerPop types but also scala runtime types and Spark 
> internals.  GryoSerializer doesn't extend either of the two serializers 
> provided by Spark.  It effectively assumes responsibility for reimplementing 
> them.
> This is problematic.  It is not totally trivial to replicate the 
> functionality of Spark's standard serializers.  It is also not easy to 
> empirically test all meaningful cases.  For instance, there is a conditional 
> within Spark that selects between two concrete Map implementations depending 
> on whether the current RDD partition count exceeds 2k 
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
>   The implementation used below this threshold serializes fine on 
> GryoSerializer.  The implementation used above the threshold does not.  Above 
> the partition threshold, I've started getting 
> {{org.apache.spark.SparkException: Job aborted due to stage failure: 
> Exception while getting task result: 
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed 
> to find one of the right cookies.}}  Google leads to 
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64.  However, just 
> switching to Spark's {{KryoSerializer}} without changing anything somehow 
> fixes the problem in my environment, implying that Spark has done something 
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.  
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph 
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.  
> These still serialize, but KryoSerializer uses its default behavior 
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's 
> StarGraphSerializer.  TinkerPop's reliance on its own internal shaded Kryo 
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just 
> by tweaking the configuration.  Besides spark.serializer, there is also a 
> setting spark.closure.serializer for which the only supported value is 
> JavaSerializer.  Key TP classes that make it into the object reference graphs 
> of Spark closures implement Serializable by resorting to TinkerPop's shaded 
> Kryo via HadoopPools (looking at Object/VertexWritable).  This leads to 
> surprises with custom property data types.  It doesn't matter if those types 
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is 
> configured to accept those types.  If  those types are reachable from 
> Object/VertexWritable, then they must be registered with TinkerPop's internal 
> shaded Kryo, or else it will choke on them (unless it was explicitly 
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice 
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if 
> they choose not to use GryoSerializer: introduce lightweight interfaces that 
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded 
> package doing the work.  TinkerPop's serialization logic would be written 
> against interfaces that replicate a minimal subset of Kryo, and then TP's 
> shaded Kryo or Spark's unshaded Kryo could be plugged in underneath without 
> having to touch the source, recompile any TinkerPop code, or munge bytecode 
> at runtime.
> This would not resolve all of the potential problems/complexity around 
> TinkerPop serialization, but it would make it possible for users to apply the 
> spark.serializer of their

[jira] [Commented] (TINKERPOP-1318) java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314431#comment-15314431
 ] 

ASF GitHub Bot commented on TINKERPOP-1318:
---

Github user pluradj commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/329
  
Yes, I ran integration tests with `-DincludeNeo4j`


> java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch
> --
>
> Key: TINKERPOP-1318
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1318
> Project: TinkerPop
>  Issue Type: Bug
>  Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> I don't recall specifically how to make this fail with {{gremlin-test}}, but 
> I did run into it at one point when writing a graph implementation. This blog 
> describes the issue and workaround. 
> https://tedvinke.wordpress.com/2013/12/17/mixing-junit-hamcrest-and-mockito-explaining-nosuchmethoderror/
> The error trace looks like this:
> {noformat}
> java.lang.NoSuchMethodError: 
> org.hamcrest.Matcher.describeMismatch(Ljava/lang/Object;Lorg/hamcrest/Description;)V
> {noformat}
> There is a dependency conflict created by an older version of {{hamcrest}} 
> coming out of {{mockito-all}}. The fix is to use {{mockito-core}} instead.
> I'll submit a patch for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-tinkerpop issue #329: TINKERPOP-1318: use mockito-core instead of ...

2016-06-03 Thread pluradj

Github user pluradj commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/329
  
Yes, I ran integration tests with `-DincludeNeo4j`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1319) several FeatureRequirement annotations are incorrect in gremlin-test

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314427#comment-15314427
 ] 

ASF GitHub Bot commented on TINKERPOP-1319:
---

GitHub user pluradj opened a pull request:

https://github.com/apache/incubator-tinkerpop/pull/330

several FeatureRequirement annotations are incorrect

https://issues.apache.org/jira/browse/TINKERPOP-1319

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1319

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-tinkerpop/pull/330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #330


commit d0878cf70778c637cafa43d09f9a134d89ee5024
Author: Jason Plurad 
Date:   2016-06-03T17:06:27Z

several FeatureRequirement annotations are incorrect




> several FeatureRequirement annotations are incorrect in gremlin-test
> 
>
> Key: TINKERPOP-1319
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1319
> Project: TinkerPop
>  Issue Type: Bug
>  Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> Several {{@FeatureRequirement}} annotations are incorrect in these 
> {{gremlin-test}} tests
> * EdgeTest.java
> * FeatureSupportTest.java
> * GraphTest.java
> * PropertyTest.java
> * VertexPropertyTest.java
> * VertexTest.java
> I'll submit a patch for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-tinkerpop pull request #330: several FeatureRequirement annotation...

2016-06-03 Thread pluradj

GitHub user pluradj opened a pull request:

https://github.com/apache/incubator-tinkerpop/pull/330

several FeatureRequirement annotations are incorrect

https://issues.apache.org/jira/browse/TINKERPOP-1319

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1319

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-tinkerpop/pull/330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #330


commit d0878cf70778c637cafa43d09f9a134d89ee5024
Author: Jason Plurad 
Date:   2016-06-03T17:06:27Z

several FeatureRequirement annotations are incorrect




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop issue #329: use mockito-core instead of mockito-all to a...

2016-06-03 Thread okram

Github user okram commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/329
  
Do integration tests pass? ... that will be all I care about. @spmallette 
will have deeper thoughts on the matter as he uses Mockito.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1318) java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314426#comment-15314426
 ] 

ASF GitHub Bot commented on TINKERPOP-1318:
---

GitHub user pluradj opened a pull request:

https://github.com/apache/incubator-tinkerpop/pull/329

use mockito-core instead of mockito-all to avoid hamcrest conflict

https://issues.apache.org/jira/browse/TINKERPOP-1318

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1318

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-tinkerpop/pull/329.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #329


commit 27eedc478b58cd9dee571960091a3c6505f2379e
Author: Jason Plurad 
Date:   2016-06-03T17:05:10Z

use mockito-core instead of mockito-all to avoid hamcrest conflict




> java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch
> --
>
> Key: TINKERPOP-1318
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1318
> Project: TinkerPop
>  Issue Type: Bug
>  Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> I don't recall specifically how to make this fail with {{gremlin-test}}, but 
> I did run into it at one point when writing a graph implementation. This blog 
> describes the issue and workaround. 
> https://tedvinke.wordpress.com/2013/12/17/mixing-junit-hamcrest-and-mockito-explaining-nosuchmethoderror/
> The error trace looks like this:
> {noformat}
> java.lang.NoSuchMethodError: 
> org.hamcrest.Matcher.describeMismatch(Ljava/lang/Object;Lorg/hamcrest/Description;)V
> {noformat}
> There is a dependency conflict created by an older version of {{hamcrest}} 
> coming out of {{mockito-all}}. The fix is to use {{mockito-core}} instead.
> I'll submit a patch for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314387#comment-15314387
 ] 

ASF GitHub Bot commented on TINKERPOP-1321:
---

Github user okram commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
@dalaro

> The problem with custom registrations is that GryoMapper allows the user 
to provide a serializer written against shaded Kryo. This is not compatible 
with Spark's KryoSerializer. I don't see a way to make it compatible without 
putting fake org.apache.tinkerpop.shaded.kryo.* classes on the classpath, which 
could get pretty ugly.

Yea, thats a problem and the reason why `GryoSerializer` exists. We don't 
want to make `SparkGraphComputer` "special." Users register their serializers 
with `Graph` and all OLTP and OLAP systems take it from there. If, for 
`SparkGraphComputer`, they ALSO have to register them with Spark, that is no 
bueno. Is there a way to create "proxies" that map between shaded and unshaded 
Kryo so that registered `GryoMapper` serializers can be used as unshaded Kryo 
serializers by Spark?


> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting 
> spark.serializer=GryoSerializer.  This makes GryoSerializer responsible for 
> serializing not just TinkerPop types but also scala runtime types and Spark 
> internals.  GryoSerializer doesn't extend either of the two serializers 
> provided by Spark.  It effectively assumes responsibility for reimplementing 
> them.
> This is problematic.  It is not totally trivial to replicate the 
> functionality of Spark's standard serializers.  It is also not easy to 
> empirically test all meaningful cases.  For instance, there is a conditional 
> within Spark that selects between two concrete Map implementations depending 
> on whether the current RDD partition count exceeds 2k 
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
>   The implementation used below this threshold serializes fine on 
> GryoSerializer.  The implementation used above the threshold does not.  Above 
> the partition threshold, I've started getting 
> {{org.apache.spark.SparkException: Job aborted due to stage failure: 
> Exception while getting task result: 
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed 
> to find one of the right cookies.}}  Google leads to 
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64.  However, just 
> switching to Spark's {{KryoSerializer}} without changing anything somehow 
> fixes the problem in my environment, implying that Spark has done something 
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.  
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph 
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.  
> These still serialize, but KryoSerializer uses its default behavior 
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's 
> StarGraphSerializer.  TinkerPop's reliance on its own internal shaded Kryo 
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just 
> by tweaking the configuration.  Besides spark.serializer, there is also a 
> setting spark.closure.serializer for which the only supported value is 
> JavaSerializer.  Key TP classes that make it into the object reference graphs 
> of Spark closures implement Serializable by resorting to TinkerPop's shaded 
> Kryo via HadoopPools (looking at Object/VertexWritable).  This leads to 
> surprises with custom property data types.  It doesn't matter if those types 
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is 
> configured to accept those types.  If  those types are reachable from 
> Object/VertexWritable, then they must be registered with TinkerPop's internal 
> shaded Kryo, or else it will choke on them (unless it was explicitly 
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice 
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if 
> they choose not to use GryoSerializer: introduce lightweight interfaces that 
> decouple

[GitHub] incubator-tinkerpop issue #327: TINKERPOP-1301 Provide Javadoc for ScriptInp...

2016-06-03 Thread lewismc

Github user lewismc commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/327
  
@spmallette LOL np


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314371#comment-15314371
 ] 

ASF GitHub Bot commented on TINKERPOP-1301:
---

Github user pluradj commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/327
  
VOTE +1


> Provide Javadoc for ScriptInput/OutputFormat's
> --
>
> Key: TINKERPOP-1301
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1301
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 3.2.0-incubating
>Reporter: Lewis John McGibbney
>Priority: Minor
>
> Right now 
> [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html]
>  and 
> [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html]
>  are not documented. Descriptions are however present over on some old 
> [TitanDB 
> documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html]
>  which can be used to provide Javadoc level documentation for developers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-tinkerpop issue #327: TINKERPOP-1301 Provide Javadoc for ScriptInp...

2016-06-03 Thread pluradj

Github user pluradj commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/327
  
VOTE +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314306#comment-15314306
 ] 

ASF GitHub Bot commented on TINKERPOP-1301:
---

GitHub user lewismc opened a pull request:

https://github.com/apache/incubator-tinkerpop/pull/327

TINKERPOP-1301 Provide Javadoc for ScriptInput/OutputFormat's ported to 
tp31 branch

Hi @dkuppitz here is the TINKERPOP-1301 ported to tp31 branch :)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lewismc/incubator-tinkerpop TINKERPOP-1301tp31

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-tinkerpop/pull/327.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #327


commit d5d2c48ef67e31671cb177d30c6a15c84b585d24
Author: Lewis John McGibbney 
Date:   2016-06-03T16:00:08Z

TINKERPOP-1301 Provide Javadoc for ScriptInput/OutputFormat's ported to 
tp31 branch




> Provide Javadoc for ScriptInput/OutputFormat's
> --
>
> Key: TINKERPOP-1301
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1301
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 3.2.0-incubating
>Reporter: Lewis John McGibbney
>Priority: Minor
>
> Right now 
> [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html]
>  and 
> [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html]
>  are not documented. Descriptions are however present over on some old 
> [TitanDB 
> documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html]
>  which can be used to provide Javadoc level documentation for developers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

2016-06-03 Thread dalaro

Github user dalaro commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
@spmallette You're right about overrides being the problem.

The last thing I did before opening this PR was to rebase on latest master, 
when I encountered conflicts solely involving the GryoMapper registration 
override feature added in d7684757734732f39970265a425b921a48d8f0fb.  I 
attempted to resolve those conflicts too hastily and got it wrong.

I went back to Stephen's original commit reworked my code around 
registration overrides, and I now think it conforms to the old behavior 
precedent.  Specifically, when handling overrides of GryoMapper type 
registrations, I departed from the precedent set by Stephen's commit in two 
ways, both of which I've fixed locally:

* Override registrations allocated a new registration ID.  Your commit used 
the old ID from the registration being overridden.
* As a side effect of the preceding point, I incremented the shared 
`currentSerializationId` for all registration calls.  Your commit incremented 
this shared counter only for non-override registration calls.

The test passes locally for me now.  I'm still doing some registrator 
refactoring and haven't pushed yet though.

@okram The problem with custom registrations is that GryoMapper allows the 
user to provide a serializer written against shaded Kryo.  This is not 
compatible with Spark's KryoSerializer.  I don't see a way to make it 
compatible without putting fake `org.apache.tinkerpop.shaded.kryo.*` classes on 
the classpath, which could get pretty ugly.

Instead, I'm adding a registrator that includes:

* all default mappings from GryoMapper (I am changing the 
shaded-Kryo-serializers to be shim serializers so that they are reusable)
* the TinkerPop classes registered in GryoSerializer's constructors (but 
not the scala and Spark classes registered in there)

If the user wants to register custom types while using KryoSerializer, they 
can extend the registrator in a subclass, then set `spark.kryo.registrator` to 
the subclass.  The interface in question is `KryoRegistrator` and it has only 
one method, `registerClasses(Kryo)`.  It's about as simple -- maybe even less 
so -- than implementing TP's `IoRegistry` interface, which has two methods.

If the user decides to continue using GryoSerializer, they can also 
continue using `GryoPool.CONFIG_IO_REGISTRY`, which should still affect 
GryoSerializer/HadoopPools as it always has.

WDYT?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

2016-06-03 Thread okram

Github user okram commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
@dalaro -- you mention:

```
spark.serializer=KryoSerializer
spark.kryo.registrator=com.tinkerpop.something.TinkerPopRegistrator
```

I think we need:

```
spark.serializer=KryoSerializer

spark.kryo.registrator=org.apache.tinkerpop.gremlin.structure.io.gryo.GryoRegistrar
```

`GryoRegistrar` would have all the `GryoMapper` classes registered which, 
when people do custom registrations (e.g. `Point`, 
`MyStupidStringImplementation`), it will be in `GryoMapper`. Can you provide 
something like that in this PR? Hard? @spmallette knows more about how to 
interact with `GryoMapper` to get our the serializers...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1196) Calls to Result.one() might block indefinitely

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313981#comment-15313981
 ] 

ASF GitHub Bot commented on TINKERPOP-1196:
---

Github user dkuppitz commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/322
  
20 runs in a row, all of them succeeded.

VOTE: +1


> Calls to Result.one() might block indefinitely
> --
>
> Key: TINKERPOP-1196
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1196
> Project: TinkerPop
>  Issue Type: Bug
>  Components: driver
>Affects Versions: 3.1.1-incubating
>Reporter: stephen mallette
>Assignee: stephen mallette
> Fix For: 3.1.3
>
>
> There are no reproduction steps for this but it does seem to happen from on 
> very rare occasion for some unknown reason.  Need to investigate more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

2016-06-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313973#comment-15313973
 ] 

ASF GitHub Bot commented on TINKERPOP-1321:
---

Github user spmallette commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
I've confirmed that this test:

```text

GremlinResultSetIntegrateTest.shouldHandleVertexResultWithLiteSerialization:174 
? Execution
```

is only failing after these changes and not on master. I sense it has 
something to do with this:

```java
private  void addOrOverrideRegistration(TypeRegistration 
newRegistration) 
```

not preserving the registration identifier, but i could be wrong.


> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting 
> spark.serializer=GryoSerializer.  This makes GryoSerializer responsible for 
> serializing not just TinkerPop types but also scala runtime types and Spark 
> internals.  GryoSerializer doesn't extend either of the two serializers 
> provided by Spark.  It effectively assumes responsibility for reimplementing 
> them.
> This is problematic.  It is not totally trivial to replicate the 
> functionality of Spark's standard serializers.  It is also not easy to 
> empirically test all meaningful cases.  For instance, there is a conditional 
> within Spark that selects between two concrete Map implementations depending 
> on whether the current RDD partition count exceeds 2k 
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
>   The implementation used below this threshold serializes fine on 
> GryoSerializer.  The implementation used above the threshold does not.  Above 
> the partition threshold, I've started getting 
> {{org.apache.spark.SparkException: Job aborted due to stage failure: 
> Exception while getting task result: 
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed 
> to find one of the right cookies.}}  Google leads to 
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64.  However, just 
> switching to Spark's {{KryoSerializer}} without changing anything somehow 
> fixes the problem in my environment, implying that Spark has done something 
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.  
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph 
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.  
> These still serialize, but KryoSerializer uses its default behavior 
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's 
> StarGraphSerializer.  TinkerPop's reliance on its own internal shaded Kryo 
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just 
> by tweaking the configuration.  Besides spark.serializer, there is also a 
> setting spark.closure.serializer for which the only supported value is 
> JavaSerializer.  Key TP classes that make it into the object reference graphs 
> of Spark closures implement Serializable by resorting to TinkerPop's shaded 
> Kryo via HadoopPools (looking at Object/VertexWritable).  This leads to 
> surprises with custom property data types.  It doesn't matter if those types 
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is 
> configured to accept those types.  If  those types are reachable from 
> Object/VertexWritable, then they must be registered with TinkerPop's internal 
> shaded Kryo, or else it will choke on them (unless it was explicitly 
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice 
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if 
> they choose not to use GryoSerializer: introduce lightweight interfaces that 
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded 
> package doing the work.  TinkerPop's serialization logic would be written 
> against interfaces that replicate a minimal subset of Kryo, and then TP's 
> shaded Kryo or Spark's unshaded Kryo could be plugged in underneath without 
> having to touch the source, recompile any TinkerPop code, or munge bytecode 
> at runtime.
> This would

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

2016-06-03 Thread spmallette

Github user spmallette commented on the issue:

https://github.com/apache/incubator-tinkerpop/pull/325
  
I've confirmed that this test:

```text

GremlinResultSetIntegrateTest.shouldHandleVertexResultWithLiteSerialization:174 
? Execution
```

is only failing after these changes and not on master. I sense it has 
something to do with this:

```java
private  void addOrOverrideRegistration(TypeRegistration 
newRegistration) 
```

not preserving the registration identifier, but i could be wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's

[GitHub] incubator-tinkerpop pull request #327: TINKERPOP-1301 Provide Javadoc for Sc...

[jira] [Commented] (TINKERPOP-1317) IoGraphTest throws error: URI is not hierarchical

[GitHub] incubator-tinkerpop pull request #328: TINKERPOP-1317: use graph class as ro...

[GitHub] incubator-tinkerpop issue #328: TINKERPOP-1317: use graph class as root clas...

[GitHub] incubator-tinkerpop issue #328: TINKERPOP-1317: use graph class as root clas...

[jira] [Updated] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP

[jira] [Updated] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP

[jira] [Created] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

[jira] [Commented] (TINKERPOP-1318) java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch

[GitHub] incubator-tinkerpop issue #329: TINKERPOP-1318: use mockito-core instead of ...

[jira] [Commented] (TINKERPOP-1319) several FeatureRequirement annotations are incorrect in gremlin-test

[GitHub] incubator-tinkerpop pull request #330: several FeatureRequirement annotation...

[GitHub] incubator-tinkerpop issue #329: use mockito-core instead of mockito-all to a...

[jira] [Commented] (TINKERPOP-1318) java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

[GitHub] incubator-tinkerpop issue #327: TINKERPOP-1301 Provide Javadoc for ScriptInp...

[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's

[GitHub] incubator-tinkerpop issue #327: TINKERPOP-1301 Provide Javadoc for ScriptInp...

[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

[jira] [Commented] (TINKERPOP-1196) Calls to Result.one() might block indefinitely

[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo

[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...

34 matches

Site Navigation

Mail list logo

Footer information