Hi,
Check this out. Yesterday, I added GraphSONSerializers/Deserializers for
Bytecode. This means that we are able to translate Bytecode to JSON. In order
to test to make sure my serializers/deserializers are good, I created a
Translator called GraphSONTranslator that is only used in our test suite. What
it does is it wraps the JavaTranslator and before sending the Bytecode to the
JavaTranslator, it first writes the Bytecode to GraphSON, then reads it from
GraphSON and then gives it to JavaTranslator. This is run against our complete
ProcessStandard and ProcessComputer suite. There are a few hiccups (failed
tests), but I know why they are failing and it has to do either with
Long/int-stuff and/or using Elements as arguments — e.g. g.V(g.V(1).next()).
What is neat though is that the serializer/deserializer code is simple and just
like that, Bytecode is language agnostic.
https://github.com/apache/tinkerpop/blob/c60e2132a6feb76ddf72073614b60dfa824193b4/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/graphson/GraphSONTraversalSerializers.java
<https://github.com/apache/tinkerpop/blob/c60e2132a6feb76ddf72073614b60dfa824193b4/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/graphson/GraphSONTraversalSerializers.java>
https://github.com/apache/tinkerpop/blob/c60e2132a6feb76ddf72073614b60dfa824193b4/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/java/translator/GraphSONTranslator.java#L69-L73
<https://github.com/apache/tinkerpop/blob/c60e2132a6feb76ddf72073614b60dfa824193b4/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/java/translator/GraphSONTranslator.java#L69-L73>
Neato… now we just need to have a standard GraphSON representation….
GraphSON2.0 is important for this work.
Marko.
http://markorodriguez.com
> On Jul 6, 2016, at 3:23 PM, Marko Rodriguez <[email protected]> wrote:
>
> Hi,
>
>> Hi Marko, this is neat stuff. Do you have any designs on making the
>> "ByteCode" construct the source of truth for Gremlin semantics?
>
> Yes and no. Bytecode will have two “standard serializations” (GraphSON and
> Gryo). From there, any gremlin-xxx will have a “new Bytecode(json)”-style
> constructor to create a Bytecode object in the respective language. From
> there, the respective language can construct a traversal object as it sees
> fit (either via reflection or via script string creation). For instance:
>
> JavaTranslator uses reflection:
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/util/JavaTranslator.java
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/util/JavaTranslator.java>
> GroovyTranslator uses script string:
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/java/translator/GroovyTranslator.java
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/java/translator/GroovyTranslator.java>
>
> Next, the only reason you would need to use GroovyTranslator is because of a
> lambda. If the Bytecode you submit to GremlinServer (or any RemoteConnection
> implementation) doesn’t have lambdas in it, it would be more efficient to use
> JavaTranslator and thus, bypass the overhead of String compilation and Groovy
> dispatch-based meta-classes.
>
>> That is to say, generating an AST is nice but I would be hesitant to
>> build and support a non-JVM implementation of Gremlin unless the
>> semantics of the bytecode was specified and the JVM implementation was
>> simply the reference implementation.
>
> So the semantics of the byte code are specified by the semantics of the
> step(arguments).
>
> out(“created”)
> V()
> path()
> etc.
>
> TinkerPop’s ProcessStandardSuite and ProcessComputerSuite verify the
> semantics of every step and thus, can be used to verify that the Bytecode
> generated in another language (e.g. gremlin-python) has valid semantics. Here
> is how I verify Gremlin-Python generated Bytecode.
>
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/test/java/org/apache/tinkerpop/gremlin/java/translator/jython/PythonJythonTranslatorProvider.java
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/test/java/org/apache/tinkerpop/gremlin/java/translator/jython/PythonJythonTranslatorProvider.java>
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/test/java/org/apache/tinkerpop/gremlin/java/translator/jython/PythonJythonTranslatorProcessStandardTest.java
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/test/java/org/apache/tinkerpop/gremlin/java/translator/jython/PythonJythonTranslatorProcessStandardTest.java>
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/test/java/org/apache/tinkerpop/gremlin/java/translator/jython/PythonJythonTranslatorProcessComputerTest.java
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/test/java/org/apache/tinkerpop/gremlin/java/translator/jython/PythonJythonTranslatorProcessComputerTest.java>
>
> Moreover, via some trickery, it would be possible to use TinkerPop’s same
> test infrastructure to verify the Gremlin-Python native traversals (via
> step-code implementations in Python) are valid.?? would have to think on
> that…. :/
>
> The general thinking is this:
>
> Graph -> data in the respective VM (e.g. JVM, CPython, V8, etc.).
> Traversal -> program in the respective VM.
> Steps -> machine code in the respective VM (think assembly)
> Bytecode -> virtual machine code for consumption by any VM
> (think Java bytecode)
>
> If a traversal’s evaluation does not require a translation (e.g. no
> RemoteStrategy), then the Bytecode does nothing. In fact, Gremlin-Java
> Traversals generate both “machine code” and “bytecode” during construction.
> Thus, there is no need for a runtime or JIT interpreter. This costs memory as
> you have two representations of the traversal, but it speeds up execution
> (—traversals are not large like programs in a programming language so their
> size is relatively small and thus worth the memory cost).
>
>> As you suggest it'd be
>> significant work to implement the standard steps in, e.g., Python;
>> it'd also be significant work to keep up with the evolution of the JVM
>> implementation.
>
> Yes. It would take some dedication. However, the beauty is that if this were
> done then Gremlin-XXX and respective graph databases/processors written in
> XXX language can intercommunicate with Gremlin-YYY via Bytecode. That is,
> Gremlin-Java traversals can talk to a Gremlin-Python graph database.
> Gremlin-Python traversals can talk to a Gremlin-JavaScript graph database (or
> a Gremlin-Go graph database (thinking Google’s Cayley), etc.).
>
> The cool thing about Gremlin is that its really simple to implement a
> respective VM in a programming language. There really aren’t that many moving
> parts. In fact, Gremlin-Python is currently stubbed so that it can, if
> someone was so inclined, be extended with step-code (“machine code”) and not
> just bytecode creation.
>
> RemoteGraph only right now — Gremlin-Python can only talk to graph
> databases/processors outside of Python (i.e. JVM-based systems)
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/remote_graph.py
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/remote_graph.py>
> TraversalStrategies
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/traversal_strategies.py
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/traversal_strategies.py>
> TraversalStrategy
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/traversal_strategy.py
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/traversal_strategy.py>
> RemoteStrategy
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/remote_strategy.py
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/remote_strategy.py>
> Bytecode
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/bytecode.py
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/bytecode.py>
>
> …as you can see Gremlin-Python is mirroring Gremlin-Java…. Pretty neat eh?
>
> Finally, check out this Python session using Gremlin-Python to talk to
> GremlinServer (JVM).
>
> https://gist.github.com/okram/53d4eae84ab1c07b5d69b99e7432eaa1
> <https://gist.github.com/okram/53d4eae84ab1c07b5d69b99e7432eaa1>
> See line #28 and #31 … the Gremlin-Python API is identical to
> Gremlin-Java.
>
> If we keep this pattern going, then Gremlin will nicely extend to all
> languages… its pretty trivial to simply provide a Bytecode traversal in
> Gremlin-XXX. Not much code in Gremlin-Python:
>
>
> https://github.com/apache/tinkerpop/tree/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython
>
> <https://github.com/apache/tinkerpop/tree/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython>
>
> …where GraphTraversal is source generated via Java reflection.
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/groovy/org/apache/tinkerpop/gremlin/python/GraphTraversalSourceGenerator.groovy
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/groovy/org/apache/tinkerpop/gremlin/python/GraphTraversalSourceGenerator.groovy>
> — creates —>
>
> https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/graph_traversal.py
>
> <https://github.com/apache/tinkerpop/blob/88b266c9224e3d19cc28e1c4935416c1fb26e9cb/gremlin-python/src/main/jython/gremlin_python/graph_traversal.py>
> … so fat fingering all that stuff in is not necessary.
>
> Easy peasy.
>
> Any who, hope that provides a good rundown of this line of work and where we
> could take it.
>
> Take care,
> Marko.
>
> http://markorodriguez.com <http://markorodriguez.com/>
>
>
>
>
>
>
>
>>
>> Cheers,
>> --
>> b
>>
>> On Sat, Jul 2, 2016 at 10:25 AM, Marko Rodriguez <[email protected]
>> <mailto:[email protected]>> wrote:
>>> HI,
>>>
>>> For fun, I created a play "toString()” that emits a JSON representation of
>>> ByteCode.
>>>
>>> g.V().as("a").out().repeat(both("created")).times(10).as("b").values("name”).range(0,2).select("a",
>>> "b").by("name”)
>>>
>>> becomes —>
>>>
>>> [["withComputer",[]],["V",[]],["as",["a"]],["out",[]],["repeat",[[["both",["created"]]]]],["times",[10]],["as",["b"]],["values",["name"]],["range",[0,2]],["select",["a","b"]],["by",["name"]]]
>>>
>>>
>>> which in “pretty print” JSON is:
>>>
>>> [
>>> [
>>> "withComputer",
>>> []
>>> ],
>>> [
>>> "V",
>>> []
>>> ],
>>> [
>>> "as",
>>> [
>>> "a"
>>> ]
>>> ],
>>> [
>>> "out",
>>> []
>>> ],
>>> [
>>> "repeat",
>>> [
>>> [
>>> [
>>> "both",
>>> [
>>> "created"
>>> ]
>>> ]
>>> ]
>>> ]
>>> ],
>>> [
>>> "times",
>>> [
>>> 10
>>> ]
>>> ],
>>> [
>>> "as",
>>> [
>>> "b"
>>> ]
>>> ],
>>> [
>>> "values",
>>> [
>>> "name"
>>> ]
>>> ],
>>> [
>>> "dedup",
>>> []
>>> ],
>>> [
>>> "select",
>>> [
>>> "a",
>>> "b"
>>> ]
>>> ],
>>> [
>>> "by",
>>> [
>>> "name"
>>> ]
>>> ]
>>> ]
>>>
>>>
>>> Which then via gets translated into Python via PythonTranslator as:
>>>
>>> g.V().as('a').out().repeat(both('created')).times(10).as('b').name[0,2].select('a','b').by('name’)
>>>
>>>
>>> Pretty simple…. once we get a standard ?JSON? serialization format for
>>> ByteCode, then we can simply have one Translator for each Gremlin variant —
>>> ByteCode -> Gremlin-XXX. What we need to do proper is typing — long, String,
>>> double, float, P, Order, Lambda, etc. Once that is square we should be good.
>>>
>>> Take care,
>>> Marko.
>>>
>>> http://markorodriguez.com <http://markorodriguez.com/>
>>>
>>>
>>>
>>> On Jul 2, 2016, at 8:04 AM, Marko Rodriguez <[email protected]> wrote:
>>>
>>>
>>> Hello,
>>>
>>> So TINKERPOP-1278 (aka “Gremlin-Python”) has introduced the notion of
>>> Traversal ByteCode.
>>>
>>> https://github.com/apache/tinkerpop/blob/c34fe9fc2cee8bf231c7d4a2d52e746053273129/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/ByteCode.java
>>>
>>> In essence, ByteCode is the construction history of a traversal and is of
>>> the form:
>>>
>>> [string, object*]* // a list of (operator, arguments[])
>>>
>>> The traversal
>>>
>>> g.V(1).outE(‘created’).inV().
>>> repeat(out(‘created’).in(‘created’)).times(5).
>>> valueMap(’name’,’age’)
>>>
>>> has a ByteCode representation as below:
>>>
>>>
>>> [
>>> [V, 1]
>>> [outE, ‘created’]
>>> [inV]
>>> [repeat, [
>>> [out, ‘created’]
>>> [in, ‘created’]
>>> ]
>>> [times, 5]
>>> [valueMap, name, age]
>>> ]
>>>
>>>
>>> Again, Gremlin is a simple language based on function concatenation and
>>> nesting. Thats all there is to it. Thus, it forms a tree and trees are easy
>>> to encode, distribute, serialize, decode, prune/optimize, search, etc.
>>> Moreover, every programming language supports function composition and
>>> nesting and thus, Gremlin is able to be hosted in any programming language.
>>> [http://tinkerpop.apache.org/gremlin.html]
>>>
>>> The benefit of ByteCode as it applies to TINKERPOP-1278 is that a Translator
>>> is able to access the ByteCode of the traversal and then use that
>>> linear-nested structure (wide-tree) to generate a traversal representation
>>> in another language — e.g. Gremlin-Python, Gremlin-Ruby, Gremlin-JavaScript,
>>> etc.
>>>
>>> Here is the Gremlin-Python translator that will turn ByteCode into
>>> Gremlin-Python:
>>> https://github.com/apache/tinkerpop/blob/c34fe9fc2cee8bf231c7d4a2d52e746053273129/gremlin-python/src/main/java/org/apache/tinkerpop/gremlin/java/translator/PythonTranslator.java
>>> Here is the Gremlin-Groovy translator that will turn ByteCode into
>>> Gremlin-Groovy:
>>> https://github.com/apache/tinkerpop/blob/c34fe9fc2cee8bf231c7d4a2d52e746053273129/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/java/translator/GroovyTranslator.java
>>>
>>> Pretty simple, eh? So, why would you want Gremlin-Java to translate to
>>> Gremlin-Groovy? Well, so you can code in Gremlin-Java and then have it
>>> execute on GremlinServer via the GremlinGroovy JSR223 ScriptEngine. However,
>>> one can imagine a Gremlin-Java->Gremlin-Java translator! What is that?!
>>> Well, it would use reflection (or some more efficient mechanism) to
>>> reconstruct the Gremlin-Java traversal from ByteCode generated from
>>> Gremlin-Java and thus, the entire cluster/sever infrastructure is simply
>>> migrating ByteCode around as opposed to worrying about language specific
>>> representation — e.g. it has nothing to do with the JVM! Also, assume a
>>> Python-based graph database exists that implements the GreminVM —
>>> Gremlin-Java can easily talk to it via ByteCode.
>>>
>>> To ensure ByteCode generation is not costly, here are the runtimes for
>>> construction and compilation of a “fairly complex” traversal in both master/
>>> and TINKERPOP-1278/
>>>
>>> master/
>>>
>>> gremlin>
>>> clock(5000){g.V().repeat(out()).times(2).as("a").union(both(),out()).dedup().as("b").select("a","b").identity()
>>> }
>>> ==>0.004184923
>>> gremlin>
>>> clock(5000){g.V().repeat(out()).times(2).as("a").union(both(),out()).dedup().as("b").select("a","b").identity().applyStrategies()
>>> }
>>> ==>0.0264354126
>>>
>>>
>>> TINKERPOP-1278/
>>>
>>> gremlin>
>>> clock(5000){g.V().repeat(out()).times(2).as("a").union(both(),out()).dedup().as("b").select("a","b").identity()
>>> }
>>> ==>0.0048526357999999995
>>> gremlin>
>>> clock(5000){g.V().repeat(out()).times(2).as("a").union(both(),out()).dedup().as("b").select("a","b").identity().applyStrategies()
>>> }
>>> ==>0.0245924514
>>>
>>>
>>> Finally, there are various entailments that come from this ByteCode
>>> representation that have started to surface in my mind:
>>>
>>> 1. We can now optimize at the ByteCode level and not at the step level —
>>> ByteCodeStrategies could be a new TraversalStrategy class that comes before
>>> DecorationStrategies.
>>> 2. Gremlin-Java can support bindings so that its Gremlin-Groovy (e.g.)
>>> compiled form uses bindings. How? By simply rewriting the ByteCode prior to
>>> compilation and replacing values with variables!
>>> 3. GraphSON can now easily support Traversal serialization — ByteCode in
>>> JSON is natural. Gremlin-GraphSON anyone? (get it?)
>>> [g, V : [1], outE : [created], inV : repeat : [out : [created], in :
>>> [created]], times : 5], valueMap : [name, age]]
>>>
>>> This is what makes Gremlin so powerful — its syntax is crazy simple as its
>>> just functions and thus, it can naturally exist in any language — even XML!
>>> … ByteCode is what is going to free Gremlin from the JVM…as we are already
>>> now on the CPython VM with relative ease:
>>>
>>> PythonGraphTraversal
>>> https://github.com/apache/tinkerpop/blob/c34fe9fc2cee8bf231c7d4a2d52e746053273129/gremlin-python/src/main/jython/gremlin_python/graph_traversal.py
>>> JythonTranslator (Gremlin-Python to Gremlin-Jython)
>>> https://github.com/apache/tinkerpop/blob/c34fe9fc2cee8bf231c7d4a2d52e746053273129/gremlin-python/src/main/jython/gremlin_python/jython_translator.py
>>> GroovyTranslator (Gremlin-Python to Gremlin-Groovy)
>>> https://github.com/apache/tinkerpop/blob/c34fe9fc2cee8bf231c7d4a2d52e746053273129/gremlin-python/src/main/jython/gremlin_python/groovy_translator.py
>>>
>>> *** NOTE: I have yet to introduce the ByteCode concepts to gremlin-python/
>>> package so use your imagination when seeing how PythonGraphTraversal will
>>> construct ByteCode and the respective translators will translate it.
>>> Finally, realize that people can now compile Python ByteCode into
>>> Python-based steps and thus, Gremlin can live and execute on the Python VM
>>> against Python-based graph systems. Its really that easy (though, lots of
>>> work to implement the 30 some standard steps in Python). What this means
>>> though is that, in the future, Gremlin can just move between languages/VMs …
>>> between Python, JVM, Ruby, JavaScript, C, etc.-based graph processing
>>> systems. One language, tailored to its host, and agnostic to the underlying
>>> virtual machine.
>>>
>>> Enjoy,
>>> Marko.
>>>
>>> http://markorodriguez.com
>>>
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Gremlin-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/gremlin-users/C8DF5760-9CC0-41A4-9CE6-3BA025CF4BE6%40gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Gremlin-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected]
>> <mailto:[email protected]>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/gremlin-users/CABQ%3D58QNnFCesip_3mRbmLdU6UnRwkAYmJ8Q-ObdwEV4DHFKgA%40mail.gmail.com
>>
>> <https://groups.google.com/d/msgid/gremlin-users/CABQ%3D58QNnFCesip_3mRbmLdU6UnRwkAYmJ8Q-ObdwEV4DHFKgA%40mail.gmail.com>.
>> For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>
