I think that TinkerPop specific types are good and the extension model sounds reasonable. It sounds like it will blend right into the GraphBinary serialization model that we have though I wonder what happens when you start blending together different graphs and serialization extensions and what types of conflicts will occur as a result of that (e.g. two providers using the same serialization "identifiers"). anyway, that's more of a serialization problem than a type problem i suppose, so perhaps a different discussion.
> - There is no way to yield a “v[1]” or “e[3][v[1]-knows->v[2]]” representation. Is that bad? Perhaps not. without that, users are forced to write queries the way we've been professing that they write queries, which is good and will perhaps force better habits. a "reference" really doesn't do much to help an application. that said, could g.V(1) be auto-converted to a Map primitive in TP4 automatically without an explicit valueMap()? maybe that will breed bad habits or allow things to happen that we don't like....just tossing a thought out there. > - What is the nature of a TPath? Its complex, but we want to return it. perhaps there is again a default primitive representation in List form?? I'd also wonder about how we treat subgraph() and tree()? could those be a List<TPath> somehow?? > - g.V().id() on an RDF graph can return a URI. Is a URI “simple”? No, the set of simple types should never grow!…. thus, URI => String. Is that wack? isn't a URI a complex type? that list is expected to grow? maybe all complex types have simple type representations? sorry, if some of these questions/ideas are a bit half-cocked, but i read this really fast and won't be at my laptop for the rest of the day and wanted to get some thoughts out. i'm really really interested in seeing this aspect of TP done "right".... On Mon, Apr 15, 2019 at 8:06 AM Marko Rodriguez <[email protected]> wrote: > Hello, > > I have a consolidated approach to handling data structures in TP4. I would > appreciate any feedback you many have. > > 1. Every object processed by TinkerPop has a TinkerPop-specific > type. > - TLong, TInteger, TString, TMap, TVertex, TEdge, TPath, > TList, … > - BENEFIT #1: A universal type system will protect us from > language platform peculiarities (e.g. Python long vs Java long). > - BENEFIT #2: The serialization format is constrained and > consistent across all languages platforms. (no more coming across a > MySpecialClass). > 2. All primitive T-type data can be directly access via get(). > - TBoolean.get() -> java.lang.Boolean | System.Boolean | > ... > - TLong.get() -> java.lang.Long | System.Int64 | ... > - TString.get() -> java.lang.String | System.String | … > - TList.get() -> java.lang.ArrayList | .. // can only > contain primitives > - TMap.get() -> java.lang.LinkedHashMap | .. // can only > contain primitives > - ... > 3. All complex T-types have no methods! (except those afforded by > Object) > - TVertex: no accessible methods. > - TEdge: no accessible methods. > - TRow: no accessible methods. > - TDocument: no accessible methods. > - TDocumentArray: no accessible methods. // a document > list field that can contain complex objects > - ... > > REQUIREMENT #1: We need to be able to support multiple graphdbs in the > same query. > - e.g., read from JanusGraph and write to Neo4j. > REQUIREMENT #2: We need to make sure complex objects can not be queried > client-side for properties/edges/etc. data. > - e.g., vertices are universally assumed to be “detached." > REQUIREMENT #3: We no longer want to maintain a structure test suite. > Operational semantics should be verified via Bytecode -> > Processor/Structure. > - i.e., the only way to read/write vertices is via > Bytecode as complex T-types don’t have APIs. > REQUIREMENT #4: We should support other database data structures besides > graph. > - e.g., reading from MySQL and writing to JanusGraph. > > ——— > > Assume the following TraversalSource: > > g.withStructure(JanusGraphStructure.class, config1). > withStructure(Neo4jStructure.class, conflg2) > > Now, assume the following traversal fragment: > > outE(’knows’).has(’stars’,5).inV() > > This would initially be written to Bytecode as: > > [[outE,knows],[has,stars,5],[inV]] > > A decoration strategy realizes that there are two structures registered in > the Bytecode source instructions and would rewrite the above as: > > [choose,[[type,TVertex]],[[outE,knows],[has,stars,5],[inV]]] > > A JanusGraph strategy would rewrite this as: > > > [choose,[[type,TVertex]],[[outE,knows],[has,stars,5],[inV]],[[type,JanusVertex]],[[jg:vertexCentric,out,knows,stars,5]]] > > A Neo4j strategy would rewrite this as: > > > [choose,[[type,TVertex]],[[outE,knows],[has,stars,5],[inV]],[[type,JanusVertex]],[[jg:vertexCentric,out,knows,stars,5]],[[type,Neo4jVertex]],[[neo:outE,knows],[neo:has,stars,5],[neo:inV]]] > > A finalization strategy would rewrite this as: > > > [choose,[[type,JanusVertex]],[[jg:vertexCentric,out,knows,stars,5]],[[type,Neo4jVertex]],[[neo:outE,knows],[neo:has,stars,5],[neo:inV]]] > > Now, when a TVertex gets to this CFunction, it will check its type, if its > a JanusVertex, it goes down the JanusGraph-specific instruction branch. If > the type is Neo4jVertex, it goes down the Neo4j-specific instruction branch. > > REQUIREMENT #1 SOLVED > > The last instruction of the root bytecode can not return a complex object. > If so, an exception is thrown. g.V() is illegal. g.V().id() is legal. > Complex objects do not exist outside the TP4-VM. Only primitives can leave > the VM-client barrier. If you want vertex property data (e.g.), you have to > access it and return it within the traversal — e.g., g.V().valueMap(). > BENEFIT #1: Language variant implementations are simple. Just > primitives. > BENEFIT #2: The serialization specification is simple. Just > primitives. (also, note that Bytecode is just a TList of primitives! — > though TBytecode will exist.) > BENEFIT #3: The concept of a “DetachedVertex” is universally > assumed. > > REQUIREMENT #2 SOLVED > > It is completely up to the structure provider to use structure-specific > instructions for dealing with their particular TVertex. They will have to > provide CFunction implementations for out, in, both, has, outE, inE, bothE, > drop, property, value, id, label … (seems like a lot, but out/in/both could > be one parameterized CFunction). > BENEFIT #1: No more structure/ API and structure/ test suite. > BENEFIT #2: The structure provider has full control of where the > vertex data is stored (cached in memory or fetch from the db or a cut > vertex or …). No assumptions are made by the TP4-VM. > BENEFIT #3: The structure provider can safely assume their > vertices will not be accessed outside the TP4-VM (outside the processor). > > REQUIREMENT #3 SOLVED > > We can support TRow for relational databases. A TRow’s data is accessible > via the instructions has, hasKey, value, property, id, ... The location of > the data in TRow is completely up to the structure provider and its > strategy analysis (if only ’name’ is accessed, then SELECT ’name’ FROM...). > We can easily support TDocument for document databases. A TDocument’s data > is accessible via the instructions has, hasKey, value, property, id, … A > value() could return yet another TDocument (or a TDocumentArray containing > TDocuments). > > Supporting a new complex type is simply a function of asking: > > “Does the TP4 VM instruction set have the requisite > instruction-types (semantically) to manipulate this structure?" > > We are no longer playing the language-specific object API game. We are > playing the language-agnostic VM instruction game. The TP4-VM instruction > set is the sole determiner of what complex objects can be processed. (i.e. > what data structures can be processed without impedance mismatch). > > REQUIREMENT #4 SOLVED > > ——— > > The TP4-VM (and, in turn, Gremlin) can naturally support: > > 1. Property graphs: as currently supported in TP3. > 2. RDF graphs: id() is a URI | Literal. g.V(1).value(‘foaf:name’) > returns multi/meta-properties *or* g.V(1).out(‘foaf:name’) returns vertices > whose id()s are xsd:string literals. > 3. Hypergraphs: inV() can return more than one vertex. > 4. Undirected graphs: in() and out() throw exceptions. Only both() > works. > 5. Meta-properties: value(‘name’) can return a TVertexProperty (a > special complex object that is structure provider specific — and that is > okay!). > 6. Multi-properties: value(‘name’) can return a TPropertyArray of > TVertexProperty objects. > > This means that the same instruction can behave differently for different > structures. This is okay as there can be property graph, RDF, hypergraph, > etc. test suites. > > Since complex objects don’t leave the TP4-VM barrier, providers can create > any complex objects they want — they just have to have corresponding > strategies to create provider-unique bytecode instructions (and thus, > CFunctions) for those complex objects. > > Finally. there are a few of problems to work out: > - There is no way to yield a “v[1]” or “e[3][v[1]-knows->v[2]]” > representation. Is that bad? Perhaps not. > - What is the nature of a TPath? Its complex, but we want to > return it. > - g.V().id() on an RDF graph can return a URI. Is a URI “simple”? > No, the set of simple types should never grow!…. thus, URI => String. Is > that wack? > - Do we add g.R() and g.D() to Gremlin to type-support TRow and > TDocument objects. g.V() would be weird :( … Hmmmm? > - However, there are only so many data structures……. or > are there? TMatrix, TXML, …. whoa. > > Thanks for reading, > Marko. > > http://rredux.com <http://rredux.com/> > > > > >
