[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715792#comment-15715792 ] Kevin Gallardo commented on TINKERPOP-1565: --- Here's my thoughts on a more broader angle then, I do not disagree with the idea of selecting which kind of serialization you want to have, in order to allow more possibilities wrt to attachment, or lightness, although I would like to keep in sight that the more complex and dynamic the GraphSON format will be, the more complex it will be to deserialize it and consequently more complex to implement new GLVs. If you need to handle 4 different *combinations* of formats (like each kind of element can have a different configuration of detachment) instead of a constant one, it might get frustrating. GraphSON*2* was introduced not only for the types, but also because GraphSON was too inconsistent. Going back to a format where there can be a field "edge" in a "g:Property" or that field could disappear to let place to another "vertexProperty" for which the format is completely different, or the field could simply disappear, or a field named "properties" that can have 2 different schemas in the same enclosing entity, does seem to make things become more inconsistent again. I understand some times the server has to serialize more data to allow reattachment to the parent element or Graph, however I believe that the additional information that could be sent by the server should not modify how the deserialization executes. I'd be advocating for a format that stays constant but can potentially add detachment information, and in case that info is sent by the server, it gets exposed via a extra method such as {{Element.getAttachmentInfo()}} that the deserializer has conveniently filled first via a "attachInfo" fields in the JSON. Then the client would be able to do whatever it wants with that, but the deserializer does not do it itself, because it hasn't got enough knowledge of the context to do so. Concerning properties (Vertices'/Edges or VPs'), the field would either be empty, or filled, in a consistent manner (and for me more consistent would mean it would be typed as either "g:Property" or "g:VertexProperty" -> which both would have a consistent schema). {code}{ @type:”g:Vertex", @detach:”reference", @value: { id:1 } }{code} Note: As explained above the "@detach" would be part of the "@value" and should not alter the type format itself because this is not relevant information at this stage of the deserialization. The type system is domain agnostic and simply allows a Type to be associated with a value. > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez >Assignee: Marko A. Rodriguez > Labels: breaking > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713071#comment-15713071 ] Marko A. Rodriguez commented on TINKERPOP-1565: --- Here is an idea that I had with [~pietermartin-tinkerpop] on gremlin-users. Then I think we need to get smart about “what data?” We sort of already do with {{HaltedTraverserStrategy}} but this is really specific to internal computing and GremlinServer. We could go deeper into this path with: {code} g.withDetachment(…) - Detach.reference // just ids - Detach.reduced // ids, labels, keys, values (basically, what you need for toString()) - Detach.rich // property data included - Detached.full // edge data included (basically, StarGraph) {code} Next, GraphSON would have to specify a “subtype” to the various g:Element types. {code} { @type:”g:Vertex", @detach:”reference", @value: { id:1 } } {code} Then, we could add a method to Element. {code} Element.detachment() -> Detach enum {code} This way, users can always know what is available to them. Finally for cleanliness: {code} Element.detach(Detach) -> Element Element.attach(Graph) -> Element (always a Detach.full element) {code} > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez >Assignee: Marko A. Rodriguez > Labels: breaking > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709204#comment-15709204 ] Marko A. Rodriguez commented on TINKERPOP-1565: --- Huh. Thats a good idea. Yea, right now its hardcoded to just use {{DetachedXXX}}. It would be cool if GraphSONSerializers took a {{Factory}} for creating the detachments -- whether {{DeatchedXXX}} or {{ReferenceXXX}} or .. someting else down the line -- e.g. {{StarXXX}}. > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709172#comment-15709172 ] Kevin Gallardo commented on TINKERPOP-1565: --- But it wouldn't be a ReferenceXXX element that you deserialize when you use {{getOrDefault()}} right? It would just be an empty DetachedXXX, that's the part I struggle to understand. With the additional module, you could say "Everything I deserialize is a DetachedXXX OR everything is ReferenceXXX". And so according to the module used, on the serialization either everything is lengthy and deserialized as Detached, or less lengthy and deserialized as Reference. Also I think it would be useful (at least for me) to define what are the fields considered sufficient for Reference elements > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708781#comment-15708781 ] Marko A. Rodriguez commented on TINKERPOP-1565: --- When Gremlin-Python gets back a {{g:Property}} and creates a {{Property.py}} object, that {{Property.py}} object should have sufficient information to support re-attaching to the source graph. For instance: {code} >>> vertex = g.V().out('created').next() v[4] >>> g.V(vertex).in_().count() 3 {code} This sort of attach-ability is necessary for all elements: Vertex, Edge, VertexProperty, Property. Regarding "detached" vs. "reference" ... The JSON serialization specified has enough information to created "detached elements" though, through the use of {{getOrDefault()}}-style semantics in deserialization, it is possible to only transmit what is sufficient to create a "reference element." > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708770#comment-15708770 ] Marko A. Rodriguez commented on TINKERPOP-1565: --- For 3.3.0 we should get rid of the concept of GraphSON 2.0 untyped (non-embedded types). This distinction is a complication and in fact, shouldn't really be used ever given the GLV direction with GraphSON and serialization format. It should all be typed GraphSON 2.0. > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705921#comment-15705921 ] Marko A. Rodriguez commented on TINKERPOP-1565: --- In the development branch, this is what the serializations look like: *VERTEX* {code} { "@type": "g:Vertex", "@value": { "id": { "@type": "g:Int32", "@value": 1 }, "label": "person", "properties": { "name": [ { "id": { "@type": "g:Int64", "@value": 0 }, "value": "marko" } ], "location": [ { "id": { "@type": "g:Int64", "@value": 6 }, "value": "san diego", "properties": { "startTime": { "@type": "g:Int32", "@value": 1997 }, "endTime": { "@type": "g:Int32", "@value": 2001 } } }, { "id": { "@type": "g:Int64", "@value": 7 }, "value": "santa cruz", "properties": { "startTime": { "@type": "g:Int32", "@value": 2001 }, "endTime": { "@type": "g:Int32", "@value": 2004 } } }, { "id": { "@type": "g:Int64", "@value": 8 }, "value": "brussels", "properties": { "startTime": { "@type": "g:Int32", "@value": 2004 }, "endTime": { "@type": "g:Int32", "@value": 2005 } } }, { "id": { "@type": "g:Int64", "@value": 9 }, "value": "santa fe", "properties": { "startTime": { "@type": "g:Int32", "@value": 2005 } } } ] } } } {code} *VERTEX PROPERTY* {code} { "@type": "g:VertexProperty", "@value": { "id": { "@type": "g:Int64", "@value": 6 }, "value": "san diego", "vertex": { "@type": "g:Int32", "@value": 1 }, "label": "location", "properties": { "startTime": { "@type": "g:Int32", "@value": 1997 }, "endTime": { "@type": "g:Int32", "@value": 2001 } } } } {code} *EDGE* {code} { "@type": "g:Edge", "@value": { "id": { "@type": "g:Int32", "@value": 16 }, "label": "uses", "inVLabel": "software", "outVLabel": "person", "inV": { "@type": "g:Int32", "@value": 11 }, "outV": { "@type": "g:Int32", "@value": 1 }, "properties": { "skill": { "@type": "g:Int32", "@value": 5 } } } } {code} *VERTEX PROPERTY PROPERTY* {code} { "@type": "g:Property", "@value": { "key": "startTime", "value": { "@type": "g:Int32", "@value": 1997 }, "vertexProperty": { "id": { "@type": "g:Int64", "@value": 6 }, "label": "location", "value": "san diego", "vertex": { "@type": "g:Int32", "@value": 1 } } } } {code} *EDGE PROPERTY* {code} { "@type": "g:Property", "@value": { "key": "skill", "value": { "@type": "g:Int32", "@value": 5 }, "edge": { "id": { "@type": "g:Int32", "@value": 16 }, "label": "uses", "inV": { "@type": "g:Int32", "@value": 11 }, "outV": { "@type": "g:Int32", "@value": 1 } } } } {code} > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703419#comment-15703419 ] Kevin Gallardo commented on TINKERPOP-1565: --- Fair enough, if the maintenance issue is deemed not important enough, just trying to make sure we don't make mistake if we go back to that code months later and all that is not fresh in our heads anymore. I understand the argument of the size efficiency/repeated data (even though I already said it was not part of the goals of GraphSON2 otherwise GraphSON2 wouldn't look like that), but it still bothers me though to represent the Same thing, in 2 different ways (on one side VertexProperties as part of a Vertex, the other as a standalone object). Also to expand on the point "3." I mentioned earlier I hoped that the work made on GraphSON2 would allow to get a JSON representation closer to the actual objects as what is serialized in the Gryo protocol so that we could establish some sort of general spec for TinkerPop serialization that would be generic enough to apply to all the TinkerPop serialization protocols and also facilitate the addition of new ones, BUT I'm maybe overreaching there... > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703421#comment-15703421 ] Kevin Gallardo commented on TINKERPOP-1565: --- When you say "supports both DetachXXX and ReferenceXXX", you mean when serializing? > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703279#comment-15703279 ] Marko A. Rodriguez commented on TINKERPOP-1565: --- There are only 4 things to consider -- Vertex, Edge, VertexProperty, and Property. Thus, maintainability of the de/serializers isn't that big of a deal. Next, because there are only 4 things to consider, the code to treat each type individually without cascade is trivial (a morning efforts tweak to what we have now). So regarding JSON size --- {{g:Property}} will have to have reference to its parents (element+vertex ids) to get attached. Thus, "cascading" introduces lots of repeated data. And when you start adding {{@type}} to ids and stuff, this gets big. For instance, if we use "cascading," then when you serialize a vertex, you have: {code} Vertex -> VertexProperties (with ref id back to vertex and label) -> Properties (with ref it back to vertex property and back to vertex and key) {code} If we treat {{Vertex}} independent of the serialization of {{VertexProperty}} and {{Property}} we won't incur repeated data and lots of {{@type}} object nesting. {code} Vertex -> label:[VertexProperties (just id/value/properties)] -> key:[Properties (just value)] {code} Finally with {{getOrCreate()}} optional fields, we don't have to have {{VertexProperty.properties()}} field if it doesn't exist. If the vertex is just a {{ReferenceVertex}}, all it has is id. etc. etc... > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703096#comment-15703096 ] Kevin Gallardo commented on TINKERPOP-1565: --- "I think we should treat each element type individually" So that's another option, as long as all do the same consistently, but why would we do that? Is it only for the size of the payload? The way I see it I find it useful to rely on that "cascade" because: 1. we have to rely on it anyway for the other "non Graph structural" types - 2. it simplifies the deserialization logic - 3. it reflects directly what the actual objects are in the actual implementation of the serialized objects - 4. it may cause inconsistencies otherwise, considering one would change one day the code of the VertexPropertyDeserializer, but do not in the VertexDeserializer, and would need to do it there as well, so all in all, it's a code maintenance inconvenience. The docs say that StarGraph is a format made for size efficiency, which we had said was not to be a concern with the GraphSON2 standard serialization. If that's not the case then fine but there is then a lot of over things we can do to make the GraphSON2 protocol more size efficient, but that was clearly not stated as a goal initially. For example types names are encoded with a descriptive name as well as the "@type" and "@value" identifiers, where all could be smaller but it was decided that we wanted to be more descriptive here, as other future protocols can be made and be much more size efficient. > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702947#comment-15702947 ] stephen mallette commented on TINKERPOP-1565: - well - we did have "beta" in mind, but we have the capability to version our formats. maybe not a big deal to stamp out GraphSON 2.1 as part of 3.3.x. > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702878#comment-15702878 ] Marko A. Rodriguez commented on TINKERPOP-1565: --- I think we should treat each element type individually and not rely on a "cascade" of de/serialization. Thus, the {{VertexPropertySerializer}} isn't used to serialize the vertex properties of a vertex. Its only used when the vertex property is standalone. Likewise for deserialization. We already do this with the graph serialization like {{StarGraph}}: http://tinkerpop.apache.org/docs/3.2.3/dev/io/#_stargraph_2 Finally, I think we should make a {{g:Graph}} type. I don't know why TinkerGraph has its own serialization/deserializer {{tinker:graph}} (thats weird). > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702865#comment-15702865 ] Marko A. Rodriguez commented on TINKERPOP-1565: --- Here is what I think the graph objects should look like: *NOTE*: I did not include typing for the numbers to keep things clear. {code} { "@type":"g:Vertex", "@value": { "label" : "person", "id" : "1", "properties" : { "name" : [ { "id" : 0, "value" : "marko", } ], "location" : [ { "id" : 6, "value" : "san diego", "properties" : { "startTime": 1997, "endTime" : 2001 } } ], } } } {code} The benefits of the above: 1. With {{getOrDefault()}} style selection from JSON, this supports both {{DetachXXX}} and {{ReferenceXXX}}. 2. Properties are consistent. They are just key/value map. No more needed. 3. All the information is there for attaching both {{DetachXXX}} and {{ReferenceXXX}}. 4. This is a very concise representation. > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1565) GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and reducing verbosity
[ https://issues.apache.org/jira/browse/TINKERPOP-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691560#comment-15691560 ] Kevin Gallardo commented on TINKERPOP-1565: --- h5. Reference and Detached Elements: I believe that this can be nicely handled by having a GraphSONModule added to the GraphSONMapper, that replaces all the current serializers and deserializers, with lighter ones, that serializes *all* the Elements as Reference Elements, and deserializes them accordingly to Reference Elements. The GraphSON2Mapper is dynamic enough to allow to override the serializers and deserializers for already existing types like the Graph structure types, so you'd have all the "g:Vertex"/"g:VertexProperty" types mapped to Vertex/VertexProperty but the implem would be Detached Elements instead of the current Detached more heavy ones. But it sounds weird to me if we had in a same response, different formats for the same "g:" type. h5. Consistency with Properties It sounds reasonable that all the classes that derive from Element, have their properties serialized in the same format, for consistency. Since we cannot go around having Vertices' VertexProperties being serialized with their types, then the other Elements (VertexProperty's Properties and Edge Properties) should follow the same path. h5. Attaching deserialized elements I understand the need of the parent info to reattach. However, the comment I made was on the way it was done that could lead to problems. Assuming the code in the PR was *creating a new* Vertex object when deserializing a VertexProperty, based on the Vertex ID info there was in the JSON. So when a VertexProperty is serialized on its own, that's fine. But when a VertexProperty is serialized as part of its parent Vertex (so the JSON is "v + vp1 + vp2 + "), then creating a new Vertex object when deserializing the VertexProperty means that the actual parent object of the VP, and what the VP thinks its parent is, are not the same (Java)objects ({{v.properties('name').next().element() != v}}). That's why I introduced that mechanism in Jackson to indicate to the VP deserializer for example, if the VP to deserialize is on its own or if it's been serialized with its parent. Hope that's clearer > GraphSON 2.0 updates -- supporting attachment, maintaining consistency, and > reducing verbosity > -- > > Key: TINKERPOP-1565 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1565 > Project: TinkerPop > Issue Type: Improvement > Components: io >Affects Versions: 3.2.3 >Reporter: Marko A. Rodriguez > Labels: breaking > Fix For: 3.2.3 > > > GraphSON 2.0 has some issues that should be recified for its "official > release" in 3.3.0. > *Supporting Attachment* > We need to make sure that every element and property can be attached back to > the main graph. > * For {{Vertex}}, this means that we need to have the ID encoded (CHECK). > * For {{Edge}}, this means we need to have the out/in vertex ids encoded > (CHECK). > * For {{VertexProperty}}, this means we need to have the vertex ID encoded > (ERROR). > * For {{Property}}, this means we need to have the the element ID (vertex > property or edge) encoded and then the subsequent vertex ID (ERROR). > *Maintaining Consistency* > Currently, property encoding in {{Edge}} is different than property encoding > in {{VertexProperty}}. > Edge --> > {code} > properties : { > key : "aKey", > value : { @type: "gProperty", @value : {"key","aKey","value",10}} > } > {code} > VertexProperty--> > {code} > properties : { > key : "aKey", > value : 10 > } > {code} > This should be consistent. > *Reducing Verbosity* > We have the problem of representing both {{DetachedElements}} and > {{ReferenceElements}}. {{DetachedElements}} contain lots of information and > is typicaly used to ensure a data rich result set. {{ReferenceElements}} > contain only the necessary information for attachment. I think we can support > both representations by making certain fields "optional." > Vertex--> > {code} > return new > Vertex(json.get("id"),json.getOrDefault("label",Vertex.DEFAULT_LABEL),json.getOrDefault("properties",Collections.emptyMap())) > {code} > That is, lots of {{getOrDefault()}} use will make sure that when desesired, > we only send/recv information that is needed for the particular > computatation. E.g. as dictated by {{HaltedTraverserStrategy}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)