Hi,
> I think we've tried a lot of things for TinkerPop 3 and that there are too
> many options for application development as well as an over-exposed API, so I
> like this direction, though a fair bit of complexity lies in making it
> happen. Before getting to that complexity, I don't really think of this as a
> "new idea". To me the description of gremlin-x is really just a GLV for Java.
> I mean - consider that gremlin-python (something we classify as a GLV) is
> basically "everything a python dev needs to build applications - nothing
> more/less" and that's pretty much what we want for java in gremlin-x.
> gremlin-x would basically be "everything a java dev needs to build
> applications - nothing more/less". So I'm not seeing much distinction there
> (nor the need for naming it "gremlin-x" really - seems to be as simple as
> "gremlin-java" to be in line with other GLVs we will have).
Ahhhhhhhhhhhhhhhh… Okay, then there is no such thing as gremlin-core/ or
gremlin-server/. Check out my packaging and subsequent reasoning:
gremlin-java/
structure/
io/
process/
remote/
client/
server/
language/
graph/
machine/
traversal/
steps/
strategies/
computer/
distributed/ **forthcoming**
And now, check out gremlin-python (as it currently stands):
gremlin-python/
structure/
io/
process/
remote/
client/
language/
graph/
Take home points:
1. Gremlin-Java has a language, machine, server, and client
implementation.
2. Gremlin-Python only has a language implementation and a remote
client implementation.
3. remote/client is RemoteConnection and remote/server is RemoteServer
(making up that interface for now).
4. We make more explicit the distinction between traversal machine and
traversal language and now you know which variants have machine implementations.
5. In the future, Gremlin-Python could have a machine implementation
and then would be native for Python-based graph systems.
- Thus, easy to grow TinkerPop into other languages.
In this way, if you are a graph system provider with a system written in
language XXX, then you come to TinkerPop and ask:
1. Does gremlin-XXX exist?
2. If so, does gremlin-XXX have a machine/ implementation?
3. If so, then implement gremlin-XXX structure API and you are now
TinkerPop-enabled.
In this way, if you are a graph system users with an application written in
language XXX, then you come to TinkerPop and ask:
1. Does gremlin-XXX exist?
2. If so, does gremlin-XXX have a language/ implementation?
3. If so, then code in gremlin-XXX over any TinkerPop-enabled graph
system.
Finally, this makes making gremlin-test/ more language agnostic all the more
important. If all the gremlin-XXX languages can be verified “natively” using
gremlin-test/ (not via ScriptEngine emulation), then we are super cool.
Finally, cause the last finally wasn’t enough, I would make it so
gremlin-driver is in gremlin-java/ and I would keep gremlin-console/ as its own
thing because, while it is a Java-based REPL, it is language agnostic given the
current work by Stephen to enable any ScriptEngine (not just Gremlin-Groovy) to
be leveraged in Gremlin Console.
Boom.
Marko.
>
> Back to the "complexity" - and I think the questions in my mind that follow
> apply with or without this gremlin-x idea (they are general issues that just
> are faced by "remote" and GLVs):
>
> + I'll come back to this in the points below, but as a big fat general
> question: how will we deal with transactions?
> + What will be the nature of Graph.io() - in one sense i'd like to see that
> as something hidden as it is the source of a fair bit of confusion (i.e.
> someone trying to load a multi GB graphson file) but on the other hand it's
> convenient for small graphs - that story could be made better. Maybe
> Graph.io() being a part of "Graph" is hidden and only used by the test suite
> to load data into graph providers implementations for testing purposes. Of
> course, we will need our story in order for loading data in different formats
> if that isn't available (which I guess we need to do anyway).
> + Can remoting Gremlin bytecode cover everything that users currently do in
> embedded/local mode? The drawback with a remote traversal is that it is bound
> to a single transaction and you therefore lose some efficiency with certain
> graphs databases as logic with three traversals that might span a single
> transaction, for example, will be executed less efficiently as three
> different transaction refreshing a transaction cache each time. Maybe the
> server protocol needs to be session based? Maybe, going back to the first
> bullet i had, the transaction model needs to change so that its not bound to
> a single thread by default?
> + Related to the last point, is there an easy way to blend client code with
> server code? Meaning, if i have a body of complex logic right now, my only
> option is to submit a script. I think the ScriptEngine stuff should really
> just be used for those situations where you need a lambda. That's a nice,
> simple, easy-to-explain rule, whereas "do it if there is complex logic" is a
> bit more subjective (and potentially troublesome). How can GLVs get complex
> logic on the "server" (let's say the logic is written in java and available
> to gremlin server) so that it could be executed as part of some bytecode in
> any GLV?
> + It would be neat to see what kind of "server" changes we could make to
> optimize for bytecode. Right now we sorta tied bytecode execution into the
> same Gremlin Server execution pipeline as ScriptEngine processing. I think we
> could probably do better on that.
>
> Not sure there are any immediate answers to any of the above, but it's what
> I've been thinking about with respect to TinkerPop 3.3.0 lately.
>
>
>
> On Wed, Nov 30, 2016 at 1:11 PM, pieter-gmail <pieter.mar...@gmail.com
> <mailto:pieter.mar...@gmail.com>> wrote:
> Ok, if Sqlg just implements its own RemoteConnection and no need for an
> additional server in the architecture then I am a lot less nervous about
> gremlin-x.
>
> As an aside, Sqlg has its own peculiarities and optimization outside the
> standard TinkerPop interface. Most providers will have such features. If
> the graph is not exposed directly there will need to be some way to call
> custom features. No idea how now, but still.
>
> E.g. for Sqlg the most used such feature is
> SqlgGraph.tx().batchMode(batchModeType) but there are others. Index
> management will be another. Without being able to access these features
> too many benefits of choosing a particular provider will be lost.
>
> Cheers
> Pieter
>
> On 30/11/2016 19:31, Marko Rodriguez wrote:
> > Hi,
> >
> >> I generally recommend the exact opposite. Neo4j shines when embedded.
> >
> > And thats fine. There would simply be a EmbeddedRemoteConnection which
> > is basically just a proxy that does nothing. No need for server stuff.
> >
> >> Sqlg already works of databases that is designed for remote access.
> >
> > And yes, thats where a full fledged RemoteConnection (like
> > GremlinServer’s) is needed for marshaling data over a network.
> >
> >> Wrapping that with yet another server is an unnecessary complication.
> >
> > Its not about GremlinServer so much as its about RemoteConnection. If
> > Sqlg has their own RemoteConnection implementation, then it uses that.
> > Likewise, Neo4jServer could provide its own RemoteConnection
> > implementation and thus, no need for GremlinServer.
> >
> >> (Not having used it I do not know the usability or performance impact,
> >> however being another layer it can only degrade performance and
> >> complexity).
> >
> > For embedded, its an implementation that just returns the traversal as
> > was submitted.
> > For remote, its as it is now. You have to send the bytecode over the
> > network and stream back results.
> >
> >> Further because Sqlg already works of a remote database it supports many
> >> jvm's directly accessing the same graph. I imagine other providers could
> >> have the same feature.
> >
> > This is fine with RemoteConnection. What is “not fine” (the reason for
> > gremlin-x) is to make it so users can’t talk with the Graph object.
> > That is a Java thing and should not be exposed to users. More
> > specifically, that is a Gremlin traversal machine <-> graph system
> > provider interface. Users only interact with a RemoteConnection
> > implementation.
> >
> >> So gremlin-x is fine but it is not the obvious best choice for all
> >> architectures.
> >
> > I was shootin’ this idea around DSEGraph people and Bob was like:
> > “pfff…where was this idea 4 years ago. Everyone is already addicting
> > to sending a Gremlin-Groovy script over the network and getting a
> > ResultSet back.”
> >
> >> All the above is for happy java devs.
> >
> > Again, the purpose is the constrain Java developers too. Things like
> > this should never happen:
> >
> > v1.property(‘name’).value()
> > v1.edges(OUT).next()
> > cluster.submit(“1+2”)
> > graph = GraphFactory.open(...)
> > graph.io <http://graph.io/> <http://graph.io
> > <http://graph.io/>>(gryo()).readGraph(‘data/blah.gryo’)
> >
> >
> >
> > It should be:
> >
> > g.V(1).values(‘name’)
> > g.V(1).outE().next()
> > g.inject(1,2).sum()
> > g.withRemote(config)
> > // hmmm… don’t know about the last one.
> >
> >
> > See ya,
> > Marko.
> >
> >
> >> Cheers
> >> Pieter
> >>
> >> On 30/11/2016 16:06, Marko Rodriguez wrote:
> >>> Hello,
> >>>
> >>> I think TinkerPop has too many ways of doing things. For instance:
> >>>
> >>> 1. Users can use graph traversal or interact with vertex/edge/etc.
> >>> Java objects directly.
> >>> 2. Users can interact with the graph system embedded (via the Graph
> >>> object) or remotely (via RemoteConnection).
> >>> 3. Users can submit Gremlin string scripts or submit Gremlin
> >>> bytecode to GremlinServer (along with other REST/etc. type
> >>> capabilities possible).
> >>> 4. Users can use GraphML, GraphSON, or Gryo for serialization.
> >>> 5. Users can use Gremlin-Groovy, Gremlin-Java, Gremlin-Python,
> >>> Gremlin-Scala, etc.
> >>> 6. Users can use Titan, Neo4j, OrientDB, etc.
> >>>
> >>> I think we should create a module called gremlin-x/ which is the
> >>> most simplified, concise, recommended way to use TinkerPop. In
> >>> correspondence to the itemization above:
> >>>
> >>> 1. Users can only use graph traversal. All returned elements are
> >>> ReferenceElements.
> >>> 2. Users can only interact with the graph remotely (where remote may
> >>> just be a localhost connection — in-memory proxy (e.g. for
> >>> TinkerGraph)).
> >>> 3. Users can only submit Gremlin bytecode to GremlinServer (thus,
> >>> only RemoteConnection can be used). No more security issues, no more
> >>> uninterruptible scripts, etc.
> >>> 4. Users can only use GraphSON as its language agnostic.
> >>> 5. Users can use any Gremlin language variant as long is generates
> >>> Gremlin bytecode.
> >>> 6. Users can use any graph system provider as long as it has a
> >>> RemoteConnection exposed (where the provider can simply leverage
> >>> GremlinServer).
> >>>
> >>> Finally, we should create a separate documentation called
> >>> gremlin-x.asciidoc which is small and discusses the rationale for
> >>> the constrained specification and references the core reference docs
> >>> accordingly. For instance, no need to have all the step examples
> >>> re-written.
> >>>
> >>> I believe that this will make it easier for users to say the following:
> >>>
> >>> “Okay, I get it. Open a RemoteConnection from a GraphTraversalSource
> >>> (g.withRemote(...)) and spawn traversals and get back results. Thats
> >>> it?! Easy peasy lemon squeezy.”
> >>>
> >>> I believe that this will make it easier for system developers to say
> >>> the following:
> >>>
> >>> “We can cover 95% of use cases by simply exposing a
> >>> RemoteConnection. Thats it?! Easy peasy lemon squeezy."
> >>>
> >>> In general, this model looks at TinkerPop as a “remote database
> >>> system” where users submit “queries” and get back “results.” That is
> >>> it. Nothing Java-specific, nothing with executing arbitrary scripts
> >>> remotely, nothing with REST, nothing with embedded (at least from a
> >>> look-and-feel perspective), etc.
> >>>
> >>> Thoughts?,
> >>> Marko.
> >>>
> >>> http://markorodriguez.com <http://markorodriguez.com/>
> >>>
> >>>
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "Gremlin-users" group.
> >> To unsubscribe from this group and stop receiving emails from it,
> >> send an email to gremlin-users+unsubscr...@googlegroups.com
> >> <mailto:gremlin-users%2bunsubscr...@googlegroups.com>
> >> <mailto:gremlin-users+unsubscr...@googlegroups.com
> >> <mailto:gremlin-users%2bunsubscr...@googlegroups.com>>.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/gremlin-users/6f8790d7-9fa8-99c8-b553-1793c42f9c04%40gmail.com
> >>
> >> <https://groups.google.com/d/msgid/gremlin-users/6f8790d7-9fa8-99c8-b553-1793c42f9c04%40gmail.com>.
> >> For more options, visit https://groups.google.com/d/optout
> >> <https://groups.google.com/d/optout>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Gremlin-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to gremlin-users+unsubscr...@googlegroups.com
> > <mailto:gremlin-users%2bunsubscr...@googlegroups.com>
> > <mailto:gremlin-users+unsubscr...@googlegroups.com
> > <mailto:gremlin-users%2bunsubscr...@googlegroups.com>>.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/gremlin-users/50D34645-5AF8-4683-8C1F-E01A3C4F7815%40gmail.com
> >
> > <https://groups.google.com/d/msgid/gremlin-users/50D34645-5AF8-4683-8C1F-E01A3C4F7815%40gmail.com>
> > <https://groups.google.com/d/msgid/gremlin-users/50D34645-5AF8-4683-8C1F-E01A3C4F7815%40gmail.com?utm_medium=email&utm_source=footer
> >
> > <https://groups.google.com/d/msgid/gremlin-users/50D34645-5AF8-4683-8C1F-E01A3C4F7815%40gmail.com?utm_medium=email&utm_source=footer>>.
> > For more options, visit https://groups.google.com/d/optout
> > <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gremlin-users+unsubscr...@googlegroups.com
> <mailto:gremlin-users%2bunsubscr...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gremlin-users/1b1639d8-4c1d-ed65-5218-8c8786b76c97%40gmail.com
>
> <https://groups.google.com/d/msgid/gremlin-users/1b1639d8-4c1d-ed65-5218-8c8786b76c97%40gmail.com>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gremlin-users+unsubscr...@googlegroups.com
> <mailto:gremlin-users+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gremlin-users/CAA-H4390oFKxksw1bxDSO7Ej83NBes6NaBD_EBG77wD6QC786Q%40mail.gmail.com
>
> <https://groups.google.com/d/msgid/gremlin-users/CAA-H4390oFKxksw1bxDSO7Ej83NBes6NaBD_EBG77wD6QC786Q%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
