Hello,
One of the things that I’m working on for TinkerPop 3.3.0 is the ability for
any GraphComputer or GraphActors to work against any Graph. That is,
TinkerGraphComputer over Neo4jGraph, SparkGraphComputer over TinkerGraph,
AkkaGraphActors over HadoopGraph, etc. In order to do this, we needed the
concept of Graph partitions. A Graph partition simply allows you to iterate all
the vertices and edges out of a “partition” (subgraph of the full graph).
Moreover, it has methods to check for the existence of an element in that
partition.
https://github.com/apache/tinkerpop/blob/TINKERPOP-1564/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/Partition.java
<https://github.com/apache/tinkerpop/blob/TINKERPOP-1564/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/Partition.java>
Next, we have the concept of a Partitioner which contains all the Partitions.
https://github.com/apache/tinkerpop/blob/TINKERPOP-1564/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/Partitioner.java
<https://github.com/apache/tinkerpop/blob/TINKERPOP-1564/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/Partitioner.java>
The problem I’m struggling with right now is how to specify a Partitioner. For
instance, right now Graph.partitioner() returns the “default partitioner” for
the Graph. For TinkerGraph, this partitioner has one partition (single a
TinkerGraph is single machine). However, it is possible to create splits so
that you can thread processing — e.g. 5 concurrent threads processing a
TinkerGraph in TinkerGraphComputer. How is this done?
partitioner = new HashPartitioner(graph.partitioner(),5)
This sorta sucks to have to start specifying programmatically such things. I
was thinking it would be nice to have Graph.partitioner() do all the work. For
instance:
Graph.partitioner(Function<Partitioner,Partitioner> partitionerMaker)
The above function would start with the “default partitioner” of the Graph and
then create new partitioners from that. Now, we can have some “non-lambda”
based default functions for people to use.
Graph.partitioner(Maker.splits(5).centricity(Vertex.class))
Anywho…trying to think it through so we have a nice clean API with serializable
and Configuration(able) objects.
Thoughts?,
Marko.
http://markorodriguez.com
