[Fwd: Re: Local children barrier advice needed]

[pieter]

Damm only sent it to Marko, here it is again to the mail list.

--- Begin Message ---

Hi

Here is an example with global children and the batching works well.
Sqlg does not currently optimize the 'where' (TraversalFilterStep).

    @Test
    public void testBatchingIncomingTraversersOnVertexStep() {
        int count = 10_000;
        for (int i = 0; i < count; i++) {
            Vertex a1 = this.sqlgGraph.addVertex(T.label, "A");
            Vertex b1 = this.sqlgGraph.addVertex(T.label, "B");
            a1.addEdge("ab", b1);
        }
        this.sqlgGraph.tx().commit();

        GraphTraversal traversal = this.sqlgGraph.traversal()
                .V().where(__.hasLabel("A"))
                .out();
        printTraversalForm(traversal);
        List<Vertex> vertices = traversal.toList();
        assertEquals(count, vertices.size());
    }

This prints out,

pre-strategy:[GraphStep(vertex,[]), 
TraversalFilterStep([HasStep([~label.eq(A)])]), VertexStep(OUT,vertex)]
post-strategy:[SqlgGraphStepCompiled(vertex,[])@[sqlgPathFakeLabel], 
HasStep([~label.eq(A)]), SqlgVertexStepCompiled@[sqlgPathFakeLabel]]

The SqlgVertexStepCompiled is able to iterate all 10 000 incoming traversers 
and execute one query for the out().

This reduced the query time from 12 seconds to 0.4 seconds. Happiness!!

An example with a local traversal.

    @Test
    public void testBatchingIncomingTraversersOnLocalVertexStep() {
        int count = 10_000;
        for (int i = 0; i < count; i++) {
            Vertex a1 = this.sqlgGraph.addVertex(T.label, "A");
            Vertex b1 = this.sqlgGraph.addVertex(T.label, "B");
            a1.addEdge("ab", b1);
        }
        this.sqlgGraph.tx().commit();

        GraphTraversal traversal = this.sqlgGraph.traversal()
                .V().hasLabel("A")
                .local(
                        __.out()
                );
        printTraversalForm(traversal);
        List<Vertex> vertices = traversal.toList();
        Assert.assertEquals(count, vertices.size());
    }

This prints out,

pre-strategy:[GraphStep(vertex,[]), HasStep([~label.eq(A)]), 
LocalStep([VertexStep(OUT,vertex)])]
post-strategy:[SqlgGraphStepCompiled(vertex,[])@[sqlgPathFakeLabel], 
LocalStep([SqlgVertexStepCompiled@[sqlgPathFakeLabel]])]

In this case SqlgVertexStepCompiled is a local traversal of the LocalStep.

Iterating the starts only returns one traverser as the LocalStep only puts one 
on the traversal at a time.

I suppose I can replace LocalStep with a custom one but there are many steps 
with local children which will make things
fragile if I were to replace so many steps in a copy paste fashion.

Thanks
Pieter


On Thu, 2017-04-20 at 09:10 -0600, Marko Rodriguez wrote:
> Hello,
> 
> > I have started optimizing Sqlg to do a bulk/barrier for its
> > VertexStep
> > optimizations.
> 
> Cool.
> 
> > Sqlg has two optimization strategies.
> > 
> > GraphStepStrategy and VertexStepStrategy. GraphStepStrategy
> > executes
> > first and then VertexStepStrategy.
> > 
> > GraphStepStrategy starts at the beginning of the traversal
> > optimizing
> > from left to right till it reaches a step that it can not optimize
> > and
> > terminates.
> 
> Makes sense.
> 
> > After that VertexStepStrategy tries to optimize what remains.
> > It ultimately replaces optimizable sequential steps with a
> > SqlgVertexStep.
> 
> Okay...
> 
> > Thus far the SqlgVertexStep always has one incoming traverser from
> > where it continues the traversal. Basically it translated to a sql
> > where clause with the incoming traversal element's id.
> > 
> > The current optimization is to bulk the incoming traversers and
> > execute
> > the traversal for all incoming traversers in one go. This reduces
> > latency and has a drastic performance improvement.
> > 
> > I do the same as the existing BarrierSteps and iterate the `starts`
> > to
> > collect all the left incoming traversers and from there I continue
> > and
> > all is well.
> 
> Smart. You got chops.
> 
> > However for local traversals there is only one start on the
> > traversal
> > so the barrier idea is not working.
> > 
> > Is there a way barrier all incoming left traversers on local
> > traversals?
> 
> 
> Eeeeeeeeeeeeeeee…… huuuuuuuhhhhhhhh…………….
> 
> There is a “easy” and there is a “hard.” Give me an example traversal
> and lets discuss from a more specific standpoint before
> generalizing...
> 
> Thanks,
> Marko.
> 
> http://markorodriguez.com

--- End Message ---