Re: SubgraphStrategies and Graph access

[Bryn Cooke]

Function doesn't extend Serializable so it won't work.
If I define:
public static interface SerializableFunction<T,R> extends Serializable, 
Function<T,R> {
}

Then yes it does work:
FileInputStream byteIn = new FileInputStream("/home/bryn/test.lambda2");
ObjectInputStream in = new ObjectInputStream(byteIn);
SerializableFunction readObject = (SerializableFunction) in.readObject();
Object apply = readObject.apply(3);
System.out.println(apply); //4

Bryn




On 11/05/15 21:56, Marko Rodriguez wrote:
What is MyRunnable? Can you try:

        a -> a+1

Not a double void.

Marko.

http://markorodriguez.com

On May 11, 2015, at 2:55 PM, Bryn Cooke <brynco...@gmail.com> wrote:

I'm not sure I follow.

I serialised this:

() -> System.out.println("test")

And loaded it with this:

ObjectInputStream in = new ObjectInputStream(new 
FileInputStream("/home/bryn/test.lambda"));
MyRunnable readObject = (MyRunnable) in.readObject();
readObject.run();//Prints test

Is this what we are looking for?

Bryn


On 11/05/15 21:45, Marko Rodriguez wrote:
Hi,

Yes, you you can serialize an instance of a Class<? extends Function>, but you 
can't just serialize a -> a+1.

This is what you mean, right?

Marko.

http://markorodriguez.com

On May 11, 2015, at 2:41 PM, Bryn Cooke <brynco...@gmail.com> wrote:

Hi,

Yes I just tried it and it did just work, no hackery required.
The proviso is that the lambda class must exist on the target VM even if it is 
not invoked.
As long as you send the lambda classes and rehydrate the pipeline in a 
classloader that contains those classes I guess you'll be OK.

Bryn



On 11/05/15 19:05, Marko Rodriguez wrote:
Hi,

Did you try doing this across JVMs? Its possible to serialize a lambda within a 
JVM (easily), but not across JVMs.

Marko.

http://markorodriguez.com

On May 11, 2015, at 9:47 AM, Bryn Cooke <brynco...@gmail.com> wrote:

Hmm, I just gave it a go and didn't have to do anything to successfully 
serialize a Lambda. It just worked.

As long as you don't reference anything outside the lambda it's fine. If you do 
then you get a java.io.NotSerializableException.

Bryn


On 11/05/15 16:22, Marko Rodriguez wrote:
Hi Bryn,

Huh. That is how you "dehydrate()" a closure in Groovy. What is the field name of the "it" in a 
Java8 lambda? If we could figure that out, then we could have nice "static dehydrate()" and "static 
hydrate()"  helpers in gremlin-core/function.

Good stuff,
Marko.

http://markorodriguez.com

On May 11, 2015, at 9:19 AM, Bryn Cooke <brynco...@gmail.com> wrote:

Hi Marko,

Just in case Kryo 3 doesn't serialialize lambdas in the way you expect it would 
be worth trying to null out the reference to the outer object in the lambda 
object using reflection. I *believe* that this is the only thing stopping 
serialization using the standard Java Seriazable mechanism.

Bryn

On 11/05/15 16:06, Marko Rodriguez wrote:
Hi,

SubgraphStrategyTest contains test cases that access both vertices incident
on edges which isn't allowed under GraphComputer semantics. It silently
passes in GraphComputerTest because the TP3 implementations ignore those
cases. I believe implementations should throw an exception on such access.
This example demonstrates how easy it is to end up with difficult to find
bugs otherwise. We would put a high burden on the user to write unit tests
for their strategies to make sure they detect those cases where it makes a
difference.
Yes. I agree that StarGraph should throw and exception when, right now, it 
returns Collections.emptyIterator().

I will add to the GraphComputer TestSuite verification of the topology of the "star 
vertex" so that all vendors have the same accessible data and the same Exceptions 
thrown.

Moreover, we should think about such FilterStrategies in general. Should
there be a way to restrict them to just stargraph access to ensure that
they always work in GraphComputer? Basically, I think it would be great to
avoid a case where a strategy works in OLTP but doesn't in OLAP because
most people would probably implement it for OLTP first. This is
particularly true for SubgraphStrategies where you try to restrict access.
There are various situations in which OLAP and OLTP do not line up. However, I 
don't think we should restrict OLTP to be that which OLAP can handle. Primarily 
because many of the limitations of OLAP will be solved in the future. Here is 
the list as I know them:

        1. OLAP does not support mutating steps (addV(), addE(), etc.). This is 
because we haven't come up with a good model for mutating in OLAP.
                - The latest idea on this is the "GraphMutator" interface that 
all OLAP vendors should supply, blah blah.
                - In essence, it will happens, just not right now (3.1+).
        2. OLAP does not support lambdas. This is because Java8 lambdas are not 
serializable.
                - We can fake this with passing Groovy strings, but its not 
"true" Gremlin-Java8.
                - Kryo3 supposedly figured out a way to serialize Java8 lambdas 
-- perhaps our savior?
                -  In essence, it will happen, just not right now.
        3. OLAP does not support mid-traversal barriers (i.e. 
g.V.count().is(gt(100))). This is because a barrier step requires a MapReduce and 
we can't go MapReduce -> VertexProgram.
                - There is a ticket to support OLAP->OLTP->OLAP which will all 
this.
                - In essence, it will happen, just not right now.
        4. OLAP does not support non-graph object processing (i.e. 
__.(1,2,3,4).order(local).by(incr).sum() in OLAP (i.e. no graph objects)).
                - This is possible (and actually not that hard) and there is a 
ticket for it.
                - In essence, it will happen, just not right now.
        5. OLAP localTraversals can not leave "the star graph."
                - This is because we don't have that data available locally, though, I 
don't see why we can't solve this with "TraverserSet -- scoping" in OLAP.
                - The solution would be complex to write, but doable. The way 
TraveralMatrix works in OLAP, the code is staged for this.
                - In essence, it will happen, just not right now.
        6. OLAP does not support MatchStep.
                - There is currently no thoughts on solving this, but I suspect 
its a natural fall out once 4 is complete.
                - In essence, it will happen, just not right now.

So, while OLAP and OLTP differ, I would not restrict OLTP to be what OLAP can 
do. What we could do is make the ComputerVerificationStrategy's analysis easily 
accessible to users. For instance:

        ComputerVerificationStrategy.check(myTraversal) // which is simply a 
call to ComputerVerificationStrategy.instance().apply(myTraversal) :)

This way, a user can easily determine if their traversal will work on OLAP without having 
to actually try and execute the job. One of the things that Daniel Kuppitz is working on 
is making sure that ComputerVerificationStrategy is able to identify the 5 areas above 
and throw the appropriate exception explaining "why" to the user. As we move 
forward with 3.1, 3.2, 3.3, etc. hopefully the list becomes smaller and smaller.

Thanks,
Marko.

http://markorodriguez.com