Along the same lines as Anthony's email on distributed classloading, QiHong
and I also hacked up a prototype of shipping Java 8 stream operations to
gemfire data stores.
What we did:
Java 8 allows people to do functional operations on a region using the new
stream API:
region.entrySet().stream()
.filter(e -> e.getKey() % 2 == 0)
.map(e -> e.getValue())
.reduce(1, Integer::sum);
It would be much more efficient if the filtering, mapping, reducing, etc.
happen on the members that actual host the data, rather than shipping all
the data to a single member.
So we implemented a new method, remoteStream, which collects the operations
as they are applied to a stream. When a terminal operation like reduce is
encountered, the whole pipeline is sent out to all members with a gemfire
function, and the results of processing that pipeline on the data stores
are brought back.
What we did is basically a quick and dirty proof of concept. There's a lot
to polish up if we want to make this a real geode feature. Is this
something people would be interested in using?
The proof of concept code is sitting in github if anyone is interested.
Checkout the stream-prototype and look at the tests in StreamsP2PDUnitTest:
https://github.com/upthewaterspout/incubator-geode
