Le 12 mars 2018 00:16, "Reuven Lax" <re...@google.com> a écrit :

I think it would be interesting to see what a Java stream-based API would
look like. As I mentioned elsewhere, we are not limited to having only one
API for Beam.

If I remember correctly, a Java stream API was considered for Dataflow back
at the very beginning. I don't completely remember why it was rejected, but
I suspect at least part of the reason might have been that Java streams
were considered too new and untested back then.

Coders are broken - typevariables dont have bounds except object - and
reducers are not trivial to impl generally I guess.

However being close of this api can help a lot so +1 to try to have a java
dsl on top of current api. Would also be neat to integrate it with
completionstage :).


On Sun, Mar 11, 2018 at 2:29 PM Romain Manni-Bucau <rmannibu...@gmail.com>

> Le 11 mars 2018 21:18, "Jean-Baptiste Onofré" <j...@nanthrax.net> a écrit :
> Hi Romain,
> I remember we have discussed about the way to express pipeline while ago.
> I was fan of a "DSL" compared to the one we have in Camel: instead of
> using apply(), use a dedicated form (like .map(), .reduce(), etc, AFAIR,
> it's the approach in flume).
> However, we agreed that apply() syntax gives a more flexible approach.
> Using Java Stream is interesting but I'm afraid we would have the same
> issue as the one we identified discussing "fluent Java SDK". However, we
> can have a Stream API DSL on top of the SDK IMHO.
> Agree and a beam stream interface (copying jdk api but making lambda
> serializable to avoid the cast need).
> On my side i think it enables user to discover the api. If you check my
> poc impl you quickly see the steps needed to do simple things like a map
> which is a first citizen.
> Also curious if we could impl reduce with pipeline result = get an output
> of a batch from the runner (client) jvm. I see how to do it for longs -
> with metrics - but not for collect().
> Regards
> JB
> On 11/03/2018 19:46, Romain Manni-Bucau wrote:
>> Hi guys,
>> don't know if you already experienced using java Stream API as a
>> replacement for pipeline API but did some tests: https://github.com/
>> rmannibucau/jbeam
>> It is far to be complete but already shows where it fails (beam doesn't
>> have a way to reduce in the caller machine for instance, coder handling is
>> not that trivial, lambda are not working well with default Stream API
>> etc...).
>> However it is interesting to see that having such an API is pretty
>> natural compare to the pipeline API
>> so wonder if beam should work on its own Stream API (with surely another
>> name for obvious reasons ;)).
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog <
>> https://rmannibucau.metawerx.net/> | Old Blog <
>> http://rmannibucau.wordpress.com> | Github <https://github.com/
>> rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-
>> ee-8-high-performance>

Reply via email to