We have something similar for broadcast variables in FlinkML. It allows you
to write ds.mapWithBcVariable(bcDS){ (dsElement, bcVar) => ... }.
I like the idea to make the life of a Scala programmer a little bit less
javaesque :-)
On Fri, Jul 24, 2015 at 5:45 PM, Stephan Ewen <se...@apache.org> wrote:
> This is really syntactic sugar in the Scala API, rather then a system
> feature.
>
> Which is good, it needs no extra runtime constructs...
>
> On Fri, Jul 24, 2015 at 5:43 PM, Aljoscha Krettek <aljos...@apache.org>
> wrote:
>
> > Yes, this might be nice. Till and I had similar ideas about using the
> > pattern to make broadcast variables more useable in Scala, in fact. :D
> >
> > On Fri, 24 Jul 2015 at 17:39 Gyula Fóra <gyf...@apache.org> wrote:
> >
> > > Hey,
> > >
> > > I would like to propose a way to extend the standard Streaming Scala
> API
> > > methods (map, flatmap, filter etc) with versions that take stateful
> > > functions as lambdas. I think this would eliminate the awkwardness of
> > > implementing RichFunctions in Scala and make statefulness more
> explicit:
> > >
> > > *For example:*
> > > def map( statefulMap: (I, Option[S]) => (O, Option[S]) )
> > > def flatMap( statefulFlatMap: (I, Option[S] ) => (Traversable[O],
> > > Option[S]))
> > >
> > > This would be translated into RichMap and RichFlatMapFunctions that
> store
> > > Option[S] as OperatorState for fault tolerance.
> > >
> > > *Example rolling sum by key:*
> > > val input: DataStream[Long] = ...
> > > val sumByKey: DataStream[Long] =
> > > input.keyBy(...).map( (next: Long, sum: Option[Long]) =>
> > > sum match {
> > > case Some(s) => (next + s, Some(next + s))
> > > case None => (next, Some(next))
> > > })
> > >
> > > What do you think?
> > >
> > > Gyula
> > >
> >
>
