Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

[Gábor Gévay]

Hello,

I am also interested in this feature for a paper that I'm writing.

I have the "slowly evolving side input" case with a complicated custom
"update precondition" that would be expressible by a stateful UDF that
makes its decisions from looking at the elements of the main stream.

Best,
Gábor




2016-11-09 3:19 GMT+01:00 吕文龙 :
> Hi, all, we are also trying to find one way to do state initialization for
> stateful job, which has some external data stored in external storage. Side
> Input is a great feature meeting the requirement. I am looking forward to a
> full proposal including runtime implementations.
>
> 2016-11-01 21:52 GMT+08:00 Aljoscha Krettek :
>
>> Hi,
>> thanks for trying to revive the discussion! I added some comments in the
>> doc.
>>
>> Cheers,
>> Aljoscha
>>
>> On Fri, 28 Oct 2016 at 12:05 venturadelmonte 
>> wrote:
>>
>> > Hello,
>> >
>> > I find this feature really cool because it would allow people to tackle
>> > scenarios requiring a more advanced "join" on multiple streams as
>> > already stated above.
>> > Here
>> > <
>> > https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_
>> db61ky7OQy1hw/edit?usp=sharing
>> > >
>> > you can find my few thoughts about this feature (a.k.a. my
>> > attempt to extend Aljoscha's initial document).
>> > I hope you will find it interesting and a good starting point for
>> > having a new FLIP soon ;-)
>> >
>> > Cheers,
>> > Ventura
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-flink-mailing-list-archive.1008284.n3.
>> nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For-
>> Streaming-API-tp11529p14287.html
>> > Sent from the Apache Flink Mailing List archive. mailing list archive at
>> > Nabble.com.
>> >
>>

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

[吕文龙]

Hi, all, we are also trying to find one way to do state initialization for
stateful job, which has some external data stored in external storage. Side
Input is a great feature meeting the requirement. I am looking forward to a
full proposal including runtime implementations.

2016-11-01 21:52 GMT+08:00 Aljoscha Krettek :

> Hi,
> thanks for trying to revive the discussion! I added some comments in the
> doc.
>
> Cheers,
> Aljoscha
>
> On Fri, 28 Oct 2016 at 12:05 venturadelmonte 
> wrote:
>
> > Hello,
> >
> > I find this feature really cool because it would allow people to tackle
> > scenarios requiring a more advanced "join" on multiple streams as
> > already stated above.
> > Here
> > <
> > https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_
> db61ky7OQy1hw/edit?usp=sharing
> > >
> > you can find my few thoughts about this feature (a.k.a. my
> > attempt to extend Aljoscha's initial document).
> > I hope you will find it interesting and a good starting point for
> > having a new FLIP soon ;-)
> >
> > Cheers,
> > Ventura
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For-
> Streaming-API-tp11529p14287.html
> > Sent from the Apache Flink Mailing List archive. mailing list archive at
> > Nabble.com.
> >
>

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

[Aljoscha Krettek]

Hi,
thanks for trying to revive the discussion! I added some comments in the
doc.

Cheers,
Aljoscha

On Fri, 28 Oct 2016 at 12:05 venturadelmonte 
wrote:

> Hello,
>
> I find this feature really cool because it would allow people to tackle
> scenarios requiring a more advanced "join" on multiple streams as
> already stated above.
> Here
> <
> https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_db61ky7OQy1hw/edit?usp=sharing
> >
> you can find my few thoughts about this feature (a.k.a. my
> attempt to extend Aljoscha's initial document).
> I hope you will find it interesting and a good starting point for
> having a new FLIP soon ;-)
>
> Cheers,
> Ventura
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For-Streaming-API-tp11529p14287.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

[venturadelmonte]

Hello,

I find this feature really cool because it would allow people to tackle
scenarios requiring a more advanced "join" on multiple streams as
already stated above.
Here

  
you can find my few thoughts about this feature (a.k.a. my
attempt to extend Aljoscha's initial document).
I hope you will find it interesting and a good starting point for 
having a new FLIP soon ;-)

Cheers,
Ventura 



--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For-Streaming-API-tp11529p14287.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

[Aljoscha Krettek]

Hi Folks,
I added a section about "Implementation Details" to the doc, it is still
very rough but I'm afraid this is somewhat tricky and might require some
changes to how we handle state.

As always, feedback is very welcome!

Cheers,
Aljoscha

On Mon, 2 May 2016 at 11:28 Aljoscha Krettek  wrote:

> Hi,
> I've put down some words about how we would go about adding support for
> side inputs to the streaming API. People are increasingly asking for such a
> feature, which would allow joining a main stream with some slowly changing
> inputs that can be used for enriching data, among other things.
>
> The design document is available here:
> https://docs.google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit?usp=sharing
>
> Any feedback is very welcome!
>
> Cheers,
> Aljoscha
>