Re: Making broadcast state queryable?

2019-09-15 Thread Yu Li 
Thanks for the reply Oytun (and sorry for the late response, somehow just
noticed).

Requirement received, interesting one. Let's see whether this could draw
any attention from the committers (smile).

Best Regards,
Yu


On Fri, 6 Sep 2019 at 22:14, Oytun Tez  wrote:

> Hi Yu,
>
> Excuse my late reply... We simply want Flink to be our centralized stream
> analysis platform, where we 1) consume data, 2) generate analysis, 3)
> present analysis. I honestly don't want "stream analysis" to spill out to
> other components in our ecosystem (e.g., sinking insights into a DB-like
> place).
>
> So the case for QS for us is centralization of input, output,
> presentation. State Processor API for instance also counts as a
> presentation tool for us (on top of migration tool).
>
> This kind of all-in-one (in, out, ui) packaging helps with small teams.
>
> ---
> Oytun Tez
>
> *M O T A W O R D*
> The World's Fastest Human Translation Platform.
> oy...@motaword.com — www.motaword.com
>
>
> On Wed, Aug 14, 2019 at 3:13 AM Yu Li  wrote:
>
>> Good to know your thoughts and the coming talk in Flink Forward Berlin
>> Oytun, interesting topic and great job! And it's great to hear the voice
>> from application perspective.
>>
>> Could you share, if possible, the reason why you need to open the state
>> to outside instead of writing the result to sink and directly query there?
>> In another thread there's a case that sink writes to different kafka topics
>> so state is the only place to get a global view, is this the same case
>> you're facing? Or some different requirements? I believe more attention
>> will be drawn to QS if more and more user requirements emerge (smile).
>>
>> Thanks.
>>
>> Best Regards,
>> Yu
>>
>>
>> On Wed, 14 Aug 2019 at 00:50, Oytun Tez  wrote:
>>
>>> Thank you for the honest response, Yu!
>>>
>>> There is so much that comes to mind when we look at Flink as a
>>> "application framework" (my talk
>>> 
>>> in Flink Forward in Berlin will be about this). QS is one of them
>>> (need-wise, not QS itself necessarily). I opened the design doc Vino Yang
>>> created, I'll try to contribute to that.
>>>
>>> Meanwhile, for the sake of opening our state to outside, we will put a
>>> stupid simple operator in between to keep a *duplicate* of the state...
>>>
>>> Thanks again!
>>>
>>>
>>>
>>>
>>>
>>> ---
>>> Oytun Tez
>>>
>>> *M O T A W O R D*
>>> The World's Fastest Human Translation Platform.
>>> oy...@motaword.com — www.motaword.com
>>>
>>>
>>> On Tue, Aug 13, 2019 at 6:29 PM Yu Li  wrote:
>>>
 Hi Oytun,

 Sorry but TBH such support will probably not be added in the
 foreseeable future due to lack of committer bandwidth (not only support
 queryable broadcast state but all about QueryableState module) as pointed
 out in other threads [1] [2].

 However, I think you could open a JIRA for this so when things changed
 it could be taken care of. Thanks.

 [1] https://s.apache.org/MaOl
 [2] https://s.apache.org/r8k8a

 Best Regards,
 Yu


 On Tue, 13 Aug 2019 at 20:34, Oytun Tez  wrote:

> Hi there,
>
> Can we set a broadcast state as queryable? I've looked around, not
> much to find about it. I am receiving UnknownKvStateLocation when I try to
> query with the descriptor/state name I give to the broadcast state.
>
> If it doesn't work, what could be the alternative? My mind goes around
> ctx.getBroadcastState and making it queryable in the operator level (I'd
> rather not).
>
> ---
> Oytun Tez
>
> *M O T A W O R D*
> The World's Fastest Human Translation Platform.
> oy...@motaword.com — www.motaword.com
>

Re: Making broadcast state queryable?

2019-09-06 Thread Oytun Tez 
Hi Yu,

Excuse my late reply... We simply want Flink to be our centralized stream
analysis platform, where we 1) consume data, 2) generate analysis, 3)
present analysis. I honestly don't want "stream analysis" to spill out to
other components in our ecosystem (e.g., sinking insights into a DB-like
place).

So the case for QS for us is centralization of input, output, presentation.
State Processor API for instance also counts as a presentation tool for us
(on top of migration tool).

This kind of all-in-one (in, out, ui) packaging helps with small teams.

---
Oytun Tez

*M O T A W O R D*
The World's Fastest Human Translation Platform.
oy...@motaword.com — www.motaword.com


On Wed, Aug 14, 2019 at 3:13 AM Yu Li  wrote:

> Good to know your thoughts and the coming talk in Flink Forward Berlin
> Oytun, interesting topic and great job! And it's great to hear the voice
> from application perspective.
>
> Could you share, if possible, the reason why you need to open the state to
> outside instead of writing the result to sink and directly query there? In
> another thread there's a case that sink writes to different kafka topics so
> state is the only place to get a global view, is this the same case you're
> facing? Or some different requirements? I believe more attention will be
> drawn to QS if more and more user requirements emerge (smile).
>
> Thanks.
>
> Best Regards,
> Yu
>
>
> On Wed, 14 Aug 2019 at 00:50, Oytun Tez  wrote:
>
>> Thank you for the honest response, Yu!
>>
>> There is so much that comes to mind when we look at Flink as a
>> "application framework" (my talk
>> 
>> in Flink Forward in Berlin will be about this). QS is one of them
>> (need-wise, not QS itself necessarily). I opened the design doc Vino Yang
>> created, I'll try to contribute to that.
>>
>> Meanwhile, for the sake of opening our state to outside, we will put a
>> stupid simple operator in between to keep a *duplicate* of the state...
>>
>> Thanks again!
>>
>>
>>
>>
>>
>> ---
>> Oytun Tez
>>
>> *M O T A W O R D*
>> The World's Fastest Human Translation Platform.
>> oy...@motaword.com — www.motaword.com
>>
>>
>> On Tue, Aug 13, 2019 at 6:29 PM Yu Li  wrote:
>>
>>> Hi Oytun,
>>>
>>> Sorry but TBH such support will probably not be added in the foreseeable
>>> future due to lack of committer bandwidth (not only support queryable
>>> broadcast state but all about QueryableState module) as pointed out in
>>> other threads [1] [2].
>>>
>>> However, I think you could open a JIRA for this so when things changed
>>> it could be taken care of. Thanks.
>>>
>>> [1] https://s.apache.org/MaOl
>>> [2] https://s.apache.org/r8k8a
>>>
>>> Best Regards,
>>> Yu
>>>
>>>
>>> On Tue, 13 Aug 2019 at 20:34, Oytun Tez  wrote:
>>>
 Hi there,

 Can we set a broadcast state as queryable? I've looked around, not much
 to find about it. I am receiving UnknownKvStateLocation when I try to query
 with the descriptor/state name I give to the broadcast state.

 If it doesn't work, what could be the alternative? My mind goes around
 ctx.getBroadcastState and making it queryable in the operator level (I'd
 rather not).

 ---
 Oytun Tez

 *M O T A W O R D*
 The World's Fastest Human Translation Platform.
 oy...@motaword.com — www.motaword.com

>>>

Re: Making broadcast state queryable?

2019-08-14 Thread Yu Li 
Good to know your thoughts and the coming talk in Flink Forward Berlin
Oytun, interesting topic and great job! And it's great to hear the voice
from application perspective.

Could you share, if possible, the reason why you need to open the state to
outside instead of writing the result to sink and directly query there? In
another thread there's a case that sink writes to different kafka topics so
state is the only place to get a global view, is this the same case you're
facing? Or some different requirements? I believe more attention will be
drawn to QS if more and more user requirements emerge (smile).

Thanks.

Best Regards,
Yu


On Wed, 14 Aug 2019 at 00:50, Oytun Tez  wrote:

> Thank you for the honest response, Yu!
>
> There is so much that comes to mind when we look at Flink as a
> "application framework" (my talk
> 
> in Flink Forward in Berlin will be about this). QS is one of them
> (need-wise, not QS itself necessarily). I opened the design doc Vino Yang
> created, I'll try to contribute to that.
>
> Meanwhile, for the sake of opening our state to outside, we will put a
> stupid simple operator in between to keep a *duplicate* of the state...
>
> Thanks again!
>
>
>
>
>
> ---
> Oytun Tez
>
> *M O T A W O R D*
> The World's Fastest Human Translation Platform.
> oy...@motaword.com — www.motaword.com
>
>
> On Tue, Aug 13, 2019 at 6:29 PM Yu Li  wrote:
>
>> Hi Oytun,
>>
>> Sorry but TBH such support will probably not be added in the foreseeable
>> future due to lack of committer bandwidth (not only support queryable
>> broadcast state but all about QueryableState module) as pointed out in
>> other threads [1] [2].
>>
>> However, I think you could open a JIRA for this so when things changed it
>> could be taken care of. Thanks.
>>
>> [1] https://s.apache.org/MaOl
>> [2] https://s.apache.org/r8k8a
>>
>> Best Regards,
>> Yu
>>
>>
>> On Tue, 13 Aug 2019 at 20:34, Oytun Tez  wrote:
>>
>>> Hi there,
>>>
>>> Can we set a broadcast state as queryable? I've looked around, not much
>>> to find about it. I am receiving UnknownKvStateLocation when I try to query
>>> with the descriptor/state name I give to the broadcast state.
>>>
>>> If it doesn't work, what could be the alternative? My mind goes around
>>> ctx.getBroadcastState and making it queryable in the operator level (I'd
>>> rather not).
>>>
>>> ---
>>> Oytun Tez
>>>
>>> *M O T A W O R D*
>>> The World's Fastest Human Translation Platform.
>>> oy...@motaword.com — www.motaword.com
>>>
>>

Re: Making broadcast state queryable?

2019-08-13 Thread Oytun Tez 
Thank you for the honest response, Yu!

There is so much that comes to mind when we look at Flink as a "application
framework" (my talk

in Flink Forward in Berlin will be about this). QS is one of them
(need-wise, not QS itself necessarily). I opened the design doc Vino Yang
created, I'll try to contribute to that.

Meanwhile, for the sake of opening our state to outside, we will put a
stupid simple operator in between to keep a *duplicate* of the state...

Thanks again!





---
Oytun Tez

*M O T A W O R D*
The World's Fastest Human Translation Platform.
oy...@motaword.com — www.motaword.com


On Tue, Aug 13, 2019 at 6:29 PM Yu Li  wrote:

> Hi Oytun,
>
> Sorry but TBH such support will probably not be added in the foreseeable
> future due to lack of committer bandwidth (not only support queryable
> broadcast state but all about QueryableState module) as pointed out in
> other threads [1] [2].
>
> However, I think you could open a JIRA for this so when things changed it
> could be taken care of. Thanks.
>
> [1] https://s.apache.org/MaOl
> [2] https://s.apache.org/r8k8a
>
> Best Regards,
> Yu
>
>
> On Tue, 13 Aug 2019 at 20:34, Oytun Tez  wrote:
>
>> Hi there,
>>
>> Can we set a broadcast state as queryable? I've looked around, not much
>> to find about it. I am receiving UnknownKvStateLocation when I try to query
>> with the descriptor/state name I give to the broadcast state.
>>
>> If it doesn't work, what could be the alternative? My mind goes around
>> ctx.getBroadcastState and making it queryable in the operator level (I'd
>> rather not).
>>
>> ---
>> Oytun Tez
>>
>> *M O T A W O R D*
>> The World's Fastest Human Translation Platform.
>> oy...@motaword.com — www.motaword.com
>>
>

Re: Making broadcast state queryable?

2019-08-13 Thread Yu Li 
Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable
future due to lack of committer bandwidth (not only support queryable
broadcast state but all about QueryableState module) as pointed out in
other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it
could be taken care of. Thanks.

[1] https://s.apache.org/MaOl
[2] https://s.apache.org/r8k8a

Best Regards,
Yu


On Tue, 13 Aug 2019 at 20:34, Oytun Tez  wrote:

> Hi there,
>
> Can we set a broadcast state as queryable? I've looked around, not much to
> find about it. I am receiving UnknownKvStateLocation when I try to query
> with the descriptor/state name I give to the broadcast state.
>
> If it doesn't work, what could be the alternative? My mind goes around
> ctx.getBroadcastState and making it queryable in the operator level (I'd
> rather not).
>
> ---
> Oytun Tez
>
> *M O T A W O R D*
> The World's Fastest Human Translation Platform.
> oy...@motaword.com — www.motaword.com
>