Re: Queryable state and TTL

[Avi Levi]

Thanks, I'll check it out.

On Sun, Jul 7, 2019 at 5:40 AM Eron Wright  wrote:

> *This Message originated outside your organization.*
> --
> Here's a PR for queryable state TLS that I closed because I didn't have
> time, and because I get the impression that the queryable state feature is
> used very often.Feel free to take it up, if you like.
> https://github.com/apache/flink/pull/6626
> 
>
> -Eron
>
> On Wed, Jul 3, 2019 at 11:21 PM Avi Levi  wrote:
>
>> Hi Yu,
>> Our sink is actually Kafka hence we cannot query it properly, from there
>> we distribute it to different consumers. We keep info in our state such as
>> entry time, some accumulated data etc' , this data is not kept elsewhere
>> hence we need to query our state.
>>
>> Best regards
>> Avi
>>
>>
>> On Thu, Jul 4, 2019 at 7:20 AM Yu Li  wrote:
>>
>>> *This Message originated outside your organization.*
>>> --
>>> Thanks for the ping Andrey.
>>>
>>> For me the general answer is yes, but TBH it will probably not be added
>>> in the foreseeable future due to lack of committer bandwidth (not only
>>> QueryableState with TTL but all about QueryableState module) as per
>>> Aljoscha pointed out in another thread [1].
>>>
>>> Although we could see emerging requirements and proposals on
>>> QueryableState recently, prioritizing is important for each open source
>>> project. And personally I think it may help if we could gather more and
>>> clearly describe the other-than-debugging use cases of QueryableState in
>>> production [2]. Could you share your case with us and why QueryableState is
>>> necessary rather than querying the data from sink @Avi? Thanks.
>>>
>>> [1] https://s.apache.org/MaOl
>>> 
>>> [2] https://s.apache.org/hJDA
>>> 
>>>
>>> Best Regards,
>>> Yu
>>>
>>>
>>> On Wed, 3 Jul 2019 at 23:13, Andrey Zagrebin 
>>> wrote:
>>>
 Hi Avi,

 It is on the road map but I am not aware about plans of any contributor
 to work on it for the next releases.
 I think the community will firstly work on the event time support for
 TTL.
 I will loop Yu in, maybe he has some plans to work on TTL for the
 queryable state.

 Best,
 Andrey

 On Wed, Jul 3, 2019 at 3:17 PM Avi Levi 
 wrote:

> Hi,
> Adding queryable state to state with ttl is not supported at 1.8.0
> (throwing java.lang.IllegalArgumentException: Queryable state is currently
> not supported with TTL)
>
> I saw in previous mailing thread
> that
> it is on the roadmap. Is it still on the roadmap ?
>
> * There is a workaround which is using timers to clear the state, but
> in our case, it means firing billons of timers on daily basis all at the
> same time, which seems no to very efficient and might cause some resources
> issues
>
> Cheers
> Avi
>
>
>

Re: Queryable state and TTL

[Eron Wright]

Here's a PR for queryable state TLS that I closed because I didn't have
time, and because I get the impression that the queryable state feature is
used very often.Feel free to take it up, if you like.
https://github.com/apache/flink/pull/6626

-Eron

On Wed, Jul 3, 2019 at 11:21 PM Avi Levi  wrote:

> Hi Yu,
> Our sink is actually Kafka hence we cannot query it properly, from there
> we distribute it to different consumers. We keep info in our state such as
> entry time, some accumulated data etc' , this data is not kept elsewhere
> hence we need to query our state.
>
> Best regards
> Avi
>
>
> On Thu, Jul 4, 2019 at 7:20 AM Yu Li  wrote:
>
>> *This Message originated outside your organization.*
>> --
>> Thanks for the ping Andrey.
>>
>> For me the general answer is yes, but TBH it will probably not be added
>> in the foreseeable future due to lack of committer bandwidth (not only
>> QueryableState with TTL but all about QueryableState module) as per
>> Aljoscha pointed out in another thread [1].
>>
>> Although we could see emerging requirements and proposals on
>> QueryableState recently, prioritizing is important for each open source
>> project. And personally I think it may help if we could gather more and
>> clearly describe the other-than-debugging use cases of QueryableState in
>> production [2]. Could you share your case with us and why QueryableState is
>> necessary rather than querying the data from sink @Avi? Thanks.
>>
>> [1] https://s.apache.org/MaOl
>> [2] https://s.apache.org/hJDA
>>
>> Best Regards,
>> Yu
>>
>>
>> On Wed, 3 Jul 2019 at 23:13, Andrey Zagrebin 
>> wrote:
>>
>>> Hi Avi,
>>>
>>> It is on the road map but I am not aware about plans of any contributor
>>> to work on it for the next releases.
>>> I think the community will firstly work on the event time support for
>>> TTL.
>>> I will loop Yu in, maybe he has some plans to work on TTL for the
>>> queryable state.
>>>
>>> Best,
>>> Andrey
>>>
>>> On Wed, Jul 3, 2019 at 3:17 PM Avi Levi  wrote:
>>>
 Hi,
 Adding queryable state to state with ttl is not supported at 1.8.0
 (throwing java.lang.IllegalArgumentException: Queryable state is currently
 not supported with TTL)

 I saw in previous mailing thread
 that
 it is on the roadmap. Is it still on the roadmap ?

 * There is a workaround which is using timers to clear the state, but
 in our case, it means firing billons of timers on daily basis all at the
 same time, which seems no to very efficient and might cause some resources
 issues

 Cheers
 Avi

Re: Queryable state and TTL

[Avi Levi]

Hi Yu,
Our sink is actually Kafka hence we cannot query it properly, from there we
distribute it to different consumers. We keep info in our state such as
entry time, some accumulated data etc' , this data is not kept elsewhere
hence we need to query our state.

Best regards
Avi


On Thu, Jul 4, 2019 at 7:20 AM Yu Li  wrote:

> *This Message originated outside your organization.*
> --
> Thanks for the ping Andrey.
>
> For me the general answer is yes, but TBH it will probably not be added in
> the foreseeable future due to lack of committer bandwidth (not only
> QueryableState with TTL but all about QueryableState module) as per
> Aljoscha pointed out in another thread [1].
>
> Although we could see emerging requirements and proposals on
> QueryableState recently, prioritizing is important for each open source
> project. And personally I think it may help if we could gather more and
> clearly describe the other-than-debugging use cases of QueryableState in
> production [2]. Could you share your case with us and why QueryableState is
> necessary rather than querying the data from sink @Avi? Thanks.
>
> [1] https://s.apache.org/MaOl
> 
> [2] https://s.apache.org/hJDA
> 
>
> Best Regards,
> Yu
>
>
> On Wed, 3 Jul 2019 at 23:13, Andrey Zagrebin  wrote:
>
>> Hi Avi,
>>
>> It is on the road map but I am not aware about plans of any contributor
>> to work on it for the next releases.
>> I think the community will firstly work on the event time support for TTL.
>> I will loop Yu in, maybe he has some plans to work on TTL for the
>> queryable state.
>>
>> Best,
>> Andrey
>>
>> On Wed, Jul 3, 2019 at 3:17 PM Avi Levi  wrote:
>>
>>> Hi,
>>> Adding queryable state to state with ttl is not supported at 1.8.0
>>> (throwing java.lang.IllegalArgumentException: Queryable state is currently
>>> not supported with TTL)
>>>
>>> I saw in previous mailing thread
>>> that
>>> it is on the roadmap. Is it still on the roadmap ?
>>>
>>> * There is a workaround which is using timers to clear the state, but in
>>> our case, it means firing billons of timers on daily basis all at the same
>>> time, which seems no to very efficient and might cause some resources
>>> issues
>>>
>>> Cheers
>>> Avi
>>>
>>>
>>>

Re: Queryable state and TTL

[Yu Li]

Thanks for the ping Andrey.

For me the general answer is yes, but TBH it will probably not be added in
the foreseeable future due to lack of committer bandwidth (not only
QueryableState with TTL but all about QueryableState module) as per
Aljoscha pointed out in another thread [1].

Although we could see emerging requirements and proposals on QueryableState
recently, prioritizing is important for each open source project. And
personally I think it may help if we could gather more and clearly describe
the other-than-debugging use cases of QueryableState in production [2].
Could you share your case with us and why QueryableState is necessary
rather than querying the data from sink @Avi? Thanks.

[1] https://s.apache.org/MaOl
[2] https://s.apache.org/hJDA

Best Regards,
Yu


On Wed, 3 Jul 2019 at 23:13, Andrey Zagrebin  wrote:

> Hi Avi,
>
> It is on the road map but I am not aware about plans of any contributor to
> work on it for the next releases.
> I think the community will firstly work on the event time support for TTL.
> I will loop Yu in, maybe he has some plans to work on TTL for the
> queryable state.
>
> Best,
> Andrey
>
> On Wed, Jul 3, 2019 at 3:17 PM Avi Levi  wrote:
>
>> Hi,
>> Adding queryable state to state with ttl is not supported at 1.8.0
>> (throwing java.lang.IllegalArgumentException: Queryable state is currently
>> not supported with TTL)
>>
>> I saw in previous mailing thread
>> that
>> it is on the roadmap. Is it still on the roadmap ?
>>
>> * There is a workaround which is using timers to clear the state, but in
>> our case, it means firing billons of timers on daily basis all at the same
>> time, which seems no to very efficient and might cause some resources
>> issues
>>
>> Cheers
>> Avi
>>
>>
>>

Re: Queryable state and TTL

[Andrey Zagrebin]

Hi Avi,

It is on the road map but I am not aware about plans of any contributor to
work on it for the next releases.
I think the community will firstly work on the event time support for TTL.
I will loop Yu in, maybe he has some plans to work on TTL for the queryable
state.

Best,
Andrey

On Wed, Jul 3, 2019 at 3:17 PM Avi Levi  wrote:

> Hi,
> Adding queryable state to state with ttl is not supported at 1.8.0
> (throwing java.lang.IllegalArgumentException: Queryable state is currently
> not supported with TTL)
>
> I saw in previous mailing thread
> that
> it is on the roadmap. Is it still on the roadmap ?
>
> * There is a workaround which is using timers to clear the state, but in
> our case, it means firing billons of timers on daily basis all at the same
> time, which seems no to very efficient and might cause some resources
> issues
>
> Cheers
> Avi
>
>
>

Queryable state and TTL

[Avi Levi]

Hi,
Adding queryable state to state with ttl is not supported at 1.8.0
(throwing java.lang.IllegalArgumentException: Queryable state is currently
not supported with TTL)

I saw in previous mailing thread
that
it is on the roadmap. Is it still on the roadmap ?

* There is a workaround which is using timers to clear the state, but in
our case, it means firing billons of timers on daily basis all at the same
time, which seems no to very efficient and might cause some resources
issues

Cheers
Avi