Hi all,
In the past, I have tried to further refine the design of this topic thread
and wrote a design document to give more detailed design images and text
description, so that it is more conducive to discussion.[1]
Note: The document is not yet completed, for example, the "Implementation"
Hi Georgi, Thanks for your feedback. And glad to hear you are using queryable
state. I agree that implementation of option 1 is easier than others. However,
when we design the new architecture we need to consider more aspects .e.g.
scalability. So it seems option 3 is more suitable. Actually,
Hi Vino,
I was investigating the current architecture and AFAIK the first proposal will
be a lot easier to implement, cause currently JM has the information about the
states (where, which etc thanks to KvStateLocationRegistry. Correct me if I’m
wrong)
We are using the feature and it’s indeed
Hi Aljoscha,
Thanks for agreeing this is a valuable idea. I know the committers and PMCs
are busy before Flink 1.9 even 2.0. We think it's a good improvement for
queryable state and even some interactive query scenarios.
I don't mind if the timeline will be pulled long.
Although, it could not be
Hi Everyone,
I think this is a good discussion and valuable ideas have come up. However, it
seems none of the committers and/or PMCs currently have time to work on this
subject. Till, who’s focusing on the distributed runtime side, which is touched
quite a bit by queryable state, is currently
Hi Elias,
OK, I think we do not need to agree on this point of view. In order to make
the discussion more efficient, we need to focus a bit, let's talk about the
query architecture's improvement.
Best,
Vino
Elias Levy 于2019年4月30日周二 上午1:06写道：
> On Fri, Apr 26, 2019 at 8:58 PM vino yang wrote:
On Fri, Apr 26, 2019 at 8:58 PM vino yang wrote:
> I agree with your opinion that "*Flink jobs don't sufficiently meet these
> requirements to work as a replacement for a data store.*". Actually, I
> think it's obviously not Flink's goal.
>
I would not be so sure. When data Artisans
Hi Yu,
OK, now I know your comments more clearly.
Now, answer your two questions:
1. the value of this work:
As I mentioned in the last reply mail to you: "we found the queryable state
is hard to use and it may cause few users to use this function. We may
think the reason and the result affect
TL;DR: IMO a more complete solution is to cover both query and meta request
serving in a new component. We could use the proposal here as step one but
we should have a global plan. And before improving a seemingly not widely
used feature, we'd better weigh the gain and efforts.
Let me clarify the
Hi yu,
Thanks for your reply. I have some inline comment.
Yu Li 于2019年4月28日周日 下午12:24写道：
> Glad to see discussions around QueryableState in mailing list, and it seems
> we have included a bigger scope in the discussion, that what's the data
> model in Flink and how to (or is it possible to)
Hi Elias,
I agree with your opinion that "*Flink jobs don't sufficiently meet these
requirements to work as a replacement for a data store.*". Actually, I
think it's obviously not Flink's goal. If we think that the database
contains the main two parts(inexactitude): data query and data store.
On Fri, Apr 26, 2019 at 1:41 AM vino yang wrote:
> You are right, currently, the queryable state has few users. And I totally
> agree with you, it makes the streaming works more like a DB.
>
Alas, I don't think queryable state will really be used much in production
other than for ad hoc queries
Hi Paul,
Thanks for your reply.
You are right, currently, the queryable state has few users. And I totally
agree with you, it makes the streaming works more like a DB.
About the architecture and the problem you concern: yes, it maybe
affect the JobManager if they are deployed together.
I think
Hi Vino,
Thanks a lot for bringing up the discussion! Queryable state has been at beta
version for a long time, and due to its complexity and instability I think
there are not many users, but there’s a great value in it which makes state as
database one step closer.
WRT the architecture, I’d
Hi Quan,
Thanks for your reply.
Actually, I did not try this way.
But, there are two factors we should consider:
1. The local state storage is not equals to RocksDB, otherwise Flink
does not need to provide a queryable state client. What's more, querying
the RocksDB is still an
Hi,
How about take states from RocksDB directly, in this case, TM host is
unnecessary.
Best
Quan Shi
From: vino yang
Sent: Thursday, April 25, 2019 10:18:20 PM
To: dev; user
Cc: Stefan Richter; Aljoscha Krettek; kklou...@gmail.com
Subject: [DISCUSS] Improve
16 matches
Mail list logo
