Hello,
To give an extra datapoint: after a not so successful experiment with
faust-streaming we moved our application to flink. Since flinks
queryable state was apparently stagnant, we implemented what was needed
to sink the state to an external data store for querying.
However, if queryable state was in good shape we would definately have
used it. Making sure that the state is always reflected correctly in our
external system turned out to be non-trivial for a number of reasons:
our state is not trivially convertable to rows in a table, and sometimes
we had (due to our own bugs, but still) inconsistencies between the
internal flink state and the externally materialized state, especially
after replaying from a checkpoint/savepoint after a crash (we cannot use
exactly_once sinks in all occasions).
Also, obviously, we could not use flinks partitioning/parallellism to
help making state querying more scalable.
Greetings,
Frank
On 04.02.22 14:06, Dawid Wysakowicz wrote:
Hi Karthik,
The reason we deprecated it is because we lacked committers who could
spend time on getting the Queryable state to a production ready state.
I might be speaking for myself here, but I think the main use case for
the queryable state is to have an insight into the current state of
the application for debugging purposes. If it is used for data serving
purposes, we believe it's better to sink the data into an external
store, which can provide better discoverability and more user friendly
APIs for querying the results.
As for debugging/tracking insights you may try to achieve similar
results with metrics.
Best,
Dawid
On 01/02/2022 16:36, Jatti, Karthik wrote:
Hi,
I see on the Flink Roadmap that Queryable state API is scheduled to
be deprecated but I couldn’t find much information on confluence or
this mailing group’s archives to understand the background as to why
it’s being deprecated and what would be a an alternative. Any
pointers to help me get some more information here would be great.
Thanks,
Karthik
------------------------------------------------------------------------
The information in the email message containing a link to this page,
including any attachments thereto (collectively, “the e-mail”), is
only for use by the intended recipient(s). The e-mail may contain
information that is confidential, proprietary and/or privileged. If
you have reason to believe that you are not the intended recipient,
please notify the sender that you may have received this e-mail in
error and delete all copies of it, including attachments, from your
computer. Any viewing, copying, disclosure or distribution of this
information by an unintended recipient is prohibited and by an
intended recipient may be governed by arrangements in place between
the sender’s and recipient’s respective firms. Eze Software does not
represent that the e-mail is virus-free, complete or accurate. Eze
Software accepts no liability for any damage sustained in connection
with the content or transmission of the e-mail.
------------------------------------------------------------------------
Copyright © 2013 Eze Castle Software LLC. All Rights Reserved.
