practically impossible to tell when a store has been
converted
from
active
to
standby, or vice versa. So having access to the
SuspendReason,
and
more
importantly having a callback guaranteed to notify you
when a
state store is recycled whether active or standby,
would
b
;>>>>>>>>> tasks.
> >>>>>>>>>>>>>>>>>> JMX metrics are not fine-grained enough to give us an
> >>>>> accurate
> >>>>>>>>>>>>>>
recycling
process.
It's
practically impossible to tell when a store has been
converted
from
active
to
standby, or vice versa. So having access to the
SuspendReason,
and
more
importantly having a callback guaranteed to notify you
when a
state store is recycled whether active or standby,
; here, since
> >> Restoration
> >>>>>>>>>>>>>>>> refers
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>&
we can.
I was actually considering proposing a short KIP with a
new
RecyclingListener (or something) specifically for this
exact
kind of
thing,
since we
currently have literally zero insight into the recycling
process.
It's
practically impossible to tell when a store has been
converted
fr
restart, and
> > > >>>>>>>>>> then
> > > >>>>>>>>>>>> after the whole restart, the Task is shuffled onto an
> > instance
> > > >>>>>>>> where
> > > >>>>>>>&
rd no-go" for me, and we will remove it
> > (I'll
> > >>>>>>>> have to
> > >>>>>>>>>>>> check with Eduwer as he is close to having a working
> > >>>>>>>> implementation). I
> > >&g
gt;>>>> first
> >>>>>>>>>>>> call to `onBatch{Restored/Updated/Processed/Loaded}()`.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Anyways, I've updated the JavaDoc in the interface;
> >>>>>>>>>>>> hope
cyclingListener (or something) specifically for this exact
kind of
thing,
since we
currently have literally zero insight into the recycling
process.
It's
practically impossible to tell when a store has been converted
from
active
to
standby, or vice versa. So having access to the SuspendReason,
ible to tell when a store has been converted
from
active
to
standby, or vice versa. So having access to the SuspendReason,
and
more
importantly having a callback guaranteed to notify you when a
state store is recycled whether active or standby, would be
amazing.
Thanks for the KIP!
-Sophie
lly considering proposing a short KIP with a new
RecyclingListener (or something) specifically for this exact
kind of
thing,
since we
currently have literally zero insight into the recycling
process.
It's
practically impossible to tell when a store has been converted
from
active
to
stan
y have literally zero insight into the recycling process.
It's
practically impossible to tell when a store has been converted
from
active
to
standby, or vice versa. So having access to the SuspendReason,
and
more
importantly having a callback guaranteed to notify you when a
state
gt; example
> >>>>>> at
> >>>>>>>> the end of the Motivation section. How are standby tasks (and the
> >>>>>> ability
> >>>>>>>> to hook into and monitor their status) related to the
> >>>>>> s
ll when a store has been converted
from
active
to
standby, or vice versa. So having access to the SuspendReason,
and
more
importantly having a callback guaranteed to notify you when a
state store is recycled whether active or standby, would be
amazing.
Thanks for the KIP!
-Sophie "ott
; > > >
> > > > > > > 3b. #onTaskCreated
> > > > > > > I know the "start" callback feels a bit different for the
> > standby
> > > > task
> > > > > >
d give us #onUpdateStart. Personally I like this better,
> > > > > > but it's ultimately up to you. However, I would push back against
> > > > anything
> > > > > > that includes the word "Task" (eg #onTaskCreated) as the listener
> > > > > > is actually not s
eel like this might be more
> > > > > confusing than it
> > > > > is helpful (tbh even I'm not completely sure I know what the
> > > earliestOffset
> > > > > is supposed to represent) More importantly, is this all infor
> > > 3c. #onBatchRestored
> > > > If we opt to use the term "update" in place of "restore" elsewhere,
> > then we
> > > > should consider doing so here as well. What do you think about
> > > > #onBatchUpdated, or even #onBatc
ed probably makes more sense for this callback. Also,
> > > I notice the StateRestoreListener passes in the total number of
> records
> > > restored to its #onRestoreSuspended. Assuming we already track
> > > that information in Streams and have it readily available to pass in at
> >
> 4b. Assuming we do 4a, let's rename PROMOTED to RECYCLED -- for standby
> > tasks it means basically the same thing, the point is that active
> > tasks can also be recycled into standbys through the same mechanism. This
> > way they can share the SuspendReason enum -- not th
thing,
> since we
> currently have literally zero insight into the recycling process. It's
> practically impossible to tell when a store has been converted from active
> to
> standby, or vice versa. So having access to the SuspendReason, and more
> importantly having a ca
or standby, would be amazing.
Thanks for the KIP!
-Sophie "otterStandbyTaskUpdateListener :P" Blee-Goldman
------ Forwarded message -----
> From: Colt McNealy
> Date: Tue, Oct 3, 2023 at 12:48 PM
> Subject: [DISCUSS] KIP-988 Streams Standby Task Update Listener
> To
Hi all,
We would like to propose a small KIP to improve the ability of Streams apps
to monitor the progress of their standby tasks through a callback interface.
We have a nearly-working implementation on our fork and are curious for
feedback.
23 matches
Mail list logo
