Thanks for the followup Nick!
This is definitely one of those things that feels like it should be easy
and useful for everyone, but "the devil is in the details" as we always
say. It'll be good to have this thread and the discussion so far should
anyone want to attempt it in the future or need
Hi folks,
Sorry I haven't got back to you until now.
It's become clear that I hadn't anticipated a significant number of
technical challenges that this KIP presents. I think expecting users to
understand the ramifications on aggregations, joins and windowing
ultimately kills it: it only becomes
>
> Well, the KIP mentions the ability to either re-try the record (eg,
> after applying some external fix that would allow Kafka Streams to now
> deserialize the record now) or to skip it by advancing the offset.
That's fair -- you're definitely right that what's described in the KIP
document
Yes, about the "drop records" case. It's a very common scenario to have
a repartition step before a windowed aggregation or a join with
grace-period.
About "add feature vs guard users": it's always a tricky question and
tradeoff. For this particular KIP, I personally think we should opt to
>
> I see way too many food-guns and complications that can be introduced.
What is a "food-gun"?? I'm picturing like a spud rifle/potato gun but I
don't think that's what you meant hahaha
I don't feel super strongly one way or another, but I have a few questions
& corrections about some of
Hey Nick,
I am sorry that I have to say that I am not a fan of this KIP. I see way
too many food-guns and complications that can be introduced.
I am also not sure if I understand the motivation. You say, CONTINUE and
FAIL is not good enough, but don't describe in detail why? If we
1.
Woops! I've fixed that now. Thanks for catching that.
2.
I agree, I'll remove the LogAndPause handler so it's clear this is an
advanced feature. I'll also add some documentation to
DeserializationExceptionResponse#SUSPEND that explains the care users
should approach it with.
3a.
This is
1. Makes sense to me! Can you just update the name of the
DeserializationHandlerResponse enum from SUSPEND to PAUSE so
we're consistent with the wording?
The drawback here would be that custom stateful Processors
> might also be impacted, but there'd be no way to know if they're safe to
> not
Hi Sophie,
Thanks for the review!
1-3.
I had a feeling this was the case. I'm thinking of adding a PAUSED state
with the following valid transitions:
- RUNNING -> PAUSED
- PAUSED -> RUNNING
- PAUSED -> SUSPENDED
The advantage of a dedicated State is it should make testing easier and
Hey Nick,
A few high-level thoughts:
1. We definitely don't want to piggyback on the SUSPENDED task state, as
this is currently more like an intermediate state that a task passes
through as it's being closed/migrated elsewhere, it doesn't really mean
that a task is "suspended" and there's no
Hi everyone,
This is a Streams KIP to add a new DeserializationHandlerResponse,
"SUSPEND", that suspends the failing Task but continues to process other
Tasks normally.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-990%3A+Capability+to+SUSPEND+Tasks+on+DeserializationException
I'm not
11 matches
Mail list logo
