we have a lot of processes that need a time-bounded checkpoint logic.
the standard use case is some consume-process-produce logic of the
following form:
while (alive) {
data = consumer.poll()
output = process(data)
producer.send(output)
if (System.millis() > nextCheckpoint) {
nextCheckpoint = System.millis() + 5 minutes;
if (!producer.flush(30 seconds) || !consumer.commitSync(30 seconds)) {
//unable to checkpoint within timeout, die (or at least raise
some alarm)
}
}
}
an unbounded flush could cause the consumer to be considered dead and
send the whole app into rebalance storms, so we'd really love to be
able to put a time bound on it.
im also fine with rethrowing InterruptedException
On Thu, Sep 12, 2019 at 3:46 PM Colin McCabe <[email protected]> wrote:
>
> Hi Radai,
>
> Thanks for the KIP. Sounds interesting. I assume that if an
> InterruptedException were caught, that would be rethrown, rather than
> returning false? It might be good to specify that. Can you give an example
> of how this would be used?
>
> best,
> Colin
>
>
> On Thu, Sep 12, 2019, at 15:26, radai wrote:
> > bump.
> >
> > if no one has any comments on this can we initiate a vote?
> >
> > On Tue, Sep 3, 2019 at 8:28 AM KUN DU <[email protected]> wrote:
> > >
> > > Hi,
> > >
> > > I would like to start discussion on KIP-514 that proposes we add a
> > > bounded flush() API to producer.
> > >
> > > Link to the KIP:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer
> > >
> > > Suggestions and feedback are welcome!
> > >
> > > Thanks,
> > > Kun
> >
