For the benchmarking, I have a couple of questions:
1. Re: the mention of exactly once, this is within a producer session,
right? And so really only idempotent. Applications still need to take extra
steps for exactly once if they, e.g., are producing data from some other
log like a DB txn log.
> Further, the results above show that there is a large improvement in
throughput and latency when we go from max.in.flight=1 to max.in.flight=2,
but then there no discernible difference for higher values of this setting.
If in the tests there's no difference with higher values, perhaps leaving
it alone is better. There are a bunch of other configs we expose and this
test only evaluates one environment. Without testing things like cross-DC
traffic, I'd be wary of jumping to the conclusion that max.in.flight > 2
never makes a difference and that some people aren't already relying on a
larger default OOTB.
3. The acks=all change is actually unrelated to the title of the KIP and
orthogonal to all the other changes. It's also the most risky since
acks=all needs more network round trips. And while I think it makes sense
to have the more durable default, this seems like it's actually fairly
likely to break things for some people (even if a minority of people). This
one seems like a setting change that needs more sensitive handling, e.g.
both release notes and log notification that the default is going to
change, followed by the actual change later.
On Tue, Aug 8, 2017 at 5:23 PM, Apurva Mehta <apu...@confluent.io> wrote:
> I've put together a new KIP which proposes to ship Kafka with its strongest
> delivery guarantees by default.
> We currently ship with at most once semantics and don't provide any
> ordering guarantees per partition. The proposal is is to provide exactly
> once in order delivery per partition by default in the upcoming 1.0.0
> The KIP linked to below also outlines the performance characteristics of
> the proposed default.
> The KIP is here:
> Please have a look, I would love your feedback!