I've had a small talk to the Kafka guys to find out a little bit more and
the oversimplified conclusion is that if
the producer version >= 3.0 and broker version < 0.11.0 with message format
version V1 then
either `enable.idempotence = false` needed or broker upgrade to 0.11.0+ is
required to make
Maybe I misunderstood this explanation.
Agreed. Spark relies on Kafka, Google Pub/Sub or any other messaging
systems to process the related streaming data via topic or topics and
present them to Spark. At this stage, Spark does not care to know how this
streaming data is produced. Spark relies on
If it requires a Kafka broker update, we should not simply bump the version
of Kafka client. Probably we should at least provide separate artifacts.
We should not enforce the upgrade of other component just because we want
to upgrade the dependency. At least it should not happen in minor versions
Well, yes, but if it requires a Kafka server-side update, it does, and that
is out of scope for us to document.
It is important that we document if and how (if we know) the client update
will impact existing Kafka installations (does it require a server-side
update or not?), and document the
Bump to try gathering more voices before taking action. For now, I see two
voices as option 2 & 5 (similar to option 2 but not in the migration node
but in the release note).
On Fri, Mar 18, 2022 at 7:15 PM Jungtaek Lim
wrote:
> CORRECTION: in option 2, we enumerate KIPs which may bring
The thing is, it is “us” who upgrades Kafka client and makes possible
divergence between client and broker in end users’ production env.
Someone can claim that end users can downgrade the kafka-client artifact
when building their app so that the version can be matched, but we don’t
test anything
I think we can assume that someone upgrading Kafka will be responsible for
thinking through the breaking changes. We can help by listing anything we
know could affect Spark-Kafka usage and calling those out in a release
note, for sure. I don't think we need to get into items that would affect
As always, I hope that the direction of the discussion would be focusing on
the topic. Let’s avoid ourselves to be side-tracked. Please consider the
mail thread as full context and feel free to ask me if there is a lack of
information for you to provide a voice.
Thanks for the voice in previous
I've just read the related PR and seems like the situation is not so black
and white as I've presumed purely from tech point of view...
On Fri, 18 Mar 2022, 12:44 Gabor Somogyi, wrote:
> Hi Jungtaek,
>
> I've taken a deeper look at the issue and here are my findings.
> As far as I'm concerned
Hi Jungtaek,
I've taken a deeper look at the issue and here are my findings.
As far as I'm concerned there are basically 2 ways with some minor
decorations:
* We care
* We don't care
I'm pretty sure users are clever enough but setting the expectation that
all users are tracking Kafka KIPs
CORRECTION: in option 2, we enumerate KIPs which may bring incompatibility
with older brokers (not all KIPs).
On Fri, Mar 18, 2022 at 7:12 PM Jungtaek Lim
wrote:
> Hi dev,
>
> I would like to initiate the discussion about how to deal with the
> migration guide on upgrading Kafka to 3.1 (from
11 matches
Mail list logo
