macdoor commented on PR #13040:
URL: https://github.com/apache/apisix/pull/13040#issuecomment-4022874321
Short summary of the current findings for Kafka 4.x support in
`kafka-logger`:
**With the current `lua-resty-kafka` implementation, Apache Kafka 4.x and
`kafka-logger` are *protocol incompatible*. Simply changing `api_version`
cannot fix this.**
Why:
1. **`lua-resty-kafka` only implements the legacy MessageSet format**
Regardless of whether `api_version` is set to 0, 1 or 2,
`lua-resty-kafka` still produces messages using **message formats v0/v1 (magic
0/1) — i.e. MessageSet**.
It does **not** implement message format v2 (magic 2) / RecordBatch.
2. **Apache Kafka 4.x no longer accepts MessageSet; it only supports
RecordBatch**
According to KIP‑724 ("Drop support for message formats v0 and v1") and
the Kafka 4.0 message format docs, Kafka 4.x brokers **only support message
format v2 (magic 2), i.e. RecordBatch**, and have completely dropped support
for formats v0/v1 (MessageSet).
See:
[KIP‑724](https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1),
[Kafka 4.0 Message
Format](https://kafka.apache.org/40/implementation/message-format/).
As a result:
> `lua-resty-kafka` can only send MessageSet, while Kafka 4.x only accepts
RecordBatch.
> The two are incompatible at the protocol level; no `api_version` setting
can change that with the current client.
So this PR’s approach — exposing `api_version` and documenting
`api_version=2` for Kafka 4.x — is still valuable for **Kafka < 4.0** (and for
making the behavior explicit), but **it cannot by itself deliver full Kafka 4.x
support**.
To truly support Kafka 4.x in `kafka-logger`, we will need:
- Either to enhance `lua-resty-kafka` to implement RecordBatch (magic 2), or
- Introduce/wrap another Kafka client that already supports RecordBatch and
have `kafka-logger` use that.
Until that happens, "`kafka-logger` + Kafka 4.x" should be treated as
**experimental / not reliably supported**, not as finished Kafka 4.x support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
