macdoor commented on issue #12984:
URL: https://github.com/apache/apisix/issues/12984#issuecomment-4022874348
In summary, this PR has revealed a more fundamental limitation:
**With the current `lua-resty-kafka` implementation, Apache Kafka 4.x and
`kafka-logger` are *protocol incompatible*. Simply changing `api_version`
cannot fix this.**
The reasons are:
1. **`lua-resty-kafka` only implements the legacy MessageSet format**
Regardless of whether `api_version` is set to 0, 1 or 2,
`lua-resty-kafka` still produces messages using **message formats v0/v1 (magic
0/1) — i.e. MessageSet**.
It does **not** implement message format v2 (magic 2) / RecordBatch.
2. **Apache Kafka 4.x no longer accepts MessageSet; it only supports
RecordBatch**
According to KIP‑724 ("Drop support for message formats v0 and v1") and
the Kafka 4.0 message format docs, Kafka 4.x brokers **only support message
format v2 (magic 2), i.e. RecordBatch**, and have completely dropped support
for formats v0/v1 (MessageSet).
See e.g.
[KIP‑724](https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1)
and [Kafka 4.0 Message
Format](https://kafka.apache.org/40/implementation/message-format/).
Putting these together:
> `lua-resty-kafka` can only send MessageSet, while Kafka 4.x only accepts
RecordBatch.
> The two are incompatible at the protocol level; no `api_version` setting
can change that with the current client.
So the idea in this issue/PR — "set default `api_version=2` and make it
configurable to support Kafka 4.x" — is still useful for **older Kafka versions
(< 4.0)**, but **it cannot solve the fundamental compatibility problem with
Kafka 4.x**.
To truly support Kafka 4.x, we would need:
- Either to extend `lua-resty-kafka` to implement RecordBatch (magic 2), or
- Introduce/wrap another Kafka client in APISIX that already supports
RecordBatch, and have `kafka-logger` use that.
Until then, "`kafka-logger` + Kafka 4.x" should be considered **experimental
/ not reliably supported**, and we should not advertise it as full Kafka 4.x
support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
