ryerraguntla commented on code in PR #4205:
URL: https://github.com/apache/iggy/pull/4205#discussion_r4031248903
##########
gateways/kafka/docs/BRIDGE_MAPPING.md:
##########
@@ -0,0 +1,204 @@
+# Kafka to Iggy record mapping
+
+Status: proposed. Direction agreed with @spetz and @hubcio on 2026-09-16
("messages should be
+stored in iggy format"); the escape hatches below still need sign-off. Closes
the last open
+scope item of [#3533](https://github.com/apache/iggy/issues/3533) and blocks
+[#3535](https://github.com/apache/iggy/issues/3535) (Produce) and
+[#3536](https://github.com/apache/iggy/issues/3536) (Fetch).
+
+## Decision
+
+One Kafka record becomes one Iggy message, in Iggy's own format: the record
value is the
+message payload, the key and the Kafka headers become Iggy user headers. The
gateway rebuilds
+a Kafka record batch on Fetch.
+
+Two properties drive this.
+
+A Kafka consumer must be able to read a topic an Iggy producer wrote. This is
the staged
+migration the maintainers described: rewrite producers to the Iggy SDK first,
leave consumers
+on the gateway until later. The gateway can only encode an arbitrary Iggy
message as a Kafka
+record if the stored form has no Kafka framing in it.
+
+Kafka offsets must line up with Iggy offsets. A Kafka record batch carries N
records under one
+base offset, while an Iggy message consumes exactly one offset. Storing a
batch whole makes
+every offset the gateway reports wrong by the batch size, and fixing that
inside Iggy means
+teaching the server to count records inside an opaque payload.
+
+Storing the Kafka payload and headers as a dump inside the Iggy payload was
the alternative
+raised in the same thread. It does not satisfy the first property on its own:
the gateway would
+still need a native path for Iggy-written messages, so it would carry two
storage formats
+instead of one. Native storage with a narrow fallback (see below) keeps that
to one.
+
+## Field mapping
+
+Produce, per record:
+
+| Kafka | Iggy |
+| ------- | ------ |
+| record value | `payload` |
+| record key | `kafka.key` user header, `Raw` |
+| record header `name` | `kafka.h.<name>` user header, `Raw` |
+| record timestamp (CreateTime) | `origin_timestamp` |
Review Comment:
While converting, make sure of the resolution, milli seconds in Kafka to
microSeconds in Iggy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
