ryerraguntla commented on code in PR #4205:
URL: https://github.com/apache/iggy/pull/4205#discussion_r4031290720
##########
gateways/kafka/docs/BRIDGE_MAPPING.md:
##########
@@ -0,0 +1,204 @@
+# Kafka to Iggy record mapping
+
+Status: proposed. Direction agreed with @spetz and @hubcio on 2026-09-16
("messages should be
+stored in iggy format"); the escape hatches below still need sign-off. Closes
the last open
+scope item of [#3533](https://github.com/apache/iggy/issues/3533) and blocks
+[#3535](https://github.com/apache/iggy/issues/3535) (Produce) and
+[#3536](https://github.com/apache/iggy/issues/3536) (Fetch).
+
+## Decision
+
+One Kafka record becomes one Iggy message, in Iggy's own format: the record
value is the
+message payload, the key and the Kafka headers become Iggy user headers. The
gateway rebuilds
+a Kafka record batch on Fetch.
+
+Two properties drive this.
+
+A Kafka consumer must be able to read a topic an Iggy producer wrote. This is
the staged
+migration the maintainers described: rewrite producers to the Iggy SDK first,
leave consumers
+on the gateway until later. The gateway can only encode an arbitrary Iggy
message as a Kafka
+record if the stored form has no Kafka framing in it.
+
+Kafka offsets must line up with Iggy offsets. A Kafka record batch carries N
records under one
+base offset, while an Iggy message consumes exactly one offset. Storing a
batch whole makes
+every offset the gateway reports wrong by the batch size, and fixing that
inside Iggy means
+teaching the server to count records inside an opaque payload.
+
+Storing the Kafka payload and headers as a dump inside the Iggy payload was
the alternative
+raised in the same thread. It does not satisfy the first property on its own:
the gateway would
+still need a native path for Iggy-written messages, so it would carry two
storage formats
+instead of one. Native storage with a narrow fallback (see below) keeps that
to one.
+
+## Field mapping
+
+Produce, per record:
+
+| Kafka | Iggy |
+| ------- | ------ |
+| record value | `payload` |
+| record key | `kafka.key` user header, `Raw` |
+| record header `name` | `kafka.h.<name>` user header, `Raw` |
+| record timestamp (CreateTime) | `origin_timestamp` |
+| record offset | partition offset, assigned by Iggy |
+| partition index | partition index, both 0-based |
+| topic | stream and topic per `TopicMapping` |
+
+Fetch reverses it. A message with no `kafka.*` headers is a message an Iggy
client wrote, and
+encodes as a Kafka record with a null key, its Iggy user headers as Kafka
headers, and
+`origin_timestamp` as the record timestamp (falling back to the
server-assigned `timestamp`
+when the origin timestamp is zero). Iggy header kinds other than `Raw` and
`String` are emitted
+as their raw value bytes.
+
+## Records Iggy cannot hold natively
+
+Iggy rejects an empty payload
(`core/common/src/types/message/iggy_message.rs:169`), caps a
+user header value at 255 bytes
(`core/common/src/types/message/user_headers.rs:631`), keys
+headers in a `BTreeMap` so a name cannot repeat, and caps all user headers of
a message at
+100 KB (`MAX_USER_HEADERS_SIZE`). Kafka allows all of the shapes those rules
exclude, so two
+mechanisms cover them.
+
+### Null and empty values
+
+A record with a null value (a tombstone) or a zero-length value is stored with
a single `0x00`
+byte payload and a `kafka.value` header holding `null` or `empty`. Fetch reads
that header and
+restores the original, discarding the placeholder byte.
+
+This keeps tombstones on the fast path rather than pushing them into the
fallback, because they
+are ordinary traffic on compacted Kafka topics. Iggy has no compaction, so a
tombstone is stored
+and served like any other record and nothing acts on it.
+
+### Everything else: the envelope fallback
+
+A record takes the fallback when any of these hold:
+
+- the key is longer than 255 bytes
+- a header name, prefixed with `kafka.h.`, is longer than 255 bytes
+- a header value is null, empty, or longer than 255 bytes
+- two headers share a name
+- the headers together would exceed the 100 KB user-header budget
Review Comment:
Is 100Kb an optimal number or decided with any data point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
