ryerraguntla commented on code in PR #4205:
URL: https://github.com/apache/iggy/pull/4205#discussion_r4031323959
##########
gateways/kafka/docs/BRIDGE_MAPPING.md:
##########
@@ -0,0 +1,204 @@
+# Kafka to Iggy record mapping
+
+Status: proposed. Direction agreed with @spetz and @hubcio on 2026-09-16
("messages should be
+stored in iggy format"); the escape hatches below still need sign-off. Closes
the last open
+scope item of [#3533](https://github.com/apache/iggy/issues/3533) and blocks
+[#3535](https://github.com/apache/iggy/issues/3535) (Produce) and
+[#3536](https://github.com/apache/iggy/issues/3536) (Fetch).
+
+## Decision
+
+One Kafka record becomes one Iggy message, in Iggy's own format: the record
value is the
+message payload, the key and the Kafka headers become Iggy user headers. The
gateway rebuilds
+a Kafka record batch on Fetch.
+
+Two properties drive this.
+
+A Kafka consumer must be able to read a topic an Iggy producer wrote. This is
the staged
+migration the maintainers described: rewrite producers to the Iggy SDK first,
leave consumers
+on the gateway until later. The gateway can only encode an arbitrary Iggy
message as a Kafka
+record if the stored form has no Kafka framing in it.
+
+Kafka offsets must line up with Iggy offsets. A Kafka record batch carries N
records under one
+base offset, while an Iggy message consumes exactly one offset. Storing a
batch whole makes
+every offset the gateway reports wrong by the batch size, and fixing that
inside Iggy means
+teaching the server to count records inside an opaque payload.
+
+Storing the Kafka payload and headers as a dump inside the Iggy payload was
the alternative
+raised in the same thread. It does not satisfy the first property on its own:
the gateway would
+still need a native path for Iggy-written messages, so it would carry two
storage formats
+instead of one. Native storage with a narrow fallback (see below) keeps that
to one.
+
+## Field mapping
+
+Produce, per record:
+
+| Kafka | Iggy |
+| ------- | ------ |
+| record value | `payload` |
+| record key | `kafka.key` user header, `Raw` |
+| record header `name` | `kafka.h.<name>` user header, `Raw` |
+| record timestamp (CreateTime) | `origin_timestamp` |
+| record offset | partition offset, assigned by Iggy |
+| partition index | partition index, both 0-based |
+| topic | stream and topic per `TopicMapping` |
+
+Fetch reverses it. A message with no `kafka.*` headers is a message an Iggy
client wrote, and
+encodes as a Kafka record with a null key, its Iggy user headers as Kafka
headers, and
+`origin_timestamp` as the record timestamp (falling back to the
server-assigned `timestamp`
+when the origin timestamp is zero). Iggy header kinds other than `Raw` and
`String` are emitted
+as their raw value bytes.
+
+## Records Iggy cannot hold natively
+
+Iggy rejects an empty payload
(`core/common/src/types/message/iggy_message.rs:169`), caps a
+user header value at 255 bytes
(`core/common/src/types/message/user_headers.rs:631`), keys
+headers in a `BTreeMap` so a name cannot repeat, and caps all user headers of
a message at
+100 KB (`MAX_USER_HEADERS_SIZE`). Kafka allows all of the shapes those rules
exclude, so two
+mechanisms cover them.
+
+### Null and empty values
+
+A record with a null value (a tombstone) or a zero-length value is stored with
a single `0x00`
+byte payload and a `kafka.value` header holding `null` or `empty`. Fetch reads
that header and
+restores the original, discarding the placeholder byte.
+
+This keeps tombstones on the fast path rather than pushing them into the
fallback, because they
+are ordinary traffic on compacted Kafka topics. Iggy has no compaction, so a
tombstone is stored
+and served like any other record and nothing acts on it.
+
+### Everything else: the envelope fallback
+
+A record takes the fallback when any of these hold:
+
+- the key is longer than 255 bytes
+- a header name, prefixed with `kafka.h.`, is longer than 255 bytes
+- a header value is null, empty, or longer than 255 bytes
+- two headers share a name
+- the headers together would exceed the 100 KB user-header budget
+
+Such a record is stored with a `kafka.envelope` header carrying the format
version, and the
+Kafka record body (key, value, headers) verbatim in the payload. Fetch checks
for that header
+first and takes the plain path only when it is absent.
+
+The cost is that these messages are opaque to Iggy consumers and connectors.
That is the point
+of confining the fallback to record shapes that are rare in practice, rather
than making it the
+default storage form.
+
+## Batch-level fields
+
+Per-record storage drops what the Kafka record batch header carries: producer
id, producer
+epoch, base sequence, the transactional flag, compression and the batch CRC.
Fetch synthesizes
+a batch with producer id `-1`, epoch `-1`, base sequence `-1`, no compression,
`CreateTime`
+timestamps, and a recomputed CRC32C.
+
+Two consequences worth stating before they surprise someone:
+
+- Idempotent-producer deduplication cannot be reconstructed from stored data
later. If
+ [#3545](https://github.com/apache/iggy/issues/3545) ever grows past a stub,
producer id,
+ epoch and sequence need their own tracking.
+- The bytes a consumer receives are not the bytes the producer sent, so
anything comparing
+ batches byte for byte across the gateway will differ.
+
+Whether producer id `-1` is what Fetch actually sends depends on the
InitProducerId decision in
+[`IDEMPOTENCE.md`](IDEMPOTENCE.md). Allocating producer ids does not change
what is stored, only
+what Produce accepts, so this section holds under either answer.
+
+Produce decompresses gzip, snappy, lz4 and zstd batches, which means turning
those features back
Review Comment:
How to handle decompressed message exceeds IGGY_KAFKA_MAX_FRAME_SIZE ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
