krishvishal opened a new pull request, #4229: URL: https://github.com/apache/iggy/pull/4229
`to_iggy` takes one Kafka record and returns one Iggy message. The value becomes the payload. The key and each header become `kafka.` user headers. A null or empty value is stored as a single placeholder byte plus a marker that says which of the two it was, so a tombstone stays on the fast path. Kafka counts timestamps in milliseconds and Iggy counts them in microseconds, so the value is multiplied by 1000 on the way in and divided on the way out. Some records do not fit. Iggy rejects an empty header value, caps a name and a value at 255 bytes, and caps all headers of one message at 100 KB. A record that trips any of those is written into an envelope instead, at the byte layout the document fixes. `from_iggy` reads both forms, and a caller cannot tell them apart. A message that no Kafka producer wrote is handled too. If it carries no `kafka.` headers, an Iggy client wrote it, so it becomes a record with a null key and its own headers. The batch layer sits on top. `decode_batches` drains a whole partition blob, because one blob holds several batches back to back. `encode_batch` writes one uncompressed batch for Fetch. Reading a compressed batch needs the four codecs, so `gzip`, `lz4`, `snappy` and `zstd` are turned back on for `kafka-protocol`. Compressed input needs a limit. `max_frame_size` only bounds the compressed bytes, and `zstd` reaches 1000 to 1 on repetitive input without being asked. One frame also holds up to 4096 partition entries, so a limit applied to each batch on its own still admits 4096 times that much. The budget therefore covers the whole request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
