krishvishal opened a new issue, #4517: URL: https://github.com/apache/iggy/issues/4517
## Summary A record with a repeated header name gets `INVALID_RECORD` (87), and Fetch returns headers sorted by name (`gateways/kafka/docs/BRIDGE_MAPPING.md`). Kafka allows both. Who hits it: - KafkaJS writes array header values as repeated keys, a documented feature. If it forwards a consumed message, it writes them again. The KafkaJS layer of confluent-kafka-javascript does the same. - Apps that append a trace header twice (https://github.com/spring-projects/spring-kafka/issues/3786). - Error 87 fails the whole partition batch in librdkafka, franz-go and Sarama. It puts a kafka-python 3.0 producer in a fatal state, so every later send fails until a restart. A kafka-go writer at its default `acks=0` loses the batch and the connection, with no error. ## Open question Does every record keep its header order, at the cost of a larger stored envelope, or only records with repeated names? ## Scope - [ ] Read headers from the raw record bytes, because `kafka_protocol` keeps only the last value of a name - [ ] Store a record with repeated names in the envelope, which keeps a list - [ ] Return headers in their original order on Fetch, as the open question decides - [ ] Update `gateways/kafka/docs/BRIDGE_MAPPING.md` ## Acceptance criteria - [ ] A KafkaJS producer with array header values succeeds, and a consumer reads every value in order - [ ] A record with the same header name twice is the same after Produce and Fetch ## References - KafkaJS: https://github.com/tulios/kafkajs/tree/v2.2.4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
