ryerraguntla commented on code in PR #4229: URL: https://github.com/apache/iggy/pull/4229#discussion_r4058482958
########## gateways/kafka/docs/BRIDGE_MAPPING.md: ########## @@ -161,9 +165,16 @@ therefore still admit 4096 times that much output. Produce keeps a single decompression budget for the whole request, set to `max_frame_size`, so a compressed request can never yield more than the same client could have sent uncompressed. A -batch that exhausts the budget is rejected with `MESSAGE_TOO_LARGE` (10) before the output is -allocated. Each decompressed record value has to clear Iggy's own `MAX_PAYLOAD_SIZE` (64 MB, -`iggy_message.rs:44`) separately, since one record becomes one message. +batch that exhausts the budget is rejected with `MESSAGE_TOO_LARGE` (10). Each decompressed +record value has to clear Iggy's own `MAX_PAYLOAD_SIZE` (64 MB, `iggy_message.rs:44`) separately, +since one record becomes one message. + +The budget bounds what a request accumulates, not what one batch allocates. `kafka_protocol`'s +decompressors write the whole stream out before they hand it over (`compression/gzip.rs:46` and +its three siblings), so a single batch reaches its full decompressed size in memory and the +budget rejects it one step later. Bounding the peak needs a size-limited reader per codec, which +means owning Kafka's snappy and lz4 framing rather than borrowing it. That is worth doing and it +is not done here. Nothing decompresses today. The record batch stays an opaque `Bytes` on both paths, so the bound Review Comment: nit - “Nothing decompresses today” + :156 still features=["broker"]. This PR adds decode_batches + codecs. Fix: say Produce decompresses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
