[
https://issues.apache.org/jira/browse/KAFKA-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-1011:
--------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.10.1.0)
Status: Resolved (was: Patch Available)
MirrorMaker was rewritten, compression was re-written, lets just assume it was
fixed :)
> Decompression and re-compression on MirrorMaker could result in messages
> being dropped in the pipeline
> ------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-1011
> URL: https://issues.apache.org/jira/browse/KAFKA-1011
> Project: Kafka
> Issue Type: Bug
> Reporter: Guozhang Wang
> Assignee: Guozhang Wang
> Attachments: KAFKA-1011.v1.patch
>
>
> The way MirrorMaker works today is that its consumers could use deep iterator
> to decompress messages received from the source brokers and its producers
> could re-compress the messages while sending them to the target brokers.
> Since MirrorMakers use a centralized data channel for its consumers to pipe
> messages to its producers, and since producers would compress messages with
> the same topic within a batch as a single produce request, this could result
> in messages accepted at the front end of the pipeline being dropped at the
> target brokers of the MirrorMaker due to MesageSizeTooLargeException if it
> happens that one batch of messages contain too many messages of the same
> topic in MirrorMaker's producer. If we can use shallow iterator at the
> MirrorMaker's consumer side to directly pipe compressed messages this issue
> can be fixed.
> Also as Swapnil pointed out, currently if the MirrorMaker lags and there are
> large messages in the MirrorMaker queue (large after decompression), it can
> run into an OutOfMemoryException. Shallow iteration will be very helpful in
> avoiding this exception.
> The proposed solution of this issue is also related to KAFKA-527.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
