Lerh Chuan Low created KAFKA-8547: ------------------------------------- Summary: 2 __consumer_offsets partitions grow very big Key: KAFKA-8547 URL: https://issues.apache.org/jira/browse/KAFKA-8547 Project: Kafka Issue Type: Bug Components: log cleaner Affects Versions: 2.1.1 Environment: Ubuntu 18.04, Kafka 2.1.12-2.1.1, running as systemd service Reporter: Lerh Chuan Low
There's a related issue here: https://issues.apache.org/jira/browse/KAFKA-3917, just thought it was a little bit outdated/dead. We observed a few out of memory errors with our Kafka servers and our theory was due to 2 overly large partitions in `__consumer_offsets`. On further digging, it looks like these 2 large partitions have segments dating up to 3 months ago. Also, these old files collectively consumed most of the data from those partitions (About 10G from the partition's 12G). When we tried dumping those old segments, we see: ``` 11:40 $ ./kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000161728257775.log --offsets-decoder --print-data-log --deep-iteration Dumping 00000000161728257775.log Starting offset: 161728257775 offset: 161728257904 position: 61 CreateTime: 1553457816168 isvalid: true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 367038 producerEpoch: 3 sequence: -1 isTransactional: true headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 746 offset: 161728258098 position: 200 CreateTime: 1553457816230 isvalid: true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 366036 producerEpoch: 3 sequence: -1 isTransactional: true headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 761 ... ``` It looks like all those old segments all contain transactional information, and the 2 partitions are 1 for the control message COMMIT, the other for the control message ABORT. (As a side note, we did take a while to figure out that for a segment with the control bit set, the key really is `endTxnMarker` and the value is `coordinatorEpoch`...otherwise in a non-control batch dump it would have value and payload. We were wondering if seeing what those 2 partitions contained in their keys may give us any clues). Our current workaround is based on this post: https://issues.apache.org/jira/browse/KAFKA-3917?focusedCommentId=16816874&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16816874. We set the cleanup policy to both compact,delete and very quickly the partition was down to below 2G. Not sure if this is something log cleaner should be able to handle normally? -- This message was sent by Atlassian JIRA (v7.6.3#76005)