[ https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
BrianLing updated KAFKA-3465: ----------------------------- Description: 1. When we enable mirrorMake to migrate Kafka event from one to other with "new.consumer" mode: java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka/kafka-app-logs -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log --consumer.config ../config/consumer.properties --new.consumer --num.streams 4 --producer.config ../config/producer-slca.properties --whitelist risk.* 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and the owner is none. bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 --topic XXXX Group Topic Pid Offset logSize Lag Owner lvs.slca.mirrormaker XXXX 0 418578332 418678347 100015 none lvs.slca.mirrormaker XXXX 1 418598026 418698338 100312 none [Root Cause] I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka internal to store offset & consumer owner information. Does it mean we can not use the below command to check new consumer’s lag since current lag formula: lag= logSize – offset https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80 https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182 => offSet Fetch from zookeeper instead of from Kafka was: When we enable mirrorMake to migrate Kafka event from one to other with "new.consumer" mode: java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka/kafka-app-logs -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log --consumer.config ../config/consumer.properties --new.consumer --num.streams 4 --producer.config ../config/producer-slca.properties --whitelist risk.* When we use ConsumerOffzsetChecker tool, notice the lag won't changed and the owner is none. bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 --topic risk.radd.acct_misc01 Group Topic Pid Offset logSize Lag Owner lvs.slca.mirrormaker XXXX 0 418578332 418678347 100015 none lvs.slca.mirrormaker XXXX 1 418598026 418698338 100312 none I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka internal to store offset & consumer owner information. Does it mean we can not use the below command to check new consumer’s lag since current lag formula: lag= logSize – offset https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80 https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182 => offSet Fetch from zookeeper instead of from Kafka > kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode > -------------------------------------------------------------------------- > > Key: KAFKA-3465 > URL: https://issues.apache.org/jira/browse/KAFKA-3465 > Project: Kafka > Issue Type: Improvement > Components: core > Affects Versions: 0.9.0.0 > Reporter: BrianLing > > 1. When we enable mirrorMake to migrate Kafka event from one to other with > "new.consumer" mode: > java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 > -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC > -Djava.awt.headless=true -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dkafka.logs.dir=/kafka/kafka-app-logs > -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties > -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* > -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log > --consumer.config ../config/consumer.properties --new.consumer --num.streams > 4 --producer.config ../config/producer-slca.properties --whitelist risk.* > 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and > the owner is none. > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info > --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 > --topic XXXX > Group Topic Pid Offset logSize > Lag Owner > lvs.slca.mirrormaker XXXX 0 418578332 418678347 100015 > none > lvs.slca.mirrormaker XXXX 1 418598026 418698338 100312 > none > [Root Cause] > I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka > internal to store offset & consumer owner information. > Does it mean we can not use the below command to check new consumer’s > lag since current lag formula: lag= logSize – offset > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80 > > https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182 > => offSet Fetch from zookeeper instead of from Kafka -- This message was sent by Atlassian JIRA (v6.3.4#6332)