[
https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BrianLing updated KAFKA-3465:
-----------------------------
Description:
1. When we enable mirrorMake to migrate Kafka event from one to other with
"new.consumer" mode:
java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC
-Djava.awt.headless=true -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka/kafka-app-logs
-Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
-cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/*
-Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log
--consumer.config ../config/consumer.properties --new.consumer --num.streams 4
--producer.config ../config/producer-slca.properties --whitelist risk.*
2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and
the owner is none.
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group
lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 --topic XXXX
Group Topic Pid Offset logSize
Lag Owner
lvs.slca.mirrormaker XXXX 0 418578332 418678347 100015
none
lvs.slca.mirrormaker XXXX 1 418598026 418698338 100312
none
[Root Cause]
I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka
internal to store offset & consumer owner information.
Does it mean we can not use the below command to check new consumer’s
lag since current lag formula: lag= logSize – offset
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80
https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
=> offSet Fetch from zookeeper instead of from Kafka
was:
When we enable mirrorMake to migrate Kafka event from one to other with
"new.consumer" mode:
java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC
-Djava.awt.headless=true -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka/kafka-app-logs
-Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
-cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/*
-Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log
--consumer.config ../config/consumer.properties --new.consumer --num.streams 4
--producer.config ../config/producer-slca.properties --whitelist risk.*
When we use ConsumerOffzsetChecker tool, notice the lag won't changed and the
owner is none.
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group
lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 --topic
risk.radd.acct_misc01
Group Topic Pid Offset logSize
Lag Owner
lvs.slca.mirrormaker XXXX 0 418578332 418678347 100015
none
lvs.slca.mirrormaker XXXX 1 418598026 418698338 100312
none
I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka
internal to store offset & consumer owner information.
Does it mean we can not use the below command to check new consumer’s
lag since current lag formula: lag= logSize – offset
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80
https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
=> offSet Fetch from zookeeper instead of from Kafka
> kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
> --------------------------------------------------------------------------
>
> Key: KAFKA-3465
> URL: https://issues.apache.org/jira/browse/KAFKA-3465
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.9.0.0
> Reporter: BrianLing
>
> 1. When we enable mirrorMake to migrate Kafka event from one to other with
> "new.consumer" mode:
> java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC
> -Djava.awt.headless=true -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dkafka.logs.dir=/kafka/kafka-app-logs
> -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
> -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/*
> -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log
> --consumer.config ../config/consumer.properties --new.consumer --num.streams
> 4 --producer.config ../config/producer-slca.properties --whitelist risk.*
> 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and
> the owner is none.
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info
> --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181
> --topic XXXX
> Group Topic Pid Offset logSize
> Lag Owner
> lvs.slca.mirrormaker XXXX 0 418578332 418678347 100015
> none
> lvs.slca.mirrormaker XXXX 1 418598026 418698338 100312
> none
> [Root Cause]
> I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka
> internal to store offset & consumer owner information.
> Does it mean we can not use the below command to check new consumer’s
> lag since current lag formula: lag= logSize – offset
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80
>
> https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
> => offSet Fetch from zookeeper instead of from Kafka
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
