[
https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3465:
---------------------------------
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
> --------------------------------------------------------------------------
>
> Key: KAFKA-3465
> URL: https://issues.apache.org/jira/browse/KAFKA-3465
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.9.0.0, 0.11.0.0
> Reporter: BrianLing
> Assignee: Vahid Hashemian
> Priority: Minor
> Fix For: 1.0.0
>
>
> 1. When we enable mirrorMake to migrate Kafka event from one to other with
> "new.consumer" mode:
> java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC
> -Djava.awt.headless=true -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dkafka.logs.dir=/kafka/kafka-app-logs
> -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
> -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/*
> -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log
> --consumer.config ../config/consumer.properties --new.consumer --num.streams
> 4 --producer.config ../config/producer-slca.properties --whitelist risk.*
> 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and
> the owner is none.
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info
> --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181
> --topic XXXX
> Group Topic Pid Offset logSize
> Lag Owner
> lvs.slca.mirrormaker XXXX 0 418578332 418678347 100015
> none
> lvs.slca.mirrormaker XXXX 1 418598026 418698338 100312
> none
> [Root Cause]
> I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka
> internal to store offset & consumer owner information.
> Does it mean we can not use the below command to check new consumer’s
> lag since current lag formula: lag= logSize – offset
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80
>
> https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
> => offSet Fetch from zookeeper instead of from Kafka
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
