[ 
https://issues.apache.org/jira/browse/KAFKA-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118941#comment-16118941
 ] 

Fernando Vega commented on KAFKA-5407:
--------------------------------------

[~jasong35] or [~hachikuji] I do not have those logs with me anymore. Sorry.

I dont remember seeing an exception on the broker logs and I wish I had that 
log file with me just to make sure.

We manage bring the cluster back by using the mirrormaker with the new 
version(10.2.1) but using the old mirrormaker config (using zookeeper). 

However what we have notice is that the throughput is not the same in the 
beginning it degraded a lot from 40Mbps to 10Mbps outbound and inbound, we  
tweak the configs a little bit and manage to pump the throughput but reached 
35Mbps still lower that when we were running 0.8.2-beta.

Any recomendation will be awesome, this is still a upgrade in process, but we 
need to run the upgrade in a more critical set of clusters. Also I will like to 
mentioned that this is still on this version until we finish the upgrade:

{noformat}
inter.broker.protocol.version=0.10.1
log.message.format.version=0.8.2
{noformat}


> Mirrormaker dont start after upgrade
> ------------------------------------
>
>                 Key: KAFKA-5407
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5407
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.10.2.1
>         Environment: Operating system
> CentOS 6.8
> HW
> Board Mfg             : HP
>  Board Product         : ProLiant DL380p Gen8
> CPU's x2
> Product Manufacturer  : Intel
>  Product Name          :  Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
>  Memory Type           : DDR3 SDRAM
>  SDRAM Capacity        : 2048 MB
>  Total Memory:             : 64GB
> Hardrives size and layout:
> 9 drives using jbod
> drive size 3.6TB each
>            Reporter: Fernando Vega
>            Priority: Critical
>
> Currently Im upgrading the cluster from 0.8.2-beta to 0.10.2.1
> So I followed the rolling procedure:
> Here the config files:
> Consumer
> {noformat}
> #
> # Cluster: repl
> # Topic list(goes into command line): 
> REPL-ams1-global,REPL-atl1-global,REPL-sjc2-global,REPL-ams1-global-PN_HXIDMAP_.*,REPL-atl1-global-PN_HXIDMAP_.*,REPL-sjc2-global-PN_HXIDMAP_.*,REPL-ams1-global-PN_HXCONTEXTUALV2_.*,REPL-atl1-global-PN_HXCONTEXTUALV2_.*,REPL-sjc2-global-PN_HXCONTEXTUALV2_.*
> bootstrap.servers=app001:9092,app002:9092,app003:9092,app004:9092
> group.id=hkg1_cluster
> auto.commit.interval.ms=60000
> partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
> {noformat}
> Producer
> {noformat}
>  hkg1
> # # Producer
> # # hkg1
> bootstrap.servers=app001:9092,app002:9092,app003:9092,app004:9092
> compression.type=gzip
> acks=0
> {noformat}
> Broker
> {noformat}
> auto.leader.rebalance.enable=true
> delete.topic.enable=true
> socket.receive.buffer.bytes=1048576
> socket.send.buffer.bytes=1048576
> default.replication.factor=2
> auto.create.topics.enable=true
> num.partitions=1
> num.network.threads=8
> num.io.threads=40
> log.retention.hours=1
> log.roll.hours=1
> num.replica.fetchers=8
> zookeeper.connection.timeout.ms=30000
> zookeeper.session.timeout.ms=30000
> inter.broker.protocol.version=0.10.2
> log.message.format.version=0.8.2
> {noformat}
> I tried also using stock configuraiton with no luck.
> The error that I get is this:
> {noformat}
> 2017-06-07 12:24:45,476] INFO ConsumerConfig values:
>       auto.commit.interval.ms = 60000
>       auto.offset.reset = latest
>       bootstrap.servers = [app454.sjc2.mytest.com:9092, 
> app455.sjc2.mytest.com:9092, app456.sjc2.mytest.com:9092, 
> app457.sjc2.mytest.com:9092, app458.sjc2.mytest.com:9092, 
> app459.sjc2.mytest.com:9092]
>       check.crcs = true
>       client.id = MirrorMaker_hkg1-1
>       connections.max.idle.ms = 540000
>       enable.auto.commit = false
>       exclude.internal.topics = true
>       fetch.max.bytes = 52428800
>       fetch.max.wait.ms = 500
>       fetch.min.bytes = 1
>       group.id = MirrorMaker_hkg1
>       heartbeat.interval.ms = 3000
>       interceptor.classes = null
>       key.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
>       max.partition.fetch.bytes = 1048576
>       max.poll.interval.ms = 300000
>       max.poll.records = 500
>       metadata.max.age.ms = 300000
>       metric.reporters = []
>       metrics.num.samples = 2
>       metrics.recording.level = INFO
>       metrics.sample.window.ms = 30000
>       partition.assignment.strategy = 
> [org.apache.kafka.clients.consumer.RoundRobinAssignor]
>       receive.buffer.bytes = 65536
>       reconnect.backoff.ms = 50
>       request.timeout.ms = 305000
>       retry.backoff.ms = 100
>       sasl.jaas.config = null
>       sasl.kerberos.kinit.cmd = /usr/bin/kinit
>       sasl.kerberos.min.time.before.relogin = 60000
>       sasl.kerberos.service.name = null
>       sasl.kerberos.ticket.renew.jitter = 0.05
>       sasl.kerberos.ticket.renew.window.factor = 0.8
>       sasl.mechanism = GSSAPI
>       security.protocol = PLAINTEXT
>       send.buffer.bytes = 131072
>       session.timeout.ms = 10000
>       ssl.cipher.suites = null
>       ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>       ssl.endpoint.identification.algorithm = null
>       ssl.key.password = null
>       ssl.keymanager.algorithm = SunX509
>       ssl.keystore.location = null
>       ssl.keystore.password = null
>       ssl.keystore.type = JKS
>       ssl.protocol = TLS
>       ssl.provider = null
>       ssl.secure.random.implementation = null
>       ssl.trustmanager.algorithm = PKIX
>       ssl.truststore.location = null
>       ssl.truststore.password = null
>       ssl.truststore.type = JKS
>       value.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
> INFO Kafka commitId : e89bffd6b2eff799 
> (org.apache.kafka.common.utils.AppInfoParser)
> [2017-06-07 12:24:45,497] INFO [mirrormaker-thread-0] Starting mirror maker 
> thread mirrormaker-thread-0 (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2017-06-07 12:24:45,497] INFO [mirrormaker-thread-1] Starting mirror maker 
> thread mirrormaker-thread-1 (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2017-06-07 12:24:48,619] INFO Discovered coordinator 
> app458.sjc2.mytest.com:9092 (id: 2147483613 rack: null) for group 
> MirrorMaker_hkg1. 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-06-07 12:24:48,620] INFO Discovered coordinator 
> app458.sjc2.mytest.com:9092 (id: 2147483613 rack: null) for group 
> MirrorMaker_hkg1. 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-06-07 12:24:48,625] INFO Revoking previously assigned partitions [] for 
> group MirrorMaker_hkg1 
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-06-07 12:24:48,625] INFO Revoking previously assigned partitions [] for 
> group MirrorMaker_hkg1 
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-06-07 12:24:48,648] INFO (Re-)joining group MirrorMaker_hkg1 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-06-07 12:24:48,649] INFO (Re-)joining group MirrorMaker_hkg1 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-06-07 12:24:53,560] FATAL [mirrormaker-thread-1] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.KafkaException: Unexpected error from SyncGroup: The 
> server experienced an unexpected error when processing the request
>       at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:548)
>       at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:521)
>       at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:784)
>       at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:765)
>       at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:186)
>       at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:149)
>       at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:116)
>       at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:493)
>       at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:322)
>       at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:253)
>       at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:172)
>       at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:347)
>       at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
>       at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
>       at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1029)
>       at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
>       at 
> kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:625)
>       at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:431)
> {noformat}
> Im using mirrormaker



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to