Hi James,
This "Cached zkVersion [x] not equal to that in zookeeper" issue bit us
once in production and I found these ticket to be relevant:
KAFKA-2729 <https://issues.apache.org/jira/browse/KAFKA-2729>
KAFKA-3042 <https://issues.apache.org/jira/browse/KAFKA-3042>
KAFKA-3083 <https://issues.apache.org/jira/browse/KAFKA-3083>
Unfortunately, I don't believe there is a fix for it yet, or in the making.
Thanks,
Michał
On 28/04/17 19:26, James Brown wrote:
For what it's worth, shutting down the entire cluster and then restarting
it did address this issue.
I'd love anyone's thoughts on what the "correct" fix would be here.
On Fri, Apr 28, 2017 at 10:58 AM, James Brown <[email protected]> wrote:
The following is also appearing in the logs a lot, if anyone has any ideas:
INFO Partition [easypost.syslog,7] on broker 1: Cached zkVersion [647] not
equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
On Fri, Apr 28, 2017 at 10:43 AM, James Brown <[email protected]> wrote:
We're running 0.10.1.0 on a five-node cluster.
I was in the process of migrating some topics from having 2 replicas to
having three replicas when two the five machines in this cluster crashed
(brokers 2 and 3).
After restarting them, all of the topics that were previously assigned to
them are unavailable and showing "Leader: -1".
Example kafka-topics output:
% kafka-topics.sh --zookeeper $ZK_HP --describe --unavailable-partitions
Topic: __consumer_offsets Partition: 9 Leader: -1 Replicas: 3,2,4 Isr:
Topic: __consumer_offsets Partition: 13 Leader: -1 Replicas: 3,2,4 Isr:
Topic: __consumer_offsets Partition: 17 Leader: -1 Replicas: 3,2,5 Isr:
Topic: __consumer_offsets Partition: 23 Leader: -1 Replicas: 5,2,1 Isr:
Topic: __consumer_offsets Partition: 25 Leader: -1 Replicas: 3,2,5 Isr:
Topic: __consumer_offsets Partition: 26 Leader: -1 Replicas: 3,2,1 Isr:
Topic: __consumer_offsets Partition: 30 Leader: -1 Replicas: 3,1,2 Isr:
Topic: __consumer_offsets Partition: 33 Leader: -1 Replicas: 1,2,4 Isr:
Topic: __consumer_offsets Partition: 35 Leader: -1 Replicas: 1,2,5 Isr:
Topic: __consumer_offsets Partition: 39 Leader: -1 Replicas: 3,1,2 Isr:
Topic: __consumer_offsets Partition: 40 Leader: -1 Replicas: 3,4,2 Isr:
Topic: __consumer_offsets Partition: 44 Leader: -1 Replicas: 3,1,2 Isr:
Topic: __consumer_offsets Partition: 45 Leader: -1 Replicas: 1,3,2 Isr:
Note that I wasn't even moving any of the __consumer_offsets partitions,
so I'm not sure if the fact that a reassignment was in progress is a red
herring or not.
The logs are full of
ERROR [ReplicaFetcherThread-0-3], Error for partition [tracking.syslog,2]
to broker 3:org.apache.kafka.common.errors.UnknownServerException: The
server experienced an unexpected error when processing the request
(kafka.server.ReplicaFetcherThread)
ERROR [ReplicaFetcherThread-0-3], Error for partition [tracking.syslog,2]
to broker 3:org.apache.kafka.common.errors.UnknownServerException: The
server experienced an unexpected error when processing the request
(kafka.server.ReplicaFetcherThread)
ERROR [ReplicaFetcherThread-0-3], Error for partition
[epostg.request_log_v1,0] to broker
3:org.apache.kafka.common.errors.UnknownServerException:
The server experienced an unexpected error when processing the request
(kafka.server.ReplicaFetcherThread)
ERROR [ReplicaFetcherThread-0-3], Error for partition
[epostg.request_log_v1,0] to broker
3:org.apache.kafka.common.errors.UnknownServerException:
The server experienced an unexpected error when processing the request
(kafka.server.ReplicaFetcherThread)
What can I do to fix this? Should I manually reassign all partitions
that were led by brokers 2 or 3 to only have whatever the third broker was
in their replica-set as their replica set? Do I need to temporarily enable
unclean elections?
I've never seen a cluster fail this way...
--
James Brown
Engineer
--
James Brown
Engineer
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: [email protected]
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
[email protected] <mailto:[email protected]> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612
