[jira] [Commented] (KAFKA-3802) log mtimes reset on broker restart
[
https://issues.apache.org/jira/browse/KAFKA-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319321#comment-15319321
]
Andrew Otto commented on KAFKA-3802:
I am running Kafka 0.9.01 on Debian Jessie, with 12 log.dirs on different
disks.
Here are my server.logs during the shutdown:
{noformat}
[2016-06-07 17:50:41,421] INFO [Kafka Server 20], shutting down
(kafka.server.KafkaServer)
[2016-06-07 17:50:41,428] INFO [Kafka Server 20], Starting controlled shutdown
(kafka.server.KafkaServer)
[2016-06-07 17:50:41,956] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_MobileWebBrowse,0]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,159] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_MobileWebBrowse,0]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,161] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [__consumer_offsets,14]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,161] INFO Truncating log __consumer_offsets-14 to offset
108. (kafka.log.Log)
[2016-06-07 17:50:42,599] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_MobileAppShareAttempts,0]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,600] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_MobileAppShareAttempts,0]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,601] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging-valid-mixed,8]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,603] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging-valid-mixed,8]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,603] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [webrequest_text,9] (kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,605] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [webrequest_text,9] (kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,606] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [__consumer_offsets,28]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,608] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [__consumer_offsets,28]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,608] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_GuidedTourButtonClick,0]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,611] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_GuidedTourButtonClick,0]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,613] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [webrequest_parsoid,1]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,619] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [webrequest_parsoid,1]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,621] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_CompletionSuggestions,0]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,624] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_CompletionSuggestions,0]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,625] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [webrequest_maps,5] (kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,625] INFO [ReplicaFetcherThread-10-12], Shutting down
(kafka.server.ReplicaFetcherThread)
[2016-06-07 17:50:42,659] INFO [ReplicaFetcherThread-10-12], Stopped
(kafka.server.ReplicaFetcherThread)
[2016-06-07 17:50:42,659] INFO [ReplicaFetcherThread-10-12], Shutdown completed
(kafka.server.ReplicaFetcherThread)
[2016-06-07 17:50:42,663] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [webrequest_maps,5] (kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,667] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging-valid-mixed,3]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,679] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging-valid-mixed,3]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,684] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [__consumer_offsets,36]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,690] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [__consumer_offsets,36]
(kafka.server.ReplicaFetcherManager)
[2016-06-07 17:50:42,691] INFO [ReplicaFetcherManager on broker 20] Removed
fetcher for partitions [eventlogging_EventError,0]
[jira] [Commented] (KAFKA-3802) log mtimes reset on broker restart
[ https://issues.apache.org/jira/browse/KAFKA-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319225#comment-15319225 ] Andrew Otto commented on KAFKA-3802: Possibly related to KAFKA-1379 > log mtimes reset on broker restart > -- > > Key: KAFKA-3802 > URL: https://issues.apache.org/jira/browse/KAFKA-3802 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.9.0.1 >Reporter: Andrew Otto > > Folks over in > http://mail-archives.apache.org/mod_mbox/kafka-users/201605.mbox/%3CCAO8=cz0ragjad1acx4geqcwj+rkd1gmdavkjwytwthkszfg...@mail.gmail.com%3E > are commenting about this issue. > In 0.9, any data log file that was on > disk before the broker has it's mtime modified to the time of the broker > restart. > This causes problems with log retention, as all the files then look like > they contain recent data to kafka. We use the default log retention of 7 > days, but if all the files are touched at the same time, this can cause us > to retain up to 2 weeks of log data, which can fill up our disks. > This happens *most* of the time, but seemingly not all. We have seen broker > restarts where mtimes were not changed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3802) log mtimes reset on broker restart
Andrew Otto created KAFKA-3802: -- Summary: log mtimes reset on broker restart Key: KAFKA-3802 URL: https://issues.apache.org/jira/browse/KAFKA-3802 Project: Kafka Issue Type: Bug Affects Versions: 0.9.0.1 Reporter: Andrew Otto Folks over in http://mail-archives.apache.org/mod_mbox/kafka-users/201605.mbox/%3CCAO8=cz0ragjad1acx4geqcwj+rkd1gmdavkjwytwthkszfg...@mail.gmail.com%3E are commenting about this issue. In 0.9, any data log file that was on disk before the broker has it's mtime modified to the time of the broker restart. This causes problems with log retention, as all the files then look like they contain recent data to kafka. We use the default log retention of 7 days, but if all the files are touched at the same time, this can cause us to retain up to 2 weeks of log data, which can fill up our disks. This happens *most* of the time, but seemingly not all. We have seen broker restarts where mtimes were not changed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2143) Replicas get ahead of leader and fail
[
https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056672#comment-15056672
]
Andrew Otto commented on KAFKA-2143:
I'd like to add that Wikimedia is experiencing this as well. It only started
happening to us recently when we started producing to Kafka via a PHP process
serving a web request. This process is short lived, so only connects to Kafka
in order to produce a single message, and then disconnects. The only topic
that this happens to is the one that this PHP process produces to.
Everything recovers as it should, but something is definitely wrong here. Why
is a follower trying to consume an offset ahead of what the leader has?
We're running 0.8.2.1 (with the snappy compression bugfix backported).
> Replicas get ahead of leader and fail
> -
>
> Key: KAFKA-2143
> URL: https://issues.apache.org/jira/browse/KAFKA-2143
> Project: Kafka
> Issue Type: Bug
> Components: replication
>Affects Versions: 0.8.2.1
>Reporter: Evan Huus
>Assignee: Jiangjie Qin
>
> On a cluster of 6 nodes, we recently saw a case where a single
> under-replicated partition suddenly appeared, replication lag spiked, and
> network IO spiked. The cluster appeared to recover eventually on its own,
> Looking at the logs, the thing which failed was partition 7 of the topic
> {{background_queue}}. It had an ISR of 1,4,3 and its leader at the time was
> 3. Here are the interesting log lines:
> On node 3 (the leader):
> {noformat}
> [2015-04-23 16:50:05,879] ERROR [Replica Manager on Broker 3]: Error when
> processing fetch request for partition [background_queue,7] offset 3722949957
> from follower with correlation id 148185816. Possible cause: Request for
> offset 3722949957 but we only have log segments in the range 3648049863 to
> 3722949955. (kafka.server.ReplicaManager)
> [2015-04-23 16:50:05,879] ERROR [Replica Manager on Broker 3]: Error when
> processing fetch request for partition [background_queue,7] offset 3722949957
> from follower with correlation id 156007054. Possible cause: Request for
> offset 3722949957 but we only have log segments in the range 3648049863 to
> 3722949955. (kafka.server.ReplicaManager)
> [2015-04-23 16:50:13,960] INFO Partition [background_queue,7] on broker 3:
> Shrinking ISR for partition [background_queue,7] from 1,4,3 to 3
> (kafka.cluster.Partition)
> {noformat}
> Note that both replicas suddenly asked for an offset *ahead* of the available
> offsets.
> And on nodes 1 and 4 (the replicas) many occurrences of the following:
> {noformat}
> [2015-04-23 16:50:05,935] INFO Scheduling log segment 3648049863 for log
> background_queue-7 for deletion. (kafka.log.Log) (edited)
> {noformat}
> Based on my reading, this looks like the replicas somehow got *ahead* of the
> leader, asked for an invalid offset, got confused, and re-replicated the
> entire topic from scratch to recover (this matches our network graphs, which
> show 3 sending a bunch of data to 1 and 4).
> Taking a stab in the dark at the cause, there appears to be a race condition
> where replicas can receive a new offset before the leader has committed it
> and is ready to replicate?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2189) Snappy compression of message batches less efficient in 0.8.2.1
[
https://issues.apache.org/jira/browse/KAFKA-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695589#comment-14695589
]
Andrew Otto commented on KAFKA-2189:
Hi all,
The Wikimedia Foundation had a serious production issue when we upgraded to
0.8.2.1 because of this bug. Snappy compression doesn't work at scale in
0.8.2.1. I know 0.8.3 is slated for release soon, maybe you should consider
doing a 0.8.2.2 release just to get this out there in a stable tag, so that
others don't run into this issue.
Snappy compression of message batches less efficient in 0.8.2.1
---
Key: KAFKA-2189
URL: https://issues.apache.org/jira/browse/KAFKA-2189
Project: Kafka
Issue Type: Bug
Components: build, compression, log
Affects Versions: 0.8.2.1
Reporter: Olson,Andrew
Assignee: Ismael Juma
Priority: Blocker
Labels: trivial
Fix For: 0.8.3
Attachments: KAFKA-2189.patch
We are using snappy compression and noticed a fairly substantial increase
(about 2.25x) in log filesystem space consumption after upgrading a Kafka
cluster from 0.8.1.1 to 0.8.2.1. We found that this is caused by messages
being seemingly recompressed individually (or possibly with a much smaller
buffer or dictionary?) instead of as a batch as sent by producers. We
eventually tracked down the change in compression ratio/scope to this [1]
commit that updated the snappy version from 1.0.5 to 1.1.1.3. The Kafka
client version does not appear to be relevant as we can reproduce this with
both the 0.8.1.1 and 0.8.2.1 Producer.
Here are the log files from our troubleshooting that contain the same set of
1000 messages, for batch sizes of 1, 10, 100, and 1000. f9d9b was the last
commit with 0.8.1.1-like behavior prior to f5ab8 introducing the issue.
{noformat}
-rw-rw-r-- 1 kafka kafka 404967 May 12 11:45
/var/kafka2/f9d9b-batch-1-0/.log
-rw-rw-r-- 1 kafka kafka 119951 May 12 11:45
/var/kafka2/f9d9b-batch-10-0/.log
-rw-rw-r-- 1 kafka kafka 89645 May 12 11:45
/var/kafka2/f9d9b-batch-100-0/.log
-rw-rw-r-- 1 kafka kafka 88279 May 12 11:45
/var/kafka2/f9d9b-batch-1000-0/.log
-rw-rw-r-- 1 kafka kafka 402837 May 12 11:41
/var/kafka2/f5ab8-batch-1-0/.log
-rw-rw-r-- 1 kafka kafka 382437 May 12 11:41
/var/kafka2/f5ab8-batch-10-0/.log
-rw-rw-r-- 1 kafka kafka 364791 May 12 11:41
/var/kafka2/f5ab8-batch-100-0/.log
-rw-rw-r-- 1 kafka kafka 380693 May 12 11:41
/var/kafka2/f5ab8-batch-1000-0/.log
{noformat}
[1]
https://github.com/apache/kafka/commit/f5ab8e1780cf80f267906e3259ad4f9278c32d28
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141002#comment-14141002 ] Andrew Otto commented on KAFKA-1367: This happens to me as well. See: https://github.com/edenhill/librdkafka/issues/147 Broker topic metadata not kept in sync with ZooKeeper - Key: KAFKA-1367 URL: https://issues.apache.org/jira/browse/KAFKA-1367 Project: Kafka Issue Type: Bug Affects Versions: 0.8.0, 0.8.1 Reporter: Ryan Berdeen Labels: newbie++ Attachments: KAFKA-1367.txt When a broker is restarted, the topic metadata responses from the brokers will be incorrect (different from ZooKeeper) until a preferred replica leader election. In the metadata, it looks like leaders are correctly removed from the ISR when a broker disappears, but followers are not. Then, when a broker reappears, the ISR is never updated. I used a variation of the Vagrant setup created by Joe Stein to reproduce this with latest from the 0.8.1 branch: https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141017#comment-14141017 ] Andrew Otto commented on KAFKA-1367: I just updated the librdkafka issue, pasting it here as well: I noticed that in my case, only 1 of the 4 brokers was ever missing in the ISRs reported by Kafka Brokers (via librdkafka). That JIRA indicated that a preferred-replica-election should fix the problem. I did this: controlled-shutdown of offending broker 21. Then actual shutdown of broker 21. Once this was done, librdkafka metadata showed the correct ISRs, since this offending broker really was not in any ISRs. I then restarted broker 21 and let its replicas catch back up. Once it caught up, zookeeper reported that all ISRs were in sync. I then checked librdkafka's metadata, and broker 21 was not listed in any ISR. I then ran a preferred-replica-election. broker 21 was then promoted to leader of some partitions. librdkafka then only showed broker 21 being in the ISRs for which it was also the leader. Any partition that has a replica on broker 21 does not show up in the ISR unless broker 21 is the leader. $ kafkacat -L -b analytics1022.eqiad.wmnet -t webrequest_upload Metadata for webrequest_upload (from broker -1: analytics1022.eqiad.wmnet:9092/bootstrap): 4 brokers: broker 12 at analytics1012.eqiad.wmnet:9092 broker 21 at analytics1021.eqiad.wmnet:9092 broker 22 at analytics1022.eqiad.wmnet:9092 broker 18 at analytics1018.eqiad.wmnet:9092 1 topics: topic webrequest_upload with 12 partitions: partition 11, leader 12, replicas: 12,21,22, isrs: 12,22 partition 5, leader 21, replicas: 21,22,12, isrs: 22,12,21 partition 10, leader 22, replicas: 22,18,21, isrs: 18,22 partition 7, leader 12, replicas: 12,18,21, isrs: 12,18 partition 8, leader 18, replicas: 18,22,12, isrs: 12,18,22 partition 3, leader 12, replicas: 12,22,18, isrs: 12,18,22 partition 4, leader 18, replicas: 18,21,22, isrs: 18,22 partition 1, leader 21, replicas: 21,18,22, isrs: 18,22,21 partition 6, leader 22, replicas: 22,12,18, isrs: 12,18,22 partition 2, leader 22, replicas: 22,21,12, isrs: 12,22 partition 9, leader 21, replicas: 21,12,18, isrs: 12,18,21 partition 0, leader 18, replicas: 18,12,21, isrs: 12,18 vs kafka-topic.sh --describe Topic:webrequest_upload PartitionCount:12 ReplicationFactor:3 Configs: Topic: webrequest_uploadPartition: 0Leader: 18 Replicas: 18,12,21 Isr: 12,18,21 Topic: webrequest_uploadPartition: 1Leader: 21 Replicas: 21,18,22 Isr: 18,22,21 Topic: webrequest_uploadPartition: 2Leader: 22 Replicas: 22,21,12 Isr: 12,22,21 Topic: webrequest_uploadPartition: 3Leader: 12 Replicas: 12,22,18 Isr: 12,18,22 Topic: webrequest_uploadPartition: 4Leader: 18 Replicas: 18,21,22 Isr: 18,22,21 Topic: webrequest_uploadPartition: 5Leader: 21 Replicas: 21,22,12 Isr: 22,12,21 Topic: webrequest_uploadPartition: 6Leader: 22 Replicas: 22,12,18 Isr: 12,18,22 Topic: webrequest_uploadPartition: 7Leader: 12 Replicas: 12,18,21 Isr: 12,18,21 Topic: webrequest_uploadPartition: 8Leader: 18 Replicas: 18,22,12 Isr: 12,18,22 Topic: webrequest_uploadPartition: 9Leader: 21 Replicas: 21,12,18 Isr: 12,18,21 Topic: webrequest_uploadPartition: 10 Leader: 22 Replicas: 22,18,21 Isr: 18,22,21 Topic: webrequest_uploadPartition: 11 Leader: 12 Replicas: 12,21,22 Isr: 12,22,21 Broker topic metadata not kept in sync with ZooKeeper - Key: KAFKA-1367 URL: https://issues.apache.org/jira/browse/KAFKA-1367 Project: Kafka Issue Type: Bug Affects Versions: 0.8.0, 0.8.1 Reporter: Ryan Berdeen Labels: newbie++ Attachments: KAFKA-1367.txt When a broker is restarted, the topic metadata responses from the brokers will be incorrect (different from ZooKeeper) until a preferred replica leader election. In the metadata, it looks like leaders are correctly removed from the ISR when a broker disappears, but followers are not. Then, when a broker reappears, the ISR is never updated. I used a variation of the Vagrant setup created by Joe Stein to reproduce this with latest from the 0.8.1 branch: https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141017#comment-14141017 ] Andrew Otto edited comment on KAFKA-1367 at 9/19/14 6:15 PM: - I just updated the librdkafka issue, pasting it here as well: I noticed that in my case, only 1 of the 4 brokers was ever missing in the ISRs reported by Kafka Brokers (via librdkafka). This JIRA indicated that a preferred-replica-election should fix the problem. I did this: controlled-shutdown of offending broker 21. Then actual shutdown of broker 21. Once this was done, librdkafka metadata showed the correct ISRs, since this offending broker really was not in any ISRs. I then restarted broker 21 and let its replicas catch back up. Once it caught up, zookeeper reported that all ISRs were in sync. I then checked librdkafka's metadata, and broker 21 was not listed in any ISR. I then ran a preferred-replica-election. broker 21 was then promoted to leader of some partitions. librdkafka then only showed broker 21 being in the ISRs for which it was also the leader. Any partition that has a replica on broker 21 does not show up in the ISR unless broker 21 is the leader. $ kafkacat -L -b analytics1022.eqiad.wmnet -t webrequest_upload Metadata for webrequest_upload (from broker -1: analytics1022.eqiad.wmnet:9092/bootstrap): 4 brokers: broker 12 at analytics1012.eqiad.wmnet:9092 broker 21 at analytics1021.eqiad.wmnet:9092 broker 22 at analytics1022.eqiad.wmnet:9092 broker 18 at analytics1018.eqiad.wmnet:9092 1 topics: topic webrequest_upload with 12 partitions: partition 11, leader 12, replicas: 12,21,22, isrs: 12,22 partition 5, leader 21, replicas: 21,22,12, isrs: 22,12,21 partition 10, leader 22, replicas: 22,18,21, isrs: 18,22 partition 7, leader 12, replicas: 12,18,21, isrs: 12,18 partition 8, leader 18, replicas: 18,22,12, isrs: 12,18,22 partition 3, leader 12, replicas: 12,22,18, isrs: 12,18,22 partition 4, leader 18, replicas: 18,21,22, isrs: 18,22 partition 1, leader 21, replicas: 21,18,22, isrs: 18,22,21 partition 6, leader 22, replicas: 22,12,18, isrs: 12,18,22 partition 2, leader 22, replicas: 22,21,12, isrs: 12,22 partition 9, leader 21, replicas: 21,12,18, isrs: 12,18,21 partition 0, leader 18, replicas: 18,12,21, isrs: 12,18 vs kafka-topic.sh --describe Topic:webrequest_upload PartitionCount:12 ReplicationFactor:3 Configs: Topic: webrequest_uploadPartition: 0Leader: 18 Replicas: 18,12,21 Isr: 12,18,21 Topic: webrequest_uploadPartition: 1Leader: 21 Replicas: 21,18,22 Isr: 18,22,21 Topic: webrequest_uploadPartition: 2Leader: 22 Replicas: 22,21,12 Isr: 12,22,21 Topic: webrequest_uploadPartition: 3Leader: 12 Replicas: 12,22,18 Isr: 12,18,22 Topic: webrequest_uploadPartition: 4Leader: 18 Replicas: 18,21,22 Isr: 18,22,21 Topic: webrequest_uploadPartition: 5Leader: 21 Replicas: 21,22,12 Isr: 22,12,21 Topic: webrequest_uploadPartition: 6Leader: 22 Replicas: 22,12,18 Isr: 12,18,22 Topic: webrequest_uploadPartition: 7Leader: 12 Replicas: 12,18,21 Isr: 12,18,21 Topic: webrequest_uploadPartition: 8Leader: 18 Replicas: 18,22,12 Isr: 12,18,22 Topic: webrequest_uploadPartition: 9Leader: 21 Replicas: 21,12,18 Isr: 12,18,21 Topic: webrequest_uploadPartition: 10 Leader: 22 Replicas: 22,18,21 Isr: 18,22,21 Topic: webrequest_uploadPartition: 11 Leader: 12 Replicas: 12,21,22 Isr: 12,22,21 was (Author: ottomata): I just updated the librdkafka issue, pasting it here as well: I noticed that in my case, only 1 of the 4 brokers was ever missing in the ISRs reported by Kafka Brokers (via librdkafka). That JIRA indicated that a preferred-replica-election should fix the problem. I did this: controlled-shutdown of offending broker 21. Then actual shutdown of broker 21. Once this was done, librdkafka metadata showed the correct ISRs, since this offending broker really was not in any ISRs. I then restarted broker 21 and let its replicas catch back up. Once it caught up, zookeeper reported that all ISRs were in sync. I then checked librdkafka's metadata, and broker 21 was not listed in any ISR. I then ran a preferred-replica-election. broker 21 was then promoted to leader of some partitions. librdkafka then only showed broker 21 being in the ISRs for which it was also the leader. Any partition that has a replica on broker 21 does not show up in the ISR unless broker 21 is the leader. $ kafkacat -L -b
[jira] [Created] (KAFKA-1564) Replace barrage of bin/*.sh with single cli wrapper
Andrew Otto created KAFKA-1564: -- Summary: Replace barrage of bin/*.sh with single cli wrapper Key: KAFKA-1564 URL: https://issues.apache.org/jira/browse/KAFKA-1564 Project: Kafka Issue Type: Improvement Components: packaging Affects Versions: 0.8.1.1 Reporter: Andrew Otto Priority: Trivial Kafka ships with a bunch of shell wrapper scripts in bin/. These are very small shell wrappers around Java CLIs. These work, but they are inconvenient to package, and not particularly elegant. If there is interest, I'd like to upstream this script: https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian/bin/kafka Wikimedia and Yelp are both using it in production, instead of the provided bin/*.sh scripts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1564) Replace barrage of bin/*.sh with single cli wrapper
[ https://issues.apache.org/jira/browse/KAFKA-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081253#comment-14081253 ] Andrew Otto commented on KAFKA-1564: Ha, actually, don't quote me on the Yelp is using this in production, all I know is that they are submitting patches to this script! :) Replace barrage of bin/*.sh with single cli wrapper --- Key: KAFKA-1564 URL: https://issues.apache.org/jira/browse/KAFKA-1564 Project: Kafka Issue Type: Improvement Components: packaging Affects Versions: 0.8.1.1 Reporter: Andrew Otto Priority: Trivial Original Estimate: 24h Remaining Estimate: 24h Kafka ships with a bunch of shell wrapper scripts in bin/. These are very small shell wrappers around Java CLIs. These work, but they are inconvenient to package, and not particularly elegant. If there is interest, I'd like to upstream this script: https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian/bin/kafka Wikimedia and Yelp are both using it in production, instead of the provided bin/*.sh scripts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1564) Replace barrage of bin/*.sh with single cli wrapper
[ https://issues.apache.org/jira/browse/KAFKA-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated KAFKA-1564: --- Description: Kafka ships with a bunch of shell wrapper scripts in bin/. These are very small shell wrappers around Java CLIs. These work, but they are inconvenient to package, and not particularly elegant. If there is interest, I'd like to upstream this script: https://github.com/wikimedia/operations-debs-kafka/blob/master/debian/bin/kafka Wikimedia and Yelp are both using it in production, instead of the provided bin/*.sh scripts. was: Kafka ships with a bunch of shell wrapper scripts in bin/. These are very small shell wrappers around Java CLIs. These work, but they are inconvenient to package, and not particularly elegant. If there is interest, I'd like to upstream this script: https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian/bin/kafka Wikimedia and Yelp are both using it in production, instead of the provided bin/*.sh scripts. Replace barrage of bin/*.sh with single cli wrapper --- Key: KAFKA-1564 URL: https://issues.apache.org/jira/browse/KAFKA-1564 Project: Kafka Issue Type: Improvement Components: packaging Affects Versions: 0.8.1.1 Reporter: Andrew Otto Priority: Trivial Original Estimate: 24h Remaining Estimate: 24h Kafka ships with a bunch of shell wrapper scripts in bin/. These are very small shell wrappers around Java CLIs. These work, but they are inconvenient to package, and not particularly elegant. If there is interest, I'd like to upstream this script: https://github.com/wikimedia/operations-debs-kafka/blob/master/debian/bin/kafka Wikimedia and Yelp are both using it in production, instead of the provided bin/*.sh scripts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1564) Replace barrage of bin/*.sh with single cli wrapper
[ https://issues.apache.org/jira/browse/KAFKA-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081259#comment-14081259 ] Andrew Otto commented on KAFKA-1564: The difficult part might be making the CLASSPATH generic. Currently it is hardcoded for *.jars provided by our .deb. Replace barrage of bin/*.sh with single cli wrapper --- Key: KAFKA-1564 URL: https://issues.apache.org/jira/browse/KAFKA-1564 Project: Kafka Issue Type: Improvement Components: packaging Affects Versions: 0.8.1.1 Reporter: Andrew Otto Priority: Trivial Original Estimate: 24h Remaining Estimate: 24h Kafka ships with a bunch of shell wrapper scripts in bin/. These are very small shell wrappers around Java CLIs. These work, but they are inconvenient to package, and not particularly elegant. If there is interest, I'd like to upstream this script: https://github.com/wikimedia/operations-debs-kafka/blob/master/debian/bin/kafka Wikimedia and Yelp are both using it in production, instead of the provided bin/*.sh scripts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1123) Broker IPv6 addresses parsed incorrectly
Andrew Otto created KAFKA-1123: -- Summary: Broker IPv6 addresses parsed incorrectly Key: KAFKA-1123 URL: https://issues.apache.org/jira/browse/KAFKA-1123 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 0.8 Reporter: Andrew Otto Assignee: Jun Rao Priority: Minor It seems that broker addresses are parsed incorrectly when IPv6 addresses are supplied. IPv6 addresses have colons in them, and Kafka seems to be interpreting the first : as the address:port separator. I have only tried this with the console-producer --broker-list option, so I don't know if this affects anything deeper than the CLI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1046) Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x
[ https://issues.apache.org/jira/browse/KAFKA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769853#comment-13769853 ] Andrew Otto commented on KAFKA-1046: Hm, I seem to be getting the same compilation error that is in the attached Screen Shot. How do I fix? Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x -- Key: KAFKA-1046 URL: https://issues.apache.org/jira/browse/KAFKA-1046 Project: Kafka Issue Type: Improvement Affects Versions: 0.8 Reporter: Christopher Freeman Assignee: Christopher Freeman Fix For: 0.8 Attachments: kafka_2_10_refactor_0.8.patch, kafka_2_10_refactor.patch, Screen Shot 2013-09-09 at 9.34.09 AM.png I refactored the project such that it will compile against Scala 2.10.1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
