[jira] [Comment Edited] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577509#comment-14577509
]
Aditya Auradkar edited comment on KAFKA-1367 at 6/8/15 5:40 PM:
[~jjkoshy] [~junrao] KAFKA-2225, even if we leave the ISR in the
TopicMetadataRequest, how do the consumers detect which of the replicas in ISR
to fetch from right? The consumers need to know which zone each of the
brokers live in and their own in order to fetch from the closest replica (which
mitigates with the bandwidth issues described in 2225).
Couple of options:
1. Return it in BrokerMetadataRequest (KIP-24)
2. Piggyback it along with the ISR field in TMR. i.e. isr : {0: zone1, 1:
zone2}
If we choose to do (2), then the TMR will evolve anyway.
was (Author: aauradkar):
[~jjkoshy] [~junrao] KAFKA-2225, even if we leave the ISR in the
TopicMetadataRequest, how do the consumers detect which of the replicas in ISR
to fetch from right? The consumers need to know which zone each of the
brokers live in and their own in order to fetch from the closest replica (which
mitigates with the bandwidth issues described in 2225).
Broker topic metadata not kept in sync with ZooKeeper
-
Key: KAFKA-1367
URL: https://issues.apache.org/jira/browse/KAFKA-1367
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.0, 0.8.1
Reporter: Ryan Berdeen
Assignee: Ashish K Singh
Labels: newbie++
Fix For: 0.8.3
Attachments: KAFKA-1367.txt
When a broker is restarted, the topic metadata responses from the brokers
will be incorrect (different from ZooKeeper) until a preferred replica leader
election.
In the metadata, it looks like leaders are correctly removed from the ISR
when a broker disappears, but followers are not. Then, when a broker
reappears, the ISR is never updated.
I used a variation of the Vagrant setup created by Joe Stein to reproduce
this with latest from the 0.8.1 branch:
https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577509#comment-14577509 ] Aditya Auradkar commented on KAFKA-1367: [~jjkoshy] [~junrao] KAFKA-2225, even if we leave the ISR in the TopicMetadataRequest, how do the consumers detect which of the replicas in ISR to fetch from right? The consumers need to know which zone each of the brokers live in and their own in order to fetch from the closest replica (which mitigates with the bandwidth issues described in 2225). Broker topic metadata not kept in sync with ZooKeeper - Key: KAFKA-1367 URL: https://issues.apache.org/jira/browse/KAFKA-1367 Project: Kafka Issue Type: Bug Affects Versions: 0.8.0, 0.8.1 Reporter: Ryan Berdeen Assignee: Ashish K Singh Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1367.txt When a broker is restarted, the topic metadata responses from the brokers will be incorrect (different from ZooKeeper) until a preferred replica leader election. In the metadata, it looks like leaders are correctly removed from the ISR when a broker disappears, but followers are not. Then, when a broker reappears, the ISR is never updated. I used a variation of the Vagrant setup created by Joe Stein to reproduce this with latest from the 0.8.1 branch: https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2225) Allow fetching from ISR
[ https://issues.apache.org/jira/browse/KAFKA-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577571#comment-14577571 ] Jun Rao commented on KAFKA-2225: A few notes on this. 1. We probably should only do this on the new java consumer. 2. We will need some kind of notion of zones in both the brokers and the clients. 3. We probably need to fix ISR in TMR first. There is some discussion about this in KAFKA-1367. Allow fetching from ISR --- Key: KAFKA-2225 URL: https://issues.apache.org/jira/browse/KAFKA-2225 Project: Kafka Issue Type: Improvement Reporter: Theo Hultberg Assignee: Parth Brahmbhatt Currently clients are not allowed to fetch from replicas, even if they are in sync with the master. If I'm not missing anything significant it shouldn't be any difference fetching from the leader or an ISR, besides maybe some extra latency. For our use case it would be very beneficial to be able to fetch from replicas instead of just the leader. We run Kafka clusters that replicate across EC2 availability zones, and each byte sent between zones costs money. This bandwith usage costs us about the same as it costs to run the instances. If we could fetch from a replica in the same zone as the client we could avoid some of this cost. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2253) Deadlock in delayed operation purgatory
[
https://issues.apache.org/jira/browse/KAFKA-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577639#comment-14577639
]
Guozhang Wang commented on KAFKA-2253:
--
Updated reviewboard https://reviews.apache.org/r/35201/diff/
against branch origin/trunk
Deadlock in delayed operation purgatory
---
Key: KAFKA-2253
URL: https://issues.apache.org/jira/browse/KAFKA-2253
Project: Kafka
Issue Type: Bug
Reporter: Mayuresh Gharat
Assignee: Guozhang Wang
Fix For: 0.8.3
Attachments: KAFKA-2253.patch, KAFKA-2253.patch,
KAFKA-2253_2015-06-08_11:47:40.patch
We hit a deadlock while running brokers with git hash:
9e894aa0173b14d64a900bcf780d6b7809368384
There's a circular wait between the removeWatchersLock and an operations
intrinsic lock.
{code}
Found one Java-level deadlock:
=
kafka-request-handler-a:
waiting for ownable synchronizer 0x0006da08f9e0, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by ExpirationReaper-xyz
ExpirationReaper-xyz:
waiting to lock monitor 0x7f454e18 (object 0x0006b0563fe8, a
java.util.LinkedList),
which is held by kafka-request-handler-b
kafka-request-handler-b:
waiting for ownable synchronizer 0x0006da08f9e0, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by ExpirationReaper-xyz
kafka-request-handler-a:
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0006da08f9e0 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:296)
at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:304)
at
kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:224)
at
kafka.server.ReplicaManager.tryCompleteDelayedFetch(ReplicaManager.scala:166)
at
kafka.cluster.Partition.kafka$cluster$Partition$$maybeIncrementLeaderHW(Partition.scala:358)
at
kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply$mcV$sp(Partition.scala:288)
at
kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply(Partition.scala:270)
at
kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply(Partition.scala:270)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:306)
at kafka.cluster.Partition.maybeExpandIsr(Partition.scala:268)
at
kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:244)
at
kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:790)
at
kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:787)
at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
at
kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:787)
at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:432)
at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:312)
at kafka.server.KafkaApis.handle(KafkaApis.scala:60)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
ExpirationReaper-xyz:
at
kafka.server.DelayedOperationPurgatory$Watchers.watched(DelayedOperation.scala:278)
- waiting to lock 0x0006b0563fe8 (a java.util.LinkedList)
at
kafka.server.DelayedOperationPurgatory$$anonfun$kafka$server$DelayedOperationPurgatory$$removeKeyIfEmpty$1.apply(DelayedOperation.scala:258)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:306)
at
kafka.server.DelayedOperationPurgatory.kafka$server$DelayedOperationPurgatory$$removeKeyIfEmpty(DelayedOperation.scala:256)
at
kafka.server.DelayedOperationPurgatory$Watchers.purgeCompleted(DelayedOperation.scala:322)
- locked 0x00071a86a478 (a java.util.LinkedList)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper$$anonfun$3.apply(DelayedOperation.scala:347)
[jira] [Updated] (KAFKA-2253) Deadlock in delayed operation purgatory
[
https://issues.apache.org/jira/browse/KAFKA-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-2253:
-
Attachment: KAFKA-2253_2015-06-08_11:47:40.patch
Deadlock in delayed operation purgatory
---
Key: KAFKA-2253
URL: https://issues.apache.org/jira/browse/KAFKA-2253
Project: Kafka
Issue Type: Bug
Reporter: Mayuresh Gharat
Assignee: Guozhang Wang
Fix For: 0.8.3
Attachments: KAFKA-2253.patch, KAFKA-2253.patch,
KAFKA-2253_2015-06-08_11:47:40.patch
We hit a deadlock while running brokers with git hash:
9e894aa0173b14d64a900bcf780d6b7809368384
There's a circular wait between the removeWatchersLock and an operations
intrinsic lock.
{code}
Found one Java-level deadlock:
=
kafka-request-handler-a:
waiting for ownable synchronizer 0x0006da08f9e0, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by ExpirationReaper-xyz
ExpirationReaper-xyz:
waiting to lock monitor 0x7f454e18 (object 0x0006b0563fe8, a
java.util.LinkedList),
which is held by kafka-request-handler-b
kafka-request-handler-b:
waiting for ownable synchronizer 0x0006da08f9e0, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by ExpirationReaper-xyz
kafka-request-handler-a:
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0006da08f9e0 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:296)
at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:304)
at
kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:224)
at
kafka.server.ReplicaManager.tryCompleteDelayedFetch(ReplicaManager.scala:166)
at
kafka.cluster.Partition.kafka$cluster$Partition$$maybeIncrementLeaderHW(Partition.scala:358)
at
kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply$mcV$sp(Partition.scala:288)
at
kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply(Partition.scala:270)
at
kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply(Partition.scala:270)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:306)
at kafka.cluster.Partition.maybeExpandIsr(Partition.scala:268)
at
kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:244)
at
kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:790)
at
kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:787)
at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
at
kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:787)
at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:432)
at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:312)
at kafka.server.KafkaApis.handle(KafkaApis.scala:60)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
ExpirationReaper-xyz:
at
kafka.server.DelayedOperationPurgatory$Watchers.watched(DelayedOperation.scala:278)
- waiting to lock 0x0006b0563fe8 (a java.util.LinkedList)
at
kafka.server.DelayedOperationPurgatory$$anonfun$kafka$server$DelayedOperationPurgatory$$removeKeyIfEmpty$1.apply(DelayedOperation.scala:258)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:306)
at
kafka.server.DelayedOperationPurgatory.kafka$server$DelayedOperationPurgatory$$removeKeyIfEmpty(DelayedOperation.scala:256)
at
kafka.server.DelayedOperationPurgatory$Watchers.purgeCompleted(DelayedOperation.scala:322)
- locked 0x00071a86a478 (a java.util.LinkedList)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper$$anonfun$3.apply(DelayedOperation.scala:347)
at
[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577728#comment-14577728 ] Gwen Shapira commented on KAFKA-1367: - By zones do we mean rack-awareness? Or more general locality notion? Sounds like something that may need its own JIRA and design. Broker topic metadata not kept in sync with ZooKeeper - Key: KAFKA-1367 URL: https://issues.apache.org/jira/browse/KAFKA-1367 Project: Kafka Issue Type: Bug Affects Versions: 0.8.0, 0.8.1 Reporter: Ryan Berdeen Assignee: Ashish K Singh Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1367.txt When a broker is restarted, the topic metadata responses from the brokers will be incorrect (different from ZooKeeper) until a preferred replica leader election. In the metadata, it looks like leaders are correctly removed from the ISR when a broker disappears, but followers are not. Then, when a broker reappears, the ISR is never updated. I used a variation of the Vagrant setup created by Joe Stein to reproduce this with latest from the 0.8.1 branch: https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577563#comment-14577563 ] Jun Rao commented on KAFKA-1367: Yes, we need some kind of notion of zones for both the brokers and the clients. Each broker and each client (producer/consumer) need a configuration for which zone it belongs to. It's probably simpler to just return the zone info in TMR. We will need to evolve TMR, but that can probably be done separately from fixing the ISR in TMR. We probably should move these design discussions to KAFKA-2225 itself. Broker topic metadata not kept in sync with ZooKeeper - Key: KAFKA-1367 URL: https://issues.apache.org/jira/browse/KAFKA-1367 Project: Kafka Issue Type: Bug Affects Versions: 0.8.0, 0.8.1 Reporter: Ryan Berdeen Assignee: Ashish K Singh Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1367.txt When a broker is restarted, the topic metadata responses from the brokers will be incorrect (different from ZooKeeper) until a preferred replica leader election. In the metadata, it looks like leaders are correctly removed from the ISR when a broker disappears, but followers are not. Then, when a broker reappears, the ISR is never updated. I used a variation of the Vagrant setup created by Joe Stein to reproduce this with latest from the 0.8.1 branch: https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35201: Fix KAFKA-2253
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35201/#review87065 --- Ship it! Ship It! - Jiangjie Qin On June 8, 2015, 6:47 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35201/ --- (Updated June 8, 2015, 6:47 p.m.) Review request for kafka. Bugs: KAFKA-2253 https://issues.apache.org/jira/browse/KAFKA-2253 Repository: kafka Description --- Incorporated Jiangjie and Onur's comments Diffs - core/src/main/scala/kafka/server/DelayedOperation.scala 123078d97a7bfe2121655c00f3b2c6af21c53015 Diff: https://reviews.apache.org/r/35201/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 35201: Fix KAFKA-2253
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35201/ --- (Updated June 8, 2015, 6:47 p.m.) Review request for kafka. Bugs: KAFKA-2253 https://issues.apache.org/jira/browse/KAFKA-2253 Repository: kafka Description --- Incorporated Jiangjie and Onur's comments Diffs (updated) - core/src/main/scala/kafka/server/DelayedOperation.scala 123078d97a7bfe2121655c00f3b2c6af21c53015 Diff: https://reviews.apache.org/r/35201/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Created] (KAFKA-2259) port offset_management_testsuite
Geoffrey Anderson created KAFKA-2259: Summary: port offset_management_testsuite Key: KAFKA-2259 URL: https://issues.apache.org/jira/browse/KAFKA-2259 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Port to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2232) make MockProducer generic
[ https://issues.apache.org/jira/browse/KAFKA-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577806#comment-14577806 ] Alexander Pakulov commented on KAFKA-2232: -- Pull request has been submitted: https://github.com/apache/kafka/pull/68 I've moved MockProducer and MockConsumer to test source set, since they only have been used for tests. make MockProducer generic - Key: KAFKA-2232 URL: https://issues.apache.org/jira/browse/KAFKA-2232 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2.1 Reporter: Jun Rao Labels: newbie Fix For: 0.8.3 Currently, MockProducer implements Producerbyte[], byte[]. Instead, we should implement MockProducerK, V. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2258) Port mirrormaker_testsuite
Geoffrey Anderson created KAFKA-2258: Summary: Port mirrormaker_testsuite Key: KAFKA-2258 URL: https://issues.apache.org/jira/browse/KAFKA-2258 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Port mirrormaker_testsuite to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: Make MockProducer generic
GitHub user apakulov opened a pull request: https://github.com/apache/kafka/pull/68 Make MockProducer generic MockConsumer and MockProducer have been moved to test source set. KeySerializer and ValueSerializer have been added to mimic actual KafkaProducer behavior. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apakulov/kafka KAFKA-2232 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/68.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #68 commit 2e06015752b1b6ccae263c03778668959271e152 Author: Alexander Pakulov a.paku...@gmail.com Date: 2015-06-08T20:40:26Z Make MockProducer generic --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578020#comment-14578020 ] Jun Rao commented on KAFKA-1690: 2. I am not sure what the best approach is. Currently, Send.writeTo works with GatheringByteChannel, which is simpler than SocketChannel. So, it's probably easier to keep it this way. new java producer needs ssl support as a client --- Key: KAFKA-1690 URL: https://issues.apache.org/jira/browse/KAFKA-1690 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Sriharsha Chintalapani Fix For: 0.8.3 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577881#comment-14577881 ] Jun Rao commented on KAFKA-1690: [~sriharsha], thanks for the latest patch. Haven't looked at it in details. A couple of comments. 1. Could you rebase now that KAFKA-1928 is done? 2. We need to think a bit more about supporting the zero-copy transfer on the consumer. Currently, the broker sends data to the consumer client using the socket.writeTo() api (in PartitionDataSend, TopicDataSend, and FetchResponseSend). We want to preserve this optimization for plain socket channels. Only for SSL channel, we will copy the bytes from the file channel into the application and then send it to the remote socket. One way is probably to have Channel implement GatheringByteChannel and ScatteringByteChannel. new java producer needs ssl support as a client --- Key: KAFKA-1690 URL: https://issues.apache.org/jira/browse/KAFKA-1690 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Sriharsha Chintalapani Fix For: 0.8.3 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2257) port replication_testsuite
Geoffrey Anderson created KAFKA-2257: Summary: port replication_testsuite Key: KAFKA-2257 URL: https://issues.apache.org/jira/browse/KAFKA-2257 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Port subset of replication_testsuite to run on ducktape. Details to follow -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34492: Patch for KAFKA-2210
On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
core/src/main/scala/kafka/security/auth/Acl.scala, lines 57-62
https://reviews.apache.org/r/34492/diff/1/?file=965651#file965651line57
I tried exactly that but it tunrs out our current Json parser does not
work when a json string has other special characters, somehow gets into
some double parsing and fails. Has been long since I wrote this code so
dont exactly remember why it was failing but I did try it and with current
JsonUtil it does not work.
Could you explain a bit which part doesn't work? The following simple test
works for me.
scala val a = [{\a\: \aa\}]
a: String = [{a: aa}]
scala JSON.parseFull(a)
res4: Option[Any] = Some(List(Map(a - aa)))
On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
core/src/main/scala/kafka/security/auth/KafkaPrincipal.scala, line 22
https://reviews.apache.org/r/34492/diff/1/?file=965653#file965653line22
I haven't added Group support yet but they will be of the form
Group:group-name. Why did you get the impression that groups will not
have :
Oh, I was just saying that if the group name itself can contain :, parsing
will be more difficult if : is the separator.
On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
core/src/main/scala/kafka/security/auth/KafkaPrincipal.scala, line 41
https://reviews.apache.org/r/34492/diff/1/?file=965653#file965653line41
Yes we can and as mentioned in the design doc when no authentication is
configured it will be set as User:DrWho?.
So, I guess authentication will always authenticate at the user level and it's
up to the Authorization model to implement the user to group mapping?
On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
core/src/main/scala/kafka/security/auth/Operation.java, line 22
https://reviews.apache.org/r/34492/diff/1/?file=965654#file965654line22
I grepped through kafka code base to understand how enums were used in
other parts and all places used java enums. I assumed that was the
convention . If that is not the case I can change all enum classes in core
to use http://www.scala-lang.org/api/current/index.html#scala.Enumeration
Under core/, we don't have java files except when defining the java api. We
implement enum using case object in scala (see BrokerStates as an example).
On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
core/src/test/scala/unit/kafka/security/auth/AclTest.scala, line 36
https://reviews.apache.org/r/34492/diff/1/?file=965662#file965662line36
can you elloborate why do you think that is a better approach?
I was thinking of just embedding the acl json string in the code.
On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
core/src/main/scala/kafka/security/auth/Authorizer.scala, line 36
https://reviews.apache.org/r/34492/diff/1/?file=965652#file965652line36
In the KIP dicussion it was proposed to add a config
authoizer.config.path which will contain path to a property files on all
broker hosts. This is how the plugin specific property file gets passed on.
Do we want to instead use configurable?
Sorry, but I missed this in the KIP review. I think it's probably better NOT to
have another config.path inside a configuration file. We already have other
pluggable logic such as the MetrisReporter and will be adding other pluggable
logic such as PrincipalExtractor in KAFKA-1690. Introducing a separate config
path for each pluggable logic may not be ideal. Also, currently, we allow
people to instantiate KafkaServerStartble directly so that people can obtain
the properties from any configuration system and pass them to Kafka, instead of
assuming that the properties are always specified in a file. So, it's probably
better to specify the properties needed by any pluggable logic in the same
property file, then pass them to the pluggable logic through the configure()
api. We have KAFKA-2249 filed to allow KafkaConfig to do this. Perhaps, we can
fix KAFKA-2249 first.
- Jun
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34492/#review86267
---
On June 4, 2015, 11:07 p.m., Parth Brahmbhatt wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34492/
---
(Updated June 4, 2015, 11:07 p.m.)
Review request for kafka.
Bugs: KAFKA-2210
https://issues.apache.org/jira/browse/KAFKA-2210
Repository: kafka
Description
---
Addressing review comments from Jun.
Adding CREATE check for offset topic only if the topic does not exist already.
Diffs
-
core/src/main/scala/kafka/api/OffsetRequest.scala
3d483bc7518ad76f9548772522751afb4d046b78
[jira] [Created] (KAFKA-2256) Port system tests
Geoffrey Anderson created KAFKA-2256: Summary: Port system tests Key: KAFKA-2256 URL: https://issues.apache.org/jira/browse/KAFKA-2256 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 This is a tracking issue for the system test suites to be ported per KIP-25 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35201: Fix KAFKA-2253
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35201/#review87077 --- Ship it! Ship It! - Onur Karaman On June 8, 2015, 6:47 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35201/ --- (Updated June 8, 2015, 6:47 p.m.) Review request for kafka. Bugs: KAFKA-2253 https://issues.apache.org/jira/browse/KAFKA-2253 Repository: kafka Description --- Incorporated Jiangjie and Onur's comments Diffs - core/src/main/scala/kafka/server/DelayedOperation.scala 123078d97a7bfe2121655c00f3b2c6af21c53015 Diff: https://reviews.apache.org/r/35201/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577758#comment-14577758 ] Jiangjie Qin commented on KAFKA-1367: - I agree with [~gwenshap], it sounds this deserves a KIP. Broker topic metadata not kept in sync with ZooKeeper - Key: KAFKA-1367 URL: https://issues.apache.org/jira/browse/KAFKA-1367 Project: Kafka Issue Type: Bug Affects Versions: 0.8.0, 0.8.1 Reporter: Ryan Berdeen Assignee: Ashish K Singh Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1367.txt When a broker is restarted, the topic metadata responses from the brokers will be incorrect (different from ZooKeeper) until a preferred replica leader election. In the metadata, it looks like leaders are correctly removed from the ISR when a broker disappears, but followers are not. Then, when a broker reappears, the ISR is never updated. I used a variation of the Vagrant setup created by Joe Stein to reproduce this with latest from the 0.8.1 branch: https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577912#comment-14577912 ] Sriharsha Chintalapani commented on KAFKA-1690: --- [~junrao] currently working on rebasing on KAFKA-1928. 2. My earlier version have channel implementing GatheringByteChannel and ScatteringByteChannel . I thought from our previous conversation you wanted Channel to own NetworkReceive and Send. I am currently keeping Send and Receive to work with socketChannel . I'll try to upload the new patch with KAFKA-1928 soon. new java producer needs ssl support as a client --- Key: KAFKA-1690 URL: https://issues.apache.org/jira/browse/KAFKA-1690 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Sriharsha Chintalapani Fix For: 0.8.3 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system
Honghai,
You are going to do a vote on this, right? I think the patch is ready to go
so we are just waiting on the KIP adoption I think.
-Jay
On Thu, May 21, 2015 at 8:50 AM, Jun Rao j...@confluent.io wrote:
Honghai,
Could you update the wiki on the preallocated size? Instead of
config.segmentSize
- 2 * config.maxMessageSize, we just want to use config.segmentSize.
Thanks,
Jun
On Mon, May 11, 2015 at 8:16 PM, Honghai Chen honghai.c...@microsoft.com
wrote:
All issues fixed, test cases added, performance result on windows
attached. The patch can help improve the consume performance around
25%~50%.
Thanks, Honghai Chen
-Original Message-
From: Jun Rao [mailto:j...@confluent.io]
Sent: Wednesday, May 6, 2015 5:39 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
Thanks. Could you updated the wiki? Also, commented on the jira.
Jun
On Tue, May 5, 2015 at 12:48 AM, Honghai Chen
honghai.c...@microsoft.com
wrote:
Use config.segmentSize should be ok. Previously add that one for make
sure the file not exceed config.segmentSize, actually the function
maybeRoll already make sure that.
When try add test case for recover, blocked by the rename related
issue, just open one jira at
https://issues.apache.org/jira/browse/KAFKA-2170 , any recommendation
for fix that issue?
Thanks, Honghai Chen
-Original Message-
From: Jun Rao [mailto:j...@confluent.io]
Sent: Tuesday, May 5, 2015 12:51 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve
consume performance under windows and some old Linux file system
This seems similar to what's in
https://issues.apache.org/jira/browse/KAFKA-1065.
Also, could you explain why the preallocated size is set to
config.segmentSize
- 2 * config.maxMessageSize, instead of just config.segmentSize?
Thanks,
Jun
On Mon, May 4, 2015 at 8:12 PM, Honghai Chen
honghai.c...@microsoft.com
wrote:
Hi guys,
I'm trying add test cases, but below case crashed at line
segReopen.recover(64*1024)-- index.trimToValidSize() , any idea
for
it?
Appreciate your help.
The case assume kafka suddenly crash, and need recover the
last segment.
kafka.log.LogSegmentTest testCreateWithInitFileSizeCrash FAILED
java.io.IOException: The requested operation cannot be performed
on a file w ith a user-mapped section open
at java.io.RandomAccessFile.setLength(Native Method)
at
kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:292)
at
kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:288)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:283)
at
kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(Offset
I
ndex.scala:272)
at
kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.s
c
ala:272)
at
kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.s
c
ala:272)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:288)
at
kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:271)
at kafka.log.LogSegment.recover(LogSegment.scala:199)
at
kafka.log.LogSegmentTest.testCreateWithInitFileSizeCrash(LogSegmentT
e
st.scala:306)
def recover(maxMessageSize: Int): Int = {
index.truncate()
index.resize(index.maxIndexSize)
var validBytes = 0
var lastIndexEntry = 0
val iter = log.iterator(maxMessageSize)
try {
while(iter.hasNext) {
val entry = iter.next
entry.message.ensureValid()
if(validBytes - lastIndexEntry indexIntervalBytes) {
// we need to decompress the message, if required, to get
the offset of the first uncompressed message
val startOffset =
entry.message.compressionCodec match {
case NoCompressionCodec =
entry.offset
case _ =
ByteBufferMessageSet.deepIterator(entry.message).next().offset
}
index.append(startOffset, validBytes)
lastIndexEntry = validBytes
}
validBytes += MessageSet.entrySize(entry.message)
}
} catch {
case e: InvalidMessageException =
logger.warn(Found invalid messages in log segment %s at
byte offset %d: %s..format(log.file.getAbsolutePath, validBytes,
e.getMessage))
}
val truncated = log.sizeInBytes - validBytes
log.truncateTo(validBytes)
index.trimToValidSize()
truncated
[jira] [Commented] (KAFKA-2208) Consumer should detect coordinator failures
[ https://issues.apache.org/jira/browse/KAFKA-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578030#comment-14578030 ] Guozhang Wang commented on KAFKA-2208: -- Created reviewboard https://reviews.apache.org/r/35231/diff/ against branch origin/trunk Consumer should detect coordinator failures --- Key: KAFKA-2208 URL: https://issues.apache.org/jira/browse/KAFKA-2208 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Onur Karaman Assignee: Guozhang Wang Attachments: KAFKA-2208.patch, KAFKA-2208.patch, KAFKA-2208_2015-05-20_19:14:58.patch, KAFKA-2208_2015-05-31_17:05:06.patch, KAFKA-2208_2015-06-02_10:45:34.patch, KAFKA-2208_2015-06-02_14:06:58.patch Add coordinator discovery and failure detection to the consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35231: Fix KAFKA-1740
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35231/ --- (Updated June 8, 2015, 11:12 p.m.) Review request for kafka. Bugs: KAFKA-1740 https://issues.apache.org/jira/browse/KAFKA-1740 Repository: kafka Description --- Move offset manager to coordinator, add validation logic for offset commit and fetch Diffs - clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java c1496a0851526f3c7d3905ce4bdff2129c83a6c1 clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 5b898c8f8ad5d0469f469b600c4b2eb13d1fc662 clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java 70844d65369f6ff300cbeb513dbb6650050c7eec clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java deec1fa480d5a5c5884a1c007b076aa64e902472 clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java 512a0ef7e619d54e74122c38119209f5cf9590e3 clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorTest.java b06c4a73e2b4e9472cd772c8bc32bf4a29f431bb core/src/main/scala/kafka/admin/TopicCommand.scala dacbdd089f96c67b7bbc1c0cd36490d4fe094c1b core/src/main/scala/kafka/cluster/Partition.scala 730a232482fdf77be5704cdf5941cfab3828db88 core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 6b4242c7cd1df9b3465db0fec35a25102c76cd60 core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 51e89c87ee2c20fc7f976536f01fa1055fb8e670 core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala c39e6de34ee531c6dfa9107b830752bd7f8fbe59 core/src/main/scala/kafka/server/KafkaApis.scala d63bc18d795a6f0e6994538ca55a9a46f7fb8ffd core/src/main/scala/kafka/server/KafkaServer.scala b320ce9f6a12c0ee392e91beb82e8804d167f9f4 core/src/main/scala/kafka/server/OffsetManager.scala 5cca85cf727975f6d3acb2223fd186753ad761dc core/src/main/scala/kafka/server/ReplicaManager.scala 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 core/src/test/scala/integration/kafka/api/ConsumerTest.scala 17b17b9b1520c7cc2e2ba96cdb1f9ff06e47bcad core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 07b1ff47bfc3cd3f948c9533c8dc977fa36d996f core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala c7136f20972614ac47aa57ab13e3c94ef775a4b7 core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 4f124af5c3e946045a78ad1519c37372a72c8985 core/src/test/scala/unit/kafka/coordinator/CoordinatorMetadataTest.scala 08854c5e6ec249368206298b2ac2623df18f266a core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 528525b719ec916e16f8b3ae3715bec4b5dcc47d Diff: https://reviews.apache.org/r/35231/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.
[ https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated KAFKA-2254: --- Status: Patch Available (was: Open) I create a patch kafka-shell-script.patch of the trunk . The shell script should be optimized , even kafka-run-class.sh has a syntax error. -- Key: KAFKA-2254 URL: https://issues.apache.org/jira/browse/KAFKA-2254 Project: Kafka Issue Type: Bug Components: build Affects Versions: 0.8.2.1 Environment: linux Reporter: Bo Wang Priority: Critical Labels: client-script, kafka-run-class.sh, shell-script Fix For: 0.8.2.1 Attachments: kafka-shell-script.patch Original Estimate: 24h Remaining Estimate: 24h kafka-run-class.sh 128 line has a syntax error(missing a space): 127-loggc) 128 if [ -z $KAFKA_GC_LOG_OPTS] ; then 129GC_LOG_ENABLED=true 130 fi And use the ShellCheck to check the shell scripts, the results shows some errors 、 warnings and notes： https://github.com/koalaman/shellcheck/wiki/SC2068 https://github.com/koalaman/shellcheck/wiki/Sc2046 https://github.com/koalaman/shellcheck/wiki/Sc2086 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35231: Fix KAFKA-1740
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35231/#review87107 --- I'll try to do a more detailed review soon. I just want to get this one comment out of the way. core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala https://reviews.apache.org/r/35231/#comment139433 I don't think the ConsumerCoordinator constructor should change. The coordinator just needs a KafkaConfig, ZkClient, and OffsetManager in order to work. It shouldn't care about ReplicaManagers, KakfaSchedulers, or OffsetManagerConfigs. Passing in these lower-level dependencies makes testing the ConsumerCoordinator harder. As an example, think about how https://reviews.apache.org/r/35086/diff/ would work with this change. You'd have to mock out dependencies(ReplicaManager) that are a level deeper than what you actually care about(OffsetManager) with the hope that OffsetManager would react nicely to your mock instead of directly mocking the OffsetManager itself. - Onur Karaman On June 8, 2015, 11:12 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35231/ --- (Updated June 8, 2015, 11:12 p.m.) Review request for kafka. Bugs: KAFKA-1740 https://issues.apache.org/jira/browse/KAFKA-1740 Repository: kafka Description --- Move offset manager to coordinator, add validation logic for offset commit and fetch Diffs - clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java c1496a0851526f3c7d3905ce4bdff2129c83a6c1 clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 5b898c8f8ad5d0469f469b600c4b2eb13d1fc662 clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java 70844d65369f6ff300cbeb513dbb6650050c7eec clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java deec1fa480d5a5c5884a1c007b076aa64e902472 clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java 512a0ef7e619d54e74122c38119209f5cf9590e3 clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorTest.java b06c4a73e2b4e9472cd772c8bc32bf4a29f431bb core/src/main/scala/kafka/admin/TopicCommand.scala dacbdd089f96c67b7bbc1c0cd36490d4fe094c1b core/src/main/scala/kafka/cluster/Partition.scala 730a232482fdf77be5704cdf5941cfab3828db88 core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 6b4242c7cd1df9b3465db0fec35a25102c76cd60 core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 51e89c87ee2c20fc7f976536f01fa1055fb8e670 core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala c39e6de34ee531c6dfa9107b830752bd7f8fbe59 core/src/main/scala/kafka/server/KafkaApis.scala d63bc18d795a6f0e6994538ca55a9a46f7fb8ffd core/src/main/scala/kafka/server/KafkaServer.scala b320ce9f6a12c0ee392e91beb82e8804d167f9f4 core/src/main/scala/kafka/server/OffsetManager.scala 5cca85cf727975f6d3acb2223fd186753ad761dc core/src/main/scala/kafka/server/ReplicaManager.scala 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 core/src/test/scala/integration/kafka/api/ConsumerTest.scala 17b17b9b1520c7cc2e2ba96cdb1f9ff06e47bcad core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 07b1ff47bfc3cd3f948c9533c8dc977fa36d996f core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala c7136f20972614ac47aa57ab13e3c94ef775a4b7 core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 4f124af5c3e946045a78ad1519c37372a72c8985 core/src/test/scala/unit/kafka/coordinator/CoordinatorMetadataTest.scala 08854c5e6ec249368206298b2ac2623df18f266a core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 528525b719ec916e16f8b3ae3715bec4b5dcc47d Diff: https://reviews.apache.org/r/35231/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 33378: Patch for KAFKA-2136
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33378/ --- (Updated June 9, 2015, 1:37 a.m.) Review request for kafka, Joel Koshy and Jun Rao. Bugs: KAFKA-2136 https://issues.apache.org/jira/browse/KAFKA-2136 Repository: kafka Description --- Changes are - protocol changes to the fetch request and response to return the throttle_time_ms to clients - New producer/consumer metrics to expose the avg and max delay time for a client - Test cases. For now the patch will publish a zero delay and return a response Diffs (updated) - clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 56281ee15cc33dfc96ff64d5b1e596047c7132a4 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 07e65d4a54ba4eef5b787eba3e71cbe9f6a920bd clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 3dc8b015afd2347a41c9a9dbc02b8e367da5f75f clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 8686d83aa52e435c6adafbe9ff4bd1602281072a clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java eb8951fba48c335095cc43fc3672de1c733e07ff clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java fabeae3083a8ea55cdacbb9568f3847ccd85bab4 clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 37ec0b79beafcf5735c386b066eb319fb697eff5 clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java 419541011d652becf0cda7a5e62ce813cddb1732 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java e3cc1967e407b64cc734548c19e30de700b64ba8 core/src/main/scala/kafka/api/FetchRequest.scala 5b38f8554898e54800abd65a7415dd0ac41fd958 core/src/main/scala/kafka/api/FetchResponse.scala 0b6b33ab6f7a732ff1322b6f48abd4c43e0d7215 core/src/main/scala/kafka/api/ProducerRequest.scala c866180d3680da03e48d374415f10220f6ca68c4 core/src/main/scala/kafka/api/ProducerResponse.scala 5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 core/src/main/scala/kafka/consumer/SimpleConsumer.scala c16f7edd322709060e54c77eb505c44cbd77a4ec core/src/main/scala/kafka/server/AbstractFetcherThread.scala 83fc47417dd7fe4edf030217fa7fd69d99b170b0 core/src/main/scala/kafka/server/DelayedFetch.scala de6cf5bdaa0e70394162febc63b50b55ca0a92db core/src/main/scala/kafka/server/DelayedProduce.scala 05078b24ef28f2f4e099afa943e43f1d00359fda core/src/main/scala/kafka/server/KafkaApis.scala d63bc18d795a6f0e6994538ca55a9a46f7fb8ffd core/src/main/scala/kafka/server/OffsetManager.scala 5cca85cf727975f6d3acb2223fd186753ad761dc core/src/main/scala/kafka/server/ReplicaFetcherThread.scala b31b432a226ba79546dd22ef1d2acbb439c2e9a3 core/src/main/scala/kafka/server/ReplicaManager.scala 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 5717165f2344823fabe8f7cfafae4bb8af2d949a core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 00d59337a99ac135e8689bd1ecd928f7b1423d79 Diff: https://reviews.apache.org/r/33378/diff/ Testing --- New tests added Thanks, Aditya Auradkar
[jira] [Issue Comment Deleted] (KAFKA-2208) Consumer should detect coordinator failures
[ https://issues.apache.org/jira/browse/KAFKA-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2208: - Comment: was deleted (was: Created reviewboard https://reviews.apache.org/r/35231/diff/ against branch origin/trunk) Consumer should detect coordinator failures --- Key: KAFKA-2208 URL: https://issues.apache.org/jira/browse/KAFKA-2208 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Onur Karaman Assignee: Guozhang Wang Attachments: KAFKA-2208.patch, KAFKA-2208.patch, KAFKA-2208_2015-05-20_19:14:58.patch, KAFKA-2208_2015-05-31_17:05:06.patch, KAFKA-2208_2015-06-02_10:45:34.patch, KAFKA-2208_2015-06-02_14:06:58.patch Add coordinator discovery and failure detection to the consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578053#comment-14578053 ] Jun Rao commented on KAFKA-1367: Yes, perhaps some kind of more general locality could be useful. That can be done in a separate jira. Here, we just want to figure out whether it's useful to maintain ISR in TMR. [~jjkoshy], another issue without ISR is that initially a client will have no idea which replica is in sync and can only guess. Broker topic metadata not kept in sync with ZooKeeper - Key: KAFKA-1367 URL: https://issues.apache.org/jira/browse/KAFKA-1367 Project: Kafka Issue Type: Bug Affects Versions: 0.8.0, 0.8.1 Reporter: Ryan Berdeen Assignee: Ashish K Singh Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1367.txt When a broker is restarted, the topic metadata responses from the brokers will be incorrect (different from ZooKeeper) until a preferred replica leader election. In the metadata, it looks like leaders are correctly removed from the ISR when a broker disappears, but followers are not. Then, when a broker reappears, the ISR is never updated. I used a variation of the Vagrant setup created by Joe Stein to reproduce this with latest from the 0.8.1 branch: https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-25 System test improvements
Hi Gwen, I don't see any problem with this as long as we're convinced there's a good use case, which seems to be true. Cheers, Geoff On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira gshap...@cloudera.com wrote: Not completely random places :) People may use Cloudera / HWX distributions which include Kafka, but want to verify that these bits match a specific upstream release. I think having the tests separately will be useful for this. In this case, finding the tests are not a big issue - we'll add a download link :) On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Gwen, Currently the test and code are downloaded at the same time. Supposedly the tests in the same repository should cover match the code. Are you saying people downloaded a release from some random place and want to verify it? If that is the case, does that mean people still need to find the correct place to download the right test artifact? Thanks, Jiangjie (Becket) Qin On 6/4/15, 4:29 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Reviving the discussion a bit :) I think it will be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. The idea is that if someone downloads packages that claim to contain something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), users can easily download the tests and verify that it indeed passes the tests for this version and therefore behaves the way this version is expected to behave. Does it make sense? Gwen On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Ashish, Looks like Ewen already hit the main points, but a few additions: 1. ducktape repo is here: https://github.com/confluentinc/ducktape ducktape itself will be pip installable in the near future, and Kafka system tests will be able to depend on a particular version of ducktape. 2. The reporting is nothing fancy. We're definitely open to feedback, but it consists of: - top level summary of the test run (simple PASS/FAIL for each test) - top level info and debug logs - per-test info and debug logs - per-test service logs gathered from each service used in the test. For example, if your test pulls up a Kafka cluster with 5 brokers, the end result will have the Kafka logs from each of those 5 machines. Cheers, Geoff On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Ashish, 1. That was the plan. We put some effort into cleanly separating the framework so it would be reusable across many projects. 2. I think you're seeing a test in progress where the final report hasn't been created yet. If you visit one of the older ones you'll see it has a landing page with links: http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently we need to adjust when we update the 'latest' symlink. The logs that are collected for tests are configurable, and service implementations include sane defaults (so, e.g., you will always get the normal log file for Kafka, but only get the data files if the test asks for them). 3. No code coverage support. Haven't looked into it, so I couldn't comment on how hard it would be to add. -Ewen On Thu, May 21, 2015 at 2:38 PM, Ashish Singh asi...@cloudera.com wrote: Geoffrey, This looks great! A few questions. 1. Will ducktape be maintained separately as a github repo? 2. How easy is viewing the test results and logs. The link in KIP, http://testing.confluent.io/confluent_platform/latest/, lists a bunch of files and dirs. Could you add to KIP how the result and logs for the tests will be organized. 3. Does it support code coverage? If not, how easy/ difficult would it be? On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson ge...@confluent.io wrote: Great, I'll work on putting together a more detailed map of this replacement process. On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira gshap...@cloudera.com wrote: Love this idea :) I took a look at Ducktape API and it looks like a good fit - clean API, extensible, easy to use and powerful enough for our use-case. Something I'd like to see as part of the KIP is a map of what system-test currently tests, which ones we want to replace and a JIRA for replacing (possibly one for each group of tests). Basically, I know we all want to use the new system for new test cases (upgrades, etc), but I really want to make sure we don't get stuck with both systems forever.
[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.
[ https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated KAFKA-2254: --- Priority: Major (was: Critical) The shell script should be optimized , even kafka-run-class.sh has a syntax error. -- Key: KAFKA-2254 URL: https://issues.apache.org/jira/browse/KAFKA-2254 Project: Kafka Issue Type: Bug Components: build Affects Versions: 0.8.2.1 Environment: linux Reporter: Bo Wang Labels: client-script, kafka-run-class.sh, shell-script Fix For: 0.8.2.1 Attachments: kafka-shell-script.patch Original Estimate: 24h Remaining Estimate: 24h kafka-run-class.sh 128 line has a syntax error(missing a space): 127-loggc) 128 if [ -z $KAFKA_GC_LOG_OPTS] ; then 129GC_LOG_ENABLED=true 130 fi And use the ShellCheck to check the shell scripts, the results shows some errors 、 warnings and notes： https://github.com/koalaman/shellcheck/wiki/SC2068 https://github.com/koalaman/shellcheck/wiki/Sc2046 https://github.com/koalaman/shellcheck/wiki/Sc2086 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2208) Consumer should detect coordinator failures
[ https://issues.apache.org/jira/browse/KAFKA-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2208: - Attachment: KAFKA-2208.patch Consumer should detect coordinator failures --- Key: KAFKA-2208 URL: https://issues.apache.org/jira/browse/KAFKA-2208 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Onur Karaman Assignee: Guozhang Wang Attachments: KAFKA-2208.patch, KAFKA-2208.patch, KAFKA-2208_2015-05-20_19:14:58.patch, KAFKA-2208_2015-05-31_17:05:06.patch, KAFKA-2208_2015-06-02_10:45:34.patch, KAFKA-2208_2015-06-02_14:06:58.patch Add coordinator discovery and failure detection to the consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 35231: Fix KAFKA-1740
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35231/ --- Review request for kafka. Bugs: KAFKA-2208 https://issues.apache.org/jira/browse/KAFKA-2208 Repository: kafka Description --- Move offset manager to coordinator, add validation logic for offset commit and fetch Diffs - clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java c1496a0851526f3c7d3905ce4bdff2129c83a6c1 clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 5b898c8f8ad5d0469f469b600c4b2eb13d1fc662 clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java 70844d65369f6ff300cbeb513dbb6650050c7eec clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java deec1fa480d5a5c5884a1c007b076aa64e902472 clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java 512a0ef7e619d54e74122c38119209f5cf9590e3 clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorTest.java b06c4a73e2b4e9472cd772c8bc32bf4a29f431bb core/src/main/scala/kafka/admin/TopicCommand.scala dacbdd089f96c67b7bbc1c0cd36490d4fe094c1b core/src/main/scala/kafka/cluster/Partition.scala 730a232482fdf77be5704cdf5941cfab3828db88 core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 6b4242c7cd1df9b3465db0fec35a25102c76cd60 core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 51e89c87ee2c20fc7f976536f01fa1055fb8e670 core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala c39e6de34ee531c6dfa9107b830752bd7f8fbe59 core/src/main/scala/kafka/server/KafkaApis.scala d63bc18d795a6f0e6994538ca55a9a46f7fb8ffd core/src/main/scala/kafka/server/KafkaServer.scala b320ce9f6a12c0ee392e91beb82e8804d167f9f4 core/src/main/scala/kafka/server/OffsetManager.scala 5cca85cf727975f6d3acb2223fd186753ad761dc core/src/main/scala/kafka/server/ReplicaManager.scala 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 core/src/test/scala/integration/kafka/api/ConsumerTest.scala 17b17b9b1520c7cc2e2ba96cdb1f9ff06e47bcad core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 07b1ff47bfc3cd3f948c9533c8dc977fa36d996f core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala c7136f20972614ac47aa57ab13e3c94ef775a4b7 core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 4f124af5c3e946045a78ad1519c37372a72c8985 core/src/test/scala/unit/kafka/coordinator/CoordinatorMetadataTest.scala 08854c5e6ec249368206298b2ac2623df18f266a core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 528525b719ec916e16f8b3ae3715bec4b5dcc47d Diff: https://reviews.apache.org/r/35231/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Commented] (KAFKA-1740) Merge Offset manager into Coordinator
[ https://issues.apache.org/jira/browse/KAFKA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578032#comment-14578032 ] Guozhang Wang commented on KAFKA-1740: -- Created reviewboard https://reviews.apache.org/r/35231/diff/ against branch origin/trunk Merge Offset manager into Coordinator - Key: KAFKA-1740 URL: https://issues.apache.org/jira/browse/KAFKA-1740 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.9.0 This JIRA involves refactoring offset manager and merge it into coordinator, including adding the logic for consumer-id / generation-id checking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KAFKA-1740) Merge Offset manager into Coordinator
[ https://issues.apache.org/jira/browse/KAFKA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-1740 started by Guozhang Wang. Merge Offset manager into Coordinator - Key: KAFKA-1740 URL: https://issues.apache.org/jira/browse/KAFKA-1740 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.9.0 This JIRA involves refactoring offset manager and merge it into coordinator, including adding the logic for consumer-id / generation-id checking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578031#comment-14578031 ] Sriharsha Chintalapani commented on KAFKA-1690: --- [~junrao] meant to say GatheringByteChannel. Keeping those interfaces same as they are trunk now. new java producer needs ssl support as a client --- Key: KAFKA-1690 URL: https://issues.apache.org/jira/browse/KAFKA-1690 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Sriharsha Chintalapani Fix For: 0.8.3 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1740) Merge Offset manager into Coordinator
[ https://issues.apache.org/jira/browse/KAFKA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1740: - Status: Patch Available (was: In Progress) Merge Offset manager into Coordinator - Key: KAFKA-1740 URL: https://issues.apache.org/jira/browse/KAFKA-1740 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.9.0 This JIRA involves refactoring offset manager and merge it into coordinator, including adding the logic for consumer-id / generation-id checking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.
[ https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated KAFKA-2254: --- Status: Patch Available (was: Open) The shell script should be optimized , even kafka-run-class.sh has a syntax error. -- Key: KAFKA-2254 URL: https://issues.apache.org/jira/browse/KAFKA-2254 Project: Kafka Issue Type: Bug Components: build Affects Versions: 0.8.2.1 Environment: linux Reporter: Bo Wang Priority: Critical Labels: client-script, kafka-run-class.sh, shell-script Fix For: 0.8.2.1 Original Estimate: 24h Remaining Estimate: 24h kafka-run-class.sh 128 line has a syntax error(missing a space): 127-loggc) 128 if [ -z $KAFKA_GC_LOG_OPTS] ; then 129GC_LOG_ENABLED=true 130 fi And use the ShellCheck to check the shell scripts, the results shows some errors 、 warnings and notes： https://github.com/koalaman/shellcheck/wiki/SC2068 https://github.com/koalaman/shellcheck/wiki/Sc2046 https://github.com/koalaman/shellcheck/wiki/Sc2086 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.
[ https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated KAFKA-2254: --- Status: Open (was: Patch Available) The shell script should be optimized , even kafka-run-class.sh has a syntax error. -- Key: KAFKA-2254 URL: https://issues.apache.org/jira/browse/KAFKA-2254 Project: Kafka Issue Type: Bug Components: build Affects Versions: 0.8.2.1 Environment: linux Reporter: Bo Wang Priority: Critical Labels: client-script, kafka-run-class.sh, shell-script Fix For: 0.8.2.1 Original Estimate: 24h Remaining Estimate: 24h kafka-run-class.sh 128 line has a syntax error(missing a space): 127-loggc) 128 if [ -z $KAFKA_GC_LOG_OPTS] ; then 129GC_LOG_ENABLED=true 130 fi And use the ShellCheck to check the shell scripts, the results shows some errors 、 warnings and notes： https://github.com/koalaman/shellcheck/wiki/SC2068 https://github.com/koalaman/shellcheck/wiki/Sc2046 https://github.com/koalaman/shellcheck/wiki/Sc2086 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.
[ https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated KAFKA-2254: --- Attachment: kafka-shell-script.patch The shell script should be optimized , even kafka-run-class.sh has a syntax error. -- Key: KAFKA-2254 URL: https://issues.apache.org/jira/browse/KAFKA-2254 Project: Kafka Issue Type: Bug Components: build Affects Versions: 0.8.2.1 Environment: linux Reporter: Bo Wang Priority: Critical Labels: client-script, kafka-run-class.sh, shell-script Fix For: 0.8.2.1 Attachments: kafka-shell-script.patch Original Estimate: 24h Remaining Estimate: 24h kafka-run-class.sh 128 line has a syntax error(missing a space): 127-loggc) 128 if [ -z $KAFKA_GC_LOG_OPTS] ; then 129GC_LOG_ENABLED=true 130 fi And use the ShellCheck to check the shell scripts, the results shows some errors 、 warnings and notes： https://github.com/koalaman/shellcheck/wiki/SC2068 https://github.com/koalaman/shellcheck/wiki/Sc2046 https://github.com/koalaman/shellcheck/wiki/Sc2086 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 33378: Patch for KAFKA-2136
On June 5, 2015, 2:43 a.m., Joel Koshy wrote: core/src/main/scala/kafka/api/FetchResponse.scala, line 143 https://reviews.apache.org/r/33378/diff/5/?file=956942#file956942line143 follow-up Can you elaborate? On June 5, 2015, 2:43 a.m., Joel Koshy wrote: core/src/main/scala/kafka/server/DelayedFetch.scala, line 58 https://reviews.apache.org/r/33378/diff/5/?file=956947#file956947line58 This is slightly unwieldy. Perhaps we can hold this patch especially since this will be impacted by the main patch (KAFKA-2084) Sure, the plan is to commit this after the main patch. I can make this simpler if I added a class to represent the arguments for the callback but I don't think it will add a great deal of value On June 5, 2015, 2:43 a.m., Joel Koshy wrote: core/src/main/scala/kafka/api/ProducerResponse.scala, line 40 https://reviews.apache.org/r/33378/diff/5/?file=956944#file956944line40 We should do this based on the response version as well right? I gather the readFrom is only used on the client side. A client running this code will only send V1 style requests and will always get the throttleTime in return. This isn't the case for consumers because they will send an old version of the request from the ReplicaFetcherThread (if the intra.cluster.replication.protocol is old). - Aditya --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33378/#review86428 --- On May 12, 2015, 9:42 p.m., Aditya Auradkar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33378/ --- (Updated May 12, 2015, 9:42 p.m.) Review request for kafka, Joel Koshy and Jun Rao. Bugs: KAFKA-2136 https://issues.apache.org/jira/browse/KAFKA-2136 Repository: kafka Description --- Changes are - protocol changes to the fetch request and response to return the throttle_time_ms to clients - New producer/consumer metrics to expose the avg and max delay time for a client - Test cases. For now the patch will publish a zero delay and return a response Diffs - clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java ef9dd5238fbc771496029866ece1d85db6d7b7a5 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java b2db91ca14bbd17fef5ce85839679144fff3f689 clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 3dc8b015afd2347a41c9a9dbc02b8e367da5f75f clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 8686d83aa52e435c6adafbe9ff4bd1602281072a clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java eb8951fba48c335095cc43fc3672de1c733e07ff clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java fabeae3083a8ea55cdacbb9568f3847ccd85bab4 clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 37ec0b79beafcf5735c386b066eb319fb697eff5 clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java 419541011d652becf0cda7a5e62ce813cddb1732 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java e3cc1967e407b64cc734548c19e30de700b64ba8 core/src/main/scala/kafka/api/FetchRequest.scala b038c15186c0cbcc65b59479324052498361b717 core/src/main/scala/kafka/api/FetchResponse.scala 75aaf57fb76ec01660d93701a57ae953d877d81c core/src/main/scala/kafka/api/ProducerRequest.scala 570b2da1d865086f9830aa919a49063abbbe574d core/src/main/scala/kafka/api/ProducerResponse.scala 5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 core/src/main/scala/kafka/consumer/SimpleConsumer.scala 31a2639477bf66f9a05d2b9b07794572d7ec393b core/src/main/scala/kafka/server/AbstractFetcherThread.scala 83fc47417dd7fe4edf030217fa7fd69d99b170b0 core/src/main/scala/kafka/server/DelayedFetch.scala de6cf5bdaa0e70394162febc63b50b55ca0a92db core/src/main/scala/kafka/server/DelayedProduce.scala 05078b24ef28f2f4e099afa943e43f1d00359fda core/src/main/scala/kafka/server/KafkaApis.scala 417960dd1ab407ebebad8fdb0e97415db3e91a2f core/src/main/scala/kafka/server/OffsetManager.scala 18680ce100f10035175cc0263ba7787ab0f6a17a core/src/main/scala/kafka/server/ReplicaFetcherThread.scala b31b432a226ba79546dd22ef1d2acbb439c2e9a3 core/src/main/scala/kafka/server/ReplicaManager.scala 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 5717165f2344823fabe8f7cfafae4bb8af2d949a
[jira] [Commented] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.
[ https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578153#comment-14578153 ] Bo Wang commented on KAFKA-2254: The trunk have fixed kafka-run-class.sh syntax error. The shell script should be optimized , even kafka-run-class.sh has a syntax error. -- Key: KAFKA-2254 URL: https://issues.apache.org/jira/browse/KAFKA-2254 Project: Kafka Issue Type: Bug Components: build Affects Versions: 0.8.2.1 Environment: linux Reporter: Bo Wang Labels: client-script, kafka-run-class.sh, shell-script Fix For: 0.8.2.1 Attachments: kafka-shell-script.patch Original Estimate: 24h Remaining Estimate: 24h kafka-run-class.sh 128 line has a syntax error(missing a space): 127-loggc) 128 if [ -z $KAFKA_GC_LOG_OPTS] ; then 129GC_LOG_ENABLED=true 130 fi And use the ShellCheck to check the shell scripts, the results shows some errors 、 warnings and notes： https://github.com/koalaman/shellcheck/wiki/SC2068 https://github.com/koalaman/shellcheck/wiki/Sc2046 https://github.com/koalaman/shellcheck/wiki/Sc2086 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 33204: Patch for KAFKA-1646 merge to latest trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33204/ --- (Updated June 9, 2015, 3:02 a.m.) Review request for kafka. Summary (updated) - Patch for KAFKA-1646 merge to latest trunk Bugs: KAFKA-1646 https://issues.apache.org/jira/browse/KAFKA-1646 Repository: kafka Description (updated) --- Patch for KAFKA-1646 merge to latest trunk Diffs (updated) - core/src/main/scala/kafka/log/FileMessageSet.scala 2522604bd985c513527fa0c863a7df677ff7a503 core/src/main/scala/kafka/log/Log.scala 84e7b8fe9dd014884b60c4fbe13c835cf02a40e4 core/src/main/scala/kafka/log/LogConfig.scala a907da09e1ccede3b446459225e407cd1ae6d8b3 core/src/main/scala/kafka/log/LogSegment.scala ed039539ac18ea4d65144073915cf112f7374631 core/src/main/scala/kafka/server/KafkaConfig.scala 2d75186a110075e0c322db4b9f7a8c964a7a3e88 core/src/main/scala/kafka/server/KafkaServer.scala b320ce9f6a12c0ee392e91beb82e8804d167f9f4 core/src/main/scala/kafka/utils/CoreUtils.scala d0a8fa701564b4c13b3cd6501e1b6218d77e8e06 core/src/test/scala/unit/kafka/log/FileMessageSetTest.scala cec1caecc51507ae339ebf8f3b8a028b12a1a056 core/src/test/scala/unit/kafka/log/LogSegmentTest.scala 03fb3512c4a4450eac83d4cd4b0919baeaa22942 Diff: https://reviews.apache.org/r/33204/diff/ Testing --- Thanks, Honghai Chen
[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578235#comment-14578235 ] Honghai Chen commented on KAFKA-1646: - Created reviewboard against branch origin/trunk Improve consumer read performance for Windows - Key: KAFKA-1646 URL: https://issues.apache.org/jira/browse/KAFKA-1646 Project: Kafka Issue Type: Improvement Components: log Affects Versions: 0.8.1.1 Environment: Windows Reporter: xueqiang wang Assignee: xueqiang wang Labels: newbie, patch Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, KAFKA-1646_20150511_AddTestcases.patch, KAFKA-1646_20150609_MergeToLatestTrunk.patch This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system. This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Honghai Chen updated KAFKA-1646: Attachment: KAFKA-1646_20150609_MergeToLatestTrunk.patch Improve consumer read performance for Windows - Key: KAFKA-1646 URL: https://issues.apache.org/jira/browse/KAFKA-1646 Project: Kafka Issue Type: Improvement Components: log Affects Versions: 0.8.1.1 Environment: Windows Reporter: xueqiang wang Assignee: xueqiang wang Labels: newbie, patch Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, KAFKA-1646_20150511_AddTestcases.patch, KAFKA-1646_20150609_MergeToLatestTrunk.patch This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system. This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1646) Improve consumer read performance for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578235#comment-14578235 ] Honghai Chen edited comment on KAFKA-1646 at 6/9/15 3:07 AM: - Merge to latest trunk, patch attached. code review no change https://reviews.apache.org/r/33204/diff/4/ was (Author: waldenchen): Created reviewboard against branch origin/trunk Improve consumer read performance for Windows - Key: KAFKA-1646 URL: https://issues.apache.org/jira/browse/KAFKA-1646 Project: Kafka Issue Type: Improvement Components: log Affects Versions: 0.8.1.1 Environment: Windows Reporter: xueqiang wang Assignee: xueqiang wang Labels: newbie, patch Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, KAFKA-1646_20150511_AddTestcases.patch, KAFKA-1646_20150609_MergeToLatestTrunk.patch This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system. This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2260) Allow specifying expected offset on produce
Ben Kirwin created KAFKA-2260:
-
Summary: Allow specifying expected offset on produce
Key: KAFKA-2260
URL: https://issues.apache.org/jira/browse/KAFKA-2260
Project: Kafka
Issue Type: Improvement
Reporter: Ben Kirwin
Priority: Minor
I'd like to propose a change that adds a simple CAS-like mechanism to the Kafka
producer. This update has a small footprint, but enables a bunch of interesting
uses in stream processing or as a commit log for process state.
h4. Proposed Change
In short:
- Allow the user to attach a specific offset to each message produced.
- The server assigns offsets to messages in the usual way. However, if the
expected offset doesn't match the actual offset, the server should fail the
produce request instead of completing the write.
This is a form of optimistic concurrency control, like the ubiquitous
check-and-set -- but instead of checking the current value of some state, it
checks the current offset of the log.
h4. Motivation
Much like check-and-set, this feature is only useful when there's very low
contention. Happily, when Kafka is used as a commit log or as a
stream-processing transport, it's common to have just one producer (or a small
number) for a given partition -- and in many of these cases, predicting offsets
turns out to be quite useful.
- We get the same benefits as the 'idempotent producer' proposal: a producer
can retry a write indefinitely and be sure that at most one of those attempts
will succeed; and if two producers accidentally write to the end of the
partition at once, we can be certain that at least one of them will fail.
- It's possible to 'bulk load' Kafka this way -- you can write a list of n
messages consecutively to a partition, even if the list is much larger than the
buffer size or the producer has to be restarted.
- If a process is using Kafka as a commit log -- reading from a partition to
bootstrap, then writing any updates to that same partition -- it can be sure
that it's seen all of the messages in that partition at the moment it does its
first (successful) write.
There's a bunch of other similar use-cases here, but they all have roughly the
same flavour.
h4. Implementation
The major advantage of this proposal over other suggested transaction /
idempotency mechanisms is its minimality: it gives the 'obvious' meaning to a
currently-unused field, adds no new APIs, and requires very little new code or
additional work from the server.
- Produced messages already carry an offset field, which is currently ignored
by the server. This field could be used for the 'expected offset', with a sigil
value for the current behaviour. (-1 is a natural choice, since it's already
used to mean 'next available offset'.)
- We'd need a new error and error code for a 'CAS failure'.
- The server assigns offsets to produced messages in
{{ByteBufferMessageSet.validateMessagesAndAssignOffsets}}. After this changed,
this method would assign offsets in the same way -- but if they don't match the
offset in the message, we'd return an error instead of completing the write.
- To avoid breaking existing clients, this behaviour would need to live behind
some config flag. (Possibly global, but probably more useful per-topic?)
I understand all this is unsolicited and possibly strange: happy to answer
questions, and if this seems interesting, I'd be glad to flesh this out into a
full KIP or patch. (And apologies if this is the wrong venue for this sort of
thing!)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [DISCUSS] KIP-25 System test improvements
Hi KIP-25 thread, I consolidated some of the questions from this thread and elsewhere. Q: Can we see a map of what system-test currently tests, which ones we want to replace and JIRAs for replacing? A: Initial draft here: https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests Q: Will ducktape be maintained separately as a github repo? A: Yes https://github.com/confluentinc/ducktape Q: How easy is viewing the test results and logs, how will test output be structured? A: Hierarchical structure as outlined here: https://github.com/confluentinc/ducktape/wiki/Design-overview#output Q: Does it support code coverage? If not, how easy/ difficult would it be to support? A: It does not, and we have no immediate plans to support this. Difficulty unclear. Q: It would be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. A: This seems reasonable and not too much extra work. Definitely open to discussion on this. Q: Why not share running services across multiple tests? A: Prefer to optimize for simplicity and correctness over what might be a questionable improvement in run-time. Q: Are regressions - in the road map? A: yes Q: Are Jepsen style tests involving network failures in the road map? A: yes Thanks much, Geoff On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Gwen, I don't see any problem with this as long as we're convinced there's a good use case, which seems to be true. Cheers, Geoff On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira gshap...@cloudera.com wrote: Not completely random places :) People may use Cloudera / HWX distributions which include Kafka, but want to verify that these bits match a specific upstream release. I think having the tests separately will be useful for this. In this case, finding the tests are not a big issue - we'll add a download link :) On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Gwen, Currently the test and code are downloaded at the same time. Supposedly the tests in the same repository should cover match the code. Are you saying people downloaded a release from some random place and want to verify it? If that is the case, does that mean people still need to find the correct place to download the right test artifact? Thanks, Jiangjie (Becket) Qin On 6/4/15, 4:29 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Reviving the discussion a bit :) I think it will be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. The idea is that if someone downloads packages that claim to contain something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), users can easily download the tests and verify that it indeed passes the tests for this version and therefore behaves the way this version is expected to behave. Does it make sense? Gwen On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Ashish, Looks like Ewen already hit the main points, but a few additions: 1. ducktape repo is here: https://github.com/confluentinc/ducktape ducktape itself will be pip installable in the near future, and Kafka system tests will be able to depend on a particular version of ducktape. 2. The reporting is nothing fancy. We're definitely open to feedback, but it consists of: - top level summary of the test run (simple PASS/FAIL for each test) - top level info and debug logs - per-test info and debug logs - per-test service logs gathered from each service used in the test. For example, if your test pulls up a Kafka cluster with 5 brokers, the end result will have the Kafka logs from each of those 5 machines. Cheers, Geoff On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Ashish, 1. That was the plan. We put some effort into cleanly separating the framework so it would be reusable across many projects. 2. I think you're seeing a test in progress where the final report hasn't been created yet. If you visit one of the older ones you'll see it has a landing page with links: http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently we need to adjust when we update the 'latest' symlink. The logs that are collected for tests are configurable, and service implementations include sane defaults (so, e.g., you will always get the normal log file for Kafka, but only get the data files if the test asks for them). 3. No code coverage support. Haven't looked into it, so I couldn't comment on how hard it would be to add. -Ewen
Re: Review Request 35201: Fix KAFKA-2253
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35201/#review86987 --- core/src/main/scala/kafka/server/DelayedOperation.scala https://reviews.apache.org/r/35201/#comment139198 Will watchForOperation sound more precise? core/src/main/scala/kafka/server/DelayedOperation.scala https://reviews.apache.org/r/35201/#comment139197 Here we are grabbing removeWatchersLock first then synchronize on the watchers. In tryCompleteWatched, we synchronize on the watcher first then grab the removeWatchersLock. Will there be a deadlock? - Jiangjie Qin On June 8, 2015, 4:46 a.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35201/ --- (Updated June 8, 2015, 4:46 a.m.) Review request for kafka. Bugs: KAFKA-2253 https://issues.apache.org/jira/browse/KAFKA-2253 Repository: kafka Description --- Incorporated Jiangjie and Onur's comments Diffs - core/src/main/scala/kafka/server/DelayedOperation.scala 123078d97a7bfe2121655c00f3b2c6af21c53015 Diff: https://reviews.apache.org/r/35201/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Commented] (KAFKA-2231) Deleting a topic fails
[ https://issues.apache.org/jira/browse/KAFKA-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577095#comment-14577095 ] Bogdan Popa commented on KAFKA-2231: I'm having the same situation. It's a staging machine on AWS with one kafka broker and one zookeeper node. I'm using kafka 8.2.1. On --list all topics remain as marked for deletion even after multiple restarts of kafka / zookeeper. It's worth noting that the actual files got deleted but not for all topics. Another thing that might be relevant is that I've obtained this server by creating an image of an existing node form a production cluster and then changing the config to make it it's own cluster. delete.topic.enable is true Deleting a topic fails -- Key: KAFKA-2231 URL: https://issues.apache.org/jira/browse/KAFKA-2231 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Environment: Windows 8.1 Reporter: James G. Haberly Priority: Minor delete.topic.enable=true is in config\server.properties. Using --list shows the topic marked for deletion. Stopping and restarting kafka and zookeeper does not delete the topic; it remains marked for deletion. Trying to recreate the topic fails with Topic XXX already exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
