date:20140618

[jira] [Updated] (KAFKA-1483) Split Brain about Leader Partitions

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1483:
-

Assignee: Sriharsha Chintalapani  (was: Jun Rao)

> Split Brain about Leader Partitions
> ---
>
> Key: KAFKA-1483
> URL: https://issues.apache.org/jira/browse/KAFKA-1483
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Guozhang Wang
>Assignee: Sriharsha Chintalapani
>  Labels: newbie++
> Fix For: 0.9.0
>
>
> Today in the server there are two places storing the leader partition info:
> 1) leaderPartitions list in the ReplicaManager.
> 2) leaderBrokerIdOpt in the Partition.
> 1) is used as the ground truth to decide if the server is the current leader 
> for serving requests; 2) is used as the ground truth for reporting leader 
> counts metrics, etc and for the background Shrinking-ISR thread to decide 
> which partition to check. There is a risk that these two ground truth caches 
> are not consistent, and we'd better only make one of them as the ground truth.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-218) ZOOKEEPER-961 is nasty, upgrade to zk 3.3.4

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-218.
---


> ZOOKEEPER-961 is nasty, upgrade to zk 3.3.4
> ---
>
> Key: KAFKA-218
> URL: https://issues.apache.org/jira/browse/KAFKA-218
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chris Burroughs
>Assignee: Pierre-Yves Ritschard
>Priority: Critical
>  Labels: newbie
> Fix For: 0.7.1
>
> Attachments: 0002-KAFKA-202-upgrade-zookeeper-to-3.3.4.patch
>
>
> 3.3.4 is out with ZOOKEEPER-961, which I think is our most reported issue.
> http://www.cloudera.com/blog/2011/11/apache-zookeeper-3-3-4-has-been-released/
> Should be a one char changes, but the jar hasn't hit the maven repos yet.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-241) ConsumerIterator throws a IllegalStateException after a ConsumerTimeout occurs

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-241.
---


> ConsumerIterator throws a IllegalStateException after a ConsumerTimeout occurs
> --
>
> Key: KAFKA-241
> URL: https://issues.apache.org/jira/browse/KAFKA-241
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.7
>Reporter: Patricio Echague
>Assignee: Jun Rao
>  Labels: newbie
> Fix For: 0.7.1
>
> Attachments: consumerIteratorTestCase.java, kafka-241.patch
>
>
> Please find the test case attached.
> After a timeout occurs (property consumer.timeout.ms > 0 ) the 
> consumerIterator throws an IllegalStateException.
> The work around seems to be to recreate the MessageStream an issue a new 
> Iterator.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-220) LogManager test fails on linux

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-220.
---


> LogManager test fails on linux
> --
>
> Key: KAFKA-220
> URL: https://issues.apache.org/jira/browse/KAFKA-220
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.7
> Environment: nnarkhed-ld:~ nnarkhed$ uname -r
> 2.6.32-131.4.1.el6.x86_64
>Reporter: Neha Narkhede
>Assignee: Jun Rao
>Priority: Critical
>  Labels: newbie
> Fix For: 0.7.1
>
> Attachments: kafka-220.patch
>
>
> On Linux, LogManagerTest fails on each and every run
> [info] Test Starting: testCleanupExpiredSegments
> [error] Test Failed: testCleanupExpiredSegments
> junit.framework.AssertionFailedError: Now there should only be only one 
> segment. expected:<1> but was:<12>
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.failNotEquals(Assert.java:277)
> at junit.framework.Assert.assertEquals(Assert.java:64)
> at junit.framework.Assert.assertEquals(Assert.java:195)
> at 
> kafka.log.LogManagerTest.testCleanupExpiredSegments(LogManagerTest.scala:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
> at 
> org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
> at 
> org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
> at 
> org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
> at 
> org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
> at 
> org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
> at 
> org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
> at 
> org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
> at 
> org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
> at 
> org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
> at 
> org.junit.internal.runners.CompositeRunner.run(CompositeRunner.java:29)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:121)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:100)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:91)
> at org.scalatest.junit.JUnitSuite$class.run(JUnitSuite.scala:261)
> at kafka.log.LogManagerTest.run(LogManagerTest.scala:28)
> at 
> org.scalatest.tools.ScalaTestFramework$ScalaTestRunner.run(ScalaTestFramework.scala:40)
> at sbt.TestRunner.run(TestFramework.scala:53)
> at sbt.TestRunner.runTest$1(TestFramework.scala:67)
> at sbt.TestRunner.run(TestFramework.scala:76)
> at 
> sbt.TestFramework$$anonfun$10$$anonfun$apply$11.runTest$2(TestFramework.scala:194)
> at 
> sbt.TestFramework$$anonfun$10$$anonfun$apply$11$$anonfun$apply$12.apply(TestFramework.scala:205)
> at 
> sbt.TestFramework$$anonfun$10$$anonfun$apply$11$$anonfun$apply$12.apply(TestFramework.scala:205)
> at sbt.NamedTestTask.run(TestFramework.scala:92)
> at 
> sbt.ScalaProject$$anonfun$sbt$ScalaProject$$toTask$1.apply(ScalaProject.scala:193)
> at 
> sbt.ScalaProject$$anonfun$sbt$ScalaProject$$toTask$1.apply(ScalaProject.scala:193)
> at sbt.TaskManager$Task.invoke(TaskManager.scala:62)
> at sbt.impl.RunTask.doRun$1(RunTask.scala:77)
> at sbt.impl.RunTask.runTask(RunTask.scala:85)
> at sbt.impl.RunTask.sbt$impl$RunTask$$runIfNotRoot(RunTask.scala:60)
> at 
> sbt.impl.RunTask$$anonfun$runTasksExceptRoot$2.apply(RunTask.scala:48)
> at 
> sbt.impl.RunTask$$anonfun$runTasksExceptRoot$2.apply(RunTask.scala:48)
> at 
> sbt.Distributor$Run$Worker$$anonfun$2.apply(ParallelRunner.scala:131)
> at 
> sbt.Distributor$Run$Worker$$anonfun$2.apply(ParallelRunner.scala:131)
> at sbt.Control$.trapUnit(Control.scala:19)
> at sbt.Distributor$Run$Worker.run(ParallelRunner.scala:131)
> [info] Test Starting: testCleanupSegmentsToMaintainSize



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-215) Improve system tests for the mirroring code

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-215.
---


> Improve system tests for the mirroring code
> ---
>
> Key: KAFKA-215
> URL: https://issues.apache.org/jira/browse/KAFKA-215
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.7
>Reporter: Neha Narkhede
>Priority: Critical
>  Labels: newbie
> Fix For: 0.8.0
>
>
> Improve the system tests for the mirroring code to *add* the following 
> testing scenarios -
> 1. Bounce the mirroring kafka cluster during the test
> 2. Add new topics to the source kafka cluster while the mirror is copying 
> data for existing topics
> 3. Bounce the source Kafka cluster during the test
> The point of this improvement is to catch any bugs in the consumer 
> rebalancing or the detection of new topics logic. Also, verification code of 
> this part of the system test can report on the % of duplicates/ % data loss 
> in the mirror cluster. Since Kafka guarantees at least once delivery, small 
> percentage of duplicates is ok, but any amount of data loss is a critical bug



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-247) max.message.size and fetch.size defaults should be consistent

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-247.
---


> max.message.size and fetch.size defaults should be consistent
> -
>
> Key: KAFKA-247
> URL: https://issues.apache.org/jira/browse/KAFKA-247
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.7
>Reporter: Blake Matheny
>Assignee: Pierre-Yves Ritschard
>Priority: Minor
>  Labels: newbie
> Fix For: 0.7.1
>
> Attachments: 0001-Fix-KAFKA-247-by-bumping-fetch.size.patch
>
>
> The default max.message.size for a producer is ~976kB. The default fetch.size 
> for a consumer is 300kB. Having the default fetch.size less than the default 
> max.message.size causes new users with messages larger than fetch.size to run 
> into the InvalidMessageSizeException issue.
> Making the default max.message.size less than or equal to the default 
> fetch.size would eliminate that problem for most new setups.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-374) Move to java CRC32 implementation

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-374.
---


> Move to java CRC32 implementation
> -
>
> Key: KAFKA-374
> URL: https://issues.apache.org/jira/browse/KAFKA-374
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
>Assignee: Jay Kreps
>Priority: Minor
>  Labels: newbie
> Attachments: KAFKA-374-draft.patch, KAFKA-374.patch
>
>
> We keep a per-record crc32. This is fairly cheap algorithm, but the java 
> implementation uses JNI and it seems to be a bit expensive for small records. 
> I have seen this before in Kafka profiles, and I noticed it on another 
> application I was working on. Basically with small records the native 
> implementation can only checksum < 100MB/sec. Hadoop has done some analysis 
> of this and replaced it with a Java implementation that is 2x faster for 
> large values and 5-10x faster for small values. Details are here HADOOP-6148.
> We should do a quick read/write benchmark on log and message set iteration 
> and see if this improves things.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-245) upgrade to zkclient 0.1

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-245.
---


> upgrade to zkclient 0.1
> ---
>
> Key: KAFKA-245
> URL: https://issues.apache.org/jira/browse/KAFKA-245
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.7
>Reporter: Pierre-Yves Ritschard
>Assignee: Pierre-Yves Ritschard
>  Labels: newbie
> Attachments: 
> 0001-KAFKA-245-zkclient-0.1-to-enable-maven-syncing.patch, 
> 0003-follow-up-to-KAFKA-245-use-maven-to-fetch-zkclient.patch
>
>
> the zkclient jar bundled with kafka should be synced with what is available 
> on maven central. the artifact which has group com.github.sgroschupf, 
> artifact id zkclient and version 0.1 is from a day after the one bundled with 
> kafka and should thus be sufficient for kafka's needs.
> I have tested it locally and find no regressions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-455) ProducerSendThread calls ListBuffer.size a whole bunch. That is a O(n) operation

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-455.
---


> ProducerSendThread calls ListBuffer.size a whole bunch. That is a O(n) 
> operation
> 
>
> Key: KAFKA-455
> URL: https://issues.apache.org/jira/browse/KAFKA-455
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.7.1, 0.7.2, 0.8.0
> Environment: NA
>Reporter: Matthew Rathbone
>Priority: Minor
>  Labels: newbie
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Hi all,
> So there are various statements throughout the async code that call 
> 'events.size', mostly for debugging purposes.
> Problem is that this call is O(n), so it could add up if the batch size is 
> high. (it's a ListBuffer)
> I see this in at least ProducerSendThread (x4), likely more. Will factor this 
> out myself soon when I start hacking on the project, just wanted to put this 
> somewhere.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-367) StringEncoder/StringDecoder use platform default character set

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-367.
---


> StringEncoder/StringDecoder use platform default character set
> --
>
> Key: KAFKA-367
> URL: https://issues.apache.org/jira/browse/KAFKA-367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
>Assignee: Eli Reisman
>  Labels: newbie
> Fix For: 0.8.0
>
> Attachments: KAFKA-367-1.patch, KAFKA-367-2.patch, KAFKA-367-3.patch, 
> KAFKA-367-3.patch, KAFKA-367-4.patch, KAFKA-367-5.patch
>
>
> StringEncoder and StringDecoder take the platform default character set. This 
> is bad since the messages they produce are sent off that machine. We should
> -- add a new required argument to these that adds the character set and 
> default to UTF-8 rather than the machine setting
> -- add a commandline parameter for the console-* tools to let you specify the 
> correct encoding.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-233) The producer's load balancing logic can send requests to dead brokers, when using the async producer option

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-233.
---


> The producer's load balancing logic can send requests to dead brokers, when 
> using the async producer option
> ---
>
> Key: KAFKA-233
> URL: https://issues.apache.org/jira/browse/KAFKA-233
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.7
>Reporter: Neha Narkhede
>  Labels: newbie
> Fix For: 0.8.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> The ZK producer, when used with the async producer option does the following 
> 1. Create a pool of async producers, one each for a broker registered under 
> /broker/ids
> 2. On each send request, apply the Partitioner, to decide the broker and 
> partition to send the data
> 3. Use the Async producer's send API to enqueue that data into the async 
> producer's queue
> 4. When the data is dequeued by the ProducerSendThread, use the underlying 
> sync producer to send it to the broker
> The load balancing decision is taken in step 2, before entering the queue. 
> This leaves a window of error, equal to the queue length, when a broker can 
> go down. When this happens, potentially, a queue worth of data can fail to 
> reach a broker, and will be dropped by the EventHandler. 
> To correct this, the Producer, with the async option, needs to be refactored 
> to allow only a single queue to hold all requests. And the application of the 
> Partitioner should be moved to the end of the queue, in the EventHandler.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-456) ProducerSendThread calls ListBuffer.size a whole bunch. That is a O(n) operation

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-456.
---


> ProducerSendThread calls ListBuffer.size a whole bunch. That is a O(n) 
> operation
> 
>
> Key: KAFKA-456
> URL: https://issues.apache.org/jira/browse/KAFKA-456
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
> Environment: NA
>Reporter: Matthew Rathbone
>Assignee: David Arthur
>Priority: Minor
>  Labels: newbie
> Fix For: 0.8.0
>
> Attachments: KAFKA-456.patch
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Hi all,
> So there are various statements throughout the async code that call 
> 'events.size', mostly for debugging purposes.
> Problem is that this call is O(n), so it could add up if the batch size is 
> high. (it's a ListBuffer)
> I see this in at least ProducerSendThread (x4), likely more. Will factor this 
> out myself soon when I start hacking on the project, just wanted to put this 
> somewhere.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1283) Log4jAppender is unable to send the message.

2014-06-18 Thread Babak Behzad (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036969#comment-14036969
 ] 

Babak Behzad commented on KAFKA-1283:
-

[~nehanarkhede]: I came up with a patch for this today! I am not sure if it's a 
general one, but it works for our case. Should I sync up with  [~harsha_ch]?

> Log4jAppender is unable to send the message.
> 
>
> Key: KAFKA-1283
> URL: https://issues.apache.org/jira/browse/KAFKA-1283
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.0
> Environment: ubuntu. eclipse.
>Reporter: Dongkyoung Kim
>Assignee: Sriharsha Chintalapani
>  Labels: newbie
> Fix For: 0.8.0
>
>
> User application can`t send any messages via KafkaLog4jAppender.
> Here is log4j.properties.
> --
> log4j.rootLogger=INFO, stdout, KAFKA
> log4j.appender.stdout=org.apache.log4j.ConsoleAppender
> log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
> log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
> log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
> log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
> log4j.appender.KAFKA.layout.ConversionPattern=%-5p: %c - %m%n
> log4j.appender.KAFKA.BrokerList=hnode01:9092
> log4j.appender.KAFKA.Topic=DKTestEvent
> #log4j.appender.KAFKA.SerializerClass=kafka.log4j.AppenderStringEncoder
> --
> And this is a sample application.
> --
> import org.apache.log4j.Logger;
> import org.apache.log4j.BasicConfigurator;
> import org.apache.log4j.PropertyConfigurator;
> public class HelloWorld {
>   static Logger logger = Logger.getLogger(HelloWorld.class.getName());
>   public static void main(String[] args) {
>   PropertyConfigurator.configure(args[0]);
>   logger.info("Entering application.");
>   logger.debug("Debugging!.");
>   logger.info("Exiting application.");
>   }
> }
> --
> Since my project is maven project, I attached pom.xml also.
> --
> http://maven.apache.org/POM/4.0.0"; 
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>   4.0.0
>   com.my.app
>   log4-appender
>   0.0.1-SNAPSHOT
>   
>   
>   org.apache.kafka
>   kafka_2.8.2
>   0.8.0
>   
>   
>   log4j
>   log4j
>   1.2.17
>   
>   
> 
> --
> And I am getting these error:
> --
> INFO [main] (Logging.scala:67) - Verifying properties
>  INFO [main] (Logging.scala:67) - Property metadata.broker.list is overridden 
> to hnode01:9092
>  INFO [main] (Logging.scala:67) - Property serializer.class is overridden to 
> kafka.serializer.StringEncoder
>  INFO [main] (HelloWorld.java:14) - Entering application.
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 0 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 1 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 2 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 3 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 4 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 5 for 1 topic(s) 
> Set(DKTestEvent)
> .
> .
> .
> INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 60 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:

Build failed in Jenkins: Kafka-trunk #207

2014-06-18 Thread Apache Jenkins Server

See 

Changes:

[neha.narkhede] KAFKA-1382 follow up patch; reviewed by Neha Narkhede

--
[...truncated 1446 lines...]
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:144)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:125)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:32)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at kafka.admin.DeleteTopicTest.setUp(DeleteTopicTest.scala:34)

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:144)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:125)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:32)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at kafka.admin.DeleteTopicTest.setUp(DeleteTopicTest.scala:34)

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:144)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:125)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:32)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at kafka.admin.DeleteTopicTest.setUp(DeleteTopicTest.scala:34)

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:144)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:125)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:32)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at kafka.admin.DeleteTopicTest.setUp(DeleteTopicTest.scala:34)

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:144)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:125)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:32)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at kafka.admin.DeleteTopicTest.setUp(DeleteTopicTest.scala:34)

kafka.admin.DeleteTopicTest > testAutoCreateAfterDeleteTopic FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:144)
at 
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:125)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:32)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at kafka.admin.DeleteTopicTest.setUp(DeleteTopicTest.scala:34)

kafka.admin.DeleteTopicTest >

[jira] [Closed] (KAFKA-1041) Number of file handles increases indefinitely in producer if broker host is unresolvable

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1041.



> Number of file handles increases indefinitely in producer if broker host is 
> unresolvable
> 
>
> Key: KAFKA-1041
> URL: https://issues.apache.org/jira/browse/KAFKA-1041
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.0
> Environment: *unix*
>Reporter: Rajasekar Elango
>Assignee: Rajasekar Elango
>  Labels: features, newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1041-patch.diff
>
>
> We found a issue that if broker host is un resolvable, the number of file 
> handle keep increasing for every message we produce and eventually it uses up 
> all available files handles in operating system. If broker itself is not 
> running and broker host name is resolvable, open file handles count stays 
> flat.
> lsof output shows number of these open file handles continue to grow for 
> every message we produce.
>  java  19631relango   81u sock0,6  0t0  
> 196966526 can't identify protocol
> I can easily reproduce this with console producer,  If I run console producer 
> with right hostname and if broker is not running, the console producer will 
> exit after three tries. But If I run console producer with unresolvable 
> broker, it throws below exception and continues to wait for user input, every 
> time I enter new message, it opens socket and file handle count keeps 
> increasing.. 
> Here is Exception in producer
> ERROR fetching topic metadata for topics [Set(test-1378245487417)] from 
> broker [ArrayBuffer(id:0,host:localhost1,port:6667)] failed 
> (kafka.utils.Utils$)
> kafka.common.KafkaException: fetching topic metadata for topics 
> [Set(test-1378245487417)] from broker 
> [ArrayBuffer(id:0,host:localhost1,port:6667)] failed
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:51)
> at 
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> at 
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79)
> at kafka.utils.Utils$.swallow(Utils.scala:186)
> at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> at kafka.utils.Utils$.swallowError(Utils.scala:45)
> at 
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79)
> at 
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at 
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at 
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> at 
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> at 
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> Caused by: java.nio.channels.UnresolvedAddressException
> at sun.nio.ch.Net.checkAddress(Net.java:30)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:487)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:59)
> at kafka.producer.SyncProducer.connect(SyncProducer.scala:151)
> at 
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:166)
> at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:73)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:117)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:37)
> ... 12 more



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-653) Allow getTopicMetadata to get metadata for all topics

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-653.
---


> Allow getTopicMetadata to get metadata for all topics
> -
>
> Key: KAFKA-653
> URL: https://issues.apache.org/jira/browse/KAFKA-653
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>  Labels: newbie
>
> Currently the topic metadata api requires a list of topics. This is good when 
> you know what you want, but for tools that need to replicate all topics or 
> those that match a pattern or something like that, they may not know their 
> topic names a priori.
> To support this it would be nice to have the behavior be that issuing the 
> getTopicMetadata request with no topics yields metadata for all topics (as 
> this is more useful than the current behavior of giving you metadata for no 
> topics).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-616) Implement acks=0

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-616.
---


> Implement acks=0
> 
>
> Key: KAFKA-616
> URL: https://issues.apache.org/jira/browse/KAFKA-616
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
>Assignee: Neha Narkhede
>  Labels: newbie
>
> For completeness it would be nice to handle the case where acks=0 in the 
> produce request. The meaning of this would be that the broker immediately 
> responds without blocking even on the local write. The advantage of this is 
> that it would often isolate the producer from any latency in the local write 
> (which we have occasionally seen).
> Since we don't block on the append the response would contain a placeholder 
> for all the fields--e.g. offset=-1 and no error.
> This should be pretty easy to implement, just an if statement in 
> KafkaApis.handleProduceRequest to send the response immediately in this case 
> (and again to avoid sending a second response later).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-1079) Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1079.



> Liars in PrimitiveApiTest that promise to test api in compression mode, but 
> don't do this actually
> --
>
> Key: KAFKA-1079
> URL: https://issues.apache.org/jira/browse/KAFKA-1079
> Project: Kafka
>  Issue Type: Test
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Kostya Golikov
>Assignee: Kostya Golikov
>Priority: Minor
>  Labels: newbie, test
> Fix For: 0.8.2
>
> Attachments: testing-with-compression-producer.patch, 
> testing-with-compression-v2.patch
>
>
> Long time ago (0.7) we had ByteBufferMessageSet as a part of api and it's 
> allowed us to control compression. Times goes on and now PrimitiveApiTest 
> have methods that promise to test api with compression enabled, but in fact 
> they don't. Moreover this methods almost entirely copy their counterparts 
> without compression. In particular I'm talking about 
> `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and 
> `testMultiProduce`/`testMultiProduceWithCompression` pairs. 
> The fix could be super-easy and soundness -- just parameterize methods with 
> producer of each type (with/without compression). Sadly but it isn't feasible 
> for junit3, so straightforward solution is to do the same ugly thing as 
> `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget 
> about class-wide producer and roll-out it's own. I will attach path if that 
> is a problem indeed. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-1146) toString() on KafkaStream gets stuck indefinitely

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1146.



> toString() on KafkaStream gets stuck indefinitely
> -
>
> Key: KAFKA-1146
> URL: https://issues.apache.org/jira/browse/KAFKA-1146
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.0
>Reporter: Arup Malakar
>Assignee: Arup Malakar
>Priority: Trivial
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1146.patch
>
>
> There is no toString implementation for KafkaStream, so if a user tries to 
> print the stream it falls back to default toString implementation which tries 
> to iterate over the collection and gets stuck indefinitely as it awaits 
> messages. KafkaStream could instead override the toString and return a 
> verbose description of the stream with topic name etc.
> println("Current stream: " + stream) // This call never returns



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1283) Log4jAppender is unable to send the message.

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1283:
-

Assignee: Sriharsha Chintalapani  (was: Jun Rao)

> Log4jAppender is unable to send the message.
> 
>
> Key: KAFKA-1283
> URL: https://issues.apache.org/jira/browse/KAFKA-1283
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.0
> Environment: ubuntu. eclipse.
>Reporter: Dongkyoung Kim
>Assignee: Sriharsha Chintalapani
>  Labels: newbie
> Fix For: 0.8.0
>
>
> User application can`t send any messages via KafkaLog4jAppender.
> Here is log4j.properties.
> --
> log4j.rootLogger=INFO, stdout, KAFKA
> log4j.appender.stdout=org.apache.log4j.ConsoleAppender
> log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
> log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
> log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
> log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
> log4j.appender.KAFKA.layout.ConversionPattern=%-5p: %c - %m%n
> log4j.appender.KAFKA.BrokerList=hnode01:9092
> log4j.appender.KAFKA.Topic=DKTestEvent
> #log4j.appender.KAFKA.SerializerClass=kafka.log4j.AppenderStringEncoder
> --
> And this is a sample application.
> --
> import org.apache.log4j.Logger;
> import org.apache.log4j.BasicConfigurator;
> import org.apache.log4j.PropertyConfigurator;
> public class HelloWorld {
>   static Logger logger = Logger.getLogger(HelloWorld.class.getName());
>   public static void main(String[] args) {
>   PropertyConfigurator.configure(args[0]);
>   logger.info("Entering application.");
>   logger.debug("Debugging!.");
>   logger.info("Exiting application.");
>   }
> }
> --
> Since my project is maven project, I attached pom.xml also.
> --
> http://maven.apache.org/POM/4.0.0"; 
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>   4.0.0
>   com.my.app
>   log4-appender
>   0.0.1-SNAPSHOT
>   
>   
>   org.apache.kafka
>   kafka_2.8.2
>   0.8.0
>   
>   
>   log4j
>   log4j
>   1.2.17
>   
>   
> 
> --
> And I am getting these error:
> --
> INFO [main] (Logging.scala:67) - Verifying properties
>  INFO [main] (Logging.scala:67) - Property metadata.broker.list is overridden 
> to hnode01:9092
>  INFO [main] (Logging.scala:67) - Property serializer.class is overridden to 
> kafka.serializer.StringEncoder
>  INFO [main] (HelloWorld.java:14) - Entering application.
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 0 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 1 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 2 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 3 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 4 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 5 for 1 topic(s) 
> Set(DKTestEvent)
> .
> .
> .
> INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 60 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hnode01,port:9092 with correlation id 61 for 1 topic(s) 
> Set(DKTestEvent)
>  INFO [main] (HelloWorld.java:14) - Fetching metadata from broker 
> id:0,host:hno

[jira] [Closed] (KAFKA-1195) windows kafka-run-class.bat not updated

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1195.



> windows kafka-run-class.bat not updated
> ---
>
> Key: KAFKA-1195
> URL: https://issues.apache.org/jira/browse/KAFKA-1195
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.0
> Environment: windows 2008R2
>Reporter: Amir Gershman
>Priority: Minor
>  Labels: newbie
>
> The kafka-run-class.bat in the windows directory doesn't work. 
> base dir is wrongly defined. ivy jars names don't match.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-1189) kafka-server-stop.sh doesn't stop broker

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1189.



> kafka-server-stop.sh doesn't stop broker
> 
>
> Key: KAFKA-1189
> URL: https://issues.apache.org/jira/browse/KAFKA-1189
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.8.0
> Environment: RHEL 6.4 64bit, Java 6u35
>Reporter: Bryan Baugher
>Assignee: Martin Kleppmann
>Priority: Minor
>  Labels: newbie
> Attachments: KAFKA-1189.patch
>
>
> Just before the 0.8.0 release this commit[1] changed the signal in the 
> kafka-server-stop.sh script from SIGTERM to SIGINT. This doesn't seem to stop 
> the broker. Changing this back to SIGTERM (or -15) fixes the issues.
> [1] - 
> https://github.com/apache/kafka/commit/51de7c55d2b3107b79953f401fc8c9530bd0eea0



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-1438) Migrate kafka client tools

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1438.



> Migrate kafka client tools
> --
>
> Key: KAFKA-1438
> URL: https://issues.apache.org/jira/browse/KAFKA-1438
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Sriharsha Chintalapani
>  Labels: newbie, tools, usability
> Fix For: 0.8.2
>
> Attachments: KAFKA-1438.patch, KAFKA-1438.patch, 
> KAFKA-1438_2014-05-27_11:45:29.patch, KAFKA-1438_2014-05-27_12:16:00.patch, 
> KAFKA-1438_2014-05-27_17:08:59.patch, KAFKA-1438_2014-05-28_08:32:46.patch, 
> KAFKA-1438_2014-05-28_08:36:28.patch, KAFKA-1438_2014-05-28_08:40:22.patch, 
> KAFKA-1438_2014-05-30_11:36:01.patch, KAFKA-1438_2014-05-30_11:38:46.patch, 
> KAFKA-1438_2014-05-30_11:42:32.patch
>
>
> Currently the console/perf client tools scatter across different packages, 
> we'd better to:
> 1. Move Consumer/ProducerPerformance and SimpleConsumerPerformance to tools 
> and remove the perf sub-project.
> 2. Move ConsoleConsumer from kafka.consumer to kafka.tools.
> 3. Move other consumer related tools from kafka.consumer to kafka.tools.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Closed] (KAFKA-1472) Add the compression ratio metrics in the new producer

2014-06-18 Thread Neha Narkhede (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1472.



> Add the compression ratio metrics in the new producer
> -
>
> Key: KAFKA-1472
> URL: https://issues.apache.org/jira/browse/KAFKA-1472
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Guozhang Wang
>Assignee: Dong Lin
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1472.patch, KAFKA-1472_2014-06-04_10:31:44.patch, 
> KAFKA-1472_2014-06-05_22:42:23.patch
>
>
> New producer's bytes throughput is based on compressed data. With the current 
> implementation, it would be very easy to get the compression ratio of each 
> closed batch upon drain(). It would be good to have such metric to help 
> understanding the throughput metrics.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1382) Update zkVersion on partition state update failures

2014-06-18 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036964#comment-14036964
 ] 

Jun Rao commented on KAFKA-1382:


Thanks for the patch. This looks good to me.

> Update zkVersion on partition state update failures
> ---
>
> Key: KAFKA-1382
> URL: https://issues.apache.org/jira/browse/KAFKA-1382
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.2
>
> Attachments: KAFKA-1382.patch, KAFKA-1382_2014-05-30_21:19:21.patch, 
> KAFKA-1382_2014-05-31_15:50:25.patch, KAFKA-1382_2014-06-04_12:30:40.patch, 
> KAFKA-1382_2014-06-07_09:00:56.patch, KAFKA-1382_2014-06-09_18:23:42.patch, 
> KAFKA-1382_2014-06-11_09:37:22.patch, KAFKA-1382_2014-06-16_13:50:16.patch, 
> KAFKA-1382_2014-06-16_14:19:27.patch
>
>
> Our updateIsr code is currently:
>   private def updateIsr(newIsr: Set[Replica]) {
> debug("Updated ISR for partition [%s,%d] to %s".format(topic, 
> partitionId, newIsr.mkString(",")))
> val newLeaderAndIsr = new LeaderAndIsr(localBrokerId, leaderEpoch, 
> newIsr.map(r => r.brokerId).toList, zkVersion)
> // use the epoch of the controller that made the leadership decision, 
> instead of the current controller epoch
> val (updateSucceeded, newVersion) = 
> ZkUtils.conditionalUpdatePersistentPath(zkClient,
>   ZkUtils.getTopicPartitionLeaderAndIsrPath(topic, partitionId),
>   ZkUtils.leaderAndIsrZkData(newLeaderAndIsr, controllerEpoch), zkVersion)
> if (updateSucceeded){
>   inSyncReplicas = newIsr
>   zkVersion = newVersion
>   trace("ISR updated to [%s] and zkVersion updated to 
> [%d]".format(newIsr.mkString(","), zkVersion))
> } else {
>   info("Cached zkVersion [%d] not equal to that in zookeeper, skip 
> updating ISR".format(zkVersion))
> }
> We encountered an interesting scenario recently when a large producer fully
> saturated the broker's NIC for over an hour. The large volume of data led to
> a number of ISR shrinks (and subsequent expands). The NIC saturation
> affected the zookeeper client heartbeats and led to a session timeout. The
> timeline was roughly as follows:
> - Attempt to expand ISR
> - Expansion written to zookeeper (confirmed in zookeeper transaction logs)
> - Session timeout after around 13 seconds (the configured timeout is 20
>   seconds) so that lines up.
> - zkclient reconnects to zookeeper (with the same session ID) and retries
>   the write - but uses the old zkVersion. This fails because the zkVersion
>   has already been updated (above).
> - The ISR expand keeps failing after that and the only way to get out of it
>   is to bounce the broker.
> In the above code, if the zkVersion is different we should probably update
> the cached version and even retry the expansion until it succeeds.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1308) Publish jar of test utilities to Maven

2014-06-18 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036961#comment-14036961
 ] 

Jun Rao commented on KAFKA-1308:


Thanks for the patch. I am not sure if it's worthwhile to create a separate jar 
just for TestUtils. There could be other classes that people may want to use. 
It seems that we just need to publish the test jar to maven. 

Joe Stein,

Do you think you can upload the test jars for 0.8.1.1 to maven? Thanks,

> Publish jar of test utilities to Maven
> --
>
> Key: KAFKA-1308
> URL: https://issues.apache.org/jira/browse/KAFKA-1308
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1
>Reporter: Martin Kleppmann
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1308.patch
>
>
> For projects that use Kafka, and want to write tests that exercise Kafka (in 
> our case, Samza), it's useful to have access to Kafka's test utility classes 
> such as kafka.zk.EmbeddedZookeeper and kafka.utils.TestUtils. We can use 
> {{./gradlew testJar}} to build jar files that contain those classes, but as 
> far as I know, these are currently not made available in a binary release.
> At the moment, we have to check those kafka*-test.jar files into the Samza 
> repository. To avoid that, would it be possible to publish those jars of 
> tests to Maven, so that they fit into the normal dependency management?
> Or perhaps, if publishing the tests themselves is not appropriate, we could 
> move the test utilities into a separate module that is published, and make 
> the tests depend on that module?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 22496: Patch for KAFKA-1096

2014-06-18 Thread Neha Narkhede


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22496/#review46184
---



core/src/main/scala/kafka/controller/KafkaController.scala


I think this should be removed as well. Basically, the only time the broker 
needs to refresh information about the controller epoch is when it gets elected 
as the controller. Also, it needs to reset it to 0 or something once 
onControllerResignation().


- Neha Narkhede


On June 12, 2014, 4:52 a.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22496/
> ---
> 
> (Updated June 12, 2014, 4:52 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1096
> https://issues.apache.org/jira/browse/KAFKA-1096
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1096. An old controller coming out of long GC could update its epoch to 
> the latest controller's epoch.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8af48ab500779d3d851d25050e1308f5e7b588a6 
> 
> Diff: https://reviews.apache.org/r/22496/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

[jira] [Commented] (KAFKA-1382) Update zkVersion on partition state update failures

2014-06-18 Thread Neha Narkhede (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036932#comment-14036932
 ] 

Neha Narkhede commented on KAFKA-1382:
--

Thanks for the follow up patch. Pushed to trunk

> Update zkVersion on partition state update failures
> ---
>
> Key: KAFKA-1382
> URL: https://issues.apache.org/jira/browse/KAFKA-1382
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.2
>
> Attachments: KAFKA-1382.patch, KAFKA-1382_2014-05-30_21:19:21.patch, 
> KAFKA-1382_2014-05-31_15:50:25.patch, KAFKA-1382_2014-06-04_12:30:40.patch, 
> KAFKA-1382_2014-06-07_09:00:56.patch, KAFKA-1382_2014-06-09_18:23:42.patch, 
> KAFKA-1382_2014-06-11_09:37:22.patch, KAFKA-1382_2014-06-16_13:50:16.patch, 
> KAFKA-1382_2014-06-16_14:19:27.patch
>
>
> Our updateIsr code is currently:
>   private def updateIsr(newIsr: Set[Replica]) {
> debug("Updated ISR for partition [%s,%d] to %s".format(topic, 
> partitionId, newIsr.mkString(",")))
> val newLeaderAndIsr = new LeaderAndIsr(localBrokerId, leaderEpoch, 
> newIsr.map(r => r.brokerId).toList, zkVersion)
> // use the epoch of the controller that made the leadership decision, 
> instead of the current controller epoch
> val (updateSucceeded, newVersion) = 
> ZkUtils.conditionalUpdatePersistentPath(zkClient,
>   ZkUtils.getTopicPartitionLeaderAndIsrPath(topic, partitionId),
>   ZkUtils.leaderAndIsrZkData(newLeaderAndIsr, controllerEpoch), zkVersion)
> if (updateSucceeded){
>   inSyncReplicas = newIsr
>   zkVersion = newVersion
>   trace("ISR updated to [%s] and zkVersion updated to 
> [%d]".format(newIsr.mkString(","), zkVersion))
> } else {
>   info("Cached zkVersion [%d] not equal to that in zookeeper, skip 
> updating ISR".format(zkVersion))
> }
> We encountered an interesting scenario recently when a large producer fully
> saturated the broker's NIC for over an hour. The large volume of data led to
> a number of ISR shrinks (and subsequent expands). The NIC saturation
> affected the zookeeper client heartbeats and led to a session timeout. The
> timeline was roughly as follows:
> - Attempt to expand ISR
> - Expansion written to zookeeper (confirmed in zookeeper transaction logs)
> - Session timeout after around 13 seconds (the configured timeout is 20
>   seconds) so that lines up.
> - zkclient reconnects to zookeeper (with the same session ID) and retries
>   the write - but uses the old zkVersion. This fails because the zkVersion
>   has already been updated (above).
> - The ISR expand keeps failing after that and the only way to get out of it
>   is to bounce the broker.
> In the above code, if the zkVersion is different we should probably update
> the cached version and even retry the expansion until it succeeds.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21899: Patch for KAFKA-1382

2014-06-18 Thread Neha Narkhede


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21899/#review46183
---

Ship it!


Ship It!

- Neha Narkhede


On June 16, 2014, 9:19 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21899/
> ---
> 
> (Updated June 16, 2014, 9:19 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1382
> https://issues.apache.org/jira/browse/KAFKA-1382
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1382. Update zkVersion on partition state update failures. Changes as 
> per Jun's suggestions.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> a9c0465b61c7f955e9b0ab93139ca5ef628585c4 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8af48ab500779d3d851d25050e1308f5e7b588a6 
>   core/src/main/scala/kafka/controller/PartitionStateMachine.scala 
> e29e47003ab000fe5560d6f5aabcc8c38632c852 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 2f0f29d9b76d847700bb64d6d54515b6a926a253 
>   core/src/main/scala/kafka/utils/ReplicationUtils.scala 
> eb538377dc1de056703d8d96447f47ff277ecf0a 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> 1a23eb43637662879e447cbda8e84a762cdd6889 
>   core/src/test/scala/unit/kafka/utils/ReplicationUtilsTest.scala 
> f3649808e452b56df39904297a4bfe03f1a776bc 
> 
> Diff: https://reviews.apache.org/r/21899/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

Re: Review Request 22762: Fix concurrent modification exception in Sender.

2014-06-18 Thread Neha Narkhede


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22762/#review46175
---

Ship it!


Ship It!

- Neha Narkhede


On June 18, 2014, 10:43 p.m., Jay Kreps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22762/
> ---
> 
> (Updated June 18, 2014, 10:43 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1316
> https://issues.apache.org/jira/browse/KAFKA-1316
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1316 Follow-up patch for concurrent modification exception.
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> c67b947c464ec86e28fbdf35af811e5e40f22a7a 
> 
> Diff: https://reviews.apache.org/r/22762/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>

Re: Review Request 22744: Patch for KAFKA-1291

2014-06-18 Thread Neha Narkhede


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22744/#review46173
---



bin/windows/kafka-simple-consumer-perf-test.bat


We should also add kafka-consumer-perf-test.bat



bin/windows/kafka-simple-consumer-perf-test.bat


This should be SimpleConsumerPerformance


- Neha Narkhede


On June 18, 2014, 5:55 p.m., Jay Kreps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22744/
> ---
> 
> (Updated June 18, 2014, 5:55 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1291
> https://issues.apache.org/jira/browse/KAFKA-1291
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1291 Add wrapper scripts and usage information to each command.
> 
> 
> Diffs
> -
> 
>   bin/kafka-consumer-offset-checker.sh PRE-CREATION 
>   bin/kafka-mirror-maker.sh PRE-CREATION 
>   bin/kafka-replica-verification.sh PRE-CREATION 
>   bin/windows/kafka-consumer-offset-checker.bat PRE-CREATION 
>   bin/windows/kafka-consumer-perf-test.bat PRE-CREATION 
>   bin/windows/kafka-mirror-maker.bat PRE-CREATION 
>   bin/windows/kafka-preferred-replica-election.bat PRE-CREATION 
>   bin/windows/kafka-producer-perf-test.bat PRE-CREATION 
>   bin/windows/kafka-reassign-partitions.bat PRE-CREATION 
>   bin/windows/kafka-replay-log-producer.bat PRE-CREATION 
>   bin/windows/kafka-replica-verification.bat PRE-CREATION 
>   bin/windows/kafka-simple-consumer-perf-test.bat PRE-CREATION 
>   bin/windows/kafka-simple-consumer-shell.bat PRE-CREATION 
>   bin/windows/zookeeper-shell.bat PRE-CREATION 
>   core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 
> 9b3c6aeaf77db6cea75272c60a42fa45955fed5b 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 2637586af99cf8a6c4ea3a4ccb244d1e76f0f81a 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 6788c2edef91a0aef28f1cb5e3fc477d9b9f2d67 
>   core/src/main/scala/kafka/tools/ConsoleConsumer.scala 
> f6bc2f1579580574bc66ec6463eafe21ffb311f9 
>   core/src/main/scala/kafka/tools/ConsoleProducer.scala 
> f4e07d4bb4e264479e196aa805018413104acebd 
>   core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala 
> 19df757d75fdbb3ff0b434b6cb10338ff5cc32da 
>   core/src/main/scala/kafka/tools/ConsumerPerformance.scala 
> 46883493e04877f3e19892ecb73015d0335d51a1 
>   core/src/main/scala/kafka/tools/DumpLogSegments.scala 
> f0ab02a4e6dd1b945b263607c26ae22f00e6b158 
>   core/src/main/scala/kafka/tools/ExportZkOffsets.scala 
> 005231f38dd9cea60ec953a28a1ab57d3e316eed 
>   core/src/main/scala/kafka/tools/GetOffsetShell.scala 
> fba652e3716a67b04431fc46790ad255201b639f 
>   core/src/main/scala/kafka/tools/ImportZkOffsets.scala 
> c8023ee60c07b6e59906da9331bd5d764fd60cba 
>   core/src/main/scala/kafka/tools/JmxTool.scala 
> 747a675455e9aee4f7c124abe78e0454e72f0b18 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 
> e75c4f8e8070c558caa8fe9a932729d26d9807db 
>   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
> 95cfbc1d24820a10dbe03613c03c5ec12aadc4dc 
>   core/src/main/scala/kafka/tools/ReplayLogProducer.scala 
> eb71e49a8f16be1422ced60c66e43d7f99943607 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
> 91f072816418040a396a0cad26bc889f539dadd6 
>   core/src/main/scala/kafka/tools/SimpleConsumerPerformance.scala 
> 8b8c4726ab04e109402d80435459ffc691d087ca 
>   core/src/main/scala/kafka/tools/SimpleConsumerShell.scala 
> 747e07280cce72d621acbc771337b909a9b2487e 
>   core/src/main/scala/kafka/tools/StateChangeLogMerger.scala 
> 97970fb941fafb04ae351e99bf1a4ff3c8e0d322 
>   core/src/main/scala/kafka/tools/TestLogCleaning.scala 
> 595dc7ca17b6e43605285d74c4a3f4f2da2a72b3 
>   core/src/main/scala/kafka/tools/VerifyConsumerRebalance.scala 
> 92c0d1f979fba3496d1f092cc073073dcf4447e6 
>   core/src/main/scala/kafka/utils/CommandLineUtils.scala 
> c1d8ba5422f42529e6e1636157eb3a50d1bbdb44 
>   core/src/test/scala/other/kafka/TestLinearWriteSpeed.scala 
> eeb8c8856200c5b2ed5fd8a7b9b28f813eabd67d 
> 
> Diff: https://reviews.apache.org/r/22744/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>

Looking for review on two trivial patches

2014-06-18 Thread Jay Kreps

Looking for review:

https://reviews.apache.org/r/22744/

https://reviews.apache.org/r/22762

-Jay

[jira] [Updated] (KAFKA-1316) Refactor Sender

2014-06-18 Thread Jay Kreps (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1316:
-

Attachment: KAFKA-1316.patch

> Refactor Sender
> ---
>
> Key: KAFKA-1316
> URL: https://issues.apache.org/jira/browse/KAFKA-1316
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1316.patch, KAFKA-1316.patch, KAFKA-1316.patch, 
> KAFKA-1316_2014-06-03_11:15:38.patch, KAFKA-1316_2014-06-03_14:33:33.patch, 
> KAFKA-1316_2014-06-07_11:20:38.patch
>
>
> Currently most of the logic of the producer I/O thread is in Sender.java.
> However we will need to do a fair number of similar things in the new 
> consumer. Specifically:
>  - Track in-flight requests
>  - Fetch metadata
>  - Manage connection lifecycle
> It may be possible to refactor some of this into a helper class that can be 
> shared with the consumer. This will require some detailed thought.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1316) Refactor Sender

2014-06-18 Thread Jay Kreps (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036582#comment-14036582
 ] 

Jay Kreps commented on KAFKA-1316:
--

Created reviewboard https://reviews.apache.org/r/22762/
 against branch trunk

> Refactor Sender
> ---
>
> Key: KAFKA-1316
> URL: https://issues.apache.org/jira/browse/KAFKA-1316
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1316.patch, KAFKA-1316.patch, KAFKA-1316.patch, 
> KAFKA-1316_2014-06-03_11:15:38.patch, KAFKA-1316_2014-06-03_14:33:33.patch, 
> KAFKA-1316_2014-06-07_11:20:38.patch
>
>
> Currently most of the logic of the producer I/O thread is in Sender.java.
> However we will need to do a fair number of similar things in the new 
> consumer. Specifically:
>  - Track in-flight requests
>  - Fetch metadata
>  - Manage connection lifecycle
> It may be possible to refactor some of this into a helper class that can be 
> shared with the consumer. This will require some detailed thought.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 22762: Fix concurrent modification exception in Sender.

2014-06-18 Thread Jay Kreps


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22762/
---

Review request for kafka.


Bugs: KAFKA-1316
https://issues.apache.org/jira/browse/KAFKA-1316


Repository: kafka


Description
---

KAFKA-1316 Follow-up patch for concurrent modification exception.


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
c67b947c464ec86e28fbdf35af811e5e40f22a7a 

Diff: https://reviews.apache.org/r/22762/diff/


Testing
---


Thanks,

Jay Kreps

[jira] [Commented] (KAFKA-1291) Make wrapper shell scripts for important tools

2014-06-18 Thread Jay Kreps (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036189#comment-14036189
 ] 

Jay Kreps commented on KAFKA-1291:
--

Specifically it would be good for someone to sanity check that the one-line 
description is correct and would convey the necessary information to someone 
trying to figure out what that command does.

> Make wrapper shell scripts for important tools
> --
>
> Key: KAFKA-1291
> URL: https://issues.apache.org/jira/browse/KAFKA-1291
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1
>Reporter: Jay Kreps
>  Labels: newbie, usability
> Fix For: 0.8.2
>
> Attachments: KAFKA-1291.patch, KAFKA-1291.patch, KAFKA-1291.patch
>
>
> It is nice to have a proper command for the important tools just to help with 
> discoverability. I noticed that mirror maker doesn't have such a wrapper. 
> Neither does consumer offset checker. It would be good to do an audit and 
> think of any tools that should have a wrapper that don't.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (KAFKA-1497) Change producer load-balancing algorithm in MirrorMaker

2014-06-18 Thread Ivan Kunz (JIRA)

Ivan Kunz created KAFKA-1497:


 Summary: Change producer load-balancing algorithm in MirrorMaker
 Key: KAFKA-1497
 URL: https://issues.apache.org/jira/browse/KAFKA-1497
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.8.1.1
Reporter: Ivan Kunz


Currently the MirrorMaker uses the following way of spreading the load into 
configured producers :
val producerId = 
Utils.abs(java.util.Arrays.hashCode(msgAndMetadata.key)) % producers.size()

This way if the producer side of MM uses different than the default 
"partitioner.class" messages within the same partition can get re-ordered. Also 
hashCode does not produce the same results on different machines (verified by 
testing) so cannot be safely used for partitioning between distributed systems 
connected via MM (for us message order preservation within a partition is a 
critical feature).
It would be great if the code above is changed to utilize the configured 
"partitioner.class". 
Something along the lines of  :
At the initialization:
  mmpartitioner = Utils.createObject[Partitioner](config.partitionerClass, 
config.props)  
During the processing:
val producerId = 
mmpartitioner.partition(msgAndMetadata.key,producers.size())

This way the messages consumed and produced by MM can remain in the same order.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1382) Update zkVersion on partition state update failures

2014-06-18 Thread Sriharsha Chintalapani (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036082#comment-14036082
 ] 

Sriharsha Chintalapani commented on KAFKA-1382:
---

[~junrao] Can you please take a look at the latest patch and let me know if it 
looks good or not.
Thanks,
Harsha

> Update zkVersion on partition state update failures
> ---
>
> Key: KAFKA-1382
> URL: https://issues.apache.org/jira/browse/KAFKA-1382
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.2
>
> Attachments: KAFKA-1382.patch, KAFKA-1382_2014-05-30_21:19:21.patch, 
> KAFKA-1382_2014-05-31_15:50:25.patch, KAFKA-1382_2014-06-04_12:30:40.patch, 
> KAFKA-1382_2014-06-07_09:00:56.patch, KAFKA-1382_2014-06-09_18:23:42.patch, 
> KAFKA-1382_2014-06-11_09:37:22.patch, KAFKA-1382_2014-06-16_13:50:16.patch, 
> KAFKA-1382_2014-06-16_14:19:27.patch
>
>
> Our updateIsr code is currently:
>   private def updateIsr(newIsr: Set[Replica]) {
> debug("Updated ISR for partition [%s,%d] to %s".format(topic, 
> partitionId, newIsr.mkString(",")))
> val newLeaderAndIsr = new LeaderAndIsr(localBrokerId, leaderEpoch, 
> newIsr.map(r => r.brokerId).toList, zkVersion)
> // use the epoch of the controller that made the leadership decision, 
> instead of the current controller epoch
> val (updateSucceeded, newVersion) = 
> ZkUtils.conditionalUpdatePersistentPath(zkClient,
>   ZkUtils.getTopicPartitionLeaderAndIsrPath(topic, partitionId),
>   ZkUtils.leaderAndIsrZkData(newLeaderAndIsr, controllerEpoch), zkVersion)
> if (updateSucceeded){
>   inSyncReplicas = newIsr
>   zkVersion = newVersion
>   trace("ISR updated to [%s] and zkVersion updated to 
> [%d]".format(newIsr.mkString(","), zkVersion))
> } else {
>   info("Cached zkVersion [%d] not equal to that in zookeeper, skip 
> updating ISR".format(zkVersion))
> }
> We encountered an interesting scenario recently when a large producer fully
> saturated the broker's NIC for over an hour. The large volume of data led to
> a number of ISR shrinks (and subsequent expands). The NIC saturation
> affected the zookeeper client heartbeats and led to a session timeout. The
> timeline was roughly as follows:
> - Attempt to expand ISR
> - Expansion written to zookeeper (confirmed in zookeeper transaction logs)
> - Session timeout after around 13 seconds (the configured timeout is 20
>   seconds) so that lines up.
> - zkclient reconnects to zookeeper (with the same session ID) and retries
>   the write - but uses the old zkVersion. This fails because the zkVersion
>   has already been updated (above).
> - The ISR expand keeps failing after that and the only way to get out of it
>   is to bounce the broker.
> In the above code, if the zkVersion is different we should probably update
> the cached version and even retry the expansion until it succeeds.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1291) Make wrapper shell scripts for important tools

2014-06-18 Thread Jay Kreps (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036041#comment-14036041
 ] 

Jay Kreps commented on KAFKA-1291:
--

Hey [~sgeller] this is great! I updated this patch a bit:
1. I removed the shell script for a couple of tools that seemed kind of 
esoteric and seemed more likely to add confusion.
2. I made all commands print a one-line explanation of what they do when run 
without arguments. Let me know if you think this description seems reasonable.
3. I refactored a little bit of the cut-and-paste in our command line args 
definition.

This should be an easy patch to review as most changes are trivial.

> Make wrapper shell scripts for important tools
> --
>
> Key: KAFKA-1291
> URL: https://issues.apache.org/jira/browse/KAFKA-1291
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1
>Reporter: Jay Kreps
>  Labels: newbie, usability
> Fix For: 0.8.2
>
> Attachments: KAFKA-1291.patch, KAFKA-1291.patch, KAFKA-1291.patch
>
>
> It is nice to have a proper command for the important tools just to help with 
> discoverability. I noticed that mirror maker doesn't have such a wrapper. 
> Neither does consumer offset checker. It would be good to do an audit and 
> think of any tools that should have a wrapper that don't.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1291) Make wrapper shell scripts for important tools

2014-06-18 Thread Jay Kreps (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1291:
-

Attachment: KAFKA-1291.patch

> Make wrapper shell scripts for important tools
> --
>
> Key: KAFKA-1291
> URL: https://issues.apache.org/jira/browse/KAFKA-1291
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1
>Reporter: Jay Kreps
>  Labels: newbie, usability
> Fix For: 0.8.2
>
> Attachments: KAFKA-1291.patch, KAFKA-1291.patch, KAFKA-1291.patch
>
>
> It is nice to have a proper command for the important tools just to help with 
> discoverability. I noticed that mirror maker doesn't have such a wrapper. 
> Neither does consumer offset checker. It would be good to do an audit and 
> think of any tools that should have a wrapper that don't.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1291) Make wrapper shell scripts for important tools

2014-06-18 Thread Jay Kreps (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036034#comment-14036034
 ] 

Jay Kreps commented on KAFKA-1291:
--

Created reviewboard https://reviews.apache.org/r/22744/
 against branch trunk

> Make wrapper shell scripts for important tools
> --
>
> Key: KAFKA-1291
> URL: https://issues.apache.org/jira/browse/KAFKA-1291
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1
>Reporter: Jay Kreps
>  Labels: newbie, usability
> Fix For: 0.8.2
>
> Attachments: KAFKA-1291.patch, KAFKA-1291.patch, KAFKA-1291.patch
>
>
> It is nice to have a proper command for the important tools just to help with 
> discoverability. I noticed that mirror maker doesn't have such a wrapper. 
> Neither does consumer offset checker. It would be good to do an audit and 
> think of any tools that should have a wrapper that don't.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 22744: Patch for KAFKA-1291

2014-06-18 Thread Jay Kreps


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22744/
---

Review request for kafka.


Bugs: KAFKA-1291
https://issues.apache.org/jira/browse/KAFKA-1291


Repository: kafka


Description
---

KAFKA-1291 Add wrapper scripts and usage information to each command.


Diffs
-

  bin/kafka-consumer-offset-checker.sh PRE-CREATION 
  bin/kafka-mirror-maker.sh PRE-CREATION 
  bin/kafka-replica-verification.sh PRE-CREATION 
  bin/windows/kafka-consumer-offset-checker.bat PRE-CREATION 
  bin/windows/kafka-consumer-perf-test.bat PRE-CREATION 
  bin/windows/kafka-mirror-maker.bat PRE-CREATION 
  bin/windows/kafka-preferred-replica-election.bat PRE-CREATION 
  bin/windows/kafka-producer-perf-test.bat PRE-CREATION 
  bin/windows/kafka-reassign-partitions.bat PRE-CREATION 
  bin/windows/kafka-replay-log-producer.bat PRE-CREATION 
  bin/windows/kafka-replica-verification.bat PRE-CREATION 
  bin/windows/kafka-simple-consumer-perf-test.bat PRE-CREATION 
  bin/windows/kafka-simple-consumer-shell.bat PRE-CREATION 
  bin/windows/zookeeper-shell.bat PRE-CREATION 
  core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 
9b3c6aeaf77db6cea75272c60a42fa45955fed5b 
  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
2637586af99cf8a6c4ea3a4ccb244d1e76f0f81a 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
6788c2edef91a0aef28f1cb5e3fc477d9b9f2d67 
  core/src/main/scala/kafka/tools/ConsoleConsumer.scala 
f6bc2f1579580574bc66ec6463eafe21ffb311f9 
  core/src/main/scala/kafka/tools/ConsoleProducer.scala 
f4e07d4bb4e264479e196aa805018413104acebd 
  core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala 
19df757d75fdbb3ff0b434b6cb10338ff5cc32da 
  core/src/main/scala/kafka/tools/ConsumerPerformance.scala 
46883493e04877f3e19892ecb73015d0335d51a1 
  core/src/main/scala/kafka/tools/DumpLogSegments.scala 
f0ab02a4e6dd1b945b263607c26ae22f00e6b158 
  core/src/main/scala/kafka/tools/ExportZkOffsets.scala 
005231f38dd9cea60ec953a28a1ab57d3e316eed 
  core/src/main/scala/kafka/tools/GetOffsetShell.scala 
fba652e3716a67b04431fc46790ad255201b639f 
  core/src/main/scala/kafka/tools/ImportZkOffsets.scala 
c8023ee60c07b6e59906da9331bd5d764fd60cba 
  core/src/main/scala/kafka/tools/JmxTool.scala 
747a675455e9aee4f7c124abe78e0454e72f0b18 
  core/src/main/scala/kafka/tools/MirrorMaker.scala 
e75c4f8e8070c558caa8fe9a932729d26d9807db 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
95cfbc1d24820a10dbe03613c03c5ec12aadc4dc 
  core/src/main/scala/kafka/tools/ReplayLogProducer.scala 
eb71e49a8f16be1422ced60c66e43d7f99943607 
  core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
91f072816418040a396a0cad26bc889f539dadd6 
  core/src/main/scala/kafka/tools/SimpleConsumerPerformance.scala 
8b8c4726ab04e109402d80435459ffc691d087ca 
  core/src/main/scala/kafka/tools/SimpleConsumerShell.scala 
747e07280cce72d621acbc771337b909a9b2487e 
  core/src/main/scala/kafka/tools/StateChangeLogMerger.scala 
97970fb941fafb04ae351e99bf1a4ff3c8e0d322 
  core/src/main/scala/kafka/tools/TestLogCleaning.scala 
595dc7ca17b6e43605285d74c4a3f4f2da2a72b3 
  core/src/main/scala/kafka/tools/VerifyConsumerRebalance.scala 
92c0d1f979fba3496d1f092cc073073dcf4447e6 
  core/src/main/scala/kafka/utils/CommandLineUtils.scala 
c1d8ba5422f42529e6e1636157eb3a50d1bbdb44 
  core/src/test/scala/other/kafka/TestLinearWriteSpeed.scala 
eeb8c8856200c5b2ed5fd8a7b9b28f813eabd67d 

Diff: https://reviews.apache.org/r/22744/diff/


Testing
---


Thanks,

Jay Kreps

[jira] [Commented] (KAFKA-1308) Publish jar of test utilities to Maven

2014-06-18 Thread Ivan Lyutov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035969#comment-14035969
 ] 

Ivan Lyutov commented on KAFKA-1308:


+1
Patch looks nice. It might be very useful.

> Publish jar of test utilities to Maven
> --
>
> Key: KAFKA-1308
> URL: https://issues.apache.org/jira/browse/KAFKA-1308
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1
>Reporter: Martin Kleppmann
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1308.patch
>
>
> For projects that use Kafka, and want to write tests that exercise Kafka (in 
> our case, Samza), it's useful to have access to Kafka's test utility classes 
> such as kafka.zk.EmbeddedZookeeper and kafka.utils.TestUtils. We can use 
> {{./gradlew testJar}} to build jar files that contain those classes, but as 
> far as I know, these are currently not made available in a binary release.
> At the moment, we have to check those kafka*-test.jar files into the Samza 
> repository. To avoid that, would it be possible to publish those jars of 
> tests to Maven, so that they fit into the normal dependency management?
> Or perhaps, if publishing the tests themselves is not appropriate, we could 
> move the test utilities into a separate module that is published, and make 
> the tests depend on that module?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1316) Refactor Sender

2014-06-18 Thread Jay Kreps (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035911#comment-14035911
 ] 

Jay Kreps commented on KAFKA-1316:
--

Ack, this is dumb bug. Thanks! Will patch.

> Refactor Sender
> ---
>
> Key: KAFKA-1316
> URL: https://issues.apache.org/jira/browse/KAFKA-1316
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1316.patch, KAFKA-1316.patch, 
> KAFKA-1316_2014-06-03_11:15:38.patch, KAFKA-1316_2014-06-03_14:33:33.patch, 
> KAFKA-1316_2014-06-07_11:20:38.patch
>
>
> Currently most of the logic of the producer I/O thread is in Sender.java.
> However we will need to do a fair number of similar things in the new 
> consumer. Specifically:
>  - Track in-flight requests
>  - Fetch metadata
>  - Manage connection lifecycle
> It may be possible to refactor some of this into a helper class that can be 
> shared with the consumer. This will require some detailed thought.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1316) Refactor Sender

2014-06-18 Thread JIRA


[ 
https://issues.apache.org/jira/browse/KAFKA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035620#comment-14035620
 ] 

Łukasz Drumiński commented on KAFKA-1316:
-

Hello everyone,

I have a problem with new Producer from the trunk. It throws concurrent 
modification exception. Exception is thrown in Sender.java from the attached 
patch by this block of code

{code}
// remove any nodes we aren't ready to send to
for (Node node : ready) {
if (!this.client.ready(node, now))
ready.remove(node);
}
{code}

We can workaround this problem for example like this:
{code}
Set nodesNotReadyToSend = new HashSet(ready.size());
// remove any nodes we aren't ready to send to
for (Node node : ready) {
if (!this.client.ready(node, now))
nodesNotReadyToSend.add(node);
}
ready.removeAll(nodesNotReadyToSend);
{code}






> Refactor Sender
> ---
>
> Key: KAFKA-1316
> URL: https://issues.apache.org/jira/browse/KAFKA-1316
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1316.patch, KAFKA-1316.patch, 
> KAFKA-1316_2014-06-03_11:15:38.patch, KAFKA-1316_2014-06-03_14:33:33.patch, 
> KAFKA-1316_2014-06-07_11:20:38.patch
>
>
> Currently most of the logic of the producer I/O thread is in Sender.java.
> However we will need to do a fair number of similar things in the new 
> consumer. Specifically:
>  - Track in-flight requests
>  - Fetch metadata
>  - Manage connection lifecycle
> It may be possible to refactor some of this into a helper class that can be 
> shared with the consumer. This will require some detailed thought.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

45 matches

Mail list logo