date:20140721

[
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068442#comment-14068442
]

nicu marasoiu edited comment on KAFKA-1549 at 7/21/14 12:07 PM:

2. Indeed, to avoid that overhead and make it atomical (i.e. brokers and topic
metadata from the same snapshot of metadata). However the atomicity is not
required by the functionality.

Agree with both comments, and anticipated this second one :)

was (Author: nmarasoi):
2. Indeed, to avoid that overhead and make it atomical (i.e. brokers and topic
metadata from the same snapshot of metadata). However the atomicity is not
required by the functionality.

dead brokers coming in the TopicMetadataResponse

Key: KAFKA-1549
URL: https://issues.apache.org/jira/browse/KAFKA-1549
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2
Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
Attachments:
KAFKA-1549__dead_brokers_coming_in_the_TopicMetadataResponse_.patch

JunRao confirming my observation that brokers are only added to the
metadataCache, never removed: The way that we update liveBrokers in
MetadataCache.updateCache() doesn't seem right. We only add newly received
live brokers to the list. However, there could be existing brokers in that
list that are now dead. Those dead brokers shouldn't be returned to the
clients. We should probably just take the new live broker list and cache it.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse


[ 
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068442#comment-14068442
 ] 

nicu marasoiu commented on KAFKA-1549:
--

2. Indeed, to avoid that overhead and make it atomical (i.e. brokers and topic 
metadata from the same snapshot of metadata). However the atomicity is not 
required by the functionality.

 dead brokers coming in the TopicMetadataResponse
 

 Key: KAFKA-1549
 URL: https://issues.apache.org/jira/browse/KAFKA-1549
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
 Attachments: 
 KAFKA-1549__dead_brokers_coming_in_the_TopicMetadataResponse_.patch


 JunRao confirming my observation that brokers are only added to the 
 metadataCache, never removed: The way that we update liveBrokers in 
 MetadataCache.updateCache() doesn't seem right. We only add newly received 
 live brokers to the list. However, there could be existing brokers in that 
 list that are now dead. Those dead brokers shouldn't be returned to the 
 clients. We should probably just take the new live broker list and cache it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse


 [ 
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nicu marasoiu updated KAFKA-1549:
-

Attachment: (was: 
KAFKA-1549__dead_brokers_coming_in_the_TopicMetadataResponse_.patch)

 dead brokers coming in the TopicMetadataResponse
 

 Key: KAFKA-1549
 URL: https://issues.apache.org/jira/browse/KAFKA-1549
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao

 JunRao confirming my observation that brokers are only added to the 
 metadataCache, never removed: The way that we update liveBrokers in 
 MetadataCache.updateCache() doesn't seem right. We only add newly received 
 live brokers to the list. However, there could be existing brokers in that 
 list that are now dead. Those dead brokers shouldn't be returned to the 
 clients. We should probably just take the new live broker list and cache it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse

[
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068442#comment-14068442
]

nicu marasoiu edited comment on KAFKA-1549 at 7/21/14 12:09 PM:

2. Indeed, to avoid that overhead and make it atomical (i.e. brokers and topic
metadata from the same snapshot of metadata). However the atomicity is not
required by the functionality.

Agree with both comments, and anticipated this second one :) - is this locking
encapsulation useful to make it safe and easy to reason on the multithreading
behavior, or not only that?

Agree with both comments, and anticipated this second one :)

dead brokers coming in the TopicMetadataResponse

Key: KAFKA-1549
URL: https://issues.apache.org/jira/browse/KAFKA-1549
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2
Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (KAFKA-1419) cross build for scala 2.11

2014-07-21 Thread Joe Stein (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein reassigned KAFKA-1419:


Assignee: Ivan Lyutov

 cross build for scala 2.11
 --

 Key: KAFKA-1419
 URL: https://issues.apache.org/jira/browse/KAFKA-1419
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 0.8.1
Reporter: Scott Clasen
Assignee: Ivan Lyutov
Priority: Blocker
 Fix For: 0.8.2


 Please publish builds for scala 2.11, hopefully just needs a small tweak to 
 the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse


 [ 
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nicu marasoiu updated KAFKA-1549:
-

Attachment: 
kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch

Implemented point 1 and reverted for point 2, attached patch

 dead brokers coming in the TopicMetadataResponse
 

 Key: KAFKA-1549
 URL: https://issues.apache.org/jira/browse/KAFKA-1549
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
 Attachments: 
 kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch


 JunRao confirming my observation that brokers are only added to the 
 metadataCache, never removed: The way that we update liveBrokers in 
 MetadataCache.updateCache() doesn't seem right. We only add newly received 
 live brokers to the list. However, there could be existing brokers in that 
 list that are now dead. Those dead brokers shouldn't be returned to the 
 clients. We should probably just take the new live broker list and cache it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1414) Speedup broker startup after hard reset


[ 
https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068561#comment-14068561
 ] 

Jay Kreps commented on KAFKA-1414:
--

[~arhagnel] I share your concern. I just want to make sure we get some data 
first. I find these things can be a little counter-intuitive some times (for 
example earlier in the thread I thought the recovery would be cpu bound, which 
doesn't seem to be true). If data shows that 2+ threads accessing the same 
drive really hurt performance we won't do it. As Anton says in that case there 
will be nothing we can really do about the RAID case. 

 Speedup broker startup after hard reset
 ---

 Key: KAFKA-1414
 URL: https://issues.apache.org/jira/browse/KAFKA-1414
 Project: Kafka
  Issue Type: Improvement
  Components: log
Affects Versions: 0.8.2, 0.9.0, 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
 Attachments: 
 0001-KAFKA-1414-Speedup-broker-startup-after-hard-reset-a.patch, 
 KAFKA-1414-rev1.patch, KAFKA-1414-rev2.fixed.patch, KAFKA-1414-rev2.patch, 
 KAFKA-1414-rev3-interleaving.patch, KAFKA-1414-rev3.patch, freebie.patch, 
 parallel-dir-loading-0.8.patch, 
 parallel-dir-loading-trunk-fixed-threadpool.patch, 
 parallel-dir-loading-trunk-threadpool.patch, parallel-dir-loading-trunk.patch


 After hard reset due to power failure broker takes way too much time 
 recovering unflushed segments in a single thread. This could be easiliy 
 improved launching multiple threads (one per data dirrectory, assuming that 
 typically each data directory is on a dedicated drive). Localy we trie this 
 simple patch to LogManager.loadLogs and it seems to work, however I'm too new 
 to scala, so do not take it literally:
 {code}
   /**
* Recover and load all logs in the given data directories
*/
   private def loadLogs(dirs: Seq[File]) {
 val threads : Array[Thread] = new Array[Thread](dirs.size)
 var i: Int = 0
 val me = this
 for(dir - dirs) {
   val thread = new Thread( new Runnable {
 def run()
 {
   val recoveryPoints = me.recoveryPointCheckpoints(dir).read
   /* load the logs */
   val subDirs = dir.listFiles()
   if(subDirs != null) {
 val cleanShutDownFile = new File(dir, Log.CleanShutdownFile)
 if(cleanShutDownFile.exists())
   info(Found clean shutdown file. Skipping recovery for all logs 
 in data directory '%s'.format(dir.getAbsolutePath))
 for(dir - subDirs) {
   if(dir.isDirectory) {
 info(Loading log ' + dir.getName + ')
 val topicPartition = Log.parseTopicPartitionName(dir.getName)
 val config = topicConfigs.getOrElse(topicPartition.topic, 
 defaultConfig)
 val log = new Log(dir,
   config,
   recoveryPoints.getOrElse(topicPartition, 0L),
   scheduler,
   time)
 val previous = addLogWithLock(topicPartition, log)
 if(previous != null)
   throw new IllegalArgumentException(Duplicate log 
 directories found: %s, %s!.format(log.dir.getAbsolutePath, 
 previous.dir.getAbsolutePath))
   }
 }
 cleanShutDownFile.delete()
   }
 }
   })
   thread.start()
   threads(i) = thread
   i = i + 1
 }
 for(thread - threads) {
   thread.join()
 }
   }
   def addLogWithLock(topicPartition: TopicAndPartition, log: Log): Log = {
 logCreationOrDeletionLock synchronized {
   this.logs.put(topicPartition, log)
 }
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1414) Speedup broker startup after hard reset

2014-07-21 Thread Anton Karamanov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068566#comment-14068566
 ] 

Anton Karamanov commented on KAFKA-1414:


[~jkreps], we are testing different combinations of parallelization levels, 
thread counts and segment sizes of logs at the moment. We'll publish acquired 
measurements as soon as they're done.

 Speedup broker startup after hard reset
 ---

 Key: KAFKA-1414
 URL: https://issues.apache.org/jira/browse/KAFKA-1414
 Project: Kafka
  Issue Type: Improvement
  Components: log
Affects Versions: 0.8.2, 0.9.0, 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
 Attachments: 
 0001-KAFKA-1414-Speedup-broker-startup-after-hard-reset-a.patch, 
 KAFKA-1414-rev1.patch, KAFKA-1414-rev2.fixed.patch, KAFKA-1414-rev2.patch, 
 KAFKA-1414-rev3-interleaving.patch, KAFKA-1414-rev3.patch, freebie.patch, 
 parallel-dir-loading-0.8.patch, 
 parallel-dir-loading-trunk-fixed-threadpool.patch, 
 parallel-dir-loading-trunk-threadpool.patch, parallel-dir-loading-trunk.patch


 After hard reset due to power failure broker takes way too much time 
 recovering unflushed segments in a single thread. This could be easiliy 
 improved launching multiple threads (one per data dirrectory, assuming that 
 typically each data directory is on a dedicated drive). Localy we trie this 
 simple patch to LogManager.loadLogs and it seems to work, however I'm too new 
 to scala, so do not take it literally:
 {code}
   /**
* Recover and load all logs in the given data directories
*/
   private def loadLogs(dirs: Seq[File]) {
 val threads : Array[Thread] = new Array[Thread](dirs.size)
 var i: Int = 0
 val me = this
 for(dir - dirs) {
   val thread = new Thread( new Runnable {
 def run()
 {
   val recoveryPoints = me.recoveryPointCheckpoints(dir).read
   /* load the logs */
   val subDirs = dir.listFiles()
   if(subDirs != null) {
 val cleanShutDownFile = new File(dir, Log.CleanShutdownFile)
 if(cleanShutDownFile.exists())
   info(Found clean shutdown file. Skipping recovery for all logs 
 in data directory '%s'.format(dir.getAbsolutePath))
 for(dir - subDirs) {
   if(dir.isDirectory) {
 info(Loading log ' + dir.getName + ')
 val topicPartition = Log.parseTopicPartitionName(dir.getName)
 val config = topicConfigs.getOrElse(topicPartition.topic, 
 defaultConfig)
 val log = new Log(dir,
   config,
   recoveryPoints.getOrElse(topicPartition, 0L),
   scheduler,
   time)
 val previous = addLogWithLock(topicPartition, log)
 if(previous != null)
   throw new IllegalArgumentException(Duplicate log 
 directories found: %s, %s!.format(log.dir.getAbsolutePath, 
 previous.dir.getAbsolutePath))
   }
 }
 cleanShutDownFile.delete()
   }
 }
   })
   thread.start()
   threads(i) = thread
   i = i + 1
 }
 for(thread - threads) {
   thread.join()
 }
   }
   def addLogWithLock(topicPartition: TopicAndPartition, log: Log): Log = {
 logCreationOrDeletionLock synchronized {
   this.logs.put(topicPartition, log)
 }
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning


[ 
https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068575#comment-14068575
 ] 

Jay Kreps commented on KAFKA-1546:
--

I think I was being a little vague. What I was trying to say is this. When each 
fetch is serviced we check
{code}
  if(fetchedData.size  maxSize)
 this.lagBegin = System.currentTimeMillis()
  else
 this.lagBegin = -1
{code}
Then the liveness criteria is
{code}
 partitionLagging = this.lagBegin  0  System.currentTimeMillis() - 
this.lagBegin  REPLICA_LAG_TIME_MS
{code}

 Automate replica lag tuning
 ---

 Key: KAFKA-1546
 URL: https://issues.apache.org/jira/browse/KAFKA-1546
 Project: Kafka
  Issue Type: Improvement
  Components: replication
Affects Versions: 0.8.0, 0.8.1, 0.8.1.1
Reporter: Neha Narkhede
  Labels: newbie++

 Currently, there is no good way to tune the replica lag configs to 
 automatically account for high and low volume topics on the same cluster. 
 For the low-volume topic it will take a very long time to detect a lagging
 replica, and for the high-volume topic it will have false-positives.
 One approach to making this easier would be to have the configuration
 be something like replica.lag.max.ms and translate this into a number
 of messages dynamically based on the throughput of the partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1546) Automate replica lag tuning


[ 
https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068575#comment-14068575
 ] 

Jay Kreps edited comment on KAFKA-1546 at 7/21/14 2:41 PM:
---

I think I was being a little vague. What I was trying to say is this. When each 
fetch is serviced we check
{code}
  if(!fetchedData.readToEndOfLog)
 this.lagBegin = System.currentTimeMillis()
  else
 this.lagBegin = -1
{code}
Then the liveness criteria is
{code}
 partitionLagging = this.lagBegin  0  System.currentTimeMillis() - 
this.lagBegin  REPLICA_LAG_TIME_MS
{code}


was (Author: jkreps):
I think I was being a little vague. What I was trying to say is this. When each 
fetch is serviced we check
{code}
  if(fetchedData.size  maxSize)
 this.lagBegin = System.currentTimeMillis()
  else
 this.lagBegin = -1
{code}
Then the liveness criteria is
{code}
 partitionLagging = this.lagBegin  0  System.currentTimeMillis() - 
this.lagBegin  REPLICA_LAG_TIME_MS
{code}

 Automate replica lag tuning
 ---

 Key: KAFKA-1546
 URL: https://issues.apache.org/jira/browse/KAFKA-1546
 Project: Kafka
  Issue Type: Improvement
  Components: replication
Affects Versions: 0.8.0, 0.8.1, 0.8.1.1
Reporter: Neha Narkhede
  Labels: newbie++

 Currently, there is no good way to tune the replica lag configs to 
 automatically account for high and low volume topics on the same cluster. 
 For the low-volume topic it will take a very long time to detect a lagging
 replica, and for the high-volume topic it will have false-positives.
 One approach to making this easier would be to have the configuration
 be something like replica.lag.max.ms and translate this into a number
 of messages dynamically based on the throughput of the partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning


[ 
https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068606#comment-14068606
 ] 

Jun Rao commented on KAFKA-1546:


Yes, that works. So we will have to return enough info in Log.read to derive 
fetchedData.readToEndOfLog.

 Automate replica lag tuning
 ---

 Key: KAFKA-1546
 URL: https://issues.apache.org/jira/browse/KAFKA-1546
 Project: Kafka
  Issue Type: Improvement
  Components: replication
Affects Versions: 0.8.0, 0.8.1, 0.8.1.1
Reporter: Neha Narkhede
  Labels: newbie++

 Currently, there is no good way to tune the replica lag configs to 
 automatically account for high and low volume topics on the same cluster. 
 For the low-volume topic it will take a very long time to detect a lagging
 replica, and for the high-volume topic it will have false-positives.
 One approach to making this easier would be to have the configuration
 be something like replica.lag.max.ms and translate this into a number
 of messages dynamically based on the throughput of the partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23705: Addressing Jun's comments

2014-07-21 Thread Manikumar Reddy O


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23705/
---

(Updated July 21, 2014, 3:16 p.m.)


Review request for kafka.


Summary (updated)
-

Addressing  Jun's comments


Bugs: KAFKA-1192
https://issues.apache.org/jira/browse/KAFKA-1192


Repository: kafka


Description (updated)
---

Support given for custom deserialization of messages and keys


Diffs (updated)
-

  core/src/main/scala/kafka/tools/DumpLogSegments.scala 
6daf87b25a48a51aafb7dbe8d0c0371e0ea7501f 

Diff: https://reviews.apache.org/r/23705/diff/


Testing
---


Thanks,

Manikumar Reddy O

Review Request 23740: Patch for KAFKA-1462

2014-07-21 Thread Jun Rao


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23740/
---

Review request for kafka.


Bugs: KAFKA-1462
https://issues.apache.org/jira/browse/KAFKA-1462


Repository: kafka


Description
---

Exclude request id from server-side request objects


Diffs
-

  core/src/main/scala/kafka/api/GenericRequestAndHeader.scala PRE-CREATION 
  core/src/main/scala/kafka/api/GenericRequestOrResponseAndHeader.scala 
fb022e8e740ba7b8a0d855e00809d23d1fa1ad36 
  core/src/main/scala/kafka/api/HeartbeatRequestAndHeader.scala 
932418bd34051c884f732d5a87e317c7a6fca0fc 
  core/src/main/scala/kafka/api/HeartbeatResponseAndHeader.scala 
556f38dd2a5b4c53b9e510918243684808f6adf2 
  core/src/main/scala/kafka/api/JoinGroupRequestAndHeader.scala 
9aea28c2d4a628bddf2643e89d5be455abca69da 
  core/src/main/scala/kafka/api/JoinGroupResponseAndHeader.scala 
7389ae6c3cbc195466c5aa58aaf7ec0f5a722c11 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
847a36bfef4ef268823e80e028d18a7e2bf936fd 

Diff: https://reviews.apache.org/r/23740/diff/


Testing
---


Thanks,

Jun Rao

[jira] [Updated] (KAFKA-1462) Add new request and response formats for the new consumer and coordinator communication


 [ 
https://issues.apache.org/jira/browse/KAFKA-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1462:
---

Attachment: KAFKA-1462.patch

 Add new request and response formats for the new consumer and coordinator 
 communication
 ---

 Key: KAFKA-1462
 URL: https://issues.apache.org/jira/browse/KAFKA-1462
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Reporter: Guozhang Wang
Assignee: Jun Rao
 Fix For: 0.8.2

 Attachments: KAFKA-1462.patch, KAFKA-1462.patch, 
 KAFKA-1462_2014-07-16_21:39:07.patch


 We need to add the request / response formats according to the new format 
 protocol once their design is final:
 https://cwiki.apache.org/confluence/display/KAFKA
 /Kafka+0.9+Consumer+Rewrite+Design



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages


 [ 
https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1192:
---

Attachment: KAFKA-1192_2014-07-21_20:44:08.patch

 Enable DumpLogSegments tool to deserialize messages
 ---

 Key: KAFKA-1192
 URL: https://issues.apache.org/jira/browse/KAFKA-1192
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Guozhang Wang
Assignee: Manikumar Reddy
  Labels: newbie
 Attachments: KAFKA-1192.patch, KAFKA-1192_2014-07-19_13:56:18.patch, 
 KAFKA-1192_2014-07-21_20:44:08.patch


 Currently the DumpLogSegments tool reads the message payloads as strings by 
 default, which will not display the messages correctly if the messages are 
 deserialized with another class. By enablding deserialization with a 
 customized class we can use this tool to debug more issues where I need to 
 read the message content.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1462) Add new request and response formats for the new consumer and coordinator communication


[ 
https://issues.apache.org/jira/browse/KAFKA-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068613#comment-14068613
 ] 

Jun Rao commented on KAFKA-1462:


There is actually an issue with the patch. The server side objects can't use 
the full request header right now since it doesn't include the request id. Will 
submit a followup patch to address this.

 Add new request and response formats for the new consumer and coordinator 
 communication
 ---

 Key: KAFKA-1462
 URL: https://issues.apache.org/jira/browse/KAFKA-1462
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Reporter: Guozhang Wang
Assignee: Jun Rao
 Fix For: 0.8.2

 Attachments: KAFKA-1462.patch, KAFKA-1462.patch, 
 KAFKA-1462_2014-07-16_21:39:07.patch


 We need to add the request / response formats according to the new format 
 protocol once their design is final:
 https://cwiki.apache.org/confluence/display/KAFKA
 /Kafka+0.9+Consumer+Rewrite+Design



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages


[ 
https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068611#comment-14068611
 ] 

Manikumar Reddy commented on KAFKA-1192:


Updated reviewboard https://reviews.apache.org/r/23705/diff/
 against branch origin/trunk

 Enable DumpLogSegments tool to deserialize messages
 ---

 Key: KAFKA-1192
 URL: https://issues.apache.org/jira/browse/KAFKA-1192
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Guozhang Wang
Assignee: Manikumar Reddy
  Labels: newbie
 Attachments: KAFKA-1192.patch, KAFKA-1192_2014-07-19_13:56:18.patch, 
 KAFKA-1192_2014-07-21_20:44:08.patch


 Currently the DumpLogSegments tool reads the message payloads as strings by 
 default, which will not display the messages correctly if the messages are 
 deserialized with another class. By enablding deserialization with a 
 customized class we can use this tool to debug more issues where I need to 
 read the message content.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1462) Add new request and response formats for the new consumer and coordinator communication


[ 
https://issues.apache.org/jira/browse/KAFKA-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068614#comment-14068614
 ] 

Jun Rao commented on KAFKA-1462:


Created reviewboard https://reviews.apache.org/r/23740/
 against branch origin/trunk

 Add new request and response formats for the new consumer and coordinator 
 communication
 ---

 Key: KAFKA-1462
 URL: https://issues.apache.org/jira/browse/KAFKA-1462
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Reporter: Guozhang Wang
Assignee: Jun Rao
 Fix For: 0.8.2

 Attachments: KAFKA-1462.patch, KAFKA-1462.patch, 
 KAFKA-1462_2014-07-16_21:39:07.patch


 We need to add the request / response formats according to the new format 
 protocol once their design is final:
 https://cwiki.apache.org/confluence/display/KAFKA
 /Kafka+0.9+Consumer+Rewrite+Design



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages


 [ 
https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1192:
---

Attachment: (was: KAFKA-1192_2014-07-19_13:56:18.patch)

 Enable DumpLogSegments tool to deserialize messages
 ---

 Key: KAFKA-1192
 URL: https://issues.apache.org/jira/browse/KAFKA-1192
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Guozhang Wang
Assignee: Manikumar Reddy
  Labels: newbie
 Attachments: KAFKA-1192_2014-07-21_20:44:08.patch


 Currently the DumpLogSegments tool reads the message payloads as strings by 
 default, which will not display the messages correctly if the messages are 
 deserialized with another class. By enablding deserialization with a 
 customized class we can use this tool to debug more issues where I need to 
 read the message content.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages


 [ 
https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1192:
---

Attachment: (was: KAFKA-1192.patch)

 Enable DumpLogSegments tool to deserialize messages
 ---

 Key: KAFKA-1192
 URL: https://issues.apache.org/jira/browse/KAFKA-1192
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Guozhang Wang
Assignee: Manikumar Reddy
  Labels: newbie
 Attachments: KAFKA-1192_2014-07-21_20:44:08.patch


 Currently the DumpLogSegments tool reads the message payloads as strings by 
 default, which will not display the messages correctly if the messages are 
 deserialized with another class. By enablding deserialization with a 
 customized class we can use this tool to debug more issues where I need to 
 read the message content.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages


[ 
https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068623#comment-14068623
 ] 

Manikumar Reddy commented on KAFKA-1192:


Introduced new command-line parameter key-decoder-class to pass customized 
Decoder class for keys. This class should implement kafka.serializer.Decoder 
trait.

 Enable DumpLogSegments tool to deserialize messages
 ---

 Key: KAFKA-1192
 URL: https://issues.apache.org/jira/browse/KAFKA-1192
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Guozhang Wang
Assignee: Manikumar Reddy
  Labels: newbie
 Attachments: KAFKA-1192_2014-07-21_20:44:08.patch


 Currently the DumpLogSegments tool reads the message payloads as strings by 
 default, which will not display the messages correctly if the messages are 
 deserialized with another class. By enablding deserialization with a 
 customized class we can use this tool to debug more issues where I need to 
 read the message content.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1536) Change the status of the JIRA to Patch Available in the kafka-review-tool


 [ 
https://issues.apache.org/jira/browse/KAFKA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1536:
---

Assignee: Neha Narkhede  (was: Manikumar Reddy)

 Change the status of the JIRA to Patch Available in the kafka-review-tool
 ---

 Key: KAFKA-1536
 URL: https://issues.apache.org/jira/browse/KAFKA-1536
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
Assignee: Neha Narkhede
 Fix For: 0.9.0

 Attachments: KAFKA-1536.patch, KAFKA-1536.patch, 
 KAFKA-1536_2014-07-18_20:41:40.patch


 When using the kafka-review-tool to upload a patch to certain jira, the 
 status remains OPEN. It makes searching for JIRAs that needs review a bit 
 hard. Would be better to make the tool also change the status of the jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1059) Improve the patch review tool to use OAuth for JIRA access


[ 
https://issues.apache.org/jira/browse/KAFKA-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068651#comment-14068651
 ] 

Manikumar Reddy commented on KAFKA-1059:


[nehanarkhede]  I am unable to contact ASF JIRA administrators. It seems only 
committers can mail to ASF infra mailing lists. First i want to confirm the 
above steps with infra team. Then I need help on steps 1 ,2 ,3. 

 Improve the patch review tool to use OAuth for JIRA access
 --

 Key: KAFKA-1059
 URL: https://issues.apache.org/jira/browse/KAFKA-1059
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Neha Narkhede
Assignee: Manikumar Reddy
  Labels: newbie

 jira-python seems to support oauth for accessing jira. It will be nice to do 
 that instead of storing the password in clear text
 http://jira-python.readthedocs.org/en/latest/#oauth



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1059) Improve the patch review tool to use OAuth for JIRA access


[ 
https://issues.apache.org/jira/browse/KAFKA-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068651#comment-14068651
 ] 

Manikumar Reddy edited comment on KAFKA-1059 at 7/21/14 3:44 PM:
-

[~nehanarkhede]  I am unable to contact ASF JIRA administrators. It seems only 
committers can mail to ASF infra mailing lists. First i want to confirm the 
above steps with infra team. Then I need help on steps 1 ,2 ,3. 


was (Author: omkreddy):
[nehanarkhede]  I am unable to contact ASF JIRA administrators. It seems only 
committers can mail to ASF infra mailing lists. First i want to confirm the 
above steps with infra team. Then I need help on steps 1 ,2 ,3. 

 Improve the patch review tool to use OAuth for JIRA access
 --

 Key: KAFKA-1059
 URL: https://issues.apache.org/jira/browse/KAFKA-1059
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Neha Narkhede
Assignee: Manikumar Reddy
  Labels: newbie

 jira-python seems to support oauth for accessing jira. It will be nice to do 
 that instead of storing the password in clear text
 http://jira-python.readthedocs.org/en/latest/#oauth



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option


 [ 
https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1493:
---

 Priority: Blocker  (was: Major)
Affects Version/s: 0.8.2

Marking this as a blocker for 0.8.2 since it's affects the api and we want to 
do it right.

 Use a well-documented LZ4 compression format and remove redundant LZ4HC option
 --

 Key: KAFKA-1493
 URL: https://issues.apache.org/jira/browse/KAFKA-1493
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2
Reporter: James Oliver
Priority: Blocker
 Fix For: 0.8.2






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1399) Drop Scala 2.8.x support


[ 
https://issues.apache.org/jira/browse/KAFKA-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068674#comment-14068674
 ] 

Jun Rao commented on KAFKA-1399:


LinkedIn no longer depends on scala 2.8.x. If there is no other objection, we 
can remove the 2.8.x support in trunk.

 Drop Scala 2.8.x support
 

 Key: KAFKA-1399
 URL: https://issues.apache.org/jira/browse/KAFKA-1399
 Project: Kafka
  Issue Type: Task
  Components: packaging
Affects Versions: 0.8.1
Reporter: Stevo Slavic
  Labels: gradle, scala

 It's been almost 4 years since [Scala 2.8 has been 
 released|http://www.scala-lang.org/old/node/7009] and 3 years since [Scala 
 2.9 has been released|http://www.scala-lang.org/old/node/9483], so there was 
 more than plenty of time to migrate.
 Continued support of old Scala 2.8 is causing issues like 
 [this|https://issues.apache.org/jira/browse/KAFKA-1362?focusedCommentId=13970390page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13970390].



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1553) TopicMetadataResponse.brokers not read in scala producer?


[ 
https://issues.apache.org/jira/browse/KAFKA-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068708#comment-14068708
 ] 

Jun Rao commented on KAFKA-1553:


Yes, that's the behavior of the old producer. We are changing that behavior in 
the new producer.

 TopicMetadataResponse.brokers not read in scala producer?
 -

 Key: KAFKA-1553
 URL: https://issues.apache.org/jira/browse/KAFKA-1553
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.8.2
Reporter: Nicolae Marasoiu
Assignee: Jun Rao

 I searched again and I do think that topicMetadataResponse.brokers is not 
 read in Scala Producer, or I have not found where it is. (In the java 
 producer it does seem used on the other hand).
 The producer seems to only parse its static configuration (properties file on 
 key metadata.broker.list on disk stating initial broker list), and cache it 
 in BrokerPartitionInfo. It never updates that broker list cache when it 
 receives a metadata response!
 It really seems that only that statically configured metadata.broker.list is 
 ever queried for metadata, not only initially but at any time e.g. this list 
 is never refreshed with the brokers listed in the topic metadata response.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

2014-07-21 Thread Sriram Subramanian (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068731#comment-14068731
 ] 

Sriram Subramanian commented on KAFKA-1546:
---

the lagBegin does not persist across shutdowns or leader transitions. A safe 
assumption to make is that all fetchers are lagging when a node becomes a 
leader till we get the first fetch. This would ensure we don't assume there is 
no lag when a fetcher is down and a new leader is elected.

 Automate replica lag tuning
 ---

 Key: KAFKA-1546
 URL: https://issues.apache.org/jira/browse/KAFKA-1546
 Project: Kafka
  Issue Type: Improvement
  Components: replication
Affects Versions: 0.8.0, 0.8.1, 0.8.1.1
Reporter: Neha Narkhede
  Labels: newbie++

 Currently, there is no good way to tune the replica lag configs to 
 automatically account for high and low volume topics on the same cluster. 
 For the low-volume topic it will take a very long time to detect a lagging
 replica, and for the high-volume topic it will have false-positives.
 One approach to making this easier would be to have the configuration
 be something like replica.lag.max.ms and translate this into a number
 of messages dynamically based on the throughput of the partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (KAFKA-1554) Corrupt index found on clean startup

2014-07-21 Thread Alexis Midon (JIRA)

Alexis Midon created KAFKA-1554:
---

 Summary: Corrupt index found on clean startup
 Key: KAFKA-1554
 URL: https://issues.apache.org/jira/browse/KAFKA-1554
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
 Environment: ubuntu 12.04, oracle jdk 1.7
Reporter: Alexis Midon
Priority: Critical


On a clean start up, corrupted index files are found.
After investigations, it appears that some pre-allocated index files are not 
compacted correctly and the end of the file is full of zeroes.
As a result, on start up, the last relative offset is zero which yields an 
offset equal to the base offset.

The workaround is to delete all index files of size 10MB (the size of the 
pre-allocated files), and restart. Index files will be re-created.

{code}
find $your_data_directory -size 10485760c -name *.index #-delete
{code}

This is issue might be related/similar to 
https://issues.apache.org/jira/browse/KAFKA-1112

{code}
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696 INFO 
main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698 INFO 
main kafka.server.KafkaServer.info - [Kafka Server 847605514], Connecting to 
zookeeper on 
zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708 INFO 
ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
 org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:host.name=i-6b948138.inst.aws.airbnb.com
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:java.version=1.7.0_55
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:java.vendor=Oracle Corporation
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:java.io.tmpdir=/tmp
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:java.compiler=NA
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client environment:os.name=Linux
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client environment:os.arch=amd64
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:os.version=3.2.0-61-virtual
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client environment:user.name=kafka
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:user.home=/srv/kafka
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 INFO 
main org.apache.zookeeper.ZooKeeper.logEnv - Client 
environment:user.dir=/srv/kafka/kafka_2.10-0.8.1
2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,718 INFO 
main org.apache.zookeeper.ZooKeeper.init - Initiating client connection,

Review Request 23743: Force fsync in OffsetCheckpoint

2014-07-21 Thread Jay Kreps


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23743/
---

Review request for kafka.


Bugs: KAFKA-1539
https://issues.apache.org/jira/browse/KAFKA-1539


Repository: kafka


Description
---

KAFKA-1539 Fsync offset checkpoint file after writing.


Diffs
-

  core/src/main/scala/kafka/server/OffsetCheckpoint.scala 
7af2f430f9f951d4c7df965d330fdcd9b810d2d5 

Diff: https://reviews.apache.org/r/23743/diff/


Testing
---


Thanks,

Jay Kreps

[jira] [Updated] (KAFKA-1539) Due to OS caching Kafka might loose offset files which causes full reset of data


 [ 
https://issues.apache.org/jira/browse/KAFKA-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1539:
-

Attachment: KAFKA-1539.patch

 Due to OS caching Kafka might loose offset files which causes full reset of 
 data
 

 Key: KAFKA-1539
 URL: https://issues.apache.org/jira/browse/KAFKA-1539
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
 Attachments: KAFKA-1539.patch


 Seen this while testing power failure and disk failures. Due to chaching on 
 OS level (eg. XFS can cache data for 30 seconds) after failure we got offset 
 files of zero length. This dramatically slows down broker startup (it have to 
 re-check all segments) and if high watermark offsets lost it simply erases 
 all data and start recovering from other brokers (looks funny - first 
 spending 2-3 hours re-checking logs and then deleting them all due to missing 
 high watermark).
 Proposal: introduce offset files rotation. Keep two version of offset file, 
 write to oldest, read from the newest valid. In this case we would be able to 
 configure offset checkpoint time in a way that at least one file is alway 
 flushed and valid.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1539) Due to OS caching Kafka might loose offset files which causes full reset of data


[ 
https://issues.apache.org/jira/browse/KAFKA-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068831#comment-14068831
 ] 

Jay Kreps commented on KAFKA-1539:
--

Created reviewboard https://reviews.apache.org/r/23743/
 against branch trunk

 Due to OS caching Kafka might loose offset files which causes full reset of 
 data
 

 Key: KAFKA-1539
 URL: https://issues.apache.org/jira/browse/KAFKA-1539
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
 Attachments: KAFKA-1539.patch


 Seen this while testing power failure and disk failures. Due to chaching on 
 OS level (eg. XFS can cache data for 30 seconds) after failure we got offset 
 files of zero length. This dramatically slows down broker startup (it have to 
 re-check all segments) and if high watermark offsets lost it simply erases 
 all data and start recovering from other brokers (looks funny - first 
 spending 2-3 hours re-checking logs and then deleting them all due to missing 
 high watermark).
 Proposal: introduce offset files rotation. Keep two version of offset file, 
 write to oldest, read from the newest valid. In this case we would be able to 
 configure offset checkpoint time in a way that at least one file is alway 
 flushed and valid.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23743: Force fsync in OffsetCheckpoint

2014-07-21 Thread Sriram Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23743/#review48256
---

Ship it!


Ship It!

- Sriram Subramanian


On July 21, 2014, 5:23 p.m., Jay Kreps wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23743/
 ---
 
 (Updated July 21, 2014, 5:23 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1539
 https://issues.apache.org/jira/browse/KAFKA-1539
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1539 Fsync offset checkpoint file after writing.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/server/OffsetCheckpoint.scala 
 7af2f430f9f951d4c7df965d330fdcd9b810d2d5 
 
 Diff: https://reviews.apache.org/r/23743/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jay Kreps

[jira] [Commented] (KAFKA-1539) Due to OS caching Kafka might loose offset files which causes full reset of data

[
https://issues.apache.org/jira/browse/KAFKA-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068837#comment-14068837
]

Jay Kreps commented on KAFKA-1539:
--

This is a really good catch, were clearly thinking flush() meant fsync, which
is totally wrong. I uploaded a patch with your fix. If you are doing testing
with this let me know that this actually fixes the issue you saw.

Due to OS caching Kafka might loose offset files which causes full reset of
data

Key: KAFKA-1539
URL: https://issues.apache.org/jira/browse/KAFKA-1539
Project: Kafka
Issue Type: Bug
Components: log
Affects Versions: 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
Attachments: KAFKA-1539.patch

Seen this while testing power failure and disk failures. Due to chaching on
OS level (eg. XFS can cache data for 30 seconds) after failure we got offset
files of zero length. This dramatically slows down broker startup (it have to
re-check all segments) and if high watermark offsets lost it simply erases
all data and start recovering from other brokers (looks funny - first
spending 2-3 hours re-checking logs and then deleting them all due to missing
high watermark).
Proposal: introduce offset files rotation. Keep two version of offset file,
write to oldest, read from the newest valid. In this case we would be able to
configure offset checkpoint time in a way that at least one file is alway
flushed and valid.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1539) Due to OS caching Kafka might loose offset files which causes full reset of data

2014-07-21 Thread Sriram Subramanian (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068848#comment-14068848
 ] 

Sriram Subramanian commented on KAFKA-1539:
---

I had encountered the same issue in another project and had to explicitly use 
fsync to fix it.

 Due to OS caching Kafka might loose offset files which causes full reset of 
 data
 

 Key: KAFKA-1539
 URL: https://issues.apache.org/jira/browse/KAFKA-1539
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
 Attachments: KAFKA-1539.patch


 Seen this while testing power failure and disk failures. Due to chaching on 
 OS level (eg. XFS can cache data for 30 seconds) after failure we got offset 
 files of zero length. This dramatically slows down broker startup (it have to 
 re-check all segments) and if high watermark offsets lost it simply erases 
 all data and start recovering from other brokers (looks funny - first 
 spending 2-3 hours re-checking logs and then deleting them all due to missing 
 high watermark).
 Proposal: introduce offset files rotation. Keep two version of offset file, 
 write to oldest, read from the newest valid. In this case we would be able to 
 configure offset checkpoint time in a way that at least one file is alway 
 flushed and valid.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages


 [ 
https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reassigned KAFKA-1192:


Assignee: Guozhang Wang  (was: Manikumar Reddy)

Assign back to Guozhang for review.

 Enable DumpLogSegments tool to deserialize messages
 ---

 Key: KAFKA-1192
 URL: https://issues.apache.org/jira/browse/KAFKA-1192
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Guozhang Wang
Assignee: Guozhang Wang
  Labels: newbie
 Attachments: KAFKA-1192_2014-07-21_20:44:08.patch


 Currently the DumpLogSegments tool reads the message payloads as strings by 
 default, which will not display the messages correctly if the messages are 
 deserialized with another class. By enablding deserialization with a 
 customized class we can use this tool to debug more issues where I need to 
 read the message content.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23567: Patch for KAFKA-1522

2014-07-21 Thread Joel Koshy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23567/#review48259
---



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84623

Can you add these new request/responses to the request-response 
serialization/deserialization test?



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84624

Method names should ideally not start with a capital. copyTransactionToTxId 
could be a reasonable name.



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84625

Case classes have a copy method. i.e., you can instead do: 
oldTxRequest.copy(requestInfo = oldTxRequest.requestInfo.copy(txId = newTxId))

Finally, since it seems this is only used in one place in the coordinator 
and given the brevity of the above copy we can inline it there and not expose 
this at the object level.





core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84626

TxControlTypes would be clearer I think (also based on what I have seen so 
far in KAFKA-1523 rb - I think you intend this to be stored in the message key 
which it should not.)



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84627

Typo - change it to transactionRequest



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84628

Would prefer calling this txGroupId



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84629

Would prefer calling groupId txGroupId



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84630

Can we use a more generic type like Seq instead?



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84632

Why does it need to be ordered? You could just use the groupedBy function.



core/src/main/scala/kafka/api/TransactionRequest.scala
https://reviews.apache.org/r/23567/#comment84633

What does this method do? I have further comments on this in rb for 
KAFKA-1523 which I'm doing in parallel.



core/src/main/scala/kafka/api/TransactionResponse.scala
https://reviews.apache.org/r/23567/#comment84634

Would it make sense to have a per-partition error-response? E.g., after a 
prepare-commit/abort: if a transaction spans a lot of partitions and one of 
those brokers goes down the coordinatorwould only need to retry a 
commit/abort for that broker. Although the alternative is to just resend all 
the txcontrol messages to all the brokers.




core/src/main/scala/kafka/api/TxCoordinatorMetadataRequest.scala
https://reviews.apache.org/r/23567/#comment84635

txGroupId



core/src/main/scala/kafka/common/ErrorMapping.scala
https://reviews.apache.org/r/23567/#comment84636

TxCoordinatorNotAvailableCode


- Joel Koshy


On July 18, 2014, 3:12 a.m., Dong Lin wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23567/
 ---
 
 (Updated July 18, 2014, 3:12 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1522
 https://issues.apache.org/jira/browse/KAFKA-1522
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1522 Tansactional messaging request/response definitions (version 2)
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/api/RequestKeys.scala 
 fbfc9d3aeaffed4ca85902125fcc1050086835db 
   core/src/main/scala/kafka/api/TransactionRequest.scala PRE-CREATION 
   core/src/main/scala/kafka/api/TransactionResponse.scala PRE-CREATION 
   core/src/main/scala/kafka/api/TxCoordinatorMetadataRequest.scala 
 PRE-CREATION 
   core/src/main/scala/kafka/api/TxCoordinatorMetadataResponse.scala 
 PRE-CREATION 
   core/src/main/scala/kafka/common/ErrorMapping.scala 
 5559d26ba2b96059f719754a351fa4598ca8a70b 
 
 Diff: https://reviews.apache.org/r/23567/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Dong Lin

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

2014-07-21 Thread Alexis Midon (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1406#comment-1406
 ] 

Alexis Midon commented on KAFKA-1112:
-

I created https://issues.apache.org/jira/browse/KAFKA-1554.
thanks

 broker can not start itself after kafka is killed with -9
 -

 Key: KAFKA-1112
 URL: https://issues.apache.org/jira/browse/KAFKA-1112
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.8.0, 0.8.1
Reporter: Kane Kim
Assignee: Jay Kreps
Priority: Critical
 Fix For: 0.8.1

 Attachments: KAFKA-1112-v1.patch, KAFKA-1112-v2.patch, 
 KAFKA-1112-v3.patch, KAFKA-1112-v4.patch, KAFKA-1112.out


 When I kill kafka with -9, broker cannot start itself because of corrupted 
 index logs. I think kafka should try to delete/rebuild indexes itself without 
 manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Issue Comment Deleted] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse


 [ 
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nicu marasoiu updated KAFKA-1549:
-

Comment: was deleted

(was: Implemented point 1 and reverted for point 2, attached patch)

 dead brokers coming in the TopicMetadataResponse
 

 Key: KAFKA-1549
 URL: https://issues.apache.org/jira/browse/KAFKA-1549
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
 Attachments: 
 kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch


 JunRao confirming my observation that brokers are only added to the 
 metadataCache, never removed: The way that we update liveBrokers in 
 MetadataCache.updateCache() doesn't seem right. We only add newly received 
 live brokers to the list. However, there could be existing brokers in that 
 list that are now dead. Those dead brokers shouldn't be returned to the 
 clients. We should probably just take the new live broker list and cache it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse

[
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068442#comment-14068442
]

nicu marasoiu edited comment on KAFKA-1549 at 7/21/14 5:53 PM:
---

Implemented point 1 and reverted for point 2, attached patch.

2. Indeed, to avoid that overhead and make it atomical (i.e. brokers and topic
metadata from the same snapshot of metadata). However the atomicity is not
required by the functionality.
I understand that this locking encapsulation useful to make it safe and easy to
reason on the multithreading behavior, that is the reason right?

Agree with both comments, and anticipated this second one :) - is this locking
encapsulation useful to make it safe and easy to reason on the multithreading
behavior, or not only that?

dead brokers coming in the TopicMetadataResponse

Key: KAFKA-1549
URL: https://issues.apache.org/jira/browse/KAFKA-1549
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2
Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
Attachments:
kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse

[
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068442#comment-14068442
]

nicu marasoiu edited comment on KAFKA-1549 at 7/21/14 6:05 PM:
---

Implemented point 1 and reverted for point 2, attached patch.
I used implicit parameter for the rwLock, so it required to make the lock the
second argument-list to compile. I think in most cases there will be a single
implicit read write lock reference available.

was (Author: nmarasoi):
Implemented point 1 and reverted for point 2, attached patch.

dead brokers coming in the TopicMetadataResponse

Key: KAFKA-1549
URL: https://issues.apache.org/jira/browse/KAFKA-1549
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2
Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
Attachments:
kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse

[
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068442#comment-14068442
]

nicu marasoiu edited comment on KAFKA-1549 at 7/21/14 6:06 PM:
---

def inReadLock[T](fun: = T)(lock: ReadWriteLock): T = {..

was (Author: nmarasoi):
Implemented point 1 and reverted for point 2, attached patch.
I used implicit parameter for the rwLock, so it required to make the lock the
second argument-list to compile. I think in most cases there will be a single
implicit read write lock reference available.

dead brokers coming in the TopicMetadataResponse

Key: KAFKA-1549
URL: https://issues.apache.org/jira/browse/KAFKA-1549
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2
Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
Attachments:
kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1553) TopicMetadataResponse.brokers not read in scala producer?


[ 
https://issues.apache.org/jira/browse/KAFKA-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068944#comment-14068944
 ] 

nicu marasoiu commented on KAFKA-1553:
--

Hi, what does the new producer mean, is a certain branch, or an entirely 
different class which will obsolete-out the current ones?

 TopicMetadataResponse.brokers not read in scala producer?
 -

 Key: KAFKA-1553
 URL: https://issues.apache.org/jira/browse/KAFKA-1553
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.8.2
Reporter: Nicolae Marasoiu
Assignee: Jun Rao

 I searched again and I do think that topicMetadataResponse.brokers is not 
 read in Scala Producer, or I have not found where it is. (In the java 
 producer it does seem used on the other hand).
 The producer seems to only parse its static configuration (properties file on 
 key metadata.broker.list on disk stating initial broker list), and cache it 
 in BrokerPartitionInfo. It never updates that broker list cache when it 
 receives a metadata response!
 It really seems that only that statically configured metadata.broker.list is 
 ever queried for metadata, not only initially but at any time e.g. this list 
 is never refreshed with the brokers listed in the topic metadata response.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23705: Addressing Jun's comments


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23705/#review48265
---



core/src/main/scala/kafka/tools/DumpLogSegments.scala
https://reviews.apache.org/r/23705/#comment84652

Would key also be null possibly?


- Guozhang Wang


On July 21, 2014, 3:16 p.m., Manikumar Reddy O wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23705/
 ---
 
 (Updated July 21, 2014, 3:16 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1192
 https://issues.apache.org/jira/browse/KAFKA-1192
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Support given for custom deserialization of messages and keys
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/tools/DumpLogSegments.scala 
 6daf87b25a48a51aafb7dbe8d0c0371e0ea7501f 
 
 Diff: https://reviews.apache.org/r/23705/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Manikumar Reddy O

[jira] [Comment Edited] (KAFKA-1553) TopicMetadataResponse.brokers not read in scala producer?


[ 
https://issues.apache.org/jira/browse/KAFKA-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068944#comment-14068944
 ] 

nicu marasoiu edited comment on KAFKA-1553 at 7/21/14 6:18 PM:
---

Hi, closing task, what does the new producer mean, is a certain branch, or an 
entirely different class which will obsolete-out the current ones?


was (Author: nmarasoi):
Hi, what does the new producer mean, is a certain branch, or an entirely 
different class which will obsolete-out the current ones?

 TopicMetadataResponse.brokers not read in scala producer?
 -

 Key: KAFKA-1553
 URL: https://issues.apache.org/jira/browse/KAFKA-1553
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.8.2
Reporter: Nicolae Marasoiu
Assignee: Jun Rao

 I searched again and I do think that topicMetadataResponse.brokers is not 
 read in Scala Producer, or I have not found where it is. (In the java 
 producer it does seem used on the other hand).
 The producer seems to only parse its static configuration (properties file on 
 key metadata.broker.list on disk stating initial broker list), and cache it 
 in BrokerPartitionInfo. It never updates that broker list cache when it 
 receives a metadata response!
 It really seems that only that statically configured metadata.broker.list is 
 ever queried for metadata, not only initially but at any time e.g. this list 
 is never refreshed with the brokers listed in the topic metadata response.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-1553) TopicMetadataResponse.brokers not read in scala producer?


 [ 
https://issues.apache.org/jira/browse/KAFKA-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nicu marasoiu resolved KAFKA-1553.
--

Resolution: Duplicate

 TopicMetadataResponse.brokers not read in scala producer?
 -

 Key: KAFKA-1553
 URL: https://issues.apache.org/jira/browse/KAFKA-1553
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.8.2
Reporter: Nicolae Marasoiu
Assignee: Jun Rao

 I searched again and I do think that topicMetadataResponse.brokers is not 
 read in Scala Producer, or I have not found where it is. (In the java 
 producer it does seem used on the other hand).
 The producer seems to only parse its static configuration (properties file on 
 key metadata.broker.list on disk stating initial broker list), and cache it 
 in BrokerPartitionInfo. It never updates that broker list cache when it 
 receives a metadata response!
 It really seems that only that statically configured metadata.broker.list is 
 ever queried for metadata, not only initially but at any time e.g. this list 
 is never refreshed with the brokers listed in the topic metadata response.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23568: Patch for KAFKA-1523

2014-07-21 Thread Joel Koshy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review48261
---



core/src/main/scala/kafka/admin/TopicCommand.scala
https://reviews.apache.org/r/23568/#comment84637

Will need to modify this error message.



core/src/main/scala/kafka/controller/ControllerChannelManager.scala
https://reviews.apache.org/r/23568/#comment84639

See comment below in KafkaApis



core/src/main/scala/kafka/message/Message.scala
https://reviews.apache.org/r/23568/#comment84645

Should also store the txcontrol in the message header.



core/src/main/scala/kafka/server/KafkaApis.scala
https://reviews.apache.org/r/23568/#comment84646

Initially, I was thinking we could just append to local log (since we 
definitely want to avoid duplicating code) until we have the API for durable 
append (to a replicated log). That is part of refactoring KafkaApis and is 
actually blocked on KAFKA-1333 so unless you need this for working through all 
the failure cases I would suggest just doing a local append for now.



core/src/main/scala/kafka/server/KafkaApis.scala
https://reviews.apache.org/r/23568/#comment84647

Looking at this method in the other patch - this only gives the head - what 
about the other partitions?



core/src/main/scala/kafka/server/KafkaApis.scala
https://reviews.apache.org/r/23568/#comment84648

Rather than do this one partition at a time we should group them by broker.



core/src/main/scala/kafka/server/KafkaApis.scala
https://reviews.apache.org/r/23568/#comment84649

I think it is fine to use a channel manager similar to the controller 
channel manager but that is no longer specific to the controller. i.e., we 
should probably move it out to become a more generic re-usable ChannelManager 
module.

In fact, given the critical nature of controller to broker communication we 
should probably dedicate a separate channel manager entirely to transactions so 
that it doesn't interfere with the controller-broker communication.



core/src/main/scala/kafka/server/KafkaApis.scala
https://reviews.apache.org/r/23568/#comment84650

Same comments here apply as the above (wrt duplicate code)



core/src/main/scala/kafka/server/KafkaConfig.scala
https://reviews.apache.org/r/23568/#comment84654

Should remove the comment on atomic commits - that was only for the 
consumer offsets topic.



core/src/main/scala/kafka/server/TransactionManager.scala
https://reviews.apache.org/r/23568/#comment84658

We generally avoid using tuples and use case classes instead - since that 
is a lot clearer.



core/src/main/scala/kafka/server/TransactionManager.scala
https://reviews.apache.org/r/23568/#comment84655

Incorrect comment.



core/src/main/scala/kafka/server/TransactionManager.scala
https://reviews.apache.org/r/23568/#comment84667

How about we come up some other name for this - or even just 
TransactionalHW but that is a bit too wordy. Just want to avoid confusion with 
the replica HW.



core/src/main/scala/kafka/server/TransactionManager.scala
https://reviews.apache.org/r/23568/#comment84661

One issue with this approach is that every commit/abort will cause a linear 
scan of this queue - we can discuss some alternative ways to maintain the set 
of pending transactions and associated txcontrol offsets.



core/src/main/scala/kafka/server/TransactionManager.scala
https://reviews.apache.org/r/23568/#comment84662

As discussed elsewhere, txid should not be the key.


- Joel Koshy


On July 18, 2014, 3:12 a.m., Dong Lin wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23568/
 ---
 
 (Updated July 18, 2014, 3:12 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1523
 https://issues.apache.org/jira/browse/KAFKA-1523
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1523 transaction manager module (version 2)
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/TopicCommand.scala 
 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf 
   core/src/main/scala/kafka/common/Topic.scala 
 ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 8763968fbff697e4c5c98ab1274627c192a4d26a 
   core/src/main/scala/kafka/message/Message.scala 
 d2a7293c7be4022af30884330924791340acc5c1 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 0b668f230c8556fdf08654ce522a11847d0bf39b 
   core/src/main/scala/kafka/server/KafkaConfig.scala 
 ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 c22e51e0412843ec993721ad3230824c0aadd2ba 
   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
   core/src/main/scala/kafka/utils/ZkUtils.scala

[jira] Subscription: outstanding kafka patches

2014-07-21 Thread jira

Issue Subscription
Filter: outstanding kafka patches (112 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1550  Patch review tool should use git format-patch to generate patch
https://issues.apache.org/jira/browse/KAFKA-1550
KAFKA-1549  dead brokers coming in the TopicMetadataResponse
https://issues.apache.org/jira/browse/KAFKA-1549
KAFKA-1543  Changing replication factor
https://issues.apache.org/jira/browse/KAFKA-1543
KAFKA-1541   Add transactional request definitions to clients package
https://issues.apache.org/jira/browse/KAFKA-1541
KAFKA-1539  Due to OS caching Kafka might loose offset files which causes full 
reset of data
https://issues.apache.org/jira/browse/KAFKA-1539
KAFKA-1536  Change the status of the JIRA to Patch Available in the 
kafka-review-tool
https://issues.apache.org/jira/browse/KAFKA-1536
KAFKA-1533  transient unit test failure in ProducerFailureHandlingTest
https://issues.apache.org/jira/browse/KAFKA-1533
KAFKA-1528  Normalize all the line endings
https://issues.apache.org/jira/browse/KAFKA-1528
KAFKA-1526  Producer performance tool should have an option to enable 
transactions
https://issues.apache.org/jira/browse/KAFKA-1526
KAFKA-1525  DumpLogSegments should print transaction IDs
https://issues.apache.org/jira/browse/KAFKA-1525
KAFKA-1524  Implement transactional producer
https://issues.apache.org/jira/browse/KAFKA-1524
KAFKA-1523  Implement transaction manager module
https://issues.apache.org/jira/browse/KAFKA-1523
KAFKA-1522  Transactional messaging request/response definitions
https://issues.apache.org/jira/browse/KAFKA-1522
KAFKA-1509  Restart of destination broker after unreplicated partition move 
leaves partitions without leader
https://issues.apache.org/jira/browse/KAFKA-1509
KAFKA-1507  Using GetOffsetShell against non-existent topic creates the topic 
unintentionally
https://issues.apache.org/jira/browse/KAFKA-1507
KAFKA-1500  adding new consumer requests using the new protocol
https://issues.apache.org/jira/browse/KAFKA-1500
KAFKA-1498  new producer performance and bug improvements
https://issues.apache.org/jira/browse/KAFKA-1498
KAFKA-1496  Using batch message in sync producer only sends the first message 
if we use a Scala Stream as the argument 
https://issues.apache.org/jira/browse/KAFKA-1496
KAFKA-1483  Split Brain about Leader Partitions
https://issues.apache.org/jira/browse/KAFKA-1483
KAFKA-1481  Stop using dashes AND underscores as separators in MBean names
https://issues.apache.org/jira/browse/KAFKA-1481
KAFKA-1477  add authentication layer and initial JKS x509 implementation for 
brokers, producers and consumer for network communication
https://issues.apache.org/jira/browse/KAFKA-1477
KAFKA-1476  Get a list of consumer groups
https://issues.apache.org/jira/browse/KAFKA-1476
KAFKA-1475  Kafka consumer stops LeaderFinder/FetcherThreads, but application 
does not know
https://issues.apache.org/jira/browse/KAFKA-1475
KAFKA-1471  Add Producer Unit Tests for LZ4 and LZ4HC compression
https://issues.apache.org/jira/browse/KAFKA-1471
KAFKA-1468  Improve perf tests
https://issues.apache.org/jira/browse/KAFKA-1468
KAFKA-1460  NoReplicaOnlineException: No replica for partition
https://issues.apache.org/jira/browse/KAFKA-1460
KAFKA-1450  check invalid leader in a more robust way
https://issues.apache.org/jira/browse/KAFKA-1450
KAFKA-1430  Purgatory redesign
https://issues.apache.org/jira/browse/KAFKA-1430
KAFKA-1414  Speedup broker startup after hard reset
https://issues.apache.org/jira/browse/KAFKA-1414
KAFKA-1394  Ensure last segment isn't deleted on expiration when there are 
unflushed messages
https://issues.apache.org/jira/browse/KAFKA-1394
KAFKA-1372  Upgrade to Gradle 1.10
https://issues.apache.org/jira/browse/KAFKA-1372
KAFKA-1367  Broker topic metadata not kept in sync with ZooKeeper
https://issues.apache.org/jira/browse/KAFKA-1367
KAFKA-1351  String.format is very expensive in Scala
https://issues.apache.org/jira/browse/KAFKA-1351
KAFKA-1343  Kafka consumer iterator thread stalls
https://issues.apache.org/jira/browse/KAFKA-1343
KAFKA-1330  Implement subscribe(TopicPartition...partitions) in the new consumer
https://issues.apache.org/jira/browse/KAFKA-1330
KAFKA-1329  Add metadata fetch and refresh functionality to the consumer
https://issues.apache.org/jira/browse/KAFKA-1329
KAFKA-1324  Debian packaging
https://issues.apache.org/jira/browse/KAFKA-1324
KAFKA-1303  metadata request in the new producer can be delayed

[jira] [Commented] (KAFKA-1553) TopicMetadataResponse.brokers not read in scala producer?

[
https://issues.apache.org/jira/browse/KAFKA-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069141#comment-14069141
]

Jay Kreps commented on KAFKA-1553:
--

We have been working on rewriting the clients. We have completed a rewrite of
the producer and are in the middle or redoing the consumer.
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite

These are in a new top-level module clients/. The intention is that eventually
we will deprecate the existing scala clients in favor of these.

TopicMetadataResponse.brokers not read in scala producer?
-

Key: KAFKA-1553
URL: https://issues.apache.org/jira/browse/KAFKA-1553
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.8.2
Reporter: Nicolae Marasoiu
Assignee: Jun Rao

I searched again and I do think that topicMetadataResponse.brokers is not
read in Scala Producer, or I have not found where it is. (In the java
producer it does seem used on the other hand).
The producer seems to only parse its static configuration (properties file on
key metadata.broker.list on disk stating initial broker list), and cache it
in BrokerPartitionInfo. It never updates that broker list cache when it
receives a metadata response!
It really seems that only that statically configured metadata.broker.list is
ever queried for metadata, not only initially but at any time e.g. this list
is never refreshed with the brokers listed in the topic metadata response.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 23767: Fix KAFKA-1430


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23767/
---

Review request for kafka.


Bugs: KAFKA-1430
https://issues.apache.org/jira/browse/KAFKA-1430


Repository: kafka


Description
---

Rebased on KAFKA-1462: 1. LogSegment.read() will also return fetch info, even 
if the corresponding message set is empty; 2. Purgatory checking satisfactory 
in checkAndMaybeWatch synchronously, and will only return false if this thread 
successfully set the satisfactory bit to true; 3. Remove the read lock on 
Partition's reading of the leaderOpt and epoch and making them volatile instead 
since these two functions are just single read; 4. Fix some minor issues in 
TestEndToEndLatency; 5. Other minor fixes


Diffs
-

  core/src/main/scala/kafka/api/FetchResponse.scala 
d117f10f724b09d6deef0df3a138d28fc91aa13a 
  core/src/main/scala/kafka/cluster/Partition.scala 
f2ca8562f833f09d96ec4bd37efcacf69cd84b2e 
  core/src/main/scala/kafka/cluster/Replica.scala 
5e659b4a5c0256431aecc200a6b914472da9ecf3 
  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
0e64632210385ef63c2ad3445b55ac4f37a63df2 
  core/src/main/scala/kafka/log/FileMessageSet.scala 
b2652ddbe2f857028d5980e29a008b2c614694a3 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogCleaner.scala 
2faa196a4dc612bc634d5ff5f5f275d09073f13b 
  core/src/main/scala/kafka/log/LogSegment.scala 
0d6926ea105a99c9ff2cfc9ea6440f2f2d37bde8 
  core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
3b15254f32252cf824d7a292889ac7662d73ada1 
  core/src/main/scala/kafka/server/DelayedFetch.scala PRE-CREATION 
  core/src/main/scala/kafka/server/DelayedProduce.scala PRE-CREATION 
  core/src/main/scala/kafka/server/FetchDataInfo.scala PRE-CREATION 
  core/src/main/scala/kafka/server/FetchRequestPurgatory.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
  core/src/main/scala/kafka/server/LogOffsetMetadata.scala PRE-CREATION 
  core/src/main/scala/kafka/server/OffsetManager.scala 
0e22897cd1c7e45c58a61c3c468883611b19116d 
  core/src/main/scala/kafka/server/ProducerRequestPurgatory.scala PRE-CREATION 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
75ae1e161769a020a102009df416009bd6710f4a 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
6a56a772c134dbf1e70c1bfe067223009bfdbac8 
  core/src/main/scala/kafka/server/RequestKey.scala PRE-CREATION 
  core/src/main/scala/kafka/server/RequestPurgatory.scala 
3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
  core/src/main/scala/kafka/tools/TestEndToEndLatency.scala 
5f8f6bc46ae6cbbe15cec596ac99d0b377d1d7ef 
  core/src/test/scala/other/kafka/StressTestLog.scala 
8fcd068b248688c40e73117dc119fa84cceb95b3 
  core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala 
9f04bd38be639cde3e7f402845dbe6ae92e87dc2 
  core/src/test/scala/unit/kafka/log/FileMessageSetTest.scala 
cec1caecc51507ae339ebf8f3b8a028b12a1a056 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
d03d4c4ee5c28fb1fbab2af0b84003ec0fac36e3 
  core/src/test/scala/unit/kafka/log/LogSegmentTest.scala 
6b7603728ae5217565d68b92dd5349e7c6508f31 
  core/src/test/scala/unit/kafka/log/LogTest.scala 
1da1393983d4b20330e7c7f374424edd1b26f2a3 
  core/src/test/scala/unit/kafka/message/BaseMessageSetTestCases.scala 
6db245c956d2172cde916defdb0749081bf891fd 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
558a5d6765a2b2c2fd33dc75ed31873a133d12c9 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
2cd3a3faf7be2bbc22a892eec78c6e4c05545e18 
  core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
0ec120a4a953114e88c575dd6b583874371a09e3 
  core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala 
4f61f8469df99e02d6ce7aad897d10e158cca8fd 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 

Diff: https://reviews.apache.org/r/23767/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1430) Purgatory redesign

[
https://issues.apache.org/jira/browse/KAFKA-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069162#comment-14069162
]

Guozhang Wang commented on KAFKA-1430:
--

Created reviewboard https://reviews.apache.org/r/23767/
against branch origin/trunk

Purgatory redesign
--

Key: KAFKA-1430
URL: https://issues.apache.org/jira/browse/KAFKA-1430
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 0.8.2
Reporter: Jun Rao
Assignee: Guozhang Wang
Attachments: KAFKA-1430.patch, KAFKA-1430.patch, KAFKA-1430.patch,
KAFKA-1430.patch, KAFKA-1430.patch, KAFKA-1430_2014-06-05_17:07:37.patch,
KAFKA-1430_2014-06-05_17:13:13.patch, KAFKA-1430_2014-06-05_17:40:53.patch,
KAFKA-1430_2014-06-10_11:22:06.patch, KAFKA-1430_2014-06-10_11:26:02.patch,
KAFKA-1430_2014-07-11_10:59:13.patch

We have seen 2 main issues with the Purgatory.
1. There is no atomic checkAndWatch functionality. So, a client typically
first checks whether a request is satisfied or not and then register the
watcher. However, by the time the watcher is registered, the registered item
could already be satisfied. This item won't be satisfied until the next
update happens or the delayed time expires, which means the watched item
could be delayed.
2. FetchRequestPurgatory doesn't quite work. This is because the current
design tries to incrementally maintain the accumulated bytes ready for fetch.
However, this is difficult since the right time to check whether a fetch (for
regular consumer) request is satisfied is when the high watermark moves. At
that point, it's hard to figure out how many bytes we should incrementally
add to each pending fetch request.
The problem has been reported in KAFKA-1150 and KAFKA-703.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1430) Purgatory redesign

[
https://issues.apache.org/jira/browse/KAFKA-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Guozhang Wang updated KAFKA-1430:
-

Attachment: KAFKA-1430.patch

Purgatory redesign
--

--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23767: Fix KAFKA-1430


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23767/
---

(Updated July 21, 2014, 7:53 p.m.)


Review request for kafka.


Bugs: KAFKA-1430
https://issues.apache.org/jira/browse/KAFKA-1430


Repository: kafka


Description
---

Rebased on KAFKA-1462: 1. LogSegment.read() will also return fetch info, even 
if the corresponding message set is empty; 2. Purgatory checking satisfactory 
in checkAndMaybeWatch synchronously, and will only return false if this thread 
successfully set the satisfactory bit to true; 3. Remove the read lock on 
Partition's reading of the leaderOpt and epoch and making them volatile instead 
since these two functions are just single read; 4. Fix some minor issues in 
TestEndToEndLatency; 5. Other minor fixes


Diffs (updated)
-

  core/src/main/scala/kafka/api/FetchResponse.scala 
d117f10f724b09d6deef0df3a138d28fc91aa13a 
  core/src/main/scala/kafka/cluster/Partition.scala 
f2ca8562f833f09d96ec4bd37efcacf69cd84b2e 
  core/src/main/scala/kafka/cluster/Replica.scala 
5e659b4a5c0256431aecc200a6b914472da9ecf3 
  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
0e64632210385ef63c2ad3445b55ac4f37a63df2 
  core/src/main/scala/kafka/log/FileMessageSet.scala 
b2652ddbe2f857028d5980e29a008b2c614694a3 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogCleaner.scala 
2faa196a4dc612bc634d5ff5f5f275d09073f13b 
  core/src/main/scala/kafka/log/LogSegment.scala 
0d6926ea105a99c9ff2cfc9ea6440f2f2d37bde8 
  core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
3b15254f32252cf824d7a292889ac7662d73ada1 
  core/src/main/scala/kafka/server/DelayedFetch.scala PRE-CREATION 
  core/src/main/scala/kafka/server/DelayedProduce.scala PRE-CREATION 
  core/src/main/scala/kafka/server/FetchDataInfo.scala PRE-CREATION 
  core/src/main/scala/kafka/server/FetchRequestPurgatory.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
  core/src/main/scala/kafka/server/LogOffsetMetadata.scala PRE-CREATION 
  core/src/main/scala/kafka/server/OffsetManager.scala 
0e22897cd1c7e45c58a61c3c468883611b19116d 
  core/src/main/scala/kafka/server/ProducerRequestPurgatory.scala PRE-CREATION 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
75ae1e161769a020a102009df416009bd6710f4a 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
6a56a772c134dbf1e70c1bfe067223009bfdbac8 
  core/src/main/scala/kafka/server/RequestKey.scala PRE-CREATION 
  core/src/main/scala/kafka/server/RequestPurgatory.scala 
3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
  core/src/main/scala/kafka/tools/TestEndToEndLatency.scala 
5f8f6bc46ae6cbbe15cec596ac99d0b377d1d7ef 
  core/src/test/scala/other/kafka/StressTestLog.scala 
8fcd068b248688c40e73117dc119fa84cceb95b3 
  core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala 
9f04bd38be639cde3e7f402845dbe6ae92e87dc2 
  core/src/test/scala/unit/kafka/log/FileMessageSetTest.scala 
cec1caecc51507ae339ebf8f3b8a028b12a1a056 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
d03d4c4ee5c28fb1fbab2af0b84003ec0fac36e3 
  core/src/test/scala/unit/kafka/log/LogSegmentTest.scala 
6b7603728ae5217565d68b92dd5349e7c6508f31 
  core/src/test/scala/unit/kafka/log/LogTest.scala 
1da1393983d4b20330e7c7f374424edd1b26f2a3 
  core/src/test/scala/unit/kafka/message/BaseMessageSetTestCases.scala 
6db245c956d2172cde916defdb0749081bf891fd 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
558a5d6765a2b2c2fd33dc75ed31873a133d12c9 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
2cd3a3faf7be2bbc22a892eec78c6e4c05545e18 
  core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
0ec120a4a953114e88c575dd6b583874371a09e3 
  core/src/test/scala/unit/kafka/server/RequestPurgatoryTest.scala 
4f61f8469df99e02d6ce7aad897d10e158cca8fd 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 

Diff: https://reviews.apache.org/r/23767/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1430) Purgatory redesign

[
https://issues.apache.org/jira/browse/KAFKA-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069171#comment-14069171
]

Guozhang Wang commented on KAFKA-1430:
--

Updated reviewboard https://reviews.apache.org/r/23767/
against branch origin/trunk

Purgatory redesign
--

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1430) Purgatory redesign

[
https://issues.apache.org/jira/browse/KAFKA-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Guozhang Wang updated KAFKA-1430:
-

Attachment: KAFKA-1430_2014-07-21_12:53:39.patch

Purgatory redesign
--

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse


[ 
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069189#comment-14069189
 ] 

nicu marasoiu commented on KAFKA-1549:
--

added an implementation without the implicit: this required a pair of function 
definitions in each client class (to my current scala knowleadge)

 dead brokers coming in the TopicMetadataResponse
 

 Key: KAFKA-1549
 URL: https://issues.apache.org/jira/browse/KAFKA-1549
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
 Attachments: 
 kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch,
  
 kafka-1549__without_implicit__only_last_seen_alive_brokers_to_be_responded_part_of_the_top.patch


 JunRao confirming my observation that brokers are only added to the 
 metadataCache, never removed: The way that we update liveBrokers in 
 MetadataCache.updateCache() doesn't seem right. We only add newly received 
 live brokers to the list. However, there could be existing brokers in that 
 list that are now dead. Those dead brokers shouldn't be returned to the 
 clients. We should probably just take the new live broker list and cache it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse


 [ 
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nicu marasoiu updated KAFKA-1549:
-

Attachment: 
kafka-1549__without_implicit__only_last_seen_alive_brokers_to_be_responded_part_of_the_top.patch

 dead brokers coming in the TopicMetadataResponse
 

 Key: KAFKA-1549
 URL: https://issues.apache.org/jira/browse/KAFKA-1549
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: trunk
Reporter: nicu marasoiu
Assignee: Jun Rao
 Attachments: 
 kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch,
  
 kafka-1549__without_implicit__only_last_seen_alive_brokers_to_be_responded_part_of_the_top.patch


 JunRao confirming my observation that brokers are only added to the 
 metadataCache, never removed: The way that we update liveBrokers in 
 MetadataCache.updateCache() doesn't seem right. We only add newly received 
 live brokers to the list. However, there could be existing brokers in that 
 list that are now dead. Those dead brokers shouldn't be returned to the 
 clients. We should probably just take the new live broker list and cache it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23593: Fix KAFKA-1533: Address Jun's comments

2014-07-21 Thread Neha Narkhede


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23593/#review48285
---



clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
https://reviews.apache.org/r/23593/#comment84688

Could you please address my previous rb comment about the naming convention?


- Neha Narkhede


On July 16, 2014, 11:38 p.m., Guozhang Wang wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23593/
 ---
 
 (Updated July 16, 2014, 11:38 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1533
 https://issues.apache.org/jira/browse/KAFKA-1533
 
 
 Repository: kafka
 
 
 Description
 ---
 
 1. Add the metadataRefreshAttemptMS in NetworkClient for backing off; 2. 
 Refactor Producer API tests using KafkaTestHarness; 3. Change default backoff 
 time to 100ms for test utils
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 d8f9ce663ee24d2b0852c974136741280c39f8f8 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java
  4aa5b01d611631db72df47d50bbe30edb8c478db 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 15fd5bcbaf175a0f7d7cf0b142e63f705ca9b6ae 
   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
 34a7db4b4ea2b720476c2b1f22a623a997faffbc 
   core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala 
 194dd70919a5f301d3131c56594e40a0ebb27311 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 3faa884f8eb83c7c00baab416d0acfb488dc39c1 
 
 Diff: https://reviews.apache.org/r/23593/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Guozhang Wang

Re: Review Request 23593: Fix KAFKA-1533: Address Jun's comments



 On July 21, 2014, 8:24 p.m., Neha Narkhede wrote:
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java, line 81
  https://reviews.apache.org/r/23593/diff/1/?file=633954#file633954line81
 
  Could you please address my previous rb comment about the naming 
  convention?

It is already addressed in the last patch.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23593/#review48285
---


On July 16, 2014, 11:38 p.m., Guozhang Wang wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23593/
 ---
 
 (Updated July 16, 2014, 11:38 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1533
 https://issues.apache.org/jira/browse/KAFKA-1533
 
 
 Repository: kafka
 
 
 Description
 ---
 
 1. Add the metadataRefreshAttemptMS in NetworkClient for backing off; 2. 
 Refactor Producer API tests using KafkaTestHarness; 3. Change default backoff 
 time to 100ms for test utils
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 d8f9ce663ee24d2b0852c974136741280c39f8f8 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java
  4aa5b01d611631db72df47d50bbe30edb8c478db 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 15fd5bcbaf175a0f7d7cf0b142e63f705ca9b6ae 
   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
 34a7db4b4ea2b720476c2b1f22a623a997faffbc 
   core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala 
 194dd70919a5f301d3131c56594e40a0ebb27311 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 3faa884f8eb83c7c00baab416d0acfb488dc39c1 
 
 Diff: https://reviews.apache.org/r/23593/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Guozhang Wang

Re: Review Request 23697: Fix KAFKA-1533: Address Jun and Neha's comments



 On July 20, 2014, 11:32 p.m., Jun Rao wrote:
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java, lines 
  376-377
  https://reviews.apache.org/r/23697/diff/1/?file=635933#file635933line376
 
  Thinking a bit more about this. There are two possibility after adding 
  the metadata request to sends: (1) The metadata fetch succeeds. In this 
  case Metadata.lastRefreshMs will be updated and this line is not needed. 
  (2) The metadata fetch fails due to a connection issue. In this case, we 
  probably want to fetch the metadata request from another node immediately, 
  instead of backing off.
  
  So, it seems that in NetworkClient, we only need to take care of the 
  case when no node is available for metadata. In this case, we should 
  backoff. So, perhaps we should rename metadataLastUpdateAttemptMs to sth 
  like lastNoNodeMs.

Actually for the second case, the metadata fetch can fail either the selected 
node is not connected any more or it cannot accept any more requests (i.e. 
inFlightRequests.canSendMore() == false). In the latter case, since this node 
is selected as least loaded, there will be no other connected nodes that is 
available for sending also, and hence we should also back off.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23697/#review48194
---


On July 18, 2014, 10:45 p.m., Guozhang Wang wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23697/
 ---
 
 (Updated July 18, 2014, 10:45 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1533
 https://issues.apache.org/jira/browse/KAFKA-1533
 
 
 Repository: kafka
 
 
 Description
 ---
 
 1. Add the metadataRefreshAttemptMs in NetworkClient for backing off; 2. 
 Refactor Producer API tests using KafkaTestHarness; 3. Change default backoff 
 time to 100ms for test utils
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 d8f9ce663ee24d2b0852c974136741280c39f8f8 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java
  4aa5b01d611631db72df47d50bbe30edb8c478db 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 15fd5bcbaf175a0f7d7cf0b142e63f705ca9b6ae 
   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
 34a7db4b4ea2b720476c2b1f22a623a997faffbc 
   core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala 
 194dd70919a5f301d3131c56594e40a0ebb27311 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 3faa884f8eb83c7c00baab416d0acfb488dc39c1 
 
 Diff: https://reviews.apache.org/r/23697/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Guozhang Wang

[jira] [Commented] (KAFKA-1420) Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with TestUtils.createTopic in all unit tests

2014-07-21 Thread Jonathan Natkins (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069394#comment-14069394
 ] 

Jonathan Natkins commented on KAFKA-1420:
-

I was starting to work on this JIRA, but I'm hitting a small stumbling block. 
I've noticed some tests that create brokers using 
TestUtils.createBrokerConfigs() and mapping over the configs with 
TestUtils.createServer(), and other tests that create brokers using 
TestUtils.createBrokersInZk().

Where this becomes a little confusing is that both implementations of 
createTopic require a Seq[KafkaServer], but createServer() returns a 
KafkaServer via a Properties object and createBrokersInZk returns a 
Seq[Broker], and I don't see a particularly obvious way to go from a Broker to 
a KakfaServer.

Am I missing something obvious?

 Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with 
 TestUtils.createTopic in all unit tests
 --

 Key: KAFKA-1420
 URL: https://issues.apache.org/jira/browse/KAFKA-1420
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
  Labels: newbie
 Fix For: 0.8.2


 This is a follow-up JIRA from KAFKA-1389



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23697: Fix KAFKA-1533: Address Jun and Neha's comments


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23697/
---

(Updated July 21, 2014, 10:45 p.m.)


Review request for kafka.


Bugs: KAFKA-1533
https://issues.apache.org/jira/browse/KAFKA-1533


Repository: kafka


Description
---

1. Add the metadataRefreshAttemptMs in NetworkClient for backing off; 2. 
Refactor Producer API tests using KafkaTestHarness; 3. Change default backoff 
time to 100ms for test utils


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
d8f9ce663ee24d2b0852c974136741280c39f8f8 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java 
4aa5b01d611631db72df47d50bbe30edb8c478db 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
15fd5bcbaf175a0f7d7cf0b142e63f705ca9b6ae 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
34a7db4b4ea2b720476c2b1f22a623a997faffbc 
  core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala 
194dd70919a5f301d3131c56594e40a0ebb27311 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
3faa884f8eb83c7c00baab416d0acfb488dc39c1 

Diff: https://reviews.apache.org/r/23697/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1533) transient unit test failure in ProducerFailureHandlingTest


 [ 
https://issues.apache.org/jira/browse/KAFKA-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1533:
-

Attachment: KAFKA-1533_2014-07-21_15:45:58.patch

 transient unit test failure in ProducerFailureHandlingTest
 --

 Key: KAFKA-1533
 URL: https://issues.apache.org/jira/browse/KAFKA-1533
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1533.patch, KAFKA-1533.patch, KAFKA-1533.patch, 
 KAFKA-1533_2014-07-21_15:45:58.patch


 Occasionally, saw the test hang on tear down. The following is the stack 
 trace.
 Test worker prio=5 tid=7f9246956000 nid=0x10e078000 in Object.wait() 
 [10e075000]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 7f4e69578 (a org.apache.zookeeper.ClientCnxn$Packet)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1344)
 - locked 7f4e69578 (a org.apache.zookeeper.ClientCnxn$Packet)
 at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:732)
 at org.I0Itec.zkclient.ZkConnection.delete(ZkConnection.java:91)
 at org.I0Itec.zkclient.ZkClient$8.call(ZkClient.java:720)
 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
 at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:716)
 at kafka.utils.ZkUtils$.deletePath(ZkUtils.scala:416)
 at kafka.utils.ZkUtils$.deregisterBrokerInZk(ZkUtils.scala:184)
 at kafka.server.KafkaHealthcheck.shutdown(KafkaHealthcheck.scala:50)
 at 
 kafka.server.KafkaServer$$anonfun$shutdown$2.apply$mcV$sp(KafkaServer.scala:243)
 at kafka.utils.Utils$.swallow(Utils.scala:172)
 at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
 at kafka.utils.Utils$.swallowWarn(Utils.scala:45)
 at kafka.utils.Logging$class.swallow(Logging.scala:94)
 at kafka.utils.Utils$.swallow(Utils.scala:45)
 at kafka.server.KafkaServer.shutdown(KafkaServer.scala:243)
 at 
 kafka.api.ProducerFailureHandlingTest.tearDown(ProducerFailureHandlingTest.scala:90)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1533) transient unit test failure in ProducerFailureHandlingTest


[ 
https://issues.apache.org/jira/browse/KAFKA-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069462#comment-14069462
 ] 

Guozhang Wang commented on KAFKA-1533:
--

Updated reviewboard https://reviews.apache.org/r/23697/
 against branch origin/trunk

 transient unit test failure in ProducerFailureHandlingTest
 --

 Key: KAFKA-1533
 URL: https://issues.apache.org/jira/browse/KAFKA-1533
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1533.patch, KAFKA-1533.patch, KAFKA-1533.patch, 
 KAFKA-1533_2014-07-21_15:45:58.patch


 Occasionally, saw the test hang on tear down. The following is the stack 
 trace.
 Test worker prio=5 tid=7f9246956000 nid=0x10e078000 in Object.wait() 
 [10e075000]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 7f4e69578 (a org.apache.zookeeper.ClientCnxn$Packet)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1344)
 - locked 7f4e69578 (a org.apache.zookeeper.ClientCnxn$Packet)
 at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:732)
 at org.I0Itec.zkclient.ZkConnection.delete(ZkConnection.java:91)
 at org.I0Itec.zkclient.ZkClient$8.call(ZkClient.java:720)
 at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
 at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:716)
 at kafka.utils.ZkUtils$.deletePath(ZkUtils.scala:416)
 at kafka.utils.ZkUtils$.deregisterBrokerInZk(ZkUtils.scala:184)
 at kafka.server.KafkaHealthcheck.shutdown(KafkaHealthcheck.scala:50)
 at 
 kafka.server.KafkaServer$$anonfun$shutdown$2.apply$mcV$sp(KafkaServer.scala:243)
 at kafka.utils.Utils$.swallow(Utils.scala:172)
 at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
 at kafka.utils.Utils$.swallowWarn(Utils.scala:45)
 at kafka.utils.Logging$class.swallow(Logging.scala:94)
 at kafka.utils.Utils$.swallow(Utils.scala:45)
 at kafka.server.KafkaServer.shutdown(KafkaServer.scala:243)
 at 
 kafka.api.ProducerFailureHandlingTest.tearDown(ProducerFailureHandlingTest.scala:90)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-322) Remove one-off Send objects


 [ 
https://issues.apache.org/jira/browse/KAFKA-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps resolved KAFKA-322.
-

Resolution: Fixed

This was fixed a long while back, cleaning up.

 Remove one-off Send objects
 ---

 Key: KAFKA-322
 URL: https://issues.apache.org/jira/browse/KAFKA-322
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.8.0
Reporter: Jay Kreps
Assignee: Jay Kreps
Priority: Minor
  Labels: replication
 Fix For: 0.9.0


 We seem to be accumulating a bunch of unnecessary classes that implement 
 Send. I am not sure why people are doing this. Example:
 ProducerResponseSend.scala
 It is not at all clear why we would add a custom send object for each 
 request/response type. They all do the same thing. The only reason for having 
 the concept of a Send object was to allow two implementations: ByteBufferSend 
 and MessageSetSend, the later let's us abstract over the difference between a 
 normal write and a sendfile() call.
 I think we can refactory ByteBufferSend to take one or more ByteBuffers 
 instead of just one and delete all of these one-offs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23740: Patch for KAFKA-1462

2014-07-21 Thread Jay Kreps


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23740/#review48309
---

Ship it!


Ship It!

- Jay Kreps


On July 21, 2014, 3:17 p.m., Jun Rao wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23740/
 ---
 
 (Updated July 21, 2014, 3:17 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1462
 https://issues.apache.org/jira/browse/KAFKA-1462
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Exclude request id from server-side request objects
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/api/GenericRequestAndHeader.scala PRE-CREATION 
   core/src/main/scala/kafka/api/GenericRequestOrResponseAndHeader.scala 
 fb022e8e740ba7b8a0d855e00809d23d1fa1ad36 
   core/src/main/scala/kafka/api/HeartbeatRequestAndHeader.scala 
 932418bd34051c884f732d5a87e317c7a6fca0fc 
   core/src/main/scala/kafka/api/HeartbeatResponseAndHeader.scala 
 556f38dd2a5b4c53b9e510918243684808f6adf2 
   core/src/main/scala/kafka/api/JoinGroupRequestAndHeader.scala 
 9aea28c2d4a628bddf2643e89d5be455abca69da 
   core/src/main/scala/kafka/api/JoinGroupResponseAndHeader.scala 
 7389ae6c3cbc195466c5aa58aaf7ec0f5a722c11 
   core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
 847a36bfef4ef268823e80e028d18a7e2bf936fd 
 
 Diff: https://reviews.apache.org/r/23740/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jun Rao

[jira] [Commented] (KAFKA-1420) Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with TestUtils.createTopic in all unit tests


[ 
https://issues.apache.org/jira/browse/KAFKA-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069473#comment-14069473
 ] 

Guozhang Wang commented on KAFKA-1420:
--

Hi Jonathan,

TestUtils.createBrokerConfigs() is usually used when the test class is 
inheriting from KafkaServerTestHarness, which already handles server creation 
and shutdown at setUp() and tearDown() time; TestUtils.createServer() should 
then be used otherwise, i.e. when a broker is just needed to be created on the 
fly.

TestUtils.createBrokersInZk() is different, though, in that it does not 
actually create a running server, but just create the registration znode in ZK, 
the Broker object is just a placeholder for the broker metadata, like broker 
id, address, etc. They should only be used when we just need to test some 
ZK-based utilities but do not necessarily need to really create a running 
server.

 Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with 
 TestUtils.createTopic in all unit tests
 --

 Key: KAFKA-1420
 URL: https://issues.apache.org/jira/browse/KAFKA-1420
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
  Labels: newbie
 Fix For: 0.8.2


 This is a follow-up JIRA from KAFKA-1389



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23740: Patch for KAFKA-1462


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23740/#review48315
---

Ship it!


Ship It!

- Guozhang Wang


On July 21, 2014, 3:17 p.m., Jun Rao wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23740/
 ---
 
 (Updated July 21, 2014, 3:17 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1462
 https://issues.apache.org/jira/browse/KAFKA-1462
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Exclude request id from server-side request objects
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/api/GenericRequestAndHeader.scala PRE-CREATION 
   core/src/main/scala/kafka/api/GenericRequestOrResponseAndHeader.scala 
 fb022e8e740ba7b8a0d855e00809d23d1fa1ad36 
   core/src/main/scala/kafka/api/HeartbeatRequestAndHeader.scala 
 932418bd34051c884f732d5a87e317c7a6fca0fc 
   core/src/main/scala/kafka/api/HeartbeatResponseAndHeader.scala 
 556f38dd2a5b4c53b9e510918243684808f6adf2 
   core/src/main/scala/kafka/api/JoinGroupRequestAndHeader.scala 
 9aea28c2d4a628bddf2643e89d5be455abca69da 
   core/src/main/scala/kafka/api/JoinGroupResponseAndHeader.scala 
 7389ae6c3cbc195466c5aa58aaf7ec0f5a722c11 
   core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
 847a36bfef4ef268823e80e028d18a7e2bf936fd 
 
 Diff: https://reviews.apache.org/r/23740/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jun Rao

[jira] [Commented] (KAFKA-1537) add request.required.acks=-2 to require acks from all replicas


[ 
https://issues.apache.org/jira/browse/KAFKA-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069509#comment-14069509
 ] 

Daniel Compton commented on KAFKA-1537:
---

Adding to that, you will have less availability (though more durability) than 
running a single broker, as you now have 3 independent nodes that can fail, any 
of which will stop further writes. More information on this is in the 
Discussion session on http://aphyr.com/posts/293-call-me-maybe-kafka

 add request.required.acks=-2 to require acks from all replicas
 --

 Key: KAFKA-1537
 URL: https://issues.apache.org/jira/browse/KAFKA-1537
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.8.1.1
Reporter: Jiang Wu
Priority: Minor

 The current options for the producer parameter request.required.acks include 
 0, positive numbers and -1. -1 means the acks from replicas in ISR.
 In a stress test, it's found that, if request.required.acks=-1 and the leader 
 is receiving at high speed, then the followers will fail out of ISR. In this 
 case, request.required.acks=-1 is equivelant to request.required.acks=1 
 because only the leader is in ISR.
 It would be desirable to add request.required.acks=-2 to require acks from 
 all replicas.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23442: Updated WIP simple consumer

2014-07-21 Thread Jay Kreps


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23442/
---

(Updated July 21, 2014, 11:44 p.m.)


Review request for kafka.


Summary (updated)
-

Updated WIP simple consumer


Bugs: KAFKA-1330
https://issues.apache.org/jira/browse/KAFKA-1330


Repository: kafka


Description (updated)
---

KAFKA-1330 Draft version of the new consumer.


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
29658d4a15f112dc0af5ce517eaab93e6f00134b 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
d8f9ce663ee24d2b0852c974136741280c39f8f8 
  clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 
227f5646ee708af1b861c15237eda2140cfd4900 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
46efc0c8483acacf42b2984ac3f3b9e0a4566187 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecord.java 
436d8a479166eda29f2672b50fc99f288bbe3fa9 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
fe93afa24fc20b03830f1d190a276041d15bd3b9 
  clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java 
c3aad3b4d6b677f759583f309061193f2f109250 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java 
4aa5b01d611631db72df47d50bbe30edb8c478db 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
8ebe7ed82c9384b71ce0cc3ddbef2c2325363ab9 
  clients/src/main/java/org/apache/kafka/common/Cluster.java 
d3299b944062d96852452de455902659ad8af757 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
b68bbf00ab8eba6c5867d346c91188142593ca6e 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
93f2f1c7b229205fc846b4e8bba527dd15355eb0 
  clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java 
444e69e7c95d5ffad19896fff0ab15cb4f5c9b4e 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
040e5b91005edb8f015afdfa76fd94e0bf3cb4ca 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
50af60198a3f20933d0e8cf89c3b95d89ee73f35 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
aae8d4a1e98279470587d397cc779a9baf6fee6c 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
1a55242e9399fa4669630b55110d530f954e1279 
  clients/src/test/java/org/apache/kafka/clients/producer/PartitionerTest.java 
f06e28ce21e80c1265258ad3ac7900b99e61493d 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
5c5e3d40819e41cab7b52a0eeaee5f2e7317b7b3 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
d61de529173314c044261ad9662bec735d67e97f 
  core/src/test/scala/integration/kafka/api/ConsumerTest.scala PRE-CREATION 
  core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala 
194dd70919a5f301d3131c56594e40a0ebb27311 
  core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala 
9f04bd38be639cde3e7f402845dbe6ae92e87dc2 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
3faa884f8eb83c7c00baab416d0acfb488dc39c1 

Diff: https://reviews.apache.org/r/23442/diff/


Testing
---


Thanks,

Jay Kreps

[jira] [Updated] (KAFKA-1330) Implement subscribe(TopicPartition...partitions) in the new consumer


 [ 
https://issues.apache.org/jira/browse/KAFKA-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1330:
-

Attachment: KAFKA-1330_2014-07-21_16:44:23.patch

 Implement subscribe(TopicPartition...partitions) in the new consumer
 

 Key: KAFKA-1330
 URL: https://issues.apache.org/jira/browse/KAFKA-1330
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Affects Versions: 0.9.0
Reporter: Neha Narkhede
Assignee: Jay Kreps
 Attachments: KAFKA-1330.patch, KAFKA-1330_2014-07-21_16:44:23.patch


 This involves adding basic fetch functionality (equivalent to SimpleConsumer) 
 to the new consumer. Effectively implementing 
 subscribe(TopicPartition...partitions) and 
 unsubscribe(TopicPartition...partitions).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1330) Implement subscribe(TopicPartition...partitions) in the new consumer


[ 
https://issues.apache.org/jira/browse/KAFKA-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069525#comment-14069525
 ] 

Jay Kreps commented on KAFKA-1330:
--

Updated reviewboard https://reviews.apache.org/r/23442/
 against branch trunk

 Implement subscribe(TopicPartition...partitions) in the new consumer
 

 Key: KAFKA-1330
 URL: https://issues.apache.org/jira/browse/KAFKA-1330
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Affects Versions: 0.9.0
Reporter: Neha Narkhede
Assignee: Jay Kreps
 Attachments: KAFKA-1330.patch, KAFKA-1330_2014-07-21_16:44:23.patch


 This involves adding basic fetch functionality (equivalent to SimpleConsumer) 
 to the new consumer. Effectively implementing 
 subscribe(TopicPartition...partitions) and 
 unsubscribe(TopicPartition...partitions).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-1539) Due to OS caching Kafka might loose offset files which causes full reset of data


 [ 
https://issues.apache.org/jira/browse/KAFKA-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps resolved KAFKA-1539.
--

Resolution: Fixed

I'm checking this in since it seems to fix a clear problem, but [~arhagnel] it 
would still be good to get confirmation that the problem you were producing is 
fixed by this.

 Due to OS caching Kafka might loose offset files which causes full reset of 
 data
 

 Key: KAFKA-1539
 URL: https://issues.apache.org/jira/browse/KAFKA-1539
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
 Attachments: KAFKA-1539.patch


 Seen this while testing power failure and disk failures. Due to chaching on 
 OS level (eg. XFS can cache data for 30 seconds) after failure we got offset 
 files of zero length. This dramatically slows down broker startup (it have to 
 re-check all segments) and if high watermark offsets lost it simply erases 
 all data and start recovering from other brokers (looks funny - first 
 spending 2-3 hours re-checking logs and then deleting them all due to missing 
 high watermark).
 Proposal: introduce offset files rotation. Keep two version of offset file, 
 write to oldest, read from the newest valid. In this case we would be able to 
 configure offset checkpoint time in a way that at least one file is alway 
 flushed and valid.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23655: Patch for KAFKA-687


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23655/#review48319
---



core/src/main/scala/kafka/consumer/PartitionAllocator.scala
https://reviews.apache.org/r/23655/#comment84791

will be within



core/src/main/scala/kafka/consumer/PartitionAllocator.scala
https://reviews.apache.org/r/23655/#comment84795

Should this be RoundRobin, and the RoundRobin be Symmetric?



core/src/main/scala/kafka/consumer/PartitionAllocator.scala
https://reviews.apache.org/r/23655/#comment84794

Is String  well defined in all JVMs? Shall we use str1.compareTo(str2)? 



core/src/main/scala/kafka/consumer/PartitionAllocator.scala
https://reviews.apache.org/r/23655/#comment84796

Comments for this logic?


- Guozhang Wang


On July 18, 2014, 10:57 p.m., Joel Koshy wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23655/
 ---
 
 (Updated July 18, 2014, 10:57 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-687
 https://issues.apache.org/jira/browse/KAFKA-687
 
 
 Repository: kafka
 
 
 Description
 ---
 
 The updated diff contains the mbeans for ownership counts.
 The comments in the code and the summary are pretty self-explanatory.
 
 Things to think about:
 * Naming - do symmetric/range/roundrobin make sense?
 * The comments briefly summarize why we needed a separate symmetric mode but 
 let me know if that is unclear.
 * Rebalance time will be slightly higher - I have not measured (will do that)
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/consumer/ConsumerConfig.scala 
 1cf2f62ba02e4aa66bfa7575865e5d57baf82212 
   core/src/main/scala/kafka/consumer/PartitionAllocator.scala PRE-CREATION 
   core/src/main/scala/kafka/consumer/TopicCount.scala 
 c79311097c5bd6718cb6a7fc403f804a1a939353 
   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
 65f518d47c7555c42c4bff39c211814831f4b8b6 
   core/src/main/scala/kafka/metrics/KafkaMetricsGroup.scala 
 a20ab90165cc7ebb1cf44078efe23a53938c8df6 
   core/src/main/scala/kafka/utils/ZkUtils.scala 
 dcdc1ce2b02c996294e19cf480736106aaf29511 
   core/src/test/scala/unit/kafka/consumer/PartitionAllocatorTest.scala 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/23655/diff/
 
 
 Testing
 ---
 
 * I did the unit tests (including the new one) as well as mirror maker system 
 test suite with roundrobin. While this is being reviewed I will run the 
 system tests with symmetric
 
 
 Thanks,
 
 Joel Koshy

[jira] [Commented] (KAFKA-1537) add request.required.acks=-2 to require acks from all replicas

2014-07-21 Thread Jiang Wu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069607#comment-14069607
 ] 

Jiang Wu commented on KAFKA-1537:
-

I agree that adding acks=-2 to require acks from all replicas is not necessary; 
mainly because users can do so by setting acks=num_of_replicas.





 add request.required.acks=-2 to require acks from all replicas
 --

 Key: KAFKA-1537
 URL: https://issues.apache.org/jira/browse/KAFKA-1537
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.8.1.1
Reporter: Jiang Wu
Priority: Minor

 The current options for the producer parameter request.required.acks include 
 0, positive numbers and -1. -1 means the acks from replicas in ISR.
 In a stress test, it's found that, if request.required.acks=-1 and the leader 
 is receiving at high speed, then the followers will fail out of ISR. In this 
 case, request.required.acks=-1 is equivelant to request.required.acks=1 
 because only the leader is in ISR.
 It would be desirable to add request.required.acks=-2 to require acks from 
 all replicas.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-1537) add request.required.acks=-2 to require acks from all replicas

2014-07-21 Thread Jiang Wu (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiang Wu resolved KAFKA-1537.
-

Resolution: Not a Problem

 add request.required.acks=-2 to require acks from all replicas
 --

 Key: KAFKA-1537
 URL: https://issues.apache.org/jira/browse/KAFKA-1537
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.8.1.1
Reporter: Jiang Wu
Priority: Minor

 The current options for the producer parameter request.required.acks include 
 0, positive numbers and -1. -1 means the acks from replicas in ISR.
 In a stress test, it's found that, if request.required.acks=-1 and the leader 
 is receiving at high speed, then the followers will fail out of ISR. In this 
 case, request.required.acks=-1 is equivelant to request.required.acks=1 
 because only the leader is in ISR.
 It would be desirable to add request.required.acks=-2 to require acks from 
 all replicas.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1420) Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with TestUtils.createTopic in all unit tests

2014-07-21 Thread Jonathan Natkins (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069626#comment-14069626
 ] 

Jonathan Natkins commented on KAFKA-1420:
-

Thanks for the context, Guozhang. So in the cases where createBrokersInZk is 
used, should AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK still be 
used? In particular, I encountered this in 
AdminTest.testManualReplicaAssignment.

 Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with 
 TestUtils.createTopic in all unit tests
 --

 Key: KAFKA-1420
 URL: https://issues.apache.org/jira/browse/KAFKA-1420
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
  Labels: newbie
 Fix For: 0.8.2


 This is a follow-up JIRA from KAFKA-1389



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1517) Messages is a required argument to Producer Performance Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Compton updated KAFKA-1517:
--

Status: Open  (was: Patch Available)

 Messages is a required argument to Producer Performance Test
 

 Key: KAFKA-1517
 URL: https://issues.apache.org/jira/browse/KAFKA-1517
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Daniel Compton
Priority: Trivial
  Labels: newbie

 When running the producer performance test without providing a messages 
 argument, you get an error:
 {noformat}
 $bin/kafka-producer-perf-test.sh --topics mirrormirror --broker-list 
 kafka-dc21:9092
 Missing required argument [messages]
 Option  Description
 --  ---
 ..
 --messages Long: countThe number of messages to send or
   consume (default:
   9223372036854775807)
 {noformat}
 However the [shell command 
 documentation|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/perf/src/main/scala/kafka/perf/PerfConfig.scala#L25]
  doesn't say that this is required and implies that 
 [2^63-1|http://en.wikipedia.org/wiki/9223372036854775807] (Long.MaxValue) 
 messages will be sent. It should probably look like the 
 [ConsoleProducer|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/core/src/main/scala/kafka/producer/ConsoleProducer.scala#L32]
  and prefix the documentation with REQUIRED. Or should we make this a 
 non-required argument and set the default value to something sane like 
 100,000 messages.
 Which option is preferable for this?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1517) Messages is a required argument to Producer Performance Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Compton updated KAFKA-1517:
--

Status: Patch Available  (was: Open)

 Messages is a required argument to Producer Performance Test
 

 Key: KAFKA-1517
 URL: https://issues.apache.org/jira/browse/KAFKA-1517
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Daniel Compton
Priority: Trivial
  Labels: newbie

 When running the producer performance test without providing a messages 
 argument, you get an error:
 {noformat}
 $bin/kafka-producer-perf-test.sh --topics mirrormirror --broker-list 
 kafka-dc21:9092
 Missing required argument [messages]
 Option  Description
 --  ---
 ..
 --messages Long: countThe number of messages to send or
   consume (default:
   9223372036854775807)
 {noformat}
 However the [shell command 
 documentation|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/perf/src/main/scala/kafka/perf/PerfConfig.scala#L25]
  doesn't say that this is required and implies that 
 [2^63-1|http://en.wikipedia.org/wiki/9223372036854775807] (Long.MaxValue) 
 messages will be sent. It should probably look like the 
 [ConsoleProducer|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/core/src/main/scala/kafka/producer/ConsoleProducer.scala#L32]
  and prefix the documentation with REQUIRED. Or should we make this a 
 non-required argument and set the default value to something sane like 
 100,000 messages.
 Which option is preferable for this?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Issue Comment Deleted] (KAFKA-1517) Messages is a required argument to Producer Performance Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Compton updated KAFKA-1517:
--

Comment: was deleted

(was: From 3cb6dd13ab487b72ff73f375617a372360418708 Mon Sep 17 00:00:00 2001
From: Daniel Compton d...@danielcompton.net
Date: Tue, 22 Jul 2014 13:10:19 +1200
Subject: [PATCH] KAFKA-1517; Make messages a required argument

---
 core/src/main/scala/kafka/tools/PerfConfig.scala | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/core/src/main/scala/kafka/tools/PerfConfig.scala 
b/core/src/main/scala/kafka/tools/PerfConfig.scala
index 129cc01..517a2cd 100644
--- a/core/src/main/scala/kafka/tools/PerfConfig.scala
+++ b/core/src/main/scala/kafka/tools/PerfConfig.scala
@@ -22,11 +22,10 @@ import joptsimple.OptionParser

 class PerfConfig(args: Array[String]) {
   val parser = new OptionParser
-  val numMessagesOpt = parser.accepts(messages, The number of messages to 
send or consume)
+  val numMessagesOpt = parser.accepts(messages, REQUIRED: The number of 
messages to send or consume)
 .withRequiredArg
 .describedAs(count)
 .ofType(classOf[java.lang.Long])
-.defaultsTo(Long.MaxValue)
   val reportingIntervalOpt = parser.accepts(reporting-interval, Interval at 
which to print progress info.)
 .withRequiredArg
 .describedAs(size)
--
1.9.2)

 Messages is a required argument to Producer Performance Test
 

 Key: KAFKA-1517
 URL: https://issues.apache.org/jira/browse/KAFKA-1517
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Daniel Compton
Priority: Trivial
  Labels: newbie

 When running the producer performance test without providing a messages 
 argument, you get an error:
 {noformat}
 $bin/kafka-producer-perf-test.sh --topics mirrormirror --broker-list 
 kafka-dc21:9092
 Missing required argument [messages]
 Option  Description
 --  ---
 ..
 --messages Long: countThe number of messages to send or
   consume (default:
   9223372036854775807)
 {noformat}
 However the [shell command 
 documentation|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/perf/src/main/scala/kafka/perf/PerfConfig.scala#L25]
  doesn't say that this is required and implies that 
 [2^63-1|http://en.wikipedia.org/wiki/9223372036854775807] (Long.MaxValue) 
 messages will be sent. It should probably look like the 
 [ConsoleProducer|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/core/src/main/scala/kafka/producer/ConsoleProducer.scala#L32]
  and prefix the documentation with REQUIRED. Or should we make this a 
 non-required argument and set the default value to something sane like 
 100,000 messages.
 Which option is preferable for this?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1517) Messages is a required argument to Producer Performance Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Compton updated KAFKA-1517:
--

Status: Patch Available  (was: Open)

From 3cb6dd13ab487b72ff73f375617a372360418708 Mon Sep 17 00:00:00 2001
From: Daniel Compton d...@danielcompton.net
Date: Tue, 22 Jul 2014 13:10:19 +1200
Subject: [PATCH] KAFKA-1517; Make messages a required argument

---
 core/src/main/scala/kafka/tools/PerfConfig.scala | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/core/src/main/scala/kafka/tools/PerfConfig.scala 
b/core/src/main/scala/kafka/tools/PerfConfig.scala
index 129cc01..517a2cd 100644
--- a/core/src/main/scala/kafka/tools/PerfConfig.scala
+++ b/core/src/main/scala/kafka/tools/PerfConfig.scala
@@ -22,11 +22,10 @@ import joptsimple.OptionParser

 class PerfConfig(args: Array[String]) {
   val parser = new OptionParser
-  val numMessagesOpt = parser.accepts(messages, The number of messages to 
send or consume)
+  val numMessagesOpt = parser.accepts(messages, REQUIRED: The number of 
messages to send or consume)
 .withRequiredArg
 .describedAs(count)
 .ofType(classOf[java.lang.Long])
-.defaultsTo(Long.MaxValue)
   val reportingIntervalOpt = parser.accepts(reporting-interval, Interval at 
which to print progress info.)
 .withRequiredArg
 .describedAs(size)
--
1.9.2

 Messages is a required argument to Producer Performance Test
 

 Key: KAFKA-1517
 URL: https://issues.apache.org/jira/browse/KAFKA-1517
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Daniel Compton
Priority: Trivial
  Labels: newbie

 When running the producer performance test without providing a messages 
 argument, you get an error:
 {noformat}
 $bin/kafka-producer-perf-test.sh --topics mirrormirror --broker-list 
 kafka-dc21:9092
 Missing required argument [messages]
 Option  Description
 --  ---
 ..
 --messages Long: countThe number of messages to send or
   consume (default:
   9223372036854775807)
 {noformat}
 However the [shell command 
 documentation|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/perf/src/main/scala/kafka/perf/PerfConfig.scala#L25]
  doesn't say that this is required and implies that 
 [2^63-1|http://en.wikipedia.org/wiki/9223372036854775807] (Long.MaxValue) 
 messages will be sent. It should probably look like the 
 [ConsoleProducer|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/core/src/main/scala/kafka/producer/ConsoleProducer.scala#L32]
  and prefix the documentation with REQUIRED. Or should we make this a 
 non-required argument and set the default value to something sane like 
 100,000 messages.
 Which option is preferable for this?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1517) Messages is a required argument to Producer Performance Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Compton updated KAFKA-1517:
--

Status: Patch Available  (was: Open)

I've attached a patch for this. Let me know if the patch isn't correct, I 
wasn't sure I was doing it correctly.

 Messages is a required argument to Producer Performance Test
 

 Key: KAFKA-1517
 URL: https://issues.apache.org/jira/browse/KAFKA-1517
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Daniel Compton
Priority: Trivial
  Labels: newbie
 Attachments: 0001-KAFKA-1517-Make-messages-a-required-argument.patch


 When running the producer performance test without providing a messages 
 argument, you get an error:
 {noformat}
 $bin/kafka-producer-perf-test.sh --topics mirrormirror --broker-list 
 kafka-dc21:9092
 Missing required argument [messages]
 Option  Description
 --  ---
 ..
 --messages Long: countThe number of messages to send or
   consume (default:
   9223372036854775807)
 {noformat}
 However the [shell command 
 documentation|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/perf/src/main/scala/kafka/perf/PerfConfig.scala#L25]
  doesn't say that this is required and implies that 
 [2^63-1|http://en.wikipedia.org/wiki/9223372036854775807] (Long.MaxValue) 
 messages will be sent. It should probably look like the 
 [ConsoleProducer|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/core/src/main/scala/kafka/producer/ConsoleProducer.scala#L32]
  and prefix the documentation with REQUIRED. Or should we make this a 
 non-required argument and set the default value to something sane like 
 100,000 messages.
 Which option is preferable for this?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1517) Messages is a required argument to Producer Performance Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Compton updated KAFKA-1517:
--

Status: Open  (was: Patch Available)

 Messages is a required argument to Producer Performance Test
 

 Key: KAFKA-1517
 URL: https://issues.apache.org/jira/browse/KAFKA-1517
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Daniel Compton
Priority: Trivial
  Labels: newbie
 Attachments: 0001-KAFKA-1517-Make-messages-a-required-argument.patch


 When running the producer performance test without providing a messages 
 argument, you get an error:
 {noformat}
 $bin/kafka-producer-perf-test.sh --topics mirrormirror --broker-list 
 kafka-dc21:9092
 Missing required argument [messages]
 Option  Description
 --  ---
 ..
 --messages Long: countThe number of messages to send or
   consume (default:
   9223372036854775807)
 {noformat}
 However the [shell command 
 documentation|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/perf/src/main/scala/kafka/perf/PerfConfig.scala#L25]
  doesn't say that this is required and implies that 
 [2^63-1|http://en.wikipedia.org/wiki/9223372036854775807] (Long.MaxValue) 
 messages will be sent. It should probably look like the 
 [ConsoleProducer|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/core/src/main/scala/kafka/producer/ConsoleProducer.scala#L32]
  and prefix the documentation with REQUIRED. Or should we make this a 
 non-required argument and set the default value to something sane like 
 100,000 messages.
 Which option is preferable for this?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1517) Messages is a required argument to Producer Performance Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Compton updated KAFKA-1517:
--

Attachment: 0001-KAFKA-1517-Make-messages-a-required-argument.patch

 Messages is a required argument to Producer Performance Test
 

 Key: KAFKA-1517
 URL: https://issues.apache.org/jira/browse/KAFKA-1517
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Daniel Compton
Priority: Trivial
  Labels: newbie
 Attachments: 0001-KAFKA-1517-Make-messages-a-required-argument.patch


 When running the producer performance test without providing a messages 
 argument, you get an error:
 {noformat}
 $bin/kafka-producer-perf-test.sh --topics mirrormirror --broker-list 
 kafka-dc21:9092
 Missing required argument [messages]
 Option  Description
 --  ---
 ..
 --messages Long: countThe number of messages to send or
   consume (default:
   9223372036854775807)
 {noformat}
 However the [shell command 
 documentation|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/perf/src/main/scala/kafka/perf/PerfConfig.scala#L25]
  doesn't say that this is required and implies that 
 [2^63-1|http://en.wikipedia.org/wiki/9223372036854775807] (Long.MaxValue) 
 messages will be sent. It should probably look like the 
 [ConsoleProducer|https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/core/src/main/scala/kafka/producer/ConsoleProducer.scala#L32]
  and prefix the documentation with REQUIRED. Or should we make this a 
 non-required argument and set the default value to something sane like 
 100,000 messages.
 Which option is preferable for this?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1420) Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with TestUtils.createTopic in all unit tests