Re: Fault Injection

[Timothy Chen]

Hi Colin,

The Kibosh code is just a README for now, is it going to be published soon?

Tim

On Tue, Aug 22, 2017 at 11:44 AM, Colin McCabe  wrote:
> Hi all,
>
> I've been working on a fault injector for Apache Kafka.  The general
> idea is to create faults such as network partitions or disk failures,
> and see what happens in the cluster.  The fault injector can run as part
> of a ducktape system test, or standalone.
>
> The fault injector has two processes: a coordinator, and an agent.  The
> agent process is responsible for actually implementing the faults.  For
> example, it might run iptables, send signals to processes, generate a
> lot of load, or do something else to disrupt the computer it is running
> on.  We run an agent process on each node where we would like to
> potentially inject faults.  So it will run alongside the brokers,
> zookeeper nodes, etc.
>
> The coordinator process is responsible for communicating with the agent
> processes and for scheduling faults.  For example, the coordinator can
> be instructed to create a fault immediately on several nodes.  Or it can
> be instructed to create faults over time, based on a pseudorandom seed.
> Both the coordinator and the agent expose a REST interface that accepts
> objects serialized via JSON.
>
> I think two kinds of faults will be especially interesting: network
> faults, and disk errors.  Simulating network faults in a Linux
> environment is relatively straightforward using iptables.  Disk errors
> are tougher to simulate, but I have written a FUSE filesystem to do
> this.  The  filesystem essentially simulates a bind mount in most cases,
> but it can take a JSON specification telling it to inject certain
> faults.  (Disk errors seem especially relevant to the ongoing work on
> JBOD.)
>
> Although it's not a user-visible component, I think having a fault
> injector will be really great for Kafka users.  It will really help us
> stress test Kafka in more situations.  I'm going to post some patches in
> a day or two-- it would be great to get some feedback.  Check out
> https://cwiki.apache.org/confluence/display/KAFKA/Fault+Injection
>
> best,
> Colin

[jira] [Assigned] (KAFKA-2339) broker becomes unavailable if bad data is passed through the protocol

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen reassigned KAFKA-2339:
---

Assignee: Timothy Chen

 broker becomes unavailable if bad data is passed through the protocol
 -

 Key: KAFKA-2339
 URL: https://issues.apache.org/jira/browse/KAFKA-2339
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
Assignee: Timothy Chen
Priority: Critical
 Fix For: 0.8.3


 I ran into a situation that a non integer value got past for the partition 
 and the brokers went bonkers.
 reproducible
 {code}
 ah=1..2
 echo don't do this in production|kafkacat -b localhost:9092 -p $ah
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2339) broker becomes unavailable if bad data is passed through the protocol

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630981#comment-14630981
 ] 

Timothy Chen commented on KAFKA-2339:
-

Somehow I can't produce this with kafkacat and latest trunk:

Timothys-MacBook-Pro :: ~/src/kafka ‹trunk*› » kafkacat -P -p 3..5 -t test -b 
localhost:9092  
   1 ↵
abc
% ERROR: Failed to produce message (3 bytes): Local: Unknown partition

Which version of Kafka are you using?

 broker becomes unavailable if bad data is passed through the protocol
 -

 Key: KAFKA-2339
 URL: https://issues.apache.org/jira/browse/KAFKA-2339
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
Assignee: Timothy Chen
Priority: Critical
 Fix For: 0.8.3


 I ran into a situation that a non integer value got past for the partition 
 and the brokers went bonkers.
 reproducible
 {code}
 ah=1..2
 echo don't do this in production|kafkacat -b localhost:9092 -p $ah
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36474: Patch for KAFKA-2188

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36474/
---

(Updated July 15, 2015, 8:19 a.m.)


Review request for kafka and Joe Stein.


Bugs: KAFKA-2188
https://issues.apache.org/jira/browse/KAFKA-2188


Repository: kafka


Description
---

KAFKA-2188 - JBOD Support


Diffs (updated)
-

  core/src/main/scala/kafka/cluster/Partition.scala 2649090 
  core/src/main/scala/kafka/common/GenericKafkaStorageException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/controller/KafkaController.scala b4fc755 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala bb6b5c8 
  core/src/main/scala/kafka/log/Log.scala e5e8007 
  core/src/main/scala/kafka/log/LogManager.scala 69386c1 
  core/src/main/scala/kafka/log/LogSegment.scala 1377e8f 
  core/src/main/scala/kafka/server/KafkaApis.scala 18f5b5b 
  core/src/main/scala/kafka/server/KafkaConfig.scala dbe170f 
  core/src/main/scala/kafka/server/OffsetCheckpoint.scala 8c5b054 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala c89d00b 
  core/src/main/scala/kafka/server/ReplicaManager.scala 795220e 
  core/src/main/scala/kafka/utils/CoreUtils.scala 168a18d 
  core/src/main/scala/kafka/utils/ZkUtils.scala 166814c 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala a13f2be 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
60cd824 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 00d5933 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 17e9fe4 

Diff: https://reviews.apache.org/r/36474/diff/


Testing
---


Thanks,

Timothy Chen

Review Request 36503: Patch for KAFKA-2188

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36503/
---

Review request for kafka.


Bugs: KAFKA-2188
https://issues.apache.org/jira/browse/KAFKA-2188


Repository: kafka


Description
---

KAFKA-2188 - JBOD Support


Diffs
-

  core/src/main/scala/kafka/cluster/Partition.scala 
2649090b6cbf8d442649f19fd7113a30d62bca91 
  core/src/main/scala/kafka/common/GenericKafkaStorageException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/controller/KafkaController.scala 
b4fc755641b9bbe8a6bf9c221a9ffaec0b94d6e8 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
bb6b5c8764522e7947bb08998256ce1deb717c84 
  core/src/main/scala/kafka/log/Log.scala 
e5e80079645ce6e6fe7bb1c2696d3dd21a07761b 
  core/src/main/scala/kafka/log/LogManager.scala 
69386c17153e5ef08a24d4f14b915e4316b121d8 
  core/src/main/scala/kafka/log/LogSegment.scala 
1377e8f322a3fedc683d93feaf27c955de528a4b 
  core/src/main/scala/kafka/server/KafkaApis.scala 
18f5b5b895af1469876b2223841fd90a2dd255e0 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
dbe170f87331f43e2dc30165080d2cb7dfe5fdbf 
  core/src/main/scala/kafka/server/OffsetCheckpoint.scala 
8c5b0546908d3b3affb9f48e2ece9ed252518783 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
c89d00b5976ffa34cafdae261239934b1b917bfe 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
795220e7f63d163be90738b4c1a39687b44c1395 
  core/src/main/scala/kafka/utils/CoreUtils.scala 
168a18d380c200ee566eccb6988dd1ae85ed5b09 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
166814c2959a429e20f400d1c0e523090ce37d91 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
a13f2bef8ee8c3d42192c9a60df092023e4d2ff9 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
60cd8249e6ec03349e20bb0a7226ea9cd66e6b17 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
17e9fe4c159a29033fe9a287db6ced2fdc3fa9d1 

Diff: https://reviews.apache.org/r/36503/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-2188) JBOD Support

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627708#comment-14627708
 ] 

Timothy Chen commented on KAFKA-2188:
-

Created reviewboard https://reviews.apache.org/r/36503/diff/
 against branch origin/trunk

 JBOD Support
 

 Key: KAFKA-2188
 URL: https://issues.apache.org/jira/browse/KAFKA-2188
 Project: Kafka
  Issue Type: Bug
Reporter: Andrii Biletskyi
Assignee: Andrii Biletskyi
 Attachments: KAFKA-2188.patch, KAFKA-2188.patch, KAFKA-2188.patch


 https://cwiki.apache.org/confluence/display/KAFKA/KIP-18+-+JBOD+Support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2188) JBOD Support

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-2188:

Attachment: KAFKA-2188.patch

 JBOD Support
 

 Key: KAFKA-2188
 URL: https://issues.apache.org/jira/browse/KAFKA-2188
 Project: Kafka
  Issue Type: Bug
Reporter: Andrii Biletskyi
Assignee: Andrii Biletskyi
 Attachments: KAFKA-2188.patch, KAFKA-2188.patch, KAFKA-2188.patch


 https://cwiki.apache.org/confluence/display/KAFKA/KIP-18+-+JBOD+Support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 36474: Patch for KAFKA-2188

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36474/
---

Review request for kafka.


Bugs: KAFKA-2188
https://issues.apache.org/jira/browse/KAFKA-2188


Repository: kafka


Description
---

KAFKA-2188 - JBOD Support


Diffs
-

  core/src/main/scala/kafka/cluster/Partition.scala 
2649090b6cbf8d442649f19fd7113a30d62bca91 
  core/src/main/scala/kafka/common/GenericKafkaStorageException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/controller/KafkaController.scala 
b4fc755641b9bbe8a6bf9c221a9ffaec0b94d6e8 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
bb6b5c8764522e7947bb08998256ce1deb717c84 
  core/src/main/scala/kafka/log/Log.scala 
e5e80079645ce6e6fe7bb1c2696d3dd21a07761b 
  core/src/main/scala/kafka/log/LogManager.scala 
69386c17153e5ef08a24d4f14b915e4316b121d8 
  core/src/main/scala/kafka/log/LogSegment.scala 
1377e8f322a3fedc683d93feaf27c955de528a4b 
  core/src/main/scala/kafka/server/KafkaApis.scala 
18f5b5b895af1469876b2223841fd90a2dd255e0 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
dbe170f87331f43e2dc30165080d2cb7dfe5fdbf 
  core/src/main/scala/kafka/server/OffsetCheckpoint.scala 
8c5b0546908d3b3affb9f48e2ece9ed252518783 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
c89d00b5976ffa34cafdae261239934b1b917bfe 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
795220e7f63d163be90738b4c1a39687b44c1395 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
166814c2959a429e20f400d1c0e523090ce37d91 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
a13f2bef8ee8c3d42192c9a60df092023e4d2ff9 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
60cd8249e6ec03349e20bb0a7226ea9cd66e6b17 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
17e9fe4c159a29033fe9a287db6ced2fdc3fa9d1 

Diff: https://reviews.apache.org/r/36474/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-2188) JBOD Support

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625910#comment-14625910
 ] 

Timothy Chen commented on KAFKA-2188:
-

Created reviewboard https://reviews.apache.org/r/36474/diff/
 against branch origin/trunk

 JBOD Support
 

 Key: KAFKA-2188
 URL: https://issues.apache.org/jira/browse/KAFKA-2188
 Project: Kafka
  Issue Type: Bug
Reporter: Andrii Biletskyi
Assignee: Andrii Biletskyi
 Attachments: KAFKA-2188.patch, KAFKA-2188.patch


 https://cwiki.apache.org/confluence/display/KAFKA/KIP-18+-+JBOD+Support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2188) JBOD Support

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-2188:

Attachment: KAFKA-2188.patch

 JBOD Support
 

 Key: KAFKA-2188
 URL: https://issues.apache.org/jira/browse/KAFKA-2188
 Project: Kafka
  Issue Type: Bug
Reporter: Andrii Biletskyi
Assignee: Andrii Biletskyi
 Attachments: KAFKA-2188.patch, KAFKA-2188.patch


 https://cwiki.apache.org/confluence/display/KAFKA/KIP-18+-+JBOD+Support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSSION] Message Metadata

[Timothy Chen]

Hi Guozhang,

I don't think that is publically accessible, can you update it to the
Kafka wiki?

Tim

On Fri, Nov 21, 2014 at 5:24 PM, Guozhang Wang wangg...@gmail.com wrote:
 Hi all,

 I have updated the wiki page (
 https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Kafka+Enriched+Message+Metadata)
 according to people's comments and discussions offline.

 Guozhang

 On Thu, Nov 13, 2014 at 9:43 AM, Guozhang Wang wangg...@gmail.com wrote:

 Hi Jun,

 Sorry for the delay on your comments in the wiki page as well as this
 thread; quite swamped now. I will get back to you as soon as I find some
 time.

 Guozhang

 On Tue, Nov 11, 2014 at 6:26 PM, Jun Rao jun...@gmail.com wrote:

 Thinking about this a bit more. For adding the auditing support, I am not
 sure if we need to change the message format by adding the application
 tags. An alternative way to do that is to add it in the producer client.
 For example, for each message payload (doesn't matter what the
 serialization mechanism is) that a producer receives, the producer can
 just
 add a header before the original payload. The header will contain all
 needed fields (e.g. timestamp, host, etc) for the purpose of auditing.
 This
 way, we don't need to change the message format and the auditing info can
 be added independent of the serialization mechanism of the message. The
 header can use a different serialization mechanism for better efficiency.
 For example, if we use Avro to serialize the header, the encoded bytes
 won't include the field names in the header. This is potentially more
 efficient than representing those fields as application tags in the
 message
 where the tags have to be explicitly store in every message.

 To make it easier for the client to add and make use of this kind of
 auditing support, I was imagining that we can add a ProducerFactory in the
 new java client. The ProducerFactory will create an instance of Producer
 based on a config property. By default, the current KafkaProducer will be
 returned. However, a user can plug in a different implementation of
 Producer that does auditing. For example, an implementation of an
 AuditProducer.send() can take the original ProducerRecord, add the header
 to the value byte array and then forward the record to an underlying
 KafkaProducer. We can add a similar ConsumerFactory to the new consumer
 client. If a user plugs in an implementation of the AuditingConsumer, the
 consumer will then be audited automatically.

 Thanks,

 Jun

 On Tue, Oct 21, 2014 at 4:06 PM, Guozhang Wang wangg...@gmail.com
 wrote:

  Hi Jun,
 
  Regarding 4) in your comment, after thinking it for a while I cannot
 come
  up a way to it along with log compaction without adding new fields into
 the
  current format on message set. Do you have a better way that do not
 require
  protocol changes?
 
  Guozhang
 
  On Mon, Oct 20, 2014 at 9:53 AM, Guozhang Wang wangg...@gmail.com
 wrote:
 
   I have updated the wiki page incorporating received comments. We can
   discuss some more details on:
  
   1. How we want to do audit? Whether we want to have in-built auditing
 on
   brokers or even MMs or use  an audit consumer to fetch all messages
 from
   just brokers.
  
   2. How we can avoid de-/re-compression on brokers and MMs with log
   compaction turned on.
  
   3. How we can resolve unclean leader election resulted data
 inconsistency
   with control messages.
  
   Guozhang
  
   On Sun, Oct 19, 2014 at 11:41 PM, Guozhang Wang wangg...@gmail.com
   wrote:
  
   Thanks for the detailed comments Jun! Some replies inlined.
  
   On Sun, Oct 19, 2014 at 7:42 PM, Jun Rao jun...@gmail.com wrote:
  
   Hi, Guozhang,
  
   Thanks for the writeup.
  
   A few high level comments.
  
   1. Associating (versioned) schemas to a topic can be a good thing
   overall.
   Yes, this could add a bit more management overhead in Kafka.
 However,
  it
   makes sure that the data format contract between a producer and a
   consumer
   is kept and managed in a central place, instead of in the
 application.
   The
   latter is probably easier to start with, but is likely to be
 brittle in
   the
   long run.
  
  
   I am actually not proposing to not support associated versioned
 schemas
   for topics, but to not let some core Kafka functionalities like
 auditing
   being depend on schemas. I think this alone can separate the schema
   management from Kafka piping management (i.e. making sure every
 single
   message is delivered, and within some latency, etc). Adding
 additional
   auditing info into an existing schema will force Kafka to be aware of
  the
   schema systems (Avro, JSON, etc).
  
  
  
   2. Auditing can be a general feature that's useful for many
  applications.
   Such a feature can be implemented by extending the low level message
   format
   with a header. However, it can also be added as part of the schema
   management. For example, you can imagine a type of audited schema
 that
   adds
   additional auditing 

Re: Welcome Kafka's newest committer

[Timothy Chen]

Congrats Guozhang!

Tim

On Wed, Nov 19, 2014 at 4:05 PM, Neha Narkhede neha.narkh...@gmail.com wrote:
 Hi everyone,

 I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
 become a committer. Guozhang has made significant contributions to Kafka
 over the past year, along with being very active on code reviews and the
 mailing list.

 Please join me in welcoming him.

 Thanks,
 Neha (on behalf of the Kafka PMC)

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

 On Aug. 6, 2014, 5:32 a.m., Timothy Chen wrote:
  core/src/main/scala/kafka/server/TransactionManager.scala, line 300
  https://reviews.apache.org/r/23568/diff/6/?file=653418#file653418line300
 
  I'm just trying to think if there is a chance to have overlapping tx 
  ids especially around failover. I'm not familiar with the tx changes so 
  just wondering if the range size can cause issues. Seems like you're 
  incrementing the zk version on each id fetch and allow a range of the batch 
  size of them. I was thinking if I sent 999 messages and then failed over, 
  and the new broker that took over has a small batch size now, even though 
  the version has incremented the new range now overlaps then right? It might 
  already been handled but just want to throw it out there.
 
 Dong Lin wrote:
 Sure. Thanks for taking time to help think about it.
 
 Here I assume the batch size should be the same across producers 
 regardless of when they start. If a producer sends 999 messages, say with 
 txid in range 1000 to 1999, and fails over, the next producer will get the 
 txid in range 2000 to 2999. The range is guaranteed to be disjoint from 
 ranges used by previous/existing producers. Does this make sense?
 
 Dong Lin wrote:
 Correction: I actually mean broker where I say producer in my response.

The next producer will do so by looking at the default batch size with the 
version, which is what I was thinking there really isn't anything guarding 
aganist the range being changed from the code.
The safest thing to do is perhaps write the range into zk node data and when 
the current range is smaller have a big warning.
But minimally we will want to leave a comment in the code on the batch size to 
have this warning so this isn't something to experiment with as it can mess up 
existing transactions.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49696
---


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23568/
 ---
 
 (Updated Aug. 6, 2014, 4:25 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1523
 https://issues.apache.org/jira/browse/KAFKA-1523
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1523 Transaction manager and its failover handling.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/TopicCommand.scala 
 003a09c6160618bc94858ebc0d806b2aa4158e0a 
   core/src/main/scala/kafka/cluster/Partition.scala 
 134aef9c88068443d4d465189f376dd78605b4f8 
   core/src/main/scala/kafka/common/Topic.scala 
 ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
   core/src/main/scala/kafka/message/Message.scala 
 d2a7293c7be4022af30884330924791340acc5c1 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
   core/src/main/scala/kafka/server/KafkaConfig.scala 
 1a45f8716ccc0398cf9395d91d66199d16882aae 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 28711182aaa70eaa623de858bc063cb2613b2a4d 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 897783cb756de548a8b634876f729b63ffe9925e 
   core/src/main/scala/kafka/server/RequestPurgatory.scala 
 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
   core/src/main/scala/kafka/utils/ZkUtils.scala 
 dcdc1ce2b02c996294e19cf480736106aaf29511 
 
 Diff: https://reviews.apache.org/r/23568/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Dong Lin
 

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

 On Aug. 6, 2014, 5:32 a.m., Timothy Chen wrote:
  core/src/main/scala/kafka/server/TransactionManager.scala, line 300
  https://reviews.apache.org/r/23568/diff/6/?file=653418#file653418line300
 
  I'm just trying to think if there is a chance to have overlapping tx 
  ids especially around failover. I'm not familiar with the tx changes so 
  just wondering if the range size can cause issues. Seems like you're 
  incrementing the zk version on each id fetch and allow a range of the batch 
  size of them. I was thinking if I sent 999 messages and then failed over, 
  and the new broker that took over has a small batch size now, even though 
  the version has incremented the new range now overlaps then right? It might 
  already been handled but just want to throw it out there.
 
 Dong Lin wrote:
 Sure. Thanks for taking time to help think about it.
 
 Here I assume the batch size should be the same across producers 
 regardless of when they start. If a producer sends 999 messages, say with 
 txid in range 1000 to 1999, and fails over, the next producer will get the 
 txid in range 2000 to 2999. The range is guaranteed to be disjoint from 
 ranges used by previous/existing producers. Does this make sense?
 
 Dong Lin wrote:
 Correction: I actually mean broker where I say producer in my response.
 
 Timothy Chen wrote:
 The next producer will do so by looking at the default batch size with 
 the version, which is what I was thinking there really isn't anything 
 guarding aganist the range being changed from the code.
 The safest thing to do is perhaps write the range into zk node data and 
 when the current range is smaller have a big warning.
 But minimally we will want to leave a comment in the code on the batch 
 size to have this warning so this isn't something to experiment with as it 
 can mess up existing transactions.
 
 Dong Lin wrote:
 It is the broker who looks at the default batch size with the version, 
 and allocate unique txId to producer upon request. In other words, this code 
 on the server side and we don't need to worry about Kafka user changing this 
 code. Am I right?

You're right we don't need to worry about it for the user, but we need to worry 
about it when we change the default and do a rolling upgrade.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49696
---


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23568/
 ---
 
 (Updated Aug. 6, 2014, 4:25 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1523
 https://issues.apache.org/jira/browse/KAFKA-1523
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1523 Transaction manager and its failover handling.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/TopicCommand.scala 
 003a09c6160618bc94858ebc0d806b2aa4158e0a 
   core/src/main/scala/kafka/cluster/Partition.scala 
 134aef9c88068443d4d465189f376dd78605b4f8 
   core/src/main/scala/kafka/common/Topic.scala 
 ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
   core/src/main/scala/kafka/message/Message.scala 
 d2a7293c7be4022af30884330924791340acc5c1 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
   core/src/main/scala/kafka/server/KafkaConfig.scala 
 1a45f8716ccc0398cf9395d91d66199d16882aae 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 28711182aaa70eaa623de858bc063cb2613b2a4d 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 897783cb756de548a8b634876f729b63ffe9925e 
   core/src/main/scala/kafka/server/RequestPurgatory.scala 
 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
   core/src/main/scala/kafka/utils/ZkUtils.scala 
 dcdc1ce2b02c996294e19cf480736106aaf29511 
 
 Diff: https://reviews.apache.org/r/23568/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Dong Lin
 

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

 On Aug. 6, 2014, 5:32 a.m., Timothy Chen wrote:
  core/src/main/scala/kafka/server/TransactionManager.scala, line 300
  https://reviews.apache.org/r/23568/diff/6/?file=653418#file653418line300
 
  I'm just trying to think if there is a chance to have overlapping tx 
  ids especially around failover. I'm not familiar with the tx changes so 
  just wondering if the range size can cause issues. Seems like you're 
  incrementing the zk version on each id fetch and allow a range of the batch 
  size of them. I was thinking if I sent 999 messages and then failed over, 
  and the new broker that took over has a small batch size now, even though 
  the version has incremented the new range now overlaps then right? It might 
  already been handled but just want to throw it out there.
 
 Dong Lin wrote:
 Sure. Thanks for taking time to help think about it.
 
 Here I assume the batch size should be the same across producers 
 regardless of when they start. If a producer sends 999 messages, say with 
 txid in range 1000 to 1999, and fails over, the next producer will get the 
 txid in range 2000 to 2999. The range is guaranteed to be disjoint from 
 ranges used by previous/existing producers. Does this make sense?
 
 Dong Lin wrote:
 Correction: I actually mean broker where I say producer in my response.
 
 Timothy Chen wrote:
 The next producer will do so by looking at the default batch size with 
 the version, which is what I was thinking there really isn't anything 
 guarding aganist the range being changed from the code.
 The safest thing to do is perhaps write the range into zk node data and 
 when the current range is smaller have a big warning.
 But minimally we will want to leave a comment in the code on the batch 
 size to have this warning so this isn't something to experiment with as it 
 can mess up existing transactions.
 
 Dong Lin wrote:
 It is the broker who looks at the default batch size with the version, 
 and allocate unique txId to producer upon request. In other words, this code 
 on the server side and we don't need to worry about Kafka user changing this 
 code. Am I right?
 
 Timothy Chen wrote:
 You're right we don't need to worry about it for the user, but we need to 
 worry about it when we change the default and do a rolling upgrade.
 
 Dong Lin wrote:
 Oh, I see your point. Sure, I agree that we should add a comment in the 
 TransactionManger code so that any future developer should be aware of the 
 effect of changing DefaultTransactionsIdBatchSize. But since this is just one 
 of the many other configuration, I am not sure if it is worth saving this 
 default value in zk node.
 
 Does that address your concern? Thanks!

Yes I don't think we need to do write it out, just a comment should be good IMO.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49696
---


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23568/
 ---
 
 (Updated Aug. 6, 2014, 4:25 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1523
 https://issues.apache.org/jira/browse/KAFKA-1523
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1523 Transaction manager and its failover handling.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/TopicCommand.scala 
 003a09c6160618bc94858ebc0d806b2aa4158e0a 
   core/src/main/scala/kafka/cluster/Partition.scala 
 134aef9c88068443d4d465189f376dd78605b4f8 
   core/src/main/scala/kafka/common/Topic.scala 
 ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
   core/src/main/scala/kafka/message/Message.scala 
 d2a7293c7be4022af30884330924791340acc5c1 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
   core/src/main/scala/kafka/server/KafkaConfig.scala 
 1a45f8716ccc0398cf9395d91d66199d16882aae 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 28711182aaa70eaa623de858bc063cb2613b2a4d 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 897783cb756de548a8b634876f729b63ffe9925e 
   core/src/main/scala/kafka/server/RequestPurgatory.scala 
 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
   core/src/main/scala/kafka/utils/ZkUtils.scala 
 dcdc1ce2b02c996294e19cf480736106aaf29511 
 
 Diff: https://reviews.apache.org/r/23568/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Dong Lin
 

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49692
---



core/src/main/scala/kafka/admin/TopicCommand.scala
https://reviews.apache.org/r/23568/#comment86976

Why not use the InternalTopics array here?



core/src/main/scala/kafka/server/KafkaApis.scala
https://reviews.apache.org/r/23568/#comment86978

that is quite a mouthful, should it be called just 
producerTxRequestToCoordinator?



core/src/main/scala/kafka/server/TransactionManager.scala
https://reviews.apache.org/r/23568/#comment86979

If the batch size changes to a smaller number does the tx Id range overlap 
then?


- Timothy Chen


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23568/
 ---
 
 (Updated Aug. 6, 2014, 4:25 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1523
 https://issues.apache.org/jira/browse/KAFKA-1523
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1523 Transaction manager and its failover handling.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/TopicCommand.scala 
 003a09c6160618bc94858ebc0d806b2aa4158e0a 
   core/src/main/scala/kafka/cluster/Partition.scala 
 134aef9c88068443d4d465189f376dd78605b4f8 
   core/src/main/scala/kafka/common/Topic.scala 
 ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
   core/src/main/scala/kafka/message/Message.scala 
 d2a7293c7be4022af30884330924791340acc5c1 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
   core/src/main/scala/kafka/server/KafkaConfig.scala 
 1a45f8716ccc0398cf9395d91d66199d16882aae 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 28711182aaa70eaa623de858bc063cb2613b2a4d 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 897783cb756de548a8b634876f729b63ffe9925e 
   core/src/main/scala/kafka/server/RequestPurgatory.scala 
 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
   core/src/main/scala/kafka/utils/ZkUtils.scala 
 dcdc1ce2b02c996294e19cf480736106aaf29511 
 
 Diff: https://reviews.apache.org/r/23568/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Dong Lin
 

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49696
---



core/src/main/scala/kafka/server/TransactionManager.scala
https://reviews.apache.org/r/23568/#comment86986

I'm just trying to think if there is a chance to have overlapping tx ids 
especially around failover. I'm not familiar with the tx changes so just 
wondering if the range size can cause issues. Seems like you're incrementing 
the zk version on each id fetch and allow a range of the batch size of them. I 
was thinking if I sent 999 messages and then failed over, and the new broker 
that took over has a small batch size now, even though the version has 
incremented the new range now overlaps then right? It might already been 
handled but just want to throw it out there. 


- Timothy Chen


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23568/
 ---
 
 (Updated Aug. 6, 2014, 4:25 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1523
 https://issues.apache.org/jira/browse/KAFKA-1523
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1523 Transaction manager and its failover handling.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/TopicCommand.scala 
 003a09c6160618bc94858ebc0d806b2aa4158e0a 
   core/src/main/scala/kafka/cluster/Partition.scala 
 134aef9c88068443d4d465189f376dd78605b4f8 
   core/src/main/scala/kafka/common/Topic.scala 
 ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
   core/src/main/scala/kafka/message/Message.scala 
 d2a7293c7be4022af30884330924791340acc5c1 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
   core/src/main/scala/kafka/server/KafkaConfig.scala 
 1a45f8716ccc0398cf9395d91d66199d16882aae 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 28711182aaa70eaa623de858bc063cb2613b2a4d 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 897783cb756de548a8b634876f729b63ffe9925e 
   core/src/main/scala/kafka/server/RequestPurgatory.scala 
 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
   core/src/main/scala/kafka/utils/ZkUtils.scala 
 dcdc1ce2b02c996294e19cf480736106aaf29511 
 
 Diff: https://reviews.apache.org/r/23568/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Dong Lin
 

Re: Issues getting IntelliJ set up

[Timothy Chen]

Hi Johnathan,

Did you update your scala version before you run gradle idea?

Also try cleaning up all the artifacts and try it again, as perhaps
your intellij is not picking up the right version and from the right
build folder.

Tim

On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins na...@streamsets.com wrote:
 Hi,

 I've been having some issues getting IntelliJ set up...I followed all the
 instructions on the wiki, and I've successfully imported the project, and
 run the jar Gradle target successfully. However, when I try to run a test
 in the IDE, I get a number of errors:

 /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
 Error:(21, 30) java: package kafka.javaapi.producer does not exist
 Error:(22, 22) java: package kafka.producer does not exist
 Error:(23, 22) java: package kafka.producer does not exist
 Error:(24, 19) java: cannot find symbol
   symbol:   class Utils
   location: package kafka.utils
 Error:(303, 39) java: cannot find symbol
   symbol:   class KeyedMessage
   location: class kafka.tools.KafkaMigrationTool.MigrationThread

 And so on.

 The two classes that seem to be causing trouble are KafkaMigrationTool and
 ConsumerConnector. Has anybody run into this? Anyone know how to get around
 this issue?

 Thanks a lot,
 Natty

Re: Review Request 23702: Patch for KAFKA-1070

[Timothy Chen]

 On July 19, 2014, 11:28 p.m., Timothy Chen wrote:
  core/src/main/scala/kafka/server/KafkaConfig.scala, line 64
  https://reviews.apache.org/r/23702/diff/2/?file=636339#file636339line64
 
  Why reduce the range to 1000? 
 
 
 Sriharsha Chintalapani wrote:
 As we go with auto-generating broker id we want to have a backward 
 compatibility where a user can continue to provide brokerId as part of 
 server.properties we will give preference to the user provided brokerId if it 
 presents instead a of generating a new seqId from zookeeper. 
 1000 here is more of a reserved number for user provided config and to 
 avoid conflict with zookeeper generated seqId. For more details
 please check comments by Jay Kreps here 
 https://issues.apache.org/jira/browse/KAFKA-1070. Thanks.

I see, a comment and a constant will be helpful for what it is.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23702/#review48184
---


On July 19, 2014, 11:06 p.m., Sriharsha Chintalapani wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23702/
 ---
 
 (Updated July 19, 2014, 11:06 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1070
 https://issues.apache.org/jira/browse/KAFKA-1070
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1070. Auto assign node id.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/common/InconsistentBrokerIdException.scala 
 PRE-CREATION 
   core/src/main/scala/kafka/server/KafkaConfig.scala 
 50b09edb73af1b45f88f919ac8c46ae056878c8e 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 def1dc2a5818d45d9ee0881137ff989cec4eb9b1 
   core/src/main/scala/kafka/utils/ZkUtils.scala 
 dcdc1ce2b02c996294e19cf480736106aaf29511 
   core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
 PRE-CREATION 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 3faa884f8eb83c7c00baab416d0acfb488dc39c1 
 
 Diff: https://reviews.apache.org/r/23702/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Sriharsha Chintalapani
 

Re: Improving the Kafka client ecosystem

[Timothy Chen]

The certified client test suite really will benefit all the client developers, 
as writing a Kafka client often is not just talking protocol but to be able to 
handle correctly all the cases, errors and situations, but also performance.

From my experience writing a C# client definitely feel that a lot of test 
scenarios could be generalized and used for all clients.

I was reviewing some other client implementation and there are errors and cases 
it didn't handle and having a suite that exposes that will allow users to not 
run knot those problems and try to determine its a client or server bug as it's 
sometimes hard to figure out.

Tim

 On Jul 18, 2014, at 3:57 PM, Jay Kreps jay.kr...@gmail.com wrote:
 
 Basically my thought with getting a separate mailing list was to have
 a place specifically to discuss issues around clients. I don't see a
 lot of discussion about them on the main list. I thought perhaps this
 was because people don't like to ask questions which are about
 adjacent projects/code bases. But basically whatever will lead to a
 robust discussion, bug tracking, etc on clients.
 
 -Jay
 
 On Fri, Jul 18, 2014 at 3:49 PM, Jun Rao jun...@gmail.com wrote:
 Another important part of eco-system could be around the adaptors of
 getting data from other systems into Kafka and vice versa. So, for the
 ingestion part, this can include things like getting data from mysql,
 syslog, apache server log, etc. For the egress part, this can include
 putting Kafka data into HDFS, S3, etc.
 
 Will a separate mailing list be convenient? Could we just use the Kafka
 mailing list?
 
 Thanks,
 
 Jun
 
 
 On Fri, Jul 18, 2014 at 2:34 PM, Jay Kreps jay.kr...@gmail.com wrote:
 
 A question was asked in another thread about what was an effective way
 to contribute to the Kafka project for people who weren't very
 enthusiastic about writing Java/Scala code.
 
 I wanted to kind of advocate for an area I think is really important
 and not as good as it could be--the client ecosystem. I think our goal
 is to make Kafka effective as a general purpose, centralized, data
 subscription system. This vision only really works if all your
 applications, are able to integrate easily, whatever language they are
 in.
 
 We have a number of pretty good non-java producers. We have been
 lacking the features on the server-side to make writing non-java
 consumers easy. We are fixing that right now as part of the consumer
 work going on right now (which moves a lot of the functionality in the
 java consumer to the server side).
 
 But apart from this I think there may be a lot more we can do to make
 the client ecosystem better.
 
 Here are some concrete ideas. If anyone has additional ideas please
 reply to this thread and share them. If you are interested in picking
 any of these up, please do.
 
 1. The most obvious way to improve the ecosystem is to help work on
 clients. This doesn't necessarily mean writing new clients, since in
 many cases we already have a client in a given language. I think any
 way we can incentivize fewer, better clients rather than many
 half-working clients we should do. However we are working now on the
 server-side consumer co-ordination so it should now be possible to
 write much simpler consumers.
 
 2. It would be great if someone put together a mailing list just for
 client developers to share tips, tricks, problems, and so on. We can
 make sure all the main contributors on this too. I think this could be
 a forum for kind of directing improvements in this area.
 
 3. Help improve the documentation on how to implement a client. We
 have tried to make the protocol spec not just a dry document but also
 have it share best practices, rationale, and intentions. I think this
 could potentially be even better as there is really a range of options
 from a very simple quick implementation to a more complex highly
 optimized version. It would be good to really document some of the
 options and tradeoffs.
 
 4. Come up with a standard way of documenting the features of clients.
 In an ideal world it would be possible to get the same information
 (author, language, feature set, download link, source code, etc) for
 all clients. It would be great to standardize the documentation for
 the client as well. For example having one or two basic examples that
 are repeated for every client in a standardized way. This would let
 someone come to the Kafka site who is not a java developer, and click
 on the link for their language and view examples of interacting with
 Kafka in the language they know using the client they would eventually
 use.
 
 5. Build a Kafka Client Compatibility Kit (KCCK) :-) The idea is this:
 anyone who wants to implement a client would implement a simple
 command line program with a set of standardized options. The
 compatibility kit would be a standard set of scripts that ran their
 client using this command line driver and validate its behavior. E.g.
 for a producer it would test that it 

[jira] [Updated] (KAFKA-1443) Add delete topic to topic commands and update DeleteTopicCommand

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1443:


Attachment: KAFKA-1443_2014-06-04_20:51:44.patch

 Add delete topic to topic commands and update DeleteTopicCommand
 

 Key: KAFKA-1443
 URL: https://issues.apache.org/jira/browse/KAFKA-1443
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
 Attachments: KAFKA-1443.patch, KAFKA-1443.patch, 
 KAFKA-1443_2014-06-04_20:51:44.patch


 Add delete topic option to current topic commands



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Failed to build Kafka using IntelliJ

[Timothy Chen]

It seems like that you generated your idea files you were on a
different scala version than what you configured it to be.

Can you try to set your scala version to 2.9.2 in your
gradle.properties, and do a ./gradlew clean idea, and try again?


Tim

On Tue, May 13, 2014 at 11:26 PM, Sheng Wang shengwan...@gmail.com wrote:
 Hi,

 I am trying to use IntelliJ to build Kafka.  I use java 1.7.0_55-b13 on Mac
 OS X Mavericks. The gradlew jar commands etc. worked fine.  However, I
 cannot use IntelliJ to build the project. I got some errors when I used
 IntelliJ 13.1 to build the project. Could anyone tell how I can solve this
 problem?

 Thanks!

 Information:Compilation completed with 38 errors and 7 warnings in 22 sec
 Error:scalac: Error:
 org.jetbrains.jps.incremental.scala.remote.ServerException
 Error compiling sbt component 'compiler-interface-2.8.0.final-51.0'
 at
 sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$apply$2.apply(AnalyzingCompiler.scala:145)
 at
 sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$apply$2.apply(AnalyzingCompiler.scala:142)
 at sbt.IO$.withTemporaryDirectory(IO.scala:285)
 at
 sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:142)
 at
 sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:139)
 at sbt.IO$.withTemporaryDirectory(IO.scala:285)
 at
 sbt.compiler.AnalyzingCompiler$.compileSources(AnalyzingCompiler.scala:139)
 at sbt.compiler.IC$.compileInterfaceJar(IncrementalCompiler.scala:33)
 at
 org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl$.org$jetbrains$jps$incremental$scala$local$CompilerFactoryImpl$$getOrCompileInterfaceJar(CompilerFactoryImpl.scala:86)
 at
 org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl$$anonfun$getScalac$1.apply(CompilerFactoryImpl.scala:43)
 at
 org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl$$anonfun$getScalac$1.apply(CompilerFactoryImpl.scala:42)
 at scala.Option.map(Option.scala:145)
 at
 org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl.getScalac(CompilerFactoryImpl.scala:42)
 at
 org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl.createCompiler(CompilerFactoryImpl.scala:21)
 at
 org.jetbrains.jps.incremental.scala.local.CachingFactory$$anonfun$createCompiler$1.apply(CachingFactory.scala:23)
 at
 org.jetbrains.jps.incremental.scala.local.CachingFactory$$anonfun$createCompiler$1.apply(CachingFactory.scala:23)
 at
 org.jetbrains.jps.incremental.scala.local.Cache$$anonfun$getOrUpdate$2.apply(Cache.scala:20)
 at scala.Option.getOrElse(Option.scala:120)
 at
 org.jetbrains.jps.incremental.scala.local.Cache.getOrUpdate(Cache.scala:19)
 at
 org.jetbrains.jps.incremental.scala.local.CachingFactory.createCompiler(CachingFactory.scala:22)
 at
 org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:21)
 at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.scala:64)
 at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:22)
 at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
 Warning:scalac:
 /var/folders/k2/_ksrnkxx6p502x6sb3v4jsvhgp/T/sbt_8d7d63cd/CompilerInterface.scala:161:
 error: object creation impossible, since method registerTopLevelSym in
 trait GlobalCompat of type (sym: this.Symbol)Unit is not defined
 Warning:scalac: new Compiler() with RangePositions // unnecessary in 2.11
 Warning:scalac: ^
 Warning:scalac:
 /var/folders/k2/_ksrnkxx6p502x6sb3v4jsvhgp/T/sbt_8d7d63cd/CompilerInterface.scala:165:
 error: class Compiler needs to be abstract, since method
 registerTopLevelSym in trait GlobalCompat of type (sym:
 Compiler.this.Symbol)Unit is not defined
 Warning:scalac: class Compiler extends CallbackGlobal(command.settings,
 dreporter, output)
 Warning:scalac:   ^
 Warning:scalac: two errors found
 /Users/aaa/git/kafka2/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
 Error:(21, 30) java: package kafka.javaapi.producer does not exist
 Error:(22, 22) java: package kafka.producer does not exist
 Error:(23, 22) java: package kafka.producer does not exist
 Error:(24, 19) java: cannot find symbol
   symbol:   class Utils
   location: package kafka.utils
 Error:(303, 39) java: cannot find symbol
   symbol:   class KeyedMessage
   location: class kafka.tools.KafkaMigrationTool.MigrationThread
 Error:(310, 57) java: cannot find symbol
   symbol:   class KeyedMessage
   location: class kafka.tools.KafkaMigrationTool.MigrationThread
 Error:(366, 39) java: cannot find symbol
   symbol:   class KeyedMessage
   

Review Request 21272: Patch for KAFKA-1443

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21272/
---

Review request for kafka.


Bugs: KAFKA-1443
https://issues.apache.org/jira/browse/KAFKA-1443


Repository: kafka


Description
---

Add delete topic option to topic commands


Diffs
-

  core/src/main/scala/kafka/admin/DeleteTopicCommand.scala 
804b331a7e4418219bc21ec920b98a0c3a21fac9 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
bdc72eaa9052b6f46bac510ab53c57d3283cdc25 

Diff: https://reviews.apache.org/r/21272/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1443) Add delete topic to topic commands and update DeleteTopicCommand

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993760#comment-13993760
 ] 

Timothy Chen commented on KAFKA-1443:
-

Created reviewboard https://reviews.apache.org/r/21272/
 against branch origin/trunk

 Add delete topic to topic commands and update DeleteTopicCommand
 

 Key: KAFKA-1443
 URL: https://issues.apache.org/jira/browse/KAFKA-1443
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
 Attachments: KAFKA-1443.patch, KAFKA-1443.patch


 Add delete topic option to current topic commands



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: How can I step through the Kafka code using a debugger

[Timothy Chen]

Hi Sheng Wang,

If you look at the Kafka website there is already a IDE setup page you
can follow.

Once your IDE is setup you can simply run Kafka via the Kafka main
method and set breakpoints in the broker.

Tim

On Tue, May 13, 2014 at 11:44 PM, Sheng Wang shengwan...@gmail.com wrote:
 Hi,

 I want to learn more about the Kafka code base. One of the easiest ways
 that I can think of is to walk through the code with a debugger. Could
 anyone tell how I can do that? Can I do it using any IDE?

 Thanks!

 -Sheng

Re: Review Request 21398: Fix KAFKA-1445 v2

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21398/#review42982
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
https://reviews.apache.org/r/21398/#comment76966

Got it!


- Timothy Chen


On May 13, 2014, 6:25 p.m., Guozhang Wang wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/21398/
 ---
 
 (Updated May 13, 2014, 6:25 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1445
 https://issues.apache.org/jira/browse/KAFKA-1445
 
 
 Repository: kafka
 
 
 Description
 ---
 
 0. Add the partitionsForNode index in Cluster;\n 1. Ready would return a list 
 of ready nodes instead of partitions;\n 2. Ready would also check if there is 
 any ready partitions with unknown leader, if yes indicate the 
 processReadyNode to force metadata refresh;\n 3. Drain would take a list of 
 nodes and drain the batches per node until the max request size is reached;\n 
 4. Collocate would not be just tranform batches per node into a producer 
 request;\n 5. Corresponding unit test changes; \n 6. One minor compilation 
 warning fix
 
 
 Diffs
 -
 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
  fbb732a57522109ac0e0eaf5c87b50cbd3a7f767 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  2d7e52d430fa267ee3689a06f8a621ce5dfd1e33 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 f0152fabbdd44e9f1a24291e84c17edf8379f4fc 
   clients/src/main/java/org/apache/kafka/common/Cluster.java 
 426bd1eec708979149cbd6fa3959e6f9e73c7e0e 
   clients/src/main/java/org/apache/kafka/common/Node.java 
 0e47ff3ff0e055823ec5a5aa4839d25b0fac8374 
   
 clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
  f37ab770b1794830154f9908a0156e7e99b4a458 
   
 clients/src/test/java/org/apache/kafka/common/utils/AbstractIteratorTest.java 
 1df226606fad29da47d81d0b8ff36209c3536c06 
 
 Diff: https://reviews.apache.org/r/21398/diff/
 
 
 Testing
 ---
 
 unit tests
 
 
 Thanks,
 
 Guozhang Wang
 

Re: Review Request 21398: Fix KAFKA-1445 v2

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21398/#review42869
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
https://reviews.apache.org/r/21398/#comment76775

Should we only call forceUpdate once if we get multiple unknown nodes?


- Timothy Chen


On May 13, 2014, 6:25 p.m., Guozhang Wang wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/21398/
 ---
 
 (Updated May 13, 2014, 6:25 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1445
 https://issues.apache.org/jira/browse/KAFKA-1445
 
 
 Repository: kafka
 
 
 Description
 ---
 
 0. Add the partitionsForNode index in Cluster;\n 1. Ready would return a list 
 of ready nodes instead of partitions;\n 2. Ready would also check if there is 
 any ready partitions with unknown leader, if yes indicate the 
 processReadyNode to force metadata refresh;\n 3. Drain would take a list of 
 nodes and drain the batches per node until the max request size is reached;\n 
 4. Collocate would not be just tranform batches per node into a producer 
 request;\n 5. Corresponding unit test changes; \n 6. One minor compilation 
 warning fix
 
 
 Diffs
 -
 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
  fbb732a57522109ac0e0eaf5c87b50cbd3a7f767 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  2d7e52d430fa267ee3689a06f8a621ce5dfd1e33 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 f0152fabbdd44e9f1a24291e84c17edf8379f4fc 
   clients/src/main/java/org/apache/kafka/common/Cluster.java 
 426bd1eec708979149cbd6fa3959e6f9e73c7e0e 
   clients/src/main/java/org/apache/kafka/common/Node.java 
 0e47ff3ff0e055823ec5a5aa4839d25b0fac8374 
   
 clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
  f37ab770b1794830154f9908a0156e7e99b4a458 
   
 clients/src/test/java/org/apache/kafka/common/utils/AbstractIteratorTest.java 
 1df226606fad29da47d81d0b8ff36209c3536c06 
 
 Diff: https://reviews.apache.org/r/21398/diff/
 
 
 Testing
 ---
 
 unit tests
 
 
 Thanks,
 
 Guozhang Wang
 

[jira] [Updated] (KAFKA-1443) Add delete topic to topic commands and update DeleteTopicCommand

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1443:


Attachment: KAFKA-1443.patch

 Add delete topic to topic commands and update DeleteTopicCommand
 

 Key: KAFKA-1443
 URL: https://issues.apache.org/jira/browse/KAFKA-1443
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
 Attachments: KAFKA-1443.patch, KAFKA-1443.patch


 Add delete topic option to current topic commands



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-06_13:21:14.patch

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
 KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch, 
 KAFKA-1384_2014-05-05_17:14:57.patch, KAFKA-1384_2014-05-06_13:21:14.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 6, 2014, 8:21 p.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
401bf1ea353d30353f9bae4a396302b15aa059c6 
  core/src/main/scala/kafka/log/LogManager.scala 
ab72cffd439c867051247e955dde2b6ade6836b8 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
be1a1ee3ac5c1dd8d658deabcfc740e88e053288 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
a78f7cfa025b0c5d968381b6d5dc7aec894a612b 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
41ebc7acae2efdd8a58964aa746c90fa71ac9ba4 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13991080#comment-13991080
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
 KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch, 
 KAFKA-1384_2014-05-05_17:14:57.patch, KAFKA-1384_2014-05-06_13:21:14.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 5, 2014, 6:05 p.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989785#comment-13989785
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
 KAFKA-1384_2014-05-05_11:05:15.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

 On May 5, 2014, 5:12 p.m., Jun Rao wrote:
  core/src/main/scala/kafka/utils/ShutdownableThread.scala, lines 28-33
  https://reviews.apache.org/r/20745/diff/6/?file=573313#file573313line28
 
  I thought you plan to remove isShuttingDown?

Odd I don't have this anymore in my box, but end up in my patch


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review42058
---


On May 2, 2014, 8:38 p.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20745/
 ---
 
 (Updated May 2, 2014, 8:38 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1397
 https://issues.apache.org/jira/browse/KAFKA-1397
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1397: Fix delete topic tests and deadlock
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/AdminUtils.scala 
 36ddeb44490e8343a4e8056c45726ac660e4b2f9 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
   core/src/main/scala/kafka/controller/KafkaController.scala 
 933de9dd324c7086efe6aa610335ef370d9e9c12 
   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
 0e47dac8cbf65a86d053a3371a18af467afd70ae 
   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
 e4bc2439ce1933c7c7571d255464ee678226a6cb 
   core/src/main/scala/kafka/log/LogManager.scala 
 ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 11c20cee83fda9a492156674d351a0096b13fd99 
   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
 cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
 9c29e144bba2c9bafa91941b6ca5c263490693b3 
   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
 5c487968014b56490eb2bc876cef1c52efd1cdad 
   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
 014e9644dba1d65142f6a9abc745858870a46230 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 00bfba41fe008fe72c17fb788e3cada3daf8eb30 
 
 Diff: https://reviews.apache.org/r/20745/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989793#comment-13989793
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
 KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
 KAFKA-1397_2014-05-02_13:38:02.patch, KAFKA-1397_2014-05-05_11:17:59.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-05_11:17:59.patch

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
 KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
 KAFKA-1397_2014-05-02_13:38:02.patch, KAFKA-1397_2014-05-05_11:17:59.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review42199
---



core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala
https://reviews.apache.org/r/20745/#comment75941

Makes sense!


- Timothy Chen


On May 5, 2014, 9 p.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20745/
 ---
 
 (Updated May 5, 2014, 9 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1397
 https://issues.apache.org/jira/browse/KAFKA-1397
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1397: Fix delete topic tests and deadlock
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/AdminUtils.scala 
 36ddeb44490e8343a4e8056c45726ac660e4b2f9 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
   core/src/main/scala/kafka/controller/KafkaController.scala 
 933de9dd324c7086efe6aa610335ef370d9e9c12 
   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
 0e47dac8cbf65a86d053a3371a18af467afd70ae 
   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
 e4bc2439ce1933c7c7571d255464ee678226a6cb 
   core/src/main/scala/kafka/log/LogManager.scala 
 ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 11c20cee83fda9a492156674d351a0096b13fd99 
   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
 cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
 9c29e144bba2c9bafa91941b6ca5c263490693b3 
   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
 5c487968014b56490eb2bc876cef1c52efd1cdad 
   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
 014e9644dba1d65142f6a9abc745858870a46230 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 00bfba41fe008fe72c17fb788e3cada3daf8eb30 
 
 Diff: https://reviews.apache.org/r/20745/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 5, 2014, 9 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
014e9644dba1d65142f6a9abc745858870a46230 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
00bfba41fe008fe72c17fb788e3cada3daf8eb30 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989968#comment-13989968
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
 KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
 KAFKA-1397_2014-05-02_13:38:02.patch, KAFKA-1397_2014-05-05_11:17:59.patch, 
 KAFKA-1397_2014-05-05_14:00:29.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

 On May 5, 2014, 8:51 p.m., Jun Rao wrote:
  Compilation error when trying to run the unit tests.
  
  kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:35:
   not enough arguments for constructor LogManager: (logDirs: 
  Array[java.io.File],topicConfigs: 
  scala.collection.Map[String,kafka.log.LogConfig],defaultConfig: 
  kafka.log.LogConfig,cleanerConfig: kafka.log.CleanerConfig,flushCheckMs: 
  Long,flushCheckpointMs: Long,retentionCheckMs: Long,scheduler: 
  kafka.utils.Scheduler,brokerState: kafka.server.BrokerState,time: 
  kafka.utils.Time)kafka.log.LogManager.
  Unspecified value parameter brokerState.
val logManagers = configs.map(config = new LogManager(logDirs = 
  config.logDirs.map(new File(_)).toArray,
^
  /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:47:
   value logDirs is not a member of Nothing
  for(manager - logManagers; dir - manager.logDirs)
 ^
  /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:67:
   value createLog is not a member of Nothing
  val log0 = logManagers(0).createLog(TopicAndPartition(topic, 0), 
  LogConfig())
^
  /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:106:
   value createLog is not a member of Nothing
  val topic1Log0 = logManagers(0).createLog(TopicAndPartition(topic1, 0), 
  LogConfig())
  ^
  /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:122:
   value createLog is not a member of Nothing
  val topic2Log0 = logManagers(0).createLog(TopicAndPartition(topic2, 0), 
  LogConfig())
  ^
  /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:64:
   not enough arguments for constructor LogManager: (logDirs: 
  Array[java.io.File],topicConfigs: 
  scala.collection.Map[String,kafka.log.LogConfig],defaultConfig: 
  kafka.log.LogConfig,cleanerConfig: kafka.log.CleanerConfig,flushCheckMs: 
  Long,flushCheckpointMs: Long,retentionCheckMs: Long,scheduler: 
  kafka.utils.Scheduler,brokerState: kafka.server.BrokerState,time: 
  kafka.utils.Time)kafka.log.LogManager.
  Unspecified value parameter brokerState.
  return new LogManager(logDirs,
 

Ah sorry, oddly I was able to run it but didn't try out all the tests. Fixed it 
in the next patch.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/#review42197
---


On May 5, 2014, 9:25 p.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20718/
 ---
 
 (Updated May 5, 2014, 9:25 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1384
 https://issues.apache.org/jira/browse/KAFKA-1384
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1384: Logging kafka state metric
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/controller/KafkaController.scala 
 933de9dd324c7086efe6aa610335ef370d9e9c12 
   core/src/main/scala/kafka/log/Log.scala 
 b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
   core/src/main/scala/kafka/log/LogManager.scala 
 ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
   core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 c208f83bed7fb91f07fae42f2b66892e6d46fecc 
   core/src/main/scala/kafka/server/KafkaServerStartable.scala 
 acda52b801714bcc182edc0ced925f0e4b493fc1 
   core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
 be1a1ee3ac5c1dd8d658deabcfc740e88e053288 
   core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
 a78f7cfa025b0c5d968381b6d5dc7aec894a612b 
   core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
 41ebc7acae2efdd8a58964aa746c90fa71ac9ba4 
 
 Diff: https://reviews.apache.org/r/20718/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989994#comment-13989994
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
 KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13990123#comment-13990123
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
 KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch, 
 KAFKA-1384_2014-05-05_17:14:57.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-05_17:14:57.patch

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
 KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch, 
 KAFKA-1384_2014-05-05_17:14:57.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 6, 2014, 12:15 a.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
be1a1ee3ac5c1dd8d658deabcfc740e88e053288 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
a78f7cfa025b0c5d968381b6d5dc7aec894a612b 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
41ebc7acae2efdd8a58964aa746c90fa71ac9ba4 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-02_12:13:24.patch

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13988111#comment-13988111
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

 On May 2, 2014, 12:26 a.m., Jun Rao wrote:
  core/src/main/scala/kafka/utils/ShutdownableThread.scala, lines 35-36
  https://reviews.apache.org/r/20745/diff/5/?file=573213#file573213line35
 
  We probably should check the return value of initiateShutdown().

I was thinking that even with concurrent calls since all calls end up just 
waiting on the shutdownLatch it should be safe to not check it.
Therefore shutdown call regardless of what state will end up waiting until the 
thread shutdown is completed.


 On May 2, 2014, 12:26 a.m., Jun Rao wrote:
  core/src/main/scala/kafka/utils/ShutdownableThread.scala, lines 40-41
  https://reviews.apache.org/r/20745/diff/5/?file=573213#file573213line40
 
  Instead of introducing isShuttingDown, could we just to compareAndSet 
  on isRunning?

Good point, consolidated it.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review41979
---


On May 2, 2014, 1:12 a.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20745/
 ---
 
 (Updated May 2, 2014, 1:12 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1397
 https://issues.apache.org/jira/browse/KAFKA-1397
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1397: Fix delete topic tests and deadlock
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/admin/AdminUtils.scala 
 36ddeb44490e8343a4e8056c45726ac660e4b2f9 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
   core/src/main/scala/kafka/controller/KafkaController.scala 
 933de9dd324c7086efe6aa610335ef370d9e9c12 
   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
 0e47dac8cbf65a86d053a3371a18af467afd70ae 
   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
 e4bc2439ce1933c7c7571d255464ee678226a6cb 
   core/src/main/scala/kafka/log/LogManager.scala 
 ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 11c20cee83fda9a492156674d351a0096b13fd99 
   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
 cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
 9c29e144bba2c9bafa91941b6ca5c263490693b3 
   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
 5c487968014b56490eb2bc876cef1c52efd1cdad 
   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
 014e9644dba1d65142f6a9abc745858870a46230 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 49c7790c995bb2e79322eb148ab80d0dcccefed4 
 
 Diff: https://reviews.apache.org/r/20745/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 2, 2014, 8:38 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
014e9644dba1d65142f6a9abc745858870a46230 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
00bfba41fe008fe72c17fb788e3cada3daf8eb30 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-02_13:38:02.patch

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
 KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
 KAFKA-1397_2014-05-02_13:38:02.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13988206#comment-13988206
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
 KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
 KAFKA-1397_2014-05-02_13:38:02.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

 On May 1, 2014, 6:26 p.m., Jun Rao wrote:
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala, lines 
  394-396
  https://reviews.apache.org/r/20745/diff/4/?file=571881#file571881line394
 
  It seems this test is not necessary since the outer loop will detect 
  that too.

It's actually necessary since isRunning might been set to false while it woke 
up from the condition, in that case we don't want it to still try to acquire 
the controller lock but to bail early.


 On May 1, 2014, 6:26 p.m., Jun Rao wrote:
  core/src/main/scala/kafka/server/ReplicaManager.scala, lines 128-136
  https://reviews.apache.org/r/20745/diff/4/?file=571885#file571885line128
 
  Do we still need to do this now that we make sure that if a replica is 
  deleted, the controller will never send a LeaderAndIsrRequest to this 
  replica until the topic is deleted?

We need this as described in the comment that if a broker that was shutdown 
during the topic deletion and brought back up, the 
stopReplicaRequest(deletePartition=true) without this extra log manager cleanup 
will never delete the log and the folders associated with as ReplicaManager 
doesn't hold that information anymore as we don't send the leader and isr 
request again as part of the retry.


 On May 1, 2014, 6:26 p.m., Jun Rao wrote:
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala, lines 86-94
  https://reviews.apache.org/r/20745/diff/4/?file=571886#file571886line86
 
  This is not going to be reliable since we can't be sure whether the 
  controller has completed the deletion before the failover.

Ok removing the test for now.


 On May 1, 2014, 6:26 p.m., Jun Rao wrote:
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala, lines 239-240
  https://reviews.apache.org/r/20745/diff/4/?file=571886#file571886line239
 
  This test seems to be the same as 
  testPartitionReassignmentDuringDeleteTopic().

Just the order of requests, but as well cannot gurantee what is processed first 
as they're just asynchronous. Will remove.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review41904
---


On April 30, 2014, 9:55 p.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20745/
 ---
 
 (Updated April 30, 2014, 9:55 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1397
 https://issues.apache.org/jira/browse/KAFKA-1397
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1397: Fix delete topic tests and deadlock
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
   core/src/main/scala/kafka/controller/KafkaController.scala 
 933de9dd324c7086efe6aa610335ef370d9e9c12 
   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
 0e47dac8cbf65a86d053a3371a18af467afd70ae 
   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
 e4bc2439ce1933c7c7571d255464ee678226a6cb 
   core/src/main/scala/kafka/log/LogManager.scala 
 ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 0b668f230c8556fdf08654ce522a11847d0bf39b 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 c208f83bed7fb91f07fae42f2b66892e6d46fecc 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 11c20cee83fda9a492156674d351a0096b13fd99 
   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
 9c29e144bba2c9bafa91941b6ca5c263490693b3 
   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
 5c487968014b56490eb2bc876cef1c52efd1cdad 
   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
 b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 384c74e7c3985abff864e61dea5e530dbd189d5d 
 
 Diff: https://reviews.apache.org/r/20745/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 1, 2014, 10:54 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
49c7790c995bb2e79322eb148ab80d0dcccefed4 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-01_15:53:57.patch

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
 KAFKA-1397_2014-05-01_15:53:57.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987127#comment-13987127
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
 KAFKA-1397_2014-05-01_15:53:57.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 2, 2014, 1:12 a.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
014e9644dba1d65142f6a9abc745858870a46230 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
49c7790c995bb2e79322eb148ab80d0dcccefed4 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-01_18:12:24.patch

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
 KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

 On May 2, 2014, 1:09 a.m., Jun Rao wrote:
  core/src/main/scala/kafka/server/KafkaServer.scala, lines 108-109
  https://reviews.apache.org/r/20718/diff/2/?file=569132#file569132line108
 
  Will that override the state to broker even though it's set to 
  controller?

The controller only sets the state when it is elected, and from the server 
startup steps it will set as broker state first, and once the controller starts 
up and the elect happens then it will be set to controller state.


 On May 2, 2014, 1:09 a.m., Jun Rao wrote:
  core/src/main/scala/kafka/server/KafkaServerStartable.scala, lines 55-58
  https://reviews.apache.org/r/20718/diff/2/?file=569133#file569133line55
 
  What's the intention of this method?

This is to allow custom state that is not defined in the enums. One example is 
in our custom kafka server startable we can inject more states that is specific 
to different use cases.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/#review41993
---


On April 26, 2014, 7:09 a.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20718/
 ---
 
 (Updated April 26, 2014, 7:09 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1384
 https://issues.apache.org/jira/browse/KAFKA-1384
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1384: Logging kafka state metric
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/controller/KafkaController.scala 
 933de9dd324c7086efe6aa610335ef370d9e9c12 
   core/src/main/scala/kafka/log/Log.scala 
 f20c232e2daa63a10d91b965af52801af656477c 
   core/src/main/scala/kafka/log/LogManager.scala 
 ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
   core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 c208f83bed7fb91f07fae42f2b66892e6d46fecc 
   core/src/main/scala/kafka/server/KafkaServerStartable.scala 
 acda52b801714bcc182edc0ced925f0e4b493fc1 
 
 Diff: https://reviews.apache.org/r/20718/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 2, 2014, 1:47 a.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987259#comment-13987259
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-01_18:47:03.patch

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
 KAFKA-1384_2014-05-01_18:47:03.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

 On April 29, 2014, 5:02 p.m., Jun Rao wrote:
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala, lines 
  209-212
  https://reviews.apache.org/r/20745/diff/3/?file=569935#file569935line209
 
  Is it better to do the check here or in the caller?

Hmm, at first glance not sure why it makes sense to propagate 
LeaderAndIsrRequest when a partition is being deleted, but moving it just to 
the delete topic path is probably the least impact. I think I'll move it to the 
caller for now.


 On April 29, 2014, 5:02 p.m., Jun Rao wrote:
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala, lines 
  385-386
  https://reviews.apache.org/r/20745/diff/3/?file=569938#file569938line385
 
  Do we need to check here at all since the caller of doWork() already 
  does the check?

Good point, since isRunning must be set to false before it can shut down this 
is redudant.


 On April 29, 2014, 5:02 p.m., Jun Rao wrote:
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala, lines 
  412-413
  https://reviews.apache.org/r/20745/diff/3/?file=569938#file569938line412
 
  Do we need to introduce isAnyReplicaInState? replicasInState() seems 
  more general and we can just check the output to implement 
  isAnyReplicaInState.

I added this because the logic here only want to see if there is any replica 
that is in a specific state, and thought returning the whole list is a waste as 
I see internally it's constructing a new set. From profiling other Scala code I 
see that any set constructing is quite expensive as it's using immutable sets. 
It is more clean to only expose one API, but I thought the trade off might be 
worth it. More thoughts?


 On April 29, 2014, 5:02 p.m., Jun Rao wrote:
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala, lines 86-91
  https://reviews.apache.org/r/20745/diff/3/?file=569942#file569942line86
 
  There seems to be no guarantee that the delete topic process is 
  completed before the controller was shutdown. So, I am not sure how 
  reliable the test is.

It's actually the reverse, where it tries to shutdown controller while the 
delete topic process is still in progress. Currently there is no way to 
gurantee that the delete topic can halt until some condition happens, so it 
might be not guranteed.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review41714
---


On April 29, 2014, 12:08 a.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20745/
 ---
 
 (Updated April 29, 2014, 12:08 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1397
 https://issues.apache.org/jira/browse/KAFKA-1397
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Fix delete topic tests and deadlock
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
   core/src/main/scala/kafka/controller/KafkaController.scala 
 933de9dd324c7086efe6aa610335ef370d9e9c12 
   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
 0e47dac8cbf65a86d053a3371a18af467afd70ae 
   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
 e4bc2439ce1933c7c7571d255464ee678226a6cb 
   core/src/main/scala/kafka/log/LogManager.scala 
 ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 0b668f230c8556fdf08654ce522a11847d0bf39b 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 c208f83bed7fb91f07fae42f2b66892e6d46fecc 
   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
 9c29e144bba2c9bafa91941b6ca5c263490693b3 
   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
 b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 
 
 Diff: https://reviews.apache.org/r/20745/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated April 30, 2014, 9:55 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description (updated)
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0b668f230c8556fdf08654ce522a11847d0bf39b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
384c74e7c3985abff864e61dea5e530dbd189d5d 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986142#comment-13986142
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-04-30_14:55:28.patch

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

 On April 28, 2014, 7:43 p.m., Neha Narkhede wrote:
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala, line 
  210
  https://reviews.apache.org/r/20745/diff/1/?file=569160#file569160line210
 
  better to not leak the logic from delete topic manager here. It is 
  worth exposing the check as an API in DeleteTopicManager. In any case, can 
  you explain the motivation behind your change?

Sorry forgot to add more comments to the rb, have been chatting mostly with Jun 
about my fixes.
One test that was failing in delete topic test was a test that was handling 
request while topic being deleted, and it made sure it can never finish by 
shutting down one of the follower broker. Sending a fetch request to a topic 
being deleted returns a NotLeaderForPartition error code instead of expected 
UnknownTopicOrPartition.
 
The cause of this is that as part of the metadata cache changes in Kafka api, 
we removed the check if a topic exists in the cache when serving fetching 
request and now it tries to read message set from the replica manager, and even 
though a stop replica request with delete partition is already been sent and 
processed, the partition was somehow still in replica manager and it returns 
NotLeaderForPartition exception since there is no leader for this topic.

I found out that the partition was being recreated in the replica manager, 
because in the process of retrying deleting the same topic it change the failed 
replica state into OfflineReplica again that triggers a LeaderAndIsrRequest to 
all brokers for shrinking the Isr for that replica. 

The brokers that received the LeaderAndIsrRequest recreates the partition with 
getOrCreatePartition, and therefore future fetch requests hits it and throws 
that error code.

I think the correct thing to do is to not send LeaderAndIsrRequest if the topic 
is being deleted when bring replica to an offline state, as it is not necessary 
anymore.


 On April 28, 2014, 7:43 p.m., Neha Narkhede wrote:
  core/src/main/scala/kafka/controller/KafkaController.scala, line 652
  https://reviews.apache.org/r/20745/diff/1/?file=569161#file569161line652
 
  did you make this change since onControllerResignation already holds 
  the same lock? 
  
  In general, for tricky bugs like deadlock, it is a good idea to explain 
  all the significant changes, so it is easier to review.

The latest fix for the deadlock decouples the await topic deletion with a new 
lock internal to delete topic manager, so that during the shutdown the await on 
the topic deletion condition and resume since it's not based on the controller 
lock anymore.

We're still seeing occasional deadlock hanging problem, and with investigation 
Jun and I believe it's because if the topic deletion resumes and right before 
it acquires the controller lock, if a shutdown happens and acquired the lock, 
the delete topic thread will be waiting forever as the shutdown also blocks and 
waits.

There are a variety of fixes, but I felt one of the cleaner fix is to shutdown 
the delete topic manager outside of the controller lock as it doesn't really 
require it anymore. Once delete topic thread shuts down then we can proceed 
holding the controller lock and finish the broker shutdown. 


 On April 28, 2014, 7:43 p.m., Neha Narkhede wrote:
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala, line 403
  https://reviews.apache.org/r/20745/diff/1/?file=569163#file569163line403
 
  * delete topic wasn't initiated for the given topic
  
  Nevertheless, it seems like the loop goes over topics that are queued 
  up for deletion, so is this comment right?

There isn't really actual bug here, but I was trying to fix the comments and 
logic as if a topic was queued to be deleted and it's the first time calling 
delete topic thread, the current code and comment seems to suggest all topics 
that are queued that is not in completed has one failed replica. But it could 
be that it is just queued up to be deleted the first time.

The effect is that in the logs you will see that the current code will log that 
it attempts to retry topic deletion again, while it's just being deleted the 
first time. 

I was trying to say that this is the first time the topic has been queued for 
deletion and not resumed to be completed or retried. I'll try to rewrite it.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review41626
---


On April 27, 2014, 6:43 a.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20745/
 ---
 
 (Updated April 27, 2014, 6:43 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1397
 https

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated April 28, 2014, 9:48 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0b668f230c8556fdf08654ce522a11847d0bf39b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-04-28_14:48:32.patch

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983619#comment-13983619
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated April 29, 2014, 12:08 a.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0b668f230c8556fdf08654ce522a11847d0bf39b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-04-28_17:08:49.patch

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983759#comment-13983759
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
 KAFKA-1397_2014-04-28_17:08:49.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

Fix delete topic tests and deadlock


Diffs
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0b668f230c8556fdf08654ce522a11847d0bf39b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397.patch

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982206#comment-13982206
 ] 

Timothy Chen commented on KAFKA-1397:
-

Created reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

 delete topic is not working 
 

 Key: KAFKA-1397
 URL: https://issues.apache.org/jira/browse/KAFKA-1397
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao
 Attachments: KAFKA-1397.patch


 All unit tests are disabled since they hang transiently (see details in 
 KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

 On April 25, 2014, 6:35 p.m., Joel Koshy wrote:
  core/src/main/scala/kafka/server/KafkaServerStartable.scala, line 55
  https://reviews.apache.org/r/20718/diff/1/?file=568568#file568568line55
 
  Unused

This is actually exposed setting custom state that is not one of the defined 
enums. One example is for our internal custom kafka server startable override 
can allow more states that is not defined in BrokerStates. The alternative is 
to just come up with all possible enums and let that be the only option 
available, but I was thinking that might be a bit too limiting. I'm not sure if 
this approach is too permissive either, you have more thoughts?


 On April 25, 2014, 6:35 p.m., Joel Koshy wrote:
  core/src/main/scala/kafka/server/BrokerStates.scala, line 28
  https://reviews.apache.org/r/20718/diff/1/?file=568566#file568566line28
 
  Just a thought: I'm wondering if it is better to have a bit-vector 
  approach for states; although that will limit the number of possible states 
  - but I think that is fine.
  
  The main reason for this is that it will enable composing simultaneous 
  states. E.g., in this approach you cannot distinguish state 5 from state 3 
  (if a shutting down broker is the controller). Although we can probably 
  infer that from the fact that the other brokers are likely in state (2) or 
  by looking at the active controller count separately. Still, we currently 
  allow more than one broker to shut down.
  
  It also _might_ help catch erroneous dual states (due to bugs).
  
  What you have is probably fine for lifecycle states (except for the 
  above caveat). However, if we ever want to allow more-than-lifecycle states 
  (e.g., under-replicated is a state we _might_ want to include on this - 
  even though we have a separate URP mbean and it's not a lifecycle state; 
  another example is loading consumer offsets).

Erroneous dual states is something I thought of, especially when state changes 
rapidly it will be harder to detect.
The down side of having a bit field is also that you need to then also have to 
unset that state when you're going into another lifecycle state.

I think coming back to the high level of what we want for broker states, since 
really a broker should really just be in a state at one point of time, setting 
a particular state I think makes more sense.

Composite states probably makes more sense for different component states. 


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/#review41501
---


On April 25, 2014, 5:25 p.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20718/
 ---
 
 (Updated April 25, 2014, 5:25 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1384
 https://issues.apache.org/jira/browse/KAFKA-1384
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1384: Logging kafka state metric
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/controller/KafkaController.scala 
 933de9dd324c7086efe6aa610335ef370d9e9c12 
   core/src/main/scala/kafka/log/Log.scala 
 46df8d99d977a3b010a9b9f4698187fa9bfb2498 
   core/src/main/scala/kafka/log/LogManager.scala 
 ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
   core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
   core/src/main/scala/kafka/server/KafkaServer.scala 
 c208f83bed7fb91f07fae42f2b66892e6d46fecc 
   core/src/main/scala/kafka/server/KafkaServerStartable.scala 
 acda52b801714bcc182edc0ced925f0e4b493fc1 
 
 Diff: https://reviews.apache.org/r/20718/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated April 26, 2014, 7:09 a.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
f20c232e2daa63a10d91b965af52801af656477c 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981909#comment-13981909
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-04-26_00:09:56.patch

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20713: Patch for KAFKA-1424

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20713/#review41473
---

Ship it!


Ship It!

- Timothy Chen


On April 25, 2014, 2:54 p.m., Jun Rao wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20713/
 ---
 
 (Updated April 25, 2014, 2:54 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1424
 https://issues.apache.org/jira/browse/KAFKA-1424
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Wait metadata is propagated before fetching.
 
 
 Diffs
 -
 
   core/src/test/scala/unit/kafka/producer/ProducerTest.scala 
 dc6a5ea4abcdff370f3b61c774bbde4e4157f412 
 
 Diff: https://reviews.apache.org/r/20713/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jun Rao
 

Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
46df8d99d977a3b010a9b9f4698187fa9bfb2498 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981272#comment-13981272
 ] 

Timothy Chen commented on KAFKA-1384:
-

I'm planning just to expose it via JMX, as we don't have a need to write it to 
zookeeper as other brokers doesn't have a need to query it.

Less contention to ZK as well, you have anything in mind why you thought 
putting it in ZK would be good?


 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen

 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981275#comment-13981275
 ] 

Timothy Chen commented on KAFKA-1384:
-

Created reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384.patch

 Log Broker state 
 -

 Key: KAFKA-1384
 URL: https://issues.apache.org/jira/browse/KAFKA-1384
 Project: Kafka
  Issue Type: New Feature
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1384.patch


 Currently we don't have visibility into what state the broker is currently 
 in, ie: Startup - Running - Waiting Controlled shutdown - Shutting down
 So without knowing what state the broker it is it's hard to figure out what 
 the current broker is performing.
 This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Apache Kafka 0.8.1.1 Release Candidate 1

[Timothy Chen]

+1 (Non-binding).

Tim

On Thu, Apr 24, 2014 at 5:32 PM, Joel Koshy jjkosh...@gmail.com wrote:
 +1

 One (non-critical) issue is that I think our packaging isn't complete
 for kafka-perf (we are only building for 2.8.0).

 On Tue, Apr 22, 2014 at 07:18:38PM -0400, Joe Stein wrote:
 This is the first candidate for release of Apache Kafka 0.8.1.1

 Release Notes for the 0.8.1.1 release
 https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/RELEASE_NOTES.html

 *** Please download, test and vote by Friday, April 25th, 4pm PT

 Kafka's KEYS file containing PGP keys we use to sign the release:
 https://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5, sha1
 and sha2 (SHA256) checksum.

 * Release artifacts to be voted upon (source and binary):
 https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/

 * Maven artifacts to be voted upon prior to release:
 https://repository.apache.org/content/groups/staging/

 * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1.1 tag
 https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=34237371b662f617dd80c75cd6282f97e047ef56


 /***
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
 /

Re: Review Request 20540: Patch for KAFKA-1410

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20540/#review41039
---



core/src/main/scala/kafka/server/KafkaApis.scala
https://reviews.apache.org/r/20540/#comment74401

Do we no longer need to check for offset management topic?


- Timothy Chen


On April 22, 2014, 4:37 p.m., Jun Rao wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20540/
 ---
 
 (Updated April 22, 2014, 4:37 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1410
 https://issues.apache.org/jira/browse/KAFKA-1410
 
 
 Repository: kafka
 
 
 Description
 ---
 
 1. Move MetadataCache to its own class. 2. Avoid updating metadata cache if 
 the controller epoch is old.
 
 
 1. Remove ensureTopicExists(). 2. Move the update of controller epoc in topic 
 metadata request to ReplicaManager.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 1a4ffcea37e32739b8360956b4a40c0101b97822 
   core/src/main/scala/kafka/server/MetadataCache.scala PRE-CREATION 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 5588f59783d5cd8f074263c207045625282af12d 
   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
 17b08e14f5e9bfec1233c80d74b50bc2ec38aa25 
 
 Diff: https://reviews.apache.org/r/20540/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jun Rao
 

Re: Review Request 20540: Patch for KAFKA-1410

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20540/#review41067
---

Ship it!


Ship It!

- Timothy Chen


On April 22, 2014, 8:51 p.m., Jun Rao wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20540/
 ---
 
 (Updated April 22, 2014, 8:51 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1410
 https://issues.apache.org/jira/browse/KAFKA-1410
 
 
 Repository: kafka
 
 
 Description
 ---
 
 1. Handle the creation of OffsetTopic properly.
 
 
 1. Move MetadataCache to its own class. 2. Avoid updating metadata cache if 
 the controller epoch is old.
 
 
 1. Remove ensureTopicExists(). 2. Move the update of controller epoc in topic 
 metadata request to ReplicaManager.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 1a4ffcea37e32739b8360956b4a40c0101b97822 
   core/src/main/scala/kafka/server/MetadataCache.scala PRE-CREATION 
   core/src/main/scala/kafka/server/ReplicaManager.scala 
 5588f59783d5cd8f074263c207045625282af12d 
   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
 17b08e14f5e9bfec1233c80d74b50bc2ec38aa25 
 
 Diff: https://reviews.apache.org/r/20540/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jun Rao
 

Re: Review Request 20536: Patch for KAFKA-1380

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20536/#review40959
---



core/src/main/scala/kafka/server/KafkaApis.scala
https://reviews.apache.org/r/20536/#comment74283

Isn't this a unnecessary check? The parent if check already checks the flag.


- Timothy Chen


On April 21, 2014, 9:34 p.m., Joel Koshy wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20536/
 ---
 
 (Updated April 21, 2014, 9:34 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1380
 https://issues.apache.org/jira/browse/KAFKA-1380
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1356; follow-up - return unknown topic partition on non-existent topic 
 if auto.create is off
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 9e569eb8c3586b2fdb3b5904c5fa5f938cffada1 
 
 Diff: https://reviews.apache.org/r/20536/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Joel Koshy
 

Re: Review Request 20534: Patch for KAFKA-1380

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20534/#review40968
---

Ship it!


Ship It!

- Timothy Chen


On April 21, 2014, 9:47 p.m., Joel Koshy wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20534/
 ---
 
 (Updated April 21, 2014, 9:47 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1380
 https://issues.apache.org/jira/browse/KAFKA-1380
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1356; follow-up - return unknown topic partition on non-existent topic 
 if auto.create is off
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 9e569eb8c3586b2fdb3b5904c5fa5f938cffada1 
 
 Diff: https://reviews.apache.org/r/20534/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Joel Koshy
 

Review Request 20252: Patch for KAFKA-1356

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20252/
---

Review request for kafka.


Bugs: KAFKA-1356
https://issues.apache.org/jira/browse/KAFKA-1356


Repository: kafka


Description
---

WIP


Diffs
-

  core/src/main/scala/kafka/api/TopicMetadata.scala 
0513a59ed94e556894b3515dc38666ee9a66ae3d 
  core/src/main/scala/kafka/controller/KafkaController.scala 
d6c03218ec0afdb65a7b5aa9b85eb1c029357d7a 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0f137c5136e2320ca27c285d6ab013f6559314c4 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 
d5644ea40ec7678b975c4775546b79fcfa9f64b7 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
1317b4c3c60b8d1835dd6a06bf9b250398f0d47d 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
500eeca2f95d901536b1363b8c4b485c4893179f 

Diff: https://reviews.apache.org/r/20252/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20252: Patch for KAFKA-1356

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20252/
---

(Updated April 11, 2014, 7:41 a.m.)


Review request for kafka.


Bugs: KAFKA-1356
https://issues.apache.org/jira/browse/KAFKA-1356


Repository: kafka


Description
---

WIP


Diffs
-

  core/src/main/scala/kafka/api/TopicMetadata.scala 
0513a59ed94e556894b3515dc38666ee9a66ae3d 
  core/src/main/scala/kafka/controller/KafkaController.scala 
d6c03218ec0afdb65a7b5aa9b85eb1c029357d7a 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0f137c5136e2320ca27c285d6ab013f6559314c4 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 
d5644ea40ec7678b975c4775546b79fcfa9f64b7 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
1317b4c3c60b8d1835dd6a06bf9b250398f0d47d 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
500eeca2f95d901536b1363b8c4b485c4893179f 

Diff: https://reviews.apache.org/r/20252/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20252: Patch for KAFKA-1356

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20252/
---

(Updated April 11, 2014, 7:42 a.m.)


Review request for kafka.


Bugs: KAFKA-1356
https://issues.apache.org/jira/browse/KAFKA-1356


Repository: kafka


Description (updated)
---

Topic metadata requests takes too long to process


Diffs
-

  core/src/main/scala/kafka/api/TopicMetadata.scala 
0513a59ed94e556894b3515dc38666ee9a66ae3d 
  core/src/main/scala/kafka/controller/KafkaController.scala 
d6c03218ec0afdb65a7b5aa9b85eb1c029357d7a 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0f137c5136e2320ca27c285d6ab013f6559314c4 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 
d5644ea40ec7678b975c4775546b79fcfa9f64b7 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
1317b4c3c60b8d1835dd6a06bf9b250398f0d47d 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
500eeca2f95d901536b1363b8c4b485c4893179f 

Diff: https://reviews.apache.org/r/20252/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1356) Topic metadata requests takes too long to process

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1356:


Attachment: KAFKA-1356.patch

 Topic metadata requests takes too long to process
 -

 Key: KAFKA-1356
 URL: https://issues.apache.org/jira/browse/KAFKA-1356
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1356.patch, KAFKA-1356.patch, KAFKA-1356.patch, 
 KAFKA-1356_2014-04-02_18:39:36.patch, KAFKA-1356_2014-04-04_14:40:18.patch, 
 KAFKA-1356_2014-04-04_17:45:37.patch, KAFKA-1356_2014-04-06_01:45:47.patch, 
 KAFKA-1356_2014-04-08_01:38:23.patch


 Currently we're seeing slow response times in handling get topic metadata 
 requests.
 Local testing shows that even locally it takes 300 avg ms to respond, even 
 though it's not doing any IO operations.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1356) Topic metadata requests takes too long to process

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966312#comment-13966312
 ] 

Timothy Chen commented on KAFKA-1356:
-

Created reviewboard https://reviews.apache.org/r/20252/
 against branch origin/0.8.1

 Topic metadata requests takes too long to process
 -

 Key: KAFKA-1356
 URL: https://issues.apache.org/jira/browse/KAFKA-1356
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Timothy Chen
Assignee: Timothy Chen
 Attachments: KAFKA-1356.patch, KAFKA-1356.patch, KAFKA-1356.patch, 
 KAFKA-1356_2014-04-02_18:39:36.patch, KAFKA-1356_2014-04-04_14:40:18.patch, 
 KAFKA-1356_2014-04-04_17:45:37.patch, KAFKA-1356_2014-04-06_01:45:47.patch, 
 KAFKA-1356_2014-04-08_01:38:23.patch


 Currently we're seeing slow response times in handling get topic metadata 
 requests.
 Local testing shows that even locally it takes 300 avg ms to respond, even 
 though it's not doing any IO operations.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20190: Patch for KAFKA-1356

[Timothy Chen]

 On April 10, 2014, 2:57 p.m., Jun Rao wrote:
  core/src/main/scala/kafka/server/KafkaApis.scala, lines 62-63
  https://reviews.apache.org/r/20190/diff/1/?file=554248#file554248line62
 
  Could we make this class private?

Unit tests seems to rely on accessing the cache to verify some behaviors, so I 
thought to leave it open as is.


 On April 10, 2014, 2:57 p.m., Jun Rao wrote:
  core/src/main/scala/kafka/server/KafkaApis.scala, lines 283-284
  https://reviews.apache.org/r/20190/diff/1/?file=554248#file554248line283
 
  I am not sure that we should just blindly set the controllerEpoch in 
  ReplicaManager. ReplicaManager is supposed to hold the latest controller 
  epoch so that it can disregard requests sent from an older controller. So, 
  we only need to update controllerEpoch if the new value is higher than the 
  current one. Perhaps we can add a util function in ReplicaManager to do 
  that.

Talked to Jun and we agree the previous check is sufficient here.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20190/#review40012
---


On April 10, 2014, 1:48 a.m., Timothy Chen wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20190/
 ---
 
 (Updated April 10, 2014, 1:48 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1356
 https://issues.apache.org/jira/browse/KAFKA-1356
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Refactor metadata cache code in kafka api and remove lock on ensureTopicExists
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/server/KafkaApis.scala 
 d96229e2d4aa7006b0dbd81055ce5a2459d8758c 
 
 Diff: https://reviews.apache.org/r/20190/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Timothy Chen
 

Re: Review Request 20130: Patch for KAFKA-1373

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20130/#review40078
---

Ship it!


Ship It!

- Timothy Chen


On April 8, 2014, 11:18 p.m., Joel Koshy wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20130/
 ---
 
 (Updated April 8, 2014, 11:18 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1373
 https://issues.apache.org/jira/browse/KAFKA-1373
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Set first dirty (uncompacted) offset to first offset of the log if no 
 checkpoint exists.
 
 
 Diffs
 -
 
   core/src/main/scala/kafka/log/LogCleanerManager.scala 
 6a981345fd7dc711e6c4f058a42bd7f8f9d350fe 
   core/src/main/scala/kafka/server/KafkaConfig.scala 
 b0506d4881176cd95914847c4275e428a8d2ca10 
   core/src/main/scala/kafka/server/OffsetCheckpoint.scala 
 19f61a9718a7f8e6f9bf743ac9eb2118d0fdd9b0 
   core/src/main/scala/kafka/server/OffsetManager.scala 
 89a88a7e4055f804429b64e85b2f65312d1e2155 
   core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 
 9aeb69d4934b9aa94e0899af14fc27e7c20f039f 
 
 Diff: https://reviews.apache.org/r/20130/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Joel Koshy
 

Re: Review Request 20143: Patch for KAFKA-1363

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20143/
---

(Updated April 9, 2014, 6:38 p.m.)


Review request for kafka.


Bugs: KAFKA-1363
https://issues.apache.org/jira/browse/KAFKA-1363


Repository: kafka


Description
---

KAFKA-1363 Avoid delete topic deadlock by checking thread run status


Diffs (updated)
-

  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
d29e556232ae549545bde1b6c5c9fabd5fa67bf5 

Diff: https://reviews.apache.org/r/20143/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1363) testTopicConfigChangesDuringDeleteTopic hangs

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964526#comment-13964526
 ] 

Timothy Chen commented on KAFKA-1363:
-

Updated reviewboard https://reviews.apache.org/r/20143/
 against branch origin/trunk

 testTopicConfigChangesDuringDeleteTopic hangs
 -

 Key: KAFKA-1363
 URL: https://issues.apache.org/jira/browse/KAFKA-1363
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
Reporter: Jun Rao
Assignee: Timothy Chen
 Fix For: 0.8.1.1

 Attachments: KAFKA-1363.patch, KAFKA-1363_2014-04-08_17:52:17.patch, 
 KAFKA-1363_2014-04-09_11:38:09.patch


 Saw the following deadlock during shutting down the delete topic manager.
 delete-topics-thread prio=10 tid=0x7fd50c003800 nid=0x7d9 waiting on 
 condition [0x7fd53d16]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0x0006b41d6318 (a 
 java.util.concurrent.locks.ReentrantLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
 at 
 java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
 at 
 java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
 at kafka.utils.Utils$.inLock(Utils.scala:535)
 at 
 kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:363)
 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
 Test worker prio=10 tid=0x7fd578928800 nid=0x763d waiting on condition 
 [0x7fd570a87000]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0x0006b5b6f580 (a 
 java.util.concurrent.CountDownLatch$Sync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
 at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
 at 
 kafka.utils.ShutdownableThread.shutdown(ShutdownableThread.scala:36)
 at 
 kafka.controller.TopicDeletionManager.shutdown(TopicDeletionManager.scala:100)
 at 
 kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:345)
 at 
 kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341)
 at 
 kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341)
 at kafka.utils.Utils$.inLock(Utils.scala:537)
 at 
 kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:341)
 at 
 kafka.controller.KafkaController$$anonfun$shutdown$1.apply$mcV$sp(KafkaController.scala:648)
 at 
 kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646)
 at 
 kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646)
 at kafka.utils.Utils$.inLock(Utils.scala:537)
 at 
 kafka.controller.KafkaController.shutdown(KafkaController.scala:646)
 at 
 kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242)
 at kafka.utils.Utils$.swallow(Utils.scala:166)
 at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
 at kafka.utils.Utils$.swallowWarn(Utils.scala:45)
 at kafka.utils.Logging$class.swallow(Logging.scala:94)
 at kafka.utils.Utils$.swallow(Utils.scala:45)
 at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242)
 at 
 kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362)
 at 
 kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362)
 at 
 scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
 at scala.collection.immutable.List.foreach(List.scala:45

[jira] [Updated] (KAFKA-1363) testTopicConfigChangesDuringDeleteTopic hangs

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1363:


Attachment: KAFKA-1363_2014-04-09_11:38:09.patch

 testTopicConfigChangesDuringDeleteTopic hangs
 -

 Key: KAFKA-1363
 URL: https://issues.apache.org/jira/browse/KAFKA-1363
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
Reporter: Jun Rao
Assignee: Timothy Chen
 Fix For: 0.8.1.1

 Attachments: KAFKA-1363.patch, KAFKA-1363_2014-04-08_17:52:17.patch, 
 KAFKA-1363_2014-04-09_11:38:09.patch


 Saw the following deadlock during shutting down the delete topic manager.
 delete-topics-thread prio=10 tid=0x7fd50c003800 nid=0x7d9 waiting on 
 condition [0x7fd53d16]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0x0006b41d6318 (a 
 java.util.concurrent.locks.ReentrantLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
 at 
 java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
 at 
 java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
 at kafka.utils.Utils$.inLock(Utils.scala:535)
 at 
 kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:363)
 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
 Test worker prio=10 tid=0x7fd578928800 nid=0x763d waiting on condition 
 [0x7fd570a87000]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0x0006b5b6f580 (a 
 java.util.concurrent.CountDownLatch$Sync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
 at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
 at 
 kafka.utils.ShutdownableThread.shutdown(ShutdownableThread.scala:36)
 at 
 kafka.controller.TopicDeletionManager.shutdown(TopicDeletionManager.scala:100)
 at 
 kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:345)
 at 
 kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341)
 at 
 kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341)
 at kafka.utils.Utils$.inLock(Utils.scala:537)
 at 
 kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:341)
 at 
 kafka.controller.KafkaController$$anonfun$shutdown$1.apply$mcV$sp(KafkaController.scala:648)
 at 
 kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646)
 at 
 kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646)
 at kafka.utils.Utils$.inLock(Utils.scala:537)
 at 
 kafka.controller.KafkaController.shutdown(KafkaController.scala:646)
 at 
 kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242)
 at kafka.utils.Utils$.swallow(Utils.scala:166)
 at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
 at kafka.utils.Utils$.swallowWarn(Utils.scala:45)
 at kafka.utils.Logging$class.swallow(Logging.scala:94)
 at kafka.utils.Utils$.swallow(Utils.scala:45)
 at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242)
 at 
 kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362)
 at 
 kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362)
 at 
 scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
 at scala.collection.immutable.List.foreach(List.scala:45)
 at 
 kafka.admin.DeleteTopicTest.testTopicConfigChangesDuringDeleteTopic(DeleteTopicTest.scala:362)



--
This message

Re: Review Request 20186: Patch for KAFKA-1377

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20186/#review39972
---



core/src/test/scala/unit/kafka/server/LogOffsetTest.scala
https://reviews.apache.org/r/20186/#comment72754

I thought we want to assert here so we know from the output it's the leader 
not being elected being the problem in test?


- Timothy Chen


On April 10, 2014, 1:22 a.m., Jun Rao wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20186/
 ---
 
 (Updated April 10, 2014, 1:22 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1377
 https://issues.apache.org/jira/browse/KAFKA-1377
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Increase the wait timeout to give the leader a better chance of being elected.
 
 
 Diffs
 -
 
   core/src/test/scala/unit/kafka/server/LogOffsetTest.scala 
 e1022850d1efc828b8135fb0aa3f19801afc 
 
 Diff: https://reviews.apache.org/r/20186/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jun Rao
 

Re: Review Request 20187: Patch for KAFKA-1363

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20187/
---

(Updated April 10, 2014, 1:27 a.m.)


Review request for kafka.


Bugs: KAFKA-1363
https://issues.apache.org/jira/browse/KAFKA-1363


Repository: kafka


Description
---

KAFKA-1363 Avoid delete topic deadlock by checking thread run status


Diffs
-

  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
40c4c5776b893890a30ad1e4b04f36aa1c112d80 

Diff: https://reviews.apache.org/r/20187/diff/


Testing
---


Thanks,

Timothy Chen