Re: Review Request 31566: Patch for KAFKA-1988
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/#review74973 --- Tong, could you address Jun's last comments before committing? - Guozhang Wang On Feb. 27, 2015, 11:16 p.m., Tong Li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/ --- (Updated Feb. 27, 2015, 11:16 p.m.) Review request for kafka. Bugs: KAFKA-1988 https://issues.apache.org/jira/browse/KAFKA-1988 Repository: kafka Description --- KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 69530c187cd1c41b8173b61de6f982aafe65c9fe clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f Diff: https://reviews.apache.org/r/31566/diff/ Testing --- Thanks, Tong Li
Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2
Jun, I have most everything looks good except I keep getting test failures from wget https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/kafka-0.8.2.1-src.tgz tar -xvf kafka-0.8.2.1-src.tgz cd kafka-0.8.2.1-src gradle ./gradlew test kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown FAILED org.scalatest.junit.JUnitTestFailedError: Expected NotEnoughReplicasException when producing to topic with fewer brokers than min.insync.replicas at org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:101) at org.scalatest.junit.JUnit3Suite.newAssertionFailedException(JUnit3Suite.scala:149) at org.scalatest.Assertions$class.fail(Assertions.scala:711) at org.scalatest.junit.JUnit3Suite.fail(JUnit3Suite.scala:149) at kafka.api.ProducerFailureHandlingTest.testNotEnoughReplicasAfterBrokerShutdown(ProducerFailureHandlingTest.scala:355) This happens to me all the time on a few different machines. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Mon, Mar 2, 2015 at 7:36 PM, Jun Rao j...@confluent.io wrote: +1 from me. Verified quickstart and unit tests. Thanks, Jun On Thu, Feb 26, 2015 at 2:59 PM, Jun Rao j...@confluent.io wrote: This is the second candidate for release of Apache Kafka 0.8.2.1. This fixes 4 critical issue in 0.8.2.0. Release Notes for the 0.8.2.1 release https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/RELEASE_NOTES.html *** Please download, test and vote by Monday, Mar 2, 3pm PT Kafka's KEYS file containing PGP keys we use to sign the release: http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2 (SHA256) checksum. * Release artifacts to be voted upon (source and binary): https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/ * Maven artifacts to be voted upon prior to release: https://repository.apache.org/content/groups/staging/ * scala-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/scaladoc/ * java-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/javadoc/ * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.1 tag https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bd1bfb63ec73c10d08432ac893a23f28281ea021 (git commit ee1267b127f3081db491fa1bf9a287084c324e36) /*** Thanks, Jun -- You received this message because you are subscribed to the Google Groups kafka-clients group. To unsubscribe from this group and stop receiving emails from it, send an email to kafka-clients+unsubscr...@googlegroups.com. To post to this group, send email to kafka-clie...@googlegroups.com. Visit this group at http://groups.google.com/group/kafka-clients. To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/CAFc58G_5FuLKx3_kM4PCgqQL8d%2B4sqE0o-%2BXfu3FJicAgn5KPw%40mail.gmail.com https://groups.google.com/d/msgid/kafka-clients/CAFc58G_5FuLKx3_kM4PCgqQL8d%2B4sqE0o-%2BXfu3FJicAgn5KPw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout.
[jira] [Updated] (KAFKA-1999) Fix failing unit-test: kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown
[ https://issues.apache.org/jira/browse/KAFKA-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1999: Attachment: KAFKA-1999.patch Simple fix to the test. Fix failing unit-test: kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown --- Key: KAFKA-1999 URL: https://issues.apache.org/jira/browse/KAFKA-1999 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-1999.patch The test checks for just one out of two possible exceptions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1999) Fix failing unit-test: kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown
[ https://issues.apache.org/jira/browse/KAFKA-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1999: Status: Patch Available (was: Open) Fix failing unit-test: kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown --- Key: KAFKA-1999 URL: https://issues.apache.org/jira/browse/KAFKA-1999 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-1999.patch The test checks for just one out of two possible exceptions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2
Hi, Good catch, Joe. Releasing with a broken test is not a good habit. I provided a small patch that fixes the issue in KAFKA-1999. Gwen On Tue, Mar 3, 2015 at 9:08 AM, Joe Stein joe.st...@stealth.ly wrote: Jun, I have most everything looks good except I keep getting test failures from wget https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/kafka-0.8.2.1-src.tgz tar -xvf kafka-0.8.2.1-src.tgz cd kafka-0.8.2.1-src gradle ./gradlew test kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown FAILED org.scalatest.junit.JUnitTestFailedError: Expected NotEnoughReplicasException when producing to topic with fewer brokers than min.insync.replicas at org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:101) at org.scalatest.junit.JUnit3Suite.newAssertionFailedException(JUnit3Suite.scala:149) at org.scalatest.Assertions$class.fail(Assertions.scala:711) at org.scalatest.junit.JUnit3Suite.fail(JUnit3Suite.scala:149) at kafka.api.ProducerFailureHandlingTest.testNotEnoughReplicasAfterBrokerShutdown(ProducerFailureHandlingTest.scala:355) This happens to me all the time on a few different machines. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Mon, Mar 2, 2015 at 7:36 PM, Jun Rao j...@confluent.io wrote: +1 from me. Verified quickstart and unit tests. Thanks, Jun On Thu, Feb 26, 2015 at 2:59 PM, Jun Rao j...@confluent.io wrote: This is the second candidate for release of Apache Kafka 0.8.2.1. This fixes 4 critical issue in 0.8.2.0. Release Notes for the 0.8.2.1 release https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/RELEASE_NOTES.html *** Please download, test and vote by Monday, Mar 2, 3pm PT Kafka's KEYS file containing PGP keys we use to sign the release: http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2 (SHA256) checksum. * Release artifacts to be voted upon (source and binary): https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/ * Maven artifacts to be voted upon prior to release: https://repository.apache.org/content/groups/staging/ * scala-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/scaladoc/ * java-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/javadoc/ * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.1 tag https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bd1bfb63ec73c10d08432ac893a23f28281ea021 (git commit ee1267b127f3081db491fa1bf9a287084c324e36) /*** Thanks, Jun -- You received this message because you are subscribed to the Google Groups kafka-clients group. To unsubscribe from this group and stop receiving emails from it, send an email to kafka-clients+unsubscr...@googlegroups.com. To post to this group, send email to kafka-clie...@googlegroups.com. Visit this group at http://groups.google.com/group/kafka-clients. To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/CAFc58G_5FuLKx3_kM4PCgqQL8d%2B4sqE0o-%2BXfu3FJicAgn5KPw%40mail.gmail.com https://groups.google.com/d/msgid/kafka-clients/CAFc58G_5FuLKx3_kM4PCgqQL8d%2B4sqE0o-%2BXfu3FJicAgn5KPw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups kafka-clients group. To unsubscribe from this group and stop receiving emails from it, send an email to kafka-clients+unsubscr...@googlegroups.com. To post to this group, send email to kafka-clie...@googlegroups.com. Visit this group at http://groups.google.com/group/kafka-clients. To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/CAA7ooCCr5oVs0oVpHY%3D2cAgTvJYEL998VfrL5zJbTpSCw8dMUg%40mail.gmail.com https://groups.google.com/d/msgid/kafka-clients/CAA7ooCCr5oVs0oVpHY%3D2cAgTvJYEL998VfrL5zJbTpSCw8dMUg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout.
[jira] [Created] (KAFKA-1999) Fix failing unit-test: kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown
Gwen Shapira created KAFKA-1999: --- Summary: Fix failing unit-test: kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown Key: KAFKA-1999 URL: https://issues.apache.org/jira/browse/KAFKA-1999 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira The test checks for just one out of two possible exceptions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-1688) Add authorization interface and naive implementation
[ https://issues.apache.org/jira/browse/KAFKA-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt reassigned KAFKA-1688: --- Assignee: Parth Brahmbhatt (was: Sriharsha Chintalapani) Add authorization interface and naive implementation Key: KAFKA-1688 URL: https://issues.apache.org/jira/browse/KAFKA-1688 Project: Kafka Issue Type: Sub-task Components: security Reporter: Jay Kreps Assignee: Parth Brahmbhatt Fix For: 0.8.3 Add a PermissionManager interface as described here: https://cwiki.apache.org/confluence/display/KAFKA/Security (possibly there is a better name?) Implement calls to the PermissionsManager in KafkaApis for the main requests (FetchRequest, ProduceRequest, etc). We will need to add a new error code and exception to the protocol to indicate permission denied. Add a server configuration to give the class you want to instantiate that implements that interface. That class can define its own configuration properties from the main config file. Provide a simple implementation of this interface which just takes a user and ip whitelist and permits those in either of the whitelists to do anything, and denies all others. Rather than writing an integration test for this class we can probably just use this class for the TLS and SASL authentication testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31566: Patch for KAFKA-1988
Yes. It was addressed in the latest patch set. A private method called toPositive was added so that the partition selection is preserved cross this and previous versions. Thanks Sent from my iPhone On Mar 4, 2015, at 12:49 AM, Guozhang Wang wangg...@gmail.com wrote: This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/ Tong, could you address Jun's last comments before committing? - Guozhang Wang On February 27th, 2015, 11:16 p.m. UTC, Tong Li wrote: Review request for kafka. By Tong Li. Updated Feb. 27, 2015, 11:16 p.m. Bugs: KAFKA-1988 Repository: kafka Description KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers Diffs clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java (dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8) clients/src/main/java/org/apache/kafka/common/utils/Utils.java (69530c187cd1c41b8173b61de6f982aafe65c9fe) clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java (4c2ea34815b63174732d58b699e1a0a9e6ec3b6f) View Diff
Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations
It would also be good to think through how we can use those new admin requests in the producer/consumer client as well. Currently, both the producer and the consumer use TopicMetadataRequest to obtain the metadata, which will trigger a topic creation if auto topic creation is enabled. This is a bit weird for the consumer since the reader shouldn't be creating new topics. With the new admin requests, we can potentially decouple topic creation from obtaining the metadata. The consumer can just be issuing the metadata requests without triggering the topic creation. The producer can fetch the metadata first and then issue a create topic request if the topic doesn't exist. We will have to think through how this works with the auto topic creation logic though. Thanks, Jun On Mon, Mar 2, 2015 at 9:16 AM, Gwen Shapira gshap...@cloudera.com wrote: Thanks for sending this out Joe. Looking forward to chatting with everyone :) On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein joe.st...@stealth.ly wrote: Hey, I just sent out a google hangout invite to all pmc, committers and everyone I found working on a KIP. If I missed anyone in the invite please let me know and can update it, np. We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA help to make a google account so we can manage better? To discuss https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals in progress and related JIRA that are interdependent and common work. ~ Joe Stein On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps jay.kr...@gmail.com wrote: Let's stay on Google hangouts that will also record and make the sessions available on youtube. -Jay On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman jholo...@cloudera.com wrote: Jay / Joe We're happy to send out a Webex for this purpose. We could record the sessions if there is interest and publish them out. Thanks Jeff On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps jay.kr...@gmail.com wrote: Let's try to get the technical hang-ups sorted out, though. I really think there is some benefit to live discussion vs writing. I am hopeful that if we post instructions and give ourselves a few attempts we can get it working. Tuesday at that time would work for me...any objections? -Jay On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein joe.st...@stealth.ly wrote: Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT I don't mind google hangout but there is always some issue or whatever so we know the apache irc channel works. We can start there and see how it goes? We can pull transcripts too and associate to tickets if need be makes it helpful for things. ~ Joestein On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps jay.kr...@gmail.com wrote: We'd talked about doing a Google Hangout to chat about this. What about generalizing that a little further...I actually think it would be good for everyone spending a reasonable chunk of their week on Kafka stuff to maybe sync up once a week. I think we could use time to talk through design stuff, make sure we are on top of code reviews, talk through any tricky issues, etc. We can make it publicly available so that any one can follow along who likes. Any interest in doing this? If so I'll try to set it up starting next week. -Jay On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Hi all, I've updated KIP page, fixed / aligned document structure. Also I added some very initial proposal for AdminClient so we have something to start from while discussing the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations Thanks, Andrii Biletskyi On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Jay, Re error messages: you are right, in most cases client will have enough context to show descriptive error message. My concern is that we will have to add lots of new error codes for each possible error. Of course, we could reuse some of existing like UknownTopicOrPartitionCode, but we will also need to add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both for topic name and config, and probably user would like to know what exactly is wrong in his config), InvalidReplicaAssignment, InternalError (e.g. zookeeper failure) etc. And this is only for TopicCommand, we will
Re: Review Request 31306: Patch for KAFKA-1755
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/#review74985 --- Ship it! Ship It! - Guozhang Wang On Feb. 26, 2015, 6:54 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/ --- (Updated Feb. 26, 2015, 6:54 p.m.) Review request for kafka. Bugs: KAFKA-1755 https://issues.apache.org/jira/browse/KAFKA-1755 Repository: kafka Description --- KAFKA-1755 v2 re-enable test Incorporate Guozhang's comments Diffs - core/src/main/scala/kafka/log/Log.scala 846023bb98d0fa0603016466360c97071ac935ea core/src/main/scala/kafka/log/LogCleaner.scala f8e7cd5fabce78c248a9027c4bb374a792508675 core/src/main/scala/kafka/log/LogCleanerManager.scala fd87d90597981c867a9b23731fca3b555bf85b7f core/src/main/scala/kafka/message/ByteBufferMessageSet.scala f46ad5cbbbad77d8d1f490d1f8aac97858da9b06 core/src/main/scala/kafka/server/OffsetManager.scala 83d52643028c5628057dc0aa29819becfda61fdb core/src/test/scala/unit/kafka/log/CleanerTest.scala d10e4f4ccbca5e50d81a243d3ab30cc7314b7fef core/src/test/scala/unit/kafka/log/LogTest.scala c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala 73a26377eb63ab9989698e0491049434f032cba2 Diff: https://reviews.apache.org/r/31306/diff/ Testing --- Thanks, Joel Koshy
Re: [VOTE] KIP-3 Mirror Maker Enhancement
KIP-3 passed with 3 binding +1 and 2 non-binding +1. On 2/26/15, 9:02 PM, Neha Narkhede n...@confluent.io wrote: +1 (binding) On Thu, Feb 26, 2015 at 2:14 PM, Joel Koshy jjkosh...@gmail.com wrote: +1 (binding) On Tue, Feb 24, 2015 at 06:37:53PM -0800, Guozhang Wang wrote: +1 binding On Tue, Feb 24, 2015 at 5:24 PM, Gwen Shapira gshap...@cloudera.com wrote: +1 (non-binding) On Tue, Feb 24, 2015 at 5:14 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+En hancement +1 -- -- Guozhang -- Thanks, Neha
Re: Review Request 29467: Patch for KAFKA-1660
On March 3, 2015, 4:10 a.m., Jay Kreps wrote: clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, line 560 https://reviews.apache.org/r/29467/diff/4/?file=882247#file882247line560 This seems to call initiateClose() twice, once in initiateClose and then again from forceClose. This seems like it depends on all the things getting closed being idempotent to repeated calls (e.g. record accumulator etc). Would it make more sense to have forceClose() just set the force flag? The issue with that is someone can just call sender.forceClose and it will never call accumulator.close which is part of initiate close. Also shouldn't the calls be idempotent given this can be called from multiple threds multiple times? On March 3, 2015, 4:10 a.m., Jay Kreps wrote: clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, line 554 https://reviews.apache.org/r/29467/diff/4/?file=882247#file882247line554 It's probably worth adding an if(timeout 0) on this. Added. On March 3, 2015, 4:10 a.m., Parth Brahmbhatt wrote: Two minor changes I noted, but otherwise looks good to me. Needs some unit tests, as you mentioned. Jay Kreps wrote: Actually one probably I didn't think of is that forceClose() leaves the in-flight requests forever incomplete. A better approach would be to fail them all with TimeoutException. To do this correctly I will need to get the imcomplete and unsent RecordBatches from RecordAccumulator. I can add methods to get these with default scope. The sender will need these to emit correct metrics and failing the bathces. For unit testing I need someway to mock RecordAccumulator as the Seneder's run method where the force close logic lives is a while(true) loop which dependes on the values of record accumulator. RecordAccumulator is a final class right now, is it ok to change that so I can create a MockRecordAccumulator? - Parth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74884 --- On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
Re: Review Request 29467: Patch for KAFKA-1660
On March 3, 2015, 5:37 a.m., Jay Kreps wrote:
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java,
line 533
https://reviews.apache.org/r/29467/diff/4/?file=882247#file882247line533
Now there is a bit of duplicate code between the two close methods.
Maybe this would be cleaner if we just made
public void close() {
close(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
}
Fixed.
- Parth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74897
---
On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/
---
(Updated March 2, 2015, 6:41 p.m.)
Review request for kafka.
Bugs: KAFKA-1660
https://issues.apache.org/jira/browse/KAFKA-1660
Repository: kafka
Description
---
Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
Conflicts:
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
Changing log levels as suggested.
Diffs
-
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
7397e565fd865214529ffccadd4222d835ac8110
clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java
6913090af03a455452b0b5c3df78f266126b3854
clients/src/main/java/org/apache/kafka/clients/producer/Producer.java
5b3e75ed83a940bc922f9eca10d4008d67aa37c9
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
ed9c63a6679e3aaf83d19fde19268553a4c107c2
Diff: https://reviews.apache.org/r/29467/diff/
Testing
---
existing unit tests passed.
Thanks,
Parth Brahmbhatt
[jira] [Updated] (KAFKA-1852) OffsetCommitRequest can commit offset on unknown topic
[ https://issues.apache.org/jira/browse/KAFKA-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1852: -- Resolution: Fixed Status: Resolved (was: Patch Available) committed to trunk OffsetCommitRequest can commit offset on unknown topic -- Key: KAFKA-1852 URL: https://issues.apache.org/jira/browse/KAFKA-1852 Project: Kafka Issue Type: Bug Affects Versions: 0.8.3 Reporter: Jun Rao Assignee: Sriharsha Chintalapani Attachments: KAFKA-1852.patch, KAFKA-1852_2015-01-19_10:44:01.patch, KAFKA-1852_2015-02-12_16:46:10.patch, KAFKA-1852_2015-02-16_13:21:46.patch, KAFKA-1852_2015-02-18_13:13:17.patch, KAFKA-1852_2015-02-27_13:50:34.patch Currently, we allow an offset to be committed to Kafka, even when the topic/partition for the offset doesn't exist. We probably should disallow that and send an error back in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1986) Producer request failure rate should not include InvalidMessageSizeException and OffsetOutOfRangeException
[ https://issues.apache.org/jira/browse/KAFKA-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1986: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk Producer request failure rate should not include InvalidMessageSizeException and OffsetOutOfRangeException -- Key: KAFKA-1986 URL: https://issues.apache.org/jira/browse/KAFKA-1986 Project: Kafka Issue Type: Sub-task Reporter: Aditya Auradkar Assignee: Aditya Auradkar Attachments: KAFKA-1986.patch InvalidMessageSizeException and OffsetOutOfRangeException should not be counted a failedProduce and failedFetch requests since they are client side errors. They should be treated similarly to UnknownTopicOrPartitionException or NotLeaderForPartitionException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2000) Delete consumer offsets from kafka once the topic is deleted
Sriharsha Chintalapani created KAFKA-2000: - Summary: Delete consumer offsets from kafka once the topic is deleted Key: KAFKA-2000 URL: https://issues.apache.org/jira/browse/KAFKA-2000 Project: Kafka Issue Type: Bug Reporter: Sriharsha Chintalapani Assignee: Sriharsha Chintalapani -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Hey Jun, Initially I was thinking about instead of 3-4 state that global config changes take effect only after broker restart. So it's just: 3-4. On each broker startup apply global config from ZK In other words, the comprehensive workflow is the following: 1. Issue ChangeGlobalConfigRequest 2. Controller validates / stores config in ZK (out of scope of this KIP) 3. Do rolling restart for brokers one by one to pick up config changes My understanding is that we won't be able to handle config change dynamically as we do for Log config. The main reason, from my point of view, is that broker config operates such settings as num.io.threads updating which would mean gracefully restart some of the broker's components (in this case SocketServer) which is, in turn, might be very tricky. Thanks, Andrii Biletskyi On Tue, Mar 3, 2015 at 7:43 PM, Jun Rao j...@confluent.io wrote: It seems the proposed workflow is the following. 1. Client issues a global config update request to the broker. 2. Broker writes the new config to ZK. 3. Controller picks up the changes from ZK. 4. Controller propagates the config changes to all brokers. Do we need to add a new request/response to propagate the config changes? Also, this logic is a bit different from how per topic config changes works: each broker reads from ZK directly. It would be good to make them consistent. Thanks, Jun On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein joe.st...@stealth.ly wrote: Created a KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management JIRA https://issues.apache.org/jira/browse/KAFKA-1786 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
[jira] [Commented] (KAFKA-1852) OffsetCommitRequest can commit offset on unknown topic
[ https://issues.apache.org/jira/browse/KAFKA-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345612#comment-14345612 ] Joel Koshy commented on KAFKA-1852: --- (As noted in RB we need a separate jira to handle for commits that occur while deleting topics) OffsetCommitRequest can commit offset on unknown topic -- Key: KAFKA-1852 URL: https://issues.apache.org/jira/browse/KAFKA-1852 Project: Kafka Issue Type: Bug Affects Versions: 0.8.3 Reporter: Jun Rao Assignee: Sriharsha Chintalapani Attachments: KAFKA-1852.patch, KAFKA-1852_2015-01-19_10:44:01.patch, KAFKA-1852_2015-02-12_16:46:10.patch, KAFKA-1852_2015-02-16_13:21:46.patch, KAFKA-1852_2015-02-18_13:13:17.patch, KAFKA-1852_2015-02-27_13:50:34.patch Currently, we allow an offset to be committed to Kafka, even when the topic/partition for the offset doesn't exist. We probably should disallow that and send an error back in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations
Thanks for sending that out Joe - I don't think I will be able to make it today, so if notes can be sent out afterward that would be great. On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira wrote: Thanks for sending this out Joe. Looking forward to chatting with everyone :) On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein joe.st...@stealth.ly wrote: Hey, I just sent out a google hangout invite to all pmc, committers and everyone I found working on a KIP. If I missed anyone in the invite please let me know and can update it, np. We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA help to make a google account so we can manage better? To discuss https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals in progress and related JIRA that are interdependent and common work. ~ Joe Stein On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps jay.kr...@gmail.com wrote: Let's stay on Google hangouts that will also record and make the sessions available on youtube. -Jay On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman jholo...@cloudera.com wrote: Jay / Joe We're happy to send out a Webex for this purpose. We could record the sessions if there is interest and publish them out. Thanks Jeff On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps jay.kr...@gmail.com wrote: Let's try to get the technical hang-ups sorted out, though. I really think there is some benefit to live discussion vs writing. I am hopeful that if we post instructions and give ourselves a few attempts we can get it working. Tuesday at that time would work for me...any objections? -Jay On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein joe.st...@stealth.ly wrote: Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT I don't mind google hangout but there is always some issue or whatever so we know the apache irc channel works. We can start there and see how it goes? We can pull transcripts too and associate to tickets if need be makes it helpful for things. ~ Joestein On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps jay.kr...@gmail.com wrote: We'd talked about doing a Google Hangout to chat about this. What about generalizing that a little further...I actually think it would be good for everyone spending a reasonable chunk of their week on Kafka stuff to maybe sync up once a week. I think we could use time to talk through design stuff, make sure we are on top of code reviews, talk through any tricky issues, etc. We can make it publicly available so that any one can follow along who likes. Any interest in doing this? If so I'll try to set it up starting next week. -Jay On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Hi all, I've updated KIP page, fixed / aligned document structure. Also I added some very initial proposal for AdminClient so we have something to start from while discussing the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations Thanks, Andrii Biletskyi On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Jay, Re error messages: you are right, in most cases client will have enough context to show descriptive error message. My concern is that we will have to add lots of new error codes for each possible error. Of course, we could reuse some of existing like UknownTopicOrPartitionCode, but we will also need to add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both for topic name and config, and probably user would like to know what exactly is wrong in his config), InvalidReplicaAssignment, InternalError (e.g. zookeeper failure) etc. And this is only for TopicCommand, we will also need to add similar stuff for ReassignPartitions, PreferredReplica. So we'll end up with a large list of error codes, used only in Admin protocol. Having said that, I agree my proposal is not consistent with other cases. Maybe we can find better solution or something in-between. Re Hangout chat: I think it is a great idea. This way we can move on faster. Let's agree somehow on date/time so people can join. Will work for me this and next week almost anytime if agreed in advance. Thanks, Andrii
Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2
Ok, lets fix the transient test failure on trunk agreed not a blocker. +1 quick start passed, verified artifacts, updates in scala https://github.com/stealthly/scala-kafka/tree/0.8.2.1 and go https://github.com/stealthly/go_kafka_client/tree/0.8.2.1 look good ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Mar 3, 2015 at 12:30 PM, Jun Rao j...@confluent.io wrote: Hi, Joe, Yes, that unit test does have transient failures from time to time. The issue seems to be with the unit test itself and not the actual code. So, this is not a blocker for 0.8.2.1 release. I think we can just fix it in trunk. Thanks, Jun On Tue, Mar 3, 2015 at 9:08 AM, Joe Stein joe.st...@stealth.ly wrote: Jun, I have most everything looks good except I keep getting test failures from wget https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/kafka-0.8.2.1-src.tgz tar -xvf kafka-0.8.2.1-src.tgz cd kafka-0.8.2.1-src gradle ./gradlew test kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown FAILED org.scalatest.junit.JUnitTestFailedError: Expected NotEnoughReplicasException when producing to topic with fewer brokers than min.insync.replicas at org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:101) at org.scalatest.junit.JUnit3Suite.newAssertionFailedException(JUnit3Suite.scala:149) at org.scalatest.Assertions$class.fail(Assertions.scala:711) at org.scalatest.junit.JUnit3Suite.fail(JUnit3Suite.scala:149) at kafka.api.ProducerFailureHandlingTest.testNotEnoughReplicasAfterBrokerShutdown(ProducerFailureHandlingTest.scala:355) This happens to me all the time on a few different machines. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Mon, Mar 2, 2015 at 7:36 PM, Jun Rao j...@confluent.io wrote: +1 from me. Verified quickstart and unit tests. Thanks, Jun On Thu, Feb 26, 2015 at 2:59 PM, Jun Rao j...@confluent.io wrote: This is the second candidate for release of Apache Kafka 0.8.2.1. This fixes 4 critical issue in 0.8.2.0. Release Notes for the 0.8.2.1 release https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/RELEASE_NOTES.html *** Please download, test and vote by Monday, Mar 2, 3pm PT Kafka's KEYS file containing PGP keys we use to sign the release: http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2 (SHA256) checksum. * Release artifacts to be voted upon (source and binary): https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/ * Maven artifacts to be voted upon prior to release: https://repository.apache.org/content/groups/staging/ * scala-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/scaladoc/ * java-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/javadoc/ * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.1 tag https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bd1bfb63ec73c10d08432ac893a23f28281ea021 (git commit ee1267b127f3081db491fa1bf9a287084c324e36) /*** Thanks, Jun -- You received this message because you are subscribed to the Google Groups kafka-clients group. To unsubscribe from this group and stop receiving emails from it, send an email to kafka-clients+unsubscr...@googlegroups.com. To post to this group, send email to kafka-clie...@googlegroups.com. Visit this group at http://groups.google.com/group/kafka-clients. To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/CAFc58G_5FuLKx3_kM4PCgqQL8d%2B4sqE0o-%2BXfu3FJicAgn5KPw%40mail.gmail.com https://groups.google.com/d/msgid/kafka-clients/CAFc58G_5FuLKx3_kM4PCgqQL8d%2B4sqE0o-%2BXfu3FJicAgn5KPw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout.
Re: Review Request 31306: Patch for KAFKA-1755
On March 3, 2015, 5:43 p.m., Mayuresh Gharat wrote: core/src/main/scala/kafka/log/LogCleaner.scala, line 413 https://reviews.apache.org/r/31306/diff/3/?file=878588#file878588line413 This will mean that if there are unkeyed messages we will neglect them and not throw an exception right? So we are aloowing unkeyed messages to go through comapction. Unkeyed messages will be rejected up front (as demonstrated in the unit tests). However, if you change a topic to be compacted (from non-compacted) and it already has unkeyed messages then those will just be ignored (i.e., deleted). - Joel --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/#review74986 --- On Feb. 26, 2015, 6:54 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/ --- (Updated Feb. 26, 2015, 6:54 p.m.) Review request for kafka. Bugs: KAFKA-1755 https://issues.apache.org/jira/browse/KAFKA-1755 Repository: kafka Description --- KAFKA-1755 v2 re-enable test Incorporate Guozhang's comments Diffs - core/src/main/scala/kafka/log/Log.scala 846023bb98d0fa0603016466360c97071ac935ea core/src/main/scala/kafka/log/LogCleaner.scala f8e7cd5fabce78c248a9027c4bb374a792508675 core/src/main/scala/kafka/log/LogCleanerManager.scala fd87d90597981c867a9b23731fca3b555bf85b7f core/src/main/scala/kafka/message/ByteBufferMessageSet.scala f46ad5cbbbad77d8d1f490d1f8aac97858da9b06 core/src/main/scala/kafka/server/OffsetManager.scala 83d52643028c5628057dc0aa29819becfda61fdb core/src/test/scala/unit/kafka/log/CleanerTest.scala d10e4f4ccbca5e50d81a243d3ab30cc7314b7fef core/src/test/scala/unit/kafka/log/LogTest.scala c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala 73a26377eb63ab9989698e0491049434f032cba2 Diff: https://reviews.apache.org/r/31306/diff/ Testing --- Thanks, Joel Koshy
Re: Review Request 31306: Patch for KAFKA-1755
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/#review74992 --- Ship it! Ship It! - Mayuresh Gharat On Feb. 26, 2015, 6:54 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/ --- (Updated Feb. 26, 2015, 6:54 p.m.) Review request for kafka. Bugs: KAFKA-1755 https://issues.apache.org/jira/browse/KAFKA-1755 Repository: kafka Description --- KAFKA-1755 v2 re-enable test Incorporate Guozhang's comments Diffs - core/src/main/scala/kafka/log/Log.scala 846023bb98d0fa0603016466360c97071ac935ea core/src/main/scala/kafka/log/LogCleaner.scala f8e7cd5fabce78c248a9027c4bb374a792508675 core/src/main/scala/kafka/log/LogCleanerManager.scala fd87d90597981c867a9b23731fca3b555bf85b7f core/src/main/scala/kafka/message/ByteBufferMessageSet.scala f46ad5cbbbad77d8d1f490d1f8aac97858da9b06 core/src/main/scala/kafka/server/OffsetManager.scala 83d52643028c5628057dc0aa29819becfda61fdb core/src/test/scala/unit/kafka/log/CleanerTest.scala d10e4f4ccbca5e50d81a243d3ab30cc7314b7fef core/src/test/scala/unit/kafka/log/LogTest.scala c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala 73a26377eb63ab9989698e0491049434f032cba2 Diff: https://reviews.apache.org/r/31306/diff/ Testing --- Thanks, Joel Koshy
[jira] [Commented] (KAFKA-1992) Following KAFKA-1697, checkEnoughReplicasReachOffset doesn't need to get requiredAcks
[ https://issues.apache.org/jira/browse/KAFKA-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345866#comment-14345866 ] Gwen Shapira commented on KAFKA-1992: - Updated reviewboard https://reviews.apache.org/r/31591/diff/ against branch trunk Following KAFKA-1697, checkEnoughReplicasReachOffset doesn't need to get requiredAcks - Key: KAFKA-1992 URL: https://issues.apache.org/jira/browse/KAFKA-1992 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-1992.patch, KAFKA-1992_2015-03-03_14:16:34.patch Follow up from Jun's review: Should we just remove requiredAcks completely since checkEnoughReplicasReachOffset() will only be called when requiredAcks is -1? Answer is: Yes, we should :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31706: Patch for KAFKA-1997
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/ --- Review request for kafka. Bugs: KAFKA-1997 https://issues.apache.org/jira/browse/KAFKA-1997 Repository: kafka Description --- Patch for KAFKA-1997: refactor mirror maker based on producer.flush() Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f core/src/main/scala/kafka/consumer/PartitionAssignor.scala e6ff7683a0df4a7d221e949767e57c34703d5aad core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 7f45a90ba6676290172b7da54c15ee5dc1a42a2e core/src/main/scala/kafka/tools/MirrorMaker.scala 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin
[VOTE] KIP-7 Security - IP Filtering
Details in the wiki. https://cwiki.apache.org/confluence/display/KAFKA/KIP-7+-+Security+-+IP+Filtering -- Jeff Holoman Systems Engineer
[jira] [Updated] (KAFKA-1997) Refactor Mirror Maker
[ https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-1997: Status: Patch Available (was: Open) Refactor Mirror Maker - Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Attachments: KAFKA-1997.patch Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1997) Refactor Mirror Maker
[ https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345879#comment-14345879 ] Jiangjie Qin commented on KAFKA-1997: - Created reviewboard https://reviews.apache.org/r/31706/diff/ against branch origin/trunk Refactor Mirror Maker - Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Attachments: KAFKA-1997.patch Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] KIP-7 Security - IP Filtering
+1 (non-binding) On 3/3/15, 1:17 PM, Gwen Shapira gshap...@cloudera.com wrote: +1 (non-binding) On Tue, Mar 3, 2015 at 12:44 PM, Jeff Holoman jholo...@cloudera.com wrote: Details in the wiki. https://cwiki.apache.org/confluence/display/KAFKA/KIP-7+-+Security+-+IP+F iltering -- Jeff Holoman Systems Engineer
Build failed in Jenkins: Kafka-trunk #415
See https://builds.apache.org/job/Kafka-trunk/415/changes Changes: [jjkoshy] KAFKA-1755; Reject compressed and unkeyed messages sent to compacted topics; reviewed by Mayuresh Gharat, Neha Narkhede and Guozhang Wang [jjkoshy] KAFKA-1852; Reject offset commits to unknown topics; reviewed by Joel Koshy [jjkoshy] KAFKA-1499; trivial follow-up (remove unnecessary parentheses) [jjkoshy] KAFKA-1986; Request failure rate should not include invalid message size and offset out of range; reviewed by Joel Koshy -- [...truncated 1822 lines...] kafka.api.test.ProducerFailureHandlingTest testSendAfterClosed FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) at kafka.zk.EmbeddedZookeeper.init(EmbeddedZookeeper.scala:33) at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerFailureHandlingTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerFailureHandlingTest.scala:38) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:67) kafka.api.test.ProducerFailureHandlingTest testBrokerFailure FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) at kafka.zk.EmbeddedZookeeper.init(EmbeddedZookeeper.scala:33) at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerFailureHandlingTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerFailureHandlingTest.scala:38) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:67) kafka.api.test.ProducerFailureHandlingTest testCannotSendToInternalTopic FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) at kafka.zk.EmbeddedZookeeper.init(EmbeddedZookeeper.scala:33) at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerFailureHandlingTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerFailureHandlingTest.scala:38) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:67) kafka.api.test.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) at kafka.zk.EmbeddedZookeeper.init(EmbeddedZookeeper.scala:33) at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerFailureHandlingTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerFailureHandlingTest.scala:38) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:67) kafka.api.test.ProducerCompressionTest testCompression[0] FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at
[jira] [Updated] (KAFKA-1997) Refactor Mirror Maker
[ https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-1997: Attachment: KAFKA-1997.patch Refactor Mirror Maker - Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Attachments: KAFKA-1997.patch Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: TL;DR; of the first KIP hangout
Thanks Gwen! One thing that I forgot to mention during the meeting is that we need to create a KIP for another producer API change also (KAFKA-1660). The code is almost ready and we just need to figure out some details about interacting with concurrent flush() calls. Guozhang On Tue, Mar 3, 2015 at 12:12 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, I put together a (very) short summary of the discussion and decisions: KIPs: We reviewed the list of KIPs posted here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals * KIP-2: Ready for formal vote * KIP-3: Discussion closed. There’s a new KIP (14) for standardizing command line options. * KIP-4: Need to think about and document APIs in detail, update KIP and discuss. * KIP-5: Need some fleshing out in the KIP - open questions about how exactly it will be used * KIP-6: Everyone should go back to review the KIP. * KIP-7: Ready for vote * KIP-12: Sriharsha will put in a blueprint for discussion * KIP-13: Need more discussion in mailing list about specific details * Ewen had good suggestion on how to fix the port binding in tests. * Create KIP to remove JDK6 support Gwen -- -- Guozhang
Re: [VOTE] KIP-7 Security - IP Filtering
+1 (non-binding) On Tue, Mar 3, 2015 at 12:44 PM, Jeff Holoman jholo...@cloudera.com wrote: Details in the wiki. https://cwiki.apache.org/confluence/display/KAFKA/KIP-7+-+Security+-+IP+Filtering -- Jeff Holoman Systems Engineer
Re: Review Request 31591: Patch for KAFKA-1992
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31591/ --- (Updated March 3, 2015, 10:16 p.m.) Review request for kafka. Bugs: KAFKA-1992 https://issues.apache.org/jira/browse/KAFKA-1992 Repository: kafka Description (updated) --- add logging per Jiangjie Qin comment Diffs (updated) - core/src/main/scala/kafka/cluster/Partition.scala c4bf48a801007ebe7497077d2018d6dffe1677d4 core/src/main/scala/kafka/server/DelayedProduce.scala 4d763bf05efb24a394662721292fc54d32467969 core/src/test/resources/log4j.properties 1b7d5d8f7d5fae7d272849715714781cad05d77b core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala ba48a636dd0b0ed06646d56bb36aa3d43228604f Diff: https://reviews.apache.org/r/31591/diff/ Testing --- Thanks, Gwen Shapira
[VOTE] KIP-2 - Refactor brokers to allow listening on multiple ports and IPs
Details are in the wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
TL;DR; of the first KIP hangout
Hi, I put together a (very) short summary of the discussion and decisions: KIPs: We reviewed the list of KIPs posted here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals * KIP-2: Ready for formal vote * KIP-3: Discussion closed. There’s a new KIP (14) for standardizing command line options. * KIP-4: Need to think about and document APIs in detail, update KIP and discuss. * KIP-5: Need some fleshing out in the KIP - open questions about how exactly it will be used * KIP-6: Everyone should go back to review the KIP. * KIP-7: Ready for vote * KIP-12: Sriharsha will put in a blueprint for discussion * KIP-13: Need more discussion in mailing list about specific details * Ewen had good suggestion on how to fix the port binding in tests. * Create KIP to remove JDK6 support Gwen
Re: [VOTE] KIP-2 - Refactor brokers to allow listening on multiple ports and IPs
+1 non-binding On Tue, Mar 3, 2015, at 12:39 PM, Jeff Holoman wrote: +1 non-binding of course On Tue, Mar 3, 2015 at 3:18 PM, Joe Stein joe.st...@stealth.ly wrote: +1 ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Mar 3, 2015 at 3:14 PM, Gwen Shapira gshap...@cloudera.com wrote: Details are in the wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs -- Jeff Holoman Systems Engineer
[jira] [Updated] (KAFKA-1992) Following KAFKA-1697, checkEnoughReplicasReachOffset doesn't need to get requiredAcks
[ https://issues.apache.org/jira/browse/KAFKA-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1992: Attachment: KAFKA-1992_2015-03-03_14:16:34.patch Following KAFKA-1697, checkEnoughReplicasReachOffset doesn't need to get requiredAcks - Key: KAFKA-1992 URL: https://issues.apache.org/jira/browse/KAFKA-1992 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-1992.patch, KAFKA-1992_2015-03-03_14:16:34.patch Follow up from Jun's review: Should we just remove requiredAcks completely since checkEnoughReplicasReachOffset() will only be called when requiredAcks is -1? Answer is: Yes, we should :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] KIP-2 - Refactor brokers to allow listening on multiple ports and IPs
+1 non-binding of course On Tue, Mar 3, 2015 at 3:18 PM, Joe Stein joe.st...@stealth.ly wrote: +1 ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Mar 3, 2015 at 3:14 PM, Gwen Shapira gshap...@cloudera.com wrote: Details are in the wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs -- Jeff Holoman Systems Engineer
Re: [VOTE] KIP-2 - Refactor brokers to allow listening on multiple ports and IPs
+1 ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Mar 3, 2015 at 3:14 PM, Gwen Shapira gshap...@cloudera.com wrote: Details are in the wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
[jira] [Created] (KAFKA-2001) OffsetCommitTest hang during setup
Jun Rao created KAFKA-2001: -- Summary: OffsetCommitTest hang during setup Key: KAFKA-2001 URL: https://issues.apache.org/jira/browse/KAFKA-2001 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.3 Reporter: Jun Rao OffsetCommitTest seems to hang in trunk reliably. The following is the stacktrace. It seems to hang during setup. Test worker prio=5 tid=7fb5ab154800 nid=0x11198e000 waiting on condition [11198c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at kafka.server.OffsetCommitTest$$anonfun$setUp$2.apply(OffsetCommitTest.scala:59) at kafka.server.OffsetCommitTest$$anonfun$setUp$2.apply(OffsetCommitTest.scala:58) at scala.collection.immutable.Stream.dropWhile(Stream.scala:825) at kafka.server.OffsetCommitTest.setUp(OffsetCommitTest.scala:58) at junit.framework.TestCase.runBare(TestCase.java:132) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:91) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31566: Patch for KAFKA-1988
On March 3, 2015, 4:48 p.m., Guozhang Wang wrote: Tong, could you address Jun's last comments before committing? Yes. absolutely, doing it now. Thanks. - Tong --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/#review74973 --- On Feb. 27, 2015, 11:16 p.m., Tong Li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/ --- (Updated Feb. 27, 2015, 11:16 p.m.) Review request for kafka. Bugs: KAFKA-1988 https://issues.apache.org/jira/browse/KAFKA-1988 Repository: kafka Description --- KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 69530c187cd1c41b8173b61de6f982aafe65c9fe clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f Diff: https://reviews.apache.org/r/31566/diff/ Testing --- Thanks, Tong Li
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1988: --- Attachment: KAFKA-1988_2015-03-03_19:00:21.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: Kafka-trunk #416
See https://builds.apache.org/job/Kafka-trunk/416/changes Changes: [jjkoshy] KAFKA-2001; Trivial commit to prevent OffsetCommitTest from hanging [jjkoshy] KAFKA-2001; Trivial commit to fix OffsetCommitTest -- [...truncated 1121 lines...] kafka.api.test.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown FAILED org.scalatest.junit.JUnitTestFailedError: Expected NotEnoughReplicasException when producing to topic with fewer brokers than min.insync.replicas at org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:101) at org.scalatest.junit.JUnit3Suite.newAssertionFailedException(JUnit3Suite.scala:149) at org.scalatest.Assertions$class.fail(Assertions.scala:711) at org.scalatest.junit.JUnit3Suite.fail(JUnit3Suite.scala:149) at kafka.api.test.ProducerFailureHandlingTest.testNotEnoughReplicasAfterBrokerShutdown(ProducerFailureHandlingTest.scala:352) kafka.network.SocketServerTest simpleRequest PASSED kafka.network.SocketServerTest tooBigRequestIsRejected PASSED kafka.network.SocketServerTest testNullResponse PASSED kafka.network.SocketServerTest testSocketsCloseOnShutdown PASSED kafka.network.SocketServerTest testMaxConnectionsPerIp PASSED kafka.network.SocketServerTest testMaxConnectionsPerIPOverrides PASSED kafka.log.OffsetIndexTest truncate PASSED kafka.log.OffsetIndexTest randomLookupTest PASSED kafka.log.OffsetIndexTest lookupExtremeCases PASSED kafka.log.OffsetIndexTest appendTooMany PASSED kafka.log.OffsetIndexTest appendOutOfOrder PASSED kafka.log.OffsetIndexTest testReopen PASSED kafka.log.OffsetMapTest testBasicValidation PASSED kafka.log.OffsetMapTest testClear PASSED kafka.log.CleanerTest testCleanSegments PASSED kafka.log.CleanerTest testCleaningWithDeletes PASSED kafka.log.CleanerTest testCleaningWithUnkeyedMessages PASSED kafka.log.CleanerTest testCleanSegmentsWithAbort PASSED kafka.log.CleanerTest testSegmentGrouping PASSED kafka.log.CleanerTest testBuildOffsetMap PASSED kafka.log.FileMessageSetTest testWrittenEqualsRead PASSED kafka.log.FileMessageSetTest testIteratorIsConsistent PASSED kafka.log.FileMessageSetTest testSizeInBytes PASSED kafka.log.FileMessageSetTest testWriteTo PASSED kafka.log.FileMessageSetTest testFileSize PASSED kafka.log.FileMessageSetTest testIterationOverPartialAndTruncation PASSED kafka.log.FileMessageSetTest testIterationDoesntChangePosition PASSED kafka.log.FileMessageSetTest testRead PASSED kafka.log.FileMessageSetTest testSearch PASSED kafka.log.FileMessageSetTest testIteratorWithLimits PASSED kafka.log.FileMessageSetTest testTruncate PASSED kafka.log.LogTest testTimeBasedLogRoll PASSED kafka.log.LogTest testTimeBasedLogRollJitter PASSED kafka.log.LogTest testSizeBasedLogRoll PASSED kafka.log.LogTest testLoadEmptyLog PASSED kafka.log.LogTest testAppendAndReadWithSequentialOffsets PASSED kafka.log.LogTest testAppendAndReadWithNonSequentialOffsets PASSED kafka.log.LogTest testReadAtLogGap PASSED kafka.log.LogTest testReadOutOfRange PASSED kafka.log.LogTest testLogRolls PASSED kafka.log.LogTest testCompressedMessages PASSED kafka.log.LogTest testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED kafka.log.LogTest testMessageSetSizeCheck PASSED kafka.log.LogTest testCompactedTopicConstraints PASSED kafka.log.LogTest testMessageSizeCheck PASSED kafka.log.LogTest testLogRecoversToCorrectOffset PASSED kafka.log.LogTest testIndexRebuild PASSED kafka.log.LogTest testTruncateTo PASSED kafka.log.LogTest testIndexResizingAtTruncation PASSED kafka.log.LogTest testBogusIndexSegmentsAreRemoved PASSED kafka.log.LogTest testReopenThenTruncate PASSED kafka.log.LogTest testAsyncDelete PASSED kafka.log.LogTest testOpenDeletesObsoleteFiles PASSED kafka.log.LogTest testAppendMessageWithNullPayload PASSED kafka.log.LogTest testCorruptLog PASSED kafka.log.LogTest testCleanShutdownFile PASSED kafka.log.LogTest testParseTopicPartitionName PASSED kafka.log.LogTest testParseTopicPartitionNameForEmptyName PASSED kafka.log.LogTest testParseTopicPartitionNameForNull PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingSeparator PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingTopic PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingPartition PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[0] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[1] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[2] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[3] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[4] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[5] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[6] PASSED kafka.log.BrokerCompressionTest
[jira] [Updated] (KAFKA-1999) Fix failing unit-test: kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown
[ https://issues.apache.org/jira/browse/KAFKA-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1999: --- Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available) Thanks for the patch. +1. Committed to trunk with a minor tweak (including the unexpected exception in the failing message). Fix failing unit-test: kafka.api.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown --- Key: KAFKA-1999 URL: https://issues.apache.org/jira/browse/KAFKA-1999 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1999.patch The test checks for just one out of two possible exceptions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31591: Patch for KAFKA-1992
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31591/#review75120
---
Thanks for the patch. A few minor comments below.
core/src/main/scala/kafka/cluster/Partition.scala
https://reviews.apache.org/r/31591/#comment122063
The last %s should be %d.
core/src/main/scala/kafka/cluster/Partition.scala
https://reviews.apache.org/r/31591/#comment122065
Our current coding convention is not to wrap single line statement in {}.
core/src/main/scala/kafka/cluster/Partition.scala
https://reviews.apache.org/r/31591/#comment122068
Space should be added after comma.
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala
https://reviews.apache.org/r/31591/#comment122070
This change is already committed.
- Jun Rao
On March 4, 2015, 1:17 a.m., Gwen Shapira wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31591/
---
(Updated March 4, 2015, 1:17 a.m.)
Review request for kafka.
Bugs: KAFKA-1992
https://issues.apache.org/jira/browse/KAFKA-1992
Repository: kafka
Description
---
add logging per Jiangjie Qin comment
revert unintentional changes to log4j
Diffs
-
core/src/main/scala/kafka/cluster/Partition.scala
c4bf48a801007ebe7497077d2018d6dffe1677d4
core/src/main/scala/kafka/server/DelayedProduce.scala
4d763bf05efb24a394662721292fc54d32467969
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala
ba48a636dd0b0ed06646d56bb36aa3d43228604f
Diff: https://reviews.apache.org/r/31591/diff/
Testing
---
Thanks,
Gwen Shapira
Re: TL;DR; of the first KIP hangout
Thanks Gwen! -Jay On Tue, Mar 3, 2015 at 12:12 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, I put together a (very) short summary of the discussion and decisions: KIPs: We reviewed the list of KIPs posted here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals * KIP-2: Ready for formal vote * KIP-3: Discussion closed. There’s a new KIP (14) for standardizing command line options. * KIP-4: Need to think about and document APIs in detail, update KIP and discuss. * KIP-5: Need some fleshing out in the KIP - open questions about how exactly it will be used * KIP-6: Everyone should go back to review the KIP. * KIP-7: Ready for vote * KIP-12: Sriharsha will put in a blueprint for discussion * KIP-13: Need more discussion in mailing list about specific details * Ewen had good suggestion on how to fix the port binding in tests. * Create KIP to remove JDK6 support Gwen
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346038#comment-14346038 ] Tong Li commented on KAFKA-1988: Updated reviewboard https://reviews.apache.org/r/31566/diff/ against branch origin/trunk org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31566: Patch for KAFKA-1988
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/ --- (Updated March 4, 2015, midnight) Review request for kafka. Bugs: KAFKA-1988 https://issues.apache.org/jira/browse/KAFKA-1988 Repository: kafka Description --- KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers Diffs (updated) - clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 69530c187cd1c41b8173b61de6f982aafe65c9fe clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f Diff: https://reviews.apache.org/r/31566/diff/ Testing --- Thanks, Tong Li
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346083#comment-14346083 ] Geoffrey Anderson commented on KAFKA-902: - Thanks for the feedback. Is there any reason to keep 0 - 0 behavior? In other words, is there any harm to making 0 less magical and perturbing it just like any other value? Also, I like the idea of scaling the upper bound on jitter with backoff_ms, though it seems like we still want some amount of jitter even if backoff_ms is small (in your example, if backoff_ms 5, then jitter is always 0) In that case, we might want to make sure that the jitter can be non-zero with something like max(3, min(20, 0.2 * backoff_ms)) But then we end up with a couple more semi-arbitrary parameters - a scaling parameter and a hard minimum. Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31706: Patch for KAFKA-1997
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/#review75110 --- core/src/main/scala/kafka/tools/MirrorMaker.scala https://reviews.apache.org/r/31706/#comment122054 On the other hand we should avoid checking for each message as System.currentTimeMillis() is not that cheap.. - Guozhang Wang On March 4, 2015, 12:28 a.m., Jiangjie Qin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/ --- (Updated March 4, 2015, 12:28 a.m.) Review request for kafka. Bugs: KAFKA-1997 https://issues.apache.org/jira/browse/KAFKA-1997 Repository: kafka Description --- Addressed Guozhang's comments. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f core/src/main/scala/kafka/consumer/PartitionAssignor.scala e6ff7683a0df4a7d221e949767e57c34703d5aad core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 7f45a90ba6676290172b7da54c15ee5dc1a42a2e core/src/main/scala/kafka/tools/MirrorMaker.scala 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin
Re: [VOTE] KIP-7 Security - IP Filtering
Jeff, I am wondering if the IP filtering rule can be enforced at the socket server level instead of the Kafka API level? Guozhang On Tue, Mar 3, 2015 at 2:24 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: +1 (non-binding) On 3/3/15, 1:17 PM, Gwen Shapira gshap...@cloudera.com wrote: +1 (non-binding) On Tue, Mar 3, 2015 at 12:44 PM, Jeff Holoman jholo...@cloudera.com wrote: Details in the wiki. https://cwiki.apache.org/confluence/display/KAFKA/KIP-7+-+Security+-+IP+F iltering -- Jeff Holoman Systems Engineer -- -- Guozhang
Re: [VOTE] KIP-2 - Refactor brokers to allow listening on multiple ports and IPs
+1 binding. On Tue, Mar 3, 2015 at 1:08 PM, Harsha ka...@harsha.io wrote: +1 non-binding On Tue, Mar 3, 2015, at 12:39 PM, Jeff Holoman wrote: +1 non-binding of course On Tue, Mar 3, 2015 at 3:18 PM, Joe Stein joe.st...@stealth.ly wrote: +1 ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Mar 3, 2015 at 3:14 PM, Gwen Shapira gshap...@cloudera.com wrote: Details are in the wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs -- Jeff Holoman Systems Engineer -- -- Guozhang
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[
https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345891#comment-14345891
]
Ashish K Singh commented on KAFKA-1994:
---
[~junrao] Your suggestion makes sense. Profiled {{createPersistent()}} with and
without the patch using {{ZooKeeperTestHarness}}, there is ~44% jump in exec
time. Will shortly attach a patch to fix it.
[~gwenshap] let me know if you think running on a real zk cluster is still
required.
Evaluate performance effect of chroot check on Topic creation
-
Key: KAFKA-1994
URL: https://issues.apache.org/jira/browse/KAFKA-1994
Project: Kafka
Issue Type: Improvement
Reporter: Ashish K Singh
Assignee: Ashish K Singh
KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks
if namespace exists before trying to create a path in ZK. This raises a
concern that checking namespace for each path creation might be unnecessary
and can potentially make creations expensive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: TL;DR; of the first KIP hangout
Thanks Gwen! On Tue, Mar 03, 2015 at 12:12:24PM -0800, Gwen Shapira wrote: Hi, I put together a (very) short summary of the discussion and decisions: KIPs: We reviewed the list of KIPs posted here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals * KIP-2: Ready for formal vote * KIP-3: Discussion closed. There’s a new KIP (14) for standardizing command line options. * KIP-4: Need to think about and document APIs in detail, update KIP and discuss. * KIP-5: Need some fleshing out in the KIP - open questions about how exactly it will be used * KIP-6: Everyone should go back to review the KIP. * KIP-7: Ready for vote * KIP-12: Sriharsha will put in a blueprint for discussion * KIP-13: Need more discussion in mailing list about specific details * Ewen had good suggestion on how to fix the port binding in tests. * Create KIP to remove JDK6 support Gwen
[jira] [Assigned] (KAFKA-2001) OffsetCommitTest hang during setup
[ https://issues.apache.org/jira/browse/KAFKA-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy reassigned KAFKA-2001: - Assignee: Joel Koshy I'll take a quick look at this OffsetCommitTest hang during setup -- Key: KAFKA-2001 URL: https://issues.apache.org/jira/browse/KAFKA-2001 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.3 Reporter: Jun Rao Assignee: Joel Koshy OffsetCommitTest seems to hang in trunk reliably. The following is the stacktrace. It seems to hang during setup. Test worker prio=5 tid=7fb5ab154800 nid=0x11198e000 waiting on condition [11198c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at kafka.server.OffsetCommitTest$$anonfun$setUp$2.apply(OffsetCommitTest.scala:59) at kafka.server.OffsetCommitTest$$anonfun$setUp$2.apply(OffsetCommitTest.scala:58) at scala.collection.immutable.Stream.dropWhile(Stream.scala:825) at kafka.server.OffsetCommitTest.setUp(OffsetCommitTest.scala:58) at junit.framework.TestCase.runBare(TestCase.java:132) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:91) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2001) OffsetCommitTest hang during setup
[ https://issues.apache.org/jira/browse/KAFKA-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-2001. --- Resolution: Fixed This was due to a change I made as part of KAFKA-1755 I added compression.type none for the offsets topic (since we now have KAFKA-1499). However, it was rejected as an invalid config. It needs to be uncompressed OffsetCommitTest hang during setup -- Key: KAFKA-2001 URL: https://issues.apache.org/jira/browse/KAFKA-2001 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.3 Reporter: Jun Rao Assignee: Joel Koshy OffsetCommitTest seems to hang in trunk reliably. The following is the stacktrace. It seems to hang during setup. Test worker prio=5 tid=7fb5ab154800 nid=0x11198e000 waiting on condition [11198c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at kafka.server.OffsetCommitTest$$anonfun$setUp$2.apply(OffsetCommitTest.scala:59) at kafka.server.OffsetCommitTest$$anonfun$setUp$2.apply(OffsetCommitTest.scala:58) at scala.collection.immutable.Stream.dropWhile(Stream.scala:825) at kafka.server.OffsetCommitTest.setUp(OffsetCommitTest.scala:58) at junit.framework.TestCase.runBare(TestCase.java:132) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:91) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31706: Patch for KAFKA-1997
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/ --- (Updated March 4, 2015, 12:28 a.m.) Review request for kafka. Bugs: KAFKA-1997 https://issues.apache.org/jira/browse/KAFKA-1997 Repository: kafka Description (updated) --- Addressed Guozhang's comments. Diffs (updated) - clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f core/src/main/scala/kafka/consumer/PartitionAssignor.scala e6ff7683a0df4a7d221e949767e57c34703d5aad core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 7f45a90ba6676290172b7da54c15ee5dc1a42a2e core/src/main/scala/kafka/tools/MirrorMaker.scala 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin
[jira] [Updated] (KAFKA-1997) Refactor Mirror Maker
[ https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-1997: Attachment: KAFKA-1997_2015-03-03_16:28:46.patch Refactor Mirror Maker - Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Attachments: KAFKA-1997.patch, KAFKA-1997_2015-03-03_16:28:46.patch Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31706: Patch for KAFKA-1997
On March 4, 2015, midnight, Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, line 682 https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line682 Instead of a wrapper rebalancer, I think it is cleaner to just instantiate the ConsumerRebalanceListener interface and pass-in the producer / mirrormaker object for flush() and commitOffsets(). I don't quite follow the comment here. Can you elaborate? On March 4, 2015, midnight, Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, line 67 https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line67 Are we sure that one producer io thread is sufficient for all cases? The assumption is that if more producers are needed, just create more mirror maker instances. On March 4, 2015, midnight, Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, lines 507-509 https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line507 It seems the flush / commit check is outside the inner while loop, and hence will only be triggered when iter.hasNext() returns false? Good catch! - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/#review75071 --- On March 4, 2015, 12:28 a.m., Jiangjie Qin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/ --- (Updated March 4, 2015, 12:28 a.m.) Review request for kafka. Bugs: KAFKA-1997 https://issues.apache.org/jira/browse/KAFKA-1997 Repository: kafka Description --- Addressed Guozhang's comments. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f core/src/main/scala/kafka/consumer/PartitionAssignor.scala e6ff7683a0df4a7d221e949767e57c34703d5aad core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 7f45a90ba6676290172b7da54c15ee5dc1a42a2e core/src/main/scala/kafka/tools/MirrorMaker.scala 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin
[jira] [Commented] (KAFKA-1997) Refactor Mirror Maker
[ https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346096#comment-14346096 ] Jiangjie Qin commented on KAFKA-1997: - Updated reviewboard https://reviews.apache.org/r/31706/diff/ against branch origin/trunk Refactor Mirror Maker - Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Attachments: KAFKA-1997.patch, KAFKA-1997_2015-03-03_16:28:46.patch Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-1976) transient unit test failure in ProducerFailureHandlingTest.testNotEnoughReplicasAfterBrokerShutdown
[ https://issues.apache.org/jira/browse/KAFKA-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1976. Resolution: Duplicate Duplicate of KAFKA-1999. transient unit test failure in ProducerFailureHandlingTest.testNotEnoughReplicasAfterBrokerShutdown --- Key: KAFKA-1976 URL: https://issues.apache.org/jira/browse/KAFKA-1976 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.3 Reporter: Jun Rao Saw the following failure a few times. kafka.api.test.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown FAILED org.scalatest.junit.JUnitTestFailedError: Expected NotEnoughReplicasException when producing to topic with fewer brokers than min.insync.replicas at org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:101) at org.scalatest.junit.JUnit3Suite.newAssertionFailedException(JUnit3Suite.scala:149) at org.scalatest.Assertions$class.fail(Assertions.scala:711) at org.scalatest.junit.JUnit3Suite.fail(JUnit3Suite.scala:149) at kafka.api.test.ProducerFailureHandlingTest.testNotEnoughReplicasAfterBrokerShutdown(ProducerFailureHandlingTest.scala:352) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31306: Patch for KAFKA-1755
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/#review75112 --- core/src/test/scala/unit/kafka/log/CleanerTest.scala https://reviews.apache.org/r/31306/#comment122056 fixed - Joel Koshy On Feb. 26, 2015, 6:54 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/ --- (Updated Feb. 26, 2015, 6:54 p.m.) Review request for kafka. Bugs: KAFKA-1755 https://issues.apache.org/jira/browse/KAFKA-1755 Repository: kafka Description --- KAFKA-1755 v2 re-enable test Incorporate Guozhang's comments Diffs - core/src/main/scala/kafka/log/Log.scala 846023bb98d0fa0603016466360c97071ac935ea core/src/main/scala/kafka/log/LogCleaner.scala f8e7cd5fabce78c248a9027c4bb374a792508675 core/src/main/scala/kafka/log/LogCleanerManager.scala fd87d90597981c867a9b23731fca3b555bf85b7f core/src/main/scala/kafka/message/ByteBufferMessageSet.scala f46ad5cbbbad77d8d1f490d1f8aac97858da9b06 core/src/main/scala/kafka/server/OffsetManager.scala 83d52643028c5628057dc0aa29819becfda61fdb core/src/test/scala/unit/kafka/log/CleanerTest.scala d10e4f4ccbca5e50d81a243d3ab30cc7314b7fef core/src/test/scala/unit/kafka/log/LogTest.scala c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala 73a26377eb63ab9989698e0491049434f032cba2 Diff: https://reviews.apache.org/r/31306/diff/ Testing --- Thanks, Joel Koshy
Review Request 31715: Patch for KAFKA-902
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31715/ --- Review request for kafka. Bugs: KAFKA-902 https://issues.apache.org/jira/browse/KAFKA-902 Repository: kafka Description --- Add simple unit test for ClientUtils.randomizeBackoff Added config for backoff jitter; added randomization to a few more client parameters; updated randomization function. Diffs - clients/src/main/java/org/apache/kafka/clients/ClientUtils.java d0da5d7a08a0c3e67e0fe14bb0b0e7c73380f416 clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 06fcfe62cc1fe76f58540221698ef076fe150e96 clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 5fb21001abd77cac839bd724afa04e377a3e82aa clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 67ceb754a52c07143c69b053fe128b9e24060b99 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 122375c473bf73caf05299b9f5174c6b226ca863 clients/src/test/java/org/apache/kafka/clients/ClientUtilsTest.java 13ce519f03d13db041e1f2dbcd6b59414d2775b8 Diff: https://reviews.apache.org/r/31715/diff/ Testing --- Thanks, Geoffrey Anderson
Re: Review Request 31711: Patch for KAFKA-1994
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/#review75115 --- Thanks for the patch. A comment below. core/src/main/scala/kafka/utils/ZkUtils.scala https://reviews.apache.org/r/31711/#comment122059 Since we may be calling the methods on ZkPath from different threads, we will need to make isNameSpaceChecked volatile. - Jun Rao On March 4, 2015, 12:45 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/ --- (Updated March 4, 2015, 12:45 a.m.) Review request for kafka. Bugs: KAFKA-1994 https://issues.apache.org/jira/browse/KAFKA-1994 Repository: kafka Description --- KAFKA-1994: Evaluate performance effect of chroot check on Topic creation Diffs - core/src/main/scala/kafka/utils/ZkUtils.scala 7ae999ec619443d35a9cb8fbcd531fca0c51c8c0 core/src/test/scala/unit/kafka/zk/ZKPathTest.scala 9897b2fa8f8261fe8ab6b62b45b9052adb07043f Diff: https://reviews.apache.org/r/31711/diff/ Testing --- Thanks, Ashish Singh
Re: Review Request 31591: Patch for KAFKA-1992
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31591/ --- (Updated March 4, 2015, 1:17 a.m.) Review request for kafka. Bugs: KAFKA-1992 https://issues.apache.org/jira/browse/KAFKA-1992 Repository: kafka Description (updated) --- add logging per Jiangjie Qin comment revert unintentional changes to log4j Diffs (updated) - core/src/main/scala/kafka/cluster/Partition.scala c4bf48a801007ebe7497077d2018d6dffe1677d4 core/src/main/scala/kafka/server/DelayedProduce.scala 4d763bf05efb24a394662721292fc54d32467969 core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala ba48a636dd0b0ed06646d56bb36aa3d43228604f Diff: https://reviews.apache.org/r/31591/diff/ Testing --- Thanks, Gwen Shapira
Jenkins build is back to normal : KafkaPreCommit #29
See https://builds.apache.org/job/KafkaPreCommit/29/changes
[jira] [Updated] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1994: -- Attachment: KAFKA-1994.patch Evaluate performance effect of chroot check on Topic creation - Key: KAFKA-1994 URL: https://issues.apache.org/jira/browse/KAFKA-1994 Project: Kafka Issue Type: Improvement Reporter: Ashish K Singh Assignee: Ashish K Singh Attachments: KAFKA-1994.patch KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks if namespace exists before trying to create a path in ZK. This raises a concern that checking namespace for each path creation might be unnecessary and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346125#comment-14346125 ] Ashish K Singh commented on KAFKA-1994: --- Created reviewboard https://reviews.apache.org/r/31711/ against branch trunk Evaluate performance effect of chroot check on Topic creation - Key: KAFKA-1994 URL: https://issues.apache.org/jira/browse/KAFKA-1994 Project: Kafka Issue Type: Improvement Reporter: Ashish K Singh Assignee: Ashish K Singh Attachments: KAFKA-1994.patch KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks if namespace exists before trying to create a path in ZK. This raises a concern that checking namespace for each path creation might be unnecessary and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31706: Patch for KAFKA-1997
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/#review75071 --- clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java https://reviews.apache.org/r/31706/#comment122005 We will try to avoid wildcard imports as much as possible; one thing you can do is in your IDE set the auto-wildcard threshold to sth. like 99. clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java https://reviews.apache.org/r/31706/#comment122006 Ditto here. Also it seems you do not have any actual changes other than imports? core/src/main/scala/kafka/consumer/PartitionAssignor.scala https://reviews.apache.org/r/31706/#comment122007 Change the comments in @return? core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala https://reviews.apache.org/r/31706/#comment122023 This is not a comment but just a thought: in the new consumer where we also have these two callbacks which do not have the global assignment information, we could possibly use metadata request to get the required information. core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java https://reviews.apache.org/r/31706/#comment122018 Add comment that it is a map of (topic - map of (partitionId - consumerThreadId)) since it is different with the assignor's output, which is the map of (consumerId - map of (topic_partition - consumerThreadId)). core/src/main/scala/kafka/tools/MirrorMaker.scala https://reviews.apache.org/r/31706/#comment122019 Are we sure that one producer io thread is sufficient for all cases? core/src/main/scala/kafka/tools/MirrorMaker.scala https://reviews.apache.org/r/31706/#comment122026 It seems the flush / commit check is outside the inner while loop, and hence will only be triggered when iter.hasNext() returns false? core/src/main/scala/kafka/tools/MirrorMaker.scala https://reviews.apache.org/r/31706/#comment122035 Instead of a wrapper rebalancer, I think it is cleaner to just instantiate the ConsumerRebalanceListener interface and pass-in the producer / mirrormaker object for flush() and commitOffsets(). - Guozhang Wang On March 3, 2015, 10:25 p.m., Jiangjie Qin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/ --- (Updated March 3, 2015, 10:25 p.m.) Review request for kafka. Bugs: KAFKA-1997 https://issues.apache.org/jira/browse/KAFKA-1997 Repository: kafka Description --- Patch for KAFKA-1997: refactor mirror maker based on producer.flush() Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f core/src/main/scala/kafka/consumer/PartitionAssignor.scala e6ff7683a0df4a7d221e949767e57c34703d5aad core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 7f45a90ba6676290172b7da54c15ee5dc1a42a2e core/src/main/scala/kafka/tools/MirrorMaker.scala 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin
Re: Review Request 31566: Patch for KAFKA-1988
Guozhang, Yes, I have rebased and updated the patch set to resolve the couple comments for code comments. Also removed the trailing spaces (which always annoying. :-) ), Please see the new patch set here. Sorry for the delay. https://reviews.apache.org/r/31566/ Thanks. Tong Li OpenStack Kafka Community Development Building 501/B205 liton...@us.ibm.com Guozhang Wang nore...@reviews.apache.org wrote on 03/03/2015 11:48:36 AM: From: Guozhang Wang wangg...@gmail.com To: kafka dev@kafka.apache.org, Guozhang Wang wangg...@gmail.com, Tong Li/Raleigh/IBM@IBMUS Date: 03/03/2015 11:49 AM Subject: Re: Review Request 31566: Patch for KAFKA-1988 Sent by: Guozhang Wang nore...@reviews.apache.org --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/#review74973 --- Tong, could you address Jun's last comments before committing? - Guozhang Wang On Feb. 27, 2015, 11:16 p.m., Tong Li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/ --- (Updated Feb. 27, 2015, 11:16 p.m.) Review request for kafka. Bugs: KAFKA-1988 https://issues.apache.org/jira/browse/KAFKA-1988 Repository: kafka Description --- KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers Diffs - clients/src/main/java/org/apache/kafka/clients/producer/ internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 69530c187cd1c41b8173b61de6f982aafe65c9fe clients/src/test/java/org/apache/kafka/common/utils/ UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f Diff: https://reviews.apache.org/r/31566/diff/ Testing --- Thanks, Tong Li
Re: Review Request 31306: Patch for KAFKA-1755
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/#review75047 --- core/src/test/scala/unit/kafka/log/CleanerTest.scala https://reviews.apache.org/r/31306/#comment121983 The comment is incorrect. We are appending keyed messages here. - Jun Rao On Feb. 26, 2015, 6:54 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31306/ --- (Updated Feb. 26, 2015, 6:54 p.m.) Review request for kafka. Bugs: KAFKA-1755 https://issues.apache.org/jira/browse/KAFKA-1755 Repository: kafka Description --- KAFKA-1755 v2 re-enable test Incorporate Guozhang's comments Diffs - core/src/main/scala/kafka/log/Log.scala 846023bb98d0fa0603016466360c97071ac935ea core/src/main/scala/kafka/log/LogCleaner.scala f8e7cd5fabce78c248a9027c4bb374a792508675 core/src/main/scala/kafka/log/LogCleanerManager.scala fd87d90597981c867a9b23731fca3b555bf85b7f core/src/main/scala/kafka/message/ByteBufferMessageSet.scala f46ad5cbbbad77d8d1f490d1f8aac97858da9b06 core/src/main/scala/kafka/server/OffsetManager.scala 83d52643028c5628057dc0aa29819becfda61fdb core/src/test/scala/unit/kafka/log/CleanerTest.scala d10e4f4ccbca5e50d81a243d3ab30cc7314b7fef core/src/test/scala/unit/kafka/log/LogTest.scala c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala 73a26377eb63ab9989698e0491049434f032cba2 Diff: https://reviews.apache.org/r/31306/diff/ Testing --- Thanks, Joel Koshy
[jira] [Commented] (KAFKA-1992) Following KAFKA-1697, checkEnoughReplicasReachOffset doesn't need to get requiredAcks
[ https://issues.apache.org/jira/browse/KAFKA-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346162#comment-14346162 ] Gwen Shapira commented on KAFKA-1992: - Updated reviewboard https://reviews.apache.org/r/31591/diff/ against branch trunk Following KAFKA-1697, checkEnoughReplicasReachOffset doesn't need to get requiredAcks - Key: KAFKA-1992 URL: https://issues.apache.org/jira/browse/KAFKA-1992 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-1992.patch, KAFKA-1992_2015-03-03_14:16:34.patch, KAFKA-1992_2015-03-03_17:17:43.patch Follow up from Jun's review: Should we just remove requiredAcks completely since checkEnoughReplicasReachOffset() will only be called when requiredAcks is -1? Answer is: Yes, we should :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1992) Following KAFKA-1697, checkEnoughReplicasReachOffset doesn't need to get requiredAcks
[ https://issues.apache.org/jira/browse/KAFKA-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1992: Attachment: KAFKA-1992_2015-03-03_17:17:43.patch Following KAFKA-1697, checkEnoughReplicasReachOffset doesn't need to get requiredAcks - Key: KAFKA-1992 URL: https://issues.apache.org/jira/browse/KAFKA-1992 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-1992.patch, KAFKA-1992_2015-03-03_14:16:34.patch, KAFKA-1992_2015-03-03_17:17:43.patch Follow up from Jun's review: Should we just remove requiredAcks completely since checkEnoughReplicasReachOffset() will only be called when requiredAcks is -1? Answer is: Yes, we should :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: KafkaPreCommit #28
See https://builds.apache.org/job/KafkaPreCommit/28/changes Changes: [jjkoshy] KAFKA-1755; Reject compressed and unkeyed messages sent to compacted topics; reviewed by Mayuresh Gharat, Neha Narkhede and Guozhang Wang [jjkoshy] KAFKA-1852; Reject offset commits to unknown topics; reviewed by Joel Koshy [jjkoshy] KAFKA-1499; trivial follow-up (remove unnecessary parentheses) [jjkoshy] KAFKA-1986; Request failure rate should not include invalid message size and offset out of range; reviewed by Joel Koshy -- [...truncated 646 lines...] kafka.network.SocketServerTest testSocketsCloseOnShutdown PASSED kafka.network.SocketServerTest testMaxConnectionsPerIp PASSED kafka.network.SocketServerTest testMaxConnectionsPerIPOverrides PASSED kafka.utils.UtilsTest testSwallow PASSED kafka.utils.UtilsTest testCircularIterator PASSED kafka.utils.UtilsTest testReadBytes PASSED kafka.utils.UtilsTest testAbs PASSED kafka.utils.UtilsTest testReplaceSuffix PASSED kafka.utils.UtilsTest testReadInt PASSED kafka.utils.UtilsTest testCsvList PASSED kafka.utils.UtilsTest testCsvMap PASSED kafka.utils.UtilsTest testInLock PASSED kafka.utils.UtilsTest testDoublyLinkedList PASSED kafka.utils.IteratorTemplateTest testIterator PASSED kafka.utils.SchedulerTest testMockSchedulerNonPeriodicTask PASSED kafka.utils.SchedulerTest testMockSchedulerPeriodicTask PASSED kafka.utils.SchedulerTest testReentrantTaskInMockScheduler PASSED kafka.utils.SchedulerTest testNonPeriodicTask PASSED kafka.utils.SchedulerTest testPeriodicTask PASSED kafka.utils.SchedulerTest testRestart PASSED kafka.utils.JsonTest testJsonEncoding PASSED kafka.utils.ReplicationUtilsTest testUpdateLeaderAndIsr PASSED kafka.utils.ReplicationUtilsTest testGetLeaderIsrAndEpochForPartition PASSED kafka.zk.ZKEphemeralTest testEphemeralNodeCleanup PASSED kafka.admin.AdminTest testReplicaAssignment PASSED kafka.admin.AdminTest testManualReplicaAssignment PASSED kafka.admin.AdminTest testTopicCreationInZK PASSED kafka.admin.AdminTest testPartitionReassignmentWithLeaderInNewReplicas PASSED kafka.admin.AdminTest testPartitionReassignmentWithLeaderNotInNewReplicas PASSED kafka.admin.AdminTest testPartitionReassignmentNonOverlappingReplicas PASSED kafka.admin.AdminTest testReassigningNonExistingPartition PASSED kafka.admin.AdminTest testResumePartitionReassignmentThatWasCompleted PASSED kafka.admin.AdminTest testPreferredReplicaJsonData PASSED kafka.admin.AdminTest testBasicPreferredReplicaElection PASSED kafka.admin.AdminTest testShutdownBroker PASSED kafka.admin.AdminTest testTopicConfigChange PASSED kafka.admin.DeleteConsumerGroupTest testGroupWideDeleteInZK PASSED kafka.admin.DeleteConsumerGroupTest testGroupWideDeleteInZKDoesNothingForActiveConsumerGroup PASSED kafka.admin.DeleteConsumerGroupTest testGroupTopicWideDeleteInZKForGroupConsumingOneTopic PASSED kafka.admin.DeleteConsumerGroupTest testGroupTopicWideDeleteInZKForGroupConsumingMultipleTopics PASSED kafka.admin.DeleteConsumerGroupTest testGroupTopicWideDeleteInZKDoesNothingForActiveGroupConsumingMultipleTopics PASSED kafka.admin.DeleteConsumerGroupTest testTopicWideDeleteInZK PASSED kafka.admin.DeleteConsumerGroupTest testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED kafka.admin.TopicCommandTest testConfigPreservationAcrossPartitionAlteration PASSED kafka.admin.DeleteTopicTest testDeleteTopicWithAllAliveReplicas PASSED kafka.admin.DeleteTopicTest testResumeDeleteTopicWithRecoveredFollower PASSED kafka.admin.DeleteTopicTest testResumeDeleteTopicOnControllerFailover PASSED kafka.admin.DeleteTopicTest testPartitionReassignmentDuringDeleteTopic PASSED kafka.admin.DeleteTopicTest testDeleteTopicDuringAddPartition PASSED kafka.admin.DeleteTopicTest testAddPartitionDuringDeleteTopic PASSED kafka.admin.DeleteTopicTest testRecreateTopicAfterDeletion PASSED kafka.admin.DeleteTopicTest testDeleteNonExistingTopic PASSED kafka.admin.DeleteTopicTest testDeleteTopicWithCleaner PASSED kafka.admin.AddPartitionsTest testTopicDoesNotExist PASSED kafka.admin.AddPartitionsTest testWrongReplicaCount PASSED kafka.admin.AddPartitionsTest testIncrementPartitions PASSED kafka.admin.AddPartitionsTest testManualAssignmentOfReplicas PASSED kafka.admin.AddPartitionsTest testReplicaPlacement PASSED kafka.message.MessageCompressionTest testSimpleCompressDecompress PASSED kafka.message.MessageTest testFieldValues PASSED kafka.message.MessageTest testChecksum PASSED kafka.message.MessageTest testEquality PASSED kafka.message.MessageTest testIsHashable PASSED kafka.message.ByteBufferMessageSetTest testIterator PASSED kafka.message.ByteBufferMessageSetTest testWrittenEqualsRead PASSED kafka.message.ByteBufferMessageSetTest testIteratorIsConsistent PASSED kafka.message.ByteBufferMessageSetTest testSizeInBytes PASSED kafka.message.ByteBufferMessageSetTest
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1988: --- Attachment: KAFKA-1988_2015-03-03_19:03:31.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31711: Patch for KAFKA-1994
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/ --- Review request for kafka. Bugs: KAFKA-1994 https://issues.apache.org/jira/browse/KAFKA-1994 Repository: kafka Description --- KAFKA-1994: Evaluate performance effect of chroot check on Topic creation Diffs - core/src/main/scala/kafka/utils/ZkUtils.scala 7ae999ec619443d35a9cb8fbcd531fca0c51c8c0 core/src/test/scala/unit/kafka/zk/ZKPathTest.scala 9897b2fa8f8261fe8ab6b62b45b9052adb07043f Diff: https://reviews.apache.org/r/31711/diff/ Testing --- Thanks, Ashish Singh
[jira] [Updated] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1994: -- Status: Patch Available (was: Open) Evaluate performance effect of chroot check on Topic creation - Key: KAFKA-1994 URL: https://issues.apache.org/jira/browse/KAFKA-1994 Project: Kafka Issue Type: Improvement Reporter: Ashish K Singh Assignee: Ashish K Singh Attachments: KAFKA-1994.patch KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks if namespace exists before trying to create a path in ZK. This raises a concern that checking namespace for each path creation might be unnecessary and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] KIP-7 Security - IP Filtering
Guozhang, The way the patch is implemented, the check is done in the acceptor thread accept() method of the Socket Server, just before connectionQuotas. Thanks Jeff On Tue, Mar 3, 2015 at 7:59 PM, Guozhang Wang wangg...@gmail.com wrote: Jeff, I am wondering if the IP filtering rule can be enforced at the socket server level instead of the Kafka API level? Guozhang On Tue, Mar 3, 2015 at 2:24 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: +1 (non-binding) On 3/3/15, 1:17 PM, Gwen Shapira gshap...@cloudera.com wrote: +1 (non-binding) On Tue, Mar 3, 2015 at 12:44 PM, Jeff Holoman jholo...@cloudera.com wrote: Details in the wiki. https://cwiki.apache.org/confluence/display/KAFKA/KIP-7+-+Security+-+IP+F iltering -- Jeff Holoman Systems Engineer -- -- Guozhang -- Jeff Holoman Systems Engineer
Jenkins build is back to normal : Kafka-trunk #417
See https://builds.apache.org/job/Kafka-trunk/417/changes
[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.
[
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345918#comment-14345918
]
Guozhang Wang commented on KAFKA-1660:
--
+1 on creating a new KIP. Especially, I think Jiangjie has some comments about
interacting with flush() that we may need some more discussion.
Ability to call close() with a timeout on the Java Kafka Producer.
---
Key: KAFKA-1660
URL: https://issues.apache.org/jira/browse/KAFKA-1660
Project: Kafka
Issue Type: Improvement
Components: clients, producer
Affects Versions: 0.8.2.0
Reporter: Andrew Stein
Assignee: Parth Brahmbhatt
Fix For: 0.8.3
Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch,
KAFKA-1660_2015-03-02_10:41:49.patch
I would like the ability to call {{close}} with a timeout on the Java
Client's KafkaProducer.
h6. Workaround
Currently, it is possible to ensure that {{close}} will return quickly by
first doing a {{future.get(timeout)}} on the last future produced on each
partition, but this means that the user has to define the partitions up front
at the time of {{send}} and track the returned {{future}}'s
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346045#comment-14346045 ] Tong Li commented on KAFKA-1988: Updated reviewboard https://reviews.apache.org/r/31566/diff/ against branch origin/trunk org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31566: Patch for KAFKA-1988
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/ --- (Updated March 4, 2015, 12:03 a.m.) Review request for kafka. Bugs: KAFKA-1988 https://issues.apache.org/jira/browse/KAFKA-1988 Repository: kafka Description --- KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers Diffs (updated) - clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 69530c187cd1c41b8173b61de6f982aafe65c9fe clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f Diff: https://reviews.apache.org/r/31566/diff/ Testing --- Thanks, Tong Li
Re: Review Request 31706: Patch for KAFKA-1997
On March 4, 2015, midnight, Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, line 67 https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line67 Are we sure that one producer io thread is sufficient for all cases? Jiangjie Qin wrote: The assumption is that if more producers are needed, just create more mirror maker instances. I think adding more MM instances is not ideal right? If we have n hosts for MM but need m producers to keep up the traffic where m n, we will end up with more-than-one MM in each host, which is a quite a waste of resources isn't it? On March 4, 2015, midnight, Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, line 682 https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line682 Instead of a wrapper rebalancer, I think it is cleaner to just instantiate the ConsumerRebalanceListener interface and pass-in the producer / mirrormaker object for flush() and commitOffsets(). Jiangjie Qin wrote: I don't quite follow the comment here. Can you elaborate? Instead of class InternalRebalanceListener(Option[ConsumerRebalanceListener]) extends ConsumerRebalanceListener could we just have class InternalRebalanceListener(KafkaProducer) extends ConsumerRebalanceListener and in the instantiated callback we can call KafkaProducer.send() and commitOffsets of MirrorMaker, right? - Guozhang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/#review75071 --- On March 4, 2015, 12:28 a.m., Jiangjie Qin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/ --- (Updated March 4, 2015, 12:28 a.m.) Review request for kafka. Bugs: KAFKA-1997 https://issues.apache.org/jira/browse/KAFKA-1997 Repository: kafka Description --- Addressed Guozhang's comments. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f core/src/main/scala/kafka/consumer/PartitionAssignor.scala e6ff7683a0df4a7d221e949767e57c34703d5aad core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 7f45a90ba6676290172b7da54c15ee5dc1a42a2e core/src/main/scala/kafka/tools/MirrorMaker.scala 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346149#comment-14346149 ] Geoffrey Anderson commented on KAFKA-902: - Created reviewboard https://reviews.apache.org/r/31715/diff/ against branch origin/trunk Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch, KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Attachment: KAFKA-902.patch Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch, KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31706: Patch for KAFKA-1997
On March 4, 2015, midnight, Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, line 67 https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line67 Are we sure that one producer io thread is sufficient for all cases? Jiangjie Qin wrote: The assumption is that if more producers are needed, just create more mirror maker instances. Guozhang Wang wrote: I think adding more MM instances is not ideal right? If we have n hosts for MM but need m producers to keep up the traffic where m n, we will end up with more-than-one MM in each host, which is a quite a waste of resources isn't it? Oh, by instance I just mean another mirror maker process. Not a mirror maker cluster. On March 4, 2015, midnight, Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, line 682 https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line682 Instead of a wrapper rebalancer, I think it is cleaner to just instantiate the ConsumerRebalanceListener interface and pass-in the producer / mirrormaker object for flush() and commitOffsets(). Jiangjie Qin wrote: I don't quite follow the comment here. Can you elaborate? Guozhang Wang wrote: Instead of class InternalRebalanceListener(Option[ConsumerRebalanceListener]) extends ConsumerRebalanceListener could we just have class InternalRebalanceListener(KafkaProducer) extends ConsumerRebalanceListener and in the instantiated callback we can call KafkaProducer.send() and commitOffsets of MirrorMaker, right? Talked to Guozhang offline. Will keep the code as is because we need to invoke the custom rebalance listener in the internal rebalance listener. - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/#review75071 --- On March 4, 2015, 12:28 a.m., Jiangjie Qin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/ --- (Updated March 4, 2015, 12:28 a.m.) Review request for kafka. Bugs: KAFKA-1997 https://issues.apache.org/jira/browse/KAFKA-1997 Repository: kafka Description --- Addressed Guozhang's comments. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f core/src/main/scala/kafka/consumer/PartitionAssignor.scala e6ff7683a0df4a7d221e949767e57c34703d5aad core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 7f45a90ba6676290172b7da54c15ee5dc1a42a2e core/src/main/scala/kafka/tools/MirrorMaker.scala 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin
Re: Review Request 31369: Patch for KAFKA-1982
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/ --- (Updated March 4, 2015, 1:51 a.m.) Review request for kafka. Bugs: KAFKA-1982 https://issues.apache.org/jira/browse/KAFKA-1982 Repository: kafka Description --- KAFKA-1982: change kafka.examples.Producer to use the new java producer Diffs (updated) - clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java PRE-CREATION clients/src/test/java/org/apache/kafka/common/serialization/SerializationTest.java f5cd61c1aa9433524da0b83826a766389de68a0b examples/README 53db6969b2e2d49e23ab13283b9146848e37434e examples/src/main/java/kafka/examples/Consumer.java 13135b954f3078eeb7394822b0db25470b746f03 examples/src/main/java/kafka/examples/KafkaConsumerProducerDemo.java 1239394190fe557e025fbd8f3803334402b0aeea examples/src/main/java/kafka/examples/Producer.java 96e98933148d07564c1b30ba8e805e2433c2adc8 examples/src/main/java/kafka/examples/SimpleConsumerDemo.java 0d66fe5f8819194c8624aed4a21105733c20cc8e Diff: https://reviews.apache.org/r/31369/diff/ Testing --- Thanks, Ashish Singh
Re: Review Request 31706: Patch for KAFKA-1997
On March 4, 2015, 12:48 a.m., Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, line 524 https://reviews.apache.org/r/31706/diff/2/?file=884005#file884005line524 On the other hand we should avoid checking for each message as System.currentTimeMillis() is not that cheap.. currentTimeMillis took about 5 nano secs on my linux box, this is probably a small cost compared with compression/decompression. - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/#review75110 --- On March 4, 2015, 12:28 a.m., Jiangjie Qin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/ --- (Updated March 4, 2015, 12:28 a.m.) Review request for kafka. Bugs: KAFKA-1997 https://issues.apache.org/jira/browse/KAFKA-1997 Repository: kafka Description --- Addressed Guozhang's comments. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f core/src/main/scala/kafka/consumer/PartitionAssignor.scala e6ff7683a0df4a7d221e949767e57c34703d5aad core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java 7f45a90ba6676290172b7da54c15ee5dc1a42a2e core/src/main/scala/kafka/tools/MirrorMaker.scala 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin
[jira] [Commented] (KAFKA-1982) change kafka.examples.Producer to use the new java producer
[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346214#comment-14346214 ] Ashish K Singh commented on KAFKA-1982: --- Updated reviewboard https://reviews.apache.org/r/31369/ against branch trunk change kafka.examples.Producer to use the new java producer --- Key: KAFKA-1982 URL: https://issues.apache.org/jira/browse/KAFKA-1982 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.3 Reporter: Jun Rao Assignee: Ashish K Singh Labels: newbie Attachments: KAFKA-1982.patch, KAFKA-1982_2015-02-24_10:34:51.patch, KAFKA-1982_2015-02-24_20:45:48.patch, KAFKA-1982_2015-02-24_20:48:12.patch, KAFKA-1982_2015-02-27_11:08:34.patch, KAFKA-1982_2015-03-03_17:50:57.patch We need to change the example to use the new java producer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31711: Patch for KAFKA-1994
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/#review75132 --- core/src/main/scala/kafka/utils/ZkUtils.scala https://reviews.apache.org/r/31711/#comment122078 We will need to remember the result of client.exist(). If isNamespaceChecked is true, we will throw a ConfigException if the namespace doesn't exist. - Jun Rao On March 4, 2015, 12:45 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/ --- (Updated March 4, 2015, 12:45 a.m.) Review request for kafka. Bugs: KAFKA-1994 https://issues.apache.org/jira/browse/KAFKA-1994 Repository: kafka Description --- KAFKA-1994: Evaluate performance effect of chroot check on Topic creation Diffs - core/src/main/scala/kafka/utils/ZkUtils.scala 7ae999ec619443d35a9cb8fbcd531fca0c51c8c0 core/src/test/scala/unit/kafka/zk/ZKPathTest.scala 9897b2fa8f8261fe8ab6b62b45b9052adb07043f Diff: https://reviews.apache.org/r/31711/diff/ Testing --- Thanks, Ashish Singh
Re: Review Request 31711: Patch for KAFKA-1994
On March 4, 2015, 1:59 a.m., Jun Rao wrote: core/src/main/scala/kafka/utils/ZkUtils.scala, lines 817-826 https://reviews.apache.org/r/31711/diff/1/?file=884018#file884018line817 We will need to remember the result of client.exist(). If isNamespaceChecked is true, we will throw a ConfigException if the namespace doesn't exist. Not sure if that is a good idea. If we do that then there won't be an easy way to fix the situation. Even if someone creates that namespace on ZK, ZkPath will keep complaining about the missing namespace. With current logic, if there is an exception then namespace will be rechecked the next time as well. Let me know if this does not make sense. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/#review75132 --- On March 4, 2015, 12:45 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/ --- (Updated March 4, 2015, 12:45 a.m.) Review request for kafka. Bugs: KAFKA-1994 https://issues.apache.org/jira/browse/KAFKA-1994 Repository: kafka Description --- KAFKA-1994: Evaluate performance effect of chroot check on Topic creation Diffs - core/src/main/scala/kafka/utils/ZkUtils.scala 7ae999ec619443d35a9cb8fbcd531fca0c51c8c0 core/src/test/scala/unit/kafka/zk/ZKPathTest.scala 9897b2fa8f8261fe8ab6b62b45b9052adb07043f Diff: https://reviews.apache.org/r/31711/diff/ Testing --- Thanks, Ashish Singh
Re: Review Request 31711: Patch for KAFKA-1994
On March 4, 2015, 1:59 a.m., Jun Rao wrote: core/src/main/scala/kafka/utils/ZkUtils.scala, lines 817-826 https://reviews.apache.org/r/31711/diff/1/?file=884018#file884018line817 We will need to remember the result of client.exist(). If isNamespaceChecked is true, we will throw a ConfigException if the namespace doesn't exist. Ashish Singh wrote: Not sure if that is a good idea. If we do that then there won't be an easy way to fix the situation. Even if someone creates that namespace on ZK, ZkPath will keep complaining about the missing namespace. With current logic, if there is an exception then namespace will be rechecked the next time as well. Let me know if this does not make sense. That makes sense. Perhaps we can rename isNamespaceChecked to isNamespacePresent? - Jun --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/#review75132 --- On March 4, 2015, 12:45 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/ --- (Updated March 4, 2015, 12:45 a.m.) Review request for kafka. Bugs: KAFKA-1994 https://issues.apache.org/jira/browse/KAFKA-1994 Repository: kafka Description --- KAFKA-1994: Evaluate performance effect of chroot check on Topic creation Diffs - core/src/main/scala/kafka/utils/ZkUtils.scala 7ae999ec619443d35a9cb8fbcd531fca0c51c8c0 core/src/test/scala/unit/kafka/zk/ZKPathTest.scala 9897b2fa8f8261fe8ab6b62b45b9052adb07043f Diff: https://reviews.apache.org/r/31711/diff/ Testing --- Thanks, Ashish Singh
Re: [KIP-DISCUSSION] KIP-7 Security - IP Filtering
Hey Joel, good questions As a first thought, my experience with customers in large corporate environments probably has me somewhat jaded :). You know it really shouldn't take 3 weeks to get ports opened on a load balancer, but, that really does happen. Coordination across those teams also can and should / does happen but I've noted that operators appreciate measures they can take that keep them out of more internal process. 1) Yes probably. After all we're really just checking what's returned from InetAddress and trusting that. The check is pretty lightweight. I think what you are getting at is that a security check that doesn't go all the way can be bad as it can engender a false sense sense of security and end up leaving the system more vulnerable to attack than if other, more standard, approaches are taken. This is a fair point. I'm not deep enough into network security to comment all that intelligently but I do think that reducing the exposure to say, IP spoofing on internal traffic vs free-for-all data consumption is a step in the right direction. 2) Yes they may have access to this, and it could be redundant. On customers that I interface with, operators typically get their root-level privileges through something like PowerBroker, so access to IPTables is not a given, and even if it's available typically does not fall within their realm of accepted responsibilities. Additionally, when I first got this request I suggested IPTables and was told that due to the difficulties and complexities of configuration and management (from their perspective) that it would not be an acceptable solution (also the it's not in the corporate standard line) I noted in the KIP that I look at this not only as a potential security measure by reducing attack vector size but also as a guard against human error. Hardcoded configs sometimes make their way all the way to production and this would help to limit that. You could argue that it might not be Kafka's responsibility to enforce this type of control, but there is precedence here with HDFS (dfs.hosts and dfs.hosts.exclude) and Flume (*https://issues.apache.org/jira/browse/FLUME-2189 https://issues.apache.org/jira/browse/FLUME-2189*). In short, I don't think that this supplants more robust security functionality but I do think it gives an additional (lightweight) control which would be useful. Security is about defense in depth, and this raises the bar a tad. Thanks Jeff On Tue, Mar 3, 2015 at 8:58 PM, Joel Koshy jjkosh...@gmail.com wrote: The proposal itself looks reasonable, but I have a couple of questions as you made reference to operators of the system; and network team in your wiki. - Are spoofing attacks a concern even with this in place? If so, it would require some sort of internal ingress filtering which presumably need cooperation with network teams right? - Also, the operators of the (Kafka) system really should have access to iptables on the Kafka brokers so wouldn't this feature be effectively redundant? Thanks, Joel On Thu, Jan 22, 2015 at 01:50:41PM -0500, Joe Stein wrote: Hey Jeff, thanks for the patch and writing this up. I think the approach to explicitly deny and then set what is allowed or explicitly allow then deny specifics makes sense. Supporting CIDR notation and ip4 and ip6 both good too. Waiting for KAFKA-1845 to get committed I think makes sense before reworking this anymore right now, yes. Andrii posted a patch yesterday for it so hopefully in the next ~ week(s). Not sure what other folks think of this approach but whatever that is would be good to have it in prior to reworking for the config def changes. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Jan 21, 2015 at 8:47 PM, Jeff Holoman jholo...@cloudera.com wrote: Posted a KIP for IP Filtering: https://cwiki.apache.org/confluence/display/KAFKA/KIP-7+-+Security+-+IP+Filtering Relevant JIRA: https://issues.apache.org/jira/browse/KAFKA-1810 Appreciate any feedback. Thanks Jeff -- Jeff Holoman Systems Engineer
[jira] [Updated] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1994: -- Attachment: KAFKA-1994_2015-03-03_18:19:45.patch Evaluate performance effect of chroot check on Topic creation - Key: KAFKA-1994 URL: https://issues.apache.org/jira/browse/KAFKA-1994 Project: Kafka Issue Type: Improvement Reporter: Ashish K Singh Assignee: Ashish K Singh Attachments: KAFKA-1994.patch, KAFKA-1994_2015-03-03_18:19:45.patch KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks if namespace exists before trying to create a path in ZK. This raises a concern that checking namespace for each path creation might be unnecessary and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31711: Patch for KAFKA-1994
On March 4, 2015, 1:59 a.m., Jun Rao wrote: core/src/main/scala/kafka/utils/ZkUtils.scala, lines 817-826 https://reviews.apache.org/r/31711/diff/1/?file=884018#file884018line817 We will need to remember the result of client.exist(). If isNamespaceChecked is true, we will throw a ConfigException if the namespace doesn't exist. Ashish Singh wrote: Not sure if that is a good idea. If we do that then there won't be an easy way to fix the situation. Even if someone creates that namespace on ZK, ZkPath will keep complaining about the missing namespace. With current logic, if there is an exception then namespace will be rechecked the next time as well. Let me know if this does not make sense. Jun Rao wrote: That makes sense. Perhaps we can rename isNamespaceChecked to isNamespacePresent? Done! - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/#review75132 --- On March 4, 2015, 2:19 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31711/ --- (Updated March 4, 2015, 2:19 a.m.) Review request for kafka. Bugs: KAFKA-1994 https://issues.apache.org/jira/browse/KAFKA-1994 Repository: kafka Description --- KAFKA-1994: Evaluate performance effect of chroot check on Topic creation Diffs - core/src/main/scala/kafka/utils/ZkUtils.scala 7ae999ec619443d35a9cb8fbcd531fca0c51c8c0 core/src/test/scala/unit/kafka/zk/ZKPathTest.scala 9897b2fa8f8261fe8ab6b62b45b9052adb07043f Diff: https://reviews.apache.org/r/31711/diff/ Testing --- Thanks, Ashish Singh
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346253#comment-14346253 ] Ashish K Singh commented on KAFKA-1994: --- Updated reviewboard https://reviews.apache.org/r/31711/ against branch trunk Evaluate performance effect of chroot check on Topic creation - Key: KAFKA-1994 URL: https://issues.apache.org/jira/browse/KAFKA-1994 Project: Kafka Issue Type: Improvement Reporter: Ashish K Singh Assignee: Ashish K Singh Attachments: KAFKA-1994.patch, KAFKA-1994_2015-03-03_18:19:45.patch KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks if namespace exists before trying to create a path in ZK. This raises a concern that checking namespace for each path creation might be unnecessary and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346274#comment-14346274 ] Guozhang Wang commented on KAFKA-1988: -- Thanks [~tongli], +1 and committed to trunk. org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1988: - Resolution: Fixed Status: Resolved (was: Patch Available) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1982) change kafka.examples.Producer to use the new java producer
[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1982: -- Attachment: KAFKA-1982_2015-03-03_17:50:57.patch change kafka.examples.Producer to use the new java producer --- Key: KAFKA-1982 URL: https://issues.apache.org/jira/browse/KAFKA-1982 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.3 Reporter: Jun Rao Assignee: Ashish K Singh Labels: newbie Attachments: KAFKA-1982.patch, KAFKA-1982_2015-02-24_10:34:51.patch, KAFKA-1982_2015-02-24_20:45:48.patch, KAFKA-1982_2015-02-24_20:48:12.patch, KAFKA-1982_2015-02-27_11:08:34.patch, KAFKA-1982_2015-03-03_17:50:57.patch We need to change the example to use the new java producer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31369: Patch for KAFKA-1982
On March 3, 2015, 5:42 a.m., Jun Rao wrote: Thanks for the review Jun! Addressed your concerns. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/#review74895 --- On March 4, 2015, 1:51 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/ --- (Updated March 4, 2015, 1:51 a.m.) Review request for kafka. Bugs: KAFKA-1982 https://issues.apache.org/jira/browse/KAFKA-1982 Repository: kafka Description --- KAFKA-1982: change kafka.examples.Producer to use the new java producer Diffs - clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java PRE-CREATION clients/src/test/java/org/apache/kafka/common/serialization/SerializationTest.java f5cd61c1aa9433524da0b83826a766389de68a0b examples/README 53db6969b2e2d49e23ab13283b9146848e37434e examples/src/main/java/kafka/examples/Consumer.java 13135b954f3078eeb7394822b0db25470b746f03 examples/src/main/java/kafka/examples/KafkaConsumerProducerDemo.java 1239394190fe557e025fbd8f3803334402b0aeea examples/src/main/java/kafka/examples/Producer.java 96e98933148d07564c1b30ba8e805e2433c2adc8 examples/src/main/java/kafka/examples/SimpleConsumerDemo.java 0d66fe5f8819194c8624aed4a21105733c20cc8e Diff: https://reviews.apache.org/r/31369/diff/ Testing --- Thanks, Ashish Singh
