Re: Can I get a review on KAFKA-1512?

2014-07-12 Thread Gwen Shapira
Looks like Jun's comments on review board were not addressed? They seem valid to me - especially the points about IPv6 (seen this at some customers) and InetAddress having both host and port. Gwen On Sat, Jul 12, 2014 at 1:21 PM, Jay Kreps jay.kr...@gmail.com wrote: Thx, -Jay

Newer Zookeeper?

2014-08-03 Thread Gwen Shapira
Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more recent 3.4.x branch? I tested the change on my system and the only impact is to EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which was refactored into its own class in

Re: Newer Zookeeper?

2014-08-04 Thread Gwen Shapira
. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira gshap

Re: Newer Zookeeper?

2014-08-04 Thread Gwen Shapira
Also, specific Zookeeper 3.4.X version where loss of quorum occurred will help. 3.4.5 fixed some pretty serious issues around hanging. Gwen On Mon, Aug 4, 2014 at 9:29 AM, Gwen Shapira gshap...@cloudera.com wrote: Thanks for the heads-up, Joe. We've been shipping Zookeeper 3.4.X for over two

Re: Newer Zookeeper?

2014-08-05 Thread Gwen Shapira
/ On Mon, Aug 4, 2014 at 12:36 PM, Gwen Shapira gshap...@cloudera.com wrote: Also, specific Zookeeper 3.4.X version where loss of quorum occurred will help. 3.4.5 fixed some pretty serious issues around hanging. Gwen On Mon, Aug 4, 2014 at 9:29 AM, Gwen

Re: Newer Zookeeper?

2014-08-05 Thread Gwen Shapira
, since we treat those clusters more like production (they¹re staging for everyone else, but we¹re infrastructure). -Todd On 8/3/14, 9:56 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more

Review Request 24723: KAFKA-1485 - Upgrade to Zookeeper 3.4.6 and create shim for ZKCLI so system tests can run

2014-08-14 Thread Gwen Shapira
/utils/kafka_system_test_utils.py 6edd64a Diff: https://reviews.apache.org/r/24723/diff/ Testing --- Unit tests + replica test from system tests Thanks, Gwen Shapira

Review Request 24782: Patch for KAFKA-1576

2014-08-16 Thread Gwen Shapira
/browse/KAFKA-1576 Repository: kafka Description --- make topic deletion more user-friendly Diffs - core/src/main/scala/kafka/admin/TopicCommand.scala 003a09c6160618bc94858ebc0d806b2aa4158e0a Diff: https://reviews.apache.org/r/24782/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 24782: Patch for KAFKA-1576

2014-08-16 Thread Gwen Shapira
list. Diffs - core/src/main/scala/kafka/admin/TopicCommand.scala 003a09c6160618bc94858ebc0d806b2aa4158e0a Diff: https://reviews.apache.org/r/24782/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 24723: KAFKA-1485 - Upgrade to Zookeeper 3.4.6 and create shim for ZKCLI so system tests can run

2014-08-25 Thread Gwen Shapira
-CREATION core/src/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala d883bde system_test/utils/kafka_system_test_utils.py fcacf0a Diff: https://reviews.apache.org/r/24723/diff/ Testing --- Unit tests + replica test from system tests Thanks, Gwen Shapira

Review Request 25044: KAFKA-1611 - Improve system test configuration

2014-08-25 Thread Gwen Shapira
/system_test_utils.py 50340f0 Diff: https://reviews.apache.org/r/25044/diff/ Testing --- Running system tests bunch of times :) Thanks, Gwen Shapira

Re: [DISCUSSION] Error Handling and Logging at Kafka

2014-08-27 Thread Gwen Shapira
The wiki is great place to record the results or make suggestions that require formatting and graphics to explain, but I think I prefer mailing lists for discussion. So, let me repeat here a question I raised in the wiki, since I don't want it to get lost :) Is there a JIRA for the issues

Re: Kafka Security

2014-09-16 Thread Gwen Shapira
Stein wrote: Hi Andrew, yes the meeting took place and we plan to-do it every two weeks (same bat time, same bat channel) moving forward. In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn), Gwen Shapira (Cloudera) and myself. Gwen updated the wiki after our

Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-21 Thread Gwen Shapira
and one broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-22 Thread Gwen Shapira
normally (all writes succeeded as long as a broker was up) * min.insync.replicas=3 returned NotEnoughReplicas when required.acks=-1 and one broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-22 Thread Gwen Shapira
=3 returned NotEnoughReplicas when required.acks=-1 and one broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-22 Thread Gwen Shapira
NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-24 Thread Gwen Shapira
--- On Sept. 23, 2014, 12:28 a.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25886/ --- (Updated Sept. 23

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-24 Thread Gwen Shapira
NotEnoughReplicas when required.acks=-1 and one broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Read a specific number of messages using kafka

2014-09-25 Thread Gwen Shapira
Using high level consumer and assuming you already created an iterator: while (msgCount maxMessages it.hasNext()) { bytes = it.next().message(); eventList.add(bytes); } (See a complete example here:

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-25 Thread Gwen Shapira
broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-29 Thread Gwen Shapira
--- On Sept. 25, 2014, 7:41 p.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25886/ --- (Updated Sept. 25, 2014, 7:41

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-30 Thread Gwen Shapira
notice that we now have separate objects for MFromConfig and MToConfig. What's the reasoning behind that? - Gwen Shapira On Sept. 25, 2014, 7:41 p.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-30 Thread Gwen Shapira
On Sept. 30, 2014, 4:55 p.m., Gwen Shapira wrote: I didn't get a chance to go over the entire review in detail. I did notice that we now have separate objects for MFromConfig and MToConfig. What's the reasoning behind that? Ignore. commenting on wrong RB. - Gwen

Re: Two open issues on Kafka security

2014-09-30 Thread Gwen Shapira
Re #1: Since the auth_to_local is a kerberos config, its up to the admin to decide how he likes the user names and set it up properly (or leave empty) and make sure the ACLs match. Simplified names may be needed if the authorization system integrates with LDAP to get groups or something fancy

Re: Two open issues on Kafka security

2014-09-30 Thread Gwen Shapira
for the client librariy developers too :) On Tue, Sep 30, 2014 at 6:44 PM, Gwen Shapira gshap...@cloudera.com wrote: Re #1: Since the auth_to_local is a kerberos config, its up to the admin to decide how he likes the user names and set it up properly (or leave empty) and make sure the ACLs

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-09-30 Thread Gwen Shapira
was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: setting up brokers

2014-10-02 Thread Gwen Shapira
We typically install Kafka on its own cluster, although it can use the existing ZK (if its not too busy). Otherwise Kafka can suffer from disk contention with the slaves and CPU/memory contention with the masters. If this is just test cluster, installing Kafka on the slaves is probably fine.

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-02 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25886/#review55187 --- On Oct. 1, 2014, 1:19 a.m., Gwen Shapira wrote

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-02 Thread Gwen Shapira
certain producer side config. My feeling is that if we make min.isr a topic level config, then it should work in the same way independent of the producer side config. So, perhaps it's better to just check inSyncSize minIsr, irrespective of the ack. What do you think? Gwen Shapira

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-06 Thread Gwen Shapira
NotEnoughReplicas when required.acks=-1 and one broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Security JIRAS

2014-10-07 Thread Gwen Shapira
I think we need to add: * Authentication of Kafka brokers with a secured ZooKeeper * Kafka should be able to generate delegation tokens for MapReduce / Spark / Yarn jobs. * Extend systest framework to allow testing secured kafka Gwen On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps jay.kr...@gmail.com

Re: Kafka AWS deployment + UI console

2014-10-07 Thread Gwen Shapira
I'm using Hue's ZooKeeper app: http://gethue.com/new-zookeeper-browser-app/ This UI looks very cute, but I didn't try it yet: https://github.com/claudemamo/kafka-web-console Gwen On Tue, Oct 7, 2014 at 5:08 PM, Shafaq s.abdullah...@gmail.com wrote: We are going to deploy Kafka in Production

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-07 Thread Gwen Shapira
NotEnoughReplicas when required.acks=-1 and one broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-07 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25886/#review55730 --- On Oct. 8, 2014, 1:46 a.m., Gwen Shapira wrote

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-08 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25886/#review55745 --- On Oct. 8, 2014, 1:46 a.m., Gwen Shapira wrote

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-09 Thread Gwen Shapira
required.acks=-1 and one broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-09 Thread Gwen Shapira
(all writes succeeded as long as a broker was up) * min.insync.replicas=3 returned NotEnoughReplicas when required.acks=-1 and one broker was down * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1 See notes about retry behavior in the JIRA. Thanks, Gwen Shapira

Re: Security JIRAS

2014-10-09 Thread Gwen Shapira
it isn't clear to me how to do good integration testing since we need a KDC to test against and it isn't clear how that happens in the test environment except possibly manually (which is not ideal). How do other projects handle this? -Jay On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira gshap

Re: including KAFKA-1555 in 0.8.2?

2014-10-10 Thread Gwen Shapira
+1 :) On Fri, Oct 10, 2014 at 9:34 AM, Joe Stein joe.st...@stealth.ly wrote: +1 On Oct 10, 2014 12:08 PM, Neha Narkhede neha.narkh...@gmail.com wrote: +1. On Thu, Oct 9, 2014 at 6:41 PM, Jun Rao jun...@gmail.com wrote: Hi, Everyone, I just committed KAFKA-1555 (min.isr support) to

Re: Security JIRAS

2014-10-16 Thread Gwen Shapira
Wondering the same here :) I think there are some parallel threads here (SSL is independent of Kerberos, as far as I can see). Kerberos work is blocked on https://issues.apache.org/jira/browse/KAFKA-1683 - Implement a session concept in the socket server. So there's no point in picking up other

Re: Security JIRAS

2014-10-17 Thread Gwen Shapira
Yes, I think we can focus on Broker to Zookeeper communication only. At least for initial stage. Gwen On Fri, Oct 17, 2014 at 2:10 PM, Todd Palino tpal...@gmail.com wrote: For the moment, consumers still need to write under the /consumers tree. Even if they are committing offsets to Kafka

Re: blog with some out of the box pains

2014-10-17 Thread Gwen Shapira
In #2, do you refer to advertising the internal hostname instead of the external one? In this case, will it be enough to use getCanonicalHostName (which uses a name service)? Note that I think the problem the blog reported (wrong name advertised) is somewhat orthogonal to the question of which

Re: blog with some out of the box pains

2014-10-17 Thread Gwen Shapira
. We may want to add this option to the FAQ. Gwen On Fri, Oct 17, 2014 at 7:56 PM, Gwen Shapira gshap...@cloudera.com wrote: It looks like we are using canonical hostname: def register() { val advertisedHostName = if(advertisedHost == null || advertisedHost.trim.isEmpty

Re: [DISCUSS] Release 0.8.2-beta before 0.8.2?

2014-10-21 Thread Gwen Shapira
Note that I added a patch with 0.8.2 documentation to KAFKA-1555 (in order to document the new min.isr behavior). We need to get this committed in SVN before the release, and probably document few more things (new producer for example). Gwen On Tue, Oct 21, 2014 at 12:07 PM, Jun Rao

Re: Documentation with patches

2014-10-23 Thread Gwen Shapira
With two patches, we'll need a two-phase commit to maintain consistency, and we all know what a PITA this can be. (Sorry, couldn't resist the pun. Carry on.) On Thu, Oct 23, 2014 at 10:32 AM, Jay Kreps jay.kr...@gmail.com wrote: That makes sense. I agree that moving to another doc system isn't a

Re: Documentation with patches

2014-10-23 Thread Gwen Shapira
:37 AM, Gwen Shapira gshap...@cloudera.com wrote: With two patches, we'll need a two-phase commit to maintain consistency, and we all know what a PITA this can be. (Sorry, couldn't resist the pun. Carry on.) On Thu, Oct 23, 2014 at 10:32 AM, Jay Kreps jay.kr...@gmail.com wrote: That makes sense. I

Re: [VOTE] 0.8.2-beta Release Candidate 1

2014-10-24 Thread Gwen Shapira
+1 (non-official community vote). Kicked the tires of the binary release. Works out of the box as expected, new producer included. On Fri, Oct 24, 2014 at 5:22 AM, Joe Stein joe.st...@stealth.ly wrote: Jun, I updated the https://people.apache.org/~joestein/kafka-0.8.2-beta-candidate1/java-doc/

Re: [VOTE] 0.8.2-beta Release Candidate 1

2014-10-24 Thread Gwen Shapira
What's the plan for moving from beta to stable? Time based? Or are there any patches that need to go in? On Tue, Oct 21, 2014 at 1:58 PM, Joe Stein joe.st...@stealth.ly wrote: This is the first candidate for release of Apache Kafka 0.8.2-beta Release Notes for the 0.8.2-beta release

Re: 082 docs

2014-10-24 Thread Gwen Shapira
I'll upload a clean patch today to make the review easier. On Fri, Oct 24, 2014 at 10:32 AM, Joel Koshy jjkosh...@gmail.com wrote: Yes I can update the doc for offset management, and review Gwen's doc on min.isr/durability guarantees On Fri, Oct 24, 2014 at 09:10:19AM -0700, Jun Rao wrote:

Re: [VOTE] 0.8.2-beta Release Candidate 1

2014-10-24 Thread Gwen Shapira
. It will be great if companies that pick up beta, deploy it, use it and report bugs. LinkedIn will be helping do this within a 4-5 weeks timeframe, if possible. Thanks, Neha On Fri, Oct 24, 2014 at 10:36 AM, Gwen Shapira gshap...@cloudera.com wrote: What's the plan for moving from beta

Review Request 27203: Patch for KAFKA-1683

2014-10-25 Thread Gwen Shapira
cee76b323e5f3e4c783749ac9e78e1ef02897e3b Diff: https://reviews.apache.org/r/27203/diff/ Testing --- Thanks, Gwen Shapira

Review Request 27204: Patch for KAFKA-1683

2014-10-25 Thread Gwen Shapira
core/src/test/scala/unit/kafka/network/SocketServerTest.scala 5f4d85254c384dcc27a5a84f0836ea225d3a901a Diff: https://reviews.apache.org/r/27204/diff/ Testing --- Thanks, Gwen Shapira

Re: Kafka Security?

2014-10-27 Thread Gwen Shapira
This is very much work in progress. You can follow the Jira here to see how it goes: https://issues.apache.org/jira/browse/KAFKA-1682 On Mon, Oct 27, 2014 at 11:49 AM, Stephenson, John L john.l.stephen...@lmco.com wrote: List Users, Does anyone know when/if Kafka security features are being

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2-beta Released

2014-10-27 Thread Gwen Shapira
Strange. I'm seeing it. Browser cache? On Mon, Oct 27, 2014 at 5:59 PM, Jay Kreps jay.kr...@gmail.com wrote: I actually don't see the beta release on that download page: http://kafka.apache.org/downloads.html -Jay On Mon, Oct 27, 2014 at 5:50 PM, Joe Stein joest...@apache.org wrote: The

Re: High Level Consumer and Close with Auto Commit On

2014-10-28 Thread Gwen Shapira
High level consumer commits before shutting down. If you'll look at ZookeeperConsumerConnector.scala (currently the only implementation of ConsumerConnector) you'll see shutdown() includes the following: if (config.autoCommitEnable) commitOffsets() Gwen On Tue, Oct 28,

Re: Review Request 27430: Fix KAFKA-1720

2014-10-31 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27430/#review59453 --- Ship it! LGTM :) - Gwen Shapira On Nov. 1, 2014, 12:21 a.m

Re: Review Request 27634: Patch for KAFKA-1667

2014-11-05 Thread Gwen Shapira
for the class or from the documentation and add them. - Gwen Shapira On Nov. 5, 2014, 6:44 p.m., Dmytro Kostiuchenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27634

Re: No longer supporting Java 6, if? when?

2014-11-06 Thread Gwen Shapira
+1 for dropping Java 6 On Thu, Nov 6, 2014 at 9:31 AM, Steven Schlansker sschlans...@opentable.com wrote: Java 6 has been End of Life since Feb 2013. Java 7 (and 8, but unfortunately that's too new still) has very compelling features which can make development a lot easier. The sooner more

Re: No longer supporting Java 6, if? when?

2014-11-06 Thread Gwen Shapira
Java6 is supported on CDH4 but not CDH5. On Thu, Nov 6, 2014 at 9:54 AM, Koert Kuipers ko...@tresata.com wrote: when is java 6 dropped by the hadoop distros? i am still aware of many clusters that are java 6 only at the moment. On Thu, Nov 6, 2014 at 12:44 PM, Gwen Shapira gshap

Re: Review Request 27634: Patch for KAFKA-1667

2014-11-06 Thread Gwen Shapira
nottrue? If String.valueOf(false) no longer works as expected, it looks like a bug waiting to happen... core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala https://reviews.apache.org/r/27634/#comment101527 why are we removing a test? - Gwen Shapira

Re: Review Request 27684: Patch for KAFKA-1743

2014-11-07 Thread Gwen Shapira
://reviews.apache.org/r/27684/#comment101725 I'd add a comment on why we have two interfaces here, since its none-obvious and someone may try to refactor in the future... Perhaps link to the email thread? - Gwen Shapira On Nov. 6, 2014, 5:13 p.m., Manikumar Reddy O wrote

Re: Review Request 27634: Patch for KAFKA-1667

2014-11-10 Thread Gwen Shapira
On Nov. 7, 2014, 5 a.m., Gwen Shapira wrote: core/src/main/scala/kafka/utils/Utils.scala, lines 514-522 https://reviews.apache.org/r/27634/diff/3/?file=751666#file751666line514 I think I'm not seeing why we need this. Shouldn't Scala's JavaConversion class handle this exact

Re: Review Request 27634: Patch for KAFKA-1667

2014-11-10 Thread Gwen Shapira
On Nov. 7, 2014, 5 a.m., Gwen Shapira wrote: core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala, line 144 https://reviews.apache.org/r/27634/diff/3/?file=751668#file751668line144 String.valueOf(false) should evaluate to false, right? Why do we need

Re: No longer supporting Java 6, if? when?

2014-11-11 Thread Gwen Shapira
Perhaps relevant: Hadoop is moving toward dropping Java6 in next release. https://issues.apache.org/jira/browse/HADOOP-10530 On Thu, Nov 6, 2014 at 11:03 AM, Jay Kreps jay.kr...@gmail.com wrote: Yeah it is a little bit silly that people are still using Java 6. I guess this is a

Re: Review Request 27990: Patch for KAFKA-1751

2014-11-15 Thread Gwen Shapira
still need to make sure this test will fail if the reassignment command magically succeeds and not exception is thrown. core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala https://reviews.apache.org/r/27990/#comment103389 Any reason for this change? - Gwen Shapira On Nov. 13

Re: Review Request 27990: Patch for KAFKA-1751

2014-11-17 Thread Gwen Shapira
://reviews.apache.org/r/27990/#comment103766 Maybe comment on why you want to fail here. Unlike assertions, these tests are not self documenting. - Gwen Shapira On Nov. 17, 2014, 2:33 p.m., Dmitry Pekar wrote: --- This is an automatically

Re: Welcome Kafka's newest committer

2014-11-19 Thread Gwen Shapira
Congrats Guozhang. Well deserved! — Sent from Mailbox On Thu, Nov 20, 2014 at 1:05 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Hi everyone, I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to become a committer. Guozhang has made significant contributions to

Re: Review Request 28268: Bump up default scala version from 2.10.1 to 2.10.4

2014-11-20 Thread Gwen Shapira
that, I applied the patch and managed to build jar and cleanTest test with no issues with Java8. Non-binding +1 from me :) bin/windows/kafka-run-class.bat https://reviews.apache.org/r/28268/#comment104378 For some reason this doesn't seem to apply cleanly (failed to apply on trunk) - Gwen

Re: Review Request 28108: KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-11-20 Thread Gwen Shapira
. persistent, epehmeral, sequence, etc... Does it make sense to put the validation in a function and use it in all create functions? - Gwen Shapira On Nov. 16, 2014, 7:49 p.m., Ashish Singh wrote: --- This is an automatically generated e-mail

Re: Review Request 28096: KAFKA-313: Add JSON/CSV output and looping options to ConsumerOffsetChecker

2014-11-20 Thread Gwen Shapira
/28096/#comment104399 Can be much cleaner using Scala's enumeration withName. - Gwen Shapira On Nov. 15, 2014, 6:06 p.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: [DISCUSSION] adding the serializer api back to the new java producer

2014-11-24 Thread Gwen Shapira
As one of the people who spent too much time building Avro repositories, +1 on bringing serializer API back. I think it will make the new producer easier to work with. Gwen On Mon, Nov 24, 2014 at 6:13 PM, Jay Kreps jay.kr...@gmail.com wrote: This is admittedly late in the release cycle to

[SECURITY DISCUSSION] Refactoring Brokers to support multiple ports

2014-11-25 Thread Gwen Shapira
multiple components, I wrote a short design document that covers the necessary changes and the upgrade process: https://cwiki.apache.org/confluence/display/KAFKA/Multiple+Listeners+for+Kafka+Brokers Comments are more than welcome :) If this is acceptable, hope to have a patch ready in few days. Gwen

Re: [SECURITY DISCUSSION] Refactoring Brokers to support multiple ports

2014-12-02 Thread Gwen Shapira
new clients. I'll update if any of my customers yells loudly and we'll need to support old clients too :) Thanks, Jun On Tue, Nov 25, 2014 at 11:13 AM, Gwen Shapira gshap...@cloudera.com wrote: Hi Everyone, One of the pre-requisites we have for supporting multiple security protocols (SSL

Re: [SECURITY DISCUSSION] Refactoring Brokers to support multiple ports

2014-12-02 Thread Gwen Shapira
in the request. 4. Should we plan to support security just on the new java clients? Supporting security in both the old and the new clients adds more work and gives people less incentive to migrate off the old clients. Thanks, Jun On Tue, Nov 25, 2014 at 11:13 AM, Gwen Shapira gshap...@cloudera.com

Re: [SECURITY DISCUSSION] Refactoring Brokers to support multiple ports

2014-12-02 Thread Gwen Shapira
eliminate this possibility, unless there's a pretty good reason to do so. -Todd On Tue, Dec 2, 2014 at 3:00 PM, Gwen Shapira gshap...@cloudera.com wrote: Hey Todd, You say lose the ability - you mean this ability actually exist now? Or is this something you hope the new patch will support? Gwen

Review Request 28769: Patch for KAFKA-1809

2014-12-05 Thread Gwen Shapira
, Gwen Shapira

Re: [SECURITY DISCUSSION] Refactoring Brokers to support multiple ports

2014-12-05 Thread Gwen Shapira
deploying from trunk will know when it's safe to use the new feature. Thanks, Jun On Tue, Dec 2, 2014 at 2:56 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks you so much for your help here Jun! Highlighting the specific protocols is very useful. See some detailed comments below

Re: Review Request 28859: Patch for KAFKA-1812

2014-12-10 Thread Gwen Shapira
it instead of the internal Array. core/src/test/scala/unit/kafka/utils/UtilsTest.scala https://reviews.apache.org/r/28859/#comment107304 Since this is a generic function, I'd add tests for: 1. list with a single host:port pair (i.e. no commas) 2. ipv4 - Gwen Shapira On Dec. 9

Re: Review Request 28859: Patch for KAFKA-1812

2014-12-10 Thread Gwen Shapira
On Dec. 10, 2014, 3:21 p.m., Gwen Shapira wrote: Non-binding LGTM from me. Any chance of cleaning up the whitespace? makes the change looks way bigger than it really is. Few suggested improvements below: My mistake - the whitespaces were in the code originally and this patch

Re: Review Request 28859: Patch for KAFKA-1812

2014-12-10 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28859/#review64689 --- Ship it! Non-binding +1 - Gwen Shapira On Dec. 11, 2014, 2:39

Re: A plugin: Kafka Sink and Source for flume

2014-12-12 Thread Gwen Shapira
I'd use the one in Flume trunk (it was far better tested) and we also have Kafka-channel :) And it works with 0.8.1. https://github.com/apache/flume/tree/trunk/flume-ng-sources/flume-kafka-source https://github.com/apache/flume/tree/trunk/flume-ng-sinks/flume-ng-kafka-sink

Re: A plugin: Kafka Sink and Source for flume

2014-12-12 Thread Gwen Shapira
is that we support committing offsets to zookeeper on every batch (rather than automatically) and batch sizes are configurable so you can tune throughput / latency / cpu utilization / zookeeper load, etc. Gwen On Fri, Dec 12, 2014 at 6:10 PM, Gwen Shapira gshap...@cloudera.com wrote: I'd use

Re: Review Request 29029: KAFKA-1512 wire in Override configuration

2014-12-14 Thread Gwen Shapira
and use both when creating the override map and in the connection loop? I think this will make the test less fragile and clearer. - Gwen Shapira On Dec. 14, 2014, 5:26 p.m., Jeff Holoman wrote: --- This is an automatically generated e-mail

Re: Review Request 29029: Patch for KAFKA-1512

2014-12-15 Thread Gwen Shapira
expected. - Gwen Shapira On Dec. 15, 2014, 2:30 a.m., Jeff Holoman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29029

API Annotations

2014-12-16 Thread Gwen Shapira
Hi, Kafka has public APIs in Java and Scala, intended for use by external developers. In addition, Kafka also exposes many public methods that are intended to use within Kafka but are not intended to be called by external developers. Also, some of the external APIs are less stable than others

Review Request 29210: Patch for KAFKA-1819

2014-12-18 Thread Gwen Shapira
/scala/kafka/log/LogManager.scala 4d2924d04bc4bd62413edb0ee2d4aaf3c0052867 core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 29cc01bcef9cacd8dec1f5d662644fc6fe4994bc Diff: https://reviews.apache.org/r/29210/diff/ Testing --- Thanks, Gwen Shapira

Re: API Annotations

2014-12-18 Thread Gwen Shapira
in the original scala code. Thoughts? -Jay On Tue, Dec 16, 2014 at 10:04 AM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka has public APIs in Java and Scala, intended for use by external developers. In addition, Kafka also exposes many public methods that are intended to use within

Review Request 29231: Patch for KAFKA-1824

2014-12-18 Thread Gwen Shapira
--- Thanks, Gwen Shapira

Re: Review Request 29210: Patch for KAFKA-1819

2014-12-18 Thread Gwen Shapira
--- On Dec. 18, 2014, 6:59 p.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29210

Re: Review Request 29231: Patch for KAFKA-1824

2014-12-18 Thread Gwen Shapira
/#review65582 --- On Dec. 19, 2014, 1:56 a.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29231

Re: Review Request 29231: Patch for KAFKA-1824

2014-12-22 Thread Gwen Shapira
/ConsoleProducer.scala 1061cc74fac69693836f1e75add06b09d459a764 core/src/test/scala/kafka/tools/ConsoleProducerTest.scala PRE-CREATION Diff: https://reviews.apache.org/r/29231/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 28769: Patch for KAFKA-1809

2014-12-25 Thread Gwen Shapira
it. - Gwen --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/#review64163 --- On Dec. 5, 2014, 8:34 p.m., Gwen Shapira wrote

Re: Review Request 28769: Patch for KAFKA-1809

2014-12-25 Thread Gwen Shapira
0c6d7a316cc6b51ac0755ca03558507db0706c31 system_test/utils/kafka_system_test_utils.py 41d511cbc310fa87e0f2cd2f772e479e8e3ae4e2 Diff: https://reviews.apache.org/r/28769/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 28769: Patch for KAFKA-1809

2014-12-25 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/#review64163 --- On Dec. 25, 2014, 7:04 p.m., Gwen Shapira wrote

Re: Review Request 29029: Patch for KAFKA-1512

2014-12-25 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29029/#review66134 --- Ship it! Non-binding +1 - the fix and test look good. - Gwen

Re: Review Request 29210: Patch for KAFKA-1819

2014-12-26 Thread Gwen Shapira
: https://reviews.apache.org/r/29210/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 28769: Patch for KAFKA-1809

2014-12-27 Thread Gwen Shapira
/kafka_system_test_utils.py 41d511cbc310fa87e0f2cd2f772e479e8e3ae4e2 Diff: https://reviews.apache.org/r/28769/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 28769: Patch for KAFKA-1809

2014-12-27 Thread Gwen Shapira
0c6d7a316cc6b51ac0755ca03558507db0706c31 system_test/utils/kafka_system_test_utils.py 41d511cbc310fa87e0f2cd2f772e479e8e3ae4e2 Diff: https://reviews.apache.org/r/28769/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 28769: Patch for KAFKA-1809

2014-12-29 Thread Gwen Shapira
/replication_testsuite/testcase_1/testcase_1_properties.json 0c6d7a316cc6b51ac0755ca03558507db0706c31 system_test/utils/kafka_system_test_utils.py 41d511cbc310fa87e0f2cd2f772e479e8e3ae4e2 Diff: https://reviews.apache.org/r/28769/diff/ Testing --- Thanks, Gwen Shapira

  1   2   3   4   5   6   7   8   9   10   >