Re: [VOTE] Release plan - Kafka 0.10.0

, Mar 7, 2016 at 3:26 PM, Joel Koshy wrote: > >> +1 >> >> On Mon, Mar 7, 2016 at 9:27 AM, Gwen Shapira wrote: >> >> > Greetings Kafka Developer Community, >> > >> > As you all know, we have few big features that are almost complete >

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

Hi Team, Since KIP-48 depends on KIP-43, which is already a bit of a risk for the next release - any chance we can delay delegation tokens to Kafka 0.10.1? With the community working on a release every 3 month, this is not a huge delay. Gwen On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh wrote:

Re: [DISCUSS] KIP-49: Fair Partition Assignment Strategy

Hi Team, Since there is a fairly busy release coming up in 2 weeks, and since partition-assignors are pluggable and don't need to be part of an Apache Kafka release in order to be useful, can we delay this KIP to release 0.10.1 or 0.11.0 (whichever is earlier)? This will give the community a chan

Re: [VOTE] Release plan - Kafka 0.10.0

e the system tests for the feature and not just the basic code. Gwen On Mon, Mar 7, 2016 at 4:53 PM, Gwen Shapira wrote: > In order to track the many KIPs that we are trying to land in the next > two weeks, I created this page: > https://cwiki.apache.org/confluence/display/KAFKA/Relea

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

ogin() only once. Again, you > can delegate to multiple classes if you wish to add some complex > mechanism-specific logic, but a single login class makes the mapping to a > single login context and the login cache more obvious (the PR has a test > that includes Kerberos and PLAIN). >

Re: [VOTE] Release plan - Kafka 0.10.0

08.2.x to > 0.10.0.0 as well, due to short release cycle. Please correct me if that is > not true. Created KAFKA-3348 and KAFKA-3349 to add tests for making sure > following paths work fine. > > 1. 0.8.2.x -> 0.10.0.0 > 2. 0.9.1.x -> 0.10.0.0 > > > On Mon, Mar 7, 2016 at 5:

Re: [VOTE] Release plan - Kafka 0.10.0

I added a section on "how to contribute to the release" to the release plan: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.0 Please help out! :) On Mon, Mar 7, 2016 at 9:27 AM, Gwen Shapira wrote: > Greetings Kafka Developer Community, > > As you all kn

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

, at 04:59 PM, Ismael Juma wrote: >> I agree that it would be good to have more time to review and discuss >> KIP-48. >> >> Ismael >> >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira wrote: >> >> > Hi Team, >> > >> > Since KIP-4

Re: [VOTE] Release plan - Kafka 0.10.0

p://bit.ly/21YCthZ > > -Flavio > >> On 08 Mar 2016, at 06:12, Harsha wrote: >> >> >> +1 >> >> Thanks, >> Harsha >> On Mon, Mar 7, 2016, at 09:49 PM, Jun Rao wrote: >>> +1 >>> >>> Thanks, >>> >>> Jun

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

ls different callback handlers based on the mechanism. > And for login, you provide a single or a compound class that provides any > additional login logic for all the login modules that you have specified > for KafkaServer. > > Hope that helps. > > > On Tue, Mar 8, 2016 at

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

ndlers configurable. So we do require users to wrap > their existing classes in our interface. And the interfaces make it > possible to plugin any number of mechanisms. Am I missing something? > > > > On Tue, Mar 8, 2016 at 4:49 PM, Gwen Shapira wrote: > >> Yes, I underst

Re: [Discuss] Remove producer configs deprecated in 0.9

We are planning on supporting direct 0.8.2 to 0.10 upgrades (and are adding tests to that effect). So we expect that quite a few users will never see 0.9 and therefore never see the deprecation notice. I don't think this allows us to remove them at this release. On Tue, Mar 8, 2016 at 3:59 PM, As

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

n't remove the need for new code altogether > since you still need to implement SmartcardCallbackHandler. Multiple > conflicting callback handlers within one broker to support complete > different behaviour is an edge case. Since the design supports > full flexibility in terms of func

KIP Call Notes

Shortest KIP discussion ever :) Here are my notes. Feel free to correct or expand: SSL: Harsh is rebasing after Kafka-1928, and resolving an issue with support for specific API KIP-4 - ISR in Metadata request: We prefer to preserve this information. KIP-25 - System Tests: Geoffrey posted an up

Re: [VOTE] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

+1 (non-binding) On Tue, Jun 9, 2015 at 11:24 PM, Honghai Chen wrote: > Hi Kafka, > > After a long discussion, please help vote again for the KIP. > Thanks. > > > > I wrote a KIP for this after some discussion on KAFKA-1646. > https://issues.apache.org/jira/browse/KAFKA-1646 > > h

Re: [VOTE] KIP-25 System test improvements

+1 (non-binding. Actually, since this is non-binding anyway, lets make it +100. I'm so so excited about having a usable testing framework) On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson wrote: > Hi Kafka, > > After a few rounds of discussion on KIP-25, there doesn't seem to be > opposition, s

Review Request 35347: Patch for KAFKA-2249

/35347/diff/ Testing --- Thanks, Gwen Shapira

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Maybe bring it up at the next KIP call, to make sure everyone is aware? On Thu, Jun 11, 2015 at 2:17 PM, Ashish Singh wrote: > Hi Guys, > > This has been lying around for quite some time. Should I start a voting > thread on this? > > On Thu, May 7, 2015 at 12:20 PM, Ashish Singh wrote: > >> Had

Re: Review Request 35347: Patch for KAFKA-2249

reason to change this to a long? It is used as LONG everywhere in the code. - Gwen --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35347/#review87643 -------

Re: [VOTE] KIP-25 System test improvements

I counted 3 binding votes (Jun, Neha, Joe) and two non-binding (Ewen and I). So I assume the KIP passed :) What's the next step? Are we expecting a patch with few initial tests soon? On Thu, Jun 11, 2015 at 5:41 PM, Jun Rao wrote: > +1 > > Thanks, > > Jun > > On Wed, Jun 10, 2015 at 6:10 PM, Geo

Review Request 35474: Patch for KAFKA-2252

://reviews.apache.org/r/35474/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 33614: Patch for KAFKA-2132

> On June 11, 2015, 10:15 p.m., Aditya Auradkar wrote: > > log4j-appender/src/main/java/org/apache/kafka/log4jappender/KafkaLog4jAppender.java, > > line 135 > > > > > > perhaps wrap this inside an isDebugEnabled check

Re: Guozhang Wang elected to Kafka PMC

Congrats Guozhang! Thanks for your contributions :) On Mon, Jun 15, 2015 at 9:59 PM, Jun Rao wrote: > Hi, Everyone, > > Guozhang Wang has been active in the Kafka community since he became a > Kafka committer about 7 months ago. I am glad to announce that Guozhang is > now a member of Kafka PMC.

Any reason why LogConfig defaults are different from KafkaConfig defaults?

Hi, I'm migrating KafkaConfig and LogConfig over to AbstractConfig interface,and I've ran into something strange: All of LogConfig parameters also exist at KafkaConfig level. However the defaults are completely different. Few examples: Maximum message size is 100 + MessageSet.LogOverhead in

Re: Review Request 35347: Patch for KAFKA-2249

now call originals. - Gwen --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35347/#review87643 --- On June 11, 2015, 6:09 a.m., Gwen Shap

Re: Review Request 35347: Patch for KAFKA-2249

/unit/kafka/server/KafkaConfigConfigDefTest.scala c487f361949b2ca2b6d1b5e2c7fb9ba83c8e53c1 Diff: https://reviews.apache.org/r/35347/diff/ Testing --- Thanks, Gwen Shapira

Re: Review Request 35347: Patch for KAFKA-2249

We can > > just remove that method. > > > > Also, perhaps it's useful and convenient to turn LogConfig into an > > AbstractConfig too. > > Gwen Shapira wrote: > I agree that we don't need to expose toProps since we can let other > cla

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

I think BikeShed will be a great name. Can you clarify the scope? The KIP discusses a framework and also few examples for connectors. Does the addition include just the framework (and perhaps an example or two), or do we plan to start accepting connectors to Apache Kafka project? Gwen On Thu, Ju

Review Request 35677: Patch for KAFKA-2288

/KafkaConfigConfigDefTest.scala 98a5b042a710d3c1064b0379db1d152efc9eabee core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 2428dbd7197a58cf4cad42ef82b385dab3a2b15e Diff: https://reviews.apache.org/r/35677/diff/ Testing --- Thanks, Gwen Shapira

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

just to include the framework and maybe one simple connector as > an example. > > Thoughts? > > -Jay > > On Fri, Jun 19, 2015 at 2:38 PM, Gwen Shapira wrote: > >> I think BikeShed will be a great name. >> >> Can you clarify the scope? The KIP discusses

Re: Review Request 35677: Patch for KAFKA-2288

This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35677/#review88723 ------- On June 20, 2015, 12:59 a.m., Gwen Shapira wrote: > >

Re: Review Request 35677: Patch for KAFKA-2288

/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 2428dbd7197a58cf4cad42ef82b385dab3a2b15e Diff: https://reviews.apache.org/r/35677/diff/ Testing --- Thanks, Gwen Shapira

Re: [GitHub] kafka pull request: Kafka 2276

Thanks, I indeed missed the original :) Is the plan to squash the commits and merge a pull request with single commit that matches the JIRA #? This will be more in line with how commits were organized until now and will make life much easier when cherry-picking. Gwen On Mon, Jun 22, 2015 at 1:58

Re: [GitHub] kafka pull request: Kafka 2276

s outlined here: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201504.mbox/%3ccad5tkzab-hkey-zcr8x4wtxawybxpojx62k1vbv+ycknuxq...@mail.gmail.com%3E > > Ismael's script is here: https://issues.apache.org/jira/browse/KAFKA-2187 > > > Thanks, > > Geoff > > On Mon,

Re: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

f we add these options, would you still need the old format. If >> >> csv/json offers more convenience, should we have a plan to phase out the >> >> old format? >> >> >> > Probably not, but having it around will not hurt. Having three output >> >

Re: Review Request 33620: Patch for KAFKA-1690

> On May 22, 2015, 12:33 a.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, > > line 231 > > > > > > Do we need config.values? Could we just pass in config.originals()

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

Re: Flume vs. CopyCat I would love to have an automagically-parallelizing, schema-aware version of Flume with great reliability guarantees. Flume has good core architecture and I'm sure that if the Flume community is interested, it can be extended in that direction. However, the Apache way is not

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

One more reason to have CopyCat as a separate project is to sidestep the entire "Why CopyCat and not X" discussion :) On Tue, Jun 23, 2015 at 6:26 PM, Gwen Shapira wrote: > Re: Flume vs. CopyCat > > I would love to have an automagically-parallelizing, schema-aware > versi

Re: [GitHub] kafka pull request: Kafka 2276

This has a bunch of stuff that I'm not sure we want in the project at all... specific IPs, Vagrant install from debian packages, etc. Gwen On Tue, Jun 23, 2015 at 10:45 AM, Gwen Shapira wrote: > Awesome, thanks :) > > On Tue, Jun 23, 2015 at 10:32 AM, Geoffrey Anderson &

Re: [GitHub] kafka pull request: Kafka 2276

; stuff like being Debian-specific can be generalized by extending the > Vagrantfile and scripts, which I think would be great, but somebody needs > to invest the time. There are loads of other improvements that could me > made too -- for example, I'd love seeing Docker support, which wo

Re: Help Us Nominate Apache Kafka for a 2015 Bossie (Best of OSS) Award - Due June 30th

Sent! Thanks for letting us know of this opportunity to promote our favorite Apache project :) For inspiration, here's what I wrote: Apache Kafka revolutionized stream processing for big data. There is a dramatic growth in the number and variety of data sources an organization has to track as wel

Re: EOL JDK 1.6 for Kafka

Huge +1. I don't think there is any other project that still supports 1.6. On Wed, Jul 1, 2015 at 8:05 AM, Harsha wrote: > Hi, > During our SSL Patch KAFKA-1690. Some of the reviewers/users > asked for support this config > https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/

Review Request 36290: Patch for KAFKA-2308

/ Testing --- Thanks, Gwen Shapira

Re: Review Request 34403: Patch for KAFKA-2198

use the CLI. Perhaps the stack trace should go under log.error(...)? - Gwen Shapira On July 10, 2015, 4:34 p.m., Manikumar Reddy O wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://rev

Re: Review Request 34641: Patch for KAFKA-2214

/kafka/admin/ReassignPartitionsCommand.scala (line 79) <https://reviews.apache.org/r/34641/#comment144625> why not re-throw? APIs that can both return error codes and throw exceptions are pretty confusing (you need to handle errors in two different ways). - Gwen Shapira On July 10

Re: Review Request 34641: Patch for KAFKA-2214

> On May 26, 2015, 7:08 a.m., Michael Noll wrote: > > core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala, line 81 > > > > > > Should we also consider reassignments that are in-progress as "errors"? > >

Re: Review Request 34641: Patch for KAFKA-2214

: > Yes, we can consider in-progress as "errors". Other option could be > returning a different error code. Let us wait for others suggestions/concerns. > > Gwen Shapira wrote: > I wouldn't consider in-progress as error, since in-progress is pretty > much the

Re: Review Request 36333: Patch for KAFKA-2123

> On July 9, 2015, 1:42 a.m., Guozhang Wang wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, > > line 775 > > > > > > I think KAFKA-1894 is already fixed in this patch + KAFKA-2

[Discussion] Limitations on topic names

Hi Kafka Fans, If you have one topic named "kafka_lab_2" and the other named "kafka.lab.2", the topic level metrics will be named kafka_lab_2 for both, effectively making it impossible to monitor them properly. The reason this happens is that using "." in topic names is pretty common, especially

Re: [Discussion] Limitations on topic names

t; " is illegal, contains a character other than ASCII alphanumerics, '.', >> > '_' and '-'") >> > } >> > } >> > } >> > >> > On Fri, Jul 10, 2015 at 2:50 PM, Todd Palino wrote: >> > >

Re: [Discussion] Limitations on topic names

ely in >> topic names here :) I also happen to think it's the right choice, >> specifically because "." has more special meanings, as you noted. >> >> -Todd >> >> >> On Fri, Jul 10, 2015 at 1:30 PM, Gwen Shapira >> wrote: >> >

Re: [Discussion] Limitations on topic names

opic names. +1 for 2) going forward only >> without breaking previously created topics with "_" though that might >> require us to patch the code somewhat awkwardly till we phase it out a >> couple (purposely left vague to stay out of Ewen's wrath :-)) versions >>

Re: [Discussion] Limitations on topic names

leave it supporting > both and document the potential conflict in metrics. And if people use both > naming schemes, they probably deserve to suffer for their inconsistency :) > > -Ewen > > On Fri, Jul 10, 2015 at 3:28 PM, Gwen Shapira wrote: > >> I find dots more

Re: [Discussion] Limitations on topic names

w about the conversion or they will fail to find their topics in their monitoring tools. Not very nice to them, but I can't think of alternatives. I'll start with the doc patch :) On Sat, Jul 11, 2015 at 12:54 AM, Ewen Cheslack-Postava wrote: > On Fri, Jul 10, 2015 at 4:41 PM, Gwe

Re: Review Request 34403: Patch for KAFKA-2198

> On July 10, 2015, 4:46 p.m., Gwen Shapira wrote: > > core/src/main/scala/kafka/admin/TopicCommand.scala, lines 72-73 > > <https://reviews.apache.org/r/34403/diff/4/?file=1008271#file1008271line72> > > > > This is a bit unclean. I think its more idiomatic w

Re: Review Request 34403: Patch for KAFKA-2198

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34403/#review91579 --- Ship it! Ship It! - Gwen Shapira On July 13, 2015, 1:57 p.m

Re: Build failed in Jenkins: KafkaPreCommit #147

t; > > Changes: > > [cshapi] KAFKA-2198: kafka-topics.sh exits with 0 status on failures; patched > by Manikumar Reddy reviewed by Gwen Shapira > > -- > [...truncated 3035 lines...] > kafka.consumer.MetricsTest >

[CFP] Developer showcase at Strata NYC

Hi Kafka fans, Strata NYC is looking for emerging, open source, big data projects to highlight in the developer showcase: http://strataconf.com/big-data-conference-ny-2015/public/cfp/409 Depending on how you look at it, it is either science faire for grownups or free conference booth for open sou

Re: [CFP] Developer showcase at Strata NYC

You (or someone working on them) should submit the projects :) On Tue, Jul 14, 2015 at 4:27 PM, Neha Narkhede wrote: > All of the ones you've mentioned there sound great. I'd want to see copycat > there. > > On Tue, Jul 14, 2015 at 4:22 PM, Gwen Shapira wrote: > >&

Re: Contributor Request

Can you share your JIRA username? I couldn't find you. On Tue, Jul 14, 2015 at 8:04 PM, Warren Green wrote: > Hi, > > I'm interested in being added to the contributor list for Apache Kafka so > that I may assign myself to a newbie JIRA ticket. > > Thank you > > Warren Green > 248.390.8849 > Compu

Re: [DISCUSS] Using GitHub Pull Requests for contributions and code review

The script just runs a git push, it will be with your user (from git configuration), so you need privileges to push. On Wed, Jul 15, 2015 at 12:59 PM, Guozhang Wang wrote: > Another questions regarding the "kafka-merge-pr.py" script: once a > committer gives a LGTM on the PR, could the contributo

Re: Contributor Request

Cool, added you. Thanks for volunteering to help out! On Wed, Jul 15, 2015 at 3:49 PM, Warren Green wrote: > Hi Gwen, > > My username is: wgreen > > Thanks! > > Warren Green > 248.390.8849 > Computer Science Major | UIUC > > On Wed, Jul 15, 2015 at 12:23 PM, Gw

Re: Review Request 34641: Patch for KAFKA-2214

seems like the return value is either 1 or 2, at random (since it depends on order of foreach on a Map). I'm not sure thats expected :) Perhaps we can come up with something better defined? - Gwen Shapira On July 14, 2015, 10:13 a.m., Manikumar Reddy O

Re: Review Request 36548: Patch for KAFKA-2336

> On July 17, 2015, 4:26 a.m., Jiangjie Qin wrote: > > Looks good to me. > > Jiangjie Qin wrote: > Actually do we need to talk to Zookeeper every time? Can we read the data > from topic metadata cache directly? Good point, Jiangjie - looks like partitionFor is called on every ConsumerMeta

Re: [VOTE] Drop support for Scala 2.9 for the next release

+1 (binding) On Fri, Jul 17, 2015 at 3:26 AM, Ismael Juma wrote: > Hi all, > > I would like to start a vote on dropping support for Scala 2.9 for the next > release. People seemed to be in favour of the idea in previous discussions: > > * http://search-hadoop.com/m/uyzND1uIW3k2fZVfU1 > * http://s

Re: Review Request 36565: Patch for KAFKA-2345

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36565/#review92106 --- Ship it! - Gwen Shapira On July 17, 2015, 5:21 p.m., Ashish

Re: Build failed in Jenkins: KafkaPreCommit #156

Ick. It seemed to work locally. I'm checking what went wrong. Let me know if you want a revert. On Fri, Jul 17, 2015 at 10:39 AM, Apache Jenkins Server wrote: > See > > Changes: > > [cshapi] KAFKA-2345; Attempt to delete a topic already

Re: Build failed in Jenkins: KafkaPreCommit #156

my bad, I missed a file while committing. I did a "trivial" commit with the missing file and I think the build looks ok now. On Fri, Jul 17, 2015 at 10:42 AM, Gwen Shapira wrote: > Ick. It seemed to work locally. I'm checking what went wrong. > > Let me know if you w

Re: Official Kafka Gitter Room?

Slack sounds good and I'm not a fan of gitter (we use it for Apache Sqoop, I dislike the GUI and the guilt-inducing emails about all the messages I missed...) Do they have free rooms for open source projects? On Sat, Jul 18, 2015 at 6:10 PM, Andrew Psaltis wrote: > +1 for slack > > On Saturday, J

Re: Official Kafka Gitter Room?

So, as an experiment, I created: https://apachekafka.slack.com I figured we'll give it a whirl for a week or two for dev discussions, see how it goes and if we have activity we can add this to the website and announce on the lists. Gwen On Sat, Jul 18, 2015 at 6:18 PM, Gwen Shapira

Re: Official Kafka Gitter Room?

. So, if you want to try it out, yell for an invite :) On Sat, Jul 18, 2015 at 6:26 PM, Gwen Shapira wrote: > So, as an experiment, I created: > https://apachekafka.slack.com > > I figured we'll give it a whirl for a week or two for dev discussions, > see how it goes and if we

New Producer and "acks" configuration

Hi, I was looking into the different between acks = 0 and acks = 1 in the new producer, and was a bit surprised at what I found. Basically, if I understand correctly, the only difference is that with acks = 0, if the leader fails to append locally, it closes the network connection silently and wi

Re: Official Kafka Gitter Room?

might seen the same error > which was answered on the mailing lists before. slack/glitter can do > that? > > > On Sat, Jul 18, 2015, at 09:59 PM, Gwen Shapira wrote: >> FYI: >> Joining Slack requires an invitation. We can take invite-requests in >> the mailing lists

Re: Official Kafka Gitter Room?

invited :) btw. anyone already in Slack can invite others with: /invite_people On Sun, Jul 19, 2015 at 9:19 AM, Edward Ribeiro wrote: > Hi, Gwen, > > I would like a Slack invite, *please*. :-) > > Cheers, > Eddie > Em 19/07/2015 02:00, "Gwen Shapira" escre

Re: Review Request 36570: Patch for KAFKA-2337

.apache.org/r/36570/#comment146412> Nit: Our code standard includes not using curly brackets on a single line "if" block. - Gwen Shapira On July 20, 2015, 5:37 p.m., Grant Henke wrote: > > --- > This is an automatically g

Re: [VOTE] Switch to GitHub pull requests for new contributions

+1 (binding) on using PRs. It sounds like we need additional discussion on how the transition will happen. Maybe move that to a separate thread, to keep the vote easy to follow. On Tue, Jul 21, 2015 at 4:28 AM, Ismael Juma wrote: > Hi all, > > I would like to start a vote on switching to GitHub

Re: Review Request 36664: Patch for KAFKA-2353

e? core/src/main/scala/kafka/network/SocketServer.scala (line 461) <https://reviews.apache.org/r/36664/#comment146708> Turns out that catching Throwable is a really bad idea: https://www.sumologic.com/2014/05/05/why-you-should-never-catch-throwable-in-scala/ - Gwen Shapira On July 21,

Re: Review Request 36664: Patch for KAFKA-2353

> On July 22, 2015, 10:40 a.m., Ismael Juma wrote: > > core/src/main/scala/kafka/network/SocketServer.scala, line 467 > > > > > > As far as I can see `ClosedChannelException`, `IllegalStateException` > > and `Illega

Re: Review Request 36664: Patch for KAFKA-2353

> On July 21, 2015, 11:15 p.m., Gwen Shapira wrote: > > core/src/main/scala/kafka/network/SocketServer.scala, line 465 > > <https://reviews.apache.org/r/36664/diff/1/?file=1018238#file1018238line465> > > > > Turns out that catching Throwable

Re: [DISCUSS] KIP-27 - Conditional Publish

Tangent: I think we should complete the move of Produce / Fetch RPC to the client libraries before we add more revisions to this protocol. On Wed, Jul 22, 2015 at 11:02 AM, Jiangjie Qin wrote: > I missed yesterday's KIP hangout. I'm currently working on another KIP for > enriched metadata of mess

Re: Review Request 36664: Patch for KAFKA-2353

lout of this particular problem all > week, I'll add a few thoughts... > > > On Wed, Jul 22, 2015 at 10:51 AM, Gwen Shapira > wrote: >> >> >> >> > On July 21, 2015, 11:15 p.m., Gwen Shapira wrote: >> > > core/src/main/scala/kafka/network/Soc

Re: Review Request 36664: Patch for KAFKA-2353

> On July 21, 2015, 11:15 p.m., Gwen Shapira wrote: > > core/src/main/scala/kafka/network/SocketServer.scala, line 465 > > <https://reviews.apache.org/r/36664/diff/1/?file=1018238#file1018238line465> > > > > Turns out that catching Throwable

Re: Round Robin Among Consumers

As crazy as it sounds, there is an actual use-case there. Writing to HDFS can be very slow, so if you do a batch dump from a topic to HDFS, you may want more consumers reading from the topic than for "normal" streaming use-cases. We ended up modifying Camus to split a partition between multiple ma

Re: Round Robin Among Consumers

Agree. On Thu, Jul 23, 2015 at 2:43 PM, Jiangjie Qin wrote: > Ah, I see. Thanks for the use case, Gwen. I guess in that case it seems the > time to use low level consumer. > > Jiangjie (Becket) Qin > > On Thu, Jul 23, 2015 at 9:52 AM, Gwen Shapira wrote: > >> As craz

Re: Query about LogConfigs

Backward compatibility, I think. At least the "ms" one is fairly new, and I think we left the others to avoid break configuration during upgrade. On Fri, Jul 24, 2015 at 6:34 PM, Mayuresh Gharat wrote: > I was thinking why we have 3 different configs for the same property (log > retention) : >

Re: Query about LogConfigs

t;log.retention.ms"? In most > cases, people would want to retain their logs for at least a few minutes > I'd think. > > Aditya > > On Fri, Jul 24, 2015 at 6:49 PM, Gwen Shapira wrote: > >> Backward compatibility, I think. >> >> At least the "

Re: Round Robin Among Consumers

have > been needed at all) or because there were other constraints on partitioning? > > -Ewen > > On Thu, Jul 23, 2015 at 2:45 PM, Gwen Shapira wrote: > >> Agree. >> >> On Thu, Jul 23, 2015 at 2:43 PM, Jiangjie Qin >> wrote: >> > Ah, I see. Tha

Re: New Producer and "acks" configuration

cks = 1 - message is written to the leader log > acks = -1 - message is committed > > -Jay > > On Sat, Jul 18, 2015 at 10:50 PM, Gwen Shapira > wrote: > >> Hi, >> >> I was looking into the different between acks = 0 and acks = 1 in the >> new producer, an

Re: New Producer and "acks" configuration

since it still has to make it through the TCP buffers it > doesn't really guarantee anything that useful. > > -Ewen > > > On Sun, Jul 26, 2015 at 1:40 PM, Gwen Shapira wrote: > >> What bugs me is that even with acks = 0, the broker will append to >> local

Re: New Producer and "acks" configuration

se since it will be handled in callbacks. So the use of >> acks = 0 sounds very rare to me and I am not able to think of an usecase >> around it. >> >> Thanks, >> >> Mayuresh >> >> On Sun, Jul 26, 2015 at 2:40 PM, Gwen Shapira >> wrote: >&g

Re: [DISCUSS] Partitioning in Kafka

If you are used to map-reduce patterns, this sounds like a perfectly natural way to process streams of data. Call the first consumer "map-combine-log", the topic "shuffle-log" and the second consumer "reduce-log" :) I like that a lot. It works well for either "embarrassingly parallel" cases, or "s

Re: [DISCUSS] Partitioning in Kafka

; in particular, is the one I'm not convinced by since in practice if you > have a skewed distribution, you probably also won't actually see every key > in every partition; each worker actually only needs to maintain a subset of > the key set (and associated aggregate data) in memory

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Hi, Since we will be discussing KIP-28 in the call tomorrow, can you update the KIP with the feature-comparison with existing solutions? I admit that I do not see a need for single-event-producer-consumer pair (AKA Flume Interceptor). I've seen tons of people implement such apps in the last year,

Re: New Producer and "acks" configuration

o data loss pipelines: >> > >> > >> http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844 >> > Short version: as things are today, you have to *really* understand the >> > producer settings, and some producer internals, to get

Re: New Producer and "acks" configuration

, to get the exact behavior > you want. > > > On Mon, Jul 27, 2015 at 9:44 AM, Gwen Shapira wrote: > >> Yeah, using acks=0 should result in higher throughput since we are not >> limited by the roundtrip time to the broker. >> >> Btw. regarding in-flight reques

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Thanks Guazhang! Much clearer now, at least for me. Few comments / questions: 1. Perhaps punctuate(int numRecords) will be a nice API addition, some use-cases have record-count based windows, rather than time-based.. 2. The diagram for "Flexible partition distribution" shows two joins. Is the ide

Re: [DISCUSS] Reviewers in commit message

My two cents: The jira comment is a way for the committer to say "thank you" to people who were involved in the review process. It doesn't have any formal implications - the responsibility for committing good code is on the committer (thats the whole point). It doesn't even have informal implicati

Re: Review Request 28096: Patch for KAFKA-313

/scala/kafka/admin/ConsumerGroupCommand.scala (line 388) <https://reviews.apache.org/r/28096/#comment147883> If only CSV and JSON are allowed, what is NONE for? - Gwen Shapira On June 24, 2015, 6:14 p.m., Ashish Singh wrote: > > --

Re: [DISCUSS] Reviewers in commit message

I guess we see the "reviewer" part with different interpretations. What are the benefits you see of formalizing who gets mentioned as reviewer? On Wed, Jul 29, 2015 at 11:06 AM, Ismael Juma wrote: > Hi Gwen, > > Thanks for the feedback. Comments below. > > On Wed, Jul

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Since it sounds like it is not a separate framework (like CopyCat) but rather a new client, it will be nice to follow existing convention. Producer, Consumer and Processor (or Transformer) make sense to me. Note that the way the API is currently described, people may want to use it inside Spark ap

<    4   5   6   7   8   9   10   11   12   13   >