[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration

[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731497#comment-14731497 ] Jay Kreps commented on KAFKA-2510: -- I actually think we shouldn't prevent this

[jira] [Commented] (KAFKA-2397) leave group request

[ https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731091#comment-14731091 ] Jay Kreps commented on KAFKA-2397: -- [~hachikuji] Makes sense. > leave group

[jira] [Commented] (KAFKA-2397) leave group request

[ https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731052#comment-14731052 ] Jay Kreps commented on KAFKA-2397: -- Couple thoughts: 1. [~hachikuji] Does this nee

[jira] [Commented] (KAFKA-2496) New consumer from trunk doesn't work with 0.8.2.1 brokers

[ https://issues.apache.org/jira/browse/KAFKA-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726448#comment-14726448 ] Jay Kreps commented on KAFKA-2496: -- The first response is expected. The group manage

[jira] [Commented] (KAFKA-2486) New consumer performance

[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726349#comment-14726349 ] Jay Kreps commented on KAFKA-2486: -- [~hachikuji], [~benstopford] Fantastic! &

[jira] [Commented] (KAFKA-2486) New consumer performance

[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724448#comment-14724448 ] Jay Kreps commented on KAFKA-2486: -- [~ewencp] Nightly will be awesome! I think wh

[jira] [Commented] (KAFKA-2486) New consumer performance

[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724079#comment-14724079 ] Jay Kreps commented on KAFKA-2486: -- Where does this leave perf for the new consume

[jira] [Commented] (KAFKA-2486) New consumer performance

[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723615#comment-14723615 ] Jay Kreps commented on KAFKA-2486: -- I don't think we need to do anything specia

[jira] [Commented] (KAFKA-2486) New consumer performance

[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720237#comment-14720237 ] Jay Kreps commented on KAFKA-2486: -- [~guozhang] I interpret you to be saying that

[jira] [Commented] (KAFKA-2486) New consumer performance

[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720050#comment-14720050 ] Jay Kreps commented on KAFKA-2486: -- [~guozhang] these are long-poll fetches so if t

[jira] [Commented] (KAFKA-2486) New consumer performance

[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14719979#comment-14719979 ] Jay Kreps commented on KAFKA-2486: -- Hmm, I think it should be {code} w

[jira] [Commented] (KAFKA-2486) New consumer performance

[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14719963#comment-14719963 ] Jay Kreps commented on KAFKA-2486: -- Wait, we sleep? That is totally messed up, you c

Re: Kafka New Consumer Performance Test ThroughPut Degradation

I think this is likely a regression. The two clients had more or less equivalent performance when we checked in the code (see my post on this earlier in the year). Looks like maybe we broke something up in the interim? On my laptop the new consumer perf seems to have dropped from about ~200MB/sec

Re: [DISCUSS] Client-side Assignment for New Consumer

Hey Joel, I really don't think we should do both. There are pros and cons but we should make a decision and work on operationalizing one approach. Much of really making something like this work is getting all the bugs out, getting monitoring in place, getting rigorous system tests in place. Trying

[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API

[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712472#comment-14712472 ] Jay Kreps commented on KAFKA-2367: -- On unions: they were kind of a disaster at Link

[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance

[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709914#comment-14709914 ] Jay Kreps commented on KAFKA-2431: -- It would be good to also do the same test(s) aga

[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

[ https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709867#comment-14709867 ] Jay Kreps commented on KAFKA-2390: -- I think there could be multiple out of r

[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().

[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709609#comment-14709609 ] Jay Kreps commented on KAFKA-2389: -- I think this may be the first time Ewen has

[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

[ https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709607#comment-14709607 ] Jay Kreps commented on KAFKA-2390: -- Thanks for the clear problem statement, I agree

[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().

[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708587#comment-14708587 ] Jay Kreps commented on KAFKA-2389: -- I personally think the current API is preferable

[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().

[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708553#comment-14708553 ] Jay Kreps commented on KAFKA-2389: -- I agree with you that few people will want

[jira] [Created] (KAFKA-2460) Fix capitalization in SSL classes

Jay Kreps created KAFKA-2460: Summary: Fix capitalization in SSL classes Key: KAFKA-2460 URL: https://issues.apache.org/jira/browse/KAFKA-2460 Project: Kafka Issue Type: Improvement Affects

[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().

[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708480#comment-14708480 ] Jay Kreps commented on KAFKA-2389: -- Why would not passing in a callback imply that

[jira] [Commented] (KAFKA-2391) Blocking call such as position(), partitionsFor(), committed() and listTopics() should have a timeout

[ https://issues.apache.org/jira/browse/KAFKA-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708478#comment-14708478 ] Jay Kreps commented on KAFKA-2391: -- Putting timeouts in the api encourages hard-co

[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

[ https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708475#comment-14708475 ] Jay Kreps commented on KAFKA-2390: -- Guys, this seems like a cure that is worse than

[jira] [Commented] (KAFKA-873) Consider replacing zkclient with curator (with zkclient-bridge)

[ https://issues.apache.org/jira/browse/KAFKA-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705513#comment-14705513 ] Jay Kreps commented on KAFKA-873: - Curator will add a transitive dependency on Guava

[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API

[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703890#comment-14703890 ] Jay Kreps commented on KAFKA-2367: -- Yeah makes sense. Obviously no one is advocating

[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API

[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703871#comment-14703871 ] Jay Kreps commented on KAFKA-2367: -- Okay there are clearly divergent opinions

Re: [copycat] How to upgrade connectors

connector > development slightly more difficult. > > > On Mon, Aug 17, 2015 at 9:08 AM, Jay Kreps wrote: > > > Yeah I meant this purely as an approach for the connectors themselves to > > implement not that we would automatically suppress configs in the > > framewor

Re: [copycat] How to upgrade connectors

gt; The complexity shows up when upgradeConfig has to actually upgrade configs, > but thats our whole point, IMO. > > Gwen > > On Fri, Aug 14, 2015 at 10:48 AM, Jay Kreps wrote: > > > I think this is a great area to think about. > > > > What about the simpler i

[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API

[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697499#comment-14697499 ] Jay Kreps commented on KAFKA-2367: -- I'm more negative on using Avro. Here'

Re: [copycat] support for "batch" jobs

I thought batch was dead? :-) Yeah I think this would be really useful. Kafka kind of allows you to unify batch and streams since you produce or consume your stream on your own schedule so you would want the ingress/egress to work the same. Ewen, rather than sleeping, I think the use case is that

Re: [copycat] How to upgrade connectors

I think this is a great area to think about. What about the simpler informal scheme of just having cc ignore config it doesn't know about and instructing connector developers to maintain compatibility when renaming params (i.e. support both old and new) as we do for Kafka config. The steps would

Re: Kafka Indentation

t; > aaurad...@linkedin.com> > > > > > wrote: > > > > > > > > > > > I'm with Neha on this one. I don't have a strong preference on 2 > > vs 4 > > > > but > > > > > > I do think that consistency is more impo

Re: [DISCUSSION] KIP-29 - Add an IsrPropagateIntervalMs config to KafkaConfig

I guess the question, which I think is what Gwen was getting at, is if, rather than making this configurable, it might be possible to make this just work reliably and with the lowest possible latency in some automatic fashion? I raised group commit because that is a way to automatically batch under

Re: [DISCUSSION] KIP-29 - Add an IsrPropagateIntervalMs config to KafkaConfig

Would it be possible to take a "group commit" approach rather than configuring a delay. I.e. all changes that come in at once while another operation is occurring automatically get batched. This can sometimes be nicer and more automatic than another tunable parameter... -Jay On Fri, Aug 7, 2015 a

[jira] [Commented] (KAFKA-2397) leave group request

[ https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660279#comment-14660279 ] Jay Kreps commented on KAFKA-2397: -- [~guozhang] Basically "session" would

[jira] [Commented] (KAFKA-2397) leave group request

[ https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659470#comment-14659470 ] Jay Kreps commented on KAFKA-2397: -- [~guozhang] I haven't thought this through

Re: [DISCUSS] KIP-27 - Conditional Publish

Hey Jun, Yeah I think Ben is right, both these cases are covered. The algorithm is something like while(true) { v = get_local_val(key) v' = modify(v) try { log_to_kafka(v') put_local_val(k, v') break } catch(CasFailureException e) { warn("optimistic lock failure) } } W

[jira] [Comment Edited] (KAFKA-2397) leave group request

[ https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14652686#comment-14652686 ] Jay Kreps edited comment on KAFKA-2397 at 8/3/15 10:3

[jira] [Commented] (KAFKA-2397) leave group request

[ https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14652686#comment-14652686 ] Jay Kreps commented on KAFKA-2397: -- Nice summary [~onurkaraman]. I agree that addi

[jira] [Commented] (KAFKA-2400) Expose heartbeat frequency in new consumer configuration

[ https://issues.apache.org/jira/browse/KAFKA-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14652663#comment-14652663 ] Jay Kreps commented on KAFKA-2400: -- Also we should set good defaults: - session tim

Re: Kafka Consumer thoughts

>> List subscription(); >> List assignment(); >> >> interface RebalanceCallback { >> void onAssignment(List partitions); >> void onRevocation(List partitions); >> >> // handle non-existing topics, etc. >> void onError(Exception e); >

Re: KAFKA-2364 migrate docs from SVN to git

The issue last time was that Apache has special infrastructure for web hosting built around svn called svnpubsub. This is what takes the content changes and pushes them live to the site. They didn't yet have a gitpubsub at the time. If they've fixed that then we should be unblocked to switch. -Jay

Re: Kafka Consumer thoughts

s, RebalanceCallback callback); > void setAssignment(List partitions); > List subscription(); > List assignment(); > > interface RebalanceCallback { > void onAssignment(List partitions); > void onRevocation(List partitions); > > // handle non-existing topics, etc. >

Re: Kafka Consumer thoughts

ave another wrapper? Or are we expecting all the user to write their own > wrapper for KafkaConsumer? Some other comments are in line. > > Thanks, > > Jiangjie (Becket) Qin > > On Wed, Jul 29, 2015 at 3:16 PM, Jay Kreps wrote: > >> Some comments on the proposal: >> >&g

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Also, the most important part of any prototype, we should have a name for this producing-consumer-thingamgigy: Various ideas: - Kafka Streams - KStream - Kafka Streaming - The Processor API - Metamorphosis - Transformer API - Verwandlung For my part I think what people are trying to do is stream

Re: Kafka Consumer thoughts

future? That seems more >>>>>>>> elegant for the API's that make sense either in both flavors. From the >>>>>>>> users perspective, it is more consistent with the new producer. One >>>>>>>> easy >>>>

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

an use the producer and consumer to manage all this directly, but it is kind of low level. The goal of the processor/transformer/streaming client is to provide a user-friendly facade over these capabilities. -Jay On Tue, Jul 28, 2015 at 11:59 AM, Jay Kreps wrote: > Here is the link to the

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

as because of the lack > of > > quotas, which are in any case getting implemented. > > {quote} > > I am not sure that I followed this point. Is your point that with quota, > it > > is possible to host all data in a single cluster? > > > > -Yi > >

Re: New Producer and "acks" configuration

response > > anyway. > > > >> > > > > > >> > > This definitely is faster since you aren't waiting for the round > > > trip, > > > >> > but > > > >> > > it seems like it is of questionable value wit

Re: Kafka Consumer thoughts

processing the events. >>> >>> Case in point would be a simple app which reads events and writes to >>> Kafka. With quotas the write latencies to kafka could range from a few >>> milliseconds all the way to a tens of seconds. As the scale of processing >>>

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

esign of KIP-28 would probably need a smaller scope of > problem to solve, hence, limiting to solving partition management in a > single cluster. However, I would really hope the design won’t prevent the > use case of processing data directly from multiple clusters. In my opinion, > makin

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Hey Yi, For your other two points: - This definitely doesn't cover any kind of SQL or anything like this. - The prototype we started with just had process() as a method but Yasuhiro had some ideas of adding additional filter/aggregate convenience methods. We should discuss how this would fit wit

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

To follow on to one of Yi's points about taking ConsumerRecord vs topic/key/value. One thing we have found is that for user-facing APIs considering future API evolution is really important. If you do topic/key/value and then realize you need offset added you end up having to break everyones code. T

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

I agree that the KIP process doesn't fit well for big areas of development like the new consumer, copycat, or this. I think the approach for copycat where we do a "should this exist" KIP vote followed by a review on code checkin isn't ideal because of course the question of "should we do it" is di

Re: Kafka Indentation

unfair comparison. -Jay On Fri, Jul 24, 2015 at 8:07 AM, Ismael Juma wrote: > On Fri, Jul 24, 2015 at 2:00 AM, Jay Kreps wrote: > > > I do agree that working with a mixture of scala and java is a pain in the > > butt. What about considering the more extreme idea

[jira] [Commented] (KAFKA-2359) New consumer - partitions auto assigned only on poll

[ https://issues.apache.org/jira/browse/KAFKA-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640730#comment-14640730 ] Jay Kreps commented on KAFKA-2359: -- Interesting. This behavior was intentional in

Re: Kafka Indentation

For my part I actually prefer it how it is. The scala code is a different language, has different naming and capitalization conventions, etc. I actually like that it has a different look and feel so I don't get confused. In general, I think 4 spaces is standard for java and 2 spaces is pretty stand

[jira] [Comment Edited] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639080#comment-14639080 ] Jay Kreps edited comment on KAFKA-2350 at 7/23/15 4:2

[jira] [Comment Edited] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639080#comment-14639080 ] Jay Kreps edited comment on KAFKA-2350 at 7/23/15 4:2

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639080#comment-14639080 ] Jay Kreps commented on KAFKA-2350: -- [~becket_qin] I think there are three proposal

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638139#comment-14638139 ] Jay Kreps commented on KAFKA-2350: -- [~hachikuji] yeah i vote for pause/resume

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637317#comment-14637317 ] Jay Kreps commented on KAFKA-2350: -- [~hachikuji] Yeah I agree that would help this

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637126#comment-14637126 ] Jay Kreps commented on KAFKA-2350: -- [~becket_qin] Cool, we're on the same page,

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636098#comment-14636098 ] Jay Kreps commented on KAFKA-2350: -- +1 on resume instead of unpause though it doe

Re: [VOTE] Switch to GitHub pull requests for new contributions

+1 -Jay On Tue, Jul 21, 2015 at 4:28 AM, Ismael Juma wrote: > Hi all, > > I would like to start a vote on switching to GitHub pull requests for new > contributions. To be precise, the vote is on whether we should: > > * Update the documentation to tell users to use pull requests instead of > pa

Re: [DISCUSS] KIP-27 - Conditional Publish

It would be worth fleshing out the use cases a bit more and thinking through the overlap with the other proposals for transactions and idempotence (since likely we will end up with both). The advantage of this proposal is that it is really simple. If we go through use cases: 1. Stream processing:

Re: New Producer and "acks" configuration

acks=0 is a one-way send, the client doesn't need to wait on the response. Whether this is useful sort of depends on the client implementation. The new java producer does all sends async so "waiting" on a response isn't really a thing. For a client that lacks this, though, as some of them do, acks=

[jira] [Commented] (KAFKA-2260) Allow specifying expected offset on produce

[ https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633721#comment-14633721 ] Jay Kreps commented on KAFKA-2260: -- Yes, exactly. > Allow specifying expected of

[jira] [Commented] (KAFKA-2260) Allow specifying expected offset on produce

[ https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631512#comment-14631512 ] Jay Kreps commented on KAFKA-2260: -- [~yasuhiro.matsuda] had a good variation on

[jira] [Commented] (KAFKA-2260) Allow specifying expected offset on produce

[ https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630622#comment-14630622 ] Jay Kreps commented on KAFKA-2260: -- I like this idea a lot and the change is q

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630141#comment-14630141 ] Jay Kreps commented on KAFKA-2032: -- I think the goal was to make the assign

[jira] [Commented] (KAFKA-2333) Add rename topic support

[ https://issues.apache.org/jira/browse/KAFKA-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628909#comment-14628909 ] Jay Kreps commented on KAFKA-2333: -- [~gwenshap] Yeah totally, agree it will be tr

[jira] [Commented] (KAFKA-2333) Add rename topic support

[ https://issues.apache.org/jira/browse/KAFKA-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628878#comment-14628878 ] Jay Kreps commented on KAFKA-2333: -- Word. Probably does need a KIP, though. Presum

[jira] [Commented] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627395#comment-14627395 ] Jay Kreps commented on KAFKA-1835: -- Changing the return type of a method is a

Re: [DISCUSS] Json libraries for Kafka

if this is too > tangential to the discussion. > > Thanks, > Geoff > > On Tue, Jul 14, 2015 at 5:23 PM, Jay Kreps wrote: > > > Ah, makes sense. Yes that addresses my concerns. > > > > -Jay > > > > On Tue, Jul 14, 2015 at 5:19 PM, Ismael Juma wrote: &g

Re: [DISCUSS] Json libraries for Kafka

Ah, makes sense. Yes that addresses my concerns. -Jay On Tue, Jul 14, 2015 at 5:19 PM, Ismael Juma wrote: > Hi Jay, > > Comments inline. > > On Tue, Jul 14, 2015 at 11:04 PM, Jay Kreps wrote: > > > Is this going to become a dependency for core and then transitively f

Re: [DISCUSS] Json libraries for Kafka

Is this going to become a dependency for core and then transitively for the old clients? The current json library is definitely not great, but it does parse json and it's not used in any context where performance is a concern. Because the older clients aren't well modularized, adding core dependen

Re: [VOTE] KIP-26 Add Copycat connector framework for data import/export

+1 Super excited! -Jay On Tue, Jul 14, 2015 at 2:09 PM, Ewen Cheslack-Postava wrote: > Hi all, > > Let's start a vote on KIP-26: Add Copycat connector framework for data > import/export > > For reference, here's the wiki: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851

Re: [Discussion] Limitations on topic names

Unfortunately '.' is pretty common too. I agree that it is perverse, but people seem to do it. Breaking all the topics with '.' in the name seems like it could be worse than combining metrics for people who have a 'foo_bar' AND 'foo.bar' (and after all, having both is DEEPLY perverse, no?). Where

Re: [ANNOUNCE] New Committer

Congrats, welcome to the team! -Jay On Mon, Jul 6, 2015 at 6:08 PM, Joe Stein wrote: > I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen > Shapira as a committer and Gwen has accepted. > > Please join me on welcoming and congratulating Gwen. > > Thanks for the contribu

Re: Build failed in Jenkins: KafkaPreCommit #136

Ha ha, hoisted on our own petard. -Jay On Mon, Jul 6, 2015 at 4:39 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See > > Changes: > > [junrao] kafka-2248; Use Apache Rat to enforce copyright headers; patched > by Ewen C

Re: EOL JDK 1.6 for Kafka

+1 -Jay On Wed, Jul 1, 2015 at 3:32 PM, Guozhang Wang wrote: > +1. > > On Wed, Jul 1, 2015 at 1:00 PM, Gwen Shapira > wrote: > > > Huge +1. > > > > I don't think there is any other project that still supports 1.6. > > > > On Wed, Jul 1, 2015 at 8:05 AM, Harsha wrote: > > > Hi, > > > D

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

We were talking on the call about a logo...so here I present "The Original Copycat": http://shirtoid.com/67790/the-original-copycat/ -Jay On Tue, Jun 23, 2015 at 6:28 PM, Gwen Shapira wrote: > One more reason to have CopyCat as a separate project is to sidestep > the entire "Why CopyCat and not

Re: Incomplete Messages

The documentation currently includes the caveat that "As an optimization the server is allowed to return a partial message at the end of the message set. Clients should handle this case." As to whether or not this is a good feature: it is definitely a bad feature. It was originally there because w

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

Hey Sriram, Good question here was the thinking: 1. I think the argument we are making is that a stream processing framework is the right way to do complex transformations. We can bake in some mechanism for simple, single-row transforms in copycat, but for anything more complex copycat is really

[jira] [Commented] (KAFKA-1215) Rack-Aware replica assignment option

[ https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598643#comment-14598643 ] Jay Kreps commented on KAFKA-1215: -- This is great! > Rack-Aware replica ass

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

ools. However, it sounds solvable with few links. > > Sorry for the lengthy essay - I'm a bit passionate about connectors > and want to see CopyCat off to a great start in life :) > > (BTW. I think Apache is a great place for CopyCat. I'll be happy to > help with the

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

ce only make sense when we > can have that guarantee on the entire data flow (not just portions of the > flow). > - Connector Variety: Flume and Sqoop already have out of the box- support > for multiple destinations and sources. > > > -roshan > > > > On 6/19/15 2:

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

for connectors. Does the addition include just the framework > (and perhaps an example or two), or do we plan to start accepting > connectors to Apache Kafka project? > > Gwen > > On Thu, Jun 18, 2015 at 3:09 PM, Jay Kreps wrote: > > I think the only problem we came up with

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

the name. KopyKat? More unique, easy to write, > pronounce, remember... > > Otis > > > > > On Jun 18, 2015, at 13:36, Jay Kreps wrote: > > > > 1. We were calling the plugins connectors (which is kind of a generic way > > to say either source or sink) and the f

Re: Review Request 34789: Patch for KAFKA-2168

e this very cheap right? Given we acquire locks for all the perf stats and there are several stat updates for each poll call I don't see the problem with just synchronizing. - Jay Kreps On June 18, 2015, 9:40 p.m., Jason Gust

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

1. We were calling the plugins connectors (which is kind of a generic way to say either source or sink) and the framework copycat. The pro of copycat is it is kind of fun. The con is that it doesn't really say what it does. The Kafka Connector Framework would be a duller but more intuitive name, bu

Re: Any reason why LogConfig defaults are different from KafkaConfig defaults?

I think this is just a mistake and you should definitely fix it. Nice catch. -Jay On Wed, Jun 17, 2015 at 4:34 PM, Gwen Shapira wrote: > Hi, > > I'm migrating KafkaConfig and LogConfig over to AbstractConfig > interface,and I've ran into something strange: > > All of LogConfig parameters also e

[jira] [Commented] (KAFKA-2092) New partitioning for better load balancing

[ https://issues.apache.org/jira/browse/KAFKA-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588420#comment-14588420 ] Jay Kreps commented on KAFKA-2092: -- Hmm, not sure if I follow this. My understandin

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582528#comment-14582528 ] Jay Kreps commented on KAFKA-2168: -- Hey [~guozhang], have you had a chance to loo

Re: [VOTE] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

+1 On Tue, Jun 9, 2015 at 11:24 PM, Honghai Chen wrote: > Hi Kafka, > > After a long discussion, please help vote again for the > KIP. Thanks. > > > > I wrote a KIP for this after some discussion on KAFKA-1646. > https://issues.apache.org/jira/browse/KAFKA-1646 > > > https://cwi

Re: [Discussion] New Consumer API / Protocol

My two cents: Overall I think our focus as far as extensions go should be on (1) things which would be incompatible if changed later and (2) operationalization. There are lots of new apis that could be nice to have, but I think if adding them later will just be an api addition we should hold off a

Re: Review Request 34789: Patch for KAFKA-2168

/KafkaConsumer.java <https://reviews.apache.org/r/34789/#comment139570> Do we need this? There is no real guarantee on the poll time, so it seems that we could just return when wakeup is called. - Jay Kreps On June 5, 2015, 7:45 p.m., Jason Gustafson

<    1   2   3   4   5   6   7   8   9   10   >