[jira] [Commented] (KAFKA-1528) Normalize all the line endings

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395967#comment-14395967 ] Jay Kreps commented on KAFKA-1528: -- [~evvers] is this still active? Normalize all

[jira] [Commented] (KAFKA-1904) run sanity failed test

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395968#comment-14395968 ] Jay Kreps commented on KAFKA-1904: -- Is this still active? run sanity failed test

[jira] [Commented] (KAFKA-1351) String.format is very expensive in Scala

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395970#comment-14395970 ] Jay Kreps commented on KAFKA-1351: -- [~nehanarkhede] is this still active? If not can we

[jira] [Resolved] (KAFKA-2050) Avoid calling .size() on java.util.ConcurrentLinkedQueue

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-2050. -- Resolution: Fixed Reviewer: Jay Kreps Assignee: Jay Kreps (was: Jun Rao) Nice catch

[jira] [Resolved] (KAFKA-2002) It does not work when kafka_mx4jenable is false

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-2002. -- Resolution: Fixed Reviewer: Jay Kreps Assignee: Jay Kreps Nice catch. It does not work

[jira] [Commented] (KAFKA-1054) Eliminate Compilation Warnings for 0.8 Final Release

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395980#comment-14395980 ] Jay Kreps commented on KAFKA-1054: -- [~nehanarkhede] I think this is waiting on you

[jira] [Resolved] (KAFKA-2024) Cleaner can generate unindexable log segments

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-2024. -- Resolution: Fixed Reviewer: Jay Kreps Nice catch and nice patch. Applied. Cleaner can generate

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-04-04 Thread Jay Kreps
. The implementation can be the same as what's in the new metrics package. Thanks, Jun On Thu, Mar 19, 2015 at 8:09 PM, Jay Kreps jay.kr...@gmail.com wrote: Yeah I was saying was that we are blocked on picking an approach for metrics

[jira] [Created] (KAFKA-2091) Expose a Partitioner interface in the new producer

2015-04-04 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-2091: Summary: Expose a Partitioner interface in the new producer Key: KAFKA-2091 URL: https://issues.apache.org/jira/browse/KAFKA-2091 Project: Kafka Issue Type

Re: Review Request 31806: Patch for KAFKA-1501

2015-04-04 Thread Jay Kreps
://reviews.apache.org/r/31806/#comment127980 Nm, figured it out. - Jay Kreps On March 25, 2015, 7:45 a.m., Ewen Cheslack-Postava wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r

[jira] [Resolved] (KAFKA-1728) update 082 docs

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-1728. -- Resolution: Fixed Closing since it seems like the patch was committed. update 082 docs

[jira] [Resolved] (KAFKA-1343) Kafka consumer iterator thread stalls

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-1343. -- Resolution: Cannot Reproduce Kafka consumer iterator thread stalls

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395959#comment-14395959 ] Jay Kreps commented on KAFKA-1367: -- [~nehanarkhede] Does this problem still exist

[jira] [Updated] (KAFKA-1501) transient unit tests failures due to port already in use

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1501: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed. [~ewencp] is my personal

[jira] [Resolved] (KAFKA-1086) Improve GetOffsetShell to find metadata automatically

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-1086. -- Resolution: Fixed Assuming this is either fixed or not going to be. Improve GetOffsetShell to find

[jira] [Resolved] (KAFKA-1500) adding new consumer requests using the new protocol

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-1500. -- Resolution: Fixed adding new consumer requests using the new protocol

[jira] [Resolved] (KAFKA-264) Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-264. - Resolution: Won't Fix I think this is no longer active. Change the consumer side load balancing

Re: [DISCUSS] New partitioning for better load balancing

2015-04-04 Thread Jay Kreps
Hey guys, I think the first step here would be to expose a partitioner interface for the new producer that would make it easy to plug in these different strategies. I filed a JIRA for this: https://issues.apache.org/jira/browse/KAFKA-2091 -Jay On Fri, Apr 3, 2015 at 9:36 AM, Harsha

Re: Review Request 31806: Patch for KAFKA-1501

2015-04-04 Thread Jay Kreps
://reviews.apache.org/r/31806/#comment127979 Why are we removing these tests? - Jay Kreps On March 25, 2015, 7:45 a.m., Ewen Cheslack-Postava wrote: --- This is an automatically generated e-mail. To reply, visit: https

[jira] [Commented] (KAFKA-1729) add doc for Kafka-based offset management in 0.8.2

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395955#comment-14395955 ] Jay Kreps commented on KAFKA-1729: -- [~jjkoshy] is this still waiting on an update from

[jira] [Commented] (KAFKA-1303) metadata request in the new producer can be delayed

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395960#comment-14395960 ] Jay Kreps commented on KAFKA-1303: -- [~junrao] What is the status of this? metadata

[jira] [Resolved] (KAFKA-583) SimpleConsumerShell may receive less data inconsistently

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-583. - Resolution: Won't Fix SimpleConsumerShell may receive less data inconsistently

[jira] [Resolved] (KAFKA-777) Add system tests for important tools

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-777. - Resolution: Won't Fix Add system tests for important tools

[jira] [Resolved] (KAFKA-552) No error messages logged for those failing-to-send messages from Producer

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-552. - Resolution: Won't Fix No error messages logged for those failing-to-send messages from Producer

[jira] [Resolved] (KAFKA-530) kafka.server.KafkaApis: kafka.common.OffsetOutOfRangeException

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-530. - Resolution: Won't Fix kafka.server.KafkaApis: kafka.common.OffsetOutOfRangeException

[jira] [Commented] (KAFKA-1293) Mirror maker housecleaning

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395991#comment-14395991 ] Jay Kreps commented on KAFKA-1293: -- [~mwarhaftig] Did we get you wiki access? Mirror

[jira] [Resolved] (KAFKA-1996) Scaladoc error: unknown tag parameter

2015-04-04 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-1996. -- Resolution: Fixed Reviewer: Jay Kreps Scaladoc error: unknown tag parameter

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-04-03 Thread Jay Kreps
package. Thanks, Jun On Thu, Mar 19, 2015 at 8:09 PM, Jay Kreps jay.kr...@gmail.com wrote: Yeah I was saying was that we are blocked on picking an approach for metrics but not necessarily the full conversion. Clearly if we pick the new

Re: [DISCUSSION] Keep docs updated per jira

2015-03-31 Thread Jay Kreps
Yeah the protocol should probably move off the wiki and into the release-versioned docs. -Jay On Mon, Mar 30, 2015 at 5:17 PM, Joel Koshy jjkosh...@gmail.com wrote: Also for the wikis - those should probably correspond to the latest released version right? So for e.g., if we add or modify the

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

2015-03-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388879#comment-14388879 ] Jay Kreps commented on KAFKA-2076: -- This makes sense, and would be useful. The question

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

2015-03-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389020#comment-14389020 ] Jay Kreps commented on KAFKA-2076: -- Hey [~granthenke] I don't really get the relationship

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

2015-03-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389700#comment-14389700 ] Jay Kreps commented on KAFKA-2076: -- I guess the question we need to answer is whether we

[jira] [Commented] (KAFKA-2035) Add a topic config cache.

2015-03-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389468#comment-14389468 ] Jay Kreps commented on KAFKA-2035: -- Currently this is the log itself--Log.config always

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

2015-03-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389483#comment-14389483 ] Jay Kreps commented on KAFKA-2076: -- Actually rather than trying to ensure we have

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

2015-03-31 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389553#comment-14389553 ] Jay Kreps commented on KAFKA-2076: -- In terms of mechanics we did a discussion

[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work

2015-03-30 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386942#comment-14386942 ] Jay Kreps commented on KAFKA-2046: -- Hmm, so we have a configuration which, if you set

Re: [DISCUSSION] Keep docs updated per jira

2015-03-30 Thread Jay Kreps
dependency. I get the feeling git is no longer as taboo as it once was in Apache... -Jay On Sat, Mar 28, 2015 at 9:46 PM, Gwen Shapira gshap...@cloudera.com wrote: On Thu, Mar 26, 2015 at 9:46 PM, Jay Kreps jay.kr...@gmail.com wrote: The reason the docs are in svn is that when we were setting

Kafka confusion

2015-03-30 Thread Jay Kreps
Gwen had a really good blog post on confusing things about Kafka: http://ingest.tips/2015/03/26/what-is-confusing-about-kafka/ A lot of these are in-flight now (e.g. consumer) or finished (e.g. delete topic, kind of, and non-sticky producer partitioning). Do we have bugs for the rest? It would

Re: Metrics package discussion

2015-03-28 Thread Jay Kreps
but in the long run we will have more useful graphs. What do people think? I can start a vote thread on this once we have a couple more opinions. Thanks, Aditya From: Jay Kreps [jay.kr...@gmail.com] Sent: Thursday, March 26, 2015 2:29 PM To: dev

[jira] [Commented] (KAFKA-2045) Memory Management on the consumer

2015-03-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385387#comment-14385387 ] Jay Kreps commented on KAFKA-2045: -- Okay so since most the discussion here

[jira] [Commented] (KAFKA-2045) Memory Management on the consumer

2015-03-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385375#comment-14385375 ] Jay Kreps commented on KAFKA-2045: -- The challenge with a rewrite is that it will be hard

[jira] [Resolved] (KAFKA-936) Kafka Metrics Memory Leak

2015-03-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-936. - Resolution: Fixed Kafka Metrics Memory Leak -- Key: KAFKA-936

[jira] [Created] (KAFKA-2063) Bound fetch response size

2015-03-28 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-2063: Summary: Bound fetch response size Key: KAFKA-2063 URL: https://issues.apache.org/jira/browse/KAFKA-2063 Project: Kafka Issue Type: Improvement Reporter

[jira] [Commented] (KAFKA-2045) Memory Management on the consumer

2015-03-27 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384100#comment-14384100 ] Jay Kreps commented on KAFKA-2045: -- [~rzidane] I do think a prototype would help show

[jira] [Commented] (KAFKA-2045) Memory Management on the consumer

2015-03-27 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384558#comment-14384558 ] Jay Kreps commented on KAFKA-2045: -- Yeah, as you say, I think bounding memory would still

Re: [VOTE] KIP-15 Add a close method with timeout to new producer

2015-03-27 Thread Jay Kreps
+1 Thanks for your patience Jiangjie! -Jay On Fri, Mar 27, 2015 at 11:48 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: https://cwiki.apache.org/confluence/display/KAFKA/KIP-15+-+Add+a+close+method+with+a+timeout+in+the+producer Let us vote again!

[jira] [Commented] (KAFKA-2045) Memory Management on the consumer

2015-03-26 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383344#comment-14383344 ] Jay Kreps commented on KAFKA-2045: -- Hey [~rzidane], one statically allocated ByteBuffer

Re: [DISCUSSION] Keep docs updated per jira

2015-03-26 Thread Jay Kreps
The reason the docs are in svn is that when we were setting up the site apache required that to publish doc changes. Two possible fixes: 1. Follow up with infra to see if they have git integration working yet 2. Move to a model where doc source is kept in the main git and we use jenkyl or

[jira] [Commented] (KAFKA-1501) transient unit tests failures due to port already in use

2015-03-26 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382431#comment-14382431 ] Jay Kreps commented on KAFKA-1501: -- Fantastic! transient unit tests failures due

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-26 Thread Jay Kreps
providing more guarantees and safer. Thanks. Jiangjie (Becket) Qin On 3/25/15, 9:20 AM, Jay Kreps jay.kr...@gmail.com wrote: Wait, actually, why would the thread block forever? I would understand throwing an error and failing immediately (fail fast) and I would

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-26 Thread Jay Kreps
the sender to just exit. On Thu, Mar 26, 2015 at 01:29:41PM -0700, Jay Kreps wrote: Hey guys, I think there are really two choices: 1. Blocking for the time the user requested 2. Blocking indefinitely irrespective of what the user requested When we were discussing I thought we were

Re: Metrics package discussion

2015-03-26 Thread Jay Kreps
these co-exist with the existing metrics. This leads to 2 metric packages being used on the server, but they are both pulled in as dependencies anyway. Using this for metrics we can quota on may not be a bad place to start. Thanks, Aditya From: Jay Kreps

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-26 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382880#comment-14382880 ] Jay Kreps commented on KAFKA-527: - Do we have any kind of before/after performance

Re: Metrics package discussion

2015-03-26 Thread Jay Kreps
Here was my understanding of the issue last time. The yammer metrics use a random sample of requests to estimate the histogram. This allocates a fairly large array of longs (their values are longs rather than floats). A reasonable sample might be 8k entries which would give about 64KB per

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-25 Thread Jay Kreps
+1 -Jay On Tue, Mar 24, 2015 at 2:43 PM, Guozhang Wang wangg...@gmail.com wrote: +1. On Tue, Mar 24, 2015 at 2:15 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: https://cwiki.apache.org/confluence/display/KAFKA/KIP-15+-+Add+a+close+method+with+a+timeout+in+the+producer As a

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-25 Thread Jay Kreps
, 2015 at 9:17 AM, Jay Kreps jay.kr...@gmail.com wrote: +1 -Jay On Tue, Mar 24, 2015 at 2:43 PM, Guozhang Wang wangg...@gmail.com wrote: +1. On Tue, Mar 24, 2015 at 2:15 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: https://cwiki.apache.org/confluence/display/KAFKA/KIP-15+-+Add

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-25 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380706#comment-14380706 ] Jay Kreps commented on KAFKA-527: - Yeah it would be great to get the [~guozhang]'s patch

[jira] [Commented] (KAFKA-2050) Avoid calling .size() on java.util.ConcurrentLinkedQueue

2015-03-25 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381337#comment-14381337 ] Jay Kreps commented on KAFKA-2050: -- Nice catch. Avoid calling .size

[jira] [Commented] (KAFKA-2045) Memory Management on the consumer

2015-03-24 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378476#comment-14378476 ] Jay Kreps commented on KAFKA-2045: -- There are really two issues: 1. Bounding fetch size

Re: [Discussion] Using Client Requests and Responses in Server

2015-03-22 Thread Jay Kreps
- the SimpleConsumer uses them. So we'll need conversion at some point. I'll try to make the conversion point just before hitting a public API that we can't modify, and hopefully it won't look too arbitrary. On Wed, Mar 18, 2015 at 5:24 PM, Jay Kreps jay.kr...@gmail.com wrote: I think either

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-19 Thread Jay Kreps
? On Wed, Mar 18, 2015 at 2:52 PM, Jay Kreps jay.kr...@gmail.com wrote: Personally I'm in favor of (1) just to reduce the number of different APIs. People will find the difference between abort and close subtle and confusing and the only instance where you want it is this somewhat

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Jay Kreps
. Dynamic Configuration management - Being discussed in KIP-5. Basically we need something that will model default quotas and allow per-client overrides. Is there something else that I'm missing? Thanks, Aditya From: Jay Kreps

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-18 Thread Jay Kreps
in progress and related JIRA that are interdependent and common work. ~ Joe Stein On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps jay.kr...@gmail.com

Re: [Discussion] Using Client Requests and Responses in Server

2015-03-18 Thread Jay Kreps
a larger replacement as a follow up? Gwen On Wed, Mar 18, 2015 at 3:32 PM, Jay Kreps jay.kr...@gmail.com wrote: Great. For (3) yeah I think we should just think through the end-to-end pattern for these versioned requests since it seems like we will have a number of them. The serialization

Re: [Discussion] Using Client Requests and Responses in Server

2015-03-18 Thread Jay Kreps
don't think any of this necessarily blocks this ticket since at the moment we don't have tons of versions of requests out there. -Jay On Wed, Mar 18, 2015 at 2:50 PM, Gwen Shapira gshap...@cloudera.com wrote: See inline responses: On Wed, Mar 18, 2015 at 2:26 PM, Jay Kreps jay.kr...@gmail.com

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-18 Thread Jay Kreps
. Is there something else that I'm missing? Thanks, Aditya From: Jay Kreps [jay.kr...@gmail.com] Sent: Wednesday, March 18, 2015 2:10 PM To: dev@kafka.apache.org Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas Hey Steven, The current proposal

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-18 Thread Jay Kreps
the user to know. Blocking as a means of doing that seems wrong and annoying. On Sun, Mar 15, 2015 at 11:56 AM, Jay Kreps jay.kr...@gmail.com wrote: Cool. I think blocking is good or alternately throwing an exception directly

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-18 Thread Jay Kreps
just drop the msg and return an error/status code indicates the drop and why. then producer can just move on and accept the drop. shared buffer won't be saturated and other 9 topics won't be penalized. Thanks, Steven On Tue, Mar 17, 2015 at 9:44 AM, Jay Kreps jay.kr...@gmail.com wrote

Re: [Discussion] Using Client Requests and Responses in Server

2015-03-18 Thread Jay Kreps
Hey Gwen, This makes sense to me. A couple of thoughts, mostly confirming what you said I think: 1. Ideally we would move completely over to the new style of request definition for server-side processing, even for the internal requests. This way all requests would have the same

[jira] [Commented] (KAFKA-1912) Create a simple request re-routing facility

2015-03-18 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367966#comment-14367966 ] Jay Kreps commented on KAFKA-1912: -- 1. I think this can be a single thread. The number

[jira] [Commented] (KAFKA-1912) Create a simple request re-routing facility

2015-03-17 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365766#comment-14365766 ] Jay Kreps commented on KAFKA-1912: -- I think this is easier than I thought. We have

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-17 Thread Jay Kreps
:10 PM, Jay Kreps jay.kr...@gmail.com wrote: My concern is that as soon as you start encoding non-error response information into error codes the next question is what to do if two such codes apply (i.e. you have a replica down and the response is quota'd). I think I am

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-17 Thread Jay Kreps
sense), isn't it a similar case? On Mon, Mar 16, 2015 at 10:10 PM, Jay Kreps jay.kr...@gmail.com wrote: My concern is that as soon as you start encoding non-error response information into error codes the next question is what to do if two

Re: Question about concurrency during Log config change

2015-03-16 Thread Jay Kreps
You are correct. Each read will be a valid value but there is no guarantee that subsequent reads will read from the same config. I don't think that is a problem, do you? If we want to strengthen the guarantee we can grab the config once in the method val config = log.config and then do however

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-16 Thread Jay Kreps
Hey Jun, I'd really really really like to avoid that. Having just spent a bunch of time on the clients, using the error codes to encode other information about the response is super dangerous. The error handling is one of the hardest parts of the client (Guozhang chime in here). Generally the

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-16 Thread Jay Kreps
-error codes. It won't be backward compatible (i.e. clients that currently do else throw will throw on non-errors), but perhaps its worthwhile. On Mon, Mar 16, 2015 at 9:42 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Jun, I'd really really really like to avoid that. Having just spent a bunch

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-15 Thread Jay Kreps
sender thread so user will notice something went wrong. Thanks. Jiangjie (Becket) Qin On 3/14/15, 11:37 AM, Jay Kreps jay.kr...@gmail.com wrote: Hey Jiangjie, I think this is going to be very confusing that close(0) waits indefinitely and close(-1) waits for 0. I understand

[jira] [Commented] (KAFKA-2020) I expect ReplicaNotAvailableException to have proper Javadocs

2015-03-14 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361868#comment-14361868 ] Jay Kreps commented on KAFKA-2020: -- Is this the case where we are sending back an error

Re: [VOTE] KIP-16: Replica Lag Tuning

2015-03-14 Thread Jay Kreps
+1 -Jay On Fri, Mar 13, 2015 at 9:54 AM, Aditya Auradkar aaurad...@linkedin.com.invalid wrote: Details in the KIP, Jira and RB. https://cwiki.apache.org/confluence/display/KAFKA/KIP+16+:+Automated+Replica+Lag+Tuning https://issues.apache.org/jira/browse/KAFKA-1546

[jira] [Commented] (KAFKA-2020) I expect ReplicaNotAvailableException to have proper Javadocs

2015-03-14 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361952#comment-14361952 ] Jay Kreps commented on KAFKA-2020: -- Yeah we need to just fix that. I actually strongly

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-14 Thread Jay Kreps
¹s comments. I¹d like to initiate the voting process if no further comments are received by tomorrow. Jiangjie (Becket) Qin On 3/8/15, 9:45 AM, Jay Kreps jay.kr...@gmail.com wrote: Hey Jiangjie, Can you capture the full motivation and use cases for the feature? This mentions your

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-12 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359829#comment-14359829 ] Jay Kreps commented on KAFKA-1646: -- Yeah I was proposing doing this across the board

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Jay Kreps
+Proposals in progress and related JIRA that are interdependent and common work. ~ Joe Stein On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps jay.kr...@gmail.com wrote: Let's stay on Google hangouts that will also record and make the sessions

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-12 Thread Jay Kreps
On Wed, Mar 11, 2015 at 10:22 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Todd, Yeah it is kind of weird to do the quota check after taking a request, but since the penalty is applied during that request and it just delays you to the right rate, I think it isn't exactly wrong. I admit

Re: 0.8.3 release plan

2015-03-11 Thread Jay Kreps
Yeah I'd be in favor of a quicker, smaller release but I think as long as we have these big things in flight we should probably keep the release criteria feature-based rather than time-based, though (e.g. when X works not every other month). Ideally the next release would have at least a beta

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

2015-03-11 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358063#comment-14358063 ] Jay Kreps commented on KAFKA-1546: -- Well iiuc the wonderfulness of this feature

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

2015-03-11 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358039#comment-14358039 ] Jay Kreps commented on KAFKA-1546: -- Personally I don't think it really needs a KIP

Re: [VOTE] KIP-15 add a close method with timeout to KafkaProducer

2015-03-11 Thread Jay Kreps
Yeah guys, I'd like to second that. I'd really really love to get the quality of these to the point where we could broadly solicit user input and use them as a permanent document of the alternatives and rationale. I know it is a little painful to have process, but I think we all saw what happened

Re: 0.8.3 release plan

2015-03-11 Thread Jay Kreps
With regard to mm, I was kind of assuming just based on the amount of work that that would go in for sure, but yeah I agree it is important. -Jay On Wed, Mar 11, 2015 at 9:39 PM, Jay Kreps jay.kr...@gmail.com wrote: What I was trying to say was let's do a real release whenever either consumer

Re: 0.8.3 release plan

2015-03-11 Thread Jay Kreps
at 9:18 PM, Jay Kreps jay.kr...@gmail.com wrote: Yeah the real question is always what will we block on? I don't think we should try to hold back smaller changes. In this bucket I would include most things you described: mm improvements, replica assignment tool improvements, flush

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-11 Thread Jay Kreps
From: Jay Kreps [jay.kr...@gmail.com] Sent: Monday, March 09, 2015 5:01 PM To: dev@kafka.apache.org Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas Hey Todd, Nice points, let me try to respond: Plugins Yeah let me explain what I mean about plugins. The contracts that matter for us

[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-11 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1646: - Reviewer: (was: Jay Kreps) Improve consumer read performance for Windows

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-11 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358125#comment-14358125 ] Jay Kreps commented on KAFKA-1646: -- Hey [~waldenchen] this patch is adding a TON

Re: 0.8.3 release plan

2015-03-11 Thread Jay Kreps
11, 2015 at 8:29 PM, Jay Kreps jay.kr...@gmail.com wrote: Yeah I'd be in favor of a quicker, smaller release but I think as long as we have these big things in flight we should probably keep the release criteria feature-based rather than time-based, though (e.g. when X works not every other

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-11 Thread Jay Kreps
as opposed to what is good (or right). But given the details of where to hold the request, I have less of a concern with the burden on the broker. -Todd On Mon, Mar 9, 2015 at 5:01 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Todd, Nice points, let me try to respond: Plugins Yeah let

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-09 Thread Jay Kreps
On Sun, Mar 8, 2015 at 10:52 AM, Jay Kreps jay.kr...@gmail.com wrote: Hey Adi, Great write-up. Here are some comments: 1. I don't think you need a way to disable quotas on a per-client basis, that is just the equivalent of setting the quota to be infinite, right? 2. I agree

[jira] [Commented] (KAFKA-1930) Move server over to new metrics library

2015-03-09 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354376#comment-14354376 ] Jay Kreps commented on KAFKA-1930: -- Hey [~gwenshap] this is wheel reinvention

[jira] [Updated] (KAFKA-527) Compression support does numerous byte copies

2015-03-08 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-527: Assignee: Yasuhiro Matsuda (was: Jay Kreps) Compression support does numerous byte copies

[jira] [Commented] (KAFKA-1313) Support adding replicas to existing topic partitions

2015-03-08 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352246#comment-14352246 ] Jay Kreps commented on KAFKA-1313: -- This is actually in the alter topic command today

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-08 Thread Jay Kreps
Hey Adi, Great write-up. Here are some comments: 1. I don't think you need a way to disable quotas on a per-client basis, that is just the equivalent of setting the quota to be infinite, right? 2. I agree that the configuration problem is a general part of doing dynamic configuration, and it is

<    1   2   3   4   5   6   7   8   9   10   >