[jira] [Comment Edited] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958698#comment-14958698 ] Ismael Juma edited comment on KAFKA-2419 at 10/15/15 10:27 AM: --- [~aauradkar]

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958698#comment-14958698 ] Ismael Juma commented on KAFKA-2419: [~aauradkar] [~jjkoshy] We are creating one thread per `Metrics`

[jira] [Created] (KAFKA-2658) Implement SASL/PLAIN

2015-10-15 Thread Rajini Sivaram (JIRA)
Rajini Sivaram created KAFKA-2658: - Summary: Implement SASL/PLAIN Key: KAFKA-2658 URL: https://issues.apache.org/jira/browse/KAFKA-2658 Project: Kafka Issue Type: Sub-task

[GitHub] kafka pull request: KAFKA-2295; Support given for dynamically load...

2015-10-15 Thread omkreddy
GitHub user omkreddy opened a pull request: https://github.com/apache/kafka/pull/314 KAFKA-2295; Support given for dynamically loaded classes (encoders, e… Rebased code.. You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (KAFKA-2295) Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958614#comment-14958614 ] ASF GitHub Bot commented on KAFKA-2295: --- GitHub user omkreddy opened a pull request:

[jira] [Commented] (KAFKA-2641) Upgrade path for ZK authentication

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958542#comment-14958542 ] ASF GitHub Bot commented on KAFKA-2641: --- GitHub user fpj opened a pull request:

[GitHub] kafka pull request: KAFKA-2641: Upgrade path for ZK authentication

2015-10-15 Thread fpj
GitHub user fpj opened a pull request: https://github.com/apache/kafka/pull/313 KAFKA-2641: Upgrade path for ZK authentication This pull request adds a configuration parameter and a migration tool. It is also based on pull request #303, which should go in first. You can merge this

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

2015-10-15 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958442#comment-14958442 ] Joel Koshy commented on KAFKA-2017: --- Ashish, that is correct - consumers can still remain detached from

Re: Slow request log in Kafka

2015-10-15 Thread Ewen Cheslack-Postava
Kafka doesn't have the same type of queries that RDBMS systems have. What "slow queries" would we be trying to capture info about? -Ewen On Wed, Oct 14, 2015 at 4:27 PM, Gwen Shapira wrote: > I had some experience with the feature in MySQL. > > Its main good use is to

[jira] [Commented] (KAFKA-2500) Make logEndOffset available in the 0.8.3 Consumer

2015-10-15 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958471#comment-14958471 ] Ewen Cheslack-Postava commented on KAFKA-2500: -- We could simplify some code in Copycat if we

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

2015-10-15 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959038#comment-14959038 ] Jiangjie Qin commented on KAFKA-2017: - My two cents. 1. About the assumption of consumer group

[jira] [Resolved] (KAFKA-2647) Migrate System Tools to work with SSL

2015-10-15 Thread Ben Stopford (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Stopford resolved KAFKA-2647. - Resolution: Won't Fix These tools are all being deprecated > Migrate System Tools to work with

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

2015-10-15 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959152#comment-14959152 ] Mayuresh Gharat commented on KAFKA-2120: [~guozhang] The check is required to be done at each

[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer

2015-10-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959035#comment-14959035 ] Jun Rao commented on KAFKA-2515: [~guozhang], right. > handle oversized messages properly in new consumer

[jira] [Updated] (KAFKA-2397) leave group request

2015-10-15 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Onur Karaman updated KAFKA-2397: Reviewer: Guozhang Wang > leave group request > --- > > Key:

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958978#comment-14958978 ] Ismael Juma commented on KAFKA-2419: [~aauradkar] Thanks for submitting a fix for `ExpireSensorTask`

[jira] [Comment Edited] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958978#comment-14958978 ] Ismael Juma edited comment on KAFKA-2419 at 10/15/15 2:26 PM: -- [~aauradkar]

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959017#comment-14959017 ] Jun Rao commented on KAFKA-2419: It's more important to garbage collect old sensors on the broker than on

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958962#comment-14958962 ] Aditya Auradkar commented on KAFKA-2419: [~ijuma] - We create only a single metrics instance for

[jira] [Comment Edited] (KAFKA-2560) Fatal error during KafkaServer startup because of Map failed error.

2015-10-15 Thread A.J. (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959521#comment-14959521 ] A.J. edited comment on KAFKA-2560 at 10/15/15 8:06 PM: --- What "bitness" of JVM are

[GitHub] kafka pull request: KAFKA-2487: change kafka.examples.Consumer to ...

2015-10-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/297 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-2487) change kafka.examples.Consumer to use the new java consumer

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959919#comment-14959919 ] ASF GitHub Bot commented on KAFKA-2487: --- Github user asfgit closed the pull request at:

[jira] [Commented] (KAFKA-2338) Warn users if they change max.message.bytes that they also need to update broker and consumer settings

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959887#comment-14959887 ] ASF GitHub Bot commented on KAFKA-2338: --- GitHub user benstopford opened a pull request:

[jira] [Resolved] (KAFKA-1282) Disconnect idle socket connection in Selector

2015-10-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1282. Resolution: Fixed Fix Version/s: 0.9.0.0 [~nmarasoi], yes, the actual problem is now fixed in trunk.

[GitHub] kafka pull request: KAFKA-2419 - Fix to prevent background thread ...

2015-10-15 Thread auradkar
GitHub user auradkar opened a pull request: https://github.com/apache/kafka/pull/323 KAFKA-2419 - Fix to prevent background thread from getting created when not required See here for more discussion: https://issues.apache.org/jira/browse/KAFKA-2419 Basically, the fix involves

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960131#comment-14960131 ] ASF GitHub Bot commented on KAFKA-2419: --- GitHub user auradkar opened a pull request:

[GitHub] kafka pull request: KAFKA-2338: Warn on max.message.bytes change

2015-10-15 Thread benstopford
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/322 KAFKA-2338: Warn on max.message.bytes change - Both TopicCommand and ConfigCommand warn if message.max.bytes increases - Log failures on the broker if replication gets stuck due to an

[jira] [Commented] (KAFKA-2659) Warn user, if increasing max message bytes, to also increase on consumer

2015-10-15 Thread Ben Stopford (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959896#comment-14959896 ] Ben Stopford commented on KAFKA-2659: - Thanks [~eribeiro]. No problem. I've pushed a PR out against

[jira] [Resolved] (KAFKA-2487) change kafka.examples.Consumer to use the new java consumer

2015-10-15 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-2487. -- Resolution: Fixed Issue resolved by pull request 297 [https://github.com/apache/kafka/pull/297]

[jira] [Commented] (KAFKA-1397) delete topic is not working

2015-10-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959968#comment-14959968 ] Jun Rao commented on KAFKA-1397: One thing that you want to make sure is that the delete.topic.enable=true

Build failed in Jenkins: kafka-trunk-jdk8 #32

2015-10-15 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-2487: change kafka.examples.Consumer to use the new java consumer -- [...truncated 4202 lines...] org.apache.kafka.copycat.file.FileStreamSourceConnectorTest >

[jira] [Created] (KAFKA-2661) Add a unit test for disconnecting idle socket connections

2015-10-15 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-2661: -- Summary: Add a unit test for disconnecting idle socket connections Key: KAFKA-2661 URL: https://issues.apache.org/jira/browse/KAFKA-2661 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

2015-10-15 Thread Jiangjie Qin
Hey Jay, If we allow consumer to subscribe to /*/my-event, does that mean we allow consumer to consume cross namespaces? In that case it seems not "hierarchical" but more like a name field filtering. i.e. user can choose to consume from topic where datacenter={x,y},

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959685#comment-14959685 ] ASF GitHub Bot commented on KAFKA-2120: --- GitHub user MayureshGharat opened a pull request:

[GitHub] kafka pull request: KAFKA-2120

2015-10-15 Thread MayureshGharat
GitHub user MayureshGharat opened a pull request: https://github.com/apache/kafka/pull/320 KAFKA-2120 Trivial fix to get rid of unused statements in kafkaProducer. You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (KAFKA-2212) KafkaAuthorizer: Add CLI for Acl management.

2015-10-15 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959503#comment-14959503 ] Parth Brahmbhatt commented on KAFKA-2212: - [~junrao] [~ijuma] # we already have a PR for

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959543#comment-14959543 ] Jay Kreps commented on KAFKA-2419: -- It would be preferable to not need additional threads--we put

[jira] [Commented] (KAFKA-2447) Add capability to KafkaLog4jAppender to be able to use SSL

2015-10-15 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959684#comment-14959684 ] Ashish K Singh commented on KAFKA-2447: --- [~gwenshap], I just updated KAFKA-2574 with ducktape based

[jira] [Comment Edited] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959543#comment-14959543 ] Jay Kreps edited comment on KAFKA-2419 at 10/15/15 8:19 PM: It would be

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959588#comment-14959588 ] Aditya Auradkar commented on KAFKA-2419: I agree it would be nice to not need a tick thread.. but

[jira] [Commented] (KAFKA-1397) delete topic is not working

2015-10-15 Thread Ian Kallen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959722#comment-14959722 ] Ian Kallen commented on KAFKA-1397: --- I'm using the ASF release of 0.8.2.2 and I can confirm, topic

[jira] [Comment Edited] (KAFKA-2560) Fatal error during KafkaServer startup because of Map failed error.

2015-10-15 Thread A.J. (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959521#comment-14959521 ] A.J. edited comment on KAFKA-2560 at 10/15/15 8:03 PM: --- What "bitness" of JVM are

[jira] [Commented] (KAFKA-2560) Fatal error during KafkaServer startup because of Map failed error.

2015-10-15 Thread A.J. Samz (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959521#comment-14959521 ] A.J. Samz commented on KAFKA-2560: -- What "bitness" of JVM are you using? I ran into this with using a

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959522#comment-14959522 ] Jun Rao commented on KAFKA-2419: Yes, I was suggesting using a single expiration value at the metrics

[jira] [Commented] (KAFKA-2574) Add ducktape based ssl test for KafkaLog4jAppender

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959663#comment-14959663 ] ASF GitHub Bot commented on KAFKA-2574: --- GitHub user SinghAsDev opened a pull request:

[GitHub] kafka pull request: KAFKA-2574: Add ducktape based ssl test for Ka...

2015-10-15 Thread SinghAsDev
GitHub user SinghAsDev opened a pull request: https://github.com/apache/kafka/pull/319 KAFKA-2574: Add ducktape based ssl test for KafkaLog4jAppender You can merge this pull request into a Git repository by running: $ git pull https://github.com/SinghAsDev/kafka KAFKA-2574

[GitHub] kafka pull request: KAFKA-2371: Add distributed support for Copyca...

2015-10-15 Thread ewencp
GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/321 KAFKA-2371: Add distributed support for Copycat. This adds coordination between DistributedHerders using the generalized consumer support, allowing automatic balancing of connectors and tasks

[jira] [Commented] (KAFKA-2371) Add distributed coordinator implementation for Copycat

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959741#comment-14959741 ] ASF GitHub Bot commented on KAFKA-2371: --- GitHub user ewencp opened a pull request:

Build failed in Jenkins: kafka-trunk-jdk8 #31

2015-10-15 Thread Apache Jenkins Server
See Changes: [wangguoz] MINOR: set up temp directories properly in StreamTaskTest -- [...truncated 6319 lines...] org.apache.kafka.copycat.file.FileStreamSourceConnectorTest >

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

2015-10-15 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959196#comment-14959196 ] Mayuresh Gharat commented on KAFKA-2120: [~guozhang] Yes sure. Will submit a PR for it. > Add a

KAFKA-658 + KStreams = exactly-once stream processing?

2015-10-15 Thread James Cheng
I just recently noticed KAFKA-658 - Implement "Exact Mirroring" functionality in mirror makerhttps://issues.apache.org/jira/browse/KAFKA-658 If you take that, in combination with the custom message handling in mirrormaker, in combination with KStreams... I think you get an exactly-once stream

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

2015-10-15 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959179#comment-14959179 ] Guozhang Wang commented on KAFKA-2120: -- I see, that makes sense. Could we only define remainingTime

[jira] [Updated] (KAFKA-2659) Warn user, if increasing max message bytes, to also increase on consumer

2015-10-15 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2659: Reviewer: Gwen Shapira > Warn user, if increasing max message bytes, to also increase on consumer >

[jira] [Commented] (KAFKA-2659) Warn user, if increasing max message bytes, to also increase on consumer

2015-10-15 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959249#comment-14959249 ] Gwen Shapira commented on KAFKA-2659: - Changing configuration in kafka-topics is getting deprecated,

[jira] [Commented] (KAFKA-2660) Correct cleanableRatio calculation

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959272#comment-14959272 ] ASF GitHub Bot commented on KAFKA-2660: --- GitHub user lindong28 opened a pull request:

[GitHub] kafka pull request: MINOR: optimize unnecessary poll(0) away

2015-10-15 Thread ymatsuda
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/315 MINOR: optimize unnecessary poll(0) away @guozhangwang This change aims to remove unnecessary ```consumer.poll(0)``` calls. * ```once``` after some partition is resumed * whenever the

[jira] [Resolved] (KAFKA-2659) Warn user, if increasing max message bytes, to also increase on consumer

2015-10-15 Thread Ben Stopford (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Stopford resolved KAFKA-2659. - Resolution: Duplicate > Warn user, if increasing max message bytes, to also increase on consumer

[GitHub] kafka pull request: KAFKA-2660; Correct cleanableRatio calculation

2015-10-15 Thread lindong28
GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/316 KAFKA-2660; Correct cleanableRatio calculation @onurkaraman Could you have a look? This is the patch I discussed with you. You can merge this pull request into a Git repository by running: $

[jira] [Created] (KAFKA-2659) Warn user, if increasing max message bytes, to also increase on consumer

2015-10-15 Thread Ben Stopford (JIRA)
Ben Stopford created KAFKA-2659: --- Summary: Warn user, if increasing max message bytes, to also increase on consumer Key: KAFKA-2659 URL: https://issues.apache.org/jira/browse/KAFKA-2659 Project: Kafka

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

2015-10-15 Thread Allen Wang
For scenario 1: - Add the rack information to broker property file or dynamically set it in the wrapper code to bootstrap Kafka server. You would do that for all brokers and restart the brokers one by one. In this scenario, the complete broker to rack mapping may not be available until every

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

2015-10-15 Thread Gwen Shapira
Thanks! Just to clarify, when some brokers have rack assignment and some don't, do we act like none of them have it? or like those without assignment are in their own rack? The first scenario is good when first setting up rack-awareness, but the second makes more sense for on-going maintenance (I

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

2015-10-15 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959266#comment-14959266 ] Joel Koshy commented on KAFKA-2017: --- bq. But if we really want to have group metadata persisted in 0.9

[jira] [Created] (KAFKA-2660) Correct cleanableRatio calculation

2015-10-15 Thread Dong Lin (JIRA)
Dong Lin created KAFKA-2660: --- Summary: Correct cleanableRatio calculation Key: KAFKA-2660 URL: https://issues.apache.org/jira/browse/KAFKA-2660 Project: Kafka Issue Type: Bug Reporter:

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959484#comment-14959484 ] Aditya Auradkar commented on KAFKA-2419: [~junrao] - That is indeed the current implementation

[jira] [Updated] (KAFKA-2660) Correct cleanableRatio calculation

2015-10-15 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2660: -- Reviewer: Joel Koshy > Correct cleanableRatio calculation > -- > >

[jira] [Updated] (KAFKA-2338) Warn users if they change max.message.bytes that they also need to update broker and consumer settings

2015-10-15 Thread Edward Ribeiro (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Ribeiro updated KAFKA-2338: -- Assignee: (was: Edward Ribeiro) > Warn users if they change max.message.bytes that they

[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959379#comment-14959379 ] ASF GitHub Bot commented on KAFKA-2515: --- GitHub user guozhangwang opened a pull request:

[jira] [Commented] (KAFKA-2654) Avoid calling Consumer.poll(0) in each iteration

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959309#comment-14959309 ] ASF GitHub Bot commented on KAFKA-2654: --- Github user asfgit closed the pull request at:

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

2015-10-15 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959300#comment-14959300 ] Guozhang Wang commented on KAFKA-2017: -- rep. [~jjkoshy] and [~becket_qin]: 1) I think with {code}

[jira] [Updated] (KAFKA-2295) Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer

2015-10-15 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2295: - Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request

[jira] [Updated] (KAFKA-2660) Correct cleanableRatio calculation

2015-10-15 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2660: Fix Version/s: 0.9.0.0 > Correct cleanableRatio calculation > -- > >

[jira] [Updated] (KAFKA-2515) handle oversized messages properly in new consumer

2015-10-15 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Onur Karaman updated KAFKA-2515: Assignee: Jason Gustafson (was: Onur Karaman) > handle oversized messages properly in new consumer

[GitHub] kafka pull request: MINOR: set up temp directories properly in Str...

2015-10-15 Thread ymatsuda
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/317 MINOR: set up temp directories properly in StreamTaskTest @guozhangwang StreamTaskTest did not set up a temp directory for each test. This rarely caused interference between tests through

[GitHub] kafka pull request: KAFKA-2515: Handle oversized messages properly...

2015-10-15 Thread guozhangwang
GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/318 KAFKA-2515: Handle oversized messages properly in new consumer You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K2515

[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer

2015-10-15 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959387#comment-14959387 ] Guozhang Wang commented on KAFKA-2515: -- I have verified following the above steps that with this

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

2015-10-15 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959438#comment-14959438 ] Jiangjie Qin commented on KAFKA-2017: - [~guozhang] 1) agreed. Pause and Resume would help much here.

[jira] [Updated] (KAFKA-2338) Warn users if they change max.message.bytes that they also need to update broker and consumer settings

2015-10-15 Thread Edward Ribeiro (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Ribeiro updated KAFKA-2338: -- Assignee: Ben Stopford > Warn users if they change max.message.bytes that they also need to

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959457#comment-14959457 ] Jun Rao commented on KAFKA-2419: [~aauradkar], I thought the implementation will be just to schedule the

[GitHub] kafka pull request: MINOR: set up temp directories properly in Str...

2015-10-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/317 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

2015-10-15 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959392#comment-14959392 ] Joel Koshy commented on KAFKA-2017: --- btw, I just realized a migration as described above probably won't

Build failed in Jenkins: kafka-trunk-jdk8 #30

2015-10-15 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-2654: optimize unnecessary poll(0) away in StreamTask [wangguoz] KAFKA-2295; Support dynamically loaded classes from context class loader -- [...truncated 4202

[jira] [Comment Edited] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-10-15 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959409#comment-14959409 ] Aditya Auradkar edited comment on KAFKA-2419 at 10/15/15 6:52 PM: --

[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer

2015-10-15 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959414#comment-14959414 ] Jason Gustafson commented on KAFKA-2515: I'm a little unclear why we'd only want to log this

Jenkins build is back to normal : kafka-trunk-jdk7 #689

2015-10-15 Thread Apache Jenkins Server
See

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

2015-10-15 Thread Allen Wang
That's a good question. I can think of three actions if the rack information is incomplete: 1. Treat the node without rack as if it is on its unique rack 2. Disregard all rack information and fallback to current algorithm 3. Fail-fast Now I think about it, one and three make more sense. The

[jira] [Resolved] (KAFKA-2654) Avoid calling Consumer.poll(0) in each iteration

2015-10-15 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-2654. -- Resolution: Fixed Fix Version/s: (was: 0.9.1) 0.9.0.0 Issue

[GitHub] kafka pull request: KAFKA-2654: optimize unnecessary poll(0) away

2015-10-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/315 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request: KAFKA-2295; Support given for dynamically load...

2015-10-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/314 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-2295) Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer

2015-10-15 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959318#comment-14959318 ] ASF GitHub Bot commented on KAFKA-2295: --- Github user asfgit closed the pull request at:

[jira] [Commented] (KAFKA-2654) Avoid calling Consumer.poll(0) in each iteration

2015-10-15 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959303#comment-14959303 ] Guozhang Wang commented on KAFKA-2654: -- github PR created: https://github.com/apache/kafka/pull/315