Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Hi Dong, Thanks for your reply. You're right that your DescribeDirsResponse contains appropriate data. The comment about the log_end_offset in the KIP says "Enable user to track movement progress by comparing LEO of the *.log and *.move". That makes me wonder whether this would only work for

[GitHub] kafka pull request #3549: KAFKA-5461: Add metric to track global topic count...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3549 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3623: HOTFIX: Ported over changes from trunk

GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/3623 HOTFIX: Ported over changes from trunk - Cherry picked from #3432 - Minor checkstyle fixes to get to build You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3615: MINOR: Avoid duplicate processing of notifications...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3615 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3622: HOTFIX: state transition cherry picking

GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/3622 HOTFIX: state transition cherry picking Cherry picked from https://github.com/apache/kafka/pull/3432 You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3628: KAFKA-5681; jarAll should build all scala versions...

GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/3628 KAFKA-5681; jarAll should build all scala versions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-5681 Alternatively

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Ismael, Thanks for the comments! Here are my answers: 1. Yes it has been considered. Here are the reasons why we don't do it through controller. - There can be use-cases where we only want to rebalance the load of log directories on a given broker. It seems unnecessary to go through

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

If we are going to have a separate configuration for expiry, I prefer my proposal of max.message.delivery.wait.ms and its semantics. However, one thing which has not come out of the JIRA discussion is the actual use cases for batch expiry. Also, the KIP document states the following: *The per

Re: [VOTE] KIP-177: Consumer perf tool should count rebalance time

+1 On Fri, Aug 4, 2017 at 11:02 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding) > > Ismael > > On Fri, Aug 4, 2017 at 3:13 AM, Hu Xi wrote: > > > Hi all, > > > > > > I'd like to kick off the vote for KIP-177: https://cwiki.apache.org/ > >

[GitHub] kafka pull request #3629: First cut at porting State Store docs to AK

GitHub user bbejeck opened a pull request: https://github.com/apache/kafka/pull/3629 First cut at porting State Store docs to AK You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbejeck/kafka docs-updates-for-kip-167

[GitHub] kafka pull request #3559: KAFKA-5535: Handle null values in ExtractField

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3559 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [VOTE] KIP-177: Consumer perf tool should count rebalance time

+1 On Fri, Aug 4, 2017 at 2:55 PM, Guozhang Wang wrote: > +1 > > On Fri, Aug 4, 2017 at 11:02 AM, Ismael Juma wrote: > > > Thanks for the KIP, +1 (binding) > > > > Ismael > > > > On Fri, Aug 4, 2017 at 3:13 AM, Hu Xi wrote: > > > > >

Re: [VOTE] KIP-177: Consumer perf tool should count rebalance time

Thanks. +1 On Thu, Aug 3, 2017 at 7:13 PM, Hu Xi wrote: > Hi all, > > > I'd like to kick off the vote for KIP-177: https://cwiki.apache.org/ > confluence/display/ARIES/KIP-177%3A+Consumer+perf+tool+ > should+count+rebalance+time. > > > A PR for this KIP can be found at

Re: [DISCUSS] KIP-178: Size-based log directory selection strategy

Hey Hu, I am not sure it is OK. Say kafka-reassign-partitions.sh is used to move 100 replicas to a broker. The the controller will send LeaderAndIsrRequest asking this broker to be the follower of these 100 partitions. While it is true that the broker will create replicas sequentially, but they

Re: [VOTE] KIP-177: Consumer perf tool should count rebalance time

Thanks for the KIP, +1 (binding) Ismael On Fri, Aug 4, 2017 at 3:13 AM, Hu Xi wrote: > Hi all, > > > I'd like to kick off the vote for KIP-177: https://cwiki.apache.org/ > confluence/display/ARIES/KIP-177%3A+Consumer+perf+tool+ > should+count+rebalance+time. > > > A PR for

[jira] [Created] (KAFKA-5703) allow debug-level logging for RequestChannel's request logger

Onur Karaman created KAFKA-5703: --- Summary: allow debug-level logging for RequestChannel's request logger Key: KAFKA-5703 URL: https://issues.apache.org/jira/browse/KAFKA-5703 Project: Kafka

[jira] [Resolved] (KAFKA-5703) allow debug-level logging for RequestChannel's request logger

[ https://issues.apache.org/jira/browse/KAFKA-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Onur Karaman resolved KAFKA-5703. - Resolution: Fixed Woops. Looks like [~ijuma] already fixed this 3 days ago:

[GitHub] kafka pull request #3627: MINOR: Fix error response handler for controlled s...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/3627 MINOR: Fix error response handler for controlled shutdown v0 You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka

Jenkins build is back to normal : kafka-0.11.0-jdk7 #259

See

[GitHub] kafka pull request #3626: KAFKA-5644: Fix Reset Consumer Group Offset tool f...

GitHub user omkreddy opened a pull request: https://github.com/apache/kafka/pull/3626 KAFKA-5644: Fix Reset Consumer Group Offset tool for ThreeLetterISO8601TimeZone You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3627: MINOR: Fix error response handler for controlled s...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3627 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (KAFKA-5704) Auto topic creation causes failure with older clusters

Ewen Cheslack-Postava created KAFKA-5704: Summary: Auto topic creation causes failure with older clusters Key: KAFKA-5704 URL: https://issues.apache.org/jira/browse/KAFKA-5704 Project: Kafka

[jira] [Created] (KAFKA-5706) log the name of the error instead of the error code in response objects

Jun Rao created KAFKA-5706: -- Summary: log the name of the error instead of the error code in response objects Key: KAFKA-5706 URL: https://issues.apache.org/jira/browse/KAFKA-5706 Project: Kafka

[jira] [Created] (KAFKA-5705) Kafka Server start failed and reports "unsafe memory access operation"

Chen He created KAFKA-5705: -- Summary: Kafka Server start failed and reports "unsafe memory access operation" Key: KAFKA-5705 URL: https://issues.apache.org/jira/browse/KAFKA-5705 Project: Kafka

[GitHub] kafka pull request #3630: MINOR: Add Streams quickstart tutorial as an arche...

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/3630 MINOR: Add Streams quickstart tutorial as an archetype project 0. Minor fixes on the existing examples to merge all on a single input topic; also do not use `common.utils.Exit` as it is for

[GitHub] kafka pull request #3537: KAFKA-5599: ConsoleConsumer : --new-consumer optio...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3537 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3555: KAFKA-5619: Make --new-consumer option as deprecat...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3555 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3611: MINOR: Added a couple of unit tests for KStreamPri...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3611 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3524: KAFKA-5588: uselss --new-consumer option

Github user ppatierno closed the pull request at: https://github.com/apache/kafka/pull/3524 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3281: KAFKA-5388 Replace zkClient.subscribe*Changes meth...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3281 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5102) Refactor core components to use zkUtils methods instead of zkUtils.zkClient

[ https://issues.apache.org/jira/browse/KAFKA-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balint Molnar resolved KAFKA-5102. -- Resolution: Fixed > Refactor core components to use zkUtils methods instead of zkUtils.zkClient

[GitHub] kafka pull request #3624: KAFKA-5702: extract refactor StreamThread

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3624 KAFKA-5702: extract refactor StreamThread Extracted `TaskManager` to handle all task related activities. Make `StandbyTaskCreator`, `TaskCreator`, and `RebalanceListener` static classes so they

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Thanks Dong. I have a few initial questions, sorry if I it has been discussed and I missed it. 1. The KIP suggests that the reassignment tool is responsible for sending the ChangeReplicaDirRequests to the relevant brokers. I had imagined that this would be done by the Controller, like the rest of

Jenkins build is back to normal : kafka-trunk-jdk7 #2604

See

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

Yeah I think consistency in present continuous tense sounds better: CompletingRebalance. Guozhang On Fri, Aug 4, 2017 at 8:52 AM, Jason Gustafson wrote: > Hey Guozhang, > > Usually I think such naming inconsistencies are best avoided. It adds > another level of confusion

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

I agree that we should make them consistent. I think RebalanceJoin and RebalanceAssignment are reasonable names. I think they are a bit more descriptive than `PreparingRebalance` and `CompletingRebalance`. If we need to add more states, it seems a little easier to do if the states are a bit more

Build failed in Jenkins: kafka-0.11.0-jdk7 #258

See Changes: [ismael] MINOR: Avoid duplicate processing of notifications in -- [...truncated 861.07 KB...] kafka.api.AdminClientWithPoliciesIntegrationTest >

Jenkins build is back to normal : kafka-trunk-jdk8 #1877

See

[GitHub] kafka pull request #3617: HOTFIX add the hyper link for storage

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3617 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Hey Tom, Thanks for your reply. Here are my thoughts: 1) I think the DescribeDirsResponse can be used by AdminClient to query the lag of follower replica as well. Here is how it works: - AdminClient sends DescribeDirsRequest to both the leader and the follower of the partition. -

[jira] [Created] (KAFKA-5701) Unit test shouldTogglePrepareForBulkLoadDuringRestoreCalls fails

Eno Thereska created KAFKA-5701: --- Summary: Unit test shouldTogglePrepareForBulkLoadDuringRestoreCalls fails Key: KAFKA-5701 URL: https://issues.apache.org/jira/browse/KAFKA-5701 Project: Kafka

[GitHub] kafka pull request #1755: KAFKA-4039: delay invocation of System.exit via Fa...

Github user maysamyabandeh closed the pull request at: https://github.com/apache/kafka/pull/1755 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

Hey Guozhang, Usually I think such naming inconsistencies are best avoided. It adds another level of confusion for people who have to dip into the code, figure out a problem, and ultimately explain it. Since we already have the PreparingRebalance state, maybe we could just rename the AwaitingSync

[jira] [Created] (KAFKA-5702) Refactor StreamThread to separate concerns and enable better testability

Damian Guy created KAFKA-5702: - Summary: Refactor StreamThread to separate concerns and enable better testability Key: KAFKA-5702 URL: https://issues.apache.org/jira/browse/KAFKA-5702 Project: Kafka

Re: [DISCUSS] KIP-178: Size-based log directory selection strategy

Hi, Thanks for the KIP. As Dong said, this is not straightforward. One option would be to keep the existing system, which is very simple and rely on KIP-113 to keep the data balanced across disks. If we don't think that's good enough, we should explain why it's not in the KIP. Thanks, Ismael On

Build failed in Jenkins: kafka-trunk-jdk7 #2603

See Changes: [ismael] KAFKA-5461; Add metric to track global topic count and global parition -- [...truncated 915.04 KB...] kafka.api.AuthorizerIntegrationTest >

[GitHub] kafka pull request #3625: fix for standby tasks using batching restore

GitHub user bbejeck opened a pull request: https://github.com/apache/kafka/pull/3625 fix for standby tasks using batching restore You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbejeck/kafka