[jira] [Created] (KAFKA-5614) pep8/flake8 checks for system tests

Ewen Cheslack-Postava created KAFKA-5614: Summary: pep8/flake8 checks for system tests Key: KAFKA-5614 URL: https://issues.apache.org/jira/browse/KAFKA-5614 Project: Kafka Issue

[jira] [Created] (KAFKA-5613) Deprecate JmxTool?

Ewen Cheslack-Postava created KAFKA-5613: Summary: Deprecate JmxTool? Key: KAFKA-5613 URL: https://issues.apache.org/jira/browse/KAFKA-5613 Project: Kafka Issue Type: Bug Affects

[jira] [Created] (KAFKA-5612) Replace JmxTool with a MetricsReporter in system tests

Ewen Cheslack-Postava created KAFKA-5612: Summary: Replace JmxTool with a MetricsReporter in system tests Key: KAFKA-5612 URL: https://issues.apache.org/jira/browse/KAFKA-5612 Project: Kafka

[GitHub] kafka pull request #3547: KAFKA-5608: Add --wait option for JmxTool and use ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3547 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5608) System test failure due to timeout starting Jmx tool

[ https://issues.apache.org/jira/browse/KAFKA-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-5608. -- Resolution: Fixed Fix Version/s: 0.11.0.1 0.11.1.0

[GitHub] kafka pull request #2851: MINOR: Clarify wording

Github user jeffwidman closed the pull request at: https://github.com/apache/kafka/pull/2851 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[VOTE] KIP-167 (Addendum): Add interface for the state store restoration process

Hi, After working on the PR for this KIP I discovered that we need to add and additional parameter (TopicPartition) to the StateRestoreListener interface methods. The addition of the TopicPartition is required as the StateRestoreListener is for the entire application, thus all tasks with

[jira] [Created] (KAFKA-5611) One or more consumers in a consumer-group stop consuming after rebalancing

Panos Skianis created KAFKA-5611: Summary: One or more consumers in a consumer-group stop consuming after rebalancing Key: KAFKA-5611 URL: https://issues.apache.org/jira/browse/KAFKA-5611 Project:

Re: [VOTE] KIP-162: Enable topic deletion by default

ah, thanks guys! (and thanks Ismael for noticing that a bunch of people voted on the discuss thread by mistake). We have 6 binding votes and 5 non-binding. So this update is in :) I'll update the wiki accordingly. On Wed, Jul 19, 2017 at 5:11 PM Becket Qin wrote: > +1 >

Re: [VOTE] KIP-162: Enable topic deletion by default

+1 On Wed, Jul 19, 2017 at 11:40 AM, Jason Gustafson wrote: > +1 > > On Wed, Jul 19, 2017 at 7:51 AM, Damian Guy wrote: > > > +1 > > > > On Tue, 6 Jun 2017 at 18:19 BigData dev wrote: > > > > > +1 (non-binding) > > > > > >

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

It makes sense. Thanks for clarifying. The KIP is updated based on your feedback. Thanks again. --Vahid From: Jason Gustafson To: dev@kafka.apache.org Cc: Kafka Users Date: 07/19/2017 05:06 PM Subject:Re: [DISCUSS] KIP-175:

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

I was just thinking that the subscription/assignment information can be quite large (especially in MM use cases), so it would be nice to keep the default output concise. I'm also not thrilled about adding more options, but --verbose is sort of a standard one. What do you think? -Jason On Wed,

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hi Jason, Thanks for sharing your feedback on the updated KIP. Your suggestions look good to me. Do you see a problem with having the `--members` provide member subscription and assignment too, so we can avoid an additional `--verbose` option? Thanks. --Vahid From: Jason Gustafson

RE: Using JMXMP to access Kafka metrics

Jolokia is not really an option for me although I've looked at it. I just want to know if there's some way to enable using JMXMP to access the metrics pushed to JMX. -Original Message- From: Svante Karlsson [mailto:svante.karls...@csi.se] Sent: Wednesday, July 19, 2017 12:24 PM To:

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

Per 1. I suggested exposing the constant since we are doing so for consumer and producer configs prefix as well (CONSUMER_PREFIX, etc). Guozhang On Wed, Jul 19, 2017 at 6:01 AM, Ismael Juma wrote: > Thanks for the KIP, Damian. +1 (binding). A couple of minor comments: > > 1.

[GitHub] kafka pull request #3550: KAFKA-5610: KafkaApis.HandleWriteTxnMarkerRequest ...

GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3550 KAFKA-5610: KafkaApis.HandleWriteTxnMarkerRequest should return UNKNOWN_TOPIC_OR_PARTITION on partition emigration. Before this patch, we would return the non-retriable

Brokers restart when ZK address changes

Good day, We are using AWS and it is quite common for us to terminate instances very often, which introduces ip address change and requires to restart brokers one by one. Maybe there is a knowledge on how to avoid such restarts? According to Confluent recommendations(on their web page), one

[jira] [Created] (KAFKA-5610) KafkaApis.handleWriteTxnMarkerRequest can return UNSUPPORTED_FOR_MESSAGE_FORMAT error on partition emigration

Apurva Mehta created KAFKA-5610: --- Summary: KafkaApis.handleWriteTxnMarkerRequest can return UNSUPPORTED_FOR_MESSAGE_FORMAT error on partition emigration Key: KAFKA-5610 URL:

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hey Vahid, Thanks for the updates. Looks pretty good. A couple comments: 1. For the --state option, should we use the same column-oriented format as we use for the other options? I realize there would only be one row, but the inconsistency is a little vexing. Also, since this tool is working

Re: [DISCUSS] 2017 October release planning and release version

Hi Stevo, Just trying to add to what Ismael has already replied you: > Practice/"features" like protocol version being a parameter, and defaulting > to latest so auto updated with dependency update which introduces new > protocol/behavior should not be used in public client APIs. To switch >

Using JMXMP to access Kafka metrics

Hey, Is there a way to use JMXMP instead of RMI to access Kafka metrics through JMX? I tried creating a JMXMP JMXConnector but the connect attempt just hangs forever. Thanks, Vijay

Re: Using JMXMP to access Kafka metrics

I've used jolokia which gets JMX metrics without RMI (actually json over http) https://jolokia.org/ Integrates nicely with telegraf (and influxdb) 2017-07-19 20:47 GMT+02:00 Vijay Prakash < vijay.prak...@microsoft.com.invalid>: > Hey, > > Is there a way to use JMXMP instead of RMI to access

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

Ismael, I debated that also, but the main point was to make users aware of the rebalance latency (with KIP-134 in mind). I'm guessing no one would notice if it required another option. Note that the KIP does preserve the existing fields (and in the same order), so if it is parsed as generic csv

Re: [VOTE] KIP-162: Enable topic deletion by default

+1 On Wed, Jul 19, 2017 at 7:51 AM, Damian Guy wrote: > +1 > > On Tue, 6 Jun 2017 at 18:19 BigData dev wrote: > > > +1 (non-binding) > > > > Thanks, > > Bharat > > > > On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha > >

Re: KIP-162: Enable topic deletion by default

+1 On Wed, Jul 19, 2017 at 1:35 PM, Jay Kreps wrote: > +1 > > On Sat, May 27, 2017 at 11:04 AM, Gwen Shapira wrote: > > > Thanks Vahid, > > > > Do you mind if we leave the command-line out of scope for this? > > > > I can see why adding confirmations,

[GitHub] kafka pull request #3222: MINOR: Code Cleanup

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3222 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Jenkins build is back to normal : kafka-trunk-jdk8 #1822

See

Re: KIP-162: Enable topic deletion by default

+1 On Sat, May 27, 2017 at 11:04 AM, Gwen Shapira wrote: > Thanks Vahid, > > Do you mind if we leave the command-line out of scope for this? > > I can see why adding confirmations, options to bypass confirmations, etc > would be an improvement. However, I've seen no

Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster

Thank you all for voting. The votes are as follows: Bidding +1: Joel, Becket, and Ismael Non-bidding +1: Dong, Onur, Mickael, Vahid, and Bill This vote is passed as per Lazy Majority . Here is the link to the PR -

[GitHub] kafka pull request #3549: KAFKA-5461: Add metric to track global topic count...

GitHub user abhishekmendhekar opened a pull request: https://github.com/apache/kafka/pull/3549 KAFKA-5461: Add metric to track global topic count and global parition count in a cluster You can merge this pull request into a Git repository by running: $ git pull

Build failed in Jenkins: kafka-trunk-jdk8 #1821

See Changes: [ismael] MINOR: Make 'Topic-Level Configs' a doc section for easier access -- [...truncated 4.33 MB...] kafka.integration.PrimitiveApiTest >

Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster

+1 On Wed, Jul 19, 2017 at 7:56 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding) from me. > > Ismael > > On Wed, Jun 21, 2017 at 1:17 PM, Abhishek Mendhekar < > abhishek.mendhe...@gmail.com> wrote: > > > Hi Kafka Dev, > > > > I did like to start the voting on - > >

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Ah, thank you! I took the number from the "Next KIP Number: 178" on the KIP index and didn't check the tables. So this is now KIP-179. The old link will point you to the right place. On 19 July 2017 at 15:55, Ismael Juma wrote: > One more thing, it looks like there is already

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

One more thing, it looks like there is already a KIP-178: KIP-178: Size-based log directory selection strategy Ismael On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley wrote: > OK, I will work on adding support for this to the KIP, with the intention > of a two part

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

I think this is a good chance although it's unfortunate that it's likely to break code that is parsing the output of the performance tool. Would it make sense to only enable this if an option is provided? Ismael On Mon, Jul 17, 2017 at 3:41 PM, Jason Gustafson wrote: >

Re: KIP-162: Enable topic deletion by default

+1 On Wed, 31 May 2017 at 13:36 Jim Jagielski wrote: > +1 > > On May 27, 2017, at 9:27 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > > Sure, that sounds good. > > > > I suggested that to keep command line behavior consistent. > > Plus, removal of ACL access

Re: [VOTE] KIP-162: Enable topic deletion by default

+1 On Tue, 6 Jun 2017 at 18:19 BigData dev wrote: > +1 (non-binding) > > Thanks, > Bharat > > On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha > wrote: > > > +1 > > > > On Tue, Jun 6, 2017 at 11:20 PM, Mickael Maison < > mickael.mai...@gmail.com> >

Re: [VOTE] KIP-162: Enable topic deletion by default

Thanks for the KIP. +1 (binding) On Tue, Jun 6, 2017 at 6:19 PM, BigData dev wrote: > +1 (non-binding) > > Thanks, > Bharat > > On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha > wrote: > > > +1 > > > > On Tue, Jun 6, 2017 at 11:20 PM, Mickael

Jenkins build is back to normal : kafka-0.11.0-jdk7 #230

See

Want some help about admin client

Hi, My requirement is that i want to know which node of cluster is alive, something like health check, and we will create some topic too with the admin client java api . In health check we use adminClient.describeCluster() to list alive node, at the beginning that works, but after sometimes

Jenkins build is back to normal : kafka-trunk-jdk7 #2532

See

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

OK, I will work on adding support for this to the KIP, with the intention of a two part implementation. On 19 July 2017 at 14:59, Ismael Juma wrote: > Hi Tom, > > It's fine for the tool not to have this functionality from the start. > However, since we're adding new Kafka

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Tom, It's fine for the tool not to have this functionality from the start. However, since we're adding new Kafka protocol APIs, we need to consider some of these details to ensure we're building towards the end state, if that makes sense. Protocol APIs are used by multiple clients, so there is

[GitHub] kafka pull request #3415: MINOR: Make 'Topic-Level Configs' a doc section fo...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3415 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Ismael, Answers in-line: 1. Have you considered how progress would be reported? Partition > reassignment can take a long time and it would be good to have a mechanism > for progress reporting. > The ReassignPartitionsCommand doesn't currently have a mechanism to track progress. All you can

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

Thanks for the KIP, Damian. +1 (binding). A couple of minor comments: 1. Do we need to expose the TOPIC_PREFIX constant? 2. The vote thread ended up inside the discuss thread in Gmail. It may be worth sending another email to make it clear that the vote is ongoing. You can link back to this

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Tom, Thanks for the KIP. Two quick comments: 1. Have you considered how progress would be reported? Partition reassignment can take a long time and it would be good to have a mechanism for progress reporting. 2. Removals can only happen in major releases. In your example, the removal could

[DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi, I've written KIP-178 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-178+-+Change+ReassignPartitionsCommand+to+use+AdminClient) for changing the ReassignPartitionsCommand (a.k.a. kafka-reassign-partitions.sh)to remove the dependency on ZooKeeper and instead provide the same

Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster

Thanks for the KIP, +1 (binding) from me. Ismael On Wed, Jun 21, 2017 at 1:17 PM, Abhishek Mendhekar < abhishek.mendhe...@gmail.com> wrote: > Hi Kafka Dev, > > I did like to start the voting on - > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

[GitHub] kafka pull request #3548: KAFKA-5607: Improve error handling in SocketServer...

GitHub user rajinisivaram opened a pull request: https://github.com/apache/kafka/pull/3548 KAFKA-5607: Improve error handling in SocketServer to avoid issues later You can merge this pull request into a Git repository by running: $ git pull

Re: [DISCUSS] 2017 October release planning and release version

Hi Stevo, Thanks for your feedback. We should definitely do a better job of documenting things. We basically follow semantic versioning, but it's currently a bit confusing because: 1. There are 4 segments in the version. The "0." part should be ignored when deciding what is major, minor and

Re: [DISCUSS] 2017 October release planning and release version

With 0.x version of project at least I found lots of unexpected painful things acceptable. Graduating from 0.*, with x.y.z semantic versioning IMO it should be clearly communicated to community is there change in meaning, what's the "SLO", what are the commitments, what change in every segment