[jira] [Created] (KAFKA-5614) pep8/flake8 checks for system tests

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-5614:


 Summary: pep8/flake8 checks for system tests
 Key: KAFKA-5614
 URL: https://issues.apache.org/jira/browse/KAFKA-5614
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Affects Versions: 0.11.0.0
Reporter: Ewen Cheslack-Postava


Currently the python code for system tests doesn't have an checks on style. 
Similar to checkstyle in java/scala, we should be checking the style for Python 
code. These consists of at least 2 parts:

* Add the configs that allow you to run these checks
* Get the Kafka CI systems to run these as a regular part of the build process, 
in addition to building & testing the java/scala code.

The latter is hard to estimate the effort for since it depends on what is 
available on the Apache Jenkins slaves that we build on. Hopefully at least 
python + pip would be there such that augmenting the Jenkins build to also run 
flake8 wouldn't be too tough.

If it makes sense while doing this, we might also want to get Python3 
compatibility and even just move to it being a requirement. Whether this is a 
requirement for this JIRA might depend on the Apache jenkins environments and 
what python versions they have available.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5613) Deprecate JmxTool?

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-5613:


 Summary: Deprecate JmxTool?
 Key: KAFKA-5613
 URL: https://issues.apache.org/jira/browse/KAFKA-5613
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.0
Reporter: Ewen Cheslack-Postava


According to git-blame, JmxTool has been around since October 2011. We use it 
in system tests, but we are thinking it might be best to replace it: 
https://issues.apache.org/jira/browse/KAFKA-5612

When making modifications for system tests, we've had to take into account 
compatibility because this tool is technically included in our distribution 
and, perhaps unintentionally, a public utility.

We know that "real" tools for JMX, like jmxtrans, are more commonly used, but 
we don't know who might be using JmxTool simply because it ships with Kafka. 
That said, it also isn't documented in the Kafka documentation, so you probably 
have to dig around to find it.

Hopefully we can deprecate this and eventually move it either to a jar that is 
only used for system tests, or even better just remove it entirely. To do any 
of this, we'd probably need to do at least a cursory survey of the community to 
get a feel for usage level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5612) Replace JmxTool with a MetricsReporter in system tests

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-5612:


 Summary: Replace JmxTool with a MetricsReporter in system tests
 Key: KAFKA-5612
 URL: https://issues.apache.org/jira/browse/KAFKA-5612
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Affects Versions: 0.11.0.0
Reporter: Ewen Cheslack-Postava


I marked this as affecting 0.11.0.0, but it affects all earlier versions as 
well, at least as far back as 0.10.1.

The discussion in https://github.com/apache/kafka/pull/3547 probably gives the 
clearest explanation, but the basic issue is that ever since JmxMixin was 
introduced to the system tests, we've faced race condition issues because the 
second process that performs the monitoring has various timing issues with the 
process it is monitoring. It can be both too fast and too slow, and the exact 
conditions it needs to wait for may not even be externally visible (e.g. that 
all metrics have been registered).

An alternative solution would be to introduce a MetricsReporter implementation 
that accomplishes the same thing, but just requires overriding some configs for 
the service that is utilizing JmxMixin. In particular, the reporter could 
output data to a simple file, ideally would not require all metrics that are 
reported to be available up front (i.e., no CSV format that requires a fixed 
header that cannot be changed), and wouldn't have any timing constraints (e.g., 
could at least guarantee that metrics are reported once at the beginning and 
end of the program).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3547: KAFKA-5608: Add --wait option for JmxTool and use ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3547


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5608) System test failure due to timeout starting Jmx tool

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-5608.
--
   Resolution: Fixed
Fix Version/s: 0.11.0.1
   0.11.1.0

Issue resolved by pull request 3547
[https://github.com/apache/kafka/pull/3547]

> System test failure due to timeout starting Jmx tool
> 
>
> Key: KAFKA-5608
> URL: https://issues.apache.org/jira/browse/KAFKA-5608
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.11.1.0, 0.11.0.1
>
>
> Began seeing this in some failing system tests:
> {code}
> [INFO  - 2017-07-18 14:25:55,375 - background_thread - _protected_worker - 
> lineno:39]: Traceback (most recent call last):
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/services/background_thread.py",
>  line 35, in _protected_worker
> self._worker(idx, node)
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/services/console_consumer.py",
>  line 261, in _worker
> self.start_jmx_tool(idx, node)
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/services/monitor/jmx.py",
>  line 73, in start_jmx_tool
> wait_until(lambda: self._jmx_has_output(node), timeout_sec=10, 
> backoff_sec=.5, err_msg="%s: Jmx tool took too long to start" % node.account)
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: ubuntu@worker7: Jmx tool took too long to start
> {code}
> This is immediately followed by a consumer timeout in the failing cases:
> {code}
> [INFO  - 2017-07-18 14:26:46,907 - runner_client - log - lineno:221]: 
> RunnerClient: 
> kafkatest.tests.core.security_rolling_upgrade_test.TestSecurityRollingUpgrade.test_rolling_upgrade_phase_two.broker_protocol=SASL_SSL.client_protocol=SASL_SSL:
>  FAIL: Consumer failed to consume messages for 60s.
> Traceback (most recent call last):
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py",
>  line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py",
>  line 148, in test_rolling_upgrade_phase_two
> self.run_produce_consume_validate(self.roll_in_secured_settings, 
> client_protocol, broker_protocol)
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 106, in run_produce_consume_validate
> self.start_producer_and_consumer()
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 79, in start_producer_and_consumer
> self.consumer_start_timeout_sec)
>   File 
> "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Consumer failed to consume messages for 60s.
> {code}
> There does not appear to be anything wrong with the consumer in the logs, so 
> the timeout seems to be caused by the Jmx tool timeout.
> Possibly due to https://github.com/apache/kafka/pull/3447/files?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #2851: MINOR: Clarify wording

[jeffwidman]

Github user jeffwidman closed the pull request at:

https://github.com/apache/kafka/pull/2851


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[VOTE] KIP-167 (Addendum): Add interface for the state store restoration process

[Bill Bejeck]

Hi,

After working on the PR for this KIP I discovered that we need to add and
additional parameter (TopicPartition) to the StateRestoreListener interface
methods.

The addition of the TopicPartition is required as the StateRestoreListener
is for the entire application, thus all tasks with recovering state stores
call the same listener instance.  The TopicPartition is needed to
disambiguate the progress of the state store recovery.

For those that have voted before, please review the updated KIP

and
re-vote.

Thanks,
Bill

[jira] [Created] (KAFKA-5611) One or more consumers in a consumer-group stop consuming after rebalancing

[Panos Skianis (JIRA)]

Panos Skianis created KAFKA-5611:


 Summary: One or more consumers in a consumer-group stop consuming 
after rebalancing
 Key: KAFKA-5611
 URL: https://issues.apache.org/jira/browse/KAFKA-5611
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.2.0
Reporter: Panos Skianis
 Attachments: kka02, Server 1, Server 2, Server 3

Scenario: 
  - 3 servers running 1 consumer each under the same consumer groupId. 
  - Servers seem to be consuming messages happily but then there is a timeout 
to an external service that causes our app to restart the Kafka Consumer on one 
of the servers (this is by design). That causes rebalancing of the group and 
upon restart of one of the Consumers seem to "block".
  - Server 3 is where the problems occur.
  - Problem fixes itself either by restarting one of the 3 servers or cause the 
group to rebalance again by using the console consumer with the autocommit set 
to false and using the same group.
 
Note: 
 - Haven't managed to recreate it at will yet.
 - Mainly happens in production environment, often enough. Hence I do not have 
any logs with DEBUG/TRACE statements yet.
 - Extracts from log of each app server are attached. Also the log of the kafka 
that seems to be dealing with the related group and generations.
 - See COMMENT lines in the files for further info.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-162: Enable topic deletion by default

[Gwen Shapira]

ah, thanks guys! (and thanks Ismael for noticing that a bunch of people
voted on the discuss thread by mistake).

We have 6 binding votes and 5 non-binding. So this update is in :)

I'll update the wiki accordingly.

On Wed, Jul 19, 2017 at 5:11 PM Becket Qin  wrote:

> +1
>
> On Wed, Jul 19, 2017 at 11:40 AM, Jason Gustafson 
> wrote:
>
> > +1
> >
> > On Wed, Jul 19, 2017 at 7:51 AM, Damian Guy 
> wrote:
> >
> > > +1
> > >
> > > On Tue, 6 Jun 2017 at 18:19 BigData dev 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > > Bharat
> > > >
> > > > On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha <
> ashwin.si...@go-jek.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Tue, Jun 6, 2017 at 11:20 PM, Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1 (non binding), thanks
> > > > > >
> > > > > > On Tue, Jun 6, 2017 at 2:16 PM, Bill Bejeck 
> > > wrote:
> > > > > > > +1
> > > > > > >
> > > > > > > -Bill
> > > > > > >
> > > > > > > On Tue, Jun 6, 2017 at 9:08 AM, Ismael Juma  >
> > > > wrote:
> > > > > > >
> > > > > > >> Thanks for the KIP, Gwen. +1 (binding).
> > > > > > >>
> > > > > > >> Ismael
> > > > > > >>
> > > > > > >> On Tue, Jun 6, 2017 at 5:37 AM, Gwen Shapira <
> g...@confluent.io
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi,
> > > > > > >> >
> > > > > > >> > The discussion has been quite positive, so I posted a JIRA,
> a
> > PR
> > > > and
> > > > > > >> > updated the KIP with the latest decisions.
> > > > > > >> >
> > > > > > >> > Lets officially vote on the KIP:
> > > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > >> > 162+-+Enable+topic+deletion+by+default
> > > > > > >> >
> > > > > > >> > JIRA is here: https://issues.apache.org/
> > jira/browse/KAFKA-5384
> > > > > > >> >
> > > > > > >> > Gwen
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks and Regards,
> > > > > Ashwin
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-162: Enable topic deletion by default

[Becket Qin]

+1

On Wed, Jul 19, 2017 at 11:40 AM, Jason Gustafson 
wrote:

> +1
>
> On Wed, Jul 19, 2017 at 7:51 AM, Damian Guy  wrote:
>
> > +1
> >
> > On Tue, 6 Jun 2017 at 18:19 BigData dev  wrote:
> >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > > Bharat
> > >
> > > On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha 
> > > wrote:
> > >
> > > > +1
> > > >
> > > > On Tue, Jun 6, 2017 at 11:20 PM, Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 (non binding), thanks
> > > > >
> > > > > On Tue, Jun 6, 2017 at 2:16 PM, Bill Bejeck 
> > wrote:
> > > > > > +1
> > > > > >
> > > > > > -Bill
> > > > > >
> > > > > > On Tue, Jun 6, 2017 at 9:08 AM, Ismael Juma 
> > > wrote:
> > > > > >
> > > > > >> Thanks for the KIP, Gwen. +1 (binding).
> > > > > >>
> > > > > >> Ismael
> > > > > >>
> > > > > >> On Tue, Jun 6, 2017 at 5:37 AM, Gwen Shapira  >
> > > > wrote:
> > > > > >>
> > > > > >> > Hi,
> > > > > >> >
> > > > > >> > The discussion has been quite positive, so I posted a JIRA, a
> PR
> > > and
> > > > > >> > updated the KIP with the latest decisions.
> > > > > >> >
> > > > > >> > Lets officially vote on the KIP:
> > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> > 162+-+Enable+topic+deletion+by+default
> > > > > >> >
> > > > > >> > JIRA is here: https://issues.apache.org/
> jira/browse/KAFKA-5384
> > > > > >> >
> > > > > >> > Gwen
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks and Regards,
> > > > Ashwin
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

[Vahid S Hashemian]

It makes sense. Thanks for clarifying.
The KIP is updated based on your feedback.

Thanks again.
--Vahid




From:   Jason Gustafson 
To: dev@kafka.apache.org
Cc: Kafka Users 
Date:   07/19/2017 05:06 PM
Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for 
ConsumerGroupCommand



I was just thinking that the subscription/assignment information can be
quite large (especially in MM use cases), so it would be nice to keep the
default output concise. I'm also not thrilled about adding more options,
but --verbose is sort of a standard one. What do you think?

-Jason

On Wed, Jul 19, 2017 at 4:39 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason,
>
> Thanks for sharing your feedback on the updated KIP.
> Your suggestions look good to me.
> Do you see a problem with having the `--members` provide member
> subscription and assignment too, so we can avoid an additional 
`--verbose`
> option?
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Jason Gustafson 
> To: Kafka Users 
> Cc: dev@kafka.apache.org
> Date:   07/19/2017 01:46 PM
> Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> ConsumerGroupCommand
>
>
>
> Hey Vahid,
>
> Thanks for the updates. Looks pretty good. A couple comments:
>
> 1. For the --state option, should we use the same column-oriented format
> as
> we use for the other options? I realize there would only be one row, but
> the inconsistency is a little vexing. Also, since this tool is working
> only
> with consumer groups, perhaps we can leave out "protocol type" and use
> "assignment strategy" in place of "protocol"? It would be nice to also
> include the group generation, but it seems we didn't add that to the
> DescribeGroup response. Perhaps we could also include a count of the
> number
> of members?
> 2. It's a little annoying that --subscription and --members share so 
much
> in common. Maybe we could drop --subscription and use a --verbose flag 
to
> control whether or not to include the subscription and perhaps the
> assignment as well? Not sure if that's more annoying or less, but maybe 
a
> generic --verbose will be useful in other contexts.
>
> As for your question on whether we need the --offsets option at all, I
> don't have a strong opinion, but it seems to make the command semantics 
a
> little more consistent.
>
> -Jason
>
> On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Jason,
> >
> > I updated the KIP based on your earlier suggestions:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand
> > The only thing I am wondering at this point is whether it's worth to
> have
> > a `--describe --offsets` option that behaves exactly like 
`--describe`.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From:   "Vahid S Hashemian" 
> > To: dev@kafka.apache.org
> > Cc: Kafka Users 
> > Date:   07/17/2017 03:24 PM
> > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views 
for
> > ConsumerGroupCommand
> >
> >
> >
> > Hi Jason,
> >
> > Thanks for your quick feedback. Your suggestions seem reasonable.
> > I'll start updating the KIP accordingly and will send out another note
> > when it's ready.
> >
> > Regards.
> > --Vahid
> >
> >
> >
> >
> > From:   Jason Gustafson 
> > To: dev@kafka.apache.org
> > Cc: Kafka Users 
> > Date:   07/17/2017 02:11 PM
> > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views 
for
> > ConsumerGroupCommand
> >
> >
> >
> > Hey Vahid,
> >
> > Hmm... If possible, it would be nice to avoid cluttering the default
> > option
> > too much, especially if it is information which is going to be the 
same
> > for
> > all members (such as the generation). My preference would be to use 
the
> > --state option that you've suggested for that info so that we can
> > represent
> > it more concisely.
> >
> > The reason I prefer the current output is that it is clear every entry
> > corresponds to a partition for which we have committed offset. Entries
> > like
> > this look strange:
> >
> > TOPIC  PARTITION  CURRENT-OFFSET 
LOG-END-OFFSET
> > LAGCONSUMER-ID
> > HOST   CLIENT-ID
> > -  -  -   -
> > -  consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff
> > /127.0.0.1
> > consumer4
> > -  -  -   -
> > -  consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea
> > /127.0.0.1
> > consumer5
> >
> > It makes me think that the consumers have committed offsets for an
> unknown
> > partition. The --members option seems like a clearer way to 
communicate
> > the
> > fact that there are some members with no assigned partitions.

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

[Jason Gustafson]

I was just thinking that the subscription/assignment information can be
quite large (especially in MM use cases), so it would be nice to keep the
default output concise. I'm also not thrilled about adding more options,
but --verbose is sort of a standard one. What do you think?

-Jason

On Wed, Jul 19, 2017 at 4:39 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason,
>
> Thanks for sharing your feedback on the updated KIP.
> Your suggestions look good to me.
> Do you see a problem with having the `--members` provide member
> subscription and assignment too, so we can avoid an additional `--verbose`
> option?
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Jason Gustafson 
> To: Kafka Users 
> Cc: dev@kafka.apache.org
> Date:   07/19/2017 01:46 PM
> Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> ConsumerGroupCommand
>
>
>
> Hey Vahid,
>
> Thanks for the updates. Looks pretty good. A couple comments:
>
> 1. For the --state option, should we use the same column-oriented format
> as
> we use for the other options? I realize there would only be one row, but
> the inconsistency is a little vexing. Also, since this tool is working
> only
> with consumer groups, perhaps we can leave out "protocol type" and use
> "assignment strategy" in place of "protocol"? It would be nice to also
> include the group generation, but it seems we didn't add that to the
> DescribeGroup response. Perhaps we could also include a count of the
> number
> of members?
> 2. It's a little annoying that --subscription and --members share so much
> in common. Maybe we could drop --subscription and use a --verbose flag to
> control whether or not to include the subscription and perhaps the
> assignment as well? Not sure if that's more annoying or less, but maybe a
> generic --verbose will be useful in other contexts.
>
> As for your question on whether we need the --offsets option at all, I
> don't have a strong opinion, but it seems to make the command semantics a
> little more consistent.
>
> -Jason
>
> On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Jason,
> >
> > I updated the KIP based on your earlier suggestions:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand
> > The only thing I am wondering at this point is whether it's worth to
> have
> > a `--describe --offsets` option that behaves exactly like `--describe`.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From:   "Vahid S Hashemian" 
> > To: dev@kafka.apache.org
> > Cc: Kafka Users 
> > Date:   07/17/2017 03:24 PM
> > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> > ConsumerGroupCommand
> >
> >
> >
> > Hi Jason,
> >
> > Thanks for your quick feedback. Your suggestions seem reasonable.
> > I'll start updating the KIP accordingly and will send out another note
> > when it's ready.
> >
> > Regards.
> > --Vahid
> >
> >
> >
> >
> > From:   Jason Gustafson 
> > To: dev@kafka.apache.org
> > Cc: Kafka Users 
> > Date:   07/17/2017 02:11 PM
> > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> > ConsumerGroupCommand
> >
> >
> >
> > Hey Vahid,
> >
> > Hmm... If possible, it would be nice to avoid cluttering the default
> > option
> > too much, especially if it is information which is going to be the same
> > for
> > all members (such as the generation). My preference would be to use the
> > --state option that you've suggested for that info so that we can
> > represent
> > it more concisely.
> >
> > The reason I prefer the current output is that it is clear every entry
> > corresponds to a partition for which we have committed offset. Entries
> > like
> > this look strange:
> >
> > TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> > LAGCONSUMER-ID
> > HOST   CLIENT-ID
> > -  -  -   -
> > -  consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff
> > /127.0.0.1
> > consumer4
> > -  -  -   -
> > -  consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea
> > /127.0.0.1
> > consumer5
> >
> > It makes me think that the consumers have committed offsets for an
> unknown
> > partition. The --members option seems like a clearer way to communicate
> > the
> > fact that there are some members with no assigned partitions.
> >
> > A few additional suggestions:
> >
> > 1. Maybe we can rename --partitions to --offsets or --committed-offsets
> > and
> > the output could match the default output (in other words, --offsets is
> > treated as the default switch). Seems no harm including the assignment
> > information if we have it.
> > 2. Along the lines of Onur's comment, it would be 

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

[Vahid S Hashemian]

Hi Jason,

Thanks for sharing your feedback on the updated KIP.
Your suggestions look good to me.
Do you see a problem with having the `--members` provide member 
subscription and assignment too, so we can avoid an additional `--verbose` 
option?

Thanks.
--Vahid




From:   Jason Gustafson 
To: Kafka Users 
Cc: dev@kafka.apache.org
Date:   07/19/2017 01:46 PM
Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for 
ConsumerGroupCommand



Hey Vahid,

Thanks for the updates. Looks pretty good. A couple comments:

1. For the --state option, should we use the same column-oriented format 
as
we use for the other options? I realize there would only be one row, but
the inconsistency is a little vexing. Also, since this tool is working 
only
with consumer groups, perhaps we can leave out "protocol type" and use
"assignment strategy" in place of "protocol"? It would be nice to also
include the group generation, but it seems we didn't add that to the
DescribeGroup response. Perhaps we could also include a count of the 
number
of members?
2. It's a little annoying that --subscription and --members share so much
in common. Maybe we could drop --subscription and use a --verbose flag to
control whether or not to include the subscription and perhaps the
assignment as well? Not sure if that's more annoying or less, but maybe a
generic --verbose will be useful in other contexts.

As for your question on whether we need the --offsets option at all, I
don't have a strong opinion, but it seems to make the command semantics a
little more consistent.

-Jason

On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason,
>
> I updated the KIP based on your earlier suggestions:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand
> The only thing I am wondering at this point is whether it's worth to 
have
> a `--describe --offsets` option that behaves exactly like `--describe`.
>
> Thanks.
> --Vahid
>
>
>
> From:   "Vahid S Hashemian" 
> To: dev@kafka.apache.org
> Cc: Kafka Users 
> Date:   07/17/2017 03:24 PM
> Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> ConsumerGroupCommand
>
>
>
> Hi Jason,
>
> Thanks for your quick feedback. Your suggestions seem reasonable.
> I'll start updating the KIP accordingly and will send out another note
> when it's ready.
>
> Regards.
> --Vahid
>
>
>
>
> From:   Jason Gustafson 
> To: dev@kafka.apache.org
> Cc: Kafka Users 
> Date:   07/17/2017 02:11 PM
> Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> ConsumerGroupCommand
>
>
>
> Hey Vahid,
>
> Hmm... If possible, it would be nice to avoid cluttering the default
> option
> too much, especially if it is information which is going to be the same
> for
> all members (such as the generation). My preference would be to use the
> --state option that you've suggested for that info so that we can
> represent
> it more concisely.
>
> The reason I prefer the current output is that it is clear every entry
> corresponds to a partition for which we have committed offset. Entries
> like
> this look strange:
>
> TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> LAGCONSUMER-ID
> HOST   CLIENT-ID
> -  -  -   -
> -  consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff
> /127.0.0.1
> consumer4
> -  -  -   -
> -  consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea
> /127.0.0.1
> consumer5
>
> It makes me think that the consumers have committed offsets for an 
unknown
> partition. The --members option seems like a clearer way to communicate
> the
> fact that there are some members with no assigned partitions.
>
> A few additional suggestions:
>
> 1. Maybe we can rename --partitions to --offsets or --committed-offsets
> and
> the output could match the default output (in other words, --offsets is
> treated as the default switch). Seems no harm including the assignment
> information if we have it.
> 2. Along the lines of Onur's comment, it would be nice if the --members
> option included the list of assignment strategies that the consumer 
joined
> with (round-robin, range, etc). This list should always be small.
> 3. Thinking a little more, I'm not sure how necessary a --topics option
> is.
> The --partitions (or --offsets) option already shows the current
> assignment. Maybe --topics could be --subscription and just list the
> topics
> that the members subscribed to?
>
> Thanks,
> Jason
>
> On Mon, Jul 17, 2017 at 11:04 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Jason, Onur, thank you for reviewing the KIP.
> >
> > Regarding the default 

RE: Using JMXMP to access Kafka metrics

[Vijay Prakash]

Jolokia is not really an option for me although I've looked at it. I just want 
to know if there's some way to enable using JMXMP to access the metrics pushed 
to JMX. 

-Original Message-
From: Svante Karlsson [mailto:svante.karls...@csi.se] 
Sent: Wednesday, July 19, 2017 12:24 PM
To: us...@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Using JMXMP to access Kafka metrics

I've used jolokia which gets JMX metrics without RMI (actually json over
http)
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjolokia.org%2F=02%7C01%7CVijay.Prakash%40microsoft.com%7Cf33af52736994dff3a6e08d4cedbaea0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636360890423012957=QZRVpJI6tPRcuHEPMutTJwICSJtoTCuPj1Ndi0xhlXo%3D=0

Integrates nicely with telegraf (and influxdb)

2017-07-19 20:47 GMT+02:00 Vijay Prakash <
vijay.prak...@microsoft.com.invalid>:

> Hey,
>
> Is there a way to use JMXMP instead of RMI to access Kafka metrics 
> through JMX? I tried creating a JMXMP JMXConnector but the connect 
> attempt just hangs forever.
>
> Thanks,
> Vijay
>

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

[Guozhang Wang]

Per 1. I suggested exposing the constant since we are doing so for consumer
and producer configs prefix as well (CONSUMER_PREFIX, etc).

Guozhang

On Wed, Jul 19, 2017 at 6:01 AM, Ismael Juma  wrote:

> Thanks for the KIP, Damian. +1 (binding). A couple of minor comments:
>
> 1. Do we need to expose the TOPIC_PREFIX constant?
> 2. The vote thread ended up inside the discuss thread in Gmail. It may be
> worth sending another email to make it clear that the vote is ongoing. You
> can link back to this thread so that the existing votes are still counted.
>
> Ismael
>
> On Mon, Jul 17, 2017 at 4:43 AM, Damian Guy  wrote:
>
> > Hi,
> >
> > I'd like to kick off the vote for KIP-173:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 173%3A+Add+prefix+to+StreamsConfig+to+enable+setting+default+internal+
> > topic+configs
> >
> > A PR for this can be found here: https://github.com/apache/
> kafka/pull/3459
> >
> > Thanks,
> > Damian
> >
>



-- 
-- Guozhang

[GitHub] kafka pull request #3550: KAFKA-5610: KafkaApis.HandleWriteTxnMarkerRequest ...

[apurvam]

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3550

KAFKA-5610: KafkaApis.HandleWriteTxnMarkerRequest should return 
UNKNOWN_TOPIC_OR_PARTITION on partition emigration.

Before this patch, we would return the non-retriable 
`UNSUPPORTED_FOR_MESSAGE_FORMAT` error when leadership changed, causing markers 
to be lost.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5610-handleWriteTxnMarker-should-handle-emigration

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3550


commit 3ceba90e31964dc13e264376c993aadfe29e632c
Author: Apurva Mehta 
Date:   2017-07-19T22:10:33Z

Return UKNOWN_TOPIC_OR_PARTITION when partition emigrates during 
KafkaApis.handleWriteTxnMarkerRequest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Brokers restart when ZK address changes

[Andrey Dyachkov]

Good day,
We are using AWS and it is quite common for us to terminate instances very
often, which introduces ip address change and requires to restart brokers
one by one.
Maybe there is a knowledge on how to avoid such restarts?

According to Confluent recommendations(on their web page), one should use
Elastic IPs, but it exposes our zookeeper to outside world.
>From our side we looked at a zk Kafka client, which uses original zk
client, and it happens that there is an issue in the library and even
almost merged PR https://github.com/apache/zookeeper/pull/150, which will
solve the existing problem.

Thank you in advance!
-- 

With great enthusiasm,
Andrey

[jira] [Created] (KAFKA-5610) KafkaApis.handleWriteTxnMarkerRequest can return UNSUPPORTED_FOR_MESSAGE_FORMAT error on partition emigration

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5610:
---

 Summary: KafkaApis.handleWriteTxnMarkerRequest can return 
UNSUPPORTED_FOR_MESSAGE_FORMAT error on partition emigration
 Key: KAFKA-5610
 URL: https://issues.apache.org/jira/browse/KAFKA-5610
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.0
Reporter: Apurva Mehta
Assignee: Apurva Mehta
Priority: Critical
 Fix For: 0.11.0.1


This bug was revealed by the following system test failure 
http://confluent-systest.s3-website-us-west-2.amazonaws.com/confluent-kafka-system-test-results/?prefix=2017-07-18--001.1500383975--apache--trunk--28c83d9/

What happened was that a commit marker to the offsets topic (sent as part of 
the {{producer.sendOffsetsToTransaction}} method) was lost, causing data to be 
reprocessed, and hence causing the test to fail. 

The bug is that the wrong error code is returned from the handleWriteTxnMarker 
request when there is partition emigration. In particular, we have: 

{code:scala}
for (marker <- markers.asScala) {
  val producerId = marker.producerId
  val (goodPartitions, partitionsWithIncorrectMessageFormat) = 
marker.partitions.asScala.partition { partition =>
replicaManager.getMagic(partition) match {
  case Some(magic) if magic >= RecordBatch.MAGIC_VALUE_V2 => true
  case _ => false
}
  }

  if (partitionsWithIncorrectMessageFormat.nonEmpty) {
val currentErrors = new ConcurrentHashMap[TopicPartition, Errors]()
partitionsWithIncorrectMessageFormat.foreach { partition => 
currentErrors.put(partition, Errors.UNSUPPORTED_FOR_MESSAGE_FORMAT) }
updateErrors(producerId, currentErrors)
  }
{code}

But the {{replicaManager.getMagic()}} call will return {{None}} when the 
partition emigrates, causing the {{handleWriteTxnMarkersRequest}} to return an 
{{UNSUPPORTED_FOR_MESSAGE_FORMAT}} error. 

>From the log, we see that the partition did emigrate a few milliseconds before 
>the {{WriteTxnMarkerRequest}} was sent.

On the old leader, worker10:
{noformat}
./worker10/debug/server.log:32245:[2017-07-18 05:43:20,950] INFO 
[GroupCoordinator 2]: Unloading group metadata for 
transactions-test-consumer-group with generation 0 
(kafka.coordinator.group.GroupCoordinator)
{noformat}

On the client: 
{noformat}
[2017-07-18 05:43:20,959] INFO [Transaction Marker Request Completion Handler 
1]: Sending my-first-transactional-id's transaction marker from partition 
__consumer_offsets-47 has failed with  UNSUPPORTED_FOR_MESSAGE_FORMAT. This 
partition will be removed from the set of partitions waiting for completion 
(kafka.coordinator.transaction.TransactionMarkerRequestCompletionHandler)
{noformat}

As you can see, the client received the response 9 ms after the emigration was 
initiated on the server.

Since it is perfectly acceptable for the LeaderISR metadata to be propagated 
asynchronously, we should have more robust handling of emgiration in KafkaApis 
so that it returns the right error code when handling a request for a partition 
for which the broker is no longer the leader.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

[Jason Gustafson]

Hey Vahid,

Thanks for the updates. Looks pretty good. A couple comments:

1. For the --state option, should we use the same column-oriented format as
we use for the other options? I realize there would only be one row, but
the inconsistency is a little vexing. Also, since this tool is working only
with consumer groups, perhaps we can leave out "protocol type" and use
"assignment strategy" in place of "protocol"? It would be nice to also
include the group generation, but it seems we didn't add that to the
DescribeGroup response. Perhaps we could also include a count of the number
of members?
2. It's a little annoying that --subscription and --members share so much
in common. Maybe we could drop --subscription and use a --verbose flag to
control whether or not to include the subscription and perhaps the
assignment as well? Not sure if that's more annoying or less, but maybe a
generic --verbose will be useful in other contexts.

As for your question on whether we need the --offsets option at all, I
don't have a strong opinion, but it seems to make the command semantics a
little more consistent.

-Jason

On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason,
>
> I updated the KIP based on your earlier suggestions:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand
> The only thing I am wondering at this point is whether it's worth to have
> a `--describe --offsets` option that behaves exactly like `--describe`.
>
> Thanks.
> --Vahid
>
>
>
> From:   "Vahid S Hashemian" 
> To: dev@kafka.apache.org
> Cc: Kafka Users 
> Date:   07/17/2017 03:24 PM
> Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> ConsumerGroupCommand
>
>
>
> Hi Jason,
>
> Thanks for your quick feedback. Your suggestions seem reasonable.
> I'll start updating the KIP accordingly and will send out another note
> when it's ready.
>
> Regards.
> --Vahid
>
>
>
>
> From:   Jason Gustafson 
> To: dev@kafka.apache.org
> Cc: Kafka Users 
> Date:   07/17/2017 02:11 PM
> Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for
> ConsumerGroupCommand
>
>
>
> Hey Vahid,
>
> Hmm... If possible, it would be nice to avoid cluttering the default
> option
> too much, especially if it is information which is going to be the same
> for
> all members (such as the generation). My preference would be to use the
> --state option that you've suggested for that info so that we can
> represent
> it more concisely.
>
> The reason I prefer the current output is that it is clear every entry
> corresponds to a partition for which we have committed offset. Entries
> like
> this look strange:
>
> TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET
> LAGCONSUMER-ID
> HOST   CLIENT-ID
> -  -  -   -
> -  consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff
> /127.0.0.1
> consumer4
> -  -  -   -
> -  consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea
> /127.0.0.1
> consumer5
>
> It makes me think that the consumers have committed offsets for an unknown
> partition. The --members option seems like a clearer way to communicate
> the
> fact that there are some members with no assigned partitions.
>
> A few additional suggestions:
>
> 1. Maybe we can rename --partitions to --offsets or --committed-offsets
> and
> the output could match the default output (in other words, --offsets is
> treated as the default switch). Seems no harm including the assignment
> information if we have it.
> 2. Along the lines of Onur's comment, it would be nice if the --members
> option included the list of assignment strategies that the consumer joined
> with (round-robin, range, etc). This list should always be small.
> 3. Thinking a little more, I'm not sure how necessary a --topics option
> is.
> The --partitions (or --offsets) option already shows the current
> assignment. Maybe --topics could be --subscription and just list the
> topics
> that the members subscribed to?
>
> Thanks,
> Jason
>
> On Mon, Jul 17, 2017 at 11:04 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Jason, Onur, thank you for reviewing the KIP.
> >
> > Regarding the default `--describe` option, so far there have been a few
> > suggestions that conflict a bit. Here are the suggestions:
> > - Keep the current behavior exactly as is (Edo, Jeff)
> > - Remove members with no assignments from the current result set (Jason)
> > - Add additional status info to the result set (Onur) -- I assume the
> > additional status (which are group related info, rather than group
> member
> > related) will appear in the result separate from the member table (e.g.,
> > before the table)
> >
> > One thing we could 

Re: [DISCUSS] 2017 October release planning and release version

[Guozhang Wang]

Hi Stevo,

Just trying to add to what Ismael has already replied you:


> Practice/"features" like protocol version being a parameter, and
defaulting
> to latest so auto updated with dependency update which introduces new
> protocol/behavior should not be used in public client APIs. To switch
> between backward incompatible APIs (contract and behaviors), ideally user
> should explicitly have to change code and not dependency only, but at
least
> it should be clearly communicated that there are breaking changes to
expect
> even with just dependency update by e.g. giving major version release
clear
> meaning. If app dependency on Kafka client library minor.patch on same
> major is updated, and if there's a change in behavior or API requiring app
> code change - it's a bug.
>
> Change introduced contrary to the SLO, is OK to be reported as bug.
> Everything else is improvement or feature request.
>
> If this was the case, and 1.0.0 was released today with APIs as they are
> now, Scala client APIs even though deprecated would not break and require
> refactoring with every 1.* minor/patch release, and would only be allowed
> to be broken or removed in future major release, like 2.0.0

Just to clarify, my proposal is that moving forward beyond the next release
we will not make any public API breaking changes in any of the major or
minor releases, but will only mark them as "deprecated", and deprecated
public APIs will be only considered for removing as early as the next major
release: so if we mark the scala consumer APIs as deprecated in 1.0.0, we
should only be consider removing it at 2.0.0 or even later.

> It should be also clear how long is each version supported - e.g. if
> minor.patch had meaning that there are no backward incompatible changes,
> it's OK to file a bug only for current major.minor.patch; previous major
> and its last minor.patch can only have patches released up to some time
> like 1 up to 3 months.

Currently in practice we have not ever done, for example a bugfix release
on an older major / minor release: i.e. once we have released say 0.10.2.0
we did not release 0.10.1.2 any more. So practically speaking we do not
have a "support period" for older versions yet, and in the next coming
release I do not have plans to propose some concrete policy for that matter.


Guozhang



On Wed, Jul 19, 2017 at 2:12 AM, Ismael Juma  wrote:

> Hi Stevo,
>
> Thanks for your feedback. We should definitely do a better job of
> documenting things. We basically follow semantic versioning, but it's
> currently a bit confusing because:
>
> 1. There are 4 segments in the version. The "0." part should be ignored
> when deciding what is major, minor and patch at the moment, but many people
> don't know this. Once we move to 1.0.0, that problem goes away.
>
> 2. To know what is a public API, you must check the Javadoc (
> https://kafka.apache.org/0110/javadoc/index.html?org/apache/
> kafka/clients/consumer/KafkaConsumer.html).
> If it's not listed there, it's not public API. Ideally, it would be obvious
> from the package name (i.e. there would be "internals" in the name), but we
> are not there yet. The exception are the old Scala APIs, but they have all
> been deprecated and they will be removed eventually (the old Scala
> consumers won't be removed until the June 2018 release at the earliest in
> order to give people time to migrate).
>
> 3. Even though we are following reasonably common practices, we haven't
> documented them all in one place. It would be great to do it during the
> next release cycle.
>
> A few comments below.
>
> On Wed, Jul 19, 2017 at 1:31 AM, Stevo Slavić  wrote:
>
> > - APIs not labeled or labeled as stable
> > -- change in major version is only one that can break backward
> > compatibility (client APIs or behavior)
> >
>
> To clarify, stable APIs should not be changed in an incompatible way
> without a deprecation cycle. Independently of whether it's a major release
> or not.
>
>
> > -- change in minor version can introduce new features, but not break
> > backward compatibility
> > -- change in patch version, is for bug fixes only.
> >
>
> Right, this has been the case for a while already. Also see annotations
> below.
>
>
> > - APIs labeled as evolving can be broken in backward incompatible way in
> > any release, but are assumed less likely to be broken compared to
> unstable
> > APIs
> > - APIs labeled as unstable can be broken in backward incompatible way in
> > any release, major, minor or patch
> >
>
> The relevant annotations do explain this:
>
> https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/
> InterfaceStability.html
> https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/
> InterfaceStability.Stable.html
> https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/
> InterfaceStability.Evolving.html
> https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/
> 

Using JMXMP to access Kafka metrics

[Vijay Prakash]

Hey,

Is there a way to use JMXMP instead of RMI to access Kafka metrics through JMX? 
I tried creating a JMXMP JMXConnector but the connect attempt just hangs 
forever.

Thanks,
Vijay

Re: Using JMXMP to access Kafka metrics

[Svante Karlsson]

I've used jolokia which gets JMX metrics without RMI (actually json over
http)
https://jolokia.org/

Integrates nicely with telegraf (and influxdb)

2017-07-19 20:47 GMT+02:00 Vijay Prakash <
vijay.prak...@microsoft.com.invalid>:

> Hey,
>
> Is there a way to use JMXMP instead of RMI to access Kafka metrics through
> JMX? I tried creating a JMXMP JMXConnector but the connect attempt just
> hangs forever.
>
> Thanks,
> Vijay
>

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

[Jason Gustafson]

Ismael, I debated that also, but the main point was to make users aware of
the rebalance latency (with KIP-134 in mind). I'm guessing no one would
notice if it required another option. Note that the KIP does preserve the
existing fields (and in the same order), so if it is parsed as generic csv
data, it should be fine. But yeah, it could break some dumb parsers. In
general, I think we should at least allow ourselves compatible changes
given the output format that we have chosen for a tool.

-Jason

On Wed, Jul 19, 2017 at 7:54 AM, Ismael Juma  wrote:

> I think this is a good chance although it's unfortunate that it's likely to
> break code that is parsing the output of the performance tool. Would it
> make sense to only enable this if an option is provided?
>
> Ismael
>
> On Mon, Jul 17, 2017 at 3:41 PM, Jason Gustafson 
> wrote:
>
> > +Users
> >
> > Thanks for the KIP. I think tracking the rebalance time separately will
> > help resolve some confusion about the performance results given the
> > rebalance delay in KIP-134. And it seems generally useful to know how
> much
> > overhead is coming from the rebalance in any case.
> >
> > -Jason
> >
> > On Thu, Jul 13, 2017 at 4:15 PM, Hu Xi  wrote:
> >
> > > Hi all, I opened up a new KIP > > confluence/display/ARIES/KIP-177%3A+Consumer+perf+tool+
> > > should+count+rebalance+time> (KIP-177) concerning consumer perf tool
> > > counting and showing rebalance time in the output. Be free to leave
> your
> > > comments here. Thanks in advance.
> > >
> >
>

Re: [VOTE] KIP-162: Enable topic deletion by default

[Jason Gustafson]

+1

On Wed, Jul 19, 2017 at 7:51 AM, Damian Guy  wrote:

> +1
>
> On Tue, 6 Jun 2017 at 18:19 BigData dev  wrote:
>
> > +1 (non-binding)
> >
> > Thanks,
> > Bharat
> >
> > On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha 
> > wrote:
> >
> > > +1
> > >
> > > On Tue, Jun 6, 2017 at 11:20 PM, Mickael Maison <
> > mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > +1 (non binding), thanks
> > > >
> > > > On Tue, Jun 6, 2017 at 2:16 PM, Bill Bejeck 
> wrote:
> > > > > +1
> > > > >
> > > > > -Bill
> > > > >
> > > > > On Tue, Jun 6, 2017 at 9:08 AM, Ismael Juma 
> > wrote:
> > > > >
> > > > >> Thanks for the KIP, Gwen. +1 (binding).
> > > > >>
> > > > >> Ismael
> > > > >>
> > > > >> On Tue, Jun 6, 2017 at 5:37 AM, Gwen Shapira 
> > > wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > The discussion has been quite positive, so I posted a JIRA, a PR
> > and
> > > > >> > updated the KIP with the latest decisions.
> > > > >> >
> > > > >> > Lets officially vote on the KIP:
> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > 162+-+Enable+topic+deletion+by+default
> > > > >> >
> > > > >> > JIRA is here: https://issues.apache.org/jira/browse/KAFKA-5384
> > > > >> >
> > > > >> > Gwen
> > > > >> >
> > > > >>
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Ashwin
> > >
> >
>

Re: KIP-162: Enable topic deletion by default

[Bill Bejeck]

+1

On Wed, Jul 19, 2017 at 1:35 PM, Jay Kreps  wrote:

> +1
>
> On Sat, May 27, 2017 at 11:04 AM, Gwen Shapira  wrote:
>
> > Thanks Vahid,
> >
> > Do you mind if we leave the command-line out of scope for this?
> >
> > I can see why adding confirmations, options to bypass confirmations, etc
> > would be an improvement. However, I've seen no complaints about the
> current
> > behavior of the command-line and the KIP doesn't change it at all. So I'd
> > rather address things separately.
> >
> > Gwen
> >
> > On Fri, May 26, 2017 at 8:10 PM Vahid S Hashemian <
> > vahidhashem...@us.ibm.com>
> > wrote:
> >
> > > Gwen, thanks for the KIP.
> > > It looks good to me.
> > >
> > > Just a minor suggestion: It would be great if the command asks for a
> > > confirmation (y/n) before deleting the topic (similar to how removing
> > ACLs
> > > works).
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > > From:   Gwen Shapira 
> > > To: "dev@kafka.apache.org" , Users
> > > 
> > > Date:   05/26/2017 07:04 AM
> > > Subject:KIP-162: Enable topic deletion by default
> > >
> > >
> > >
> > > Hi Kafka developers, users and friends,
> > >
> > > I've added a KIP to improve our out-of-the-box usability a bit:
> > > KIP-162: Enable topic deletion by default:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 162+-+Enable+topic+deletion+by+default
> > >
> > >
> > > Pretty simple :) Discussion and feedback are welcome.
> > >
> > > Gwen
> > >
> > >
> > >
> > >
> > >
> >
>

[GitHub] kafka pull request #3222: MINOR: Code Cleanup

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3222


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk8 #1822

[Apache Jenkins Server]

See 

Re: KIP-162: Enable topic deletion by default

[Jay Kreps]

+1

On Sat, May 27, 2017 at 11:04 AM, Gwen Shapira  wrote:

> Thanks Vahid,
>
> Do you mind if we leave the command-line out of scope for this?
>
> I can see why adding confirmations, options to bypass confirmations, etc
> would be an improvement. However, I've seen no complaints about the current
> behavior of the command-line and the KIP doesn't change it at all. So I'd
> rather address things separately.
>
> Gwen
>
> On Fri, May 26, 2017 at 8:10 PM Vahid S Hashemian <
> vahidhashem...@us.ibm.com>
> wrote:
>
> > Gwen, thanks for the KIP.
> > It looks good to me.
> >
> > Just a minor suggestion: It would be great if the command asks for a
> > confirmation (y/n) before deleting the topic (similar to how removing
> ACLs
> > works).
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From:   Gwen Shapira 
> > To: "dev@kafka.apache.org" , Users
> > 
> > Date:   05/26/2017 07:04 AM
> > Subject:KIP-162: Enable topic deletion by default
> >
> >
> >
> > Hi Kafka developers, users and friends,
> >
> > I've added a KIP to improve our out-of-the-box usability a bit:
> > KIP-162: Enable topic deletion by default:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 162+-+Enable+topic+deletion+by+default
> >
> >
> > Pretty simple :) Discussion and feedback are welcome.
> >
> > Gwen
> >
> >
> >
> >
> >
>

Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster

[Abhishek Mendhekar]

Thank you all for voting.

The votes are as follows:

Bidding +1: Joel, Becket, and Ismael
Non-bidding +1: Dong, Onur, Mickael, Vahid, and Bill

This vote is passed as per Lazy Majority
.

Here is the link to the PR - https://github.com/apache/kafka/pull/3549


On Wed, Jul 19, 2017 at 8:21 AM, Bill Bejeck  wrote:

> +1
>
> On Wed, Jul 19, 2017 at 7:56 AM, Ismael Juma  wrote:
>
> > Thanks for the KIP, +1 (binding) from me.
> >
> > Ismael
> >
> > On Wed, Jun 21, 2017 at 1:17 PM, Abhishek Mendhekar <
> > abhishek.mendhe...@gmail.com> wrote:
> >
> > > Hi Kafka Dev,
> > >
> > > I did like to start the voting on -
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 168%3A+Add+TotalTopicCount+metric+per+cluster
> > >
> > > Discussions will continue on -
> > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201706.
> > > mbox/%3CCAMcwe-ugep-UiSn9TkKEMwwTM%3DAzGC4jPro9LnyYRezyZg_NKA%
> > > 40mail.gmail.com%3E
> > >
> > > Thanks,
> > > Abhishek
> > >
> >
>



-- 
Abhishek Mendhekar
abhishek.mendhe...@gmail.com | 818.263.7030

[GitHub] kafka pull request #3549: KAFKA-5461: Add metric to track global topic count...

[abhishekmendhekar]

GitHub user abhishekmendhekar opened a pull request:

https://github.com/apache/kafka/pull/3549

KAFKA-5461: Add metric to track global topic count and global parition 
count in a cluster



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abhishekmendhekar/kafka KAFKA-5461

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3549.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3549


commit 718b12659b6d21b783457e50565bb2ca3f2621da
Author: Abhishek Mendhekar 
Date:   2017-06-12T22:26:44Z

KAFKA-5461: Add metric to track global topic count and global parition 
count in a cluster




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #1821

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Make 'Topic-Level Configs' a doc section for easier access

--
[...truncated 4.33 MB...]

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
STARTED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics STARTED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithInvalidReplication 
STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithInvalidReplication 
PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.FetcherTest > testFetcher STARTED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
STARTED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride SKIPPED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
STARTED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
SKIPPED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig STARTED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

unit.kafka.server.KafkaApisTest > 
shouldRespondWithUnsupportedForMessageFormatOnHandleWriteTxnMarkersWhenMagicLowerThanRequired
 STARTED

unit.kafka.server.KafkaApisTest > 
shouldRespondWithUnsupportedForMessageFormatOnHandleWriteTxnMarkersWhenMagicLowerThanRequired
 PASSED

unit.kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleTxnOffsetCommitRequestWhenInterBrokerProtocolNotSupported
 STARTED

unit.kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleTxnOffsetCommitRequestWhenInterBrokerProtocolNotSupported
 PASSED

unit.kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleAddPartitionsToTxnRequestWhenInterBrokerProtocolNotSupported
 STARTED

unit.kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleAddPartitionsToTxnRequestWhenInterBrokerProtocolNotSupported
 PASSED

unit.kafka.server.KafkaApisTest > testReadUncommittedConsumerListOffsetLatest 
STARTED

unit.kafka.server.KafkaApisTest > testReadUncommittedConsumerListOffsetLatest 
PASSED

unit.kafka.server.KafkaApisTest > 

Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster

[Bill Bejeck]

+1

On Wed, Jul 19, 2017 at 7:56 AM, Ismael Juma  wrote:

> Thanks for the KIP, +1 (binding) from me.
>
> Ismael
>
> On Wed, Jun 21, 2017 at 1:17 PM, Abhishek Mendhekar <
> abhishek.mendhe...@gmail.com> wrote:
>
> > Hi Kafka Dev,
> >
> > I did like to start the voting on -
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 168%3A+Add+TotalTopicCount+metric+per+cluster
> >
> > Discussions will continue on -
> > http://mail-archives.apache.org/mod_mbox/kafka-dev/201706.
> > mbox/%3CCAMcwe-ugep-UiSn9TkKEMwwTM%3DAzGC4jPro9LnyYRezyZg_NKA%
> > 40mail.gmail.com%3E
> >
> > Thanks,
> > Abhishek
> >
>

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

[Tom Bentley]

Ah, thank you! I took the number from the "Next KIP Number: 178" on the KIP
index and didn't check the tables. So this is now KIP-179. The old link
will point you to the right place.

On 19 July 2017 at 15:55, Ismael Juma  wrote:

> One more thing, it looks like there is already a KIP-178:
>
> KIP-178: Size-based log directory selection strategy
>
> Ismael
>
> On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley 
> wrote:
>
> > OK, I will work on adding support for this to the KIP, with the intention
> > of a two part implementation.
> >
> > On 19 July 2017 at 14:59, Ismael Juma  wrote:
> >
> > > Hi Tom,
> > >
> > > It's fine for the tool not to have this functionality from the start.
> > > However, since we're adding new Kafka protocol APIs, we need to
> consider
> > > some of these details to ensure we're building towards the end state,
> if
> > > that makes sense. Protocol APIs are used by multiple clients, so there
> is
> > > value in thinking ahead a bit when it comes to the design. The
> > > implementation can often be done in stages.
> > >
> > > Does that make sense?
> > >
> > > Ismael
> > >
> > > On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley 
> > > wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > Answers in-line:
> > > >
> > > > 1. Have you considered how progress would be reported? Partition
> > > > > reassignment can take a long time and it would be good to have a
> > > > mechanism
> > > > > for progress reporting.
> > > > >
> > > >
> > > > The ReassignPartitionsCommand doesn't currently have a mechanism to
> > track
> > > > progress. All you can do at the moment is initiate a reassignment
> (with
> > > > --execute), and later check whether the assignment is in the state
> you
> > > > asked for (with --verify). I agree it would be nice to be able to
> track
> > > > progress.
> > > >
> > > > This will be the first 'big' bit of work I've done on Kafka, so I
> would
> > > > prefer to limit the scope of this KIP where possible. That said, I
> > > suppose
> > > > it could be done by having receiving controllers publish their
> progress
> > > to
> > > > ZooKeeper, and adding Protocol and AdminClient API for getting this
> > > > information. If you're keen on this I can certainly modify the KIP to
> > add
> > > > this.
> > > >
> > > > Alternatively I could write a second KIP to add this ability. What
> > other
> > > > long running tasks are there for which we'd like the ability to
> report
> > > > progress? If there are others it might be possible to come up with a
> > > common
> > > > mechanism.
> > > >
> > > >
> > > > > 2. Removals can only happen in major releases. In your example, the
> > > > removal
> > > > > could only happen in 2.0.0.
> > > > >
> > > >
> > > > OK, I'll update the KIP.
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

[Ismael Juma]

One more thing, it looks like there is already a KIP-178:

KIP-178: Size-based log directory selection strategy

Ismael

On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley  wrote:

> OK, I will work on adding support for this to the KIP, with the intention
> of a two part implementation.
>
> On 19 July 2017 at 14:59, Ismael Juma  wrote:
>
> > Hi Tom,
> >
> > It's fine for the tool not to have this functionality from the start.
> > However, since we're adding new Kafka protocol APIs, we need to consider
> > some of these details to ensure we're building towards the end state, if
> > that makes sense. Protocol APIs are used by multiple clients, so there is
> > value in thinking ahead a bit when it comes to the design. The
> > implementation can often be done in stages.
> >
> > Does that make sense?
> >
> > Ismael
> >
> > On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley 
> > wrote:
> >
> > > Hi Ismael,
> > >
> > > Answers in-line:
> > >
> > > 1. Have you considered how progress would be reported? Partition
> > > > reassignment can take a long time and it would be good to have a
> > > mechanism
> > > > for progress reporting.
> > > >
> > >
> > > The ReassignPartitionsCommand doesn't currently have a mechanism to
> track
> > > progress. All you can do at the moment is initiate a reassignment (with
> > > --execute), and later check whether the assignment is in the state you
> > > asked for (with --verify). I agree it would be nice to be able to track
> > > progress.
> > >
> > > This will be the first 'big' bit of work I've done on Kafka, so I would
> > > prefer to limit the scope of this KIP where possible. That said, I
> > suppose
> > > it could be done by having receiving controllers publish their progress
> > to
> > > ZooKeeper, and adding Protocol and AdminClient API for getting this
> > > information. If you're keen on this I can certainly modify the KIP to
> add
> > > this.
> > >
> > > Alternatively I could write a second KIP to add this ability. What
> other
> > > long running tasks are there for which we'd like the ability to report
> > > progress? If there are others it might be possible to come up with a
> > common
> > > mechanism.
> > >
> > >
> > > > 2. Removals can only happen in major releases. In your example, the
> > > removal
> > > > could only happen in 2.0.0.
> > > >
> > >
> > > OK, I'll update the KIP.
> > >
> >
>

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

[Ismael Juma]

I think this is a good chance although it's unfortunate that it's likely to
break code that is parsing the output of the performance tool. Would it
make sense to only enable this if an option is provided?

Ismael

On Mon, Jul 17, 2017 at 3:41 PM, Jason Gustafson  wrote:

> +Users
>
> Thanks for the KIP. I think tracking the rebalance time separately will
> help resolve some confusion about the performance results given the
> rebalance delay in KIP-134. And it seems generally useful to know how much
> overhead is coming from the rebalance in any case.
>
> -Jason
>
> On Thu, Jul 13, 2017 at 4:15 PM, Hu Xi  wrote:
>
> > Hi all, I opened up a new KIP > confluence/display/ARIES/KIP-177%3A+Consumer+perf+tool+
> > should+count+rebalance+time> (KIP-177) concerning consumer perf tool
> > counting and showing rebalance time in the output. Be free to leave your
> > comments here. Thanks in advance.
> >
>

Re: KIP-162: Enable topic deletion by default

[Damian Guy]

+1

On Wed, 31 May 2017 at 13:36 Jim Jagielski  wrote:

> +1
> > On May 27, 2017, at 9:27 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
> >
> > Sure, that sounds good.
> >
> > I suggested that to keep command line behavior consistent.
> > Plus, removal of ACL access is something that can be easily undone, but
> > topic deletion is not reversible.
> > So, perhaps a new follow-up JIRA to this KIP to add the confirmation for
> > topic deletion.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From:   Gwen Shapira 
> > To: dev@kafka.apache.org, us...@kafka.apache.org
> > Date:   05/27/2017 11:04 AM
> > Subject:Re: KIP-162: Enable topic deletion by default
> >
> >
> >
> > Thanks Vahid,
> >
> > Do you mind if we leave the command-line out of scope for this?
> >
> > I can see why adding confirmations, options to bypass confirmations, etc
> > would be an improvement. However, I've seen no complaints about the
> > current
> > behavior of the command-line and the KIP doesn't change it at all. So I'd
> > rather address things separately.
> >
> > Gwen
> >
> > On Fri, May 26, 2017 at 8:10 PM Vahid S Hashemian
> > 
> > wrote:
> >
> >> Gwen, thanks for the KIP.
> >> It looks good to me.
> >>
> >> Just a minor suggestion: It would be great if the command asks for a
> >> confirmation (y/n) before deleting the topic (similar to how removing
> > ACLs
> >> works).
> >>
> >> Thanks.
> >> --Vahid
> >>
> >>
> >>
> >> From:   Gwen Shapira 
> >> To: "dev@kafka.apache.org" , Users
> >> 
> >> Date:   05/26/2017 07:04 AM
> >> Subject:KIP-162: Enable topic deletion by default
> >>
> >>
> >>
> >> Hi Kafka developers, users and friends,
> >>
> >> I've added a KIP to improve our out-of-the-box usability a bit:
> >> KIP-162: Enable topic deletion by default:
> >>
> >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default
> >
> >>
> >>
> >> Pretty simple :) Discussion and feedback are welcome.
> >>
> >> Gwen
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
> >
>
>

Re: [VOTE] KIP-162: Enable topic deletion by default

[Damian Guy]

+1

On Tue, 6 Jun 2017 at 18:19 BigData dev  wrote:

> +1 (non-binding)
>
> Thanks,
> Bharat
>
> On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha 
> wrote:
>
> > +1
> >
> > On Tue, Jun 6, 2017 at 11:20 PM, Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> > > +1 (non binding), thanks
> > >
> > > On Tue, Jun 6, 2017 at 2:16 PM, Bill Bejeck  wrote:
> > > > +1
> > > >
> > > > -Bill
> > > >
> > > > On Tue, Jun 6, 2017 at 9:08 AM, Ismael Juma 
> wrote:
> > > >
> > > >> Thanks for the KIP, Gwen. +1 (binding).
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Tue, Jun 6, 2017 at 5:37 AM, Gwen Shapira 
> > wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > The discussion has been quite positive, so I posted a JIRA, a PR
> and
> > > >> > updated the KIP with the latest decisions.
> > > >> >
> > > >> > Lets officially vote on the KIP:
> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > 162+-+Enable+topic+deletion+by+default
> > > >> >
> > > >> > JIRA is here: https://issues.apache.org/jira/browse/KAFKA-5384
> > > >> >
> > > >> > Gwen
> > > >> >
> > > >>
> > >
> >
> >
> >
> > --
> > Thanks and Regards,
> > Ashwin
> >
>

Re: [VOTE] KIP-162: Enable topic deletion by default

[Rajini Sivaram]

Thanks for the KIP.

+1 (binding)

On Tue, Jun 6, 2017 at 6:19 PM, BigData dev  wrote:

> +1 (non-binding)
>
> Thanks,
> Bharat
>
> On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha 
> wrote:
>
> > +1
> >
> > On Tue, Jun 6, 2017 at 11:20 PM, Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> > > +1 (non binding), thanks
> > >
> > > On Tue, Jun 6, 2017 at 2:16 PM, Bill Bejeck  wrote:
> > > > +1
> > > >
> > > > -Bill
> > > >
> > > > On Tue, Jun 6, 2017 at 9:08 AM, Ismael Juma 
> wrote:
> > > >
> > > >> Thanks for the KIP, Gwen. +1 (binding).
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Tue, Jun 6, 2017 at 5:37 AM, Gwen Shapira 
> > wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > The discussion has been quite positive, so I posted a JIRA, a PR
> and
> > > >> > updated the KIP with the latest decisions.
> > > >> >
> > > >> > Lets officially vote on the KIP:
> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > 162+-+Enable+topic+deletion+by+default
> > > >> >
> > > >> > JIRA is here: https://issues.apache.org/jira/browse/KAFKA-5384
> > > >> >
> > > >> > Gwen
> > > >> >
> > > >>
> > >
> >
> >
> >
> > --
> > Thanks and Regards,
> > Ashwin
> >
>

Jenkins build is back to normal : kafka-0.11.0-jdk7 #230

[Apache Jenkins Server]

See 

Want some help about admin client

[Joe Chen]

Hi,

My requirement is that i want to know which node of cluster is alive,
something like health check, and we will create some topic too with the
admin client java api .

In health check we use adminClient.describeCluster() to list alive node, at
the beginning that works, but after sometimes (allways default timeout
milliseconds) , it get “no node found” exception, but actually the kafka
server is alive. It seems cause by the previous call is timeout and then
the kafka client disconnect, but  actually that call  have get the response
and successful. Is someone can help?

Below is the detail logs：

2017-07-19 11:49:50.397 main
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: queueing Call(callName=createTopics,
deadlineMs=1500436310397) with a timeout 12 ms from now.
2017-07-19 11:49:50.398 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
KafkaClient#poll retrieved 0 response(s)
2017-07-19 11:49:50.398 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
assigned Call(callName=createTopics, deadlineMs=1500436310397) to
192.168.0.3:9092 (id: 0 rack: null)
2017-07-19 11:49:50.398 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
sending (type=CreateTopicsRequest,
topics={_INTERNAL_SUBSCRIPTION_568cfbce-dda4-4c36-822c-09c446c49b08=(numPartitions=1,
replicationFactor=1, replicasAssignments={}, configs={})}, timeout=11,
validateOnly=false) to 192.168.0.3:9092 (id: 0 rack: null). correlationId=4
2017-07-19 11:49:50.398 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
entering KafkaClient#poll(timeout=111824)
2017-07-19 11:49:50.398 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
KafkaClient#poll retrieved 0 response(s)
2017-07-19 11:49:50.398 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
entering KafkaClient#poll(timeout=111824)
2017-07-19 11:49:50.412 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
KafkaClient#poll retrieved 1 response(s)
2017-07-19 11:49:50.413 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
Call(callName=createTopics, deadlineMs=1500436310397) got response
org.apache.kafka.common.requests.CreateTopicsResponse@c6f92a
2017-07-19 11:49:50.413 kafka-admin-client-thread
org.apache.kafka.clients.admin.KafkaAdminClient {} GenericKafkaAdminClient:
entering KafkaClient#poll(timeout=111810)

2017-07-19 11:51:32.908 monitor-schedulers1
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: queueing Call(callName=listNodes,
deadlineMs=1500436412908) with a timeout 12 ms from now.
2017-07-19 11:51:32.908 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: KafkaClient#poll retrieved 0 response(s)
2017-07-19 11:51:32.908 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: assigned Call(callName=listNodes,
deadlineMs=1500436412908) to 192.168.0.3:9092 (id: 0 rack: null)
2017-07-19 11:51:32.908 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: sending (type=MetadataRequest, topics=) to
192.168.0.3:9092 (id: 0 rack: null). correlationId=12
2017-07-19 11:51:32.909 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: entering KafkaClient#poll(timeout=9314)
2017-07-19 11:51:32.909 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: KafkaClient#poll retrieved 0 response(s)
2017-07-19 11:51:32.909 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: entering KafkaClient#poll(timeout=9313)
2017-07-19 11:51:32.909 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: KafkaClient#poll retrieved 1 response(s)
2017-07-19 11:51:32.910 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: Call(callName=listNodes, deadlineMs=1500436412908)
got response org.apache.kafka.common.requests.MetadataResponse@40de203e
2017-07-19 11:51:32.910 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: entering KafkaClient#poll(timeout=9313)
2017-07-19 11:51:32.910 monitor-schedulers1  com.monitor.health.HealthCheck
{} Kafka status: UP {Alive Nodes=[192.168.0.3]}

2017-07-19 11:51:42.223 kafka-admin-client-thread
 org.apache.kafka.clients.admin.KafkaAdminClient {}
GenericKafkaAdminClient: KafkaClient#poll retrieved 0 response(s)
2017-07-19 11:51:42.223 kafka-admin-client-thread  GenericKafkaAdminClient
DEBUG 

Jenkins build is back to normal : kafka-trunk-jdk7 #2532

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

[Tom Bentley]

OK, I will work on adding support for this to the KIP, with the intention
of a two part implementation.

On 19 July 2017 at 14:59, Ismael Juma  wrote:

> Hi Tom,
>
> It's fine for the tool not to have this functionality from the start.
> However, since we're adding new Kafka protocol APIs, we need to consider
> some of these details to ensure we're building towards the end state, if
> that makes sense. Protocol APIs are used by multiple clients, so there is
> value in thinking ahead a bit when it comes to the design. The
> implementation can often be done in stages.
>
> Does that make sense?
>
> Ismael
>
> On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley 
> wrote:
>
> > Hi Ismael,
> >
> > Answers in-line:
> >
> > 1. Have you considered how progress would be reported? Partition
> > > reassignment can take a long time and it would be good to have a
> > mechanism
> > > for progress reporting.
> > >
> >
> > The ReassignPartitionsCommand doesn't currently have a mechanism to track
> > progress. All you can do at the moment is initiate a reassignment (with
> > --execute), and later check whether the assignment is in the state you
> > asked for (with --verify). I agree it would be nice to be able to track
> > progress.
> >
> > This will be the first 'big' bit of work I've done on Kafka, so I would
> > prefer to limit the scope of this KIP where possible. That said, I
> suppose
> > it could be done by having receiving controllers publish their progress
> to
> > ZooKeeper, and adding Protocol and AdminClient API for getting this
> > information. If you're keen on this I can certainly modify the KIP to add
> > this.
> >
> > Alternatively I could write a second KIP to add this ability. What other
> > long running tasks are there for which we'd like the ability to report
> > progress? If there are others it might be possible to come up with a
> common
> > mechanism.
> >
> >
> > > 2. Removals can only happen in major releases. In your example, the
> > removal
> > > could only happen in 2.0.0.
> > >
> >
> > OK, I'll update the KIP.
> >
>

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

[Ismael Juma]

Hi Tom,

It's fine for the tool not to have this functionality from the start.
However, since we're adding new Kafka protocol APIs, we need to consider
some of these details to ensure we're building towards the end state, if
that makes sense. Protocol APIs are used by multiple clients, so there is
value in thinking ahead a bit when it comes to the design. The
implementation can often be done in stages.

Does that make sense?

Ismael

On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley  wrote:

> Hi Ismael,
>
> Answers in-line:
>
> 1. Have you considered how progress would be reported? Partition
> > reassignment can take a long time and it would be good to have a
> mechanism
> > for progress reporting.
> >
>
> The ReassignPartitionsCommand doesn't currently have a mechanism to track
> progress. All you can do at the moment is initiate a reassignment (with
> --execute), and later check whether the assignment is in the state you
> asked for (with --verify). I agree it would be nice to be able to track
> progress.
>
> This will be the first 'big' bit of work I've done on Kafka, so I would
> prefer to limit the scope of this KIP where possible. That said, I suppose
> it could be done by having receiving controllers publish their progress to
> ZooKeeper, and adding Protocol and AdminClient API for getting this
> information. If you're keen on this I can certainly modify the KIP to add
> this.
>
> Alternatively I could write a second KIP to add this ability. What other
> long running tasks are there for which we'd like the ability to report
> progress? If there are others it might be possible to come up with a common
> mechanism.
>
>
> > 2. Removals can only happen in major releases. In your example, the
> removal
> > could only happen in 2.0.0.
> >
>
> OK, I'll update the KIP.
>

[GitHub] kafka pull request #3415: MINOR: Make 'Topic-Level Configs' a doc section fo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3415


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

[Tom Bentley]

Hi Ismael,

Answers in-line:

1. Have you considered how progress would be reported? Partition
> reassignment can take a long time and it would be good to have a mechanism
> for progress reporting.
>

The ReassignPartitionsCommand doesn't currently have a mechanism to track
progress. All you can do at the moment is initiate a reassignment (with
--execute), and later check whether the assignment is in the state you
asked for (with --verify). I agree it would be nice to be able to track
progress.

This will be the first 'big' bit of work I've done on Kafka, so I would
prefer to limit the scope of this KIP where possible. That said, I suppose
it could be done by having receiving controllers publish their progress to
ZooKeeper, and adding Protocol and AdminClient API for getting this
information. If you're keen on this I can certainly modify the KIP to add
this.

Alternatively I could write a second KIP to add this ability. What other
long running tasks are there for which we'd like the ability to report
progress? If there are others it might be possible to come up with a common
mechanism.


> 2. Removals can only happen in major releases. In your example, the removal
> could only happen in 2.0.0.
>

OK, I'll update the KIP.

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

[Ismael Juma]

Thanks for the KIP, Damian. +1 (binding). A couple of minor comments:

1. Do we need to expose the TOPIC_PREFIX constant?
2. The vote thread ended up inside the discuss thread in Gmail. It may be
worth sending another email to make it clear that the vote is ongoing. You
can link back to this thread so that the existing votes are still counted.

Ismael

On Mon, Jul 17, 2017 at 4:43 AM, Damian Guy  wrote:

> Hi,
>
> I'd like to kick off the vote for KIP-173:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 173%3A+Add+prefix+to+StreamsConfig+to+enable+setting+default+internal+
> topic+configs
>
> A PR for this can be found here: https://github.com/apache/kafka/pull/3459
>
> Thanks,
> Damian
>

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

[Ismael Juma]

Hi Tom,

Thanks for the KIP. Two quick comments:

1. Have you considered how progress would be reported? Partition
reassignment can take a long time and it would be good to have a mechanism
for progress reporting.
2. Removals can only happen in major releases. In your example, the removal
could only happen in 2.0.0.

Thanks.
Ismael

On Wed, Jul 19, 2017 at 5:49 AM, Tom Bentley  wrote:

> Hi,
>
> I've written KIP-178 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-178+-+Change+
> ReassignPartitionsCommand+to+use+AdminClient)
> for changing the ReassignPartitionsCommand (a.k.a.
> kafka-reassign-partitions.sh)to remove the dependency on ZooKeeper and
> instead provide the same functionality via the AdminClient API.
>
> I'd be grateful for any feedback people may have on this.
>
> Cheers,
>
> Tom
>

[DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

[Tom Bentley]

Hi,

I've written KIP-178 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-178+-+Change+ReassignPartitionsCommand+to+use+AdminClient)
for changing the ReassignPartitionsCommand (a.k.a.
kafka-reassign-partitions.sh)to remove the dependency on ZooKeeper and
instead provide the same functionality via the AdminClient API.

I'd be grateful for any feedback people may have on this.

Cheers,

Tom

Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster

[Ismael Juma]

Thanks for the KIP, +1 (binding) from me.

Ismael

On Wed, Jun 21, 2017 at 1:17 PM, Abhishek Mendhekar <
abhishek.mendhe...@gmail.com> wrote:

> Hi Kafka Dev,
>
> I did like to start the voting on -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 168%3A+Add+TotalTopicCount+metric+per+cluster
>
> Discussions will continue on -
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201706.
> mbox/%3CCAMcwe-ugep-UiSn9TkKEMwwTM%3DAzGC4jPro9LnyYRezyZg_NKA%
> 40mail.gmail.com%3E
>
> Thanks,
> Abhishek
>

[GitHub] kafka pull request #3548: KAFKA-5607: Improve error handling in SocketServer...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/3548

KAFKA-5607: Improve error handling in SocketServer to avoid issues later



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-5607

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3548.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3548


commit 9d656f066dbdb86c78e150593c94cc118c843038
Author: Rajini Sivaram 
Date:   2017-07-19T09:42:43Z

KAFKA-5607: Improve error handling in SocketServer to avoid issues later




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] 2017 October release planning and release version

[Ismael Juma]

Hi Stevo,

Thanks for your feedback. We should definitely do a better job of
documenting things. We basically follow semantic versioning, but it's
currently a bit confusing because:

1. There are 4 segments in the version. The "0." part should be ignored
when deciding what is major, minor and patch at the moment, but many people
don't know this. Once we move to 1.0.0, that problem goes away.

2. To know what is a public API, you must check the Javadoc (
https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html).
If it's not listed there, it's not public API. Ideally, it would be obvious
from the package name (i.e. there would be "internals" in the name), but we
are not there yet. The exception are the old Scala APIs, but they have all
been deprecated and they will be removed eventually (the old Scala
consumers won't be removed until the June 2018 release at the earliest in
order to give people time to migrate).

3. Even though we are following reasonably common practices, we haven't
documented them all in one place. It would be great to do it during the
next release cycle.

A few comments below.

On Wed, Jul 19, 2017 at 1:31 AM, Stevo Slavić  wrote:

> - APIs not labeled or labeled as stable
> -- change in major version is only one that can break backward
> compatibility (client APIs or behavior)
>

To clarify, stable APIs should not be changed in an incompatible way
without a deprecation cycle. Independently of whether it's a major release
or not.


> -- change in minor version can introduce new features, but not break
> backward compatibility
> -- change in patch version, is for bug fixes only.
>

Right, this has been the case for a while already. Also see annotations
below.


> - APIs labeled as evolving can be broken in backward incompatible way in
> any release, but are assumed less likely to be broken compared to unstable
> APIs
> - APIs labeled as unstable can be broken in backward incompatible way in
> any release, major, minor or patch
>

The relevant annotations do explain this:

https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/InterfaceStability.html
https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/InterfaceStability.Stable.html
https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/InterfaceStability.Evolving.html
https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/InterfaceStability.Unstable.html

But we should have a section in our documentation as well.


> - deprecated stable APIs are treated as any stable APIs, they can be
> removed only in major release, are not allowed to be changed in backward
> incompatible way in either patch or minor version release
>

Right, but note that stable non-deprecated APIs provide stronger guarantees
in major releases (they can't be changed in an incompatible way).

>
> This means one should be able to upgrade server and recompile/deploy apps
> with clients to new minor.patch release with dependency version change
> being only change needed and there would be no drama.
>

That should have been the case for a while as long as you are using stable
public APIs.

>
> Practice/"features" like protocol version being a parameter, and defaulting
> to latest so auto updated with dependency update which introduces new
> protocol/behavior should not be used in public client APIs. To switch
> between backward incompatible APIs (contract and behaviors), ideally user
> should explicitly have to change code and not dependency only, but at least
> it should be clearly communicated that there are breaking changes to expect
> even with just dependency update by e.g. giving major version release clear
> meaning. If app dependency on Kafka client library minor.patch on same
> major is updated, and if there's a change in behavior or API requiring app
> code change - it's a bug.
>

Hmm, if the protocol bump provides improved behaviour, that is not a
backwards incompatible change though. So, I don't think I agree with this.
Of course,
it does mean that _downgrading_ may cause loss of functionality. That's OK,
in my opinion.

Change introduced contrary to the SLO, is OK to be reported as bug.
> Everything else is improvement or feature request.
>
> If this was the case, and 1.0.0 was released today with APIs as they are
> now, Scala client APIs even though deprecated would not break and require
> refactoring with every 1.* minor/patch release, and would only be allowed
> to be broken or removed in future major release, like 2.0.0
>

Yes, that is the plan for any _public_ Scala client APIs that are still
present in 1.0.0. The public Scala client APIs are the producer and
consumer, basically. Again, we should make this clear in our documentation.
Note that we have made an effort to keep those APIs compatible for quite a
while. It sounds like you have had some issues, were they related to usage
of internal Admin APIs by any chance 

Re: [DISCUSS] 2017 October release planning and release version

[Stevo Slavić]

With 0.x version of project at least I found lots of unexpected painful
things acceptable.
Graduating from 0.*, with x.y.z semantic versioning IMO it should be
clearly communicated to community is there change in meaning, what's the
"SLO", what are the commitments, what change in every segment means.

E.g.
- APIs not labeled or labeled as stable
-- change in major version is only one that can break backward
compatibility (client APIs or behavior)
-- change in minor version can introduce new features, but not break
backward compatibility
-- change in patch version, is for bug fixes only.
- APIs labeled as evolving can be broken in backward incompatible way in
any release, but are assumed less likely to be broken compared to unstable
APIs
- APIs labeled as unstable can be broken in backward incompatible way in
any release, major, minor or patch
- deprecated stable APIs are treated as any stable APIs, they can be
removed only in major release, are not allowed to be changed in backward
incompatible way in either patch or minor version release

This means one should be able to upgrade server and recompile/deploy apps
with clients to new minor.patch release with dependency version change
being only change needed and there would be no drama.

Practice/"features" like protocol version being a parameter, and defaulting
to latest so auto updated with dependency update which introduces new
protocol/behavior should not be used in public client APIs. To switch
between backward incompatible APIs (contract and behaviors), ideally user
should explicitly have to change code and not dependency only, but at least
it should be clearly communicated that there are breaking changes to expect
even with just dependency update by e.g. giving major version release clear
meaning. If app dependency on Kafka client library minor.patch on same
major is updated, and if there's a change in behavior or API requiring app
code change - it's a bug.

Change introduced contrary to the SLO, is OK to be reported as bug.
Everything else is improvement or feature request.

If this was the case, and 1.0.0 was released today with APIs as they are
now, Scala client APIs even though deprecated would not break and require
refactoring with every 1.* minor/patch release, and would only be allowed
to be broken or removed in future major release, like 2.0.0

It should be also clear how long is each version supported - e.g. if
minor.patch had meaning that there are no backward incompatible changes,
it's OK to file a bug only for current major.minor.patch; previous major
and its last minor.patch can only have patches released up to some time
like 1 up to 3 months.

If there are changes in release cadence with new versioning, it should be
clear too.

Kind regards,
Stevo Slavic.

On Wed, Jul 19, 2017 at 1:21 AM, Ismael Juma  wrote:

> With regards to the annotations, I think we should expect that we'll always
> have some @Evolving APIs. Even though much of the platform is mature, we'll
> continue to improve and extend it. I'm generally not a fan of @Unstable
> (since there's rarely a reason to make breaking changes in bug fix release)
> and I would not mind if we removed them from the codebase for good.
>
> Ismael
>
> On Tue, Jul 18, 2017 at 4:07 PM, Guozhang Wang  wrote:
>
> > Currently the only @unstable annotations left are on Streams and one
> class
> > of Security modules, and I think we have a good chance of removing them
> all
> > in the next release.
> >
> > We also have a few @evolving annotations on the Admin, Streams, Security
> > modules etc. And I think we can try to also eliminate as many of them as
> > possible if people feel confident about these APIs but maybe a stretch
> goal
> > to get rid of all of them.
> >
> > Guozhang
> >
> > On Tue, Jul 18, 2017 at 3:49 PM, Gwen Shapira  wrote:
> >
> > > Also fine with the change in general.
> > >
> > > As you mentioned, 1.x indicates mature APIs, compatibility and
> stability.
> > > Are we going to remove the @unstable annotations in this release?
> > >
> > > Gwen
> > >
> > > On Tue, Jul 18, 2017 at 3:43 PM Ismael Juma  wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > > Thanks for volunteering to be the release manager for the next
> release!
> > > >
> > > > I am +1 on naming the next release 1.0.0. As you said, Kafka is
> mature
> > > > enough and this will make it easier for others to understand our
> > > versioning
> > > > policy.
> > > >
> > > > A couple of minor questions inline.
> > > >
> > > > On Tue, Jul 18, 2017 at 3:36 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > major.minor.bugfix[.release-candidate]
> > > > >
> > > >
> > > > I think you mean major.minor.bugfix-rc (i.e. we typically use a dash
> > > > instead of dot for the RCx qualifier).
> > > >
> > > > >
> > > > > How do people feel about 1.0.0.x as the next Kafka version?
> > > >
> > > >
> > > > Do you mean 1.0.0?
> > > >
> > > > Ismael
>