Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

I think this model is more confusing than it needs to be. We end up with 4 prefixes despite only have 3 types of consumers. We have prefixes for: "base", "main", "global", and "restore". However, we only instantiate consumers of type "main", "global", and "restore". Until now, we've only had two

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Thanks Guozhang! I already updated the pull request and KIP to deprecate getConsumerConfigs() function. Do you think we could move to a voting stage now? From: Guozhang Wang Sent: Thursday, April 5, 2018 9:52 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KI

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Cool, I think this argument makes sense. I will deprecate `getConsumerConfigs` instead of refactoring. From: Matthias J. Sax Sent: Thursday, April 5, 2018 9:06 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different con

[jira] [Created] (KAFKA-6747) kafka-streams Invalid transition attempted from state READY to state ABORTING_TRANSACTION

Ted Yu created KAFKA-6747: - Summary: kafka-streams Invalid transition attempted from state READY to state ABORTING_TRANSACTION Key: KAFKA-6747 URL: https://issues.apache.org/jira/browse/KAFKA-6747 Project: Ka

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

I agree that renaming the method in this case may not worth it. Let's keep the existing function names. On Wed, Apr 4, 2018 at 6:06 PM, Matthias J. Sax wrote: > Thanks for updating the KIP. > > One more comment. Even if we don't expect users to call > `StreamsConfig#getConsumerConfigs()` it is s

SSL Keymanager implementation using OS managed keystore

Hi, Running kafka on Windows OS, I would like to take advantage of windows managed keystore (eg: Windows-MY, can be loaded without a user, pass) .Eventually wants to use a custom keymanager implementation. >From the code I see, SSLFactory doesn't allow to use an existing keystore and >without

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

I was just reading the whole KIP for the first time. Nice work! One minor comment. In the table of the standalone consumer, the first line, first column says: > = Empty > (protocolType = Some("consumer")) I think this should be > = Empty > (protocolType != None) Note, that for example KafkaStr

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Thanks for updating the KIP. One more comment. Even if we don't expect users to call `StreamsConfig#getConsumerConfigs()` it is still public API. Thus, we cannot just rename the method. I think we have two options: either we keep the current name or we deprecate the method and introduce `getMainC

Re: [VOTE] KIP-274: Kafka Streams Skipped Records Metrics

+1 (binding) on the latest proposal. -Matthias On 4/2/18 12:10 PM, Richard Yu wrote: > +1 > > On Mon, Apr 2, 2018 at 8:42 AM, Guozhang Wang wrote: > >> +1 (binding). >> >> On Mon, Apr 2, 2018 at 7:22 AM, Ted Yu wrote: >> >>> +1 >>> >>> On Mon, Apr 2, 2018 at 7:11 AM, Bill Bejeck wrote: >>>

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

Thanks for updating the KIP. LGTM. -Matthias On 4/4/18 12:46 PM, John Roesler wrote: > Cool, if you're satisfied with the KIP now, maybe I can lobby for your vote > ;) > > The vote thread is still at only one binding +1, I think. > > Thanks, > -John > > On Tue, Apr 3, 2018 at 9:05 PM, Matthia

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Thanks Jun, That time also works for me. -john On Wed, Apr 4, 2018 at 6:28 PM, Guozhang Wang wrote: > @Jun, yeah that works for me too. > > @Jan, just to clarify on my previous email: assuming we do the reshuffling > out of the user input topics, Streams has the advantage that the > repartition

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

@Jun, yeah that works for me too. @Jan, just to clarify on my previous email: assuming we do the reshuffling out of the user input topics, Streams has the advantage that the repartition topic is purely owned by itself: the only producer writing to this repartition topic is Streams itself; so then

[jira] [Resolved] (KAFKA-6642) Rack aware task assignment in kafka streams

[ https://issues.apache.org/jira/browse/KAFKA-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-6642. Resolution: Duplicate I am closing this a duplicate of KAFKA-6718 -- let us know if you disa

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Thanks Jun! The time works for me. On Thu, 5 Apr 2018 at 4:34 AM Jun Rao wrote: > Hi, Jan, Dong, John, Guozhang, > > Perhaps it will be useful to have a KIP meeting to discuss this together as > a group. Would Apr. 9 (Monday) at 9:00am PDT work? If so, I will send out > an invite to the mailing

[jira] [Resolved] (KAFKA-6652) The controller should log failed attempts to transition a replica to OfflineReplica state if there is no leadership info

[ https://issues.apache.org/jira/browse/KAFKA-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lucas Wang resolved KAFKA-6652. --- Resolution: Won't Fix > The controller should log failed attempts to transition a replica to > Offlin

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Hi, Jan, Dong, John, Guozhang, Perhaps it will be useful to have a KIP meeting to discuss this together as a group. Would Apr. 9 (Monday) at 9:00am PDT work? If so, I will send out an invite to the mailing list. Thanks, Jun On Wed, Apr 4, 2018 at 1:25 AM, Jan Filipiak wrote: > Want to quickl

Re: Permissions to create a KIP

Done. On 4/4/18 6:44 AM, Luís Cabral wrote: > As advised on the proposed feature submission referenced below, I seem to > first need to create a KIP on your confluence page. > Could you kindly grant me the required privileges to create a KIP? > > Wiki ID: blaghed > > Thank you in advance! > >

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

Cool, if you're satisfied with the KIP now, maybe I can lobby for your vote ;) The vote thread is still at only one binding +1, I think. Thanks, -John On Tue, Apr 3, 2018 at 9:05 PM, Matthias J. Sax wrote: > Sounds great! > > The cryptic topic names can be an issue -- however, people can > `de

Re: [DISCUSS] KIP-257 - Configurable Quota Management

Sounds good to me Rajini. Good catch spotting this before it's included in a release. :) Ismael On Wed, Apr 4, 2018 at 11:13 AM, Rajini Sivaram wrote: > For compatibility reasons, we are now using Java rather than Scala for all > pluggable interfaces including those on the broker. There is alre

Support Cast of LogicalTypes to String

I've created a ticket - https://issues.apache.org/jira/browse/KAFKA-6684 and PR - https://github.com/apache/kafka/pull/4820 Could one of the maintainers take a look? Thanks!

Re: [DISCUSS] KIP-257 - Configurable Quota Management

For compatibility reasons, we are now using Java rather than Scala for all pluggable interfaces including those on the broker. There is already a KIP to move Authorizer to Java as well. As we will be removing support for Java 7 in the next release, we can also use default methods in Java when we ne

Permissions to create a KIP

Hi, As advised on the proposed feature submission referenced below, I seem to first need to create a KIP on your confluence page.Could you kindly grant me the required privileges to create a KIP? ID: blaghedEmail: Thank you in advance!Kind Regards,Luís Cabral Ref.:https://github.com/apache/kafk

Re: [DISCUSS] KIP-278: Add version option to Kafka's commands

Jason: Maybe your reply was intended for another KIP ? KIP-278 is about adding version option, not timeout. Cheers On Wed, Apr 4, 2018 at 9:36 AM, Jason Gustafson wrote: > Hi Sasaki, > > Thanks for the KIP. I think the timeout controls the maximum allowed time > that the consumer will block fo

[jira] [Created] (KAFKA-6746) Allow ZK Znode configurable for Kafka broker

Biju Nair created KAFKA-6746: Summary: Allow ZK Znode configurable for Kafka broker Key: KAFKA-6746 URL: https://issues.apache.org/jira/browse/KAFKA-6746 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-278: Add version option to Kafka's commands

Hi Sasaki, Thanks for the KIP. I think the timeout controls the maximum allowed time that the consumer will block for the next record. Maybe the meaning would be clearer with the more concise name `--timeout`? That also fits with the old consumer which overrides the `consumer.timeout.ms` property.

Re: [DISCUSS] KIP-277 - Fine Grained ACL for CreateTopics API

Hi Edo, Mickael, The intent of this KIP seems to be rather similar to KIP-231 (Improve the Required ACL of ListGroups API). The feedback I received on that KIP was to allow for backward compatibility, and, as a result, the Describe(Cluster) ACL was preserved; and a Describe(Group) ACL was intro

Permissions to create a KIP

As advised on the proposed feature submission "https://github.com/apache/kafka/pull/4822";, I seem to first need to create a KIP on your confluence page. Could you kindly grant me the required privileges to create a KIP? https://cwiki.apache.org/confluence/users/viewuserprofile.action?username=

Permissions to create a KIP

As advised on the proposed feature submission referenced below, I seem to first need to create a KIP on your confluence page. Could you kindly grant me the required privileges to create a KIP? Wiki ID: blaghed Thank you in advance! Ref.: https://github.com/apache/kafka/pull/4822

[DISCUSS] KIP-278: Add version option to Kafka's commands

Hello everyone, I would like to start a discussion for KIP 278. Cloud you please give comments and advice ? JIRA ticket and Pull Request are bellow:

Re: Permissions to edit KIP page / add pages

Done. On Wed, Apr 4, 2018 at 3:41 AM, Moshe Blumberg wrote: > https://cwiki.apache.org/confluence/display/KAFKA/ > Kafka+Improvement+Proposals > > Can you please add permissions to the following users: > > mine: > https://cwiki.apache.org/confluence/users/viewuserprofile.action? > username=mblum

Re: Permission request

Done. On Wed, Apr 4, 2018 at 2:04 AM, Alex Dunayevsky wrote: > Hello, > > I would like to add a KIP for kafka-consumer-perf-test according to this > PR: > https://github.com/apache/kafka/pull/4818 > (please check the details in comments) but do not have a permission to > create a page. > > Thank

Permission request

Hello, I would like to add a KIP for kafka-consumer-perf-test according to this PR: https://github.com/apache/kafka/pull/4818 (please check the details in comments) but do not have a permission to create a page. Thank you https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Propos

Permissions to edit KIP page / add pages

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals Can you please add permissions to the following users: mine: https://cwiki.apache.org/confluence/users/viewuserprofile.action?username=mblumberg a customer: https://cwiki.apache.org/confluence/users/viewuserprofile.acti

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Want to quickly step in here again because it is going places again. The last part of the discussion is just a pain to read and completely diverged from what I suggested without making the reasons clear to me. I don't know why this happens here are my comments anyway. @Guozhang: That Stre

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi, Thanks again for the feedback. Is there already ongoing work for having an own consensus implementation within Kafka? If that work haven't started yet, we think there is value in having an interim solution, that allows the use of another consensus system besides Zookeeper. We ask the communit