[jira] [Resolved] (KAFKA-7226) kafka-console-consumer.sh doesn't use security.protocol provided in config file

[ https://issues.apache.org/jira/browse/KAFKA-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-7226. -- Resolution: Information Provided Closing the issue related to HDP Kafka distribution. >

[jira] [Resolved] (KAFKA-7083) maxTickMessages in ConsumerGroup option

[ https://issues.apache.org/jira/browse/KAFKA-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-7083. -- Resolution: Information Provided Closing the issue related to node.js client. Please contact client

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hey Vahid, I think we're in agreement. Perhaps since none of us feel too strongly, we should go for the more minimal change? I'll vote for the KIP either way as long as we can change the default group.id. Thanks, Jason On Thu, Aug 2, 2018 at 11:55 AM, Vahid S Hashemian <

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

Hey Kevin, Thanks for the KIP. I like Mickael's suggestion to add --under-minisr-partitions since it fits with the metric we already expose. It's also a good question whether there should be a separate category for partitions which are right at min.isr. I'm reluctant to add new categories, but I

Build failed in Jenkins: kafka-trunk-jdk8 #2858

See Changes: [github] KAFKA-3514: Part II, Choose tasks with data on all partitions to process -- [...truncated 2.47 MB...] org.apache.kafka.streams.TopologyTest >

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

Thanks John! +1 On Mon, 30 Jul 2018 at 23:58 Guozhang Wang wrote: > Yes, the addendum lgtm as well. Thanks! > > On Mon, Jul 30, 2018 at 3:34 PM, John Roesler wrote: > > > Another thing that came up after I started working on an implementation > is > > that in addition to deprecating

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

Hi Manikumar, +1 (binding) Thanks for the KIP! On Fri, Aug 3, 2018 at 3:46 AM, Ted Yu wrote: > +1 > > On Thu, Aug 2, 2018 at 7:33 PM Manikumar > wrote: > > > Hi All, > > > > I would like to start voting on KIP-332 which allows AclCommand to use > > AdminClient API for acl management. > > > >

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Updated the KIP. added a note to KIP-340 discussion thread. On Fri, Aug 3, 2018 at 2:52 PM Rajini Sivaram wrote: > Thanks Manikumar. Can you add a note to the KIP-340 discussion thread? > > On Fri, Aug 3, 2018 at 10:04 AM, Manikumar > wrote: > > > Hi Rajini, > > > > Yes, I too prefer using

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hi Manikumar, We have some tools already (ConfigCommand, ConsumerGroupCommand, DelegationTokenCommand) which use "--command-config" option to specify config file. Perhaps use should use the same name for AclCommand as well? On Thu, Aug 2, 2018 at 7:23 PM, Colin McCabe wrote: > +1 for starting

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hi, I also believe the minimal change might be better. In my opinion, the less logic we can stuff in the broker - the better. The use ofempty group.id is not ideal and should be discouraged, but I don't inherently believe strict validation in the broker is useful, since it doesn't break anything

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

Hi Dong, In KIP-332 discussion, It was agreed to use "--command-config" option name for passing config property file. We can also use same name in here, to make it consistent across all tools. Thanks, On Thu, Jul 12, 2018 at 9:20 AM Dong Lin wrote: > wiki page is currently read-only and it

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hi Rajini, Yes, I too prefer using "--command-config" . Recently in one of the other KIPs (KIP-340), it was suggested to use "--config-file", So I just followed the recent suggestion. But I think we should use same name in all tools (at least in core tools). If there are no concerns, I will

Re: [VOTE] KIP-342 - Add support for custom SASL extensions in OAuthBearer authentication

Thank you, Jun. Noted in the KIP. On Fri, Aug 3, 2018 at 2:28 AM Jun Rao wrote: > Hi, Stanislav, > > Thanks for the KIP. +1 > > Just one minor comment. Since the JWT token supports customizable claim > fields, it would be useful to clarify when to use the SASL extension vs the > customized

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Thanks Manikumar. Can you add a note to the KIP-340 discussion thread? On Fri, Aug 3, 2018 at 10:04 AM, Manikumar wrote: > Hi Rajini, > > Yes, I too prefer using "--command-config" . Recently in one of the other > KIPs (KIP-340), it was suggested to use > "--config-file", So I just followed

[jira] [Created] (KAFKA-7241) Reassignment partitiona to non existent broker

Igor Martemyanov created KAFKA-7241: --- Summary: Reassignment partitiona to non existent broker Key: KAFKA-7241 URL: https://issues.apache.org/jira/browse/KAFKA-7241 Project: Kafka Issue

[jira] [Created] (KAFKA-7242) Externalized secrets are revealed in task configuration

Bahdan Siamionau created KAFKA-7242: --- Summary: Externalized secrets are revealed in task configuration Key: KAFKA-7242 URL: https://issues.apache.org/jira/browse/KAFKA-7242 Project: Kafka

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hi Jason, Stanislav, Thanks for your feedback. I'll update the KIP later today based on your suggestion. I'll send a note once the update is complete. Thanks. --Vahid From: Stanislav Kozlovski To: dev@kafka.apache.org Date: 08/03/2018 02:34 AM Subject:Re: [DISCUSS] KIP-289:

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hey All, I think I've addressed all pending review. If there is no additional feedback, I'll plan to start a vote thread next week. Thanks, Jason On Tue, Jul 31, 2018 at 9:46 AM, Dong Lin wrote: > Hey Jason, > > Thanks for your reply. I will comment below. > > Regarding 1, we probably can not

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hi Dong, I don't have a preference for the option itself, but we are using " --command-config" in ConfigCommand, ConsumerGroupCommand, BrokerApiVersionsCommand, DeleteRecordsCommand. So it feels like we should use the same here. I think we are currently only using "--config-file" StreamsResetter.

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hey Rajini, Manikumar, Currently kafka-streams-application-reset.sh uses "--config-file" and a few other tools uses "--command-config". So the config name will be inconsistent no matter which config name we choose. Not sure we currently distinquish between core commands and non-core commands. I

[jira] [Resolved] (KAFKA-4479) Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator

[ https://issues.apache.org/jira/browse/KAFKA-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-4479. -- Resolution: Won't Fix > Streams tests should pass without hardcoded Time.SYSTEM in

[jira] [Resolved] (KAFKA-3577) Partial cluster breakdown

[ https://issues.apache.org/jira/browse/KAFKA-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang resolved KAFKA-3577. --- Resolution: Duplicate > Partial cluster breakdown > - > >

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

+1 On 8/3/18, 2:13 PM, "Ted Yu" wrote: +1 On Fri, Aug 3, 2018 at 10:40 AM Jason Gustafson wrote: > +1 Thanks Vahid. > > On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com > > wrote: > > > Hi everyone, > > > > I

Re: KIP-352: Distinguish URPs caused by reassignment

Hey Ismael, Yeah, my initial inclination was to redefine URP as well. My only doubt was how it would affect existing tools which might depend on URPs to track the progress of a reassignment. I decided to be conservative in the end, but I'd reconsider if we think it is not a major concern. It is

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

+1 Thanks Vahid. On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian wrote: > Hi everyone, > > I believe the feedback on this KIP has been addressed so far. So I'd like > to start a vote. > The KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

[jira] [Resolved] (KAFKA-3094) Kafka process 100% CPU when no message in topic

[ https://issues.apache.org/jira/browse/KAFKA-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang resolved KAFKA-3094. --- Resolution: Cannot Reproduce > Kafka process 100% CPU when no message in topic >

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

+1 On Fri, Aug 3, 2018 at 10:40 AM Jason Gustafson wrote: > +1 Thanks Vahid. > > On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com > > wrote: > > > Hi everyone, > > > > I believe the feedback on this KIP has been addressed so far. So I'd like > > to start a vote. >

[jira] [Created] (KAFKA-7243) Add unit integration tests to validate metrics in Kafka Streams

Guozhang Wang created KAFKA-7243: Summary: Add unit integration tests to validate metrics in Kafka Streams Key: KAFKA-7243 URL: https://issues.apache.org/jira/browse/KAFKA-7243 Project: Kafka

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, It seems like there's already some confusion about the roles of registry id vs member id. My understanding is that the protocol is currently that the broker assigns a unique "member id" to each consumer in a group, and those consumers must include this id in each subsequent request.

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

Thanks Manikumar! +1 On Fri, Aug 3, 2018 at 4:33 PM, Jason Gustafson wrote: > +1 Thanks Manikumar! > > On Fri, Aug 3, 2018 at 4:24 PM, Colin McCabe wrote: > > > +1 (non-binding) > > > > regards, > > Colin > > > > On Fri, Aug 3, 2018, at 02:27, Rajini Sivaram wrote: > > > Hi Manikumar, > > > >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

@Boyang, * Yes registry id should be generated via a config, similar to what you proposed in the KIP currently for member id. And "even if multiple consumers are using the same member id" (I think you meant registry id here?), we can fence against them. BTW I'm not married (but cannot speak for

Build failed in Jenkins: kafka-trunk-jdk10 #364

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H31 (ubuntu xenial) in workspace

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hi, Jason, Thanks for the updated KIP. Well thought-through. Just a few minor comments below. 50. For seek(TopicPartition partition, OffsetAndMetadata offset), I guess under the cover, it will make OffsetsForLeaderEpoch request to determine if the seeked offset is still valid before fetching? If

Re: [DISCUSS] KIP-353: Allow Users to Configure Kafka Streams Timestamp Synchronization

Hello Ted, I think dynamic configuration itself would worth an independent KIP, if you meant to allow users changing the config on-the-fly while not bouncing the instance. Guozhang On Fri, Aug 3, 2018 at 3:33 PM, Ted Yu wrote: > Guozhang: > > Do you plan to support dynamic config for the new

Jenkins build is back to normal : kafka-2.0-jdk8 #102

See

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Thanks for the KIP. One general question: The Jira is marked as "stream", so I am wondering what the intended scope the KIP is because, it suggests a new consumer API only. Can you clarify? -Matthias On 8/3/18 1:13 PM, Afshartous, Nick wrote: > > Hi All, > > This message is to initiate

Re: KIP-352: Distinguish URPs caused by reassignment

@Dong Ugh, you are right. This is indeed trickier than I imagined. You could argue that the problem here is that there is no single source of truth for the expected replication factor. While a reassignment is in progress, the reassignment itself has the expected replica set. Otherwise, it is

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Here's hoping someone has time to pick up KIP-14 again: https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization. The inconsistencies are so irritating. -Jason On Fri, Aug 3, 2018 at 10:27 AM, Rajini Sivaram wrote: > Hi Dong, > > I don't have a preference for the option

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

Hi Stanislav, Thanks for the KIP. As as user, "FaultyRecordException" seems a little vague. What's faulty about it? Is it too big? Does it not match the schema of the other records? Was it not found? Of course, we know that the specific problem is with [de]serializing the record, not

Build failed in Jenkins: kafka-trunk-jdk10 #362

See Changes: [lindong28] KAFKA-6648; Fetcher.getTopicMetadata() should return all partitions for -- [...truncated 1.54 MB...] kafka.zookeeper.ZooKeeperClientTest >

[DISCUSS] KIP-349 Priorities for Source Topics

Hi All, This message is to initiate discussion on a feature to add optional priorities to source topics. Please review https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics Cheers, -- Nick

Build failed in Jenkins: kafka-trunk-jdk10 #363

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H31 (ubuntu xenial) in workspace

Re: [DISCUSS] KIP-350: Allow kafka-topics.sh to take brokerid as parameter to show partitions associated with it

Thanks for the KIP, Ratish. We should probably specify that it is an error when --broker is specified with operations other than --list, right? best, Colin On Wed, Aug 1, 2018, at 21:28, Ratish Ravindran wrote: > Hi, > > I would like to open a discussion thread on KIP-350: > >

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

+1 (non-binding) regards, Colin On Fri, Aug 3, 2018, at 02:27, Rajini Sivaram wrote: > Hi Manikumar, > > +1 (binding) > > Thanks for the KIP! > > On Fri, Aug 3, 2018 at 3:46 AM, Ted Yu wrote: > > > +1 > > > > On Thu, Aug 2, 2018 at 7:33 PM Manikumar > > wrote: > > > > > Hi All, > > > > > >

Re: [DISCUSS] KIP-353: Allow Users to Configure Kafka Streams Timestamp Synchronization

Guozhang: Do you plan to support dynamic config for the new config entry ? Cheers On Fri, Aug 3, 2018 at 2:00 PM Guozhang Wang wrote: > Hello all, > > I would like to kick off a discussion on the following KIP, to allow users > control when a task can be processed based on its buffered

Re: [VOTE] KIP-342 - Add support for custom SASL extensions in OAuthBearer authentication

+1 Thanks Stanislav! On Fri, Aug 3, 2018 at 2:13 AM, Stanislav Kozlovski wrote: > Thank you, Jun. Noted in the KIP. > > On Fri, Aug 3, 2018 at 2:28 AM Jun Rao wrote: > > > Hi, Stanislav, > > > > Thanks for the KIP. +1 > > > > Just one minor comment. Since the JWT token supports customizable

Moving build job Was: Build failed in Jenkins: kafka-trunk-jdk10 #364

Should the build job run on nodes with the {{Hadoop}} label ? Based on recent builds@a.o discussion about how small the disk capacity is on the general {{ubuntu}} label, I think we should try {{Hadoop}} labeled machines. Cheers On Fri, Aug 3, 2018 at 3:08 PM Apache Jenkins Server <

Re: [DISCUSS] KIP-348 Eliminate null from SourceTask#poll()

"No changes to public interface" doesn't seem accurate here. SourceTask#poll is a public interface, right? Kafka connectors that are out-of-tree would certainly break if we disallowed returning null from this method. However, reading the KIP more closely, it seems like both null and the empty

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

KIP 321 has passed. Here is the vote break down: Binding: - Matthias J. Sax - Guozhang Wong - Ewen Cheslack-Postava Non-Binding: - Ted Yu - Bill Bejeck - Damian Guy Thanks to all those who voted and provided feedback! Best, Nishanth Pradeep On Thu, Aug 2, 2018 at 12:42 PM

Re: [DISCUSS] KIP-353: Allow Users to Configure Kafka Streams Timestamp Synchronization

Guozhang: I agree. Probably note this on your KIP. Thanks On Fri, Aug 3, 2018 at 6:08 PM Guozhang Wang wrote: > Hello Ted, > > I think dynamic configuration itself would worth an independent KIP, if you > meant to allow users changing the config on-the-fly while not bouncing the > instance. >

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

Thanks Vahid! +1 On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian wrote: > Hi everyone, > > I believe the feedback on this KIP has been addressed so far. So I'd like > to start a vote. > The KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

Build failed in Jenkins: kafka-2.0-jdk8 #101

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H34 (ubuntu xenial) in workspace

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey Guozhang, do you think my understanding makes sense? I would also like to hear from Jason, Mike, John and everyone's opinion on this new approach. Thanks! From: Boyang Chen Sent: Thursday, August 2, 2018 1:41 PM To: dev@kafka.apache.org Subject: Re:

Re: KIP-352: Distinguish URPs caused by reassignment

Hey Jason, Thanks for the KIP! I have some questions below. The KIP defines "UnderSynchronized" as partitions which have an in-sync replica set that is smaller than the topic's replication factor. Can you clarify how we decide the topic's replication factor? Currently the possible sources

Re: KIP-352: Distinguish URPs caused by reassignment

On Fri, Aug 3, 2018 at 11:23 AM, Jason Gustafson wrote: > Hey Ismael, > > Yeah, my initial inclination was to redefine URP as well. My only doubt was > how it would affect existing tools which might depend on URPs to track the > progress of a reassignment. I decided to be conservative in the

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

The KIP has been updated. If it looks good and there are no further comments I'll start a vote early next week. Thanks. --Vahid From: "Vahid S Hashemian" To: dev@kafka.apache.org Date: 08/03/2018 08:19 AM Subject:Re: [DISCUSS] KIP-289: Improve the default group id behavior

[jira] [Reopened] (KAFKA-4479) Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator

[ https://issues.apache.org/jira/browse/KAFKA-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-4479: The code still passes Time.SYSTEM: {code} /* start group coordinator */ // Hardcode

Re: KIP-352: Distinguish URPs caused by reassignment

RIght, that was my thinking too. Ismael On Fri, Aug 3, 2018 at 12:04 PM Gwen Shapira wrote: > On Fri, Aug 3, 2018 at 11:23 AM, Jason Gustafson > wrote: > > > Hey Ismael, > > > > Yeah, my initial inclination was to redefine URP as well. My only doubt > was > > how it would affect existing

[jira] [Created] (KAFKA-7244) Add note about memory map kernel limits to documentation

Ray Chiang created KAFKA-7244: - Summary: Add note about memory map kernel limits to documentation Key: KAFKA-7244 URL: https://issues.apache.org/jira/browse/KAFKA-7244 Project: Kafka Issue Type:

[DISCUSS] KIP-353: Allow Users to Configure Kafka Streams Timestamp Synchronization

Hello all, I would like to kick off a discussion on the following KIP, to allow users control when a task can be processed based on its buffered records, and how the stream time of a task be advanced.

Jenkins build is back to normal : kafka-trunk-jdk8 #2859

See

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

+1 Thanks Manikumar! On Fri, Aug 3, 2018 at 4:24 PM, Colin McCabe wrote: > +1 (non-binding) > > regards, > Colin > > On Fri, Aug 3, 2018, at 02:27, Rajini Sivaram wrote: > > Hi Manikumar, > > > > +1 (binding) > > > > Thanks for the KIP! > > > > On Fri, Aug 3, 2018 at 3:46 AM, Ted Yu wrote: > >

[jira] [Resolved] (KAFKA-5153) KAFKA Cluster : 0.10.2.0 : Servers Getting disconnected : Service Impacting

[ https://issues.apache.org/jira/browse/KAFKA-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang resolved KAFKA-5153. --- Resolution: Information Provided Fix Version/s: 0.11.0.1 Upgrading fixed the problem based on

[jira] [Resolved] (KAFKA-6662) Consumer use offsetsForTimes() get offset return None.

[ https://issues.apache.org/jira/browse/KAFKA-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Wang resolved KAFKA-6662. -- Resolution: Not A Problem > Consumer use offsetsForTimes() get offset return None. >

Build failed in Jenkins: kafka-trunk-jdk8 #2860

See Changes: [mjsax] MINOR: Require final variables in Streams (#5452) -- [...truncated 2.47 MB...] org.apache.kafka.streams.TopologyTest >

[jira] [Created] (KAFKA-7245) Deprecate WindowStore#put(key, value)

Matthias J. Sax created KAFKA-7245: -- Summary: Deprecate WindowStore#put(key, value) Key: KAFKA-7245 URL: https://issues.apache.org/jira/browse/KAFKA-7245 Project: Kafka Issue Type:

Build failed in Jenkins: kafka-trunk-jdk10 #366

See Changes: [github] MINOR: Add Timer to simplify timeout bookkeeping and use it in the -- [...truncated 1.54 MB...]

Build failed in Jenkins: kafka-trunk-jdk10 #365

See Changes: [mjsax] MINOR: Require final variables in Streams (#5452) -- [...truncated 1.98 MB...] org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce