Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

Hi Konstantine, b) I thought hard but briefly about this. Although there's potential for > ambiguity if someone inspects the key on its own, I believe that if we > combine this information with what we save in the value, the ambiguity is > removed. With respect to whether a connector can override

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Great to hear and well deserved! Congratulations and thanks to all of you! On Thu, 16 Jan 2020 at 07:44, Vahid Hashemian wrote: > Thank you all. > > Regards, > --Vahid > > On Wed, Jan 15, 2020 at 9:15 AM Colin McCabe wrote: > > > Thanks, everyone! > > > > best, > > Colin > > > > On Wed, Jan

Build failed in Jenkins: kafka-trunk-jdk11 #1085

See Changes: [jason] KAFKA-9410; Make groupId Optional in KafkaConsumer (#7943) -- [...truncated 5.75 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Thank you all. Manikumar On Thu, Jan 16, 2020 at 3:03 PM Sönke Liebau wrote: > Great to hear and well deserved! Congratulations and thanks to all of you! > > On Thu, 16 Jan 2020 at 07:44, Vahid Hashemian > wrote: > > > Thank you all. > > > > Regards, > > --Vahid > > > > On Wed, Jan 15, 2020

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

Vito, Thanks for the updates, the KIP LGTM. -Bill On Wed, Jan 15, 2020 at 11:31 PM John Roesler wrote: > Hi Vito, > > Haha, your archive game is on point! > > What Matthias said in that email is essentially what I figured was the > rationale. It makes sense, but the point I was making is that

Re: Role of CompositeReadOnlyKeyValueStore for point queries

Hey Navinder, I agree this is not ideal. The answer is that until KIP-535, you wouldn't necessarily know which partition you were querying, just the instance. Now, though, the information is front-and-center, and we should update the store() API to favor efficiently routing the query directly to

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

MIckael, On Thu, 16 Jan 2020 at 14:01, Mickael Maison wrote: > Hi Manna, > > In your example, the topic 'dummy' is not under replicated. It just > has 1 replica. A topic under replicated is a topic with less ISRs than > replicas. > > Having under replicated topics is relatively common in a

[jira] [Created] (KAFKA-9444) Cannot connect to zookeeper after updating the kafka config

Rishabh Bohra created KAFKA-9444: Summary: Cannot connect to zookeeper after updating the kafka config Key: KAFKA-9444 URL: https://issues.apache.org/jira/browse/KAFKA-9444 Project: Kafka

[jira] [Created] (KAFKA-9443) Producer Can Fail with NPE

David Mollitor created KAFKA-9443: - Summary: Producer Can Fail with NPE Key: KAFKA-9443 URL: https://issues.apache.org/jira/browse/KAFKA-9443 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

Hi David, Thanks for taking a look. 1) Yes, updated 2) I had not considered that but indeed this would be useful if the request contained multiple states and would avoid doing another call. The ListGroups response already includes the group ProtocolType, so I guess we could add the State as

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

Hi Manna, In your example, the topic 'dummy' is not under replicated. It just has 1 replica. A topic under replicated is a topic with less ISRs than replicas. Having under replicated topics is relatively common in a Kafka cluster, it happens everytime is broker is down. However Kafka does not

Re: [VOTE] KIP-216: IQ should throw different exceptions for different errors

Thanks for the KIP. +1 (binding) -Bill On Tue, Jan 14, 2020 at 9:41 AM Navinder Brar wrote: > +1 (non-binding) With a small comment which was mentioned by Vinoth as > well. Did we fix on the flag for StreamsRebalancingException, I don't see > it in the KIP. > -Navinder > > > On Tuesday,

Re: Streams, Kafka windows

Hi John, Thanks for answering my questions! I observe behavior which I can not understand. The code is working, but when delay between records larger then window duration I receive duplicated records. With the code below I received duplicated records in the output kstream. Count of duplicate

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

> Nevertheless, if we omit the categorization, it’s moot. Ack. I am fine to remove the middle tier. As John pointed out, it might be weird to have only one concrete exception type per category. We can also explain in detail how to handle each exception in their JavaDocs. -Matthias On 1/16/20

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

Thanks for the follow up Tom. Replies inline below this time: On Thu, Jan 16, 2020 at 1:16 AM Tom Bentley wrote: > Hi Konstantine, > > b) I thought hard but briefly about this. Although there's potential for > > ambiguity if someone inspects the key on its own, I believe that if we > > combine

Re: Role of CompositeReadOnlyKeyValueStore for point queries

Thanks John for the prompt response. Yeah, store(int partition) is exactly what I meant. Sorry for the confusion. Since we have already merged PR:7962 I can take the new trunk and start working on the KIP in the meantime. I agree that we should replace the newly added param. It would be a good

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

Hi Colin, That’s fair though I am unsure if a delay + metric + log message would really serve our purpose. There would be no action required from the operator in almost all cases. A signal that is not actionable in 99% cases may not be very useful, in my opinion. Additionally, if we add in a

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

Hi Rajini. Thanks for the feedback. I have now come around to the idea of abandoning the ZooKeeper camelCase style in favor of being consistent with the existing Kafka broker configs. Here's what I've done: #1) ZK style vs. Kafka style I've updated the KIP to convert all of the following

[jira] [Created] (KAFKA-9446) Integration test library should provide utilities to assert connector state

Chris Egerton created KAFKA-9446: Summary: Integration test library should provide utilities to assert connector state Key: KAFKA-9446 URL: https://issues.apache.org/jira/browse/KAFKA-9446 Project:

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

Hi, Viktor, Thanks for the KIP. +1 from me. Just a couple of minor comments below. 100. CreateDelegationTokenResponse/DescribeDelegationTokenResponse. It seems that "validVersions" should be "0-2". 101. The option --users "owner1,owner2" in AclCommand. Since SSL user name can include comma,

Re: Streams, Kafka windows

Hi Viktor, I’m starting to wonder what exactly “duplicate” means in this context. Can you elaborate? In case it helps, with your window definition, if I send a record with timestamp 20, it would actually belong to three different windows: [0,30) [10,40) [20,50) Because of this, you would

[jira] [Created] (KAFKA-9445) Allow fetching a key from a single partition rather than iterating over all the stores on an instance

Navinder Brar created KAFKA-9445: Summary: Allow fetching a key from a single partition rather than iterating over all the stores on an instance Key: KAFKA-9445 URL:

KIP-560 Discuss

Hello, Starting this thread to discuss KIP-560: wiki link : https://cwiki.apache.org/confluence/display/KAFKA/KIP-560%3A+Auto+infer+external+topic+partitions+in+stream+reset+tool I'm newbie I would like to receive feedback on the following features!

subject KIP-560

hello! I'm newbie I would like to receive feedback on the following features! https://cwiki.apache.org/confluence/display/KAFKA/KIP-560%3A+Auto+infer+external+topic+partitions+in+stream+reset+tool thank you!

[jira] [Created] (KAFKA-9448) Meta data for topic

Pradeep Bansal created KAFKA-9448: - Summary: Meta data for topic Key: KAFKA-9448 URL: https://issues.apache.org/jira/browse/KAFKA-9448 Project: Kafka Issue Type: Improvement

Re: [VOTE] KIP-560: Auto infer external topic partitions in stream reset tool

Hey Sang, thanks so much for driving this effort. I have just a couple of suggestions and questions: 1. In the Motivation part, would be better to phrase "user has to specify `--input-topic` in the stream reset to be able to reset offset to a specific position" as "user has to specify input

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration value that turns it on is zookeeper.ssl.client.enable (which used to be called zookeeper.client.secure before we renamed everything to be consistent with Kafka's config naming style and eliminate confusion). The default value

[jira] [Resolved] (KAFKA-9356) Potential data loss in InMemoryWindowStore and InMemorySessionStore

[ https://issues.apache.org/jira/browse/KAFKA-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman resolved KAFKA-9356. Resolution: Not A Problem > Potential data loss in InMemoryWindowStore and

[jira] [Created] (KAFKA-9447) Add examples for EOS standalone and group mode under kafka/examples

Boyang Chen created KAFKA-9447: -- Summary: Add examples for EOS standalone and group mode under kafka/examples Key: KAFKA-9447 URL: https://issues.apache.org/jira/browse/KAFKA-9447 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk11 #1088

See Changes: [jason] MINOR: Remove unnecessary call to `super` in `MetricConfig` constructor -- Started by an SCM change Running as SYSTEM [EnvInject] - Loading node

Build failed in Jenkins: kafka-trunk-jdk11 #1087

See Changes: [github] MINOR: fix flaky StreamsUpgradeTestIntegrationTest (#7974) -- [...truncated 5.75 MB...] org.apache.kafka.streams.test.ConsumerRecordFactoryTest >

Build failed in Jenkins: kafka-2.4-jdk8 #127

See Changes: [jason] KAFKA-9235; Ensure transaction coordinator is stopped after replica -- [...truncated 5.48 MB...] org.apache.kafka.streams.test.OutputVerifierTest >

[jira] [Resolved] (KAFKA-6144) Allow serving interactive queries from in-sync Standbys

[ https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-6144. - Resolution: Fixed > Allow serving interactive queries from in-sync Standbys >

[jira] [Resolved] (KAFKA-9431) Expose API in KafkaStreams to fetch all local offset lags

[ https://issues.apache.org/jira/browse/KAFKA-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-9431. - Resolution: Fixed > Expose API in KafkaStreams to fetch all local offset lags >

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: > Hi Ron/Colin, > > Thanks for the responses. I think we need to consider these cases: > > 1) The recommended approach: This is the one we document. Configs > consistent with Kafka make sense here. It seems a lot more confusing to > configure

[jira] [Resolved] (KAFKA-9235) Transaction state not cleaned up following StopReplica request

[ https://issues.apache.org/jira/browse/KAFKA-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-9235. Fix Version/s: 2.4.1 Resolution: Fixed > Transaction state not cleaned up following

Re: [VOTE] On the new KIP-158: Kafka Connect allows source connectors to set topic settings when creating new topics

Thanks everyone. Having completed three full days since the vote was opened, I'll go ahead and conclude voting on KIP-158. The KIP is being adopted, with: 3 binding +1 votes (Randall Hauch, Bill Bejeck and Gwen Shapira) 1 non-binding +1 vote (Tom Bentley) Best regards, Konstantine On Wed, Jan

Re: [VOTE] KIP-551: Expose disk read and write metrics

With binding +1 votes from Gwen Shapira, Manikumar Reddy, Mickael Maison, and David Arthur, and non-binding +1 votes from Jose Garcia Sancio, M. Manna, Lucas Bradstreet, Mitchell, and Sönke Liebau, the vote passes. Thanks, all! Colin On Wed, Jan 15, 2020, at 09:25, Colin McCabe wrote: >

[jira] [Resolved] (KAFKA-9428) Expose standby information in KafkaStreams via queryMetadataForKey API

[ https://issues.apache.org/jira/browse/KAFKA-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-9428. - Resolution: Fixed > Expose standby information in KafkaStreams via queryMetadataForKey API >

[VOTE] KIP-560: Auto infer external topic partitions in stream reset tool

this KIP, I would like to go forward with the voting process. Jira Ticket: https://issues.apache.org/jira/browse/KAFKA-9042 KIP : https://cwiki.apache.org/confluence/display/KAFKA/KIP-560%3A+Auto+infer+external+topic+partitions+in+stream+reset+tool thankyou!

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

On Thu, Jan 16, 2020, at 10:29, Dhruvil Shah wrote: > Hi Colin, > > That’s fair though I am unsure if a delay + metric + log message would > really serve our purpose. There would be no action required from the > operator in almost all cases. A signal that is not actionable in 99% cases > may not

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Congratulations everyone! On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira wrote: > Hi everyone, > > I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar > Reddy are now members of Apache Kafka PMC. > > Colin and Manikumar became committers on Sept 2018 and Vahid on Jan > 2019.

Build failed in Jenkins: kafka-trunk-jdk8 #4164

See Changes: [github] KAFKA-9235; Ensure transaction coordinator is stopped after replica -- [...truncated 5.71 MB...]

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

Thanks for new comments Randall. Following up with my replies inline below. I'll also go ahead and update the KIP with the suggestions that are outstanding right now and post a summary of the changes. On Wed, Jan 15, 2020 at 2:37 PM Randall Hauch wrote: > My responses are inline: > > On Wed,

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

Thanks Bill, John and Matthias. Glad you guys joined this discussion. I got a lot out of the discussion. I would like to update KIP-216 base on John's suggestion to remove the category. --- Vito On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax wrote: > > Nevertheless, if we omit the

[jira] [Resolved] (KAFKA-8832) We should limit the maximum size read by a fetch request on the kafka server.

[ https://issues.apache.org/jira/browse/KAFKA-8832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-8832. Resolution: Fixed This is resolved by

[jira] [Resolved] (KAFKA-9429) Allow ability to control whether stale reads out of state stores are desirable

[ https://issues.apache.org/jira/browse/KAFKA-9429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-9429. - Resolution: Fixed > Allow ability to control whether stale reads out of state stores are

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

Hi Jun, Can the SSL username really include the comma? >From what I could tell, when I searched it up, I couldn't find anything that indicated comma can be a delimiter. A related doc below: https://knowledge.digicert.com/solution/SO12401.html Cheers, Richard On Thu, Jan 16, 2020 at 1:37 PM

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

Thanks for your patience, Vito! -John On Thu, Jan 16, 2020, at 19:12, Vito Jeng wrote: > Thanks Bill, John and Matthias. Glad you guys joined this discussion. > I got a lot out of the discussion. > > I would like to update KIP-216 base on John's suggestion to remove the > category. > > > --- >

Jenkins build is back to normal : kafka-trunk-jdk11 #1086

See

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

Hi Ron/Colin, Thanks for the responses. I think we need to consider these cases: 1) The recommended approach: This is the one we document. Configs consistent with Kafka make sense here. It seems a lot more confusing to configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

Forgot to mention: Overriding system property with a differently named Kafka configuration option is something we already do for JAAS configs - we default to System property `java.security.auth.login.config` if `sasl.jaas.config` is not configured. On Thu, Jan 16, 2020 at 10:58 AM Rajini Sivaram

Build failed in Jenkins: kafka-trunk-jdk8 #4163

See Changes: [jason] KAFKA-9410; Make groupId Optional in KafkaConsumer (#7943) -- [...truncated 2.83 MB...] org.apache.kafka.streams.test.ConsumerRecordFactoryTest >

[DISCUSS] KIP-561: Regex Expressions Support for ConsumerGroupCommand

Hello, guys, Please review Regex Expressions Support for ConsumerGroupCommand improvement proposal - *Previous Discussion 1*: Re: Multiple Consumer Group Management - *Previous Discussion 2*: Re: ConsumerGroupCommand tool