Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

Hi Jason, Thanks for the written KIP. I think this is going to be a very useful tool for operational improvements since with eos in its current stage, we cannot confidently assert that we are bug-free, and even in the future when we are confident this is still going to be leveraged by older

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

Thanks Ning for the KIP. Having stronger guarantees when mirroring data would be a nice improvement! A few comments: 1. How does offset translation work with this new sink connector? Should we also include a CheckpointSinkConnector? 2. Migrating to this new connector could be tricky as

[jira] [Created] (KAFKA-10441) Connect consumers are missing the fetch metrics

Pradyumna created KAFKA-10441: - Summary: Connect consumers are missing the fetch metrics Key: KAFKA-10441 URL: https://issues.apache.org/jira/browse/KAFKA-10441 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-10440) MM2 using Kconnect Framework Creates topics recursively on source cluster

Nandini Anagondi created KAFKA-10440: Summary: MM2 using Kconnect Framework Creates topics recursively on source cluster Key: KAFKA-10440 URL: https://issues.apache.org/jira/browse/KAFKA-10440

Re: [VOTE] KIP-635: GetOffsetShell: support for multiple topics and consumer configuration override

Hi all, Please vote if you'd like to see this implemented. This one fixes a long-time debt, would be nice to see it pass. Thank you, Daniel Dániel Urbán ezt írta (időpont: 2020. aug. 18., K, 14:06): > Hello everyone, > > Please, if you are interested in this KIP and PR, don't forget to vote.

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Hi Wlaker (pun intended), Thank you for your feedback! Please find my answers inline. Best, Bruno On 26.08.20 18:46, Walker Carlson wrote: Hello Burno, Thanks for the KIP! Not to pile on, but I also had a couple additional questions. I am not super familiar with the StreamThread internals

[jira] [Created] (KAFKA-10442) Tooling to detect and abort hanging transactions (KIP-664)

Jason Gustafson created KAFKA-10442: --- Summary: Tooling to detect and abort hanging transactions (KIP-664) Key: KAFKA-10442 URL: https://issues.apache.org/jira/browse/KAFKA-10442 Project: Kafka

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Hi John, Thank you for your feedback! Please find my answers inline. Best, Bruno On 26.08.20 17:49, John Roesler wrote: Hi Bruno, Thanks for the well motivated and throrough KIP! It's a good point that the record cache should be re- distributed over the threads. Reading your KIP leads me

[DISCUSS] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

Hello all, I would like to propose the following KIP to throw a new non-fatal exception whilst aborting transactions with non-flushed data. This will help users distinguish non-fatal errors and potentially retry the batch. *Issue *- https://issues.apache.org/jira/browse/KAFKA-10186

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Thank you Matthias for the feedback. Please find my answers inline. Best, Bruno On 26.08.20 19:54, Matthias J. Sax wrote: Thanks for the KIP Bruno. While reading it, I had the same questions as raised by John and Walker (so I won't repeat them). In addition, I think that adding/removing

Re: [VOTE] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

+1 (Non Binding). Good Kip Bruno Walker On Tue, Aug 25, 2020 at 11:17 AM Guozhang Wang wrote: > +1. Thanks Bruno! > > > Guozhang > > On Tue, Aug 25, 2020 at 4:00 AM Bruno Cadonna wrote: > > > Hi, > > > > I would like to start the vote for > > > > > > >

Re: [VOTE] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

Thanks for the KIP Bruno. +1 (binding) -Bill On Thu, Aug 27, 2020 at 3:15 PM Walker Carlson wrote: > +1 (Non Binding). Good Kip Bruno > > Walker > > On Tue, Aug 25, 2020 at 11:17 AM Guozhang Wang wrote: > > > +1. Thanks Bruno! > > > > > > Guozhang > > > > On Tue, Aug 25, 2020 at 4:00 AM

[jira] [Created] (KAFKA-10443) Consider providing standard set of users in system tests

Ron Dagostino created KAFKA-10443: - Summary: Consider providing standard set of users in system tests Key: KAFKA-10443 URL: https://issues.apache.org/jira/browse/KAFKA-10443 Project: Kafka

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

Hi Jason, This looks like a very useful tool, thanks for writing it up. Given that it's possible for replica producer states to diverge from each other, it would be very useful if DescribeProducers(Request,Response) and tooling is able to query all partition replicas for their producers. One way

Re: Access to submit KIP

Hi, Mandeep, Thanks for your interest. Just gave you the wiki permissions. Jun On Wed, Aug 26, 2020 at 9:08 PM mandeep gandhi wrote: > Hi all, > > Please grant me access to create a KIP. > > Email - welcomemand...@gmail.com > Username - ifconfig > > Thanks, > Mandeep Gandhi >

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

Hey Lucas, Thanks for the comments. Responses below: > Given that it's possible for replica producer states to diverge from each other, it would be very useful if DescribeProducers(Request,Response) and tooling is able to query all partition replicas for their producers Yes, it makes sense to

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

>> Would it be worth returning transactional.id.expiration.ms in the DescribeProducersResponse? > That's an interesting thought as well. Are you trying to avoid the need to specify it through the command line? The tool could also query the value with DescribeConfigs I suppose. Basically. I'm not

Re: [VOTE] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

+1 (binding) On 8/27/20 1:10 PM, John Roesler wrote: > Thanks, Bruno! > > I'm +1 (binding) > > -John > > On Thu, 2020-08-27 at 15:35 -0400, Bill Bejeck wrote: >> Thanks for the KIP Bruno. >> >> +1 (binding) >> >> -Bill >> >> On Thu, Aug 27, 2020 at 3:15 PM Walker Carlson >> wrote: >> >>> +1

Re: [DISCUSS] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

Thanks for the KIP. Looks good overall. However, I am wondering if the new exception should extend `KafkaException`? It seems, extending `ApiException` or maybe even `RetriableException` might be better? About the name itself. I would prefer something simpler like `AbortedTransactionException`.

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #35

See Changes: [github] KAFKA-9924: Add RocksDB metric num-entries-active-mem-table (#9177) -- [...truncated 3.22 MB...]

Re: [DISCUSS] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

Hey Gokul, thanks for taking up this KIP! I agree with Matthias that directly extending KafkaException may not be ideal, and we should instead extend APIException or RetriableException. Of the two, I think APIException would be more appropriate. My understanding is that RetriableException is

Re: [DISCUSS] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Ok I'm definitely feeling pretty dumb now, but I was just thinking how ridiculous it is that the Consumer forces you to configure your Deserializer through actual config maps instead of just taking the ones you pass in directly. So I thought "why not just fix the Consumer to allow passing in an

[GitHub] [kafka-site] scott-confluent opened a new pull request #300: [MINOR] adding Itau Unibanco and OTICS to the powered-by page

scott-confluent opened a new pull request #300: URL: https://github.com/apache/kafka-site/pull/300 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

Hello Mickael, > 1. How does offset translation work with this new sink connector? > Should we also include a CheckpointSinkConnector? CheckpointSourceConnector will be re-used as the same as current. When EOS is enabled, we will run 3 connectors: MirrorSinkConnector (based on SinkConnector)

Requesting permission

I’d like to request permission to create a KIP. My username is brbrown35. Thanks, Brandon Brandon Brown (240) 506-8335 (m)

Re: Requesting permission

Hi, Brandon, Thanks for your interest. Just gave you the wiki permission. Jun On Thu, Aug 27, 2020 at 1:17 PM Brandon Brown wrote: > I’d like to request permission to create a KIP. My username is brbrown35. > > Thanks, > Brandon > > Brandon Brown > (240) 506-8335 (m)

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

Hi Boyang, Thanks for the comments. Responses below: > 1. For the analysis section, is there any consistency guarantee for `ListTransactions` and `DescribeTransactions`? Let's say the coordinator receives a DescribeTransactions while the transaction is almost complete at the same time, should we

Re: [VOTE] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

Thanks, Bruno! I'm +1 (binding) -John On Thu, 2020-08-27 at 15:35 -0400, Bill Bejeck wrote: > Thanks for the KIP Bruno. > > +1 (binding) > > -Bill > > On Thu, Aug 27, 2020 at 3:15 PM Walker Carlson > wrote: > > > +1 (Non Binding). Good Kip Bruno > > > > Walker > > > > On Tue, Aug 25,

Re: Requesting permission

Thanks! I was able to make my first KIP. -Brandon Brandon Brown (240) 506-8335 (m) > On Aug 27, 2020, at 4:19 PM, Jun Rao wrote: > > Hi, Brandon, > > Thanks for your interest. Just gave you the wiki permission. > > Jun > >> On Thu, Aug 27, 2020 at 1:17 PM Brandon Brown >> wrote: >> >>

[DISCUSS] KIP-665 Kafka Connect Hash SMT

https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT The current pr with the proposed changes https://github.com/apache/kafka/pull/9057 and the original 3rd party contribution which initiated this change

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

Thanks Jason for the tooling proposal. A couple of comments: 1. For the analysis section, is there any consistency guarantee for `ListTransactions` and `DescribeTransactions`? Let's say the coordinator receives a DescribeTransactions while the transaction is almost complete at the same time,

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #37

See

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #36

See Changes: [github] KAFKA-9924: Add RocksDB metric num-entries-active-mem-table (#9177) -- [...truncated 6.48 MB...]