[ANNOUNCE] New committer: Chia-Ping Tsai

2020-10-19 Thread Guozhang Wang
Hello all, I'm happy to announce that Chia-Ping Tsai has accepted his invitation to become an Apache Kafka committer. Chia-Ping has been contributing to Kafka since March 2018 and has made 74 commits: https://github.com/apache/kafka/commits?author=chia7712 He's also authored several major

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

2020-10-19 Thread Guozhang Wang
Congratulations Sophie! Very well deserved. Guozhang On Mon, Oct 19, 2020 at 9:57 AM Tom Bentley wrote: > Congratulations Sophie! > > On Mon, Oct 19, 2020 at 5:55 PM Walker Carlson > wrote: > > > Congratulations Sophie! > > > > On Mon, Oct 19, 2020 at 9:43 AM Navinder Brar > > wrote: > > > >

Re: [ANNOUNCE] New committer: David Jacot

2020-10-16 Thread Guozhang Wang
Congrats David! Guozhang On Fri, Oct 16, 2020 at 10:23 AM Raymond Ng wrote: > Congrats David! > > Cheers, > Ray > > On Fri, Oct 16, 2020 at 10:08 AM Rajini Sivaram > wrote: > > > Congratulations, David! > > > > Regards, > > > > Rajini > > > > On Fri, Oct 16, 2020 at 5:45 PM Matthias J. Sax

[jira] [Resolved] (KAFKA-10613) Broker should not set leader epoch if the list-offset request version < 4

2020-10-15 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10613. --- Fix Version/s: 2.7.0 Resolution: Fixed > Broker should not set leader ep

[jira] [Created] (KAFKA-10614) Group coordinator onElection/onResignation should guard against leader epoch

2020-10-14 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10614: - Summary: Group coordinator onElection/onResignation should guard against leader epoch Key: KAFKA-10614 URL: https://issues.apache.org/jira/browse/KAFKA-10614

[jira] [Created] (KAFKA-10613) Broker should not set leader epoch if the list-offset request version < 4

2020-10-14 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10613: - Summary: Broker should not set leader epoch if the list-offset request version < 4 Key: KAFKA-10613 URL: https://issues.apache.org/jira/browse/KAFKA-10613 Proj

[jira] [Resolved] (KAFKA-10271) Performance regression while fetching a key from a single partition

2020-10-08 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10271. --- Fix Version/s: (was: 2.5.2) 2.7.0 Resolution: Fixed

[jira] [Resolved] (KAFKA-10362) When resuming Streams active task with EOS, the checkpoint file should be deleted

2020-10-07 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10362. --- Fix Version/s: 2.7.0 Resolution: Fixed > When resuming Streams active task with

[jira] [Resolved] (KAFKA-10122) Consumer should allow heartbeat during rebalance as well

2020-10-07 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10122. --- Fix Version/s: 2.6.1 2.7.0 Resolution: Fixed > Consumer sho

Re: [DISCUSSION] Upgrade system tests to python 3

2020-10-07 Thread Guozhang Wang
pull/9196 > >> > >> > >>> 21 сент. 2020 г., в 12:58, Nikolay Izhikov > >> написал(а): > >>> > >>> Hello. > >>> > >>> I fixed two system tests that fails in trunk, also. > >>> > >>>

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

2020-10-06 Thread Guozhang Wang
Sorry I'm late to the party. Matthias raised a point to me regarding the recent development of moving restoration from stream threads to separate restore threads and allowing the stream threads to process any processible tasks even when some other tasks are still being restored by the restore

[jira] [Created] (KAFKA-10577) StreamThread should be able to process any processible tasks regardless of its state

2020-10-05 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10577: - Summary: StreamThread should be able to process any processible tasks regardless of its state Key: KAFKA-10577 URL: https://issues.apache.org/jira/browse/KAFKA-10577

[jira] [Created] (KAFKA-10575) StateRestoreListener#onRestoreEnd should always be triggered

2020-10-05 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10575: - Summary: StateRestoreListener#onRestoreEnd should always be triggered Key: KAFKA-10575 URL: https://issues.apache.org/jira/browse/KAFKA-10575 Project: Kafka

Re: [DISCUSS] KIP-598: Augment TopologyDescription with store and source / sink serde information

2020-10-05 Thread Guozhang Wang
gt; > Or, if KIP-591 ever gets resurrected, maybe we will have a new store type > enum or > other public API to identify the stores that we can leverage here. But that > KIP seems > to have gone dormant as well :) > > On Fri, Oct 2, 2020 at 6:18 PM Guozhang Wang wro

Re: [VOTE] KIP-614: Add Prefix Scan support for State Stores

2020-10-05 Thread Guozhang Wang
tails, so you could just update the KIP > >> document to also mention, "We will also implement the new > >> method in the InMemoryKeyValueStore." > >> > >> Thanks for your contribution to Apache Kafka! > >> -John > >> > >> O

Re: [DISCUSS] KIP-598: Augment TopologyDescription with store and source / sink serde information

2020-10-02 Thread Guozhang Wang
escription? > > On Tue, May 19, 2020 at 1:22 PM Guozhang Wang wrote: > > > We already has a Serdes actually, which is a factory class. What we > really > > need is to add new functions to `Serde`, `Serializer` and `Deserializer` > > interfaces, but since we already dro

Re: [VOTE] KIP-630: Kafka Raft Snapshot

2020-10-02 Thread Guozhang Wang
Thanks Jose! +1 from me. On Fri, Oct 2, 2020 at 3:18 PM Jose Garcia Sancio wrote: > Hi all, > > I would like to start a vote on KIP-630. > > KIP: https://cwiki.apache.org/confluence/x/exV4CQ > Discussion Thread: > >

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

2020-10-01 Thread Guozhang Wang
Thanks for the clarification Jose, that clears my confusions already :) Guozhang On Thu, Oct 1, 2020 at 10:51 AM Jose Garcia Sancio wrote: > Thanks for the email Guozhang. > > > Thanks for the replies and the KIP updates. Just want to clarify one more > > thing regarding my previous comment

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

2020-09-30 Thread Guozhang Wang
> wrote: > > > Hi Guozhang, > > > > Thanks for your feedback. It was very helpful. See my comments below. > > > > Changes to the KIP: > > > > > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=28=27 > > > >

[jira] [Resolved] (KAFKA-10326) Both serializer and deserializer should be able to see the generated client id

2020-09-30 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10326. --- Fix Version/s: 2.7.0 Resolution: Fixed > Both serializer and deserializer sho

Re: [DISCUSS] KIP-674: API to Aggregate Metrics in Kafka Streams

2020-09-30 Thread Guozhang Wang
Hello Bruno, Thanks for the proposed KIP! Here are some thoughts: 1) Regarding Sophie's comment 1), I think there may be one merit to defining the aggregated metric on a different level (though only on a higher level would generally make sense). Although in order for the aggregated metric to

[jira] [Resolved] (KAFKA-9061) StreamStreamJoinIntegrationTest flaky test failures

2020-09-29 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9061. -- Resolution: Cannot Reproduce > StreamStreamJoinIntegrationTest flaky test failu

Re: [VOTE] KIP-671: Add method to Shutdown entire Streams Application

2020-09-29 Thread Guozhang Wang
2020 at 4:36 PM John Roesler > wrote: > > > > > Thanks for the KIP, Walker! > > > > > > I’m +1 (binding) > > > > > > -John > > > > > > On Mon, Sep 21, 2020, at 17:04, Guozhang Wang wrote: > > > >

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-29 Thread Guozhang Wang
nt to provide a more elaborated default handler, I would propose to > > discuss that on a separate KIP to not block this KIP on that discussion. > > > > This is what I am currently doing. Before it is set it defaults to a lambda > to just SHUTDOWN_STREAM_THREAD and if they r

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-28 Thread Guozhang Wang
eamsUncaughtExceptionHandler > > > > On Sun, Sep 27, 2020 at 7:28 PM Guozhang Wang wrote: > > > I think single-threaded clients may be common in practice, and what > > Matthias raised is a valid concern. > > > > We had a related discussion in KIP-663, that m

[jira] [Created] (KAFKA-10532) Do not wipe state store under EOS when closing a RESTORING active or RUNNING standby task

2020-09-28 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10532: - Summary: Do not wipe state store under EOS when closing a RESTORING active or RUNNING standby task Key: KAFKA-10532 URL: https://issues.apache.org/jira/browse/KAFKA-10532

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

2020-09-27 Thread Guozhang Wang
Hello Jose, Thanks for the KIP. Overall it looks great. I have a few meta / minor question, or maybe just clarifications below: Meta: 1. I want to clarify that if only the active controller would generate snapshots, OR would any voter in the quorum would generate snapshots, OR would even

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-27 Thread Guozhang Wang
the KIP Walker. > >>>>> > >>>>> In the KIP we mentioned "In order to communicate the shutdown request > >>> from > >>>>> one client to the others we propose to update the > SubcriptionInfoData to > >>>>> include a s

[jira] [Resolved] (KAFKA-10502) Threadlocal may can not set null,because it may create a memory leak

2020-09-27 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10502. --- Fix Version/s: 2.7.0 Resolution: Fixed > Threadlocal may can not set null,beca

Re: [VOTE] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

2020-09-21 Thread Guozhang Wang
Thanks Bruno. I'm +1 on the KIP. On Mon, Sep 21, 2020 at 2:49 AM Bruno Cadonna wrote: > Hi, > > I would like to restart from zero the voting on KIP-663 that proposes to > add methods to the Kafka Streams client to add and remove stream threads > during execution. > > >

Re: [VOTE] KIP-671: Add method to Shutdown entire Streams Application

2020-09-21 Thread Guozhang Wang
Thanks for finalizing the KIP. +1 (binding) Guozhang On Mon, Sep 21, 2020 at 1:38 PM Walker Carlson wrote: > Hello all, > > I would like to start a thread to vote for KIP-671 to add a method to close > all clients in a kafka streams application. > > KIP: > >

Re: [DISCUSSION] Upgrade system tests to python 3

2020-09-19 Thread Guozhang Wang
with the release, please, let me > know. > > [1] https://github.com/confluentinc/ducktape/issues/245 > [2] https://github.com/apache/kafka/pull/9196 > [3] > https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39 > > > 16 сент. 2020 г., в

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-19 Thread Guozhang Wang
so be > fixed if we implemented our own handler so we can rebalance before the > StreamThread closes. > > With that in mind I believe that `initiateClosingAllClients` would be an > appropriate name. WDYT? > > Walker > > > On Wed, Sep 16, 2020 at 11:43 AM Guozhang Wang

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

2020-09-19 Thread Guozhang Wang
ght exception handler, this guarantee > > >> makes KafkaStreams#localThreadsMetadata() completely backward > compatible > > >> to the current behavior, because if KafkaStreams#addStreamThread() and > > >> KafkaStreams#removeStreamThread() ar

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-16 Thread Guozhang Wang
exceptional situation to which we can react in > >> multiple ways depending on the context. The exception that you propose > >> seems rather a well defined user command than a exceptional situation to > >> me. IMO, we should not use exceptions to control program flow be

Re: [DISCUSSION] Upgrade system tests to python 3

2020-09-15 Thread Guozhang Wang
Hello Nikolay, I can help run the test suite once your PR is cleanly rebased to verify the whole suite works and then I can merge (I'm trusting Ivan and Magnus here for their reviews :) Guozhang On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov wrote: > Hello! > > I got 2 approvals from Ivan

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-15 Thread Guozhang Wang
Hello Walker, Thanks for proposing the KIP! I have a couple more comments: 1. ShutdownRequestedException: my understanding is that this exception is only used if the application-shutdown was initiated by by the user triggered "shutdownApplication()", otherwise e.g. if it is due to source topic

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

2020-09-15 Thread Guozhang Wang
at 2:58 AM Bruno Cadonna wrote: > Hi Guozhang, > > Thank you for your feedback. I replied inline. > > Best, > Bruno > > On 09.09.20 23:43, Guozhang Wang wrote: > > Hello Bruno, > > > > Finally got some time to review your KIP and the discussion thread n

[jira] [Created] (KAFKA-10485) Use a separate error code for replication related errors

2020-09-15 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10485: - Summary: Use a separate error code for replication related errors Key: KAFKA-10485 URL: https://issues.apache.org/jira/browse/KAFKA-10485 Project: Kafka

Re: [VOTE] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

2020-09-09 Thread Guozhang Wang
+1. Thanks for the KIP Gokul ! Guozhang On Tue, Sep 8, 2020 at 6:52 PM Matthias J. Sax wrote: > +1 (binding) > > On 9/8/20 2:49 PM, Jason Gustafson wrote: > > +1 Thanks for the KIP! > > > > On Thu, Sep 3, 2020 at 12:25 PM Gokul Srinivas > wrote: > > > >> Hi, > >> > >> I would like to call a

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

2020-09-09 Thread Guozhang Wang
Hello Bruno, Finally got some time to review your KIP and the discussion thread now.. a few comments below: 1) I'm with Matthias about the newly added numberOfAliveStreamThreads v.s. existing localThreadsMetadata: to me it seems we can always achieve the first based on the second. It seems not

Re: [VOTE] KIP-664: Provide tooling to detect and abort hanging transactions

2020-09-08 Thread Guozhang Wang
+1. Thanks! Guozhang On Tue, Sep 8, 2020 at 3:04 PM Ron Dagostino wrote: > +1 (non-binding) -- Thanks, Jason! > > Ron > > On Tue, Sep 8, 2020 at 2:04 PM Jason Gustafson wrote: > > > Hi All, > > > > I'd like to start a vote on KIP-664: > > > > >

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

2020-09-08 Thread Guozhang Wang
which > this > > config doesn't handle. I like the name change and updated the wiki to > > reflect that, and to clarify that the default value will still be > > Long.MAX_VALUE. > > > > Thanks for your feedback! > > Leah > > > > On Tue, Sep 8, 2020 at

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

2020-09-08 Thread Guozhang Wang
esn't handle. I like the name change and updated the wiki to > reflect that, and to clarify that the default value will still be > Long.MAX_VALUE. > > Thanks for your feedback! > Leah > > On Tue, Sep 8, 2020 at 11:54 AM Guozhang Wang wrote: > > > Hello Leah, > &

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

2020-09-08 Thread Guozhang Wang
Hello Leah, Thanks for initiating this. I just have one minor clarification question here: the config "window.size.ms" seems to be used as the default window size when reading from a topic that represents windowed records right? I.e. if there are multiple topics that represent windowed records

Re: [VOTE] KIP-614: Add Prefix Scan support for State Stores

2020-09-02 Thread Guozhang Wang
Thanks for the KIP Sagar. I'm +1 (binding) too. Guozhang On Tue, Sep 1, 2020 at 1:24 PM Bill Bejeck wrote: > Thanks for the KIP! This is a great addition to the streams API. > > +1 (binding) > > -Bill > > On Tue, Sep 1, 2020 at 12:33 PM Sagar wrote: > > > Hi All, > > > > Bumping the thread

Kafka Meetup hosted Online, Wednesday 5:00pm, Sep 2nd, 2020

2020-09-01 Thread Guozhang Wang
Hello folks, The Bay Area Kafka meetup will continue to be hosted online this month, tomorrow (Sep 2nd) at 5:00pm. This time we will have guest speakers from Twitter to talk about their journey to adopt Apache Kafka. *RSVP*: https://www.meetup.com/KafkaBayArea/events/272643868/ *Date* 5:00pm,

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

2020-08-31 Thread Guozhang Wang
n the schema. > Fixed now! > > -Jason > > On Mon, Aug 31, 2020 at 3:04 PM Guozhang Wang wrote: > > > On Mon, Aug 31, 2020 at 12:28 PM Jason Gustafson > > wrote: > > > > > Hey Guozhang, > > > > > > Thanks for the detailed comment

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

2020-08-31 Thread Guozhang Wang
overall it looks great to me. > > > > I'm not sure how much value the MaxActiveTransactionDuration metric adds, > > given that we have the --find-hanging option in the tool. As you mention, > > instances of these transactions are expected to be rare, and a > > partition-l

Re: Request to subscribe to kafka mailing list

2020-08-31 Thread Guozhang Wang
Hi SaiTejia, You can add yourself to the mailing list, it's self service: https://kafka.apache.org/contact Guozhang On Sat, Aug 29, 2020 at 8:35 AM SaiTeja Ramisetty wrote: > Regards, > SaiTeja - Data Engineer > -- -- Guozhang

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

2020-08-27 Thread Guozhang Wang
Hi Jason, Thanks for the written KIP. I think this is going to be a very useful tool for operational improvements since with eos in its current stage, we cannot confidently assert that we are bug-free, and even in the future when we are confident this is still going to be leveraged by older

Re: [VOTE] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

2020-08-25 Thread Guozhang Wang
+1. Thanks Bruno! Guozhang On Tue, Aug 25, 2020 at 4:00 AM Bruno Cadonna wrote: > Hi, > > I would like to start the vote for > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-662%3A+Throw+Exception+when+Source+Topics+of+a+Streams+App+are+Deleted > > Best, > Bruno > -- -- Guozhang

Re: [DISCUSS] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

2020-08-24 Thread Guozhang Wang
Hello Bruno, Thanks for the KIP, it sounds good to me as well. Just a minor comment: we would include which package the new "MissingSourceTopicException" class belongs to. Guozhang On Fri, Aug 21, 2020 at 11:53 AM John Roesler wrote: > Thanks for the KIP, Bruno! > > Your proposal sounds

Re: KIP idea: Separate publish request from the subscribe request

2020-08-24 Thread Guozhang Wang
Hello Ming, Thanks for bringing this to the community. Just to clarify your proposal, are you suggesting to use a separate port for fetch requests from all other requests including produce, but also e.g. metadata, commit/fetch-offsets, and other inter-broker requests? If yes that would mean the

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

2020-08-18 Thread Guozhang Wang
I'm leaning towards design B primarily because it reminds me of the Firefox logo which I like a lot. But I also share Adam's concern that it should better not obscure the Kafka logo --- so if we can tweak a bit to fix it my vote goes to B, otherwise A :) Guozhang On Tue, Aug 18, 2020 at 9:48 AM

[jira] [Resolved] (KAFKA-10391) Streams should overwrite checkpoint excluding corrupted partitions

2020-08-12 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10391. --- Fix Version/s: 2.7.0 Resolution: Fixed > Streams should overwrite checkpo

[jira] [Created] (KAFKA-10391) Streams should overwrite checkpoint excluding corrupted partitions

2020-08-11 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10391: - Summary: Streams should overwrite checkpoint excluding corrupted partitions Key: KAFKA-10391 URL: https://issues.apache.org/jira/browse/KAFKA-10391 Project: Kafka

[jira] [Resolved] (KAFKA-9450) Decouple inner state flushing from committing

2020-08-11 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9450. -- Fix Version/s: 2.7.0 Resolution: Fixed > Decouple inner state flushing from committ

Re: [ANNOUNCE] New Kafka PMC Member: John Roesler

2020-08-10 Thread Guozhang Wang
Congratulations! On Mon, Aug 10, 2020 at 1:11 PM Jun Rao wrote: > Hi, Everyone, > > John Roesler has been a Kafka committer since Nov. 5, 2019. He has remained > active in the community since becoming a committer. It's my pleasure to > announce that John is now a member of Kafka PMC. > >

[jira] [Resolved] (KAFKA-9659) Kafka Streams / Consumer configured for static membership fails on "fatal exception: group.instance.id gets fenced"

2020-08-10 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9659. -- Fix Version/s: 2.6.0 Resolution: Fixed > Kafka Streams / Consumer configured for sta

Re: [ANNOUNCE] Apache Kafka 2.6.0

2020-08-06 Thread Guozhang Wang
; > Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, > avalsa, > > Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, > Brian > > Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, > Chris > > Egerton, Colin Patrick

[jira] [Created] (KAFKA-10362) When resuming Streams active task with EOS, the checkpoint file should be deleted

2020-08-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10362: - Summary: When resuming Streams active task with EOS, the checkpoint file should be deleted Key: KAFKA-10362 URL: https://issues.apache.org/jira/browse/KAFKA-10362

[jira] [Created] (KAFKA-10357) Handle accidental deletion of repartition-topics as exceptional failure

2020-08-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10357: - Summary: Handle accidental deletion of repartition-topics as exceptional failure Key: KAFKA-10357 URL: https://issues.apache.org/jira/browse/KAFKA-10357 Project

[jira] [Created] (KAFKA-10356) Handle accidental deletion of sink-topics as exceptional failure

2020-08-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10356: - Summary: Handle accidental deletion of sink-topics as exceptional failure Key: KAFKA-10356 URL: https://issues.apache.org/jira/browse/KAFKA-10356 Project: Kafka

[jira] [Created] (KAFKA-10355) Handle accidental deletion of source-topics as exceptional failure

2020-08-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10355: - Summary: Handle accidental deletion of source-topics as exceptional failure Key: KAFKA-10355 URL: https://issues.apache.org/jira/browse/KAFKA-10355 Project: Kafka

Re: New Website Layout

2020-08-03 Thread Guozhang Wang
Hello Ben, I took a look at the corresponding PR and that looks great to me. Just one minor comment otherwise I'm all in for the new UX! * The "see full list" at the homepage just has a drop-down list, maybe better to re-direct to "powered by" if we click it. Guozhang On Fri, Jun 26, 2020 at

Re: [ANNOUNCE] New Kafka PMC Member: Mickael Maison

2020-07-31 Thread Guozhang Wang
Congratulations! On Fri, Jul 31, 2020 at 9:24 AM Rajini Sivaram wrote: > Congratulations, Mickael! > > Regards, > > Rajini > > > On Fri, Jul 31, 2020 at 5:08 PM Boyang Chen > wrote: > > > Congrats Mickael! > > > > On Fri, Jul 31, 2020 at 8:47 AM Bill Bejeck wrote: > > > > > Congrats Mickael!

Re: [VOTE] KIP-450: Sliding Window Aggregations in the DSL

2020-07-28 Thread Guozhang Wang
+1 (binding) On Tue, Jul 28, 2020 at 4:44 PM Matthias J. Sax wrote: > +1 (binding) > > On 7/28/20 4:35 PM, Sophie Blee-Goldman wrote: > > Thanks for the KIP! It's been an enlightening discussion > > > > +1 (non-binding) > > > > Sophie > > > > On Tue, Jul 28, 2020 at 8:03 AM Leah Thomas >

Re: [VOTE] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

2020-07-27 Thread Guozhang Wang
+1. Thanks Jorge for bringing in this KIP! Guozhang On Mon, Jul 27, 2020 at 10:07 AM Leah Thomas wrote: > Hi Jorge, > > Looks great. +1 (non-binding) > > Best, > Leah > > On Thu, Jul 16, 2020 at 6:39 PM Sophie Blee-Goldman > wrote: > > > Hey Jorge, > > > > Thanks for the reminder -- +1

Re: [VOTE] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-27 Thread Guozhang Wang
> >> -Matthias > > > >> > > > >> On 5/15/20 11:48 AM, John Roesler wrote: > > > >>> Thanks, Bruno! > > > >>> > > > >>> I’m +1 (binding) > > > >>> > > > >>> -John >

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-27 Thread Guozhang Wang
se block > cache. Compaction uses the OS to read in the data to compact. But it > also uses fadvise to tell the kernel to not cache the data in the OS > buffer cache. > > Hope that clears up the conflicts! ;-) > > Best, > Bruno > > On 24.07.20 19:37, Guozhang Wang w

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-24 Thread Guozhang Wang
memtables are actually not hard > > bounded at all, while the block cache is soft bounded by default but can > > be configured to be hard bounded. And obviously the OS cache is not > > exactly bounded but it shouldn't cause you to run out of usable memory > > (like the

Re: Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-23 Thread Guozhang Wang
right? On Wed, Jul 22, 2020 at 4:28 AM Bruno Cadonna wrote: > Hi Guozhang, > > Thank you for your feedback! > > I answered inline. > > Best, > Bruno > > > On 21.07.20 00:39, Guozhang Wang wrote: > > Hello Bruno, > > > > Thanks for the update

Re: [DISCUSS] KIP-450: Sliding Windows

2020-07-20 Thread Guozhang Wang
follow a few > > simple, composable rules than when they have a lot of different, specific > > rules. > > > > > > == > > > > New point: (4): > > It would be nice to include some examples of user code that would use the > > new API, which shou

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-20 Thread Guozhang Wang
in. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Report+Properties+of+RocksDB > > Best, > Bruno > > On 15.05.20 17:11, Bill Bejeck wrote: > > Thanks for the KIP, Bruno. Having sensible, easy to access RocksDB memory > &g

[jira] [Created] (KAFKA-10294) Consider whether some of ProcessorStateException should be auto-handled by Streams

2020-07-19 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10294: - Summary: Consider whether some of ProcessorStateException should be auto-handled by Streams Key: KAFKA-10294 URL: https://issues.apache.org/jira/browse/KAFKA-10294

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-07-19 Thread Guozhang Wang
hen we would include the partition as a tag in the metric in addition to the `type` and `metric-name`. Does that work? > > Thanks again for the KIP, > > Tom > > > > On Mon, Jul 13, 2020 at 5:50 PM Jason Gustafson > wrote: > > > Hi All, > > > > Just a

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-07-19 Thread Guozhang Wang
> > Just a quick update on the proposal. We have decided to move quorum > > reassignment to a separate KIP: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-642%3A+Dynamic+quorum+reassignment > > . > > The way this ties into cluster bootstrapping is

[jira] [Created] (KAFKA-10283) Consolidate client-level and consumer-level assignment within ClientState

2020-07-16 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10283: - Summary: Consolidate client-level and consumer-level assignment within ClientState Key: KAFKA-10283 URL: https://issues.apache.org/jira/browse/KAFKA-10283 Project

Re: [DISCUSS] KIP-450: Sliding Windows

2020-07-14 Thread Guozhang Wang
Hello Leah, Thanks for the nice written KIP. A few thoughts: 1) I echo the other reviewer's comments regarding the typing: why extending TimeWindow instead of just extending Window? 2) I also feel that emitting policy for this type of windowing aggregation may be different from the existing

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-10 Thread Guozhang Wang
Hello Colin, Thanks for the nice written KIP. A few meta comments: 1) We need to talk a bit about broker failure detection: is that piggy backed with fencing? i.e. should the controller immediately migrate leader partitions from the fenced brokers? On one side, when a broker is fenced it cannot

[jira] [Created] (KAFKA-10263) Do not create standbys for those revoking active tasks if it is not stateful

2020-07-09 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10263: - Summary: Do not create standbys for those revoking active tasks if it is not stateful Key: KAFKA-10263 URL: https://issues.apache.org/jira/browse/KAFKA-10263

[jira] [Resolved] (KAFKA-10134) High CPU issue during rebalance in Kafka consumer after upgrading to 2.5

2020-07-08 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10134. --- Resolution: Fixed > High CPU issue during rebalance in Kafka consumer after upgrad

Re: Kafka Streams: Dynamic Topic Routing & Nonexistent Topics

2020-07-04 Thread Guozhang Wang
Hello, Thanks for reaching out to the community for this. I think (maybe you've also suggested) it is rather an observation on producer client than on streams client. Generally speaking we want to know if we can fail fast if the metadata cannot be found in producer.send() call. And here are my

Re: [VOTE] KIP-623: Add "internal-topics" option to streams application reset tool

2020-06-29 Thread Guozhang Wang
Thanks Joel, the KIP lgtm. A minor suggestion is to explain where users can get the list of internal topics of a given application, and maybe also add it as part of the helper scripts, for example via topology description. Overall, I'm +1 as well (binding). Guozhang On Sat, Jun 27, 2020 at

Re: [VOTE] KIP-418: A method-chaining way to branch KStream

2020-06-29 Thread Guozhang Wang
+1 (binding). Thanks Ivan! Guozhang On Mon, Jun 29, 2020 at 3:55 AM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > This will be a great addition. Thanks Ivan! > > +1 (non-binding) > > On Fri, Jun 26, 2020 at 7:07 PM John Roesler wrote: > > > Thanks, Ivan! > > > > I’m +1

Re: [VOTE] KIP-629: Use racially neutral terms in our codebase

2020-06-25 Thread Guozhang Wang
+1 (binding). On Thu, Jun 25, 2020 at 4:32 PM John Roesler wrote: > Thanks, Xavier! > > I’m +1 (binding) > > -John > > On Thu, Jun 25, 2020, at 17:43, Xavier Léauté wrote: > > Hi Everyone, > > > > I would like to initiate the voting process for KIP-629. > > >

[jira] [Created] (KAFKA-10199) Separate state restoration into separate threads

2020-06-25 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10199: - Summary: Separate state restoration into separate threads Key: KAFKA-10199 URL: https://issues.apache.org/jira/browse/KAFKA-10199 Project: Kafka Issue

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-06-24 Thread Guozhang Wang
new one. Guozhang On Fri, Jun 12, 2020 at 9:26 AM Boyang Chen wrote: > Thanks for the suggestions Guozhang. > > On Thu, Jun 11, 2020 at 2:51 PM Guozhang Wang wrote: > > > Hello Boyang, > > > > Thanks for the updated information. A few questions here: > > &g

[ANNOUNCE] New committer: Xi Hu

2020-06-24 Thread Guozhang Wang
The PMC for Apache Kafka has invited Xi Hu as a committer and we are pleased to announce that he has accepted! Xi Hu has been actively contributing to Kafka since 2016, and is well recognized especially for his non-code contributions: he maintains a tech blog post evangelizing Kafka in the

Re: [DISCUSS] KIP-629: Use racially neutral terms in our codebase

2020-06-22 Thread Guozhang Wang
Xavier, thanks for the KIP! The proposed names make sense to me. Guozhang On Mon, Jun 22, 2020 at 4:24 PM Xavier Léauté wrote: > Please check the list for updated config / argument names. > > I also added a proposal to replace the term "blackout" with "backoff", > which is used internally in

[ANNOUNCE] New committer: Boyang Chen

2020-06-22 Thread Guozhang Wang
The PMC for Apache Kafka has invited Boyang Chen as a committer and we are pleased to announce that he has accepted! Boyang has been active in the Kafka community more than two years ago. Since then he has presented his experience operating with Kafka Streams at Pinterest as well as several

[jira] [Resolved] (KAFKA-10167) Streams EOS-Beta should not try to get end-offsets as read-committed

2020-06-18 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10167. --- Resolution: Fixed > Streams EOS-Beta should not try to get end-offsets as read-commit

[jira] [Created] (KAFKA-10184) Flaky HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores

2020-06-18 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10184: - Summary: Flaky HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores Key: KAFKA-10184 URL: https://issues.apache.org/jira/browse/KAFKA

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-06-17 Thread Guozhang Wang
Thanks for the KIP Boyang, +1 from me. Guozhang On Wed, Jun 17, 2020 at 1:40 PM Colin McCabe wrote: > Thanks, Boyang! +1 (binding) > > best, > Colin > > On Mon, Jun 15, 2020, at 12:59, Boyang Chen wrote: > > Thanks for more feedback Colin! I have addressed them in the KIP. > > > > Boyang > >

[jira] [Created] (KAFKA-10167) Streams EOS-Beta should not try to get end-offsets as read-committed

2020-06-15 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10167: - Summary: Streams EOS-Beta should not try to get end-offsets as read-committed Key: KAFKA-10167 URL: https://issues.apache.org/jira/browse/KAFKA-10167 Project

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-06-11 Thread Guozhang Wang
t; > > >> Hi, Guozhang and Jason, > >> > >> Thanks for the reply. A couple of more replies. > >> > >> 102. Still not sure about this. How is the tombstone issue addressed in > >> the > >> non-voter and the observer. They can die at any point

[jira] [Resolved] (KAFKA-9182) Flaky Test org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosEnabled

2020-06-10 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9182. -- Fix Version/s: 2.6.0 Resolution: Fixed Resolved as part of https://issues.apache.org

[jira] [Resolved] (KAFKA-9991) Flaky Test KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled

2020-06-10 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9991. -- Fix Version/s: 2.6.0 Resolution: Fixed > Flaky T

Re: [VOTE] KIP-572: Improve timeouts and retires in Kafka Streams

2020-06-09 Thread Guozhang Wang
i page > itself, I would like to proceed with the VOTE. > > > -Matthias > > On 5/20/20 12:30 PM, Guozhang Wang wrote: > > Thanks Matthias, > > > > I agree with you on all the bullet points above. Regarding the > admin-client > > outer-loop retries insid

  1   2   3   4   5   6   7   8   9   10   >