[jira] [Created] (KAFKA-12549) Allow state stores to opt-in transactional support

2021-03-24 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12549: - Summary: Allow state stores to opt-in transactional support Key: KAFKA-12549 URL: https://issues.apache.org/jira/browse/KAFKA-12549 Project: Kafka Issue

[jira] [Created] (KAFKA-12527) Remove deprecated "PartitionAssignor" interface

2021-03-23 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12527: - Summary: Remove deprecated "PartitionAssignor" interface Key: KAFKA-12527 URL: https://issues.apache.org/jira/browse/KAFKA-12527 Project: Kafka

[jira] [Created] (KAFKA-12526) Remove deprecated long ms overloads

2021-03-22 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12526: - Summary: Remove deprecated long ms overloads Key: KAFKA-12526 URL: https://issues.apache.org/jira/browse/KAFKA-12526 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-12524) Remove deprecated WindowBytesStoreSupplier#segments

2021-03-22 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12524: - Summary: Remove deprecated WindowBytesStoreSupplier#segments Key: KAFKA-12524 URL: https://issues.apache.org/jira/browse/KAFKA-12524 Project: Kafka Issue

Re: [DISCUSS] KIP-693: Client-side Circuit Breaker for Partition Write Errors

2021-03-22 Thread Guozhang Wang
Thanks for the updated KIP! Some more comments inlined. On Sun, Mar 7, 2021 at 6:43 PM Guoqiang Shu wrote: > > > Guozhang, many thanks for taking a look! Sorry for the late reply, we have > iterated the prototype on our production setup with your question in mind > and updated the KIP correspond

[jira] [Resolved] (KAFKA-12514) NPE in SubscriptionState

2021-03-22 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-12514. --- Fix Version/s: 3.0.0 Resolution: Fixed > NPE in SubscriptionSt

Re: [VOTE] KIP-708: Rack awareness for Kafka Streams

2021-03-17 Thread Guozhang Wang
SGTM for going back to encoding the full names too --- flexibility wins here, and if users do hit limits on bytes, probably they'd consider giving some shorter names anyways :) Guozhang On Wed, Mar 17, 2021 at 11:25 AM Levani Kokhreidze wrote: > Hi Sophie, > > No worries! And thanks for taking

[jira] [Created] (KAFKA-12472) Add a Consumer / Streams metric to indicate the current rebalance status

2021-03-15 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12472: - Summary: Add a Consumer / Streams metric to indicate the current rebalance status Key: KAFKA-12472 URL: https://issues.apache.org/jira/browse/KAFKA-12472 Project

Re: [ANNOUNCE] New committer: Tom Bentley

2021-03-15 Thread Guozhang Wang
Congratulations Tom! Guozhang On Mon, Mar 15, 2021 at 11:25 AM Bill Bejeck wrote: > Congratulations, Tom! > > -Bill > > On Mon, Mar 15, 2021 at 2:08 PM Bruno Cadonna > wrote: > > > Congrats, Tom! > > > > Best, > > Bruno > > > > On 15.03.21 18:59, Mickael Maison wrote: > > > Hi all, > > > > > >

Re: [ANNOUNCE] New Kafka PMC Member: Chia-Ping Tsai

2021-03-12 Thread Guozhang Wang
Congratulations Chia-Ping! Really glad to have you on the PMC. Guozhang On Fri, Mar 12, 2021 at 11:14 AM Jun Rao wrote: > Hi, Everyone, > > Chia-Ping Tsai has been a Kafka committer since Oct. 15, 2020. He has been > very instrumental to the community since becoming a committer. It's my > ple

Re: [VOTE] KIP-708: Rack awareness for Kafka Streams

2021-03-11 Thread Guozhang Wang
Hello Levani, Thanks for the great write-up! I think this proposal makes sense, though I have one minor suggestion regarding the protocol format change: note the subscription info is part of the group metadata message that we need to write into the internal topic, and hence it's always better if w

[jira] [Created] (KAFKA-12428) Add a last-heartbeat-seconds-ago metric to Kafka Consumer

2021-03-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12428: - Summary: Add a last-heartbeat-seconds-ago metric to Kafka Consumer Key: KAFKA-12428 URL: https://issues.apache.org/jira/browse/KAFKA-12428 Project: Kafka

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

2021-03-04 Thread Guozhang Wang
Thanks Marco, Just a quick thought: what if we reuse the existing Materialized object for both subscription and join stores, instead of introducing a new param / class? Guozhang On Tue, Mar 2, 2021 at 1:07 AM Marco Aurélio Lotz wrote: > Hi folks, > > I would like to invite everyone to discuss

[jira] [Created] (KAFKA-12419) Remove Deprecated APIs of Kafka Streams in 3.0

2021-03-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12419: - Summary: Remove Deprecated APIs of Kafka Streams in 3.0 Key: KAFKA-12419 URL: https://issues.apache.org/jira/browse/KAFKA-12419 Project: Kafka Issue Type

Re: [VOTE] KIP-715: Expose Committed offset in streams

2021-03-01 Thread Guozhang Wang
Thanks Walker for the updated KIP, +1 (binding) Guozhang On Mon, Mar 1, 2021 at 3:47 PM Sophie Blee-Goldman wrote: > Thanks for the KIP! +1 (binding) > > Sophie > > On Mon, Mar 1, 2021 at 10:04 AM Leah Thomas wrote: > > > Hey Walker, > > > > Thanks for leading this discussion. +1 from me, non

Re: [DISCUSS] KIP-715: Expose Committed offset in streams

2021-02-25 Thread Guozhang Wang
nk that if the topology were to evolve there should be fair > > warning anyways. Maybe you can clarify what would be limited by exposing > > the interior topics here? I thought a user could find them in other ways. > > If it is the name we could aynomise them before exposi

[jira] [Resolved] (KAFKA-12323) Record timestamps not populated in event

2021-02-23 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-12323. --- Resolution: Fixed > Record timestamps not populated in ev

[jira] [Created] (KAFKA-12370) Refactor KafkaStreams exposed metadata hierarchy

2021-02-23 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12370: - Summary: Refactor KafkaStreams exposed metadata hierarchy Key: KAFKA-12370 URL: https://issues.apache.org/jira/browse/KAFKA-12370 Project: Kafka Issue

Re: [DISCUSS] KIP-715: Expose Committed offset in streams

2021-02-22 Thread Guozhang Wang
Hello Walker, thanks for the KIP. A few thoughts: 1) Have you considered just relying on the `KafkaStreams#metrics()` that includes embedded consumer metrics that have the committed offsets instead of adding a new API? Not advocating that this is a better approach but want to make sure we consider

Re: Re: [VOTE] KIP-695: Improve Streams Time Synchronization

2021-02-22 Thread Guozhang Wang
break if spurious > > > > wakeups > > > > > > > > > were suddenly introduced in a release, rather than warned > about > > > > from > > > > > > > > > the get-go (not a perfect analogy, far more programs rely > on > &g

Re: [DISCUSS] Kafka 3.0

2021-02-22 Thread Guozhang Wang
at 10:47 AM Ron Dagostino > > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi Michael. This is discussed in the KIP. > >>>>>>>>> > >>>>>>>>> > >>>

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-02-22 Thread Guozhang Wang
; > Hope this helps! If you have any further questions please hit me with them; > I doubt you'll be the only one wondering about these things. > > Cheers, > > Chris > > On Mon, Feb 22, 2021 at 5:30 PM Guozhang Wang wrote: > > > Thanks Chris, yeah I think I ag

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-02-22 Thread Guozhang Wang
ehavior with incremental > > rebalancing; the only difference is that, instead of waiting for the > > rebalance to complete before halting the tasks for that connector, the > > worker will halt them in preparation for the rebalance. This increases > the > > time that the

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-02-21 Thread Guozhang Wang
Hello Chris, Thanks for the great write-up! I was mainly reviewing the admin fenceProducers API of the KIP. I think it makes sense overall. I'm just wondering if we can go one step further, that instead of forcing to fence out all producers of the previous generation, could we try to achieve "part

[jira] [Created] (KAFKA-12352) Improve debuggability with continuous consumer rebalances

2021-02-20 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12352: - Summary: Improve debuggability with continuous consumer rebalances Key: KAFKA-12352 URL: https://issues.apache.org/jira/browse/KAFKA-12352 Project: Kafka

Re: [VOTE] KIP-695: Improve Streams Time Synchronization

2021-02-05 Thread Guozhang Wang
t; > > > > > pollForRecordsOrMetadata(Duration) or something. > > > > > > > > > > > > > > > > > > It seems unreliable to expect the broker to return a > > > > > > > > > particular record within a particu

Re: [VOTE] KIP-691: Transactional Producer Exception Handling

2021-01-28 Thread Guozhang Wang
Thanks Boyang for the thorough discussion. The current KIP LGTM. On Thu, Jan 28, 2021 at 1:36 PM Boyang Chen wrote: > Hey everyone, > > I would like to start the voting for KIP-691 to improve transactional > producer exception handling: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

2021-01-28 Thread Guozhang Wang
bclasses of > KafkaException, and we already mixed its handling with other specific > exception types that could be caught as superclass KafkaException. That > could be addressed in a separate proposal if we see necessary. > > Boyang > > On Thu, Jan 28, 2021 at 3:28 PM Guozhang Wa

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

2021-01-28 Thread Guozhang Wang
t; > > > already have? it's gonna introduce a compatibility issue. > > > > > > > > The original intention was to simplify `send` callback error handling > > by > > > doing exception wrapping, as on Streams level > > > we have to pre

Re: [VOTE] KIP-698: Add Explicit User Initialization of Broker-side State to Kafka Streams

2021-01-28 Thread Guozhang Wang
from folks that have already voted in the > next > > > > couple of days, I will assume their vote is still valid. You can also > > > > confirm your vote if you want. > > > > > > > > KIP: https://cwiki.apache.org/confluence/x/7CnZCQ > >

[jira] [Created] (KAFKA-12242) Decouple state store materialization enforcement from name/serde provider

2021-01-26 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12242: - Summary: Decouple state store materialization enforcement from name/serde provider Key: KAFKA-12242 URL: https://issues.apache.org/jira/browse/KAFKA-12242 Project

Re: [VOTE] KIP-698: Add Explicit User Initialization of Broker-side State to Kafka Streams

2021-01-25 Thread Guozhang Wang
ec 16, 2020 at 4:43 AM Bruno Cadonna wrote: > Hi Guozhang, > > Thank for the feedback! > > Please find my answers inline. > > Best, > Bruno > > > On 14.12.20 23:33, Guozhang Wang wrote: > > Hello Bruno, > > > > Just a few more questions about

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

2021-01-22 Thread Guozhang Wang
or state, so that abortTransaction call would not > throw. > > > > On Fri, Jan 22, 2021 at 11:02 AM Guozhang Wang > wrote: > > > >> Or are you going to maintain some internal state such that, the > >> `abortTransaction` in the catch block would never throw

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

2021-01-22 Thread Guozhang Wang
Or are you going to maintain some internal state such that, the `abortTransaction` in the catch block would never throw again? On Fri, Jan 22, 2021 at 11:01 AM Guozhang Wang wrote: > Hi Boyang/Jason, > > I've also thought about this (i.e. using CommitFailed for all non-fatal),

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

2021-01-22 Thread Guozhang Wang
e encouraged IMHO. > > Let me know your thoughts on the new approach here, thank you! > > Best, > Boyang > > On Tue, Jan 19, 2021 at 11:11 AM Guozhang Wang wrote: > > > Thanks for your clarification on 2)/3), that makes sense. > > > > On Tue, Jan 19, 2021 a

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

2021-01-19 Thread Guozhang Wang
Thanks for your clarification on 2)/3), that makes sense. On Tue, Jan 19, 2021 at 10:16 AM Boyang Chen wrote: > Thanks for the input Guozhang, replied inline. > > On Mon, Jan 18, 2021 at 8:57 PM Guozhang Wang wrote: > > > Hello Boyang, > > > > Thanks for the up

Re: Need permission to create a KIP

2021-01-19 Thread Guozhang Wang
Hello Henry, You should be able to create KIP now. Thanks! Guozhang On Tue, Jan 19, 2021 at 8:45 AM Henry Cai wrote: > Hi, > > Can someone give me permission to create a KIP? My Apache Id is hcai ( > h...@pinterest.com) > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement

[jira] [Resolved] (KAFKA-12219) Potential race condition in InMemoryKeyValueStore

2021-01-19 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-12219. --- Fix Version/s: 2.8.0 Resolution: Fixed > Potential race condition

Re: [DISCUSS] KIP-705: Selectively Disable Topology Optimizations

2021-01-18 Thread Guozhang Wang
Thanks for the KIP Almog! I am in favor of the motivation of this proposal, and just throwing out an alternative about the API here: what do you think if we reuse the `TOPOLOGY_OPTIMIZATION` config but allow it it accept a list of strings instead of a single string, in the following semantics: *

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

2021-01-18 Thread Guozhang Wang
lass and windowed.serializer.inner.class > > > > > > On Tue, Sep 8, 2020 at 2:07 PM Matthias J. Sax > wrote: > > > > > > > From my understanding, the KIP aims for the case when a user does not > > > > control the code, eg, when using the command l

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

2021-01-18 Thread Guozhang Wang
type (i.e. > >> configuration) of the producer. That makes it difficult to have a super > >> exception that encodes the retriability as mentioned above. Would it be > >> possible to introduce exceptions that inherit from RetriableException > >> and that are thr

Re: [ANNOUNCE] Apache Kafka 2.7.0

2020-12-21 Thread Guozhang Wang
117 contributors to this release! > > > > > > A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Adem Efe Gencer, > > > albert02lowis, Alex Diachenko, Andras Katona, Andre Araujo, Andrew > Choi, > > > Andrew Egelhofer, Andy Coates, Ankit Kumar, Anna Povzner, A

Re: [DISCUSS] KIP-693: Client-side Circuit Breaker for Partition Write Errors

2020-12-14 Thread Guozhang Wang
Hello George, Thanks for the KIP. Just a few questions to help us understanding the design details: 1) Why does the default implementation have a criterion of when it is enabled -- i.e. after a certain number of messages have been successfully sent -- instead of always enabled? Also how would thi

Re: [DISCUSS] KIP-694: Support Reducing Partitions for Topics

2020-12-14 Thread Guozhang Wang
Hello George, Thanks for submitting this KIP. On the high-level I think I agree that handling keyed messages is a very complicated issue and maybe we can just start with the easy scenario that does not involve them. Pushing the burden to admin users to determine if it is really safe to delete part

Re: [VOTE] KIP-698: Add Explicit User Initialization of Broker-side State to Kafka Streams

2020-12-14 Thread Guozhang Wang
Hello Bruno, Just a few more questions about the KIP: 1) If the internal topics exist but the calculated num.partitions do not match the existing topics, what would Streams do; 2) Since `init()` is a blocking call (we only return after all topics are confirmed to be created), should we have a tim

Re: [VOTE] KIP-695: Improve Streams Time Synchronization

2020-12-11 Thread Guozhang Wang
Hello John, Thanks for the updates! I've made a pass on the KIP and also the POC PR, here are some minor comments: 1) nit: "receivedTimestamp" -> it seems the metadata keep getting updated, and we do not create a new object but just update the values in-place, so maybe calling it `lastUpdateTimst

Re: [VOTE] KIP-696: Update Streams FSM to clarify ERROR state meaning

2020-12-09 Thread Guozhang Wang
all to close() on ERROR so I added this > >> > line. > >> > "Close() called on ERROR will be idempotent and not throw an > exception, > >> but > >> > we will log a warning." > >> > > >> > I have linked those tickets an

[jira] [Created] (KAFKA-10830) Kafka Producer API should throw unwrapped exceptions

2020-12-08 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10830: - Summary: Kafka Producer API should throw unwrapped exceptions Key: KAFKA-10830 URL: https://issues.apache.org/jira/browse/KAFKA-10830 Project: Kafka Issue

[jira] [Created] (KAFKA-10829) Kafka Streams handle produce exception improvement

2020-12-08 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10829: - Summary: Kafka Streams handle produce exception improvement Key: KAFKA-10829 URL: https://issues.apache.org/jira/browse/KAFKA-10829 Project: Kafka Issue

Re: [DISCUSS] KIP-696: Update Streams FSM to clarify ERROR state meaning

2020-12-08 Thread Guozhang Wang
Hello Walker, Thanks for the KIP! Overall it looks reasonable to me. Just a few minor comments for the wiki page itself: 1) Could you clarify the conditions when RUNNING / REBALANCING -> PENDING_ERROR will happen; and when PENDING_ERROR -> ERROR will happen. E.g. when I read "Streams will only re

Re: [DISCUSSION] KIP-650: Enhance Kafkaesque Raft semantics

2020-12-04 Thread Guozhang Wang
Hi Boyang, Thanks for raising this up. I have a few thoughts about the "Non-leader Linearizable Read", and I think there are two goals we can consider achieving here from a user's perspective: 1) that each of your queries on the raft log is always going to return the latest committed result. 2)

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

2020-12-04 Thread Guozhang Wang
Thanks Boyang for the proposal! I made a pass over the list and here are some thoughts: 0) Although this is not part of the public API, I think we should make sure that the suggested pattern (i.e. user can always call abortTxn() when handling non-fatal errors) are indeed supported. E.g. if the txn

Re: create KIP permission

2020-12-04 Thread Guozhang Wang
Hello Ziming, I saw your ID has been added, you should be able to create KIP page now. On Fri, Dec 4, 2020 at 5:24 AM deng ziming wrote: > Could you please grant me access to Create KIP's. thank you, My ID is > dengziming > -- -- Guozhang

Re: [VOTE] KIP-689: Extend `StreamJoined` to allow more store configs

2020-12-03 Thread Guozhang Wang
+1 (binding) Thanks Leah! On Wed, Dec 2, 2020 at 11:27 AM Sophie Blee-Goldman wrote: > Thanks for the KIP! +1 (binding) > > Sophie > > On Wed, Dec 2, 2020 at 8:42 AM Walker Carlson > wrote: > > > +1 (non-binding) > > > > Thank you, > > walker > > > > On Wed, Dec 2, 2020 at 8:15 AM Bruno Cadonn

Re: Can I get a review for a documentation update (KAFKA-10473)?

2020-12-03 Thread Guozhang Wang
Hello James, I will take a look at your PR. Thanks for the contribution! Guozhang On Wed, Dec 2, 2020 at 5:59 PM James Cheng wrote: > Hi, > > Can I get a review for this small documentation update? > > Thanks, > -James > > > On Oct 9, 2020, at 5:23 PM, James Cheng wrote: > > > > Hi, > > > > W

Re: [VOTE] 2.6.1 RC2

2020-12-01 Thread Guozhang Wang
Hello Gary, Thanks for pointing this out, that change was for making other exceptions easier to debug but I think MemberIdRequiredException is overlooked. I can provide a hotfix to separate this exception from others in this log entry to be included in a future release. Guozhang On Sat, Nov 28,

Re: [VOTE] KIP-680: TopologyTestDriver should not require a Properties argument

2020-11-17 Thread Guozhang Wang
+1, thanks Rohit. Guozhang On Sun, Nov 15, 2020 at 11:53 AM Rohit Deshpande wrote: > Hello all, > I would like to start voting on KIP-680: TopologyTestDriver should not > require a Properties argument. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-680%3A+TopologyTestDriver+should+n

[jira] [Created] (KAFKA-10725) Merge StoreQueryIntegrationTest and QueryableStateIntegrationTest

2020-11-14 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10725: - Summary: Merge StoreQueryIntegrationTest and QueryableStateIntegrationTest Key: KAFKA-10725 URL: https://issues.apache.org/jira/browse/KAFKA-10725 Project: Kafka

Re: [DISCUSS] KIP-680: TopologyTestDriver should not require a Properties argument

2020-11-13 Thread Guozhang Wang
Hello Rohit, I made a pass on the KIP and it looks good to me too. Please feel free to start a voting thread. On Fri, Nov 13, 2020 at 7:36 AM John Roesler wrote: > Thanks Rohit! > > This looks good to me. As far as I’m concerned, you could start the voting > thread. > > -John > > On Wed, Nov 11

[jira] [Created] (KAFKA-10688) Handle accidental truncation of repartition topics as exceptional failure

2020-11-05 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10688: - Summary: Handle accidental truncation of repartition topics as exceptional failure Key: KAFKA-10688 URL: https://issues.apache.org/jira/browse/KAFKA-10688 Project

Re: Formal specification of Kafka.

2020-11-01 Thread Guozhang Wang
Hello Matas, You may find this repo useful: https://github.com/hachikuji/kafka-specification It does not have a very helpful read-me (yet) but if you have questions let us know. Guozhang On Sun, Nov 1, 2020 at 9:36 AM Savickis Matas wrote: > Hello everyone, I am Software engineering Master's

[jira] [Resolved] (KAFKA-10616) StreamThread killed by "IllegalStateException: The processor is already closed"

2020-10-26 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10616. --- Resolution: Fixed > StreamThread killed by "IllegalStateException: The processor is

[ANNOUNCE] New committer: Chia-Ping Tsai

2020-10-19 Thread Guozhang Wang
Hello all, I'm happy to announce that Chia-Ping Tsai has accepted his invitation to become an Apache Kafka committer. Chia-Ping has been contributing to Kafka since March 2018 and has made 74 commits: https://github.com/apache/kafka/commits?author=chia7712 He's also authored several major impro

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

2020-10-19 Thread Guozhang Wang
Congratulations Sophie! Very well deserved. Guozhang On Mon, Oct 19, 2020 at 9:57 AM Tom Bentley wrote: > Congratulations Sophie! > > On Mon, Oct 19, 2020 at 5:55 PM Walker Carlson > wrote: > > > Congratulations Sophie! > > > > On Mon, Oct 19, 2020 at 9:43 AM Navinder Brar > > wrote: > > > >

Re: [ANNOUNCE] New committer: David Jacot

2020-10-16 Thread Guozhang Wang
Congrats David! Guozhang On Fri, Oct 16, 2020 at 10:23 AM Raymond Ng wrote: > Congrats David! > > Cheers, > Ray > > On Fri, Oct 16, 2020 at 10:08 AM Rajini Sivaram > wrote: > > > Congratulations, David! > > > > Regards, > > > > Rajini > > > > On Fri, Oct 16, 2020 at 5:45 PM Matthias J. Sax >

[jira] [Resolved] (KAFKA-10613) Broker should not set leader epoch if the list-offset request version < 4

2020-10-15 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10613. --- Fix Version/s: 2.7.0 Resolution: Fixed > Broker should not set leader epoch if

[jira] [Created] (KAFKA-10614) Group coordinator onElection/onResignation should guard against leader epoch

2020-10-14 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10614: - Summary: Group coordinator onElection/onResignation should guard against leader epoch Key: KAFKA-10614 URL: https://issues.apache.org/jira/browse/KAFKA-10614

[jira] [Created] (KAFKA-10613) Broker should not set leader epoch if the list-offset request version < 4

2020-10-14 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10613: - Summary: Broker should not set leader epoch if the list-offset request version < 4 Key: KAFKA-10613 URL: https://issues.apache.org/jira/browse/KAFKA-10613 Proj

[jira] [Resolved] (KAFKA-10271) Performance regression while fetching a key from a single partition

2020-10-08 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10271. --- Fix Version/s: (was: 2.5.2) 2.7.0 Resolution: Fixed

[jira] [Resolved] (KAFKA-10362) When resuming Streams active task with EOS, the checkpoint file should be deleted

2020-10-07 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10362. --- Fix Version/s: 2.7.0 Resolution: Fixed > When resuming Streams active task with

[jira] [Resolved] (KAFKA-10122) Consumer should allow heartbeat during rebalance as well

2020-10-07 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10122. --- Fix Version/s: 2.6.1 2.7.0 Resolution: Fixed > Consumer sho

Re: [DISCUSSION] Upgrade system tests to python 3

2020-10-07 Thread Guozhang Wang
afka/pull/9196 > >> > >> > >>> 21 сент. 2020 г., в 12:58, Nikolay Izhikov > >> написал(а): > >>> > >>> Hello. > >>> > >>> I fixed two system tests that fails in trunk, also. > >>> > >&g

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

2020-10-06 Thread Guozhang Wang
Sorry I'm late to the party. Matthias raised a point to me regarding the recent development of moving restoration from stream threads to separate restore threads and allowing the stream threads to process any processible tasks even when some other tasks are still being restored by the restore thre

[jira] [Created] (KAFKA-10577) StreamThread should be able to process any processible tasks regardless of its state

2020-10-05 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10577: - Summary: StreamThread should be able to process any processible tasks regardless of its state Key: KAFKA-10577 URL: https://issues.apache.org/jira/browse/KAFKA-10577

[jira] [Created] (KAFKA-10575) StateRestoreListener#onRestoreEnd should always be triggered

2020-10-05 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10575: - Summary: StateRestoreListener#onRestoreEnd should always be triggered Key: KAFKA-10575 URL: https://issues.apache.org/jira/browse/KAFKA-10575 Project: Kafka

Re: [DISCUSS] KIP-598: Augment TopologyDescription with store and source / sink serde information

2020-10-05 Thread Guozhang Wang
on as well. > > Or, if KIP-591 ever gets resurrected, maybe we will have a new store type > enum or > other public API to identify the stores that we can leverage here. But that > KIP seems > to have gone dormant as well :) > > On Fri, Oct 2, 2020 at 6:18 PM Guozhang Wang

Re: [VOTE] KIP-614: Add Prefix Scan support for State Stores

2020-10-05 Thread Guozhang Wang
t; implementation details, so you could just update the KIP > >> document to also mention, "We will also implement the new > >> method in the InMemoryKeyValueStore." > >> > >> Thanks for your contribution to Apache Kafka! > >> -John > >>

Re: [DISCUSS] KIP-598: Augment TopologyDescription with store and source / sink serde information

2020-10-02 Thread Guozhang Wang
the augmented description? > > On Tue, May 19, 2020 at 1:22 PM Guozhang Wang wrote: > > > We already has a Serdes actually, which is a factory class. What we > really > > need is to add new functions to `Serde`, `Serializer` and `Deserializer` > > interfaces, but since

Re: [VOTE] KIP-630: Kafka Raft Snapshot

2020-10-02 Thread Guozhang Wang
Thanks Jose! +1 from me. On Fri, Oct 2, 2020 at 3:18 PM Jose Garcia Sancio wrote: > Hi all, > > I would like to start a vote on KIP-630. > > KIP: https://cwiki.apache.org/confluence/x/exV4CQ > Discussion Thread: > > https://lists.apache.org/thread.html/r9468d1f276385695a2d6d48f6dfbdc504c445fc574

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

2020-10-01 Thread Guozhang Wang
Thanks for the clarification Jose, that clears my confusions already :) Guozhang On Thu, Oct 1, 2020 at 10:51 AM Jose Garcia Sancio wrote: > Thanks for the email Guozhang. > > > Thanks for the replies and the KIP updates. Just want to clarify one more > > thing regarding my previous comment 3)

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

2020-09-30 Thread Guozhang Wang
Sep 28, 2020 at 9:25 PM Jose Garcia Sancio > wrote: > > > Hi Guozhang, > > > > Thanks for your feedback. It was very helpful. See my comments below. > > > > Changes to the KIP: > > > > > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.ac

[jira] [Resolved] (KAFKA-10326) Both serializer and deserializer should be able to see the generated client id

2020-09-30 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10326. --- Fix Version/s: 2.7.0 Resolution: Fixed > Both serializer and deserializer should

Re: [DISCUSS] KIP-674: API to Aggregate Metrics in Kafka Streams

2020-09-30 Thread Guozhang Wang
Hello Bruno, Thanks for the proposed KIP! Here are some thoughts: 1) Regarding Sophie's comment 1), I think there may be one merit to defining the aggregated metric on a different level (though only on a higher level would generally make sense). Although in order for the aggregated metric to repo

[jira] [Resolved] (KAFKA-9061) StreamStreamJoinIntegrationTest flaky test failures

2020-09-29 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9061. -- Resolution: Cannot Reproduce > StreamStreamJoinIntegrationTest flaky test failu

Re: [VOTE] KIP-671: Add method to Shutdown entire Streams Application

2020-09-29 Thread Guozhang Wang
p 24, 2020 at 4:36 PM John Roesler > wrote: > > > > > Thanks for the KIP, Walker! > > > > > > I’m +1 (binding) > > > > > > -John > > > > > > On Mon, Sep 21, 2020, at 17:04, Guozhang Wang wrote: > > >

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-29 Thread Guozhang Wang
to provide a more elaborated default handler, I would propose to > > discuss that on a separate KIP to not block this KIP on that discussion. > > > > This is what I am currently doing. Before it is set it defaults to a lambda > to just SHUTDOWN_STREAM_THREAD and if they reset if

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-28 Thread Guozhang Wang
tExceptionHandler > > > > On Sun, Sep 27, 2020 at 7:28 PM Guozhang Wang wrote: > > > I think single-threaded clients may be common in practice, and what > > Matthias raised is a valid concern. > > > > We had a related discussion in KIP-663, that maybe we ca

[jira] [Created] (KAFKA-10532) Do not wipe state store under EOS when closing a RESTORING active or RUNNING standby task

2020-09-28 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10532: - Summary: Do not wipe state store under EOS when closing a RESTORING active or RUNNING standby task Key: KAFKA-10532 URL: https://issues.apache.org/jira/browse/KAFKA-10532

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

2020-09-27 Thread Guozhang Wang
Hello Jose, Thanks for the KIP. Overall it looks great. I have a few meta / minor question, or maybe just clarifications below: Meta: 1. I want to clarify that if only the active controller would generate snapshots, OR would any voter in the quorum would generate snapshots, OR would even observe

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-27 Thread Guozhang Wang
ks for the KIP Walker. > >>>>> > >>>>> In the KIP we mentioned "In order to communicate the shutdown request > >>> from > >>>>> one client to the others we propose to update the > SubcriptionInfoData to > >>>>&

[jira] [Resolved] (KAFKA-10502) Threadlocal may can not set null,because it may create a memory leak

2020-09-27 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10502. --- Fix Version/s: 2.7.0 Resolution: Fixed > Threadlocal may can not set null,beca

Re: [VOTE] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

2020-09-21 Thread Guozhang Wang
Thanks Bruno. I'm +1 on the KIP. On Mon, Sep 21, 2020 at 2:49 AM Bruno Cadonna wrote: > Hi, > > I would like to restart from zero the voting on KIP-663 that proposes to > add methods to the Kafka Streams client to add and remove stream threads > during execution. > > > https://cwiki.apache.org/c

Re: [VOTE] KIP-671: Add method to Shutdown entire Streams Application

2020-09-21 Thread Guozhang Wang
Thanks for finalizing the KIP. +1 (binding) Guozhang On Mon, Sep 21, 2020 at 1:38 PM Walker Carlson wrote: > Hello all, > > I would like to start a thread to vote for KIP-671 to add a method to close > all clients in a kafka streams application. > > KIP: > > https://cwiki.apache.org/confluence

Re: [DISCUSSION] Upgrade system tests to python 3

2020-09-19 Thread Guozhang Wang
need any help with the release, please, let me > know. > > [1] https://github.com/confluentinc/ducktape/issues/245 > [2] https://github.com/apache/kafka/pull/9196 > [3] > https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39 > > > 16 се

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-19 Thread Guozhang Wang
ld also be > fixed if we implemented our own handler so we can rebalance before the > StreamThread closes. > > With that in mind I believe that `initiateClosingAllClients` would be an > appropriate name. WDYT? > > Walker > > > On Wed, Sep 16, 2020 at 11:43 AM Guozhang

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

2020-09-19 Thread Guozhang Wang
stream thread in its uncaught exception handler, this guarantee > > >> makes KafkaStreams#localThreadsMetadata() completely backward > compatible > > >> to the current behavior, because if KafkaStreams#addStreamThread() and > > >> KafkaStreams#r

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-16 Thread Guozhang Wang
t;> > >> An exception signals an exceptional situation to which we can react in > >> multiple ways depending on the context. The exception that you propose > >> seems rather a well defined user command than a exceptional situation to > >> me. IMO, we should not

Re: [DISCUSSION] Upgrade system tests to python 3

2020-09-15 Thread Guozhang Wang
Hello Nikolay, I can help run the test suite once your PR is cleanly rebased to verify the whole suite works and then I can merge (I'm trusting Ivan and Magnus here for their reviews :) Guozhang On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov wrote: > Hello! > > I got 2 approvals from Ivan Das

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

2020-09-15 Thread Guozhang Wang
Hello Walker, Thanks for proposing the KIP! I have a couple more comments: 1. ShutdownRequestedException: my understanding is that this exception is only used if the application-shutdown was initiated by by the user triggered "shutdownApplication()", otherwise e.g. if it is due to source topic no

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

2020-09-15 Thread Guozhang Wang
2020 at 2:58 AM Bruno Cadonna wrote: > Hi Guozhang, > > Thank you for your feedback. I replied inline. > > Best, > Bruno > > On 09.09.20 23:43, Guozhang Wang wrote: > > Hello Bruno, > > > > Finally got some time to review your KIP and the discussion thr

<    1   2   3   4   5   6   7   8   9   10   >