Re: [ANNOUNCE] Apache Kafka 3.7.0

2024-02-29 Thread Guozhang Wang
; > Abhijeet Kumar, Akhilesh Chaganti, Alieh, Alieh Saeedi, Almog Gavra, > > > Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew > > > Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal, > > > Arnout Engelen, Arpit Goyal, Artem Livshits,

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-12-24 Thread Guozhang Wang
xternal stores outside of the context of Kafka Streams (with the > > understanding that there are fewer semantic guarantees). > > > > On Sat, Dec 2, 2023 at 9:38 AM Guozhang Wang > > wrote: > > > >> Hey Almog, > >> > >> Sorry for the late r

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-12-02 Thread Guozhang Wang
te-Project (stores: []) > --> KTABLE-TOSTREAM-06 > <-- Aggregate-Aggregate-ToOutputSchema > Processor: KTABLE-TOSTREAM-06 (stores: []) > --> KSTREAM-SINK-000007 > <-- Aggregate-Project > Sink: KSTREAM-SINK-07 (t

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-11-18 Thread Guozhang Wang
3 at 4:55 PM Almog Gavra > > >> wrote: > > >>> > > >>>> OK! I think I got everything, but I'll give the KIP another read with > > >>>> fresh eyes later. Just a reminder that the voting is open, so go out > > and &g

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

2023-11-18 Thread Guozhang Wang
Thanks Alieh, I read through the wiki page and the DISCUSS thread, all LGTM except a minor thing in javadoc: "The query returns the records with a global ascending order of keys. The records with the same key are ordered based on their insertion timestamp in ascending order. Both the global and

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2023-11-18 Thread Guozhang Wang
Hi Rohan, I took another look at the updated wiki page and do not have any major questions. Regarding returning a plugin object v.s. configuring a plugin object, I do not have a strong opinion except that the latter seems more consistent with existing patterns. Just curious, any other motivations

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2023-11-09 Thread Guozhang Wang
Hello Rohan, Thanks for the KIP! Overall it looks very nice. Just some quick thoughts : 1. All the API logic is granular at the Task level, except the previousOwnerForPartition func. I’m not clear what’s the motivation behind it, does our controller also want to change how the partitions->tasks

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

2023-10-29 Thread Guozhang Wang
ently includes the caveat that READ_UNCOMMITTED is > >> not > >>>>> available under EOS. I think this is the coupling you are alluding > >> to? > >>>>> > >>>>> This isn't intended to be a restriction of the API, but is currently > >> a >

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

2023-10-28 Thread Guozhang Wang
Congratulations Satish! On Sat, Oct 28, 2023 at 12:59 AM Luke Chen wrote: > > Congrats Satish! > > Luke > > On Sat, Oct 28, 2023 at 11:16 AM ziming deng > wrote: > > > Congratulations Satish! > > > > > On Oct 27, 2023, at 23:03, Jun Rao wrote: > > > > > > Hi, Everyone, > > > > > > Satish

Re: [VOTE] KIP-988 Streams StandbyUpdateListener

2023-10-17 Thread Guozhang Wang
+1 from me. On Mon, Oct 16, 2023 at 1:56 AM Lucas Brutschy wrote: > > Hi, > > thanks again for the KIP! > > +1 (binding) > > Cheers, > Lucas > > > > On Sun, Oct 15, 2023 at 9:13 AM Colt McNealy wrote: > > > > Hello there, > > > > I'd like to call a vote on KIP-988 (co-authored by my friend and

Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

2023-10-17 Thread Guozhang Wang
Seems my previous msg was sent to the wrong recipient, just resending.. On Fri, Oct 13, 2023 at 7:06 PM Guozhang Wang wrote: > > Thanks Hanyu. I made a pass on the KIP and read through the DISCUSS > thread. Do not have any comments. +1 > > On Fri, Oct 13, 2023 at 9:29 AM Hany

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

2023-10-14 Thread Guozhang Wang
> > normal circumstances (no rebalances or task migrations), the changelog > > reader will be in STANDBY_UPDATING, which means it will be updating standby > > tasks as long as there are new records in the changelog topic. That's why I > > prefer onStandbyUpdated instead of

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

2023-10-13 Thread Guozhang Wang
Hello Nick, First of all, thanks a lot for the great effort you've put in driving this KIP! I really like it coming through finally, as many people in the community have raised this. At the same time I honestly feel a bit ashamed for not putting enough of my time supporting it and pushing it

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

2023-10-13 Thread Guozhang Wang
Hello Colt, Thanks for writing the KIP! I have read through the updated KIP and overall it looks great. I only have minor naming comments (well, aren't naming the least boring stuff to discuss and that takes the most of the time for KIPs :P): 1. I tend to agree with Sophie regarding whether or

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

2023-10-13 Thread Guozhang Wang
Thanks Alieh for the KIP, as well as a nice summary of all the discussions! Just my 2c regarding your open questions: 1. I just checked KIP-889 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores) and we used "VersionedRecord get(K key, long asOfTimestamp);", so I

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

2023-09-22 Thread Guozhang Wang
Congratulations! On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai wrote: > > Congratulations Justine! > > On Fri, Sep 22, 2023, 19:25 Philip Nee wrote: > > > Congrats Justine! > > > > On Fri, Sep 22, 2023 at 7:07 PM Luke Chen wrote: > > > > > Hi, Everyone, > > > > > > Justine Olshan has

Re: [ANNOUNCE] New committer: Lucas Brutschy

2023-09-21 Thread Guozhang Wang
Congratulations, Lucas! On Thu, Sep 21, 2023 at 8:45 AM Bruno Cadonna wrote: > > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > Lucas Brutschy. > > Lucas' major contributions are around Kafka Streams. > > Lucas' significantly contributed to the state updater >

Re: [ANNOUNCE] New committer: Yash Mayya

2023-09-21 Thread Guozhang Wang
Congrats, Yash! On Thu, Sep 21, 2023 at 8:28 AM Bruno Cadonna wrote: > > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > Yash Mayya. > > Yash's major contributions are around Connect. > > Yash authored the following KIPs: > > KIP-793: Allow sink connectors to

Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams

2023-08-06 Thread Guozhang Wang
> thanks for pointing out ValueJoinerWithKey. In the end, it's just a > documentation change, ie, point out that the passed in key could be > `null` and similar? > > -Matthias > > > On 8/2/23 3:20 PM, Guozhang Wang wrote: > > Thanks Florin for the writeup, >

Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams

2023-08-02 Thread Guozhang Wang
Thanks Florin for the writeup, One quick thing I'd like to bring up is that in KIP-149 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner) we introduced ValueJoinerWithKey which is aimed to enhance ValueJoiner.

Re: [VOTE] KIP-954: expand default DSL store configuration to custom types

2023-07-29 Thread Guozhang Wang
Thanks Almog! I made a pass over the updated wiki and have no more questions. +1 Guozhang On Wed, Jul 26, 2023 at 8:14 AM Almog Gavra wrote: > > Hello Everyone, > > Opening the voting for KIP-954. The discussion is converging, but please > feel free to chime in on the last few conversation

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-07-27 Thread Guozhang Wang
r InMemory, based on what > is returned for the #persistent >API by the StreamJoined's WindowStoreSupplier > > Does that sound right? We can clarify this further in the KIP if need be > > On Thu, Jul 27, 2023 at 10:48 AM Guozhang Wang > wrote: > > > Hi al

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-07-27 Thread Guozhang Wang
ng a LIST_VALUE subtype on a > >>>>>>>> WindowStore). Thoughts? > >>>>>>>> > >>>>>>>> Lastly, regarding 3.b: > >>>>>>>> > >>>>>>>> I love that you brought this up because that is actually what I > >

Re: plz help me code review

2023-07-27 Thread Guozhang Wang
Hello Xiangyuan, Thanks for your contributions and thanks for raising it to community's attention. I will take a look at it sooner than later. On Thu, Jul 27, 2023 at 2:41 AM Xiangyuan LI wrote: > > Hi kafka team: > I raise a pr https://github.com/apache/kafka/pull/13965 to fix >

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-07-23 Thread Guozhang Wang
Thanks everyone for the great discussions so far! I first saw the JIRA and left some quick thoughts without being aware of the already-written KIP (kudos to Almog, very great one) and the DISCUSS thread here. And I happily find some of my initial thoughts align with the KIP already :) Would like

Re: Request permission to contribute

2023-07-23 Thread Guozhang Wang
Hello Taras, I saw your ID in the contributors list already, could you check if you can create JIRAs now? Guozhang On Fri, Jul 21, 2023 at 4:00 AM Taras Ledkov wrote: > > Hi, Kafka Team. > > I'm following this wiki to request permission to contribute to Apache Kafka >

Re: [ANNOUNCE] New committer: Greg Harris

2023-07-10 Thread Guozhang Wang
Congratulations! On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch wrote: > > Congratulations, Greg. > > On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison > wrote: > > > Congratulations Greg! > > > > On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck > > wrote: > > > > > > Congrats Greg! > > > > > > -Bill

Re: [ANNOUNCE] New PMC chair: Mickael Maison

2023-04-24 Thread Guozhang Wang
Congratulations Mickael! Guozhang On Sat, Apr 22, 2023 at 9:30 PM Kowshik Prakasam wrote: > > Thanks a lot Jun for your hard work and contributions over the years. > Congrats Mickael on your new role, well deserved! Wishing both of you, and > the rest of the community the very best! > > >

Re: Fwd: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

2023-04-11 Thread Guozhang Wang
elease. > >>> > >>> Apologies for not noticing these additional semantics implications > >>> earlier, > >>> Victoria > >>> > >>> -- Forwarded message - > >>> From: Victoria Xia > >>> Date:

Re: Failing tests in Kafka stream is under investigation now

2023-04-06 Thread Guozhang Wang
Hi Luke, Thanks for investigating it, I think my PR broke some tests somehow --- I'm still looking through it why the local runs did not capture them.. my apologies. Guozhang On Thu, Apr 6, 2023 at 4:57 AM Chia-Ping Tsai wrote: > > hi Luke > > Thanks for your time and effort. It would be nice

[jira] [Created] (KAFKA-14847) Separate the callers of commitAllTasks v.s. commitTasks for EOS(-v2) and ALOS

2023-03-24 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-14847: - Summary: Separate the callers of commitAllTasks v.s. commitTasks for EOS(-v2) and ALOS Key: KAFKA-14847 URL: https://issues.apache.org/jira/browse/KAFKA-14847

Re: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-20 Thread Guozhang Wang
+1, thank you Victoria! On Sat, Mar 18, 2023 at 8:27 AM Victoria Xia wrote: > > Hi all, > > I'd like to start a vote on KIP-914 for updating the Kafka Streams join > processors to use proper timestamp-based semantics in applications with > versioned stores: >

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-17 Thread Guozhang Wang
n question) > > because we query older than available history. If you agree, can we add > > this to the KIP? > > > > For left-table-table joins, there seems to be no special impact, but it > > should be called out, too. The lookup itself does not go into the > &g

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-15 Thread Guozhang Wang
ned stores (which makes sense anyway), there will be no > compatibility concerns with introducing the config in a future release. > It's out of scope for this particular KIP as a result, but can hopefully be > introduced as part of the next release after 3.5. > > Best, > Victoria > > On Wed, M

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-15 Thread Guozhang Wang
Thanks Victoria for the great writeup, with a thorough analysis and trade-offs. I do not have any major questions about the proposal. Just one thing that might be out of scope: if users want to enable the versioned table feature across the topology, should we allow them to do it via a single

Re: Regarding Jira Account Creation

2023-03-14 Thread Guozhang Wang
Hi Rohit, You can request the account creation in this url: https://selfserve.apache.org/jira-account.html On Tue, Mar 14, 2023 at 7:19 AM Rohit - wrote: > > Hi, I am Rohit, working as a Software Engineer at Confluent. Recently I > started working on

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

2023-03-10 Thread Guozhang Wang
Congrats David! On Fri, Mar 10, 2023 at 7:29 AM Randall Hauch wrote: > > Congratulations, David! > > Randall > > On Fri, Mar 10, 2023 at 5:17 AM Viktor Somogyi-Vass > wrote: > > > Congrats David! > > > > On Fri, Mar 10, 2023 at 9:12 AM Tom Bentley wrote: > > > > > Congratulations! > > > > > >

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-10 Thread Guozhang Wang
Congrats Chris! On Fri, Mar 10, 2023 at 8:41 AM Jeremy Custenborder wrote: > > Congrats buddy! > > On Fri, Mar 10, 2023 at 9:28 AM Randall Hauch wrote: > > > > Congratulations, Chris! > > > > Randall > > > > On Fri, Mar 10, 2023 at 9:07 AM David Arthur > > wrote: > > > > > Congrats, Chris! > >

[jira] [Resolved] (KAFKA-12639) AbstractCoordinator ignores backoff timeout when joining the consumer group

2023-02-28 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-12639. --- Fix Version/s: 3.5.0 Resolution: Fixed > AbstractCoordinator ignores backoff time

Re: [VOTE] KIP-907: Add Boolean Serde to public interface

2023-02-27 Thread Guozhang Wang
+1, thanks! On Fri, Feb 24, 2023 at 5:43 AM SpacRocket wrote: > Hi Everyone, > > I'd like to call for a vote on KIP-907, which proposes new public classes > to the package org.apache.kafka.common.serialization: > - BooleanSerde > - BooleanSerializer > - BooleanDeserializer > > KIP: >

Re: [VOTE] KIP-904: Kafka Streams - Guarantee subtractor is called before adder if key has not changed

2023-02-27 Thread Guozhang Wang
+1. On Sun, Feb 26, 2023 at 4:27 PM Fq Public wrote: > > Hi everyone, > > I'd like to start the vote on KIP-904: Kafka Streams - Guarantee subtractor > is called before adder if key has not changed. > The KIP is available here: https://cwiki.apache.org/confluence/x/P5VbDg > The easiest way to

Re: [VOTE] KIP-890: Transactions Server Side Defense

2023-02-23 Thread Guozhang Wang
> >> On Fri, Feb 3, 2023 at 1:36 AM Matthias J. Sax wrote: > >> > >> > Thanks for the KIP! > >> > > >> > +1 (binding) > >> > > >> > > >> > On 2/2/23 4:18 PM, Artem Livshits wrote: > >> > >

[jira] [Resolved] (KAFKA-14253) StreamsPartitionAssignor should print the member count in assignment logs

2023-02-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-14253. --- Fix Version/s: 3.5.0 Resolution: Fixed > StreamsPartitionAssignor should pr

Re: [ANNOUNCE] New committer: Lucas Bradstreet

2023-02-16 Thread Guozhang Wang
Congratulations, Lucas! On Thu, Feb 16, 2023 at 3:18 PM Kowshik Prakasam wrote: > > Congratulations, Lucas! > > > Cheers, > Kowshik > > On Thu, Feb 16, 2023, 2:07 PM Justine Olshan > wrote: > > > Congratulations Lucas! > > > > Thanks for your mentorship on some of my KIPs as well :) > > > > On

Re: Requesting permissions to contribute to Apache Kafka.

2023-02-15 Thread Guozhang Wang
Hello Zhangheng, I've granted your permission on the Apache Kafka wiki space. You should be able to create new KIP pages. Guozhang On Wed, Feb 15, 2023 at 6:02 AM hzh0425 wrote: > > Hello, > > My name is zhangheng huang. I want to start contributing to the Apache Kafka > project and want the

Re: Re: [DISCUSS] KIP-904 Kafka Streams - Guarantee subtractor is called before adder if key has not changed

2023-02-14 Thread Guozhang Wang
sistent" states. In the worst case, users in this situation will > > need to update any strict tests that check specifically for the presence of > > transient "inconsistent" states. > > > > What do you think? > > > > Thanks, > > Farooq > >

Re: [DISCUSS] KIP-904 Kafka Streams - Guarantee subtractor is called before adder if key has not changed

2023-02-07 Thread Guozhang Wang
Hello Farooq, Thanks for the very detailed proposal! I think this is a great idea. Just a few thoughts: 1. I regret that we over-optimized the Changed serde format for footprint while making it less extensible. It seems to me that a two rolling bounce migration is unavoidable.. Regarding the

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

2023-01-26 Thread Guozhang Wang
+1 (binding) > > -Matthias > > On 1/24/23 1:17 PM, Guozhang Wang wrote: > > Hi Matthias: > > > > re "paused" -> "suspended": I got your point now, thanks. Just to > > clarify the two functions are a bit different: "paused&quo

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

2023-01-24 Thread Guozhang Wang
ot;: that's a good point. I like it to function in the same way as the "records-rate" metrics. Will update the wiki. re: making "restore-remaining-records-total" at INFO level: sounds good to me too. I will also update the metric name a bit to be more specific. On Thu, Jan 19, 20

Re: [VOTE] KIP-890: Transactions Server Side Defense

2023-01-23 Thread Guozhang Wang
Thanks Justine, I have no further comments on the KIP. +1. On Tue, Jan 17, 2023 at 10:34 AM Jason Gustafson wrote: > > +1. Thanks Justine! > > -Jason > > On Tue, Jan 10, 2023 at 3:46 PM Colt McNealy wrote: > > > (non-binding) +1. Thank you for the KIP, Justine! I've read it; it makes > > sense

Re: [DISCUSS] KIP-890 Server Side Defense

2023-01-20 Thread Guozhang Wang
n > many cases, the producer would be disconnected. (See point 1) and this > would just be an error to return from the server. I did plan to think about > other cases, so let me know if you think of any as well! > > Lots to say! Let me know if you have further thoughts! > Justin

Re: [DISCUSS] KIP-890 Server Side Defense

2023-01-20 Thread Guozhang Wang
Hello Justine, Thanks for the great write-up! I made a quick pass through it and here are some thoughts (I have not been able to read through this thread so pardon me if they have overlapped or subsumed by previous comments): First are some meta ones: 1. I think we need to also improve the

[jira] [Created] (KAFKA-14641) Cleanup CommitNeeded after EOS-V1 is removed

2023-01-19 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-14641: - Summary: Cleanup CommitNeeded after EOS-V1 is removed Key: KAFKA-14641 URL: https://issues.apache.org/jira/browse/KAFKA-14641 Project: Kafka Issue Type

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

2023-01-19 Thread Guozhang Wang
, > >> > >> Thanks for the KIP! > >> > >> +1 (binding) > >> > >> Best, > >> Bruno > >> > >> On 25.10.22 22:07, Walker Carlson wrote: > >>> +1 non binding > >>> > >>> Thanks for the kip!

Re: [ANNOUNCE] New committer: Josep Prat

2023-01-17 Thread Guozhang Wang
Congratulations, Josep! On Tue, Jan 3, 2023 at 11:23 AM Josep Prat wrote: > Thanks all again! :) > > On Tue, Jan 3, 2023 at 6:19 PM Bill Bejeck > wrote: > > > Congratulations, Josep! > > > > -Bill > > > > On Tue, Dec 20, 2022 at 9:03 PM Luke Chen wrote: > > > > > Congratulations, Josep! > > >

Re: [ANNOUNCE] New committer: Edoardo Comar

2023-01-17 Thread Guozhang Wang
Congratulations, Edoardo! On Tue, Jan 10, 2023 at 9:11 AM Bruno Cadonna wrote: > Congrats! > > Best, > Bruno > > On 10.01.23 11:00, Edoardo Comar wrote: > > Many thanks everyone ! > > > > On Mon, 9 Jan 2023 at 19:40, Rajini Sivaram > wrote: > > > >> Congratulations, Edo! > >> > >> Regards, >

Re: [ANNOUNCE] New committer: Satish Duggana

2023-01-17 Thread Guozhang Wang
Congratulations, Satish! On Tue, Jan 10, 2023 at 11:31 AM Rajini Sivaram wrote: > Congratulations, Satish! > > Regards, > > Rajini > > On Tue, Jan 10, 2023 at 5:12 PM Bruno Cadonna wrote: > > > Congrats! > > > > Best, > > Bruno > > > > On 24.12.22 12:44, Manikumar wrote: > > > Congrats,

Re: [ANNOUNCE] New committer: Justine Olshan

2023-01-17 Thread Guozhang Wang
Congratulations, Justine (I'm also late)! On Wed, Jan 11, 2023 at 12:17 AM Bruno Cadonna wrote: > Hi Justine, > > Re-reading my message I realized that my message might be > misinterpreted. I meant that I am late with congratulating you due to > the holidays, NOT that it took you long becoming

Re: [ANNOUNCE] New committer: Walker Carlson

2023-01-17 Thread Guozhang Wang
Congrats, Walker! On Tue, Jan 17, 2023 at 2:20 PM Chris Egerton wrote: > Congrats, Walker! > > On Tue, Jan 17, 2023, 17:07 Bill Bejeck wrote: > > > Congratulations, Walker! > > > > -Bill > > > > On Tue, Jan 17, 2023 at 4:57 PM Matthias J. Sax > wrote: > > > > > Dear community, > > > > > > I

[ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-01 Thread Guozhang Wang
to announce that Bruno has agreed to join the Kafka PMC. Congratulations, Bruno! -- Guozhang Wang, on behalf of Apache Kafka PMC

[VOTE] KIP-869: Improve Streams State Restoration Visibility

2022-10-12 Thread Guozhang Wang
Hello all, I'd like to start a vote for the following KIP, aiming to improve Kafka Stream's restoration visibility via new metrics and callback methods: https://cwiki.apache.org/confluence/display/KAFKA/KIP-869%3A+Improve+Streams+State+Restoration+Visibility Thanks! -- Guozhang

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-10-12 Thread Guozhang Wang
I've updated the KIP doc and would start calling for a vote for this KIP now. On Tue, Oct 11, 2022 at 4:26 AM Nick Telford wrote: > Hi Guozhang, > > What you propose sounds good to me. Having the more detailed Task-level > metrics at DEBUG makes sense. > > Regards, > > Nick > -- -- Guozhang

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-10-10 Thread Guozhang Wang
state updater > > thread that might potentially contain multiple state stores. > > Furthermore, do we need to issue a remote call to record this metric or > > do we get this information from each poll? > > > > 3. > > Could you please give an example where &

Re: [ANNOUNCE] New committer: Deng Ziming

2022-10-10 Thread Guozhang Wang
Congrats Ziming! Thanks a lot for your contributions. Guozhang On Mon, Oct 10, 2022 at 9:31 AM Jason Gustafson wrote: > Hi All > > The PMC for Apache Kafka has invited Deng Ziming to become a committer, > and we are excited to announce that he has accepted! > > Ziming has been contributing to

Re: [DISCUSS] Apache Kafka 3.4.0 release

2022-10-06 Thread Guozhang Wang
4.0 is gonna be the next big milestone :) Thanks Sophie! On Wed, Oct 5, 2022 at 6:08 PM David Jacot wrote: > +1. Thanks, Sophie! > > Le mer. 5 oct. 2022 à 19:57, Luke Chen a écrit : > > > Hi Sophie, > > > > Thanks for volunteering! > > > > Luke > > > > On Thu, Oct 6, 2022 at 6:17 AM José

Re: [VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-10-04 Thread Guozhang Wang
Hello David, I've made my final pass on the doc and I think it looks good now. +1. Guozhang On Wed, Sep 14, 2022 at 1:37 PM Guozhang Wang wrote: > Thanks David, > > There are a few minor comments pending in the discussion thread, and one > is about whether we

Re: Request for contributor permissions

2022-10-02 Thread Guozhang Wang
Hello Ashmeet, Thanks for your interest in contributing. I've added you with contributor permissions. Guozhang On Sat, Oct 1, 2022 at 4:20 AM Ashmeet Lamba wrote: > Hello, > > I'd like to request for contributor permissions. JIRA ID: ashmeetlamba > > Thank you. > Ashmeet > -- -- Guozhang

Re: I would like to contribute

2022-09-29 Thread Guozhang Wang
clude them in the proposal. Guozhang On Thu, Sep 29, 2022 at 1:21 PM Collins, Alex wrote: > Awesome. Yes. alex_collins > ____ > From: Guozhang Wang > Sent: Thursday, September 29, 2022 12:55:27 PM > To: dev@kafka.apache.org > Subject: Re: I would like

Re: I would like to contribute

2022-09-29 Thread Guozhang Wang
Hello Alex, Thank you for your interests!Do you already have an apache id? If yes I can help letting you be able to create corresponding wiki pages. On Thu, Sep 29, 2022 at 11:56 AM Collins, Alex wrote: > I have three things to contribute: > > > * Bug fix for console producer/consumers.

Re: [DISCUSS] KIP-871: Fix ByteBufferSerializer#serialize(String, ByteBuffer) compatible problem

2022-09-26 Thread Guozhang Wang
Hello ShunKang, Thanks for filing the report. I looked at the source code and I agree it's a bug. The reason we did not get it in the past is probably just because we were inefficiently keep byte-copying and hence for almost all cases, the offset is set to 0 when serialization happens. Note that

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-26 Thread Guozhang Wang
ith Streams though so my > intuition could be wrong here. > > David > > On Mon, Sep 26, 2022 at 7:26 PM Guozhang Wang wrote: > > > > Regarding the version, what I was thinking is that in the HB request, for > > "serverAssignor" field, instead

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-26 Thread Guozhang Wang
ror prone because it > is not clear whether a partition is usable or not. Or do you think > that we should not provide the partitions but only the metadata? > > Best, > David > > On Fri, Sep 23, 2022 at 9:40 PM Guozhang Wang wrote: > > > > Hello David, > > > &

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-23 Thread Guozhang Wang
asks as a result of the metadata bookkeeping does not need to wait until all the partitions that are yet-assigned have been completely assigned to the member. Such information may not always be updatable in an incremental manner as the partitions-revoked / partitions-assigned. In such a case, it's bet

Re: [VOTE] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-22 Thread Guozhang Wang
+1, thanks ShunKang. Though its proposed motivation is on consumer fetcher's deserialization, I think adding an overloaded method with ByteBuffer would help with other serde places on the client side as well. Guozhang On Thu, Sep 22, 2022 at 9:41 AM ShunKang Lin wrote: > Hi everyone, > > I'd

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-22 Thread Guozhang Wang
Could you start a separate VOTE email thread calling for votes? On Thu, Sep 22, 2022 at 9:19 AM ShunKang Lin wrote: > Hi Guozhang, > > Thanks for your help! By the way, what should I do next? > > Best, > ShunKang > > Guozhang Wang 于2022年9月22日周四 23:21写道: > > >

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-22 Thread Guozhang Wang
Thanks ShunKang, I made a few nit edits on the Motivation section as well. LGTM for me now. On Thu, Sep 22, 2022 at 7:33 AM ShunKang Lin wrote: > Hi Guozhang, > > I've updated the "Motivation" section of the KIP, please take a look. > > Thanks. > ShunKang > >

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-09-20 Thread Guozhang Wang
Hello Bruno/Nick, I've updated the KIP wiki to reflect the incorporated comments from you, please feel free to take another look and let me know what you think. Guozhang On Tue, Sep 20, 2022 at 9:37 AM Guozhang Wang wrote: > Hi Nick, > > Thanks for the reviews, and I think these

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-20 Thread Guozhang Wang
Guozhang On Tue, Sep 20, 2022 at 10:24 AM Guozhang Wang wrote: > 1. Ack, thanks. > 2. Sounds good, thanks for clarifying. > > On Tue, Sep 20, 2022 at 9:50 AM ShunKang Lin > wrote: > >> Hi Guozhang, >> >> Thanks for your comments! >> >> 1. We

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-20 Thread Guozhang Wang
d `default ByteBuffer serializeToByteBuffer(String > topic, Headers headers, T data)` in Serializer to reduce memory copy in > `KafkaProducer#doSend(ProducerRecord, Callback)`, but this change is a bit > big, I prefer to submit another one KIP to do the job. > > Thanks. > ShunKang

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-09-20 Thread Guozhang Wang
useful information like the > estimated time remaining for restoration (by dividing the remaining total > by the restoration rate). > > Regards, > > Nick > > On Mon, 19 Sept 2022 at 19:57, Guozhang Wang wrote: > > > Hello Bruno, > > > > Thanks for your

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-19 Thread Guozhang Wang
A separate question regarding the proposed API as well: what do you think to also augment the serializers with ByteBuffer return type in order to be symmetric with deserializers? On Mon, Sep 19, 2022 at 3:32 PM Guozhang Wang wrote: > Hello ShunKang, > > Thanks for filing the

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-19 Thread Guozhang Wang
Hello ShunKang, Thanks for filing the proposal, and sorry for the late reply! I looked over your KIP proposal and the PR, in general I think I agree that adding an overloaded function with `ByteBuffer` param is beneficial, but I have a meta question regarding it's impact on Kafka consumer: my

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-09-19 Thread Guozhang Wang
s finite. Standby tasks rather > update continuously their states. They can be up-to-date or lagging. I > see that you could say "restored" instead of "up-to-date" and "not > restored" instead of "lagging", but IMO it does not describe well the >

Re: KafkaConsumer refactor proposal

2022-09-16 Thread Guozhang Wang
naming. > 2. Great suggestions. > 3. I think I'll start with defining a few basic events, like assign, > commits, and fetch. Then we can modify that as we progress with the > project. > > Thanks, > P > > > On Thu, Sep 15, 2022 at 3:02 PM Guozhang Wang wrote: > > > H

[DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-09-16 Thread Guozhang Wang
Hello everyone, I'd like to start a discussion for the following KIP, aiming to improve Kafka Stream's restoration visibility via new metrics and callback methods: https://cwiki.apache.org/confluence/display/KAFKA/KIP-869%3A+Improve+Streams+State+Restoration+Visibility Thanks! -- Guozhang

[jira] [Created] (KAFKA-14239) Merge StateRestorationIntegrationTest into RestoreIntegrationTest

2022-09-16 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-14239: - Summary: Merge StateRestorationIntegrationTest into RestoreIntegrationTest Key: KAFKA-14239 URL: https://issues.apache.org/jira/browse/KAFKA-14239 Project: Kafka

Re: Request for contributor permissions

2022-09-16 Thread Guozhang Wang
Hello Akshay, Thanks for your interest! You should have the permissions now. Guozhang On Fri, Sep 16, 2022 at 7:40 AM Akshay Bhadange wrote: > Hello sir/madam , > > I am really interested in this project and want to contribute.so please > provide me the contributor permission.my jira id is >

Re: KafkaConsumer refactor proposal

2022-09-15 Thread Guozhang Wang
Hello Philip, Thanks for writing down the 1-pager. Just to clarify, the reason we wrote this as a 1-pager instead of a KIP is that so far all the implementations should be internal, and hence not impacting anything on the public APIs. If, e.g. we found along with the implementation that some

Re: [VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-14 Thread Guozhang Wang
Thanks David, There are a few minor comments pending in the discussion thread, and one is about whether we should merge PreparePartitionAssignment with HB. But I think the KIP itself is in pretty good shape now. Thanks! Guozhang On Fri, Sep 9, 2022 at 1:32 AM David Jacot wrote: > Hi all, > >

Re: [VOTE] KIP-862: Self-join optimization for stream-stream joins

2022-09-13 Thread Guozhang Wang
iou wrote: > > Hey Guozhang, > > > > Great suggestion, I made the change. > > > > Best, > > Vicky > > > > On Fri, Sep 9, 2022 at 10:43 PM Guozhang Wang > wrote: > > > >> Thanks Vicky, that reads much clearer now. > >> > >

Re: [VOTE] KIP-862: Self-join optimization for stream-stream joins

2022-09-09 Thread Guozhang Wang
the new value > strings for each optimization rule. Let me rephrase that in the KIP to make > it clearer. > Is it better now? > > Best, > Vicky > > On Thu, Sep 8, 2022 at 9:07 PM Guozhang Wang wrote: > > > Thanks Vicky, > > > > I read through the KIP agai

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-09 Thread Guozhang Wang
; Best, > David > > On Fri, Sep 9, 2022 at 7:10 AM Guozhang Wang wrote: > > > > Hello David, > > > > One of Jun's comments make me thinking: > > > > ``` > > In this case, a new assignment is triggered by the client side > > assignor. When const

Re: [VOTE] KIP-865: Support --bootstrap-server in kafka-streams-application-reset

2022-09-09 Thread Guozhang Wang
+1. Thanks. Guozhang On Fri, Sep 9, 2022 at 9:52 AM Николай Ижиков wrote: > Hello. > > I'd like to start a vote on KIP-865 which adds support of > —bootstrap-server parameter in kafka-streams-application-reset tool > > Discuss Thread: >

Re: [DISCUSSION] KIP-864: Support --bootstrap-server in kafka-streams-application-reset

2022-09-09 Thread Guozhang Wang
Hi, I think we can vote right away. Guozhang On Fri, Sep 9, 2022 at 2:24 AM Николай Ижиков wrote: > Hello, Guozhang. > > Thanks for the feedback. > As this KIP very straightforward, is it worth to be voted right now? > Or should we wait more feedback? > > > 9 сент. 20

Re: [DISCUSSION] KIP-864: Support --bootstrap-server in kafka-streams-application-reset

2022-09-08 Thread Guozhang Wang
Hello Николай, Thanks for writing the KIP, I think it's rather straightforward and better to be consistent in tooling params. I'm +1. Guozhang On Mon, Sep 5, 2022 at 11:25 PM Николай Ижиков wrote: > Hello. > > Do we still want to make parameter names consistent in tools? > If yes, please,

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-08 Thread Guozhang Wang
gt; > Best, > > > > > > >> > David > > > > > > >> > > > > > > > >> > On Mon, Aug 8, 2022 at 2:41 PM David Jacot < > dja...@confluent.io> > > > > wrote: > > > > > > >> > > > > >

Re: [DISCUSS] KIP-857: Streaming recursion in Kafka Streams

2022-09-08 Thread Guozhang Wang
sively" method requires significantly less code than > >recursion via an explicit topic, and is significantly easier to > understand. > > > > Ultimately, I don't think repartitioning inside the unary operator adds > > much complexity to the implementation. Certai

Re: [VOTE] KIP-862: Self-join optimization for stream-stream joins

2022-09-08 Thread Guozhang Wang
Thanks Vicky, I read through the KIP again and it looks good to me. Just a quick question regarding the public config changes: you mentioned "No public interfaces will be impacted. The config TOPOLOGY_OPTIMIZATION_CONFIG will be extended to accept a list of optimization rule configs in addition

Re: Last sprint to finish line: Replace EasyMock/Powermock with Mockito

2022-09-08 Thread Guozhang Wang
Thanks Christo for the updates. As we are adding new unit tests we are also keen on using the new Mockito packages and so far I'd like to say it's much easier to use :) would chime in to help on reviewing some of the PRs as well. Guozhang On Tue, Sep 6, 2022 at 11:02 PM Christo Lolov wrote: >

Re: [DISCUSS] KIP-857: Streaming recursion in Kafka Streams

2022-09-06 Thread Guozhang Wang
implementation should be, and is, essentially > >> independent from the > >> design of the API itself -- in other words, why does calling this > >> operator/method `recursion` > >> not work, or have anything at all to do with what Streams "knows" or how > &g

  1   2   3   4   5   6   7   8   9   10   >