Re: [DISCUSS] KIP-825: introduce a new API to control when aggregated results are produced

Hello Hao, Thanks for the proposal, I have some preference among the options here so I will copy them here: I'm now thinking if it's better to not add this new config on each of the Window interfaces, but instead add that at the KGroupedStream#windowedBy function. Also instead of adding just a

Re: Need permissions to edit the KIP-714

Hello Sarat, What's your apache ID? I cannot find it via your full name in apache wiki space. On Mon, Mar 14, 2022 at 3:48 PM Sarat Kakarla wrote: > Hi Admin, > > Would you please grant me the permissions to edit the following kip: >

[jira] [Created] (KAFKA-13737) Flaky kafka.admin.LeaderElectionCommandTest.testPreferredReplicaElection

Guozhang Wang created KAFKA-13737: - Summary: Flaky kafka.admin.LeaderElectionCommandTest.testPreferredReplicaElection Key: KAFKA-13737 URL: https://issues.apache.org/jira/browse/KAFKA-13737 Project

[jira] [Reopened] (KAFKA-13736) Flaky kafka.network.SocketServerTest.closingChannelWithBufferedReceives

[ https://issues.apache.org/jira/browse/KAFKA-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang reopened KAFKA-13736: --- > Flaky kafka.network.SocketServerTest.closingChannelWithBufferedRecei

[jira] [Resolved] (KAFKA-13736) Flaky kafka.network.SocketServerTest.closingChannelWithBufferedReceives

[ https://issues.apache.org/jira/browse/KAFKA-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13736. --- Resolution: Duplicate > Fl

[jira] [Created] (KAFKA-13736) Flaky kafka.network.SocketServerTest.closingChannelWithBufferedReceives

Guozhang Wang created KAFKA-13736: - Summary: Flaky kafka.network.SocketServerTest.closingChannelWithBufferedReceives Key: KAFKA-13736 URL: https://issues.apache.org/jira/browse/KAFKA-13736 Project

[jira] [Created] (KAFKA-13735) Flaky kafka.network.SocketServerTest.remoteCloseWithoutBufferedReceives

Guozhang Wang created KAFKA-13735: - Summary: Flaky kafka.network.SocketServerTest.remoteCloseWithoutBufferedReceives Key: KAFKA-13735 URL: https://issues.apache.org/jira/browse/KAFKA-13735 Project

[jira] [Reopened] (KAFKA-13421) Fix ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup

[ https://issues.apache.org/jira/browse/KAFKA-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang reopened KAFKA-13421: --- Re-opening this ticket since the test is still failing. > Fix > ConsumerBoun

[jira] [Created] (KAFKA-13722) Update internal interfaces that use ProcessorContext to use StateStoreContext instead

Guozhang Wang created KAFKA-13722: - Summary: Update internal interfaces that use ProcessorContext to use StateStoreContext instead Key: KAFKA-13722 URL: https://issues.apache.org/jira/browse/KAFKA-13722

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

t; > > > > > > > Other possibilities to address this issue might just become too ugly? > > It > > > > seems it would require to add a new `ValueProcessorContext` that > offers > > > > a `#forward(ValueRecord)` method (with `ValueRecord` being a `Record` > > > &

Re: [DISCUSS] Should we automatically close stale PRs?

Hey David, Just reviving on this thread, do you have some final decision on this now with all the feedbacks received so far? On Sun, Feb 13, 2022 at 8:41 PM Ismael Juma wrote: > Hi David, > > I think it's a good idea to use the bot for auto closing stale PRs. The > ideal flow would be: > > 1.

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

d ask if it's a good short term solution. If we believe it is, > > I > > > > am fine with it. > > > > > > > > (I am more worried about the header's KIP...) > > > > > > > > Btw: I am still wondering if we can change existing `process()` as >

Re: [DISCUSS] KIP-761: Add total blocked time metric to streams

Thanks Rohan, I've reviewed the new PR and had a question regarding whether we should have the new metric in ms or ns, maybe we can first discuss about that before we finalize the KIP? Guozhang On Fri, Feb 25, 2022 at 5:48 AM Bruno Cadonna wrote: > Hi Rohan, > > Thank you for the heads up! >

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

owns" the implementation, and we can flag this properly. > > > > With a hypothetical v2.0 based on Record API, this will be harder to > > enforce with the current APIs. e.g. with `mapValues(Record > record)`, > > nothing would stop users from using > `record.withKey("needs

Re: [VOTE] KIP-812: Introduce another form of the `KafkaStreams.close()` API that forces the member to leave the consumer group

Thanks Sueng-chan, I'm +1 on the proposal. Guozhang On Tue, Feb 15, 2022 at 7:06 AM Seung-chan Ahn wrote: > Hi team, > > I feel like we have a finalized proposal for this improvement > . > I want to know what you think. > > Thanks in advance for

Re: [VOTE] KIP-820: Extend KStream process with new Processor API

Thanks Jorge, overall looks good to me. Maybe we can clarify a bit in the wiki that the reason we have to not include the additional `final String... stateStoreNames` params in the new `process` API is that we need to have overloaded functions which takes `ProcessorSupplier<...> ` where the

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

> > > >>>>>> Should this deprecation also consider including > > >>>> `KTable#transformValues`? > > >>>>>> The approach proposed on the KIP: > > >>>>>> `ktable.toStream().processValues().toTable(

Re: [DISCUSS] KIP-813 Shared State Stores

rd to read. > > The KIP mentions a `ProcessorProvider` -- do you mean `ProcessorSupplier`? > > About timestamp synchronization: why do you propose to disable timestamp > synchronization (similar to global state stores)? It seems to be an > unnecessary limitation? -- Given that we could

Re: [DISCUSS] KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocols

avior. > > Thank you. > Luke > > On Mon, Feb 14, 2022 at 10:09 AM Guozhang Wang wrote: > > > Hello Riven, > > > > > > Thanks for bringing this proposal. As we discussed on the JIRA I'm > > personally in favor of this fix. But if all the propos

Re: [DISCUSS] KIP-816: Topology changes without local state reset

g the format of the > > Task ID to change the sub-topology ID from an ordinal to a stable > > identifier. Although I'm not convinced that this option is viable, or > even > > desirable. > > > > Regards, > > > > Nick > > > > On Sat, 12 Feb 2022 at 0

Re: [VOTE] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

efinition is one thing, but what kind of > metric should it be is what I wanted to know. > > Thanks! > Sagar. > > On Mon, Feb 14, 2022 at 8:28 AM Sagar wrote: > > > Thanks Guozhang. > > > > Thanks! > > Sagar. > > > > On Mon, Feb 14, 2022 at

Re: [DISCUSS] KIP-792: Add "generation" field into consumer protocol

s + default > generation(-1)] in V0, or \[current ownedPartitions + default > generation(-1)] in V1. For both V0 and V1 case, we'll directly use the > ownedPartition and generation info in the subscription userData bytes. * > > Please let me know if you have other comments. > > Thank yo

Re: [DISCUSS] KIP-812: Introduce another form of the `KafkaStreams.close()` API that forces the member to leave the consumer group

e control object pattern too. It will save us from > the "telescoping constructors" pattern. Also, I found that we already > introduced this way on `AdminClient`. It sounds consistent to have the same > pattern in this case. > > I updated the `Public Interfaces` section in the KIP

Re: [VOTE] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

ulated by a task. > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186878390 > > > > Thanks! > > Sagar. > > > > On Mon, Jan 24, 2022 at 7:55 AM Guozhang Wang > wrote: > > > >> Thanks Sagar, I'm +1 on the

Re: [DISCUSS] KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocols

Hello Riven, Thanks for bringing this proposal. As we discussed on the JIRA I'm personally in favor of this fix. But if all the proposed changes are in `ConsumerCoordinator`, then we do not need a KIP since that class is internal only. Guozhang On Sat, Feb 12, 2022 at 1:35 AM Riven Sun

Re: [DISCUSS] Should we automatically close stale PRs?

ns/stale#close-pr-message > > Thanks. > Luke > > > On Fri, Feb 11, 2022 at 6:36 AM Guozhang Wang wrote: > > > Just going back to the PRs, @David Jacot, do you know if the > actions/stale > > <https://github.com/actions/stale> tool is able to send notification

Re: [DISCUSS] KIP-816: Topology changes without local state reset

A) to be independent of the sub-topologyID since state store names within a topology should be sufficiently unique would make a lot of sense. On Mon, Feb 7, 2022 at 3:52 PM Guozhang Wang wrote: > Hello Nick, > > I think I'm on the same page of the scope of your KIP, and what I was > t

[jira] [Resolved] (KAFKA-12256) auto commit causes delays due to retriable UNKNOWN_TOPIC_OR_PARTITION

[ https://issues.apache.org/jira/browse/KAFKA-12256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-12256. --- Fix Version/s: 3.2.0 Resolution: Fixed > auto commit causes delays due to retria

Re: [DISCUSS] KIP-818: Introduce cache-size-bytes-total Task Level Metric

> Thanks! > Sagar. > > On Thu, Feb 10, 2022 at 9:54 AM Sagar wrote: > > > Hi Guozhang, > > > > Sure. I will add it to the KIP. > > > > Thanks! > > Sagar. > > > > On Mon, Feb 7, 2022 at 6:22 AM Guozhang Wang wrote: > > > >> Sin

Re: [DISCUSS] Should we automatically close stale PRs?

>>>>>> would definitely save some time for committer reviews. > >>>>>>> > >>>>>>> Nikolay, great that you are willing to do reviews. We do not have a > >>>>>> separate list of PRs that need pre-reviews. You can consult

[ANNOUNCE] New committer: Luke Chen

The PMC for Apache Kafka has invited Luke Chen (showuon) as a committer and we are pleased to announce that he has accepted! Luke has been actively contributing to Kafka since early 2020. He has made more than 120 commits on various components of Kafka, with notable contributions to the rebalance

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

ck if the key did not > > change: if it is just a reference equality check maybe it's okay. What's > > your take on this? > > > > Agree, reference equality should cover this validation and the overhead > > impact should not be meaningful. > > Will update the KIP to

[jira] [Resolved] (KAFKA-13310) KafkaConsumer cannot jump out of the poll method, and the consumer is blocked in the ConsumerCoordinator method maybeAutoCommitOffsetsSync(Timer timer). Cpu and traffic

[ https://issues.apache.org/jira/browse/KAFKA-13310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13310. --- Fix Version/s: 3.2.0 Resolution: Fixed > KafkaConsumer cannot jump out of the p

Re: [DISCUSS] KIP-813 Shared State Stores

where the actual hookup happens (might help to dig into it > more if you want): > > https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/InternalStreamsBuilder.java#L353-L356 > > > -Matthias > > > On 1/20/22 10:04 AM,

Re: [DISCUSS] KIP-792: Add "generation" field into consumer protocol

Hello Luke, Thanks for the updated KIP, I've taken a look at it and still LGTM. Just a couple minor comments in the wiki: * Both `StickyAssignor` and `CooperativeStickyAssignor` that there's already generation is encoded in user-data bytes, the difference is that the `StickyAssignor`'s user

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

Hello Jorge, Thanks for bringing this KIP! I think this is a nice idea to consider using a single overloaded function name for #process, just a couple quick questions after reading the proposal: 1) Does `ValueProcessorContext` have to be a public API? It seems to me that this can be completely

Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?

t; > > > I agree with Guozhang about the waiting times for 3.2.0, 3.0.1, and > 3.1.1. > > > > Obviously that does not satisfy the organizational requirements you > > mentioned. > > > > Best, > > Bruno > > > > On 07.02.22 23:24, Guozhang W

Re: [DISCUSS] KIP-816: Topology changes without local state reset

etecting > > > when a topology has changed unexpectedly, for example. On > > > the other hand, to safely move a state directory from one > > > task directory to the other, we have to be sure no other > > > thread is using either directory. To do that, we could > >

Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?

othetical > > 3.1.1 or 3.0.1 with the RocksDB upgrade instead of 3.2.0 on a platform > > that was not supported before (i.e., 3.1.0 and 3.0.0) might be? > > > > In the end, it is always a risk to upgrade a library in a bugfix release > > without a critica

Re: [DISCUSS] Apache Kafka 3.0.1

+1, thanks Mickael! On Mon, Feb 7, 2022 at 9:27 AM Bruno Cadonna wrote: > +1 Thank you! > > Best, > Bruno > > On 07.02.22 18:24, Ismael Juma wrote: > > Thanks Mickael, +1. > > > > Ismael > > > > On Mon, Feb 7, 2022 at 9:17 AM Mickael Maison > wrote: > > > >> Hi, > >> > >> I'd like to volunteer

Re: [DISCUSS] KIP-812: Introduce another form of the `KafkaStreams.close()` API that forces the member to leave the consumer group

Hello Seung-chan, Thanks for the KIP writeup and summary! I made a pass on it and want to share some of my thoughts: On the very high level, we want to be able to effectively differentiate several cases as follows: 1) There's a network partition / soft failure hence clients cannot reach the

Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?

a valid reason? I was > hoping it would be but I won't submit a PR if it isn't. > > Thanks, > > Jon > > "Guozhang Wang" wrote on 2022-02-03 02:14:34 PM: > > > From: "Guozhang Wang" > > To: "dev" > > Date: 2022-02-03 02:15 PM &

Re: [DISCUSS] KIP-818: Introduce cache-size-bytes-total Task Level Metric

he explanation Guozhang on why this should be > a task level metric. What are your thoughts on this @Sophie? > > Thanks! > Sagar. > > > On Fri, Feb 4, 2022 at 4:47 AM Guozhang Wang wrote: > > > Thanks Sagar for proposing the KIP, and Sophie for sharing your thoughts. > >

[jira] [Resolved] (KAFKA-13563) FindCoordinatorFuture never get cleared in non-group mode( consumer#assign)

[ https://issues.apache.org/jira/browse/KAFKA-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13563. --- Fix Version/s: 3.2.0 3.1.1 Resolution: Fixed

Re: [DISCUSS] Should we automatically close stale PRs?

Thanks for bringing this up David. I'm in favor of some automatic ways to clean up stale PRs. More specifically: * I think there are indeed many root causes why we have so many stale PRs that we should consider, and admittedly the reviewing manpower cannot keep up with the contributing pace is a

[jira] [Resolved] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception

[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13346. --- Resolution: Not A Problem > Kafka Streams fails due to RocksDB Locks Not Availa

Re: [DISCUSS] Apache Kafka 3.2.0 release

Thanks Bruno! +1 On Fri, Feb 4, 2022 at 4:14 PM Ismael Juma wrote: > Thanks for volunteering, Bruno. +1! > > Ismael > > On Fri, Feb 4, 2022 at 7:03 AM Bruno Cadonna wrote: > > > Hi, > > > > I'd like to volunteer to be the release manager for our next > > feature release, 3.2.0. If there are no

Re: [DISCUSS] KIP-816: Topology changes without local state reset

> of it and leave anything beyond that for me to do as followup. > > > > By the way: one advantage of the NamedTopology feature is that we don't > > have to worry about > > any compatibility issues or upgrade/migration path -- it's opt-in by > > definition. (Of course we woul

Re: [VOTE] KIP-591: Add Kafka Streams config to set default state store

t; >> Therefore, I think there's significant value in modifying > >> >> the DSL to allow users to orthogonally specify the storage > >> >> engine and the name of the store, as in your KIP as written. > >> >> > >> >> On the other ha

Re: [DISCUSS] KIP-816: Topology changes without local state reset

Hello Nick, Thanks for bringing this up and for the proposed options. I read though your writeup and here are some of my thoughts: 1) When changing the topology of Kafka Streams, the developer need to first decide if the whole topology's persisted state (including both the state store as well as

Re: [DISCUSS] KIP-818: Introduce cache-size-bytes-total Task Level Metric

Thanks Sagar for proposing the KIP, and Sophie for sharing your thoughts. Here're my 2c: I think I agree with Sophie for making the two metrics (both the added and the newly proposed) on INFO level since we are always calculating them anyways. Regarding the level of the cache-size though, I'm

Re: Kafka <= 3.1 upgrade RocksDB to v6.27.3?

Hello Jonathan, I think Bruno's point is that we can only upgrade in the bugfix releases if the old version of rocksDB has a critical bug that the new version would fix. For 6.27.3 it seems not fixing a critical bug but for a new feature, and hence unless we change the policy we cannot upgrade in

Re: [VOTE] KIP-814: Static membership protocol should let the leader skip assignment

+1. Thanks David! On Tue, Feb 1, 2022 at 9:39 AM Ryan Leslie (BLOOMBERG/ 919 3RD A) < rles...@bloomberg.net> wrote: > Thanks, David. > > +1 (non-binding) > > From: show...@gmail.com At: 01/31/22 22:13:57 UTC-5:00To: > dev@kafka.apache.org > Subject: Re: [VOTE] KIP-814: Static membership protocol

Re: [VOTE] KIP-591: Add Kafka Streams config to set default state store

get the store supplier by the store type: > > >>> > > >>> > > >>> > > >>> *public static SessionBytesStoreSupplier > > >>> sessionStoreSupplierByStoreType()public static > WindowBytesStoreSupplier > > >>> windowStoreSu

Re: Request to Contribute

Hello, Thanks for your interests in contributing, I've added you to JIRA. Guozhang On Thu, Jan 27, 2022 at 4:54 AM Tomonari Yamashita wrote: > Hi Apache Kafka team, > > Request to Contribute > > Could you add me to the contributor list? > > I’ve found a issue: "[#KAFKA-13619]

Re: [VOTE] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

gt; > >> >>> Thanks for the KIP Sagar! > >> >>> ——— > >> >>> Josep Prat > >> >>> > >> >>> Aiven Deutschland GmbH > >> >>> > >> >>> Immanuelkirchstr

Re: [VOTE] KIP-591: Add Kafka Streams config to set default state store

Thanks Luke! I'm +1 on the KIP. Guozhang On Wed, Jan 19, 2022 at 5:58 PM Luke Chen wrote: > Hi devs, > > I'd like to start a vote for the KIP-591: Add Kafka Streams config to set > default state store. The goal is to allow users to set a default store in > the config, so it can apply to all

Re: [DISCUSS] KIP-813 Shared State Stores

Hello Daan, Thanks for writing the KIP. I just read through it and just my 2c here: to me it seems that one of the goal would be to "externalize" the internal changelog topic of an application (say A) so that other consumers can directly read them --- though technically without any auth, anyone

Re: Please add me to the contributors JIRA list

Hello Sayantanu, Thanks for your interests, I've just added you to the list. Cheers, Guozhang On Sat, Jan 15, 2022 at 3:16 AM Sayantanu Dey wrote: > Hi, devs, > I was looking to contribute to Kafka, so can someone please add me to the > contributors' list on JIRA? > JIRA username: dsayan > >

[jira] [Created] (KAFKA-13561) Consider deprecating `StreamsBuilder#build(props)` function

Guozhang Wang created KAFKA-13561: - Summary: Consider deprecating `StreamsBuilder#build(props)` function Key: KAFKA-13561 URL: https://issues.apache.org/jira/browse/KAFKA-13561 Project: Kafka

Re: [DISCUSS] KIP-591: Add Kafka Streams config to set default state store

down the scope and introduce the default > built-in > > enum store types first. > > And if there's further request, we can consider a better way to support > > default store impl. > > > > I'll update the KIP next week, unless there are other opinions from other >

Re: [ANNOUNCE] New Kafka PMC member: David Jacot

Congrats David! On Sat, Dec 18, 2021 at 2:49 PM Matthias J. Sax wrote: > Congrats! > > On 12/17/21 15:46, Bill Bejeck wrote: > > Congratulations David! Well deserved. > > > > -Bill > > > > On Fri, Dec 17, 2021 at 6:43 PM José Armando García Sancio > > wrote: > > > >> Congrats David! > >> > >>

Re: 并发问题请教

Hello, One blog post I can think of would be this: https://www.confluent.io/blog/kafka-fastest-messaging-system/ Here's one Chinese translation version that I found: https://www.sohu.com/a/417379110_355140 Hope it helps, Guozhang On Thu, Dec 16, 2021 at 9:27 PM 酒虫 wrote: > 你好， >

Re: [VOTE] KIP-806: Add session and window query over KV-store in IQv2

estion which operation in WindowStore this query should map to, > given > > > that WindowKeyQuery is templated against WindowStoreIterator and the > > > current use of WindowRangeQuery.fromKey is to call SessionStore.fetch > which > > > returns a KeyValueIterator.

Re: [VOTE] KIP-806: Add session and window query over KV-store in IQv2

> > On Mon, 2021-12-13 at 16:35 +0800, Luke Chen wrote: > > Hi Patrick, > > > > Thanks for the KIP! > > > > I have some comments, in addition to Guozhang's comments: > > 4. The parameter names `windowLower` and `windowUpper` are kind of > > ambiguo

Re: [VOTE] KIP-778 KRaft upgrades

Thanks David! +1. Guozhang On Fri, Dec 10, 2021 at 7:12 PM deng ziming wrote: > Hi, David > > Looking forwarding to this feature > > +1 (non-binding) > > Thanks! > > Ziming Deng > > > On Dec 11, 2021, at 4:49 AM, David Arthur > wrote: > > > > Hey everyone, I'd like to start a vote for KIP-778

Re: [VOTE] KIP-806: Add session and window query over KV-store in IQv2

Hi Patrick, I made a pass on the KIP and have a few comments below: 1. The `WindowRangeQuery` has a private constructor while the `WindowKeyQuery` has not, is that intentional? 2. The `WindowRangeQuery` seems not allowing to range over both window and key, but only window with a fixed key, in

Re: [VOTE] KIP-805: Add range and scan query support in IQ v2

Thanks Vicky, I'd suggest we change the KIP title as "add range and scan query over kv-store in IQv2" just for clarification, otherwise I'm +1. Guozhang On Wed, Dec 8, 2021 at 4:18 PM Matthias J. Sax wrote: > Thanks for the KIP. > > +1 (binding) > > On 12/5/21 7:03 PM, Luke Chen wrote: > > Hi

Re: [DISCUSS] KIP-591: Add Kafka Streams config to set default state store

Thanks Luke for the updated KIP. One major change the new proposal has it to change the original enum store type with a new interface. Where in the enum approach our internal implementations would be something like: " Stores#keyValueBytesStoreSupplier(storeImplTypes, storeName, ...)

Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2

Hi Vicky, Thanks for the KIP. Just for a bit more clarification, could you elaborate an example for windowed stores, beyond a key-value store (I think the `myStore` is for kv-store right?). Otherwise LGTM. Guozhang On Wed, Dec 8, 2021 at 4:18 PM Matthias J. Sax wrote: > Thanks for the

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

d versioned leader will just read the head > part of the data. > > Thanks for the reminder! > > Luke > > On Thu, Dec 9, 2021 at 4:00 AM Guozhang Wang wrote: > > > Hi Luke, > > > > Thanks for the KIP. > > > > One thing I'd like to double check

Re: [DISCUSS] KIP-807: Refactor KafkaStreams exposed metadata hierarchy

Hi Josep, Thanks for the proposed KIP. It looks good to me overall. One meta comment is that in order to illustrate whether that fits more cleanly to the actual metadata use cases, we could add a few examples each with its code snippet based on the new APIs. The ones I'm having in mind are: 1)

Re: [DISCUSS] KIP-796: Interactive Query v2

at all, then we will return the per-partition > FailureReason.DOES_NOT_EXIST. > > Sorry for the oversight. The KIP has been updated. > > Thanks, > -John > > On Wed, 2021-11-17 at 15:48 -0800, Guozhang Wang wrote: > > Thanks John. > > > > I made another pass

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

Hi Luke, Thanks for the KIP. One thing I'd like to double check is that, since the ConsumerProtocolSubscription is not auto generated from the json file, we need to make sure the old-versioned leader would be able to ignore the new field during an upgrade e.g. without crashing. Other than that,

Re: Contributor access

Hello Tamara, Thanks for your interest, I've added you to the contributor list. Guozhang On Mon, Dec 6, 2021 at 9:24 PM Tamara Skokova wrote: > Hi, > I would like to contribute to Apache Kafka. > Could you please grant me contributor access? > ID: tamara_skokova > -- -- Guozhang

Re: [ANNOUNCE] New Kafka PMC Member: Tom Bentley

Congrats Tom! Guozhang On Thu, Nov 18, 2021 at 7:49 AM Jun Rao wrote: > Hi, Everyone, > > Tom Bentley has been a Kafka committer since Mar. 15, 2021. He has been > very instrumental to the community since becoming a committer. It's my > pleasure to announce that Tom is now a member of Kafka

Re: [DISCUSS] KIP-796: Interactive Query v2

tackle it when we propose > the actual queries. > > 12: Cool. I went ahead and dropped the "serdesForStore" > method. I think you're onto something there, and we should > tackle it separately when we propose the actual queries. > > > > > On Tue, 2021-11-16 at 15:59

Re: [DISCUSS] KIP-796: Interactive Query v2

; I do think it would be neat to have queries that don't > deserialize the value by default and give you the option to > do it on demand, or maybe just de-structure some parts of > the value out (eg just reading the timestamp without > deserializing the rest of the value). But, now that I'

Re: [DISCUSS] KIP-778 KRaft Upgrades

that's supported by a majority of the quorum. Guozhang On Tue, Nov 16, 2021 at 2:02 PM Colin McCabe wrote: > On Tue, Nov 16, 2021, at 13:36, Guozhang Wang wrote: > > Hi Colin, > > > > If we allow downgrades which would be appended in metadata.version, then > > the length

Re: [DISCUSS] KIP-778 KRaft Upgrades

other voters, those > voters will truncate their logs to match the new leader. This will > downgrade them. Basically this is the case where the feature upgrade was > proposed, but never fully completed. > > best, > Colin > > > > -David > > > > On Mon, Nov 15, 2

Re: [DISCUSS] KIP-778 KRaft Upgrades

gn and inform the client to retry the update later. > > We may eventually want to consider the metadata.version when electing a > leader, but as long as we have the majority requirement before committing a > new metadata.version, I think we should be safe. > > -David > >

Re: [DISCUSS] KIP-796: Interactive Query v2

Hello John, Great, great, great writeup! :) And thank you for bringing this up finally. I have made a pass on the KIP as well as the POC PR of it, here are some initial thoughts: First are some meta ones: 1. Today the serdes do not only happen at the metered-store layer, unfortunately. For

Re: [DISCUSS] KIP-778 KRaft Upgrades

am I missing > something > > else? > > > > > > > it's inconvenient for a user to manually upgrade every feature version > > > > For this, we would probably want to extend the capabilities of KIP-584. I > > don't think anything we've discussed for KIP-

Re: [DISCUSS] KIP-778 KRaft Upgrades

Hello David, Thanks for the very nice writeup! It helped me a lot to refresh my memory on KIP-630/590/584 :) I just had two clarification questions after reading through the KIP: 1. For the initialization procedure, do we guarantee that all the quorum nodes (inactive candidates and leaders,

Re: [VOTE] KIP-791: Add Record Metadata to State Store Context

+1, thanks Patrick! Guozhang On Mon, Nov 8, 2021 at 5:44 AM Vasiliki Papavasileiou wrote: > Hi Patrick, > > Having the recordMetadata available in the state stores is fundamental for > the consistency work and the proposed approach is reasonable. > > +1 (non-binding) > > Thank you, > Vicky >

Re: [DISCUSS] KIP-791: Add Record Metadata to State Store Context

Thanks Patrick, I looked at the KIP and it looks good to me overall. I think we need to double check whether the record metadata reflect the "last processed record" or the "currently processed record" where the latter may not have been completely processed. In `ProcessorContext#recordMetadata` it

Re: Wiki Permissions

Hello Diego, I saw your id has already been in the contributors list. Cheers, Guozhang On Wed, Nov 3, 2021 at 10:02 AM Diego Erdody wrote: > Hello, > > Can I please have "permissions to contribute to Apache Kafka". > Context: propose a new KIP. > User (both jira and wiki): erdody. > Thanks! >

Re: [VOTE] Add TaskId field to StreamsException

Thanks for the KIP Sophie, I'm big +1 on this idea :) On Tue, Oct 19, 2021 at 6:48 AM Bill Bejeck wrote: > Thanks for the KIP Sophie, I think this will be helpful. > > +1(binding) > > -Bill > > On Mon, Oct 18, 2021 at 5:04 PM John Roesler wrote: > > > Thanks, Sophie, > > > > +1 from me. > > >

[jira] [Resolved] (KAFKA-13319) Do not send AddOffsetsToTxn/TxnOffsetCommit if offsets map is empty

[ https://issues.apache.org/jira/browse/KAFKA-13319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13319. --- Fix Version/s: 3.1.0 Assignee: Guozhang Wang (was: Ryan) Resolution: Fixed

[jira] [Created] (KAFKA-13371) Consider consolidating Joined / StreamJoined / TableJoined

Guozhang Wang created KAFKA-13371: - Summary: Consider consolidating Joined / StreamJoined / TableJoined Key: KAFKA-13371 URL: https://issues.apache.org/jira/browse/KAFKA-13371 Project: Kafka

[jira] [Resolved] (KAFKA-13268) Add more integration tests for Table Table FK joins with repartitioning

[ https://issues.apache.org/jira/browse/KAFKA-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13268. --- Resolution: Duplicate > Add more integration tests for Table Table FK jo

[jira] [Created] (KAFKA-13356) Use "delete" retention policy only for stream-stream join windowed stores

Guozhang Wang created KAFKA-13356: - Summary: Use "delete" retention policy only for stream-stream join windowed stores Key: KAFKA-13356 URL: https://issues.apache.org/jira/browse/KAFKA-13356

[jira] [Created] (KAFKA-13349) Allow Iterator.remove on KeyValueIterator

Guozhang Wang created KAFKA-13349: - Summary: Allow Iterator.remove on KeyValueIterator Key: KAFKA-13349 URL: https://issues.apache.org/jira/browse/KAFKA-13349 Project: Kafka Issue Type

Re: [VOTE] KIP-775: Custom partitioners in foreign key joins

+1. Thanks Victoria! On Tue, Sep 28, 2021 at 2:40 PM John Roesler wrote: > +1 (binding) > > Thanks, Victoria! > > -John > > On Tue, Sep 28, 2021, at 16:29, Adam Bellemare wrote: > > +1 (non-binding) > > > > Glad to see this in here :) > > > > On Tue, Sep 28, 2021 at 5:11 PM Bill Bejeck wrote:

Re: [DISCUSS] KIP-775: Custom partitioners in foreign key joins

Thanks Victoria for writing the KIP! I think this is a miss when we designed KIP-213 and should be fixed in syntax. Regarding how large its scope should be, here's my thoughts: 1) Today Streams does not take any indicator on how the input stream/table are partitioned, instead it simply assumes

Re: [ANNOUNCE] Apache Kafka 3.0.0

avid > Mao, David Osvath, Davor Poldrugo, Dejan Stojadinović, Dhruvil Shah, Diego > Erdody, Dong Lin, Dongjoon Hyun, Dániel Urbán, Edoardo Comar, Edwin Hobor, > Eric Beaudet, Ewen Cheslack-Postava, Gardner Vickers, Gasparina Damien, > Geordie, Greg Harris, Gunnar Morling, Guozhang Wang, G

Re: [DISCUSS] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

M. > >> >> > >> >> Just one thought, if we "rename" the config directly as mentioned in > >> the > >> >> KIP, would that break existing applications? > >> >> Should we deprecate the old one first, and make the old/new names

[jira] [Resolved] (KAFKA-13301) The relationship between request.timeout. ms and max.poll.interval.ms in the Consumer Configs is incorrect.

[ https://issues.apache.org/jira/browse/KAFKA-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13301. --- Fix Version/s: 3.1.0 Resolution: Fixed > The relationship between request.time

[jira] [Resolved] (KAFKA-13216) Streams left/outer joins cause new internal changelog topic to grow unbounded

[ https://issues.apache.org/jira/browse/KAFKA-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13216. --- Resolution: Fixed > Streams left/outer joins cause new internal changelog topic to g

[jira] [Resolved] (KAFKA-13249) Checkpoints do not contain latest offsets on shutdown when using EOS

[ https://issues.apache.org/jira/browse/KAFKA-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-13249. --- Fix Version/s: 3.1.0 Resolution: Fixed > Checkpoints do not contain latest offs

<    1   2   3   4   5   6   7   8   9   10   >