Re: [DISCUSS] KIP-1065: Add "retry" return-option to ProductionExceptionHandler

dler? Most serdes are probably (hopefully!) deterministic, but could deserialization fail due to network errors or other transient issues with schema registry? On Sun, Jun 30, 2024 at 12:35 PM Matthias J. Sax wrote: Hi, as a follow up to https://issues.apache.org/jira/browse/KAFKA-16508 which is rel

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

the scope from any error that might take place with producing a record to Kafka, to only the ones that are thrown directly from Producer::send; is that the intended behavior here? And if so, do you have thoughts on how we might design a follow-up KIP that would catch all erro

Re: [VOTE] KIP-1041: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)

I was just made aware of this KIP, and I believe we cannot drop this config in 4.0, as we guarantee a one year deprecation period. Did not read the discuss thread. Is there any good reason for not following our guaranteed deprecation period? -Matthias On 5/13/24 5:21 AM, David Jacot wrote:

Re: [DISCUSS] KIP-1067: Remove ReplicaVerificationTool in 4.0 (deprecate in 3.9)

eprecate something in 3.8 and then remove it in 4.0. ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=303794933) Maybe it is a good time to have a discussion about the deprecation cycle. Personally, the second rule is easy to follow, but the side effect is that the "deprecation cycle&qu

Re: [DISCUSS] KIP-1067: Remove ReplicaVerificationTool in 4.0 (deprecate in 3.9)

Hi, If I understand correctly, instead of maintaining the tool further (via KIP-752) we want to deprecate the tool? This is fine, but it seem the KIP deadline for 3.9 passed, so we cannot deprecate it in 3.9, but only in 4.0? And even if we make an exception, we usually have a deprecation pe

[DISCUSS] KIP-1065: Add "retry" return-option to ProductionExceptionHandler

Hi, as a follow up to https://issues.apache.org/jira/browse/KAFKA-16508 which is related to KIP-1038, I would like to prose adding a RETRY option to production error handler responses: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311627309 Looking forward to your feedback

[jira] [Created] (KAFKA-17057) Add "retry" option to ProductionExceptionHandler

Matthias J. Sax created KAFKA-17057: --- Summary: Add "retry" option to ProductionExceptionHandler Key: KAFKA-17057 URL: https://issues.apache.org/jira/browse/KAFKA-17057 Project: Kafka

[jira] [Resolved] (KAFKA-16508) Infinite loop if output topic does not exisit

[ https://issues.apache.org/jira/browse/KAFKA-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16508. - Fix Version/s: 3.9.0 Resolution: Fixed > Infinite loop if output topic does

Re: [VOTE] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Thanks for the KIP Alieh! +1 (binding) -Matthias On 6/26/24 5:29 AM, Andrew Schofield wrote: Hi Alieh, Thanks for the KIP. I think we’ve settled on a good solution. +1 (non-binding) Thanks, Andrew On 25 Jun 2024, at 13:17, Alieh Saeedi wrote: Hi all, I would like to open voting for KIP

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Thanks Alieh, it seems this KIP can just pick between a couple of tradeoffs. Adding an overloaded `send()` as the KIP propose makes sense to me and seems to provides the cleanest solution compare to there options we discussed. Given the explicit name of the passed-in option that highlights th

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

w error handling to take place during implicit flushes, via producer callback? On Mon, Jun 24, 2024 at 9:21 PM Matthias J. Sax wrote: My point it, that it does not seem to be safe to allow users to ignore errors with an implicit flush, and I think it's better to only allow it with

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

send(…) flush() commitTransaction(new CommitTransactionOptions().transactionVerified(true)) I’d tend towards the enum here because I doubt we need as much flexibility as the admin API requires. Thanks, Andrew On 24 Jun 2024, at 18:39, Matthias J. Sax wrote: I am ok either way (ie, fl

[jira] [Resolved] (KAFKA-16965) Add a "root cause" exception as a nested exception to TimeoutException for Producer

[ https://issues.apache.org/jira/browse/KAFKA-16965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16965. - Fix Version/s: 3.9.0 Resolution: Fixed > Add a "root cause" e

Re: 回复: apply a jira account

发件人: Matthias J. Sax 发送时间: 2024年6月24日 05:43 收件人: dev@kafka.apache.org 主题: Re: apply a jira account Thanks for you interest to contribute to Kafka. Unfortunately, I could not find your account. Can you double check the Jira ID? -Matthias On 6/23/24 8:01 AM, 周 正轲 wrote: Hi Guys

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

If the answer to question (1) is NO and to question (2) is YES, do we need to change the current `flush` or `commitTnx` at all? Cheers, Alieh On Sat, Jun 22, 2024 at 3:21 AM Matthias J. Sax wrote: Hey Kirk, can you elaborate on a few points? Otherwise users would have to know to explic

Re: apply a jira account

Thanks for you interest to contribute to Kafka. Unfortunately, I could not find your account. Can you double check the Jira ID? -Matthias On 6/23/24 8:01 AM, 周 正轲 wrote: Hi Guys, I want to contribute to Apache Kafka. Would you please give me the permission as a contributor? My JIRA ID is zzzk

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

Matthias J. Sax wrote: I believe the class name was picked on purpose, to make clear that it should not be used -- the problem is, that the class is in a public package and is by itself public (that's unfortunately require, given how Java works). Of course, it's also in the JavaDocs that

Re: [VOTE] KIP-1056 - Deprecate `default.` prefix for exception handler in StreamsConfig

+1 (binding) from my side. I understand the concerns raised, but would personally move forward with this KIP as-is. If we cannot get three votes, it would get naturally discarded. -Matthias On 6/19/24 11:33 AM, Muralidhar Basani wrote: Hi all, I would like to call a vote on KIP-1056 - De

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

d still be to put the producer into error state, and the user would need to call `abortTransaction()` to move forward. -Matthias On 6/21/24 5:26 PM, Kirk True wrote: Hi Matthias, On Jun 21, 2024, at 12:28 PM, Matthias J. Sax wrote: If we want to limit it to `RecordTooLargeException` thr

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

-large exception), the error is actually noticed by the producer before send() returns, so it should be possible to throw directly. Cheers, Chris On Fri, Jun 21, 2024, 14:25 Matthias J. Sax wrote: Not sure if we can change send and make it throw, given that send() is async? That is why users

[jira] [Created] (KAFKA-17019) Producer TimeoutException should include root cause

Matthias J. Sax created KAFKA-17019: --- Summary: Producer TimeoutException should include root cause Key: KAFKA-17019 URL: https://issues.apache.org/jira/browse/KAFKA-17019 Project: Kafka

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

ble to stop a commit from proceeding even on fatal errors. But I guess we'll just have to clearly document it. In some way we are basically adding a flag to optionally restore the https://issues.apache.org/jira/browse/KAFKA-9279 bug, which is the motivation for all these changes, anyway :

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

tion to tweak a configuration file that gets fed to producers spun up inside it. Cheers, Chris On Wed, Jun 19, 2024 at 10:32 PM Matthias J. Sax wrote: Thanks for the KIP Alieh. I actually like the KIP as-is, but think Arthem raises very good points... Seems we have four options on how to mo

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Thanks for the KIP Alieh. I actually like the KIP as-is, but think Arthem raises very good points... Seems we have four options on how to move forward? 1. add config to allow "silent error clearance" as the KIP proposes 2. change flush() to clear error and let it throw 3. add new flushAndThr

[jira] [Created] (KAFKA-16994) Flaky Test SlidingWindowedKStreamIntegrationTest.shouldRestoreAfterJoinRestart

Matthias J. Sax created KAFKA-16994: --- Summary: Flaky Test SlidingWindowedKStreamIntegrationTest.shouldRestoreAfterJoinRestart Key: KAFKA-16994 URL: https://issues.apache.org/jira/browse/KAFKA-16994

[jira] [Created] (KAFKA-16993) Flaky test RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener.shouldInvokeUserDefinedGlobalStateRestoreListener()

Matthias J. Sax created KAFKA-16993: --- Summary: Flaky test RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener.shouldInvokeUserDefinedGlobalStateRestoreListener() Key: KAFKA-16993 URL: https

[jira] [Resolved] (KAFKA-16992) Flaky Test org.apache.kafka.streams.integration.EOSUncleanShutdownIntegrationTest.shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once_v2]

[ https://issues.apache.org/jira/browse/KAFKA-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16992. - Resolution: Duplicate > Flaky T

[jira] [Created] (KAFKA-16992) Flaky Test org.apache.kafka.streams.integration.EOSUncleanShutdownIntegrationTest.shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once_v2]

Matthias J. Sax created KAFKA-16992: --- Summary: Flaky Test org.apache.kafka.streams.integration.EOSUncleanShutdownIntegrationTest.shouldWorkWithUncleanShutdownWipeOutStateStore[exactly_once_v2] Key: KAFKA-16992

[jira] [Created] (KAFKA-16991) Flaky Test org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState

Matthias J. Sax created KAFKA-16991: --- Summary: Flaky Test org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState Key: KAFKA-16991 URL: https://issues.apache.org/jira/browse

Re: [VOTE] KIP-1038: Add Custom Error Handler to Producer

I just replied to the DISCUSS thread, proposing to DISCARD this KIP in favor of a different solution to the problem. -Matthias On 5/20/24 9:27 AM, Kirk True wrote: +1 (non-binding) Thanks Alieh! On May 20, 2024, at 6:00 AM, Walker Carlson wrote: Hey Alieh, Thanks for the KIP. +1 bindin

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

e.org/37/javadoc/org/apache/kafka/connect/mirror/DefaultReplicationPolicy.html [3] - https://kafka.apache.org/37/javadoc/org/apache/kafka/connect/mirror/DefaultReplicationPolicy.html#SEPARATOR_CONFIG , https://kafka.apache.org/37/javadoc/org/apache/kafka/co

Re: [DISCUSS] Apache Kafka 3.9.0 release

In general I prefer to stick with the original time frame, but I believe an additional month would be good for KIP-848. Maybe we can cut down 2 weeks for the 4.1 and 4.2 releases each, to make up for the time? Not a must; just an idea. We can also just accept a one-time shift of the release ti

Re: [DISCUSS] Apache Kafka 3.8.0 release

optional in some sense and can be descoped if time runs out. For example, we can drop support for JDK 8 without immediately refactoring everything that could benefit from the improvements in JDK9+. best, Colin On Fri, Jun 14, 2024, at 15:37, Matthias J. Sax wrote: That's my understanding,

Re: [VOTE] 3.7.1 RC1

Replied on https://issues.apache.org/jira/browse/KAFKA-16960 On 6/14/24 6:05 AM, Luke Chen wrote: Hi all, After running system tests, here are failing tests: 1. kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests: KAFKA-16962 2.

Re: [DISCUSS] Apache Kafka 3.9.0 release

+1 Btw: I just looked into Jira, and we have 90 tickets with "fixed version" set to 4.0.0. I would assume that most of them need to be updated to 3.9.0? https://issues.apache.org/jira/browse/KAFKA-16924?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%204.0.0 For anybody who has context on

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

the old config if the new one is not set. If bothare set, the new one wins and we just log a WARN that the old one isignored in favor of the new one. -- It might be best to stick theestablished pattern?* Agree, we should follow the same pattern. Updated kip. Thanks, Murali On Fri, Jun 14, 2024 at

Re: [DISCUSS] Apache Kafka 3.8.0 release

That's my understanding, and I would advocate strongly to keep the 4.0 release schedule as planed originally. The 3.9 one should really be an additional "out of schedule" release not impacting any other releases. -Matthias On 6/14/24 1:29 PM, David Jacot wrote: The plan sounds good to me.

Re: [QUESTION] What to do about conflicted KIP numbers?

I don't think that there is an official guideline. Personally, I would suggest that the corresponding KIP owners agree who is keeping the conflicting number, and how is changing it. For the ones changing the number, I would propose to restart a new DISCUSS thread using the new number to separ

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

cope of the change. Similarly in "Test Plan" section Updated kip. 102: The "Deprecation" section should explain the behavior if both, old and new configs, are set. Updated kip. I think a ConfigException has to be thrown here if both configs are set. Thanks, Murali On

Re: [DISCUSS] KIP-1052: Align the naming convention for config and default variables in *Config classes

Might be good to restart a new discuss thread using the new KIP-1035 number to split both threads? I added the KIP to the table in the wiki page in the mean time. In Kafka Streams we did rename config names in the past in the exact same way as proposed: deprecate the old config, and add the

[jira] [Resolved] (KAFKA-16001) Migrate ConsumerNetworkThreadTest away from ConsumerTestBuilder

[ https://issues.apache.org/jira/browse/KAFKA-16001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16001. - Resolution: Fixed > Migrate ConsumerNetworkThreadTest away from ConsumerTestBuil

[jira] [Resolved] (KAFKA-13298) Improve documentation on EOS KStream requirements

[ https://issues.apache.org/jira/browse/KAFKA-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13298. - Fix Version/s: 3.1.0 Assignee: Andy Chambers Resolution: Fixed

Re: [DISCUSS] KIP-655: Add deduplication processor in kafka-streams

key to deduplication key, and second map to get back the initial key). Second map may imply a second repartitioning. However, if we do a per-task deduplication, the user may adapt to his specific use-case. Let me know what you think Ayoub Le mar. 11 juin 2024 à 20:21, Matthias J. Sax a écrit :

Re: Subscribe for discussion on Kafka® development

This is self service. Please follow the instructions from the web page: https://kafka.apache.org/contact -Matthias On 6/11/24 9:38 AM, Punsak Incham wrote: I want to subscribe for discussion on Kafka® development. Best regards, A black and blue logo Description automatically generated

Re: [VOTE] KIP-1035: StateStore managed changelog offsets

Thanks Nick. +1 (binding) Looking forward to get this all merged! -Matthias On 6/12/24 9:36 AM, Nick Telford wrote: Hi everyone, I'd like to call a vote on KIP-1035[1]. Regards, Nick 1: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets

Re: [VOTE] KIP-1049: Add config log.summary.interval.ms to Kafka Streams

+1 (binding) On 6/11/24 1:17 PM, Sophie Blee-Goldman wrote: +1 (binding) Thanks for the KIP! On Tue, Jun 11, 2024 at 5:37 AM jiang dou wrote: HI I would like to start a vote for KIP-1049: Add config log.summary.interval.ms to Kafka Streams KIP： https://cwiki.apache.org/confluence/display/

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

2024 at 5:07 PM Matthias J. Sax wrote: Shaswhat, any updates on this KIP? -- I still think that recommending to use `InternalFixedKeyRecordFactory` is not the best way to write test code. Changing `FixedKeyRecord` constructor (as I mentioned in my last email) might not be a good solution either.

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

er. I'm happy to defer to the experts in this case :) Thanks, Greg On Mon, Jun 10, 2024 at 3:47 PM Matthias J. Sax wrote: Thanks for the KIP Murali, Overall LGTM. A few comments. 100: Config names are part of the public interface, and the KIP should not say "none" in this section, b

[jira] [Resolved] (KAFKA-13561) Consider deprecating `StreamsBuilder#build(props)` function

[ https://issues.apache.org/jira/browse/KAFKA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13561. - Resolution: Duplicate > Consider deprecating `StreamsBuilder#build(props)` funct

[jira] [Created] (KAFKA-16945) Cleanup StreamsBuilder and TopologyConfig

Matthias J. Sax created KAFKA-16945: --- Summary: Cleanup StreamsBuilder and TopologyConfig Key: KAFKA-16945 URL: https://issues.apache.org/jira/browse/KAFKA-16945 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-15600) KIP-990: Capability to PAUSE Tasks on DeserializationException

[ https://issues.apache.org/jira/browse/KAFKA-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-15600. - Resolution: Won't Fix > KIP-990: Capability to PAUSE Tasks on DeserializationE

Re: [DISCUSS] KIP-655: Add deduplication processor in kafka-streams

not be detected. This is why I would have created two different methods, as is done for GroupBy: deduplicateByKey(...) deduplicate(...) If deduplicateByKey is used, the input stream does not need to be repartitioned. thanks Sébastien ____ De : Matthias J. Sax Envoyé

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

, it would become a non-user-facing / internal change to the "factory" we provide. -Matthias On 5/22/24 12:02 AM, Matthias J. Sax wrote: I was not aware of `InternalFixedKeyRecordFactory`. As the name indicates, it's considered an internal class, so not sure if we should recommend

Re: [DISCUSS] KIP-1049: Add config log.summary.interval.ms to Kafka Streams

Thanks for updating the KIP. Overall LGTM. However, "rejected alternatives" still says: add a new configuration log.summary.interval.ms to control the output interval of summary log This seems not to be rejected, but what the KIP actually proposes and should be removed? I think you can s

Re: [DISCUSS] KIP-655: Add deduplication processor in kafka-streams

rk on late records, or we discard them (which is the option I suggest). In the second case, It would be up to the user to choose any deduplicationInterval that may sufficiently cover all his late data. What do you think ? Thanks, Ayoub Le mar. 4 juin 2024 à 23:58, Matthias J. Sax a écrit

Re: [DISCUSS] KIP-1050: Consistent error handling for Transactions

Thanks for this KIP. Great to see it. I would assume it will make KIP-691 unnecessary? I don't think I fully understand the proposal yet. It's clear, that you propose to add new sub-classed to group existing exceptions. But it's not clear to me, which of the existing exceptions (which implemen

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

Thanks for the KIP Murali, Overall LGTM. A few comments. 100: Config names are part of the public interface, and the KIP should not say "none" in this section, but call out which configs are deprecated and which ones are newly added. 101: Nit. In "Propose Changes" there is the template pl

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

o completion that we can guarantee getting it all done by the next release. -- Cheers, Nick On Tue, 4 Jun 2024 at 20:34, Matthias J. Sax wrote: Nick, Thanks a lot for updating the KIP. I made a pass over it. Overall LGTM. A few nits and some more minor questions: 200: nit (Javadocs for `

[jira] [Resolved] (KAFKA-14567) Kafka Streams crashes after ProducerFencedException

[ https://issues.apache.org/jira/browse/KAFKA-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-14567. - Assignee: (was: Kirk True) Resolution: Fixed Not 100% sure either, but I feel

[jira] [Resolved] (KAFKA-16911) Kafka Streams topology optimization docs incomplete

[ https://issues.apache.org/jira/browse/KAFKA-16911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16911. - Fix Version/s: 3.8.0 Resolution: Fixed > Kafka Streams topology optimizat

[jira] [Created] (KAFKA-16911) Kafka Streams topology optimization docs incomplete

Matthias J. Sax created KAFKA-16911: --- Summary: Kafka Streams topology optimization docs incomplete Key: KAFKA-16911 URL: https://issues.apache.org/jira/browse/KAFKA-16911 Project: Kafka

[jira] [Resolved] (KAFKA-16903) Task should consider producer error previously occurred for different task

[ https://issues.apache.org/jira/browse/KAFKA-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16903. - Resolution: Fixed > Task should consider producer error previously occurred

Re: [DISCUSS] KIP-1049: Add config log.summary.interval.ms to Kafka Streams

Jiang, Thanks for the KIP. I think it make sense. I agree with Sophie that the KIP writeup should be improved a little bit, with regard to public API (ie, config) which are changed. The only other idea I had to avoid this issue would be some internal change: we would introduce a new logger c

Re: [DISCUSS] KIP-655: Add deduplication processor in kafka-streams

Ayoub, thanks for resurrecting this KIP. I think a built-in de-duplication operator will be very useful. Couple of questions: 100: `deduplicationKeySelector` Is this the best name? It might indicate that we select a "key" what is an overloaded term... Maybe we could use `Field` or `Id` o

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

t got around to updating the KIP yet. Now that I've wrapped up KIP-989, I'm going to be working on 1035 starting today. I'll update the KIP first, and then call a vote. Regards, Nick On Wed, 29 May 2024 at 07:25, Bruno Cadonna wrote: Totally agree on moving forward and starting

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

codes listed in the KIP: enum AssignmentError { NONE, ACTIVE_TASK_ASSIGNED_MULTIPLE_TIMES, ACTIVE_AND_STANDBY_TASK_ASSIGNED_TO_SAME_KAFKASTREAMS, INVALID_STANDBY_TASK, UNKNOWN_PROCESS_ID, UNKNOWN_TASK_ID } Anything missing? (also updated all the code

Re: Hello world - please subscribe me!

Chris, Subscription is self-service. Please follow the instruction from the web page: https://kafka.apache.org/contact -Matthias On 6/3/24 1:59 AM, Chris wrote:

[jira] [Created] (KAFKA-16863) Consider removing `default.` prefix for exception handler config

Matthias J. Sax created KAFKA-16863: --- Summary: Consider removing `default.` prefix for exception handler config Key: KAFKA-16863 URL: https://issues.apache.org/jira/browse/KAFKA-16863 Project

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

dering if opening the state stores of unassigned tasks whose state directory exists locally is actually such a big issue due to the expected higher latency since it happens actually before the Kafka Streams client joins the rebalance. Best, Bruno On 5/4/24 12:05 AM, Matthias J. Sax wrote: T

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

/blob/main/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java On Mon, May 20, 2024 at 9:05 PM Matthias J. Sax wrote: Had a discussion on https://issues.apache.org/jira/browse/KAFKA-15242 and it was pointed out, that we also need to do something about `FixedKeyRecord`. It does not h

Re: Permission to contribute to Apache Kafka

You should be all set. On 5/21/24 8:30 AM, Harry Fallows wrote: Hello, I am following the[Getting Started guide for writing KIPs](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals). Could someone give me the permissions to write a KIP please? My Wiki ID is harryfa

Re: Request for Authorization to Create KIP

You should be all set. On 5/21/24 6:58 PM, 黃竣陽 wrote: I want to create a KIP, and my wiki id : m1a2st and Jira id : m1a2st, Thanks for your help. jiang dou 於 2024年5月22日 上午9:01 寫道： You should send your jira ID and wiki ID， Please refer to this address ： https://cwiki.apache.org/confluence/dis

[jira] [Resolved] (KAFKA-15242) FixedKeyProcessor testing is unusable

[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-15242. - Assignee: (was: Alexander Aghili) Resolution: Duplicate > FixedKeyProces

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

would extend `FixedKeyRecord`? -- Or maybe an even better solution? I could not think of something else so far. Thoughts? On 5/3/24 9:46 AM, Matthias J. Sax wrote: Please also update the KIP. To get a wiki account created, please request it via a commet on this ticket: https

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

the age, this does have the benefit of a more obvious metric name. Nit: the KIP says it's a store-level metric, but I think it would be good to say explicitly that it's recorded with DEBUG level only? Yes, I've already updated the KIP with this information in the table. Rega

Re: [VOTE] KIP-989: RocksDB Iterator Metrics

+1 (binding) On 5/14/24 9:19 AM, Lucas Brutschy wrote: Hi Nick! Thanks for the KIP. +1 (binding) On Tue, May 14, 2024 at 5:16 PM Nick Telford wrote: Hi everyone, I'd like to call a vote on the Kafka Streams KIP-989: RocksDB Iterator Metrics: https://cwiki.apache.org/confluence/display/KAF

Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

+1 (binding) On 5/13/24 5:54 PM, Sophie Blee-Goldman wrote: Thanks for the KIP guys! +1 (binding) On Mon, May 13, 2024 at 6:02 AM Bill Bejeck wrote: Thanks for the KIP, this will be a great addition! +1(binding) -Bill On Fri, May 3, 2024 at 4:48 AM Bruno Cadonna wrote: Hi Damien, Séba

Re: Creating kafka wiki id

Self-service to create an account is currently not working. Please reply on https://issues.apache.org/jira/browse/INFRA-25451 to request a wiki account. I'll update the wiki page for now until the issue is resolved. -Matthias On 5/7/24 8:25 AM, 黃竣陽 wrote: Hello, I want to create a KIP, but I

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

liztionExceptionHandler. Whereas serialization exceptions are just a subset of what might get passed in to the ProductionExceptionHandler... Just explaining my reasoning -- in the end I leave it up to the KIP authors and anyone who will actually be using this feature in their applications :)

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

adding a producer config property is the cleanest one. - changed the *ClientExceptionHandler* to *ProducerExceptionHandler* to be closer to what we are changing. - added the ProducerRecord as the input parameter of the handle() method as well. - increased the response types to 3 to have

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

@Loic, yes, what you describe is exactly what I had in mind. @Sophie, can you elaborate a little bit? First of all, I agree that it makes sense to maintain the two separate callbacks for the ProductionExceptionHandler, since one of them is specifically for serialization exceptions while the o

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

Best, Bruno On 5/3/24 6:29 PM, Matthias J. Sax wrote: 101: Yes, but what I am saying is, that we don't need to flush the .position file to disk periodically, but only maintain it in main memory, and only write it to disk on close() to preserve it across restarts. This way, it would

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

, May 1, 2024 at 11:00 PM Matthias J. Sax wrote: Any updates on this KIP? On 3/28/24 4:11 AM, Matthias J. Sax wrote: It seems that `MockRecordMetadata` is a private class, and thus not part of the public API. If there are any changes required, we don't need to discuss on the KIP.

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

/24 2:43 AM, Matthias J. Sax wrote: Thanks Sophie. My bad. You are of course right about `TaskAssignment` and the StreamsPartitionAssignor's responsibitliy to map tasks of a instance to consumers. When I wrote my reply, I forgot about this detail. Seems you did not add `UNKNOWN_TASK_ID` error ye

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L397 On 5/1/24 3:00 AM, Matthias J. Sax wrote: Thanks Bruno. 101: I think I understand this better now. But just want to make sure I do. What do you mean by "they can diverge" and "Recovering after a failure might

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

+1 (binding) On 5/3/24 8:52 AM, Federico Valeri wrote: Hi Fred, this is a useful addition. +1 non binding Thanks On Fri, May 3, 2024 at 4:11 PM Andrew Schofield wrote: Hi Fred, Thanks for the KIP. It’s turned out nice and elegant I think. Definitely a worthwhile improvement. +1 (non-bind

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

What about (106) to unify both exiting callback methods of `ProductionExceptionHandler` into a single one, instead of adding two new ones? Damien's last reply about it was: I will think about unifying, I do agree it would be cleaner. There was not follow up on this question, and the KIP rig

Re: [VOTE] KIP-924: customizable task assignment for Streams

I left one more nit on the discuss thread. But overall LGTM. +1 (binding) Thanks Rohan and Sophie for driving this KIP. -Matthias On 4/29/24 2:07 PM, Sophie Blee-Goldman wrote: +1 (binding) thanks for driving this KIP! On Tue, Apr 16, 2024 at 1:46 PM Rohan Desai wrote: https://cwiki.ap

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

f: * I think some bullet points under `User APIs` and `Read-only APIs` should have a lower level indention? It caught me for a sec until I realized there are just two categories. * In TaskAssignmentUtils , why not let those util functions return `TaskAssignment` (to me it feels more consi

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

Any updates on this KIP? On 3/28/24 4:11 AM, Matthias J. Sax wrote: It seems that `MockRecordMetadata` is a private class, and thus not part of the public API. If there are any changes required, we don't need to discuss on the KIP. For `CapturedPunctuator` and `CapturedForward` i

[jira] [Resolved] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete

[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16644. - Resolution: Duplicate > FK join emits duplicate tombstone on left-side del

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Thanks for the update. I am wondering if we should use `ReadOnlyHeaders` instead of `ImmutableHeaders` as interface name? Also, the returned `Header` interface is technically not immutable either, because `Header#key()` returns a mutable byte-array... Would we need a `ReadOnlyHeader` interfa

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

104 I agree that we should try to support downgrades without wipes. At least Nick should state in the KIP why we do not support it. Best, Bruno On 4/23/24 8:13 AM, Matthias J. Sax wrote: Thanks for splitting out this KIP. The discussion shows, that it is a complex beast by itself, so worth

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

I like the idea of error codes. Not sure if the name are ideal? UNKNOWN_PROCESS_ID makes sense, but the other two seems a little bit difficult to understand? Should we be very descriptive (and also try to avoid coupling it to the threading model -- important for the first error code): - ACTI

[jira] [Created] (KAFKA-16644) FK join emit duplicate tombstone on left-side delete

Matthias J. Sax created KAFKA-16644: --- Summary: FK join emit duplicate tombstone on left-side delete Key: KAFKA-16644 URL: https://issues.apache.org/jira/browse/KAFKA-16644 Project: Kafka

Wiki account sign-up

Hi, as many of you know, there is a issue with creating new wiki accounts right now: https://issues.apache.org/jira/browse/INFRA-25451 A fix might take some more time. In the meantime, please reply to the INFRA ticket with your email address, and accounts will be created manually in the mean

Re: Assistance Needed with Creating Wiki ID for Kafka Contribution

It's a known issue and INFRA is working on a solution: https://issues.apache.org/jira/browse/INFRA-25451 In the mean time, users can be added manually (cf the ticket, and reply there to get added). -Matthias On 3/28/24 5:10 AM, Prashant Jagtap wrote: Hi, I hope this email finds you well.

Re: Confluence edit access

Thanks. You should be all set. -Matthias On 4/25/24 10:49 PM, Claude Warren wrote: My Confluence ID is "claude" On Thu, Apr 25, 2024 at 8:40 PM Matthias J. Sax wrote: What's your wiki ID? We can grant write access on our side if you have already an account. -Matthias On

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

ratch. WDYT? 117. That's fair. I agree this is probably the most tricky of the open questions, and I'm happy to defer. Although I do strongly believe that we shouldn't call certain kinds of assignments "invalid" (such as a task not being assigned to anyone). For the clearly-in

Re: Confluence edit access

What's your wiki ID? We can grant write access on our side if you have already an account. -Matthias On 4/25/24 4:06 AM, Claude Warren wrote: I would like to get edit access to the Kafka confluence so that I can work on KIP-936. Can someone here do that or do I need to go through Infra? Clau

<    1   2   3   4   5   6   7   8   9   10   >