Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

Hi Dongjin, Sorry for the delay. I'm glad you're still pushing this forward. It would be nice to get this in to the 2.7 release. I just took another look at the KIP, and it looks good to me! I think this is ready for a vote. Thanks, -John On Wed, 2020-08-05 at 22:04 +0900, Dongjin Lee wrote:

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #53

See Changes: [github] MINOR: remove DelayedOperations.checkAndCompleteFetch (#9278) -- [...truncated 3.28 MB...]

Re: [DISCUSS] KIP idea: Support of multipart messages

Thanks for the KIP. I think the main question is what's the upper bound for message size you are looking to support. Is it hundreds of MBs, GBs, tens of GBs, something else? That would inform the options. Ismael On Thu, Sep 10, 2020 at 8:03 AM Gwen Shapira wrote: > There is another option of

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #54

See Changes: [github] MINOR: remove DelayedOperations.checkAndCompleteFetch (#9278) -- [...truncated 3.29 MB...]

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

I was just thinking that the processor would null out the record context after it finished processing the record, so I'm not sure I follow why this would not be possible? AFAIK we never call a punctuator in the middle of processing a record through the topology, and even if we did, we still know

Jenkins build is back to normal : Kafka » kafka-trunk-jdk8 #52

See

[jira] [Resolved] (KAFKA-10311) Flaky test KafkaAdminClientTest#testMetadataRetries

[ https://issues.apache.org/jira/browse/KAFKA-10311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-10311. - Resolution: Fixed > Flaky test KafkaAdminClientTest#testMetadataRetries >

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

Thanks for the reply, Paul! I certainly intend to make sure that the changelogging layer continues to work the way it does now, by hook or by crook. I think the easiest path for me is to just "cheat" and get the real ProcessorContext into the ChangeLoggingStore implementation somehow. I'll tag

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

Thanks, Sophie, Yes, now that you point it out, I can see that the record context itself should be nulled out by Streams before invoking punctuators. From that perspective, we don't need to think about the second-order problem of what's in the context for the state store when called from a

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

Ah, thanks Sophie, I'm sorry for misinterpreting your resonse. Yes, we absolutely can and should clear the context before punctuating. My secondary concern is maybe more far-fetched. I was thinking that inside process(key,value), a Processor might do a get/put of a _different_ key. Consider, for

Re: [DISCUSS] Apache Kafka 2.7.0 release

Hi Gary, It's been added. Regards, Bill On Thu, Sep 10, 2020 at 4:14 PM Gary Russell wrote: > Can someone add a link to the release plan page [1] to the Future Releases > page [2]? > > I have the latter bookmarked. > > Thanks. > > [1]: >

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

> > when you use a HashMap or RocksDB or other "state stores", you don't > expect them to automatically know extra stuff about the record you're > storing. So, I don't think there is any reason we *can't* retain the record context > in the StateStoreContext, and if any users came along with a

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

Thanks for the conversation, Sophie! Sorry for the ambiguity I introduced into it, though. Thanks, -John On Thu, 2020-09-10 at 16:10 -0700, Sophie Blee-Goldman wrote: > Aha, I did misinterpret the example in your previous response regarding the > range query after all. I thought you just meant a

Re: [DISCUSS] Apache Kafka 2.7.0 release

Can someone add a link to the release plan page [1] to the Future Releases page [2]? I have the latter bookmarked. Thanks. [1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158872629 [2]: https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #55

See Changes: [github] KAFKA-10134: Enable heartbeat during PrepareRebalance and Depend On State For Poll Timeout (#8834) -- [...truncated 3.29 MB...]

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

> > Regarding your first sentence, "...the processor would null > out the record context...", this is not possible, since the > processor doesn't have write access to the context. We could > add it, > Sorry, this was poorly phrased, I definitely did not mean to imply that we should make the

Re: KIP-669: Preserve Source Partition in Kafka Streams from context

Hey Balan, thanks for the KIP! The motivation here makes sense to me, but I have a few questions about the proposed API I guess the main thing to point out is that if we just add new addSink() overloads to Topology, then only the lower level Processor API will benefit and users of the DSL won't

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #54

See Changes: [github] MINOR: Fix JSON generation of nested structs with non-matching type/name (#9277) [github] MINOR: Address flaky `KafkaAdminClient` tests (#9091) [github] MINOR: Add debug logs

Re: [DISCUSS] KIP idea: Support of multipart messages

On 2020/09/04 07:17:36, Alexander Sibiryakov wrote: > Hello, > > I would like to get your opinions on this KIP idea. > > In short it will allow to transfer messages of bigger size than allowed by > the broker. > >

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

Aha, I did misinterpret the example in your previous response regarding the range query after all. I thought you just meant a time-range query inside a punctuator. It genuinely did not occur to me that users might be looking up and/or updating records of other keys from within a Processor. Sorry

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #55

See Changes: [github] MINOR: Fix JSON generation of nested structs with non-matching type/name (#9277) -- [...truncated 6.57 MB...]

Jenkins build is back to normal : Kafka » kafka-trunk-jdk15 #56

See

[jira] [Created] (KAFKA-10475) Using same key reports different count of records for groupBy() and groupByKey() in Kafka Streaming Application

Saad Rasool created KAFKA-10475: --- Summary: Using same key reports different count of records for groupBy() and groupByKey() in Kafka Streaming Application Key: KAFKA-10475 URL:

[jira] [Created] (KAFKA-10474) Kafka Java client introduces CPU overhead when there are many consumers

Bozhidar Bozhanov created KAFKA-10474: - Summary: Kafka Java client introduces CPU overhead when there are many consumers Key: KAFKA-10474 URL: https://issues.apache.org/jira/browse/KAFKA-10474

Ping about KAFKA-7421

Hi, guys! Please, could you help us with  https://issues.apache.org/jira/browse/KAFKA-7421  ? We face it very often, it’s really sad for us and our customers.   There are discussion and suggestions in the ticket comments, but nobody have replied in long time. Do you agree that synchronization

Re: [DISCUSS] KIP idea: Support of multipart messages

Hey Ben, thanks for the link. My proposal is partially based on Becket's ideas, but I haven't reached out to him directly. +Becket Hi Becket, would you mind to have a look at my proposal (link is in the first message) ? A. On Tue, Sep 8, 2020 at 12:35 PM Ben Stopford wrote: > LinkedIn had

Re: [DISCUSS] KIP idea: Support of multipart messages

There is also another approach (harder to design, but may be easier to use and maintain), which is to make Kafka handle large messages better and allow users to set higher limits - for example, can Kafka provide really high throughput with 1GB messages? Some systems do it well. I don't know where

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

Sounds good to me, thanks! On Wed, Sep 9, 2020 at 5:30 PM Jason Gustafson wrote: > Hey Tom, > > Yeah, that's fair. I will update the proposal. I was also thinking of > adding a separate column for duration, just to save users the trouble of > computing it. > > Thanks, > Jason > > On Wed, Sep 9,

Re: [DISCUSS] KIP-516: Topic Identifiers

Thanks Lucas/Justine for the nice KIP. It has several benefits which also include simplifying the topic deletion process by controller and logs cleanup by brokers in corner cases. Best, Satish. On Wed, Sep 9, 2020 at 10:07 PM Justine Olshan wrote: > > Hello all, it's been almost a year! I've

Re: KIP-669: Preserve Source Partition in Kafka Streams from context

Forgot to add the link https://cwiki.apache.org/confluence/display/KAFKA/KIP-669%3A+Preserve+Source+Partition+in+Kafka+Streams+from+context On 2020/09/10 13:40:02, satyanarayan komandur wrote: > Hi, > > I have submitted a new KIP for preserving processor record context partition > from

Re: [DISCUSS] KIP idea: Support of multipart messages

There is another option of doing the splitting on the server and hiding this from the clients. My personal (and highly controversial) take is that Kafka clients could use less complexity rather than more. They are incredibly difficult to reason about as is. But maybe this splitting/merging won't

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

Thanks for the thoughts, Sophie. I agree that the extra information could be useful. My only concern is that it doesn’t seem like we can actually supply that extra information correctly. So, then we have a situation where the system offers useful API calls that are only correct in a narrow

KIP-669: Preserve Source Partition in Kafka Streams from context

Hi, I have submitted a new KIP for preserving processor record context partition from source. I am looking for suggestions/comments. In most use cases where source message is getting transformed and sent to a target topic, where 1. number of partitions on source and sink topic are same 2.