[DISCUSS] KIP-428: Add in-memory window store

Streams currently only has support for a RocksDB window store, but users have been requesting an in-memory version. This KIP introduces a design for an in-memory window store implementation. https://cwiki.apache.org/confluence/display/KAFKA/KIP-428%3A+Add+in-memory+window+store

Re: [DISCUSS] KIP-428: Add in-memory window store

store. > > > > Thanks, > > Dongjin > > > > On Fri, Feb 8, 2019 at 1:41 PM Boyang Chen wrote: > > > > > Thanks Sophie for proposing this new feature! In-memory window store is > > > very useful in long term. One meta comment is that we don'

[VOTE] KIP-428: Add in-memory window store

Hi all, I would like to call for a vote on KIP-428 regarding adding an in-memory version of the window store. The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-428%3A+Add+in-memory+window+store Cheers, Sophie

Re: [VOTE] KIP-428: Add in-memory window store

4, 2019 at 8:59 PM Guozhang Wang wrote: > > > +1 (binding). > > > > On Thu, Feb 14, 2019 at 4:07 PM Matthias J. Sax > > wrote: > > > > > +1 (binding) > > > > > > > > > -Matthias > > > > > > On 2/14/19 3:36 P

Re: [DISCUSS] KIP-439: Deprecate Interface WindowStoreIterator

I remember thinking this while working on window stores, am definitely for it. On Mon, Mar 11, 2019 at 9:20 AM John Roesler wrote: > Sounds great to me. Thanks, Matthias! > -John > > On Sun, Mar 10, 2019 at 11:58 PM Matthias J. Sax > wrote: > > > Hi, > > > > I would like to propose KIP-439 to d

[DISCUSS] KIP-445: In-memory Session Store

Hello all, This KIP proposes adding an in-memory session store to round-out the API and support built-in persistent and in-memory versions of ALL store types. Any feedback is appreciated https://cwiki.apache.org/confluence/display/KAFKA/KIP-445%3A+In-memory+Session+Store I made a first pass at a

[DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

Hello all, I would like to kick off discussion of this KIP aimed at providing sliding window semantics to DSL aggregations. https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL Please take a look and share any thoughts you have regarding the API, se

[VOTE] KIP-445: In-memory session store

Hello all, There has been a positive reception so I'd like to call for a vote on KIP-445, augmenting our session store options with an in-memory version. This would round out our store API to offer in-memory and persistent versions of all three types of stores. KIP: https://cwiki.apache.org/confl

Re: [DISCUSS] KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils

Hi Yishun, thanks for the KIP! I have a few initial questions/comments: 1) It may be useful to capture the iterator results as well (eg with a MockIterator that wraps the underlying iterator and records the same way the MockStore wraps/records the underlying store) 2) a. Where is the "persistent"

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

memory while making other buckets optionally on persistent > storage. In practice, as long as M is large enough (we probably need it to > be large enough to have sufficiently sensitive expiration anyways) then > each bucket's aggregate data is small enough to be in memory. > This sou

Re: [VOTE] KIP-445: In-memory session store

phie! > > > > > > +1 (binding) > > > > > > > > > -Matthias > > > > > > On 4/8/19 5:26 PM, Sophie Blee-Goldman wrote: > > > > Hello all, > > > > > > > > There has been a positive reception so I'

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

(total_aggregate, bucket_aggregate); with 2) when expiring the > > oldest bucket we can re-compute the total_aggregate by > > sum(bucket_aggregate) over other buckets again. > > > > 6. Meta comment: it is reasonable to assume in practice out-of-ordering > > data is not very common, hence mos

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Hey Boyang, I think this sounds great but one thing you might want to update is the "Compatibility, Deprecation, and Migration Plan" -- I agree having two instances in the same JVM is probably a rare occurrence but the (presumably less rare) situation Matthias described would also be affected in c

Re: [DISCUSSION] KIP-418: A method-chaining way to branch KStream

I'd like to +1 what Michael said about the issues with the existing branch method, I agree with what he's outlined and I think we should proceed by trying to alleviate these problems. Specifically it seems important to be able to cleanly access the individual branches (eg by mapping name->stream),

Re: [DISCUSS] 2.2.1 Bug Fix Release

Hey Vahid, We also have another minor bug fix we just uncovered and are hoping to get in today although I don't think there's a ticket for it atm...just waiting for the build to pass. Thanks for volunteering! Cheers, Sophie On Fri, May 3, 2019 at 3:16 PM Vahid Hashemian wrote: > Hi John, > >

[VOTE] KIP-453: Add close() method to RocksDBConfigSetter

Hi all, I'd like to call for a vote on a minor KIP that adds a close() method to the RocksDBConfigSetter interface of Streams. Link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-453%3A+Add+close%28%29+method+to+RocksDBConfigSetter This is important for users who have created RocksOBject

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hi Daniyar, Thanks for the KIP! I had to write my own List serde for testing a while back and this definitely would have saved me some time. Regarding the static declaration, I believe you're missing a between "public static" and the return type "Serde>" -- Java should allow this Cheers, Sophie

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Good point about serdes for other Collections. On the one hand I'd guess that non-List Collections are probably relatively rare in practice (if anyone disagrees please correct me!) but on the other hand, a) even if just a small number of people benefit I think it's worth the extra effort and b) if

Re: [VOTE] KIP-453: Add close() method to RocksDBConfigSetter

actical approach. > > > > > > > > +1 (non-binding) > > > > > > > > On Mon, May 6, 2019 at 2:23 PM Matthias J. Sax < > matth...@confluent.io> > > > > wrote: > > > > > > > > > +1 (binding) > > &

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

Hey Bruno! This all looks pretty good to me, but one suggestion I have is to supplement each of the metrics with some info on how the user can control them. In other words, which options could/should they set in RocksDBConfigSetter should they discover a particular bottleneck? I don't think this

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

how much of it is allocated to each. I'm not sure how common this really is but I think it would be invaluable to those who do. RocksDB performance can be quite opaque.. Cheers, Sophie On Fri, May 17, 2019 at 5:01 PM Sophie Blee-Goldman wrote: > Hey Bruno! > > This all looks pre

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

; > available RocksDB metrics, and I'm not sure. But users can control >> how >> > > many L0 files it takes to trigger compaction so if it is possible; it >> may >> > > be useful. >> > > >> > > Thanks, >> > > Bil

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

th you and Sophie about user education and that we should > > > work on it after this KIP. > > > > > > 4. I agree also on the user API. However, I would like to open a > > > separate KIP for it because I still need a bit of thinking to get it. > > > I also

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

ut it is 1) future-proof if we want to > > consolidate to 1-1 physical store -> logical store implementation, and 2) > > it is as simple and not needing to bookkeep older segments who should be > > rarely accessed. My question is though, if upon segment rolling our > metrics > >

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

Hi Patrik, As of 2.3 you will be able to use the RocksDBConfigSetter to effectively bound the total memory used by RocksDB for a single app instance. You should already be able to limit the memory used per rocksdb store, though as you mention there can be a lot of them. I'm not sure you can monito

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

Hey John, thanks for writing this up! I like the proposal but there's one point that I think may be too restrictive: "A processor that happens to use a typed store is actually emitting the same types that it is storing." I can imagine someone could want to leverage this new type safety without al

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

We would like to tack on some rebalance-related metrics as part of this KIP as well. The details can be found in the sub-task JIRA: https://issues.apache.org/jira/browse/KAFKA-8609 On Thu, May 30, 2019 at 5:09 PM Guozhang Wang wrote: > +1 (binding) from me as well. > > Thanks to everyone who hav

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

, Jun 27, 2019 at 8:15 PM Boyang Chen wrote: > Thank you Sophie for the update. Is this also reflected on the KIP? > > On Thu, Jun 27, 2019 at 3:28 PM Sophie Blee-Goldman > wrote: > > > We would like to tack on some rebalance-related metrics as part of this > KIP > > as

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

I'm +1 on Windowed, was about to suggest that as I was catching up on the discussion but Bill beat me to it :) On Tue, Jul 16, 2019 at 2:23 PM Bill Bejeck wrote: > Hi John, > > Thanks for the updates. > > I like RunningCount and RunningSum. > > What about WindowedCount, WindowedSum instead of Mo

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

from my understanding. > > > > The issue is still, that `Sum` and `Count` which seem to be the cleanest > > names cannot be used. While I agree that `TotalSum` and `TotalCount` is > > somewhat redundant, I still think it the best suggestion so far. > > > > For the &

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hey Levani, Thanks for the KIP! Can you clarify one thing for me -- for the KStream#repartition signature taking a Repartitioned, will the topic be auto-created by Streams (which seems to be the case for the signature without a Repartitioned) or does it have to be pre-created? The wording in the K

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

is the > unweighted version? In that case, I buy the argument that we don't > need "Simple" and we can just go with: > > WindowedSum, WindowedCount > CumulativeSum, CumulativeCount > > Sound good? > Thanks, > -John > > On Wed, Jul 17, 2019 at 11:53 AM

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Regards, > Levani > > > On Jul 17, 2019, at 9:02 PM, Sophie Blee-Goldman > wrote: > > > > Hey Levani, > > > > Thanks for the KIP! Can you clarify one thing for me -- for the > > KStream#repartition signature taking a Repartitioned, will the topic be > >

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

> >>>>> > >>>>> Does that make sense, or did I maybe miss something important? > >>>>> > >>>>> Thanks, > >>>>> -John > >>>>> > >>>>> On Wed, Jul 17, 2019 at 2:48 PM Le

Re: [VOTE] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

Looks good! Thanks Levani, +1 (non-binding) Sophie On Tue, Jul 23, 2019 at 2:16 PM Levani Kokhreidze wrote: > Hello, > > I’d like to initialize voting on KIP-221: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint

[DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hey all, I'd like to kick off discussion on KIP-441, aimed at the long restore times in Streams during which further active processing and IQ are blocked. Please give it a read and let us know your thoughts https://cwiki.apache.org/confluence/display/KAFKA/KIP-441:+Smooth+Scaling+Out+for+Kafka+St

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

to the exact line and better > > > > > displayed there. > > > > > > > > > > 3.a Another wild thought about the threshold itself: today the > > assignment > > > > > itself is memoryless, so we would not know if th

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

; then active and standbys share almost nothing. But that seems like follow-up work to the current KIP :) On Thu, Aug 8, 2019 at 5:31 PM Sophie Blee-Goldman wrote: > Stateful tasks with logging disabled seem to be an interesting edge case. > On the one hand, > for balancing purposes they sh

Re: [DISCUSS] KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils

#x27;t have any backward compatibility > > >>> guarantees. > > >>>>> > > >>>>> Hence, I would suggest that the corresponding classes go into a new > > >>>>> package `org.apache.kafka.streams.state`. > > >>>>> >

Re: [DISCUSS] KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils

ing a more comprehensive mock store? > > Thanks, > Yishun > > On Thu, Aug 29, 2019 at 12:18 AM Sophie Blee-Goldman > wrote: > > > > Hey Yishun! Glad to see this is in the works :) > > > > Within the past month or so, needing state stores for unit tests has b

Re: [VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

+1 (non-binding) On Wed, Sep 11, 2019 at 11:38 AM Vinoth Chandar wrote: > +1 (non-binding). > > On Fri, Sep 6, 2019 at 12:46 AM Bruno Cadonna wrote: > > > +1 (non-binding) > > > > On Fri, Sep 6, 2019 at 12:32 AM Guozhang Wang > wrote: > > > > > > +1 (binding). > > > > > > On Thu, Sep 5, 2019 a

Re: [DISCUSS] 2.2.2 Bug Fix Release

Matthias is at Kafka Summit, we should be able to get the fix for it merged by the end of today though. Will let you know when it's done. Thanks! Sophie On Sat, Sep 28, 2019 at 8:24 PM Randall Hauch wrote: > Sounds fine, Matthias. Do you have an ETA for the fix? > > On Sat, Sep 28, 2019 at 12:5

Re: [DISCUSS] 2.2.2 Bug Fix Release

The fix <https://github.com/apache/kafka/pull/7426> has been merged to the 2.2 branch. Thanks! Sophie On Mon, Sep 30, 2019 at 11:03 AM Sophie Blee-Goldman wrote: > Matthias is at Kafka Summit, we should be able to get the fix for it > merged by the end of today though. Will let y

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

consider `Repartitioned` as an "extension" to > `Produced`, > >>>>>> that adds topic configuration, is a good way to think about it and > helps > >>>>>> to keep the API "clean". > >>>>>> > >>>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Hey Walker, Thanks for the KIP! I have just a couple of questions: 1) It seems a little awkward to me that with the current API, we have a nearly identical "add stream to cogroup" method, except for the first which has a different signature (ie the first stream is joined as stream.cogroup(Aggrega

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

ject made in the initializer. > Does that make sense? > > This is a good question and I will include this explanation in the kip as > well. > > Thanks, > Walker > > On Tue, Oct 22, 2019 at 8:59 PM Sophie Blee-Goldman > wrote: > > > Hey Walker, > > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Initializer` is a mandatory parameter and not a > configuration and should be passed directly, and not via a configuration > object. > > > -Matthias > > On 10/23/19 11:37 AM, Sophie Blee-Goldman wrote: > > Thanks for the explanation, makes sense to me! As for the API, one

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Just to chime in on the "report lag vs timestamp difference" issue, I would actually advocate for both. As mentioned already, time difference is probably a lot easier and/or more useful to reason about in terms of "freshness" of the state. But in the case when all queried stores are far behind, lag

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

eadStateStoreProvider.java > > > > > > > (which > > > > > > > > will return all active stores which are running/restoring and > > > replica > > > > > > > > stores which are running), we will add new functions and not >

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

? > > > > 4. There needs to be some caching/interval somewhere though since we > don't > > want to make 1 kafka read per 1 IQ potentially. But I think its a valid > > suggestion, to make this call just synchronous and leave the caching or > how > > often you

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

e to > add `numberOfPartitions` field to `Grouped` class instead of introducing > new `KStream#groupBy(Repartitioned)` method overload. > > I would really appreciate communities feedback on this. > > Kind regards, > Levani > > > > > On Oct 17, 2019, at 12:57 AM, Sophie

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi Manikumar, We have another potential blocker in 2.4, which affects Streams members that fall out of the group and may cause them to die upon rejoin. The ticket is here https://issues.apache.org/jira/browse/KAFKA-9178 The fix is very straightforward, and we have a small PR out for it here http

Re: [DISCUSS] Apache Kafka 2.6.0 release

> > > > > already complete will need to push out to the next release > (either > > > 2.7 > > > > or > > > > > > 3.0). There are a few PRs for KIPs that are nearing completion, > and > > > > we're > > > > > > having s

Re: [DISCUSS] Apache Kafka 2.6.0 release

eated > as a > > > > > blocker for 2.6.0, please mark the issue's `fix version` as `2.6.0` > > > _and_ > > > > > respond to this thread with details, and I will work with you to > > > > determine > > > > > whether it is indeed a blocker. > >

Re: [DISCUSS] KIP-450: Sliding Windows

Thanks Leah! This kind of assumes an implicit answer to Matthias's question, but I was wondering if we should take this opportunity to choose a better default value for the grace period. Note that the default of -1 in the TimeWindows class, for example, ultimately gets translated into a default va

Re: [VOTE] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

Hey Jorge, Thanks for the reminder -- +1 (non-binding) Cheers, Sophie On Thu, Jul 16, 2020 at 4:06 PM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Bumping this vote thread to check if there's any feedback. > > Cheers, > Jorge. > > On Sat, Jul 4, 2020 at 6:20 PM John Roesler

Re: [DISCUSS] KIP-645: Replace abstract class Windows with a proper interface

Hey John, Thanks for the KIP. I know this has been bugging you :) That said, I think the KIP is missing some elaboration in the Motivation section. You mention a number of problems we've had and lived with in the past -- could you give an example of one, and how it would be solved by your proposa

Re: [DISCUSS] KIP-450: Sliding Windows

Hey John, Just a few follow-up questions/comments about the whole Windows thing: That's a good way of looking at things; in particular the point about SessionWindows for example requiring a Merger while other "statically enumerable" windows require only an adder seems to touch on the heart of the

Re: [DISCUSS] KIP-450: Sliding Windows

#x27;t > > know the user's intent. > > > > As a matter of fact, even if the user might not be aware of it, the > > algorithm we use does already leak into the API (if a user extends > > `Windows` is must work with our hopping/tumbling window algorithm

Re: Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

some discussion about > > >> RocksDB metrics, I decided to make some major modifications to this > KIP > > >> and kick off discussion again. > > >> > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAF

Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Thanks all, +1 (non-binding) Cheers, Sophie On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna wrote: > Thanks Will and Piotr, > > +1 (non-binding) > > Best, > Bruno > > On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax wrote: > > > > Thanks for the KIP. > > > > +1 (binding) > > > > > > -Matthias > > >

Re: [DISCUSS] KIP-645: Replace abstract class Windows with a proper interface

n the actual downside of using a > > base class vs the interface, is it making the code fragile, or requiring > > redundant implementation, etc. > > > > Boyang > > > > On Tue, Jul 21, 2020 at 2:19 PM Sophie Blee-Goldman > > > wrote: > > > > >

Re: [DISCUSS] KIP-645: Replace abstract class Windows with a proper interface

ss of the actual type > > of the window, so that you wouldn't be able to join a TimeWindow > > with a SessionWindow, for example. > > > > However, because of covariance, it's more painful to change Window > > than Windows, so it might not be worth it rig

Re: [DISCUSS] KIP-450: Sliding Windows

;> for it to be extensible. Different implementations can (and do) > enumerate > >>>> different windows to suit different use cases. > >>>> > >>>> On the other hand, I can’t think of any way to extend SessionWindows > to > >>>

Re: [VOTE] KIP-450: Sliding Window Aggregations in the DSL

Thanks for the KIP! It's been an enlightening discussion +1 (non-binding) Sophie On Tue, Jul 28, 2020 at 8:03 AM Leah Thomas wrote: > Hi all, > > I'd like to kick-off the vote for KIP-450 > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL

Re: [VOTE] KIP-614: Add Prefix Scan support for State Stores

Thanks Sagar! +1 (non-binding) Sophie On Sun, Aug 2, 2020 at 11:37 PM Sagar wrote: > Hi All, > > Just thought of bumping this voting thread again to see if we can form any > consensus around this. > > Thanks! > Sagar. > > > On Mon, Jul 20, 2020 at 4:21 AM Adam Bellemare > wrote: > > > LGTM > >

Re: [DISCUSS] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Just want to make a quick comment on the question that John raised about whether we should introduce a separate config for "key" and "value" window sizes: My short answer is No, I don't think that's necessary. First of all, as you said, there is no first-class concept of a "Windowed value" in the

Re: [DISCUSS] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

Thanks for the KIP! I'm totally in favor of this approach and to be honest, have always wondered why we just silently shut down instead of throwing an exception. This has definitely been a source of confusion for users in my personal experience. I was originally hesitant to extend StreamsException

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Thanks for the KIP! +1 (non-binding) Sophie On Mon, Aug 24, 2020 at 5:06 PM John Roesler wrote: > Thanks Leah, > I’m +1 (binding) > > -John > > On Mon, Aug 24, 2020, at 16:54, Leah Thomas wrote: > > Hi everyone, > > > > I'd like to kick-off the vote for KIP-659: Improve > > TimeWindowedDeserial

Re: [DISCUSS] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

Hey Gokul, thanks for taking up this KIP! I agree with Matthias that directly extending KafkaException may not be ideal, and we should instead extend APIException or RetriableException. Of the two, I think APIException would be more appropriate. My understanding is that RetriableException is gener

Re: [DISCUSS] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Based on my reading of the conversation, it sounds like I > > > > have some legwork to do in KIP-645, but our collective > > > > instinct is that Leah's proposal doesn't need to change to > > > > account for whatever we might decide to do in KIP-

Re: There is an error on the main page

Thanks for the heads up. Would you be interested in submitting a PR to fix this? The typo seems to appear in two files, quickstart-docker.html and quickstart-zookeeper.html, of the kafka-site repo . On Mon, Aug 31, 2020 at 8:43 AM Liu Lycos wrote: > Hello,

Re: [DISCUSS] KIP-666: Add Instant-based methods to ReadOnlySessionStore

Thanks for bringing the IQ API into alignment -- the proposal looks good, although one nit: you missed updating the startTime long to Instant in both appearances of the fetchSession(key, startTime, sessionEndTime) method. Also, I think by "startTime" you actually meant "earliestSessionEndTime". On

Re: [DISCUSS] KIP-667: Remove deprecated methods from ReadOnlyWindowStore

Thanks for this KIP as well! Seems like the methods were deprecated in 2.1. What's our rule for how something has to stay deprecated before we can go ahead and remove it? Assuming 3.0 comes after 2.8, it certainly seems like enough time/releases have passed for us to do so in 3.0. But I'm pretty s

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

If we're going to add a new GLOBAL_RESTORING state to the KafkaStreams FSM, maybe it would make sense to add a new plain RESTORING state that we transition to when restoring non-global state stores following a rebalance. Right now all restoration occurs within the REBALANCING state, which is pretty

Re: [DISCUSS] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

; > >> Extending from `RetriableException` seems reasonable as well. I guess > >> the > >> only question is whether it's safe to catch it as a `RetriableException` > >> and apply common retry logic. For a transactional producer, my > >> expectat

Re: [VOTE] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Hey, sorry for the late reply, I just have one minor suggestion. Since we don't make any guarantees about which thread gets removed or allow the user to specify, I think we should return either the index or full name of the thread that does get removed by removeThread(). I know you just updated th

Re: [DISCUSS] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

first ever contribution - is the next step to initiate the > voting on this KIP? > > -Gokul > > On 04-09-2020 00:34, Sophie Blee-Goldman wrote: > > I think the current proposal looks good to me. One minor suggestion I > have > > is to consider keeping the same error

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

; > > > When saying "pausing the processing threads" I only considered them in > > `RUNNING` and thought we figure out the detail on the PR... Excellent > catch! > > > > Changing state transitions is to some extend backward incompatible, but > > I thin

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

d, Sep 2, 2020 at 7:23 PM Matthias J. Sax > wrote: > > > > > > > +1 (binding) > > > > > > > > On 8/26/20 8:02 AM, John Roesler wrote: > > > > > Hi all, > > > > > > > > > > I've just sent a new messag

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

hod might not be a bad idea. Then > > again, it seems sort of optional. A caller can catch an > > exception or react to a `null` return value just the same > > either way. Having both add/remove methods behave similarly > > is probably more valuable. > > > > Thanks, > &

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

hreads() should report. If others also think that > the proposed name in the KIP is too clumsy, I am open to rename it, though. > > Best, > Bruno > > > On 08.09.20 20:12, Sophie Blee-Goldman wrote: > >> it's never a good sign when the discussion moves into the vote

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

ning the thread's name (as `Optional`) for both add() and > remove() is fine with me. > > > -Matthias > > On 9/8/20 12:58 PM, Sophie Blee-Goldman wrote: > > Sorry Bruno, I think I missed the end of your message with the > > numberOfAliveStreamThreads() > >

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

m > threads of a Kafka Streams client. > > > > > > -Matthias > > > > On 9/8/20 2:37 PM, Sophie Blee-Goldman wrote: > >> Ah, I forgot about localThreadsMetadata(). In that. case I agree, > there's > >> no reason > >> to introduce a

Re: [VOTE] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

+1 from me as well (non-binding) Gokul, it looks you've now received enough binding votes and the vote has been open for sufficiently long. You can conclude the vote and open the PR for review when it's ready Cheers, Sophie On Wed, Sep 9, 2020 at 2:47 PM Guozhang Wang wrote: > +1. Thanks for t

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

> > If you were to call "put" from a punctuator, or do a > `range()` query and then update one of those records with > `put()`, you'd have a very subtle bug on your hands. Can you elaborate on this a bit? I agree that the punctuator case is an obvious exemption to the assumption that store invoca

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

ldn’t solve these problems. Just that it > seems a little that we can be conservative and avoid them for now. If it > turns out we really need to solve them, we can always do it later. > > Thanks, > John > > On Wed, Sep 9, 2020, at 22:46, Sophie Blee-Goldman wrote: > > &g

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

seems better just to go for a single channel for > now. > > It sounds like you're basically in favor of the conservative > approach, and you just wanted to understand the blockers > that I implied. Does my clarification make sense? > > Thanks, > -John > > O

Re: KIP-669: Preserve Source Partition in Kafka Streams from context

Hey Balan, thanks for the KIP! The motivation here makes sense to me, but I have a few questions about the proposed API I guess the main thing to point out is that if we just add new addSink() overloads to Topology, then only the lower level Processor API will benefit and users of the DSL won't b

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

explain the lurking bug that they have had in their > stores all along, as well as the new recommended pattern > (just pass everything you need in the value). If that's > unsatisfying, _then_ we should consider amending the API. > > Thanks, > -John > > On Thu, 2020-09-

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

Hey Walker, The proposal makes sense to me, but while reading up on those old tickets I started wondering if we should give users two options: one that would shut down the entire application, as described in the current KIP, and another that would only shut down an individual instance. I think th

Re: [DISCUSS] KIP-655: Windowed "Distinct" Operation for KStream

Hey all, I'm not convinced either epoch-aligned or data-aligned will fit all possible use cases. Both seem totally reasonable to me: data-aligned is useful for example when you know that a large number of updates to a single key will occur in short bursts, and epoch- aligned when you specifically

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

turned out that we can fix our > >>> existing close() method to accomplish the shutdown from inside an > >>> uncaught exception handler. But I completely agree with you that we > need > >>> to ensure that we do not run into a deadlock in this case. > >

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Makes sense to me :) On Thu, Sep 17, 2020 at 9:34 AM Bruno Cadonna wrote: > Hi Sophie, > > Thank you for the feedback! I replied inline. > > Best, > Bruno > > On 16.09.20 19:19, Sophie Blee-Goldman wrote: > >> > >> We guarantee that the metadata of the

Re: [VOTE] KIP-666: Add Instant-based methods to ReadOnlySessionStore

Thanks for pointing out the vote in the discussion thread, this email somehow skipped my inbox ¯\_(ツ)_/¯ I'm +1 (non-binding) -Sophie On Mon, Sep 7, 2020 at 4:18 AM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Hi everyone, > > I'd like to start a thread to vote for KIP-666

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

d how to > > design those levers...) It's for sure a good idea to call out "use with > > case" in the JavaDocs for the new method. > > > > > > -Matthias > > > > On 7/21/23 3:34 PM, Sophie Blee-Goldman wrote: > > > I guess I felt a bit une

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

one that this would be a welcome > improvement and that this KIP is aligned with moving in that direction. > Given how much discussion there was on this KIP, which is minor relative to > making the changes to StoreBuilder API, I'd rather not tie the two > together. > > Cheers &am

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

gt; > > > > > Cheers, > > > Almog > > > > > > On Tue, Jul 25, 2023 at 9:20 AM Almog Gavra > > wrote: > > > > > >> Glad you like my KIP-secretary skills ;) > > >> > > >> A2. I'm definitely happy to take

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

t would be an example for such an extension? > > In general, I would propose to limit the scope of a config. In this case > > the config should provide suppliers for state stores for the DSL. > > > > BTW, maybe it is a good idea to let DslStorePlugin extend Configurable. >

Re: [VOTE] KIP-759: Unneeded repartition canceling

+1 (binding) thanks Shay! On Wed, Aug 2, 2023 at 1:19 AM Bruno Cadonna wrote: > Hi, > > +1 (binding) > > Thanks for the KIP! > > Best, > Bruno > > On 8/2/23 1:19 AM, Bill Bejeck wrote: > > I caught up on the discussion thread and the KIP LGTM. > > > > +1(binding) > > > > On Tue, Aug 1, 2023 at

  1   2   3   4   5   6   7   8   9   >