Hi Dongjin, Thanks! Sorry, I missed your prior message. The proposed API looks good to me.
I’m wondering if we should specify what kind of store view would be returned when querying the operation result. It seems like it must be a ReadOnlyKeyValueStore. Does that sound right? Thanks! John On Mon, Sep 28, 2020, at 10:06, Dongjin Lee wrote: > Hi John, > > I updated the KIP with the discussion above. The 'Public Interfaces' > section describes the new API, and the 'Rejected Alternatives' section > describes the reasoning about why we selected this API design and rejected > the other alternatives. > > Please have a look when you are free. And please note that the KIP freeze > for 2.7.0 is imminent. > > Thanks, > Dongjin > > On Mon, Sep 21, 2020 at 11:35 PM Dongjin Lee <dong...@apache.org> wrote: > > > Hi John, > > > > I updated the PR applying the API changes we discussed above. I am now > > updating the KIP document. > > > > Thanks, > > Dongjin > > > > On Sat, Sep 19, 2020 at 10:42 AM John Roesler <vvcep...@apache.org> wrote: > > > >> Hi Dongjin, > >> > >> Yes, that’s right. My the time of KIP-307, we had no choice but to add a > >> second name. But we do have a choice with Suppress. > >> > >> Thanks! > >> -John > >> > >> On Thu, Sep 17, 2020, at 13:14, Dongjin Lee wrote: > >> > Hi John, > >> > > >> > I just reviewed KIP-307. As far as I understood, ... > >> > > >> > 1. There was Materialized name initially. > >> > 2. With KIP-307, Named Operations were added. > >> > 3. Now we have two options for materializing suppression. If we take > >> > Materialized name here, we have two names for the same operation, which > >> is > >> > not feasible. > >> > > >> > Do I understand correctly? > >> > > >> > > Do you have a use case in mind for having two separate names for the > >> > operation and the view? > >> > > >> > No. I am now entirely convinced with your suggestion. > >> > > >> > I just started to update the draft implementation. If I understand > >> > correctly, please notify me; I will update the KIP by adding the > >> discussion > >> > above. > >> > > >> > Best, > >> > Dongjin > >> > > >> > On Thu, Sep 17, 2020 at 11:06 AM John Roesler <vvcep...@apache.org> > >> wrote: > >> > > >> > > Hi Dongjin, > >> > > > >> > > Thanks for the reply. Yes, that’s correct, we added that method to > >> name > >> > > the operation. But the operation seems synonymous with the view > >> produced > >> > > the operation, right? > >> > > > >> > > During KIP-307, I remember thinking that it’s unfortunate the we had > >> to > >> > > have two different “name” concepts for the same thing just because > >> setting > >> > > the name on Materialized is equivalent both to making it queriable and > >> > > actually materializing it. > >> > > > >> > > If we were to reconsider the API, it would be nice to treat these > >> three as > >> > > orthogonal: > >> > > * specify a name > >> > > * flag to make the view queriable > >> > > * flag to materialize the view > >> > > > >> > > That was the context behind my suggestion. Do you have a use case in > >> mind > >> > > for having two separate names for the operation and the view? > >> > > > >> > > Thanks, > >> > > John > >> > > > >> > > On Wed, Sep 16, 2020, at 11:43, Dongjin Lee wrote: > >> > > > Hi John, > >> > > > > >> > > > It seems like the available alternatives in this point is clear: > >> > > > > >> > > > 1. Pass queriable name as a separate parameter (i.e., > >> > > > `KTable#suppress(Suppressed, String)`) > >> > > > 2. Make use of the Suppression processor name as a queryable name by > >> > > adding > >> > > > `enableQuery` optional flag to `Suppressed`. > >> > > > > >> > > > However, I doubt the second approach a little bit; As far as I > >> know, the > >> > > > processor name is introduced in KIP-307[^1] to make debugging > >> topology > >> > > easy > >> > > > and understandable. Since the processor name is an independent > >> concept > >> > > with > >> > > > the materialization, I feel the first approach is more natural and > >> > > > consistent. Is there any specific reason that you prefer the second > >> > > > approach? > >> > > > > >> > > > Thanks, > >> > > > Dongjin > >> > > > > >> > > > [^1]: > >> > > > > >> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL > >> > > > > >> > > > > >> > > > > >> > > > On Wed, Sep 16, 2020 at 11:48 PM John Roesler <vvcep...@apache.org> > >> > > wrote: > >> > > > > >> > > > > Hi Dongjin, > >> > > > > > >> > > > > Yes, that's where I was leaning. Although, I'd prefer adding > >> > > > > the option to Suppressed instead of adding a new argument to > >> > > > > the method call. > >> > > > > > >> > > > > What do you think about: > >> > > > > > >> > > > > class Suppressed<K> { > >> > > > > + public Suppressed<K> enableQuery(); > >> > > > > } > >> > > > > > >> > > > > Since Suppressed already has `withName(String)`, it seems > >> > > > > like all we need to add is a boolean flag. > >> > > > > > >> > > > > Does that seem sensible to you? > >> > > > > > >> > > > > Thanks, > >> > > > > -John > >> > > > > > >> > > > > On Wed, 2020-09-16 at 21:50 +0900, Dongjin Lee wrote: > >> > > > > > Hi John, > >> > > > > > > >> > > > > > > Although it's not great to have "special snowflakes" in the > >> API, > >> > > > > Choice B > >> > > > > > does seem safer in the short term. We would basically be > >> proposing a > >> > > > > > temporary API to make the suppressed view queriable without a > >> > > > > Materialized > >> > > > > > argument. > >> > > > > > > >> > > > > > Then, it seems like you prefer `KTable#suppress(Suppressed, > >> String)` > >> > > > > (i.e., > >> > > > > > queriable name only as a parameter) for this time, and refine > >> API > >> > > with > >> > > > > the > >> > > > > > other related KIPs later. > >> > > > > > > >> > > > > > Do I understand correctly? > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Dongjin > >> > > > > > > >> > > > > > On Wed, Sep 16, 2020 at 2:17 AM John Roesler < > >> vvcep...@apache.org> > >> > > > > wrote: > >> > > > > > > >> > > > > > > Hi Dongjin, > >> > > > > > > > >> > > > > > > Thanks for presenting these options. The concern that > >> > > > > > > Matthias brought up is a very deep problem that afflicts all > >> > > > > > > operations downstream of windowing operations. It's the same > >> > > > > > > thing that derailed KIP-300. For the larger context, I have > >> > > > > > > developed a couple of approaches to resolve this situation, > >> > > > > > > but I think it makes sense to finish up KIP-478 before > >> > > > > > > presenting them. > >> > > > > > > > >> > > > > > > However, I don't think that we need in particular to block > >> > > > > > > the current proposal on solving that long-running and deep > >> > > > > > > issue with the DSL. Instead, we should make a top-level > >> > > > > > > decision whether to: > >> > > > > > > > >> > > > > > > A: Make Suppress just like all the other KTable operations. > >> > > > > > > It will have the same pathological behavior that the keyset > >> > > > > > > is unbounded while the store implementation is only a > >> > > > > > > KeyValueStore. Again, this exact pathology currently affects > >> > > > > > > all KTable operations that follow from windowing operations. > >> > > > > > > For example, it applies to the current workaround that > >> > > > > > > Dongjin documented in the KIP: > >> > > > > > > suppress().filter(Materialized<KeyValueStore>). This is > >> > > > > > > Option 2 that Dongjin presented. > >> > > > > > > > >> > > > > > > B: Do something different with Suppress in order to side- > >> > > > > > > step the problem. For example, Suppress does not _need_ to > >> > > > > > > have a separate state store at all. If we just give people a > >> > > > > > > switch to make the operation queriable, we can implement a > >> > > > > > > ReadOnlyKeyValueStore interface by querying the "priorValue" > >> > > > > > > of the buffer first and then querying the upstream > >> > > > > > > ValueGetter. This broad category of "do something different > >> > > > > > > with Suppress" encompases Option 1 and Option 3 that Dongjin > >> > > > > > > presented. > >> > > > > > > > >> > > > > > > > >> > > > > > > Speaking personally, I think Choice A would be the most > >> > > > > > > obvious and least weird choice, but it presents a serious > >> > > > > > > risk of escalating the severity of the problem of unbounded > >> > > > > > > state. This is currently a risk that we're aware of, but has > >> > > > > > > not yet become a big problem in practice. As Matthias > >> > > > > > > pointed out, Suppress is far more likely to be used > >> > > > > > > downstream of windowed tables than other operations, so > >> > > > > > > having a Materialized<KVStore> overload has the significant > >> > > > > > > risk of getting people into a bad state. Note, broadly > >> > > > > > > advertising the workaround from the KIP would have the exact > >> > > > > > > same impact, so we should be careful about recommending it. > >> > > > > > > > >> > > > > > > Although it's not great to have "special snowflakes" in the > >> > > > > > > API, Choice B does seem safer in the short term. We would > >> > > > > > > basically be proposing a temporary API to make the > >> > > > > > > suppressed view queriable without a Materialized argument. > >> > > > > > > Then, once we fix the main KIP-300 problem, we would look at > >> > > > > > > converging Suppress with the rest of the KTable > >> > > > > > > materialization APIs. > >> > > > > > > > >> > > > > > > WDYT? > >> > > > > > > Thanks, > >> > > > > > > -John > >> > > > > > > > >> > > > > > > > >> > > > > > > On Wed, 2020-09-16 at 00:01 +0900, Dongjin Lee wrote: > >> > > > > > > > Hi Matthias, > >> > > > > > > > > >> > > > > > > > Thank you very much for the detailed feedback. Here are my > >> > > opinions: > >> > > > > > > > > >> > > > > > > > > Because there is no final result for non-windowed > >> KTables, it > >> > > seems > >> > > > > > > that > >> > > > > > > > this new feature only make sense for the > >> windowed-aggregation > >> > > case? > >> > > > > > > > > >> > > > > > > > I think a little bit different. Of course, for windowed > >> KTable, > >> > > this > >> > > > > > > > feature provides the final state; for non-windowed KTables, > >> it > >> > > > > provides a > >> > > > > > > > view to the records received more than the predefined > >> waiting > >> > > time > >> > > > > ago - > >> > > > > > > > excluding the records that are waiting for more events. > >> > > > > > > > > >> > > > > > > > > Thus, the signature of `Materialized` should take a > >> > > `WindowStore` > >> > > > > > > instead > >> > > > > > > > of a `KeyValueStore`? > >> > > > > > > > > >> > > > > > > > I reviewed the implementation following your comments and > >> found > >> > > the > >> > > > > > > > following: > >> > > > > > > > > >> > > > > > > > 1. `Materialized` instance includes the following: KeySerde, > >> > > > > ValueSerde, > >> > > > > > > > StoreSupplier, and Queriable Name. > >> > > > > > > > 2. The other `Materialized` method variants in `KTable` are > >> > > making > >> > > > > use of > >> > > > > > > > KeySerde, ValueSerde, and Queriable Name only. (That is, > >> > > > > StoreSupplier is > >> > > > > > > > ignored.) > >> > > > > > > > 3. `KTable#suppress(Suppressed, Materialized)` uses the > >> Queriable > >> > > > > Name > >> > > > > > > > only. StoreSupplier is also ignored. > >> > > > > > > > > >> > > > > > > > So, we have three choices for the method signature: > >> > > > > > > > > >> > > > > > > > 1. `KTable#suppress(Suppressed, String)` (i.e., passing the > >> > > Queriable > >> > > > > > > Name > >> > > > > > > > only): > >> > > > > > > > > >> > > > > > > > This is the simplest; however, it is inconsistent with the > >> > > other > >> > > > > > > > Materialized variant methods. > >> > > > > > > > > >> > > > > > > > 2. `KTable#suppress(Suppressed, Materialized<K, V, > >> > > KeyValueStore>)` > >> > > > > > > (i.e., > >> > > > > > > > current proposal) > >> > > > > > > > > >> > > > > > > > This approach is harmless at this point, for > >> StoreSupplier is > >> > > > > ignored; > >> > > > > > > > However, since suppression can be used to both of > >> > > `KeyValueStore` and > >> > > > > > > > `WindowStore`, this approach is not only weird but also > >> leaving > >> > > some > >> > > > > > > > potential risk to the future. (On second thoughts, I agree, > >> this > >> > > API > >> > > > > > > design > >> > > > > > > > is bad and dangerous.) > >> > > > > > > > > >> > > > > > > > 3. `KTable#suppress(Suppressed, Materialized<K, V, > >> StateStore>)` > >> > > > > > > > > >> > > > > > > > This approach embraces both of `KeyValueStore` and > >> > > `WindowStore` > >> > > > > cases. > >> > > > > > > > Since the concrete class type of `Suppressed` instance > >> differs > >> > > for > >> > > > > the > >> > > > > > > > `StateStore`[^1], it seems like we can validate the > >> arguments on > >> > > the > >> > > > > > > method > >> > > > > > > > call. (e.g., throw `IllegalArgumentException` if when > >> > > `Suppressed` > >> > > > > > > instance > >> > > > > > > > on `KeyValueStore` is given with `Materialized` instance of > >> > > > > > > `WindowStore`.) > >> > > > > > > > This approach not only breaks the API consistency but also > >> guards > >> > > > > from a > >> > > > > > > > miss-usage of the API. > >> > > > > > > > > >> > > > > > > > How do you think? I am now making a try on the third > >> approach. > >> > > > > > > > > >> > > > > > > > Thanks, > >> > > > > > > > Dongjin > >> > > > > > > > > >> > > > > > > > [^1]: `SuppressedInternal` for `KeyValueStore` and > >> > > > > > > > `FinalResultsSuppressionBuilder` for `WindowStore`. > >> > > > > > > > > >> > > > > > > > On Sat, Sep 12, 2020 at 3:29 AM Matthias J. Sax < > >> > > mj...@apache.org> > >> > > > > > > wrote: > >> > > > > > > > > Thanks for updating the KIP. > >> > > > > > > > > > >> > > > > > > > > I think there is still one open question. `suppress()` > >> can be > >> > > used > >> > > > > on > >> > > > > > > > > non-windowed KTable for rate control, as well as on a > >> > > > > windowed-KTable > >> > > > > > > > > (also for rate control, but actually mainly) for only > >> emitting > >> > > the > >> > > > > > > final > >> > > > > > > > > result of a windowed aggregation. For the non-windowed > >> case, we > >> > > > > use a > >> > > > > > > > > KeyValueStore while for the windowed cases, we use a > >> > > WindowStore. > >> > > > > > > > > > >> > > > > > > > > Because there is no final result for non-windowed > >> KTables, it > >> > > seems > >> > > > > > > that > >> > > > > > > > > this new feature only make sense for the > >> windowed-aggregation > >> > > case? > >> > > > > > > > > Thus, the signature of `Materialized` should take a > >> > > `WindowStore` > >> > > > > > > > > instead of a `KeyValueStore`? > >> > > > > > > > > > >> > > > > > > > > If that's correct, I am wondering: > >> > > > > > > > > > >> > > > > > > > > - Can we guard from a miss-usage of the API if the > >> upstream > >> > > > > KTable is > >> > > > > > > > > not windowed (or maybe it's not necessary to guard)? > >> > > > > > > > > - Can we actually implement it? We had issues with > >> regard to > >> > > > > KIP-300 > >> > > > > > > to > >> > > > > > > > > materialize windowed-KTables? > >> > > > > > > > > > >> > > > > > > > > Would be worth to clarify upfront. Maybe, we even need a > >> POC > >> > > > > > > > > implementation to verify that it works? > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > -Matthias > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > On 9/11/20 12:26 AM, Dongjin Lee wrote: > >> > > > > > > > > > Hi All, > >> > > > > > > > > > > >> > > > > > > > > > Here is the voting thread: > >> > > > > > > > > > > >> > > > > > > > >> > > > > > >> > > > >> https://lists.apache.org/thread.html/r5653bf2dafbb27b247bf20dbe6f070c151b3823d96c9c9ca94183e20%40%3Cdev.kafka.apache.org%3E > >> > > > > > > > > > Thanks, > >> > > > > > > > > > Dongjin > >> > > > > > > > > > > >> > > > > > > > > > On Fri, Sep 11, 2020 at 4:23 PM Dongjin Lee < > >> > > dong...@apache.org> > >> > > > > > > wrote: > >> > > > > > > > > > > Hi John, > >> > > > > > > > > > > > >> > > > > > > > > > > Thanks for the feedback. I will open the Vote thread > >> now. > >> > > > > > > > > > > > >> > > > > > > > > > > Best, > >> > > > > > > > > > > Dongjin > >> > > > > > > > > > > > >> > > > > > > > > > > On Fri, Sep 11, 2020 at 2:00 AM John Roesler < > >> > > > > vvcep...@apache.org> > >> > > > > > > > > wrote: > >> > > > > > > > > > > > Hi Dongjin, > >> > > > > > > > > > > > > >> > > > > > > > > > > > Sorry for the delay. I'm glad you're still pushing > >> this > >> > > > > > > > > > > > forward. It would be nice to get this in to the 2.7 > >> > > release. > >> > > > > > > > > > > > > >> > > > > > > > > > > > I just took another look at the KIP, and it looks > >> good to > >> > > > > > > > > > > > me! > >> > > > > > > > > > > > > >> > > > > > > > > > > > I think this is ready for a vote. > >> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > -John > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Wed, 2020-08-05 at 22:04 +0900, Dongjin Lee > >> wrote: > >> > > > > > > > > > > > > Hi All, > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > I updated the KIP > >> > > > > > > > > > > > > < > >> > > > > > > > >> > > > > > >> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-508%3A+Make+Suppression+State+Queriable > >> > > > > > > > > > > > > and the implementation, following the discussion > >> here. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > You must be working hard preparing the release of > >> > > 2.6.0, so > >> > > > > > > please > >> > > > > > > > > have > >> > > > > > > > > > > > a > >> > > > > > > > > > > > > look after your work is done. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > Dongjin > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Sun, Mar 8, 2020 at 12:20 PM John Roesler < > >> > > > > > > vvcep...@apache.org> > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > Thanks Matthias, > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Good idea. I've changed the ticket name and > >> added a > >> > > note > >> > > > > > > > > > > > > > clarifying that this ticket is not the same as > >> > > > > > > > > > > > > > > >> https://issues.apache.org/jira/browse/KAFKA-7224 > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Incidentally, I learned that I never documented > >> my > >> > > > > reasons > >> > > > > > > > > > > > > > for abandoning my work on KAFKA-7224 ! I've now > >> > > updated > >> > > > > > > > > > > > > > that ticket, too, so your question had an > >> unexpected > >> > > > > > > side-benefit. > >> > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > -John > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Sat, Mar 7, 2020, at 18:01, Matthias J. Sax > >> wrote: > >> > > > > > > > > > Thanks for clarification. > >> > > > > > > > > > > >> > > > > > > > > > Can you maybe update the Jira ticket? Do we have a > >> ticket for > >> > > > > > > > > > spill-to-disk? Maybe link to it and explain that it's > >> two > >> > > > > different > >> > > > > > > > > > things? Maybe even rename the ticket to something more > >> > > clear, ie, > >> > > > > > > > > > "make suppress result queryable" or simliar? > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > -Matthias > >> > > > > > > > > > > >> > > > > > > > > > On 3/7/20 1:58 PM, John Roesler wrote: > >> > > > > > > > > > > > > > > > > Hey Matthias, > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > I’m sorry if the ticket was poorly > >> stated. The > >> > > > > ticket > >> > > > > > > is to add a > >> > > > > > > > > > DSL overload to pass a Materialized argument to > >> suppress. As > >> > > a > >> > > > > > > > > > > > > result, > >> > > > > > > > > > the result of the suppression would be queriable. > >> > > > > > > > > > > > > > > > > This is unrelated to “persistent buffer” > >> aka > >> > > > > > > “spill-to-disk”. > >> > > > > > > > > > > > > > > > > There was some confusion before about > >> whether > >> > > this > >> > > > > > > ticket could be > >> > > > > > > > > > implemented as “query the buffer”. Maybe it can, but not > >> > > > > trivially. > >> > > > > > > > > > The obvious way is just to add a new state store which > >> we > >> > > write > >> > > > > the > >> > > > > > > > > > results into just before we forward. I.e., it’s exactly > >> like > >> > > the > >> > > > > > > > > > materialized variant of any stateless KTable operation. > >> > > > > > > > > > > > > > > > > Thanks, John > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Sat, Mar 7, 2020, at 15:32, Matthias > >> J. Sax > >> > > > > wrote: > >> > > > > > > Thanks for > >> > > > > > > > > > > > > > > > > the KIP Dongjin, > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > I am still not sure if I can follow, what > >> might > >> > > > > also > >> > > > > > > be caused by > >> > > > > > > > > > > > > > > > > the backing JIRA ticket (maybe John can > >> > > clarify the > >> > > > > > > intent of the > >> > > > > > > > > > > > > > > > > ticket as he created it): > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Currently, suppress() only uses an > >> in-memory > >> > > buffer > >> > > > > > > and my > >> > > > > > > > > > > > > > > > > understanding of the Jira is, to add the > >> > > ability to > >> > > > > > > use a > >> > > > > > > > > > > > > > > > > persistent buffer (ie, spill to disk > >> backed by > >> > > > > > > RocksDB). > >> > > > > > > > > > > > > > > > > Adding a persistent buffer is completely > >> > > unrelated > >> > > > > to > >> > > > > > > allow > >> > > > > > > > > > > > > > > > > querying the buffer. In fact, one could > >> query > >> > > an > >> > > > > > > in-memory buffer, > >> > > > > > > > > > > > > > > > > too. However, querying the buffer does not > >> > > really > >> > > > > seem > >> > > > > > > to be > >> > > > > > > > > > > > > useful > >> > > > > > > > > > > > > > > > > as pointed out by John, as you can always > >> > > query the > >> > > > > > > upstream > >> > > > > > > > > > > > > KTable > >> > > > > > > > > > > > > > > > > store. > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Also note that for the > >> emit-on-window-close > >> > > case > >> > > > > the > >> > > > > > > result is > >> > > > > > > > > > > > > > > > > deleted from the buffer when it is > >> emitted, and > >> > > > > thus > >> > > > > > > cannot be > >> > > > > > > > > > > > > > > > > queried any longe r. > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Can you please clarify if you intend to > >> allow > >> > > > > spilling > >> > > > > > > to disk or > >> > > > > > > > > > > > > > > > > if you intent to enable IQ (even if I > >> don't > >> > > see why > >> > > > > > > querying make > >> > > > > > > > > > > > > > > > > sense, as the data is either upstream or > >> > > deleted). > >> > > > > > > Also, if you > >> > > > > > > > > > > > > > > > > want to enable IQ, why do we need all > >> those new > >> > > > > > > interfaces? The > >> > > > > > > > > > > > > > > > > result of a suppress() is a KTable that > >> is the > >> > > > > same as > >> > > > > > > any other > >> > > > > > > > > > > > > > > > > key-value/windowed/sessions store? > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > We should also have corresponding Jira > >> tickets > >> > > for > >> > > > > > > different cases > >> > > > > > > > > > > > > > > > > to avoid the confusion I am in atm :) > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > -Matthias > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On 2/27/20 8:21 AM, John Roesler wrote: > >> > > > > > > > > > > > > > > > > > > > Hi Dongjin, > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > No problem; glad we got it sorted > >> out. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks again for picking this up! > >> -John > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Wed, Feb 26, 2020, at 09:24, > >> Dongjin > >> > > Lee > >> > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > I was under the impression that > >> you > >> > > > > wanted > >> > > > > > > to expand the > >> > > > > > > > > > > > > > > > > > > > > > scope of the KIP > >> > > > > > > > > > > > > > > > > > > > > to additionally allow querying the > >> > > internal > >> > > > > > > buffer, not > >> > > > > > > > > > > > > > > > > > > > > just the result. Can you clarify > >> > > whether > >> > > > > you > >> > > > > > > are proposing > >> > > > > > > > > > > > > > > > > > > > > to allow querying the state of the > >> > > internal > >> > > > > > > buffer, the > >> > > > > > > > > > > > > > > > > > > > > result, or both? > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Sorry for the confusion. As we > >> already > >> > > > > talked > >> > > > > > > with, we > >> > > > > > > > > > > > > only > >> > > > > > > > > > > > > > > > > > > > > need to query the suppressed > >> output, > >> > > not > >> > > > > the > >> > > > > > > internal > >> > > > > > > > > > > > > > > > > > > > > buffer. The current > >> implementation is > >> > > > > wrong. > >> > > > > > > After > >> > > > > > > > > > > > > refining > >> > > > > > > > > > > > > > > > > > > > > the KIP and implementation > >> accordingly > >> > > I > >> > > > > will > >> > > > > > > notify you - > >> > > > > > > > > > > > > > > > > > > > > I must be confused, also. > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > On Tue, Feb 25, 2020 at 12:17 AM > >> John > >> > > > > Roesler > >> > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> wrote: > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Hi Dongjin, > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Ah, I think I may have been > >> > > confused. I > >> > > > > 100% > >> > > > > > > agree that > >> > > > > > > > > > > > > > > > > > > > > > we need a materialized variant > >> for > >> > > > > > > suppress(). Then, you > >> > > > > > > > > > > > > > > > > > > > > > could do: ...suppress(..., > >> > > > > > > > > > > > > > > > > > > > > > Materialized.as(“final-count”)) > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > If that’s your proposal, then > >> we are > >> > > on > >> > > > > the > >> > > > > > > same page. > >> > > > > > > > > > > > > > > > > > > > > > I was under the impression that > >> you > >> > > > > wanted > >> > > > > > > to expand the > >> > > > > > > > > > > > > > > > > > > > > > scope of the KIP to additionally > >> > > allow > >> > > > > > > querying the > >> > > > > > > > > > > > > > > > > > > > > > internal buffer, not just the > >> > > result. Can > >> > > > > > > you clarify > >> > > > > > > > > > > > > > > > > > > > > > whether you are proposing to > >> allow > >> > > > > querying > >> > > > > > > the state of > >> > > > > > > > > > > > > > > > > > > > > > the internal buffer, the > >> result, or > >> > > both? > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thanks, John > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 20, 2020, at 08:41, > >> > > Dongjin > >> > > > > Lee > >> > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > Hi John, Thanks for your kind > >> > > > > explanation > >> > > > > > > with an > >> > > > > > > > > > > > > > > > > > > > > > > example. > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > But it feels like you're > >> saying > >> > > > > you're > >> > > > > > > trying to do > >> > > > > > > > > > > > > > > > > > > > > > > > something different > >> > > > > > > > > > > > > > > > > > > > > > > than just query the windowed > >> key > >> > > and > >> > > > > get > >> > > > > > > back the > >> > > > > > > > > > > > > > > > > > > > > > > current count? > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Yes, for example, what if we > >> need > >> > > to > >> > > > > > > retrieve the (all > >> > > > > > > > > > > > > > > > > > > > > > > or range) keys > >> > > > > > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > > a closed window? In this > >> example, > >> > > let's > >> > > > > > > imagine we > >> > > > > > > > > > > > > need > >> > > > > > > > > > > > > > > > > > > > > > > to retrieve only (key=A, > >> > > window=10), > >> > > > > not > >> > > > > > > (key=A, > >> > > > > > > > > > > > > > > > > > > > > > > window=20). > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Of course, the value > >> accompanied > >> > > by a > >> > > > > > > flushed key is > >> > > > > > > > > > > > > > > > > > > > > > > exactly the same to the one > >> in the > >> > > > > > > upstream KTable; > >> > > > > > > > > > > > > > > > > > > > > > > However, if our intention is > >> not > >> > > > > pointing > >> > > > > > > out a > >> > > > > > > > > > > > > > > > > > > > > > > specific key but retrieving a > >> > > group of > >> > > > > > > unspecified > >> > > > > > > > > > > > > > > > > > > > > > > keys, we stuck > >> > > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > trouble - since we can't be > >> sure > >> > > which > >> > > > > key > >> > > > > > > is flushed > >> > > > > > > > > > > > > > > > > > > > > > > out beforehand. > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > One workaround would be > >> > > materializing > >> > > > > it > >> > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > > `suppressed.filter(e -> > >> > > > > > > > > > > > > > > > > > > > > > true, > >> > > > > > > > > > > > > > > > > > > > > > > > >> Materialized.as("final-count"))`. > >> > > But I > >> > > > > > > think > >> > > > > > > > > > > > > providing > >> > > > > > > > > > > > > > > > > > > > > > > a materialized variant for > >> suppress > >> > > > > method > >> > > > > > > is better > >> > > > > > > > > > > > > > > > > > > > > > > than this workaround. > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 20, 2020 at 1:26 > >> AM > >> > > John > >> > > > > > > Roesler > >> > > > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> > >> > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > Thanks for the response, > >> Dongjin, > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'm sorry, but I'm still not > >> > > > > following. > >> > > > > > > It seems > >> > > > > > > > > > > > > like > >> > > > > > > > > > > > > > > > > > > > > > > > the view you > >> > > > > > > > > > > > > > > > > > > > > > would > >> > > > > > > > > > > > > > > > > > > > > > > > get on the "current state > >> of the > >> > > > > buffer" > >> > > > > > > would > >> > > > > > > > > > > > > always > >> > > > > > > > > > > > > > > > > > > > > > > > be equivalent to the view > >> of the > >> > > > > > > upstream table. > >> > > > > > > > > > > > > > > > > > > > > > > > Let me try an example, and > >> maybe > >> > > you > >> > > > > can > >> > > > > > > point out > >> > > > > > > > > > > > > > > > > > > > > > > > the flaw in my reasoning. > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Let's say we're doing 10 ms > >> > > windows > >> > > > > with > >> > > > > > > a grace > >> > > > > > > > > > > > > > > > > > > > > > > > period of zero. Let's also > >> say > >> > > we're > >> > > > > > > computing a > >> > > > > > > > > > > > > > > > > > > > > > > > windowed count, and that we > >> have > >> > > a > >> > > > > > > "final results" > >> > > > > > > > > > > > > > > > > > > > > > > > suppression after the count. > >> > > Let's > >> > > > > > > materialize the > >> > > > > > > > > > > > > > > > > > > > > > > > count as "Count" and the > >> > > suppressed > >> > > > > > > result as "Final > >> > > > > > > > > > > > > > > > > > > > > > > > Count". > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Suppose we get an input > >> event: > >> > > > > (time=10, > >> > > > > > > key=A, > >> > > > > > > > > > > > > > > > > > > > > > > > value=...) > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look like: > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | | > >> 10 > >> > > | A > >> > > > > > > | 1 | > >> > > > > > > > > > > > > > > > > > > > > > > > The (internal) suppression > >> buffer > >> > > > > will > >> > > > > > > contain: > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | | > >> 10 > >> > > | A > >> > > > > > > | 1 | > >> > > > > > > > > > > > > > > > > > > > > > > > The record is still buffered > >> > > because > >> > > > > the > >> > > > > > > window > >> > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > >> Count is > >> > > an > >> > > > > > > empty table: > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > --------------- > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Now, we get a second event: > >> > > (time=15, > >> > > > > > > key=A, > >> > > > > > > > > > > > > > > > > > > > > > > > value=...) > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look like: > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | | > >> 10 > >> > > | A > >> > > > > > > | 2 | > >> > > > > > > > > > > > > > > > > > > > > > > > The (internal) suppression > >> buffer > >> > > > > will > >> > > > > > > contain: > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | | > >> 10 > >> > > | A > >> > > > > > > | 2 | > >> > > > > > > > > > > > > > > > > > > > > > > > The record is still buffered > >> > > because > >> > > > > the > >> > > > > > > window > >> > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > >> Count is > >> > > an > >> > > > > > > empty table: > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > --------------- > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Finally, we get a third > >> event: > >> > > > > (time=20, > >> > > > > > > key=A, > >> > > > > > > > > > > > > > > > > > > > > > > > value=...) > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look like: > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | | > >> 10 > >> > > | A > >> > > > > > > | 2 | | > >> > > > > > > > > > > > > > > > > > > > > > > > 20 | A | 1 | > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > The (internal) suppression > >> buffer > >> > > > > will > >> > > > > > > contain: > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | | > >> 20 > >> > > | A > >> > > > > > > | 1 | > >> > > > > > > > > > > > > > > > > > > > > > > > Note that window 10 has been > >> > > flushed > >> > > > > > > out, because > >> > > > > > > > > > > > > > > > > > > > > > > > it's now closed. And window > >> 20 is > >> > > > > > > buffered because > >> > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > >> Count is > >> > > now: > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | | > >> 10 > >> > > | A > >> > > > > > > | 2 | > >> > > > > > > > > > > > > > > > > > > > > > > > --------------- > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Reading your email, I can't > >> > > figure > >> > > > > out > >> > > > > > > what value > >> > > > > > > > > > > > > > > > > > > > > > > > there is in querying > >> > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > internal suppression buffer, > >> > > since it > >> > > > > > > only contains > >> > > > > > > > > > > > > > > > > > > > > > > > exactly the same > >> > > > > > > > > > > > > > > > > > > > > > value > >> > > > > > > > > > > > > > > > > > > > > > > > as the upstream table, for > >> each > >> > > key > >> > > > > that > >> > > > > > > is still > >> > > > > > > > > > > > > > > > > > > > > > > > buffered. But it feels > >> > > > > > > > > > > > > > > > > > > > > > like > >> > > > > > > > > > > > > > > > > > > > > > > > you're saying you're trying > >> to do > >> > > > > > > something > >> > > > > > > > > > > > > different > >> > > > > > > > > > > > > > > > > > > > > > > > than just query > >> > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > windowed key and get back > >> the > >> > > current > >> > > > > > > count? > >> > > > > > > > > > > > > > > > > > > > > > > > Thanks, -John > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 19, 2020, at > >> 09:49, > >> > > > > Dongjin > >> > > > > > > Lee wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > Hi John, > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > 'The intermediate state > >> of the > >> > > > > > > suppression' in KIP > >> > > > > > > > > > > > > > > > > > > > > > > > > does not mean the > >> > > > > > > > > > > > > > > > > > > > > > > > state > >> > > > > > > > > > > > > > > > > > > > > > > > > of upstream KTable - > >> sure, the > >> > > > > state > >> > > > > > > of the > >> > > > > > > > > > > > > > > > > > > > > > > > > upstream KTable can be > >> > > > > > > > > > > > > > > > > > > > > > > > queried > >> > > > > > > > > > > > > > > > > > > > > > > > > by materializing the > >> operator > >> > > > > > > immediately before > >> > > > > > > > > > > > > > > > > > > > > > > > > the suppress as you > >> > > > > > > > > > > > > > > > > > > > > > > > shown. > >> > > > > > > > > > > > > > > > > > > > > > > > > What I meant in KIP was > >> the > >> > > final > >> > > > > > > state of the > >> > > > > > > > > > > > > > > > > > > > > > > > > buffer, which is not > >> > > > > > > > > > > > > > > > > > > > > > > > emitted > >> > > > > > > > > > > > > > > > > > > > > > > > > yet. (I agree, the current > >> > > > > description > >> > > > > > > may be > >> > > > > > > > > > > > > > > > > > > > > > > > > confusing; it would be > >> > > > > > > > > > > > > > > > > > > > > > > > better > >> > > > > > > > > > > > > > > > > > > > > > > > > to change it with 'the > >> current > >> > > > > state > >> > > > > > > of the > >> > > > > > > > > > > > > > > > > > > > > > > > > suppression' or 'the > >> > > > > > > > > > > > > > > > > > > > > > results > >> > > > > > > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > > > > > the suppression', like > >> the Jira > >> > > > > issue > >> > > > > > > > > > > > > > > > > > > > > > > > > < > >> > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > >> > > > > > > > > > > > > > > > > > > > > > > > > states.) > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > For a little bit more > >> about the > >> > > > > > > motivation, here > >> > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > > > > > one of my > >> > > > > > > > > > > > > > > > > > > > > > > > experience: I > >> > > > > > > > > > > > > > > > > > > > > > > > > had to build a monitoring > >> > > > > application > >> > > > > > > which > >> > > > > > > > > > > > > > > > > > > > > > > > > collects signals from IoT > >> > > devices > >> > > > > > > (say, a > >> > > > > > > > > > > > > > > > > > > > > > > > > semiconductor production > >> > > line.) If > >> > > > > the > >> > > > > > > number of > >> > > > > > > > > > > > > > > > > > > > > > > > collected > >> > > > > > > > > > > > > > > > > > > > > > > > > signals within the time > >> window > >> > > is > >> > > > > much > >> > > > > > > less than > >> > > > > > > > > > > > > > > > > > > > > > > > > the expected, there > >> > > > > > > > > > > > > > > > > > > > > > may > >> > > > > > > > > > > > > > > > > > > > > > > > be > >> > > > > > > > > > > > > > > > > > > > > > > > > some problems like network > >> > > hiccup > >> > > > > in > >> > > > > > > the systems. > >> > > > > > > > > > > > > > > > > > > > > > > > > We wanted to build > >> > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > system in the form of a > >> > > dashboard, > >> > > > > but > >> > > > > > > could not > >> > > > > > > > > > > > > by > >> > > > > > > > > > > > > > > > > > > > > > > > > lack of > >> > > > > > > > > > > > > > > > > > > > > > materializing > >> > > > > > > > > > > > > > > > > > > > > > > > > feature. It was precisely > >> the > >> > > case > >> > > > > of > >> > > > > > > querying > >> > > > > > > > > > > > > only > >> > > > > > > > > > > > > > > > > > > > > > > > > the final > >> > > > > > > > > > > > > > > > > > > > > > results of > >> > > > > > > > > > > > > > > > > > > > > > > > a > >> > > > > > > > > > > > > > > > > > > > > > > > > windowed aggregation, as > >> the > >> > > Jira > >> > > > > issue > >> > > > > > > > > > > > > > > > > > > > > > > > > < > >> > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > >> > > > > > > > > > > > > > > > > > > > > > > > > states. We > >> > > > > > > > > > > > > > > > > > > > > > finally > >> > > > > > > > > > > > > > > > > > > > > > > > ended > >> > > > > > > > > > > > > > > > > > > > > > > > > in implementing the > >> system in > >> > > an > >> > > > > email > >> > > > > > > alerting > >> > > > > > > > > > > > > > > > > > > > > > > > > system like this < > >> > > > > > > > >> https://www.confluent.io/blog/kafka-streams-take-on-watermarks-an > >> > > > > > > > > > d-t > >> > > > > > > > > > riggers/ > >> > > > > > > > > > > > > > > > > and had to collect the keys and windows of > >> > > trouble > >> > > > > by > >> > > > > > > hand. > >> > > > > > > > > > > > > > > > > > > > > > > > > I think these kinds of use > >> > > cases > >> > > > > would > >> > > > > > > be much > >> > > > > > > > > > > > > > > > > > > > > > > > > common. Should it be > >> described > >> > > in > >> > > > > the > >> > > > > > > KIP much > >> > > > > > > > > > > > > more > >> > > > > > > > > > > > > > > > > > > > > > > > > in detail? > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Feb 15, 2020 at > >> 4:43 AM > >> > > > > John > >> > > > > > > Roesler > >> > > > > > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> > >> > > > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Dongjin, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP! > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Can you explain more > >> about > >> > > why > >> > > > > the > >> > > > > > > internal data > >> > > > > > > > > > > > > > > > > > > > > > > > > > structures of > >> > > > > > > > > > > > > > > > > > > > > > > > suppression > >> > > > > > > > > > > > > > > > > > > > > > > > > > should be queriable? The > >> > > > > motivation > >> > > > > > > just says > >> > > > > > > > > > > > > > > > > > > > > > > > > > that users might > >> > > > > > > > > > > > > > > > > > > > > > want to > >> > > > > > > > > > > > > > > > > > > > > > > > do > >> > > > > > > > > > > > > > > > > > > > > > > > > > it, which seems like it > >> could > >> > > > > > > justify literally > >> > > > > > > > > > > > > > > > > > > > > > > > > > anything :) > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > One design point of > >> > > Suppression > >> > > > > is > >> > > > > > > that if you > >> > > > > > > > > > > > > > > > > > > > > > > > > > wanted to query the > >> > > > > > > > > > > > > > > > > > > > > > > > “final > >> > > > > > > > > > > > > > > > > > > > > > > > > > state”, you can > >> Materialize > >> > > the > >> > > > > > > suppress itself > >> > > > > > > > > > > > > > > > > > > > > > > > > > (which is why it > >> > > > > > > > > > > > > > > > > > > > > > needs > >> > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > variant); if you wanted > >> to > >> > > query > >> > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > “intermediate state”, > >> you can > >> > > > > > > materialize the > >> > > > > > > > > > > > > > > > > > > > > > > > > > operator immediately > >> before > >> > > the > >> > > > > > > suppress. > >> > > > > > > > > > > > > > > > > > > > > > > > > > Example: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > ...count(Materialized.as(“intermediate”)) > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> .supress(untilWindowClosed(), > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Materialized.as(“final”)) > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > I’m not sure what use > >> case > >> > > would > >> > > > > > > require > >> > > > > > > > > > > > > > > > > > > > > > > > > > actually fetching from > >> the > >> > > > > internal > >> > > > > > > buffers. > >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, John > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 14, 2020, at > >> > > 07:55, > >> > > > > > > Dongjin Lee > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to reboot the > >> > > > > discussion > >> > > > > > > on KIP-508, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > which aims to > >> > > > > > > > > > > > > > > > > > > > > > support a > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Materialized variant > >> of > >> > > > > > > KTable#suppress. It > >> > > > > > > > > > > > > > > > > > > > > > > > > > > was initially > >> > > > > > > > > > > > > > > > > > > > > > submitted > >> > > > > > > > > > > > > > > > > > > > > > > > > > several > >> > > > > > > > > > > > > > > > > > > > > > > > > > > months ago but closed > >> by > >> > > the > >> > > > > > > inactivity. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > - KIP: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-508%3A+Make > >> > > > > > > > > > +Su > >> > > > > > > > > > ppression+State+Queriable > >> > > > > > > > > > > > > > > > > - Jira: > >> > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > >> > > > > > > > > > > > > > > > > > > > > > > > > > > All kinds of feedback > >> will > >> > > be > >> > > > > > > greatly > >> > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Best, Dongjin > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > >> > > > > mathematical > >> > > > > > > world.* > >> > > > > > > > > > > > > > > > > > > > > > > > > > > *github: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > < > >> http://goog_969573159/> > >> > > > > > > > > > > > > github.com/dongjinleekr > >> > > > > > > > > > <https://github.com/dongjinleekr>linkedin: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > kr.linkedin.com/in/dongjinleekr > >> > > > > > > > > > > > > > > > > > > > > > > > > > > < > >> > > > > > > https://kr.linkedin.com/in/dongjinleekr > >> > > > > > > > > > > > > > speakerdeck: > >> > > > > > > > > > speakerdeck.com/dongjin > >> > > > > > > > > > > > > > > > > > > > > > > > > > > < > >> > > > > https://speakerdeck.com/dongjin>* > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > >> > > mathematical > >> > > > > > > world.* *github: > >> > > > > > > > > > > > > > > > > > > > > > > > > <http://goog_969573159/> > >> > > > > > > github.com/dongjinleekr > >> > > > > > > > > > > > > > > > > > > > > > > > > < > >> > > https://github.com/dongjinleekr > >> > > > > > > > linkedin: > >> > > > > > > > > > > > > > > > > > > > > > > > > >> kr.linkedin.com/in/dongjinleekr > >> > > > > > > > > > > > > > > > > > > > > > > > > < > >> > > > > > > https://kr.linkedin.com/in/dongjinleekr > >> > > > > > > > > > > > > > speakerdeck: > >> > > > > > > > > > speakerdeck.com/dongjin > >> > > > > > > > > > > > > > > > > > > > > > > > > < > >> > > https://speakerdeck.com/dongjin>* > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > >> mathematical > >> > > > > world.* > >> > > > > > > *github: > >> > > > > > > > > > > > > > > > > > > > > > > <http://goog_969573159/> > >> > > > > > > github.com/dongjinleekr > >> > > > > > > > > > > > > > > > > > > > > > > < > >> https://github.com/dongjinleekr > >> > > > > >linkedin: > >> > > > > > > > > > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > >> > > > > > > > > > > > > > > > > > > > > > > < > >> > > > > https://kr.linkedin.com/in/dongjinleekr > >> > > > > > > > speakerdeck: > >> > > > > > > > > > > > > > > > > > > > > > speakerdeck.com/dongjin > >> > > > > > > > > > > > > > > > > > > > > > > < > >> https://speakerdeck.com/dongjin>* > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical > >> > > world.* > >> > > > > > > *github: > >> > > > > > > > > > > > > > > > > > > > > <http://goog_969573159/> > >> > > > > > > github.com/dongjinleekr > >> > > > > > > > > > > > > > > > > > > > > <https://github.com/dongjinleekr > >> > > >linkedin: > >> > > > > > > > > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > >> > > > > > > > > > > > > > > > > > > > > < > >> > > https://kr.linkedin.com/in/dongjinleekr > >> > > > > > > > speakerdeck: > >> > > > > > > > > > > > > > > > > > > > > speakerdeck.com/dongjin < > >> > > > > > > https://speakerdeck.com/dongjin > >> > > > > > > > > > > > > > * > >> > > > > > > > > > > > >> > > > > > > > > > > -- > >> > > > > > > > > > > *Dongjin Lee* > >> > > > > > > > > > > > >> > > > > > > > > > > *A hitchhiker in the mathematical world.* > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > *github: <http://goog_969573159/> > >> github.com/dongjinleekr > >> > > > > > > > > > > <https://github.com/dongjinleekr>keybase: > >> > > > > > > > > https://keybase.io/dongjinleekr > >> > > > > > > > > > > <https://keybase.io/dongjinleekr>linkedin: > >> > > > > > > > > kr.linkedin.com/in/dongjinleekr > >> > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> > > > > > > > > speakerdeck.com/dongjin > >> > > > > > > > > > > <https://speakerdeck.com/dongjin>* > >> > > > > > > > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > -- > >> > > > *Dongjin Lee* > >> > > > > >> > > > *A hitchhiker in the mathematical world.* > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > *github: <http://goog_969573159/>github.com/dongjinleekr > >> > > > <https://github.com/dongjinleekr>keybase: > >> > > https://keybase.io/dongjinleekr > >> > > > <https://keybase.io/dongjinleekr>linkedin: > >> > > kr.linkedin.com/in/dongjinleekr > >> > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> > > speakerdeck.com/dongjin > >> > > > <https://speakerdeck.com/dongjin>* > >> > > > > >> > > > >> > > >> > > >> > -- > >> > *Dongjin Lee* > >> > > >> > *A hitchhiker in the mathematical world.* > >> > > >> > > >> > > >> > > >> > *github: <http://goog_969573159/>github.com/dongjinleekr > >> > <https://github.com/dongjinleekr>keybase: > >> https://keybase.io/dongjinleekr > >> > <https://keybase.io/dongjinleekr>linkedin: > >> kr.linkedin.com/in/dongjinleekr > >> > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> speakerdeck.com/dongjin > >> > <https://speakerdeck.com/dongjin>* > >> > > >> > > > > > > -- > > *Dongjin Lee* > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr > > <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > speakerdeck.com/dongjin > > <https://speakerdeck.com/dongjin>* > > > > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr > <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>* >
