Added the .tostream() discussion in the Wiki.
Thanks
Eno
> On 21 Apr 2017, at 18:48, Matthias J. Sax <matth...@confluent.io> wrote:
>
> I agree with Eno about the renaming.
>
> @Eno: can you add this to the Wiki Discussion page?
>
>
> -Matthias
>
>
> On 4/21/17 1:11 AM, Eno Thereska wrote:
>> Hi Guozhang,
>>
>> Thanks for the feedback. Comments inline:
>>
>>> 1. Regarding the user-facing semantics, I thought we will claim that
>>> "KTables generated from functions that do NOT specify a table name will NOT
>>> be queryable"; but it seems you're proposing to claim it "may not possible
>>> to be queryable", i.e. if users happen to know the internal name if it is
>>> materialized, she can still query it. I feel its potential benefits are
>>> well overwhelmed by the confusion it may introduce. So I'd suggest we just
>>> be strict and say "no store name, not queryable".
>>
>> Sure. "No store name, not queryable" sounds fine. If the user is brave and
>> digs deep they will be able to query these stores that are always created
>> (like when we do aggregates), but I agree that there is no reason we need to
>> make a promise to them if they don't provide a name. I'll change the wording.
>>
>>>
>>> 2. Is there a difference between "calling the overloaded function with
>>> store name, but specify the value as null" and "calling the overloaded
>>> function without store name"? I thought they will be implemented the same
>>> way. But after reading through the wiki I'm not sure. So just clarifying.
>>>
>>
>> There is no difference. I'll clarify.
>>
>>
>>> 3. Personally I'm still a bit preferring renaming "KTable#toStream" to sth.
>>> like "KTable#getChangelog()" or "#toChangelog", since to me it feels more
>>> understandable from user's point of view. WDPT?
>>>
>>
>> So I left this out of this KIP, since it's not directly related to the
>> scope. Perhaps we can do it in a cleanup KIP?
>>
>> Thanks
>> Eno
>>
>>
>>>
>>> Guozhang
>>>
>>>
>>> On Tue, Apr 11, 2017 at 11:53 AM, Matthias J. Sax <matth...@confluent.io>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On 4/11/17 10:34 AM, Eno Thereska wrote:
>>>>> Hi Matthias,
>>>>>
>>>>>
>>>>>> On 11 Apr 2017, at 09:41, Matthias J. Sax <matth...@confluent.io>
>>>> wrote:
>>>>>>
>>>>>> Not sure, if we are on the same page already?
>>>>>>
>>>>>>> "A __store__ can be queryable whether is't materialized or not"
>>>>>>
>>>>>> This does not make sense -- there is nothing like a non-materialized
>>>>>> store -- only non-materialized KTables.
>>>>>
>>>>> Yes, there are stores that are simple views, i.e., non-materialized.
>>>> Damian has such a prototype for Global Tables (it didn't go into trunk).
>>>>> It's still a store, e.g., a KeyValueStore, but when you do a get() it
>>>> recomputes the result on the fly (e.g., it applies a filter).
>>>>>
>>>>> Eno
>>>>>
>>>>>>
>>>>>>> "Yes, there is nothing that will prevent users from querying
>>>>>> internally generated stores, but they cannot assume a store will
>>>>>> necessarily be queryable."
>>>>>>
>>>>>> That is what I disagree on. Stores should be queryable all the time.
>>>>>>
>>>>>> Furthermore, we should have all non-materialized KTables to be
>>>>>> queryable, too.
>>>>>>
>>>>>>
>>>>>> Or maybe there is just some missunderstand going as, and there is some
>>>>>> mix-up between "store" and "KTable"
>>>>>>
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>> On 4/11/17 9:34 AM, Eno Thereska wrote:
>>>>>>> Hi Matthias,
>>>>>>>
>>>>>>> See my note: "A store can be queryable whether it's materialized or
>>>> not". I think we're on the same page. Stores with an internal name are also
>>>> queryable.
>>>>>>>
>>>>>>> I'm just pointing out that. although that is the case today and with
>>>> this KIP, I don't think we have an obligation to make stores with internal
>>>> names queryable in the future. However, that is a discussion for a future
>>>> point.
>>>>>>>
>>>>>>> Eno
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 11 Apr 2017, at 08:56, Matthias J. Sax <matth...@confluent.io>
>>>> wrote:
>>>>>>>>
>>>>>>>> +1 on including GlobalKTable
>>>>>>>>
>>>>>>>> But I am not sure about the materialization / queryable question. For
>>>>>>>> full consistency, all KTables should be queryable nevertheless if they
>>>>>>>> are materialized or not. -- Maybe this is a second step though (even
>>>> if
>>>>>>>> I would like to get this done right away)
>>>>>>>>
>>>>>>>> If we don't want all KTables to be queryable, ie, only those KTables
>>>>>>>> that are materialized, then we should have a clear definition about
>>>>>>>> this, and only allow to query stores, the user did specify a name for.
>>>>>>>> This will simply the reasoning for users, what stores are queryable
>>>> and
>>>>>>>> what not. Otherwise, we still end up confusing user.
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>> On 4/11/17 8:23 AM, Damian Guy wrote:
>>>>>>>>> Eno, re: GlobalKTable - yeah that seems fine.
>>>>>>>>>
>>>>>>>>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com>
>>>> wrote:
>>>>>>>>>
>>>>>>>>>> About GlobalKTables, I suppose there is no reason why they cannot
>>>> also use
>>>>>>>>>> this KIP for consistency, e.g., today you have:
>>>>>>>>>>
>>>>>>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K>
>>>> keySerde,
>>>>>>>>>> final Serde<V> valSerde,
>>>>>>>>>> final String topic,
>>>>>>>>>> final String storeName)
>>>>>>>>>>
>>>>>>>>>> For consistency with the KIP you could also have an overload
>>>> without the
>>>>>>>>>> store name, for people who want to construct a global ktable, but
>>>> don't
>>>>>>>>>> care about querying it directly:
>>>>>>>>>>
>>>>>>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K>
>>>> keySerde,
>>>>>>>>>> final Serde<V> valSerde,
>>>>>>>>>> final String topic)
>>>>>>>>>>
>>>>>>>>>> Damian, what do you think? I'm thinking of adding this to KIP.
>>>> Thanks to
>>>>>>>>>> Michael for bringing it up.
>>>>>>>>>>
>>>>>>>>>> Eno
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Michael, comments inline:
>>>>>>>>>>>
>>>>>>>>>>>> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io>
>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the updates, Eno!
>>>>>>>>>>>>
>>>>>>>>>>>> In addition to what has already been said: We should also
>>>> explicitly
>>>>>>>>>>>> mention that this KIP is not touching GlobalKTable. I'm sure
>>>> that some
>>>>>>>>>>>> users will throw KTable and GlobalKTable into one conceptual
>>>> "it's all
>>>>>>>>>>>> tables!" bucket and then wonder how the KIP might affect global
>>>> tables.
>>>>>>>>>>>
>>>>>>>>>>> Good point, I'll add.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Damian wrote:
>>>>>>>>>>>>> I think if no store name is provided users would still be able
>>>> to query
>>>>>>>>>>>> the
>>>>>>>>>>>>> store, just the store name would be some internally generated
>>>> name.
>>>>>>>>>> They
>>>>>>>>>>>>> would be able to discover those names via the IQ API.
>>>>>>>>>>>>
>>>>>>>>>>>> I, too, think that users should be able to query a store even if
>>>> its
>>>>>>>>>> name
>>>>>>>>>>>> was internally generated. After all, the data is already there /
>>>>>>>>>>>> materialized.
>>>>>>>>>>>
>>>>>>>>>>> Yes, there is nothing that will prevent users from querying
>>>> internally
>>>>>>>>>> generated stores, but they cannot
>>>>>>>>>>> assume a store will necessarily be queryable. So if it's there,
>>>> they can
>>>>>>>>>> query it. If it's not there, and they didn't
>>>>>>>>>>> provide a queryable name, they cannot complain and say "hey, where
>>>> is my
>>>>>>>>>> store". If they must absolutely be certain that
>>>>>>>>>>> a store is queryable, then they must provide a queryable name.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Damian wrote:
>>>>>>>>>>>>> I think for some stores it will make sense to not create a
>>>> physical
>>>>>>>>>>>> store, i.e.,
>>>>>>>>>>>>> for thinks like `filter`, as this will save the rocksdb
>>>> overhead. But i
>>>>>>>>>>>> guess that
>>>>>>>>>>>>> is more of an implementation detail.
>>>>>>>>>>>>
>>>>>>>>>>>> I think it would help if the KIP would clarify what we'd do in
>>>> such a
>>>>>>>>>>>> case. For example, if the user did not specify a store name for
>>>>>>>>>>>> `KTable#filter` -- would it be queryable? If so, would this
>>>> imply we'd
>>>>>>>>>>>> always materialize the state store, or...?
>>>>>>>>>>>
>>>>>>>>>>> I'll clarify in the KIP with some more examples. Materialization
>>>> will be
>>>>>>>>>> an internal concept. A store can be queryable whether it's
>>>> materialized or
>>>>>>>>>> not
>>>>>>>>>>> (e.g., through advanced implementations that compute the value of a
>>>>>>>>>> filter on a fly, rather than materialize the answer).
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Eno
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy <damian....@gmail.com
>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eno,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the update. I agree with what Matthias said. I wonder
>>>> if
>>>>>>>>>> the KIP
>>>>>>>>>>>>> should talk less about materialization and more about querying?
>>>> After
>>>>>>>>>> all,
>>>>>>>>>>>>> that is what is being provided from an end-users perspective.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think if no store name is provided users would still be able to
>>>>>>>>>> query the
>>>>>>>>>>>>> store, just the store name would be some internally generated
>>>> name.
>>>>>>>>>> They
>>>>>>>>>>>>> would be able to discover those names via the IQ API
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think for some stores it will make sense to not create a
>>>> physical
>>>>>>>>>> store,
>>>>>>>>>>>>> i.e., for thinks like `filter`, as this will save the rocksdb
>>>>>>>>>> overhead. But
>>>>>>>>>>>>> i guess that is more of an implementation detail.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Damian
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska <
>>>> eno.there...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, this still forces users, to provide a name for store
>>>> that we
>>>>>>>>>>>>>>> must materialize, even if users are not interested in querying
>>>> the
>>>>>>>>>>>>>>> stores. Thus, I would like to have overloads for all currently
>>>>>>>>>> existing
>>>>>>>>>>>>>>> methods having mandatory storeName paremeter, with overloads,
>>>> that do
>>>>>>>>>>>>>>> not require the storeName parameter.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't
>>>> make it
>>>>>>>>>>>>>> clear, I'll clarify.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax <
>>>> matth...@confluent.io>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for pushing this KIP Eno.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The update give a very clear description about the scope, that
>>>> is
>>>>>>>>>> super
>>>>>>>>>>>>>>> helpful for the discussion!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - To put it into my own words, the KIP focus is on enable to
>>>> query
>>>>>>>>>> all
>>>>>>>>>>>>>>> KTables.
>>>>>>>>>>>>>>> ** The ability to query a store is determined by providing a
>>>> name for
>>>>>>>>>>>>>>> the store.
>>>>>>>>>>>>>>> ** At the same time, providing a name -- and thus making a
>>>> store
>>>>>>>>>>>>>>> queryable -- does not say anything about an actual
>>>> materialization
>>>>>>>>>> (ie,
>>>>>>>>>>>>>>> being queryable and being materialized are orthogonal).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I like this overall a lot. However, I would go one step
>>>> further.
>>>>>>>>>> Right
>>>>>>>>>>>>>>> now, you suggest to add new overload methods that allow users
>>>> to
>>>>>>>>>>>>> specify
>>>>>>>>>>>>>>> a storeName -- if `null` is provided and the store is not
>>>>>>>>>> materialized,
>>>>>>>>>>>>>>> we ignore it completely -- if `null` is provided but the store
>>>> must
>>>>>>>>>> be
>>>>>>>>>>>>>>> materialized we generate a internal name. So far so good.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, this still forces users, to provide a name for store
>>>> that we
>>>>>>>>>>>>>>> must materialize, even if users are not interested in querying
>>>> the
>>>>>>>>>>>>>>> stores. Thus, I would like to have overloads for all currently
>>>>>>>>>> existing
>>>>>>>>>>>>>>> methods having mandatory storeName paremeter, with overloads,
>>>> that do
>>>>>>>>>>>>>>> not require the storeName parameter.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Otherwise, we would still have some methods which optional
>>>> storeName
>>>>>>>>>>>>>>> parameter and other method with mandatory storeName parameter
>>>> --
>>>>>>>>>> thus,
>>>>>>>>>>>>>>> still some inconsistency.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 4/9/17 8:35 AM, Eno Thereska wrote:
>>>>>>>>>>>>>>>> Hi there,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've now done a V2 of the KIP, that hopefully addresses the
>>>> feedback
>>>>>>>>>>>>> in
>>>>>>>>>>>>>> this discussion thread:
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>> 114%3A+KTable+materialization+and+improved+semantics
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>> 114:+KTable+materialization+and+improved+semantics>.
>>>>>>>>>>>>>> Notable changes:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - clearly outline what is in the scope of the KIP and what is
>>>> not.
>>>>>>>>>> We
>>>>>>>>>>>>>> ran into the issue where lots of useful, but somewhat tangential
>>>>>>>>>>>>>> discussions came up on interactive queries, declarative DSL
>>>> etc. The
>>>>>>>>>>>>> exact
>>>>>>>>>>>>>> scope of this KIP is spelled out.
>>>>>>>>>>>>>>>> - decided to go with overloaded methods, not .materialize(),
>>>> to stay
>>>>>>>>>>>>>> within the spirit of the current declarative DSL.
>>>>>>>>>>>>>>>> - clarified the depreciation plan
>>>>>>>>>>>>>>>> - listed part of the discussion we had under rejected
>>>> alternatives
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If you have any further feedback on this, let's continue on
>>>> this
>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you
>>>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska <
>>>> eno.there...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP so
>>>> I'll do
>>>>>>>>>>>>>> that and we can see how it looks and continue the discussion
>>>> from
>>>>>>>>>> there.
>>>>>>>>>>>>>> Stay tuned.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax <
>>>> matth...@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think Eno's separation is very clear and helpful. In
>>>> order to
>>>>>>>>>>>>>>>>>> streamline this discussion, I would suggest we focus back
>>>> on point
>>>>>>>>>>>>> (1)
>>>>>>>>>>>>>>>>>> only, as this is the original KIP question.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Even if I started to DSL design discussion somehow, because
>>>> I
>>>>>>>>>>>>> thought
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> might be helpful to resolve both in a single shot, I feel
>>>> that we
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> too many options about DSL design and we should split it up
>>>> in two
>>>>>>>>>>>>>>>>>> steps. This will have the disadvantage that we will change
>>>> the API
>>>>>>>>>>>>>>>>>> twice, but still, I think it will be a more focused
>>>> discussion.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I just had another look at the KIP, an it proposes 3
>>>> changes:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1. add .materialized() -> IIRC it was suggested to name this
>>>>>>>>>>>>>>>>>> .materialize() though (can you maybe update the KIP Eno?)
>>>>>>>>>>>>>>>>>> 2. remove print(), writeAsText(), and foreach()
>>>>>>>>>>>>>>>>>> 3. rename toStream() to toKStream()
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I completely agree with (2) -- not sure about (3) though
>>>> because
>>>>>>>>>>>>>>>>>> KStreamBuilder also hast .stream() and .table() as methods.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However, we might want to introduce a KStream#toTable() --
>>>> this
>>>>>>>>>> was
>>>>>>>>>>>>>>>>>> requested multiple times -- might also be part of a
>>>> different KIP.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thus, we end up with (1). I would suggest to do a step
>>>> backward
>>>>>>>>>> here
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> instead of a discussion how to express the changes in the
>>>> DSL (new
>>>>>>>>>>>>>>>>>> overload, new methods...) we should discuss what the actual
>>>> change
>>>>>>>>>>>>>>>>>> should be. Like (1) materialize all KTable all the time (2)
>>>> all
>>>>>>>>>> the
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>> to force a materialization to enable querying the KTable
>>>> (3) allow
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> queryable non-materialized KTable.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On more question is, if we want to allow a user-forced
>>>>>>>>>>>>> materialization
>>>>>>>>>>>>>>>>>> only as as local store without changelog, or both (together
>>>> /
>>>>>>>>>>>>>>>>>> independently)? We got some request like this already.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote:
>>>>>>>>>>>>>>>>>>> Hi Eno,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> thanks for putting into different points. I want to put a
>>>> few
>>>>>>>>>>>>> remarks
>>>>>>>>>>>>>>>>>>> inline.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote:
>>>>>>>>>>>>>>>>>>>> So I think there are several important discussion threads
>>>> that
>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>> emerging here. Let me try to tease them apart:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1. inconsistency in what is materialized and what is not,
>>>> what
>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> queryable and what is not. I think we all agree there is
>>>> some
>>>>>>>>>>>>>>>>>>>> inconsistency there and this will be addressed with any
>>>> of the
>>>>>>>>>>>>>>>>>>>> proposed approaches. Addressing the inconsistency is the
>>>> point
>>>>>>>>>> of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> original KIP.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2. the exact API for materializing a KTable. We can
>>>> specify 1) a
>>>>>>>>>>>>>>>>>>>> "store name" (as we do today) or 2) have a
>>>> ".materialize[d]"
>>>>>>>>>> call
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4)
>>>> have a
>>>>>>>>>>>>> builder
>>>>>>>>>>>>>>>>>>>> construct. So we have discussed 4 options. It is
>>>> important to
>>>>>>>>>>>>>> remember
>>>>>>>>>>>>>>>>>>>> in this discussion that IQ is not designed for just local
>>>>>>>>>> queries,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> also for distributed queries. In all cases an identifying
>>>>>>>>>> name/id
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> needed for the store that the user is interested in
>>>> querying. So
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> end up with a discussion on who provides the name, the
>>>> user (as
>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>>>>> today) or if it is generated automatically (as Jan
>>>> suggests, as
>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> understand it). If it is generated automatically we need
>>>> a way
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> expose these auto-generated names to the users and link
>>>> them to
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> KTables they care to query.
>>>>>>>>>>>>>>>>>>> Hi, the last sentence is what I currently arguing against.
>>>> The
>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>> would never see a stringtype indentifier name or anything.
>>>> All he
>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>>> is the queryHandle if he executes a get(K) that will be an
>>>>>>>>>>>>>> interactive
>>>>>>>>>>>>>>>>>>> query get. with all the finding the right servers that
>>>> currently
>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>> copy of this underlying store stuff going on. The nice
>>>> part is
>>>>>>>>>> that
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>> someone retrieves a queryHandle, you know that you have to
>>>>>>>>>>>>>> materialized
>>>>>>>>>>>>>>>>>>> (if you are not already) as queries will be coming. Taking
>>>> away
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> confusion mentioned in point 1 IMO.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 3. The exact boundary between the DSL, that is the
>>>> processing
>>>>>>>>>>>>>>>>>>>> language, and the storage/IQ queries, and how we jump
>>>> from one
>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> other. This is mostly for how we get a handle on a store
>>>> (so
>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>> related to point 2), rather than for how we query the
>>>> store. I
>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>> we all agree that we don't want to limit ways one can
>>>> query a
>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>> (e.g., using gets or range queries etc) and the query
>>>> APIs are
>>>>>>>>>> not
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the scope of the DSL.
>>>>>>>>>>>>>>>>>>> Does the IQ work with range currently? The range would
>>>> have to be
>>>>>>>>>>>>>>>>>>> started on all stores and then merged by maybe the client.
>>>> Range
>>>>>>>>>>>>>> force a
>>>>>>>>>>>>>>>>>>> flush to RocksDB currently so I am sure you would get a
>>>>>>>>>> performance
>>>>>>>>>>>>>> hit
>>>>>>>>>>>>>>>>>>> right there. Time-windows might be okay, but I am not sure
>>>> if the
>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>> version should offer the user range access.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 4. The nature of the DSL and whether its declarative
>>>> enough, or
>>>>>>>>>>>>>>>>>>>> flexible enough. Damian made the point that he likes the
>>>> builder
>>>>>>>>>>>>>>>>>>>> pattern since users can specify, per KTable, things like
>>>> caching
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> logging needs. His observation (as I understand it) is
>>>> that the
>>>>>>>>>>>>>>>>>>>> processor API (PAPI) is flexible but doesn't provide any
>>>> help at
>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>> to users. The current DSL provides declarative
>>>> abstractions, but
>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>> not fine-grained enough. This point is much broader than
>>>> the
>>>>>>>>>> KIP,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> discussing it in this KIPs context is ok, since we don't
>>>> want to
>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>>> small piecemeal changes and then realise we're not in the
>>>> spot
>>>>>>>>>> we
>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>>>> to be.
>>>>>>>>>>>>>>>>>>> This is indeed much broader. My guess here is that's why
>>>> both
>>>>>>>>>> API's
>>>>>>>>>>>>>>>>>>> exists and helping the users to switch back and forth
>>>> might be a
>>>>>>>>>>>>>> thing.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Feel free to pitch in if I have misinterpreted something.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak <
>>>>>>>>>> jan.filip...@trivago.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Eno,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I have a really hard time understanding why we can't.
>>>> From my
>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>> of view everything could be super elegant DSL only +
>>>> public api
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> the PAPI-people as already exist.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The above aproach implementing a .get(K) on KTable is
>>>> foolisch
>>>>>>>>>> in
>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>> opinion as it would be to late to know that
>>>> materialisation
>>>>>>>>>> would
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> required.
>>>>>>>>>>>>>>>>>>>>> But having an API that allows to indicate I want to
>>>> query this
>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>> and then wrapping the say table's processorname can work
>>>> out
>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>>>>>>> really nice. The only obstacle I see is people not
>>>> willing to
>>>>>>>>>>>>> spend
>>>>>>>>>>>>>>>>>>>>> the additional time in implementation and just want a
>>>> quick
>>>>>>>>>> shot
>>>>>>>>>>>>>>>>>>>>> option to make it work.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> For me it would look like this:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> table = builder.table()
>>>>>>>>>>>>>>>>>>>>> filteredTable = table.filter()
>>>>>>>>>>>>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the
>>>> materialisation,
>>>>>>>>>>>>>>>>>>>>> really all names possible but id rather hide the
>>>> implication of
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> materializes
>>>>>>>>>>>>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() //
>>>> this
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>> _not_ materialize again of course, the source or the
>>>> aggregator
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>> stay the only materialized processors
>>>>>>>>>>>>>>>>>>>>> streams = new streams(builder)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This middle part is highly flexible I could imagin to
>>>> force the
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>> todo something like this. This implies to the user that
>>>> his
>>>>>>>>>>>>> streams
>>>>>>>>>>>>>>>>>>>>> need to be running
>>>>>>>>>>>>>>>>>>>>> instead of propagating the missing initialisation back by
>>>>>>>>>>>>>> exceptions.
>>>>>>>>>>>>>>>>>>>>> Also if the users is forced to pass the appropriate
>>>> streams
>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>> back can change.
>>>>>>>>>>>>>>>>>>>>> I think its possible to build multiple streams out of
>>>> one
>>>>>>>>>>>>> topology
>>>>>>>>>>>>>>>>>>>>> so it would be easiest to implement aswell. This is just
>>>> what I
>>>>>>>>>>>>>> maybe
>>>>>>>>>>>>>>>>>>>>> had liked the most
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> streams.start();
>>>>>>>>>>>>>>>>>>>>> rawHandle.prepare(streams)
>>>>>>>>>>>>>>>>>>>>> filteredHandle.prepare(streams)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> later the users can do
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> V value = rawHandle.get(K)
>>>>>>>>>>>>>>>>>>>>> V value = filteredHandle.get(K)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This could free DSL users from anything like storenames
>>>> and how
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> what to materialize. Can someone indicate what the
>>>> problem
>>>>>>>>>> would
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> implementing it like this.
>>>>>>>>>>>>>>>>>>>>> Yes I am aware that the current IQ API will not support
>>>>>>>>>> querying
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> KTableProcessorName instread of statestoreName. But I
>>>> think
>>>>>>>>>> that
>>>>>>>>>>>>>> had
>>>>>>>>>>>>>>>>>>>>> to change if you want it to be intuitive
>>>>>>>>>>>>>>>>>>>>> IMO you gotta apply the filter read time
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Looking forward to your opinions
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote:
>>>>>>>>>>>>>>>>>>>>>> Hi there,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The inconsistency will be resolved, whether with
>>>> materialize
>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>> overloaded methods.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> With the discussion on the DSL & stores I feel we've
>>>> gone in a
>>>>>>>>>>>>>>>>>>>>>> slightly different tangent, which is worth discussing
>>>>>>>>>>>>> nonetheless.
>>>>>>>>>>>>>>>>>>>>>> We have entered into an argument around the scope of
>>>> the DSL.
>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>> DSL has been designed primarily for processing. The DSL
>>>> does
>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> dictate ways to access state stores or what hind of
>>>> queries to
>>>>>>>>>>>>>>>>>>>>>> perform on them. Hence, I see the mechanism for
>>>> accessing
>>>>>>>>>>>>> storage
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>> decoupled from the DSL.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We could think of ways to get store handles from part
>>>> of the
>>>>>>>>>>>>> DSL,
>>>>>>>>>>>>>>>>>>>>>> like the KTable abstraction. However, subsequent
>>>> queries will
>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm not
>>>> sure we
>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing
>>>> that the
>>>>>>>>>>>>>>>>>>>>>> current way of getting a handle on state stores is fine.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang <
>>>> wangg...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thinking loud here about the API options (materialize
>>>> v.s.
>>>>>>>>>>>>>> overloaded
>>>>>>>>>>>>>>>>>>>>>>> functions) and its impact on IQ:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 1. The first issue of the current DSL is that, there is
>>>>>>>>>>>>>>>>>>>>>>> inconsistency upon
>>>>>>>>>>>>>>>>>>>>>>> whether / how KTables should be materialized:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> a) in many cases the library HAS TO materialize
>>>> KTables no
>>>>>>>>>>>>>>>>>>>>>>> matter what,
>>>>>>>>>>>>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables,
>>>> and hence
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>> users to provide store names and throw RTE if it is
>>>> null;
>>>>>>>>>>>>>>>>>>>>>>> b) in some other cases, the KTable can be materialized
>>>> or
>>>>>>>>>> not;
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> example in KStreamBuilder.table(), store names can be
>>>>>>>>>> nullable
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> in which
>>>>>>>>>>>>>>>>>>>>>>> case the KTable would not be materialized;
>>>>>>>>>>>>>>>>>>>>>>> c) in some other cases, the KTable will never be
>>>>>>>>>> materialized,
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> example KTable.filter() resulted KTables, and users
>>>> have no
>>>>>>>>>>>>>> options to
>>>>>>>>>>>>>>>>>>>>>>> enforce them to be materialized;
>>>>>>>>>>>>>>>>>>>>>>> d) this is related to a), where some KTables are
>>>> required to
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>> materialized, but we do not enforce users to provide a
>>>> state
>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>>>> name,
>>>>>>>>>>>>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown
>>>> not
>>>>>>>>>>>>>>>>>>>>>>> immediately but
>>>>>>>>>>>>>>>>>>>>>>> later in this case.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2. The second issue is related to IQ, where state
>>>> stores are
>>>>>>>>>>>>>>>>>>>>>>> accessed by
>>>>>>>>>>>>>>>>>>>>>>> their state stores; so only those KTable's that have
>>>>>>>>>>>>>> user-specified
>>>>>>>>>>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>>>>>> stores will be queryable. But because of 1) above, many
>>>>>>>>>> stores
>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>> not be
>>>>>>>>>>>>>>>>>>>>>>> interested to users for IQ but they still need to
>>>> provide a
>>>>>>>>>>>>>>>>>>>>>>> (dummy?) state
>>>>>>>>>>>>>>>>>>>>>>> store name for them; while on the other hand users
>>>> cannot
>>>>>>>>>> query
>>>>>>>>>>>>>>>>>>>>>>> some state
>>>>>>>>>>>>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() as
>>>> there
>>>>>>>>>> is
>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>>>>> APIs for
>>>>>>>>>>>>>>>>>>>>>>> them to specify a state store name.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 3. We are aware from user feedbacks that such backend
>>>> details
>>>>>>>>>>>>>> would be
>>>>>>>>>>>>>>>>>>>>>>> better be abstracted away from the DSL layer, where app
>>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>> just focus on processing logic, while state stores
>>>> along with
>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>>>>>> changelogs etc would better be in a different
>>>> mechanism; same
>>>>>>>>>>>>>>>>>>>>>>> arguments
>>>>>>>>>>>>>>>>>>>>>>> have been discussed for serdes / windowing triggers as
>>>> well.
>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>> serdes
>>>>>>>>>>>>>>>>>>>>>>> specifically, we had a very long discussion about it
>>>> and
>>>>>>>>>>>>>> concluded
>>>>>>>>>>>>>>>>>>>>>>> that, at
>>>>>>>>>>>>>>>>>>>>>>> least in Java7, we cannot completely abstract serde
>>>> away in
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> DSL, so we
>>>>>>>>>>>>>>>>>>>>>>> choose the other extreme to enforce users to be
>>>> completely
>>>>>>>>>>>>> aware
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> serde requirements when some KTables may need to be
>>>>>>>>>>>>> materialized
>>>>>>>>>>>>>> vis
>>>>>>>>>>>>>>>>>>>>>>> overloaded API functions. While for the state store
>>>> names, I
>>>>>>>>>>>>> feel
>>>>>>>>>>>>>>>>>>>>>>> it is a
>>>>>>>>>>>>>>>>>>>>>>> different argument than serdes (details below).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded
>>>> functions
>>>>>>>>>>>>>>>>>>>>>>> directions,
>>>>>>>>>>>>>>>>>>>>>>> the first thing I'd like to resolve is the
>>>> inconsistency
>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>>>>>>>>>> above. So in either case: KTable materialization will
>>>> not be
>>>>>>>>>>>>>> affect
>>>>>>>>>>>>>>>>>>>>>>> by user
>>>>>>>>>>>>>>>>>>>>>>> providing state store name or not, but will only be
>>>> decided
>>>>>>>>>> by
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>>>>>>>>>> when it is necessary. More specifically, only join
>>>> operator
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> builder.table() resulted KTables are not always
>>>> materialized,
>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>> are still
>>>>>>>>>>>>>>>>>>>>>>> likely to be materialized lazily (e.g. when
>>>> participated in a
>>>>>>>>>>>>>> join
>>>>>>>>>>>>>>>>>>>>>>> operator).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> For overloaded functions that would mean:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> a) we have an overloaded function for ALL operators
>>>> that
>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>> result
>>>>>>>>>>>>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the
>>>> function
>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>> param it is null by default);
>>>>>>>>>>>>>>>>>>>>>>> b) null-state-store-name do not indicate that a KTable
>>>> would
>>>>>>>>>>>>>>>>>>>>>>> not be
>>>>>>>>>>>>>>>>>>>>>>> materialized, but that it will not be used for IQ at
>>>> all
>>>>>>>>>>>>>> (internal
>>>>>>>>>>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>>>>>> store names will be generated when necessary).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> For materialize() that would mean:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> a) we will remove state store names from ALL operators
>>>> that
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>> result in a KTable.
>>>>>>>>>>>>>>>>>>>>>>> b) KTables that not calling materialized do not
>>>> indicate that
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> KTable
>>>>>>>>>>>>>>>>>>>>>>> would not be materialized, but that it will not be
>>>> used for
>>>>>>>>>> IQ
>>>>>>>>>>>>>> at all
>>>>>>>>>>>>>>>>>>>>>>> (internal state store names will be generated when
>>>>>>>>>> necessary).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Again, in either ways the API itself does not "hint"
>>>> about
>>>>>>>>>>>>>> anything
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> materializing a KTable or not at all; it is still
>>>> purely
>>>>>>>>>>>>>> determined
>>>>>>>>>>>>>>>>>>>>>>> by the
>>>>>>>>>>>>>>>>>>>>>>> library when parsing the DSL for now.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Following these thoughts, I feel that 1) we should
>>>> probably
>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>> the name
>>>>>>>>>>>>>>>>>>>>>>> "materialize" since it may be misleading to users as
>>>> what
>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>>>>>>> happened
>>>>>>>>>>>>>>>>>>>>>>> behind the scene, to e.g. Damian suggested
>>>>>>>>>>>>> "queryableStore(String
>>>>>>>>>>>>>>>>>>>>>>> storeName)",
>>>>>>>>>>>>>>>>>>>>>>> which returns a QueryableStateStore, and can replace
>>>> the
>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those two
>>>> options
>>>>>>>>>>>>>>>>>>>>>>> assuming we
>>>>>>>>>>>>>>>>>>>>>>> get rid of the misleading function name, I personally
>>>> favor
>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> adding more
>>>>>>>>>>>>>>>>>>>>>>> overloading functions as it keeps the API simpler.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak
>>>>>>>>>>>>>>>>>>>>>>> <jan.filip...@trivago.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> thanks for your mail, felt like this can clarify some
>>>>>>>>>> things!
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>>>>>>>>> unfortunately split but as all branches close in on
>>>> what my
>>>>>>>>>>>>>>>>>>>>>>>> suggestion was
>>>>>>>>>>>>>>>>>>>>>>>> about Ill pick this to continue
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Of course only the table the user wants to query
>>>> would be
>>>>>>>>>>>>>>>>>>>>>>>> materialized.
>>>>>>>>>>>>>>>>>>>>>>>> (retrieving the queryhandle implies materialisation).
>>>> So In
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> example of
>>>>>>>>>>>>>>>>>>>>>>>> KTable::filter if you call
>>>>>>>>>>>>>>>>>>>>>>>> getIQHandle on both tables only the one source that
>>>> is there
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>> materialize and the QueryHandleabstraction would make
>>>> sure
>>>>>>>>>> it
>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>>>>>>>> mapped
>>>>>>>>>>>>>>>>>>>>>>>> and filtered and what not uppon read as usual.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe
>>>> only
>>>>>>>>>> wrap
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> storeName / table unique identifier and a way to
>>>> access the
>>>>>>>>>>>>>> streams
>>>>>>>>>>>>>>>>>>>>>>>> instance and then basically uses the same mechanism
>>>> that is
>>>>>>>>>>>>>>>>>>>>>>>> currently used.
>>>>>>>>>>>>>>>>>>>>>>>> From my point of view this is the least confusing way
>>>> for
>>>>>>>>>> DSL
>>>>>>>>>>>>>>>>>>>>>>>> users. If
>>>>>>>>>>>>>>>>>>>>>>>> its to tricky to get a hand on the streams instance
>>>> one
>>>>>>>>>> could
>>>>>>>>>>>>>> ask
>>>>>>>>>>>>>>>>>>>>>>>> the user
>>>>>>>>>>>>>>>>>>>>>>>> to pass it in before executing queries, therefore
>>>> making
>>>>>>>>>> sure
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> streams
>>>>>>>>>>>>>>>>>>>>>>>> instance has been build.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The effort to implement this is indeed some orders of
>>>>>>>>>>>>> magnitude
>>>>>>>>>>>>>>>>>>>>>>>> higher
>>>>>>>>>>>>>>>>>>>>>>>> than the overloaded materialized call. As long as I
>>>> could
>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>>>>>>> getting a
>>>>>>>>>>>>>>>>>>>>>>>> different view I am happy.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I understand your concern. One implication of not
>>>> passing
>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>> store name
>>>>>>>>>>>>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables
>>>> would
>>>>>>>>>> need
>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>>>>>>>> materialised. Currently the store name (or proposed
>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() call)
>>>>>>>>>>>>>>>>>>>>>>>>> act as hints on whether to materialise the KTable or
>>>> not.
>>>>>>>>>>>>>>>>>>>>>>>>> Materialising
>>>>>>>>>>>>>>>>>>>>>>>>> every KTable can be expensive, although there are
>>>> some
>>>>>>>>>> tricks
>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>>>>>>> can play,
>>>>>>>>>>>>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed by
>>>> a
>>>>>>>>>> Kafka
>>>>>>>>>>>>>> topic.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> However, even with the above, after getting an IQ
>>>> handle,
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> user would
>>>>>>>>>>>>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As
>>>> such, we
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>>>>>> continue to be outside the original DSL so this
>>>> wouldn't
>>>>>>>>>>>>>> address
>>>>>>>>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>>>>> original concern.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs by
>>>>>>>>>> removing
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>>>>>> name, at the cost of having to materialise every
>>>> KTable.
>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>>>>>>> definitely an
>>>>>>>>>>>>>>>>>>>>>>>>> option we'll consider as part of this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak <
>>>>>>>>>>>>>> jan.filip...@trivago.com>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Exactly
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I know it works from the Processor API, but my
>>>> suggestion
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>>> prevent
>>>>>>>>>>>>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> In general I am pro switching between DSL and
>>>> Processor
>>>>>>>>>> API
>>>>>>>>>>>>>>>>>>>>>>>>>> easily. (In
>>>>>>>>>>>>>>>>>>>>>>>>>> my Stream applications I do this a lot with
>>>> reflection and
>>>>>>>>>>>>>>>>>>>>>>>>>> instanciating
>>>>>>>>>>>>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that
>>>> there
>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>> be a DSL
>>>>>>>>>>>>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. This
>>>> can be
>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> Method like
>>>>>>>>>>>>>>>>>>>>>>>>>> KTable::retrieveIQHandle():InteractiveQueryHandle,
>>>> the
>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>> would know
>>>>>>>>>>>>>>>>>>>>>>>>>> to materialize, and the user had a reference to the
>>>> "store
>>>>>>>>>>>>>> and the
>>>>>>>>>>>>>>>>>>>>>>>>>> distributed query mechanism by the Interactive Query
>>>>>>>>>> Handle"
>>>>>>>>>>>>>>>>>>>>>>>>>> under the hood
>>>>>>>>>>>>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people
>>>> again.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I hope you see my point J
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but
>>>> can also
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>> used for
>>>>>>>>>>>>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism
>>>> that does
>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>>>>>>>>>> for PAPI
>>>>>>>>>>>>>>>>>>>>>>>>>>> and DSL.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we could
>>>>>>>>>> provide
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>> better API
>>>>>>>>>>>>>>>>>>>>>>>>>>> for KTable stores including the discovery of remote
>>>>>>>>>> shards
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> the same
>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of
>>>> overloads
>>>>>>>>>> and
>>>>>>>>>>>>> I
>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>>>>>>>>>> not a
>>>>>>>>>>>>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a builder
>>>>>>>>>>>>> pattern.
>>>>>>>>>>>>>>>>>>>>>>>>>>> But that
>>>>>>>>>>>>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, if
>>>> we
>>>>>>>>>> would
>>>>>>>>>>>>>> aim
>>>>>>>>>>>>>>>>>>>>>>>>>>> for a API
>>>>>>>>>>>>>>>>>>>>>>>>>>> rework, we should get the changes with regard to
>>>> stores
>>>>>>>>>>>>> right
>>>>>>>>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later
>>>> on.)
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> something like:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> stream.groupyByKey()
>>>>>>>>>>>>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000))
>>>>>>>>>>>>>>>>>>>>>>>>>>> .aggregate(...)
>>>>>>>>>>>>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde())
>>>>>>>>>>>>>>>>>>>>>>>>>>> .withStoreName("storeName);
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy --
>>>> maybe a
>>>>>>>>>>>>>> personal
>>>>>>>>>>>>>>>>>>>>>>>>>>> pain
>>>>>>>>>>>>>>>>>>>>>>>>>>> point right now :))
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i
>>>> don't
>>>>>>>>>> find
>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>>>>>>>>>>>> anywhere
>>>>>>>>>>>>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is
>>>> that
>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to know
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Store name), so we are working on different
>>>> levels
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> achieve a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> single goal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> What is your peoples opinion on having a method on
>>>>>>>>>> KTABLE
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> returns
>>>>>>>>>>>>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is of
>>>> course
>>>>>>>>>>>>>>>>>>>>>>>>>>>> problems like
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are
>>>> going and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> groupmembership
>>>>>>>>>>>>>>>>>>>>>>>>>>>> is established..." but the benefit would be that
>>>> for the
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it
>>>> materialized as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> querries gonna
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can
>>>> execute
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> querries on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 step.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to
>>>> say
>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialize on this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find
>>>> its store
>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for
>>>> the
>>>>>>>>>> store
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land and
>>>>>>>>>>>>> therefore
>>>>>>>>>>>>>>>>>>>>>>>>>>>> maybe
>>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse him less.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always
>>>> need to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it
>>>> doesn't
>>>>>>>>>>>>>> need yet
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> physical state store.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll <
>>>>>>>>>>>>>> mich...@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am more
>>>> in
>>>>>>>>>> favor
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods rather than introducing `materialize()`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for
>>>> e.g.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A related but slightly different question is
>>>> what e.g.
>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipiak
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why
>>>> KIP-114
>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always
>>>> materializing
>>>>>>>>>>>>>> tables/state
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stores.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be
>>>> querable by
>>>>>>>>>> IQ?
>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you uses
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the filter
>>>> and
>>>>>>>>>>>>>> should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or IQ is
>>>>>>>>>>>>> accessing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it? How
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new method help?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional
>>>>>>>>>> materialize
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method being
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have
>>>> strong
>>>>>>>>>>>>> opinions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be unrelated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too
>>>> in case
>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" <
>>>> matth...@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable
>>>>>>>>>>>>> materialization
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> improved
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to
>>>> materialize a
>>>>>>>>>>>>> KTable:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- users
>>>> should
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aware about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better
>>>> expressed by
>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by
>>>> using a
>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overload of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a method.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact and
>>>> mine
>>>>>>>>>> are
>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ;-P
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not really a
>>>> huge
>>>>>>>>>>>>> impact
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMO. It is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability,
>>>> usability of
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> API.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J.
>>>> Sax <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not
>>>> agree with
>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" is
>>>> not as
>>>>>>>>>>>>>> nice)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> than the second version. Adding a
>>>> stateStoreName
>>>>>>>>>>>>>> parameter
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is quite
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- thus, I
>>>> prefer
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verbose
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but explicit version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think it
>>>>>>>>>> interrupts
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> table.mapValue(..).materialize().join(..).materialize()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide
>>>> overloaded
>>>>>>>>>>>>> methods
>>>>>>>>>>>>>> where
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> people can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want,
>>>> otherwise
>>>>>>>>>> we
>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2
>>>> cents:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about
>>>> removing
>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names
>>>> (however, I
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not
>>>> force
>>>>>>>>>> users
>>>>>>>>>>>>>> to call
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but add
>>>> one more
>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the
>>>> KTable is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thus,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a
>>>> parameter
>>>>>>>>>>>>>> storeName
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (ie,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a
>>>> null store
>>>>>>>>>>>>>> name (to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but
>>>> throw an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see
>>>> below).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call
>>>> materialize
>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another
>>>>>>>>>> StateStore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but
>>>> see
>>>>>>>>>> below.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from
>>>> Guozhang, there
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worry
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no
>>>> exception
>>>>>>>>>>>>>> must be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a
>>>> "materialized
>>>>>>>>>>>>>> flag" (ie,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for
>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really
>>>> required. We
>>>>>>>>>>>>> also
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream`
>>>> would make
>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the
>>>>>>>>>>>>>> KStreamBuilder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this KIP).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods but
>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A general note:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments
>>>> "Rejected
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" was
>>>>>>>>>>>>> rejected.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog
>>>> abstraction
>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> secondary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after those changes and the "view" abstraction
>>>> is
>>>>>>>>>> what
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view
>>>>>>>>>> abstraction"
>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> optional).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few
>>>> meta
>>>>>>>>>>>>> comments
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a few
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() function in
>>>>>>>>>> general,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be
>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>
>
