For my own clarity, is there any actual distinction between
`stream.to('topic')`
where `topic` is set to compact and the upcoming `stream.toTable('topic')`
if you're not going to immediately use the table in this topology, i.e. if
you want to use it as a table in some other processor application? Am I
right in my understanding that the table aspect is related to behaviour
within this specific processor (GlobalKTables aside)?
On 10 February 2017 at 21:59, Matthias J. Sax <[email protected]> wrote:
> I agree that the API can be improved and we are working on that.
>
> Btw: KStream#toTable() was already suggested in KIP-114 discussion:
>
> http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=
> Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics
>
>
> However for now, you can only choose from the two options as described.
>
>
> -Matthias
>
>
>
> On 2/10/17 1:49 PM, Nick DeCoursin wrote:
> > To be honest, I don't think either of these options are very good.
> >
> >
> > stream.to("some-other-topic");
> > builder.table("some-other-topic");
> >
> >
> > As explained here
> > <http://mail-archives.apache.org/mod_mbox/kafka-users/
> 201702.mbox/browser>,
> > if the underlying topic doesn't have cleanup.policy=compact, the state
> > store may invisibly lose data!
> >
> > As for the second option, it's obviously verbose, and it doesn't let me
> > choose the topic name. As a result, I end with some long weird changelog
> > topic name; and, some of the Kafka Connectors map topics to the table
> name,
> > eeeh.
> >
> > To improve this, just add a topic parameter to both of the overloaded
> > KGroupedStream.reduce methods:
> >
> > reduce(Reducer<V> reducer, String, topicName, String storeName)
> >
> >
> > Furthermore, there should be a short cut to the second option, like this:
> >
> > // Creates a KTable from the KStream where
> > // the key of the KStream is the key of the KTable.
> > // Any latter key overwrites the former.
> > someStream.table(
> > Serde<K>,
> > Serde<V>,
> > topicName,
> > tableName
> > );
> >
> >
> > or maybe the Serdes can be inferred? Either way, this would be a nice
> clean
> > approach to a (maybe?) common use case.
> >
> > Thank you,
> > Nick
> >
> > On 25 January 2017 at 07:46, Nick DeCoursin <[email protected]>
> > wrote:
> >
> >> Thank you very much, both suggestions are wonderful, and I will try
> them.
> >> Have a great day!
> >>
> >> Kind regards,
> >> Nick
> >>
> >> On 24 January 2017 at 19:46, Matthias J. Sax <[email protected]>
> >> wrote:
> >>
> >>> If your data is already partitioned by key, you can save writing to a
> >>> topic by doing a dummy reduce instead:
> >>>
> >>> stream
> >>> .groupByKey()
> >>> .reduce(new Reducer() {
> >>> V apply(V value1, V value2) {
> >>> return value2;
> >>> }
> >>> },
> >>> "yourStoreName");
> >>>
> >>> (replace V with your actuall value type)
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 1/24/17 8:53 AM, Damian Guy wrote:
> >>>> Hi Nick,
> >>>>
> >>>> I guess there is some reason why you can't just build it as a table to
> >>>> begin with?
> >>>> There isn't a convenient method for doing this right now, but you
> could
> >>> do
> >>>> something like:
> >>>>
> >>>> stream.to("some-other-topic");
> >>>> builder.table("some-other-topic");
> >>>>
> >>>> Thanks,
> >>>> Damian
> >>>>
> >>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <
> [email protected]>
> >>>> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
> >>> want to
> >>>>> create a table from it backed by a state store. The key of the stream
> >>> could
> >>>>> be the same as the table.
> >>>>>
> >>>>> I've tried following examples, but it seems all examples use
> `groupBy`
> >>> or
> >>>>> `count` to convert `KStream`s into `KTable`s. Is there any other way?
> >>>>>
> >>>>> Thank you very much,
> >>>>> Nick DeCoursin
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Nick DeCoursin
> >>>>> Software Engineer
> >>>>> foodpanda
> >>>>>
> >>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>>>>
> >>>>> Mail | [email protected]
> >>>>>
> >>>>> Skype | nick.foodpanda
> >>>>>
> >>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >>>>> USt-ID-Nr | DE 283789080
> >>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>>>>
> >>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >>>>> confidential and may be privileged. It may be read, copied and used
> >>> only by
> >>>>> the intended recipient. If you have received it in error please
> >>> contact the
> >>>>> sender (by return e-mail) immediately and delete this message. Any
> >>>>> unauthorized use or dissemination of this message in whole or in
> parts
> >>> is
> >>>>> strictly prohibited.
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >> --
> >>
> >> Nick DeCoursin
> >> Software Engineer
> >> foodpanda
> >>
> >> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>
> >> Mail | [email protected]
> >>
> >> Skype | nick.foodpanda
> >>
> >> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >> USt-ID-Nr | DE 283789080
> >> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>
> >> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >> confidential and may be privileged. It may be read, copied and used
> only by
> >> the intended recipient. If you have received it in error please contact
> the
> >> sender (by return e-mail) immediately and delete this message. Any
> >> unauthorized use or dissemination of this message in whole or in parts
> is
> >> strictly prohibited.
> >>
> >
> >
> >
>
>
