Sounds good, thank you! Kind regards, Nick
On 10 February 2017 at 22:59, Matthias J. Sax <matth...@confluent.io> wrote: > I agree that the API can be improved and we are working on that. > > Btw: KStream#toTable() was already suggested in KIP-114 discussion: > > http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj= > Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics > > > However for now, you can only choose from the two options as described. > > > -Matthias > > > > On 2/10/17 1:49 PM, Nick DeCoursin wrote: > > To be honest, I don't think either of these options are very good. > > > > > > stream.to("some-other-topic"); > > builder.table("some-other-topic"); > > > > > > As explained here > > <http://mail-archives.apache.org/mod_mbox/kafka-users/ > 201702.mbox/browser>, > > if the underlying topic doesn't have cleanup.policy=compact, the state > > store may invisibly lose data! > > > > As for the second option, it's obviously verbose, and it doesn't let me > > choose the topic name. As a result, I end with some long weird changelog > > topic name; and, some of the Kafka Connectors map topics to the table > name, > > eeeh. > > > > To improve this, just add a topic parameter to both of the overloaded > > KGroupedStream.reduce methods: > > > > reduce(Reducer<V> reducer, String, topicName, String storeName) > > > > > > Furthermore, there should be a short cut to the second option, like this: > > > > // Creates a KTable from the KStream where > > // the key of the KStream is the key of the KTable. > > // Any latter key overwrites the former. > > someStream.table( > > Serde<K>, > > Serde<V>, > > topicName, > > tableName > > ); > > > > > > or maybe the Serdes can be inferred? Either way, this would be a nice > clean > > approach to a (maybe?) common use case. > > > > Thank you, > > Nick > > > > On 25 January 2017 at 07:46, Nick DeCoursin <n.decour...@foodpanda.com> > > wrote: > > > >> Thank you very much, both suggestions are wonderful, and I will try > them. > >> Have a great day! > >> > >> Kind regards, > >> Nick > >> > >> On 24 January 2017 at 19:46, Matthias J. Sax <matth...@confluent.io> > >> wrote: > >> > >>> If your data is already partitioned by key, you can save writing to a > >>> topic by doing a dummy reduce instead: > >>> > >>> stream > >>> .groupByKey() > >>> .reduce(new Reducer() { > >>> V apply(V value1, V value2) { > >>> return value2; > >>> } > >>> }, > >>> "yourStoreName"); > >>> > >>> (replace V with your actuall value type) > >>> > >>> > >>> -Matthias > >>> > >>> On 1/24/17 8:53 AM, Damian Guy wrote: > >>>> Hi Nick, > >>>> > >>>> I guess there is some reason why you can't just build it as a table to > >>>> begin with? > >>>> There isn't a convenient method for doing this right now, but you > could > >>> do > >>>> something like: > >>>> > >>>> stream.to("some-other-topic"); > >>>> builder.table("some-other-topic"); > >>>> > >>>> Thanks, > >>>> Damian > >>>> > >>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin < > n.decour...@foodpanda.com> > >>>> wrote: > >>>> > >>>>> Hello, > >>>>> > >>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I > >>> want to > >>>>> create a table from it backed by a state store. The key of the stream > >>> could > >>>>> be the same as the table. > >>>>> > >>>>> I've tried following examples, but it seems all examples use > `groupBy` > >>> or > >>>>> `count` to convert `KStream`s into `KTable`s. Is there any other way? > >>>>> > >>>>> Thank you very much, > >>>>> Nick DeCoursin > >>>>> > >>>>> -- > >>>>> > >>>>> Nick DeCoursin > >>>>> Software Engineer > >>>>> foodpanda > >>>>> > >>>>> Tel | +1 920 450 5434 <(920)%20450-5434> > >>>>> > >>>>> Mail | n.decour...@foodpanda.com > >>>>> > >>>>> Skype | nick.foodpanda > >>>>> > >>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany > >>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B | > >>>>> USt-ID-Nr | DE 283789080 > >>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel > >>>>> > >>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is > >>>>> confidential and may be privileged. It may be read, copied and used > >>> only by > >>>>> the intended recipient. If you have received it in error please > >>> contact the > >>>>> sender (by return e-mail) immediately and delete this message. Any > >>>>> unauthorized use or dissemination of this message in whole or in > parts > >>> is > >>>>> strictly prohibited. > >>>>> > >>>> > >>> > >>> > >> > >> > >> -- > >> > >> Nick DeCoursin > >> Software Engineer > >> foodpanda > >> > >> Tel | +1 920 450 5434 <(920)%20450-5434> > >> > >> Mail | n.decour...@foodpanda.com > >> > >> Skype | nick.foodpanda > >> > >> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany > >> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B | > >> USt-ID-Nr | DE 283789080 > >> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel > >> > >> CONFIDENTIALITY NOTICE: This message (including any attachments) is > >> confidential and may be privileged. It may be read, copied and used > only by > >> the intended recipient. If you have received it in error please contact > the > >> sender (by return e-mail) immediately and delete this message. Any > >> unauthorized use or dissemination of this message in whole or in parts > is > >> strictly prohibited. > >> > > > > > > > > -- Nick DeCoursin Software Engineer foodpanda Tel | +1 920 450 5434 Mail | n.decour...@foodpanda.com Skype | nick.foodpanda Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B | USt-ID-Nr | DE 283789080 Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel CONFIDENTIALITY NOTICE: This message (including any attachments) is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error please contact the sender (by return e-mail) immediately and delete this message. Any unauthorized use or dissemination of this message in whole or in parts is strictly prohibited.
