I agree that the API can be improved and we are working on that. Btw: KStream#toTable() was already suggested in KIP-114 discussion:
http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics However for now, you can only choose from the two options as described. -Matthias On 2/10/17 1:49 PM, Nick DeCoursin wrote: > To be honest, I don't think either of these options are very good. > > > stream.to("some-other-topic"); > builder.table("some-other-topic"); > > > As explained here > <http://mail-archives.apache.org/mod_mbox/kafka-users/201702.mbox/browser>, > if the underlying topic doesn't have cleanup.policy=compact, the state > store may invisibly lose data! > > As for the second option, it's obviously verbose, and it doesn't let me > choose the topic name. As a result, I end with some long weird changelog > topic name; and, some of the Kafka Connectors map topics to the table name, > eeeh. > > To improve this, just add a topic parameter to both of the overloaded > KGroupedStream.reduce methods: > > reduce(Reducer<V> reducer, String, topicName, String storeName) > > > Furthermore, there should be a short cut to the second option, like this: > > // Creates a KTable from the KStream where > // the key of the KStream is the key of the KTable. > // Any latter key overwrites the former. > someStream.table( > Serde<K>, > Serde<V>, > topicName, > tableName > ); > > > or maybe the Serdes can be inferred? Either way, this would be a nice clean > approach to a (maybe?) common use case. > > Thank you, > Nick > > On 25 January 2017 at 07:46, Nick DeCoursin <[email protected]> > wrote: > >> Thank you very much, both suggestions are wonderful, and I will try them. >> Have a great day! >> >> Kind regards, >> Nick >> >> On 24 January 2017 at 19:46, Matthias J. Sax <[email protected]> >> wrote: >> >>> If your data is already partitioned by key, you can save writing to a >>> topic by doing a dummy reduce instead: >>> >>> stream >>> .groupByKey() >>> .reduce(new Reducer() { >>> V apply(V value1, V value2) { >>> return value2; >>> } >>> }, >>> "yourStoreName"); >>> >>> (replace V with your actuall value type) >>> >>> >>> -Matthias >>> >>> On 1/24/17 8:53 AM, Damian Guy wrote: >>>> Hi Nick, >>>> >>>> I guess there is some reason why you can't just build it as a table to >>>> begin with? >>>> There isn't a convenient method for doing this right now, but you could >>> do >>>> something like: >>>> >>>> stream.to("some-other-topic"); >>>> builder.table("some-other-topic"); >>>> >>>> Thanks, >>>> Damian >>>> >>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <[email protected]> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I >>> want to >>>>> create a table from it backed by a state store. The key of the stream >>> could >>>>> be the same as the table. >>>>> >>>>> I've tried following examples, but it seems all examples use `groupBy` >>> or >>>>> `count` to convert `KStream`s into `KTable`s. Is there any other way? >>>>> >>>>> Thank you very much, >>>>> Nick DeCoursin >>>>> >>>>> -- >>>>> >>>>> Nick DeCoursin >>>>> Software Engineer >>>>> foodpanda >>>>> >>>>> Tel | +1 920 450 5434 <(920)%20450-5434> >>>>> >>>>> Mail | [email protected] >>>>> >>>>> Skype | nick.foodpanda >>>>> >>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany >>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B | >>>>> USt-ID-Nr | DE 283789080 >>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel >>>>> >>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is >>>>> confidential and may be privileged. It may be read, copied and used >>> only by >>>>> the intended recipient. If you have received it in error please >>> contact the >>>>> sender (by return e-mail) immediately and delete this message. Any >>>>> unauthorized use or dissemination of this message in whole or in parts >>> is >>>>> strictly prohibited. >>>>> >>>> >>> >>> >> >> >> -- >> >> Nick DeCoursin >> Software Engineer >> foodpanda >> >> Tel | +1 920 450 5434 <(920)%20450-5434> >> >> Mail | [email protected] >> >> Skype | nick.foodpanda >> >> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany >> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B | >> USt-ID-Nr | DE 283789080 >> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel >> >> CONFIDENTIALITY NOTICE: This message (including any attachments) is >> confidential and may be privileged. It may be read, copied and used only by >> the intended recipient. If you have received it in error please contact the >> sender (by return e-mail) immediately and delete this message. Any >> unauthorized use or dissemination of this message in whole or in parts is >> strictly prohibited. >> > > >
signature.asc
Description: OpenPGP digital signature
