-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Yes. You need to use Processor API to access the topic name via ProcessorContext that is given via Processor#init().
If you use DSL, you can mix-and-match with Processor API via KStream#process() - -Matthias On 11/14/16 12:42 AM, Timur Yusupov wrote: > How can I distinguish messages from different topics within one > KStream if I subscribed to topics using regex ? > > On Thu, Nov 10, 2016 at 7:32 AM, Hans Jespersen <[email protected]> > wrote: > >> I believe that the new topics are picked up at the next metadata >> refresh which is controlled by the metadata.max.age.ms parameter. >> The default value is 300000 (which is 5 minutes). >> >> -hans >> >> /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. >> * [email protected] (650)924-2670 */ >> >> On Wed, Nov 9, 2016 at 7:39 PM, Haopu Wang <[email protected]> >> wrote: >> >>> From the document, the new topics are picked up after some >>> period of >> time. >>> >>> My question is: how long is the duration before the new topics >>> are detected and is the duration configurable? >>> >>> Much thanks! >>> >>> -----Original Message----- From: Matthias J. Sax >>> [mailto:[email protected]] Sent: 2016年11月10日 7:28 To: >>> [email protected] Subject: Re: Is it possible to >>> resubcribe KafkaStreams in runtime to different set of topics? >>> > Yes. > > If new topics are created that match the regex, Streams will > automatically subscribed to them. > > -Mathias > > On 11/9/16 3:11 PM, Haopu Wang wrote: >>>>> Hi, do you mean that the new matched topics should be >>>>> consumed after the regex subscription has been established? >>>>> Thanks! >>>>> >>>>> -----Original Message----- From: Guozhang Wang >>>>> [mailto:[email protected]] Sent: 2016年11月10日 3:41 To: >>>>> [email protected] Subject: Re: Is it possible to >>>>> resubcribe KafkaStreams in runtime to different set of >>>>> topics? >>>>> >>>>> Timur, >>>>> >>>>> As Michael said one thing you can consider is the regex >>>>> subscription if those topics have some common prefix / >>>>> suffix / pattern. >>>>> >>>>> Another idea I can think of is to have two Streams app, one >>>>> for reading the input topics and pipe them into the >>>>> repartitioned intermediate topics that can be controlled by >>>>> the streams app itself, and the second app reading from the >>>>> intermediate topics and process the rest of the topology. >>>>> When input topics change, you can simple bounce the first >>>>> app only, which is a very thin client and hence should take >>>>> much less time to restart. >>>>> >>>>> >>>>> Guozhang >>>>> >>>>> >>>>> On Wed, Nov 9, 2016 at 12:30 PM, Michael Noll >>>>> <[email protected]> wrote: >>>>> >>>>>> I am not aware of any short-term plans to support that, >>>>>> but perhaps others in the community / mailing list are. >>>>>> >>>>>> On Wed, Nov 9, 2016 at 11:15 AM, Timur Yusupov >>>>>> <[email protected]> wrote: >>>>>> >>>>>>> Are there any nearest plans to support that? >>>>>>> >>>>>>> On Wed, Nov 9, 2016 at 1:11 PM, Michael Noll >>>>>>> <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>>> This is not possible at the moment. However, >>>>>>>> depending on your use >>>>>> case, >>>>>>>> you might be able to leverage regex topic >>>>>>>> subscriptions (think: "b*" to read from all topics >>>>>>>> starting with letter `b`). >>>>>>>> >>>>>>>> On Wed, Nov 9, 2016 at 10:56 AM, Timur Yusupov >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> In our system it is possible to add/remove topics >>>>>>>>> in runtime and we >>>>>> are >>>>>>>>> trying to use KafkaStreams for incoming messages >>>>>>>>> processing. >>>>>>>>> >>>>>>>>> It is possible to resubscribe KafkaStreams instance >>>>>>>>> to updated set of topics? >>>>>>>>> >>>>>>>>> For now I see the only way is to shutdown exiting >>>>>>>>> KafkaStreams >>>>>> instance >>>>>>>> and >>>>>>>>> start a new one, but sometimes that takes up to >>>>>>>>> 30-40 seconds... >>>>>>>>> >>>>>>>>> Thanks, Timur. >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>> >> > > > -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJYKgOmAAoJECnhiMLycopPAwUP/iZsyWmjFuBMM/gnUwlb8tIh 3SQbg85JHlBz/qy3/zd98tdEamltCzACA4TTH6/esX/sJJBhFJFEdS38CioPNtuV YnKamDQ/5IqVUpywjaq3d9eaKfabjcKpt+9Dy1WngKHAYrw7QJlW0V/5nSmPF2t/ +9xhudC8gczdW+9QeEp7OKHDZc3r0zMJ0Vqv7v0gaGasJGQp9mk+Qe6ImxR6KN8i 1qwXBjF1QBnU6gHNpcJkmk/YuoARiEKNZuajo9Acoj8vC7J5yCNTwdsu7IRW8IIa eX7PyU/f6OBQU5n3mK2vvonGxW7lm0Sx8VgyC73Fyv0hWdRHsNovMxwdlOEFJwd1 vtrMNEAwSr40z0PQGYgL0t+Q5WOGrCaNblyvTJDfFtC7tQOIhiFdA+/A9OY43vDx qXyKUmlj9L6FbfnGs/yQJbcSRqUvuthcnUrRRVpixwh5Tp/AQ0A+fexNKnvAOiUS 5jBt27IQ5PlK4L01h0b+Xv1fxs8JWlGJg1+j3U8IPtDcoovBRPsZVcouGMZK4ozb DRQ16btIF7/6/SvY4RVJxV4NkbfwLqDS8aTbVuVH177cQtGYCjmsem1E9LnN3hmd j9cURQ6I3ekhu9X2HlvqLBKeyLRaIDdQQO7W22jdFzDGim0wd2klGq3eXCmlim/Q Q8mKg4y5miDMrV2YyAqb =9CBM -----END PGP SIGNATURE-----
