How can I distinguish messages from different topics within one KStream if I subscribed to topics using regex ?
On Thu, Nov 10, 2016 at 7:32 AM, Hans Jespersen <[email protected]> wrote: > I believe that the new topics are picked up at the next metadata refresh > which is controlled by the metadata.max.age.ms parameter. The default > value > is 300000 (which is 5 minutes). > > -hans > > /** > * Hans Jespersen, Principal Systems Engineer, Confluent Inc. > * [email protected] (650)924-2670 > */ > > On Wed, Nov 9, 2016 at 7:39 PM, Haopu Wang <[email protected]> wrote: > > > From the document, the new topics are picked up after some period of > time. > > > > My question is: how long is the duration before the new topics are > > detected and is the duration configurable? > > > > Much thanks! > > > > -----Original Message----- > > From: Matthias J. Sax [mailto:[email protected]] > > Sent: 2016年11月10日 7:28 > > To: [email protected] > > Subject: Re: Is it possible to resubcribe KafkaStreams in runtime to > > different set of topics? > > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA512 > > > > Yes. > > > > If new topics are created that match the regex, Streams will > > automatically subscribed to them. > > > > - -Mathias > > > > On 11/9/16 3:11 PM, Haopu Wang wrote: > > > Hi, do you mean that the new matched topics should be consumed > > > after the regex subscription has been established? Thanks! > > > > > > -----Original Message----- From: Guozhang Wang > > > [mailto:[email protected]] Sent: 2016年11月10日 3:41 To: > > > [email protected] Subject: Re: Is it possible to resubcribe > > > KafkaStreams in runtime to different set of topics? > > > > > > Timur, > > > > > > As Michael said one thing you can consider is the regex > > > subscription if those topics have some common prefix / suffix / > > > pattern. > > > > > > Another idea I can think of is to have two Streams app, one for > > > reading the input topics and pipe them into the repartitioned > > > intermediate topics that can be controlled by the streams app > > > itself, and the second app reading from the intermediate topics and > > > process the rest of the topology. When input topics change, you can > > > simple bounce the first app only, which is a very thin client and > > > hence should take much less time to restart. > > > > > > > > > Guozhang > > > > > > > > > On Wed, Nov 9, 2016 at 12:30 PM, Michael Noll > > > <[email protected]> wrote: > > > > > >> I am not aware of any short-term plans to support that, but > > >> perhaps others in the community / mailing list are. > > >> > > >> On Wed, Nov 9, 2016 at 11:15 AM, Timur Yusupov > > >> <[email protected]> wrote: > > >> > > >>> Are there any nearest plans to support that? > > >>> > > >>> On Wed, Nov 9, 2016 at 1:11 PM, Michael Noll > > >>> <[email protected]> > > >> wrote: > > >>> > > >>>> This is not possible at the moment. However, depending on > > >>>> your use > > >> case, > > >>>> you might be able to leverage regex topic subscriptions > > >>>> (think: "b*" to read from all topics starting with letter > > >>>> `b`). > > >>>> > > >>>> On Wed, Nov 9, 2016 at 10:56 AM, Timur Yusupov > > >>>> <[email protected]> wrote: > > >>>> > > >>>>> Hello, > > >>>>> > > >>>>> In our system it is possible to add/remove topics in > > >>>>> runtime and we > > >> are > > >>>>> trying to use KafkaStreams for incoming messages > > >>>>> processing. > > >>>>> > > >>>>> It is possible to resubscribe KafkaStreams instance to > > >>>>> updated set of topics? > > >>>>> > > >>>>> For now I see the only way is to shutdown exiting > > >>>>> KafkaStreams > > >> instance > > >>>> and > > >>>>> start a new one, but sometimes that takes up to 30-40 > > >>>>> seconds... > > >>>>> > > >>>>> Thanks, Timur. > > >>>>> > > >>>> > > >>> > > >> > > > > > > > > > > > -----BEGIN PGP SIGNATURE----- > > Comment: GPGTools - https://gpgtools.org > > > > iQIcBAEBCgAGBQJYI7DxAAoJECnhiMLycopPrOoP/2+6CXMFvE1jl1lvuR46fFeR > > XQoNB1m0waXOcSBfQVJnIy0sOCQakMHeW6rpc1Ji0w43K80H5Z7K0fC83X/awg1u > > 6LiEaAymTNaFuKyd49CzXSLiwKZKZiw4NpIXvJWPp5gw5EQazEfO84gA5uDBOm5F > > KdXATw8dREhv6PZtnxQ1+1vKsobNS1iGyI/ecjbDVjSFQtCY/97jY0xgoxe3W3H6 > > 2+1NgxVlQ6o27Snkk7umcEPAmkPN1q+HhkZ9pBRUt9xr7MfAZSQqmlgoo28kNlAB > > M4CX+wV8XfL4JfOJqmvSv4FaLLuAKDvVyEHSfiNzxEayeIgnZTqlahRHkr7jW8T6 > > /TiJW+iwuAeERJwfH+0GoQgZDFodVh3/JfeRdTiS4KpnK6MywQP37H3Wp7y90sQt > > eYC9GOr0ZZF1xHjhYN974Wlb+RHmPY8c/FNcXkaZP6nBvUk5NoP4mvGJGImFZbXr > > 9/ENrJRW8HKc7XyXaL/vg8v0d7Hvpv3XTyHsfhUbR9NSufiML655cGRtDRF7ebDb > > MK2RyngdKf0YvXLv4uyQqWhIXSQvtPrqvJbzuUJhTIBOToNf0uupvXJ8CdVQPg80 > > ev7/eDFFTzUkPhYwinfP/Dn2lF6ob2BBmNeDI68mO/Z0vez/zda0lu3ofITo5mDv > > cTYKRiimCr471hNFmpPN > > =EYdQ > > -----END PGP SIGNATURE----- > > > -- С наилучшими пожеланиями, Тимур.
