I believe that the new topics are picked up at the next metadata refresh which is controlled by the metadata.max.age.ms parameter. The default value is 300000 (which is 5 minutes).
-hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * [email protected] (650)924-2670 */ On Wed, Nov 9, 2016 at 7:39 PM, Haopu Wang <[email protected]> wrote: > From the document, the new topics are picked up after some period of time. > > My question is: how long is the duration before the new topics are > detected and is the duration configurable? > > Much thanks! > > -----Original Message----- > From: Matthias J. Sax [mailto:[email protected]] > Sent: 2016年11月10日 7:28 > To: [email protected] > Subject: Re: Is it possible to resubcribe KafkaStreams in runtime to > different set of topics? > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Yes. > > If new topics are created that match the regex, Streams will > automatically subscribed to them. > > - -Mathias > > On 11/9/16 3:11 PM, Haopu Wang wrote: > > Hi, do you mean that the new matched topics should be consumed > > after the regex subscription has been established? Thanks! > > > > -----Original Message----- From: Guozhang Wang > > [mailto:[email protected]] Sent: 2016年11月10日 3:41 To: > > [email protected] Subject: Re: Is it possible to resubcribe > > KafkaStreams in runtime to different set of topics? > > > > Timur, > > > > As Michael said one thing you can consider is the regex > > subscription if those topics have some common prefix / suffix / > > pattern. > > > > Another idea I can think of is to have two Streams app, one for > > reading the input topics and pipe them into the repartitioned > > intermediate topics that can be controlled by the streams app > > itself, and the second app reading from the intermediate topics and > > process the rest of the topology. When input topics change, you can > > simple bounce the first app only, which is a very thin client and > > hence should take much less time to restart. > > > > > > Guozhang > > > > > > On Wed, Nov 9, 2016 at 12:30 PM, Michael Noll > > <[email protected]> wrote: > > > >> I am not aware of any short-term plans to support that, but > >> perhaps others in the community / mailing list are. > >> > >> On Wed, Nov 9, 2016 at 11:15 AM, Timur Yusupov > >> <[email protected]> wrote: > >> > >>> Are there any nearest plans to support that? > >>> > >>> On Wed, Nov 9, 2016 at 1:11 PM, Michael Noll > >>> <[email protected]> > >> wrote: > >>> > >>>> This is not possible at the moment. However, depending on > >>>> your use > >> case, > >>>> you might be able to leverage regex topic subscriptions > >>>> (think: "b*" to read from all topics starting with letter > >>>> `b`). > >>>> > >>>> On Wed, Nov 9, 2016 at 10:56 AM, Timur Yusupov > >>>> <[email protected]> wrote: > >>>> > >>>>> Hello, > >>>>> > >>>>> In our system it is possible to add/remove topics in > >>>>> runtime and we > >> are > >>>>> trying to use KafkaStreams for incoming messages > >>>>> processing. > >>>>> > >>>>> It is possible to resubscribe KafkaStreams instance to > >>>>> updated set of topics? > >>>>> > >>>>> For now I see the only way is to shutdown exiting > >>>>> KafkaStreams > >> instance > >>>> and > >>>>> start a new one, but sometimes that takes up to 30-40 > >>>>> seconds... > >>>>> > >>>>> Thanks, Timur. > >>>>> > >>>> > >>> > >> > > > > > > > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - https://gpgtools.org > > iQIcBAEBCgAGBQJYI7DxAAoJECnhiMLycopPrOoP/2+6CXMFvE1jl1lvuR46fFeR > XQoNB1m0waXOcSBfQVJnIy0sOCQakMHeW6rpc1Ji0w43K80H5Z7K0fC83X/awg1u > 6LiEaAymTNaFuKyd49CzXSLiwKZKZiw4NpIXvJWPp5gw5EQazEfO84gA5uDBOm5F > KdXATw8dREhv6PZtnxQ1+1vKsobNS1iGyI/ecjbDVjSFQtCY/97jY0xgoxe3W3H6 > 2+1NgxVlQ6o27Snkk7umcEPAmkPN1q+HhkZ9pBRUt9xr7MfAZSQqmlgoo28kNlAB > M4CX+wV8XfL4JfOJqmvSv4FaLLuAKDvVyEHSfiNzxEayeIgnZTqlahRHkr7jW8T6 > /TiJW+iwuAeERJwfH+0GoQgZDFodVh3/JfeRdTiS4KpnK6MywQP37H3Wp7y90sQt > eYC9GOr0ZZF1xHjhYN974Wlb+RHmPY8c/FNcXkaZP6nBvUk5NoP4mvGJGImFZbXr > 9/ENrJRW8HKc7XyXaL/vg8v0d7Hvpv3XTyHsfhUbR9NSufiML655cGRtDRF7ebDb > MK2RyngdKf0YvXLv4uyQqWhIXSQvtPrqvJbzuUJhTIBOToNf0uupvXJ8CdVQPg80 > ev7/eDFFTzUkPhYwinfP/Dn2lF6ob2BBmNeDI68mO/Z0vez/zda0lu3ofITo5mDv > cTYKRiimCr471hNFmpPN > =EYdQ > -----END PGP SIGNATURE----- >
