One more additional note to new consumer. The new topic will not be picked up by new consumer immediately. It takes as long as metadata.max.age.ms to refresh metadata and pick up topics that match the pattern at the time of check.
On Thu, 25 Feb 2016 at 08:48 Jason Gustafson <ja...@confluent.io> wrote: > Hey Luke, > > I took a look at the code and it does look like the whitelist argument is > handled differently between the old and new consumers. For the new > consumer, we just treat it as a raw regular expression, but the old > consumer does some preprocessing. We should probably do the preprocessing > in both cases for consistency. Would you mind opening a JIRA? > > Thanks, > Jason > > On Wed, Feb 24, 2016 at 4:17 PM, Luke Steensen < > luke.steen...@braintreepayments.com> wrote: > > > Hello, > > > > I've been experimenting with mirror maker and am a bit confused about > some > > behavior I'm seeing. > > > > I have two simple single broker clusters setup locally, source and > target. > > The source cluster has a topic foo with a few message in it. > > > > When I run mirror maker with --whitelist '*' (as noted in the > > documentation), nothing is written to the target cluster. Switching '.*' > > makes no difference. > > > > With --whitelist '\w*', everything is mirrored, including > > __consumer_offsets. > > > > With --whitelist foo, the foo topic is mirrored correctly from source to > > target. > > > > With --whitelist 'foo|bar' (where foo already exists and bar does not > yet), > > foo is mirrored correctly, but bar is ignored when it gets created. > > Restarting mirror maker with the same command causes it to pick up on > bar. > > > > I assume there is some outdated documentation around old vs new consumer > in > > mirror maker, but it's still not obvious how to automatically mirror all > > non-internal topics. Any pointers would be appreciated. > > > > Thanks, > > Luke > > >
