For most of our clusters, we just use auto topic creation and it’s handled that way. Periodically we’ll go through and clean up partition counts across everything if there’s a new high-volume topic. We also have the ability for people to pre-create topics using a central management system.
For the special mirror maker that we have that does 1-to-1 mappings between partitions for clusters that do not have auto topic creation enabled, the topic creation (or partition count changes) are taken care of in the message handler. -Todd On Mon, Dec 5, 2016 at 12:42 AM, James Cheng <wushuja...@gmail.com> wrote: > Hi, > > We are using mirrormaker to mirror topics from one cluster to another, and > I wanted to get some advice from the community on how people are doing > mirroring. In particular, how are people dealing with topic creation? > > Do you turn on auto-topic creation in your destination clusters > (auto.create.topics.enable=true)? > > If not, do you manually create the individual destination topics? > > If so, how does that work with mirroring based on a whitelist (regex)? > > The way we are doing it right now is, we have our regex in a file > somewhere. The regex is used in 2 ways: > 1) Passed to mirrormaker, to do the mirroring. > 2) Passed to a program which looks up all the topics on the source > cluster, finds the ones that match the regex, and then creates them on the > destination cluster. (We have auto-topic creation turned off > auto.create.topics.enable=false) > > One downside of that approach is there a potential race, where if the > regex changes, then mirrormaker (in #1) might start trying to produce to a > new destination topic before the topic was created (by #2). > > Some other hand-wavy ideas that came to mind might be: > * handling topic creation in a MirrorMakerMessageHandler > * handling topic creation in an interceptor > > Anyway, was hoping to get some thoughts from people who are already doing > this. > > Thanks! > -James > > -- *Todd Palino* Staff Site Reliability Engineer Data Infrastructure Streaming linkedin.com/in/toddpalino
