+1 to this general idea. If Wikimedia ever gets around to switching to MM2, we'll need something like this as our apps manually do the topic prefixing for active <-> active replication.
Perhaps 'Legacy' isn't a good name for this type of replication? There may be more uses cases like Marcin's where folks want to do this not just for migration to MM2 reasons. 'Prefixless' as Marcin suggests is pretty descriptive. :) -Andrew Otto SRE, Wikimedia Foundation On Mon, Mar 16, 2020 at 1:23 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Marcin, I just created this jira per your suggestion: > https://issues.apache.org/jira/browse/KAFKA-9726 > > I've assigned it to myself but I'm happy to give it up. > > Ryanne > > On Mon, Mar 16, 2020 at 2:37 AM Marcin Wieloch <marcin.wiel...@gmail.com> > wrote: > > > Hello Ryanne, > > > > Thank you for these clarifications. I am glad that the use case of > > “prefixless” replication is in-line with your vision for further > > development of MirrorMaker2; I forgot indeed that already KIP-382 > mentioned > > that LegacyReplicationPolicy. > > > > I have not found however any JIRA issue addressing specifically > > LegacyReplicationPolicy, and the main ticket for KIP-382 (KAFKA-7500) is > > already resolved. > > > > So what would be the next step you recommend I take? Should I file my > > feature request (rephrased to adjust for your comments)? Or is the need > for > > the legacy policy already tracked somehow and we don’t have to open any > new > > tickets for it? > > > > Marcin > > > > On 2020/03/11 14:20:44, Ryanne Dolan <ryannedo...@gmail.com> wrote: > > > Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not > yet > > > released) which is similar to your PrefixlessReplicationPolicy and the > > > MigrationReplicationPolicy mentioned. I don't think these are hacky, > > though > > > maybe the documentation could better address what to do when a method > > > doesn't necessarily apply cleanly. > > > > > > Frankly, I anticipated needing to make changes to MM2 itself to > > accommodate > > > these sort of use-cases -- the fact that you got this working as-is is > > > encouraging :) > > > > > > Re the proposal, I think we could ship one or more of these replication > > > policies with MM2 to support prefixless replication. There may be > > slightly > > > different implementations depending on the use-case, i.e. "migration" > > from > > > an old cluster vs active/passive w/o renaming topics vs mimicking MM1 > > with > > > LegacyReplicationPolicy. Maybe they are all identical and we just need > > one? > > > > > > I like that use-cases are driving these requests rather than us try to > > > enumerate all possible ReplicationPolicy implementations. Given a > > > particular use-case and a successful application of a custom > > > ReplicationPolicy, I for one would support including it in Apache > Kafka. > > > > > > Ryanne > > > > > > On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch <marcin.wiel...@gmail.com > > > > > wrote: > > > > > > > Dear All, > > > > > > > > I would like to raise a feature request for ‘prefixless’ topic > > > > mirroring with MirrorMaker2 (see detailed description below). > > > > > > > > As after quite a few tries I am not 100% sure, could you please > > > > confirm that by now > > > > the ‘prefixless’ mirroring is not supported and that my feature > > > > request is valid? I would then file it in JIRA. > > > > > > > > Best regards, > > > > > > > > Marcin > > > > > > > > > > > > Feature Request > > > > > > > > We would like to use MirrorMaker2 as a DRP solution for a simple > setup > > > > with two sites working in active-passive configuration > > > > where all topics are always mirrored in one direction only and the > > > > downstream topics keep their original names, i.e., without being > > > > prefixed by MirrorMaker2. > > > > > > > > After a few rounds of 'trial and error' we came up with a customized > > > > replication policy which, although hacky, seems to fulfill our need; > > > > it is presented in stackoverflow here: > > > > https://stackoverflow.com/a/60619233 > > > > and source code is available here: > > > > > > > > > > > https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java > > > > > > > > This solution is 'hacky' in that it violates specification for the > > > > topicSource and upstreamTopic methods as expressed in javadoc of the > > > > ReplicationPolicy interface. > > > > > > > > That makes us suspect that MirrorMaker2 has been never meant to work > > > > in such a configuration and that our solution may stop > > > > working with future releases of Kafka. > > > > > > > > We would like then to ask for making the possibility of 'prefixless' > > > > topic mirroring in a simplified two-cluster setup a standard > > > > feature of Kafka Mirror Maker 2. > > > > > > > > > >
