Or perhaps we could provide a more general ReplicationPolicy that renames according to a pattern string, and the Legacy and Prefixless use-cases could just use that.
On Mon, Mar 16, 2020 at 12:32 PM Andrew Otto <o...@wikimedia.org> wrote: > +1 to this general idea. If Wikimedia ever gets around to switching to > MM2, we'll need something like this as our apps manually do the topic > prefixing for active <-> active replication. > > Perhaps 'Legacy' isn't a good name for this type of replication? There may > be more uses cases like Marcin's where folks want to do this not just for > migration to MM2 reasons. 'Prefixless' as Marcin suggests is pretty > descriptive. :) > > -Andrew Otto > SRE, Wikimedia Foundation > > > On Mon, Mar 16, 2020 at 1:23 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > Marcin, I just created this jira per your suggestion: > > https://issues.apache.org/jira/browse/KAFKA-9726 > > > > I've assigned it to myself but I'm happy to give it up. > > > > Ryanne > > > > On Mon, Mar 16, 2020 at 2:37 AM Marcin Wieloch <marcin.wiel...@gmail.com > > > > wrote: > > > > > Hello Ryanne, > > > > > > Thank you for these clarifications. I am glad that the use case of > > > “prefixless” replication is in-line with your vision for further > > > development of MirrorMaker2; I forgot indeed that already KIP-382 > > mentioned > > > that LegacyReplicationPolicy. > > > > > > I have not found however any JIRA issue addressing specifically > > > LegacyReplicationPolicy, and the main ticket for KIP-382 (KAFKA-7500) > is > > > already resolved. > > > > > > So what would be the next step you recommend I take? Should I file my > > > feature request (rephrased to adjust for your comments)? Or is the need > > for > > > the legacy policy already tracked somehow and we don’t have to open any > > new > > > tickets for it? > > > > > > Marcin > > > > > > On 2020/03/11 14:20:44, Ryanne Dolan <ryannedo...@gmail.com> wrote: > > > > Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not > > yet > > > > released) which is similar to your PrefixlessReplicationPolicy and > the > > > > MigrationReplicationPolicy mentioned. I don't think these are hacky, > > > though > > > > maybe the documentation could better address what to do when a method > > > > doesn't necessarily apply cleanly. > > > > > > > > Frankly, I anticipated needing to make changes to MM2 itself to > > > accommodate > > > > these sort of use-cases -- the fact that you got this working as-is > is > > > > encouraging :) > > > > > > > > Re the proposal, I think we could ship one or more of these > replication > > > > policies with MM2 to support prefixless replication. There may be > > > slightly > > > > different implementations depending on the use-case, i.e. "migration" > > > from > > > > an old cluster vs active/passive w/o renaming topics vs mimicking MM1 > > > with > > > > LegacyReplicationPolicy. Maybe they are all identical and we just > need > > > one? > > > > > > > > I like that use-cases are driving these requests rather than us try > to > > > > enumerate all possible ReplicationPolicy implementations. Given a > > > > particular use-case and a successful application of a custom > > > > ReplicationPolicy, I for one would support including it in Apache > > Kafka. > > > > > > > > Ryanne > > > > > > > > On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch < > marcin.wiel...@gmail.com > > > > > > > wrote: > > > > > > > > > Dear All, > > > > > > > > > > I would like to raise a feature request for ‘prefixless’ topic > > > > > mirroring with MirrorMaker2 (see detailed description below). > > > > > > > > > > As after quite a few tries I am not 100% sure, could you please > > > > > confirm that by now > > > > > the ‘prefixless’ mirroring is not supported and that my feature > > > > > request is valid? I would then file it in JIRA. > > > > > > > > > > Best regards, > > > > > > > > > > Marcin > > > > > > > > > > > > > > > Feature Request > > > > > > > > > > We would like to use MirrorMaker2 as a DRP solution for a simple > > setup > > > > > with two sites working in active-passive configuration > > > > > where all topics are always mirrored in one direction only and the > > > > > downstream topics keep their original names, i.e., without being > > > > > prefixed by MirrorMaker2. > > > > > > > > > > After a few rounds of 'trial and error' we came up with a > customized > > > > > replication policy which, although hacky, seems to fulfill our > need; > > > > > it is presented in stackoverflow here: > > > > > https://stackoverflow.com/a/60619233 > > > > > and source code is available here: > > > > > > > > > > > > > > > > https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java > > > > > > > > > > This solution is 'hacky' in that it violates specification for the > > > > > topicSource and upstreamTopic methods as expressed in javadoc of > the > > > > > ReplicationPolicy interface. > > > > > > > > > > That makes us suspect that MirrorMaker2 has been never meant to > work > > > > > in such a configuration and that our solution may stop > > > > > working with future releases of Kafka. > > > > > > > > > > We would like then to ask for making the possibility of > 'prefixless' > > > > > topic mirroring in a simplified two-cluster setup a standard > > > > > feature of Kafka Mirror Maker 2. > > > > > > > > > > > > > > >
