Fair enough with separate github plugin. I think we have consensus here from previous discussions to go ahead and create such repo.
We should probably create a separate discussion now on a plugin repo, and move back to the quorum discussion here. But I believe we have a solution in place? On Tue, Mar 17, 2020 at 9:46 PM michael.andre.pearce <michael.andre.pea...@me.com.invalid> wrote: > > I was more refereeing to supporting the plugin.Concerns raised were:Other > systems client dependencies e.g. kafka client lib or Prometheus lib in core > broker, getting out of dateExposure of cves in core broker that every plugin > with its dependencies could bring.Plugin code becoming un-maintained e.g. the > older vertex ones that were removed. People seemed to be against having > plugins being in broker and then lapsing in maintenance.All I care for is we > have a solution that we agree on and stop having to revisit this, and its a > simply process to contribute new ones. I disagree strongly that it should be > case by case, we should have an agreed place and process.Both the above can > be sorted by either being simply accepted risk, or by being separately > maintained in a seperate plugin repo.My personal is take the risk as will be > also easier to deliver.But I understand the concerns of others, it might be > easier to go the new git repo for plugins (i see this alot in other projects) > Sent from my Samsung Galaxy smartphone. > -------- Original message --------From: Clebert Suconic > <clebert.suco...@gmail.com> Date: 18/03/2020 01:25 (GMT+00:00) To: > dev@activemq.apache.org Subject: Re: Re: [DISCUSSION] New Quorum vote > pluggable implementation >> no longer no one supports it <<I think this is > just apache license here.As long as you have community around it.. code is > maintained.. it'sall fine. I mean... in apache terms you guarantee things as > in apachelicense. I try to do good work on the open source community, > fixingbugs.. everything... but as you know the limits of the apache > licenseterms.The term support here I think applies to third parties.. > companieslisted here, (Including the company I work for... Red Hat / > IBM).:https://activemq.apache.org/support.htmlIt will be up to them what they > consider supported or not.. and thatgoes both for plugins or core > functionalities. Company X may decide tonot support certain functionality as > in providing you patches, help..etc. and you would be on your own on such > functionality like any otherapache license.that's pretty standard... not just > for ActiveMQ.. I think that'scommon for any other project.On Tue, Mar 17, > 2020 at 9:05 PM michael.andre.pearce<michael.andre.pea...@me.com.invalid> > wrote:>> I totally get it.As said im for the camp that we have plugin modules > in the broker. And if someone wants to commit one, we allow it without need > to votes etc and all this discussion every time.(e.g. Justin and I shouldnt > need to make discussion thread, it should just be accepted and added)If > something gets un-maintained (some peoples concerns) we simply remove it / > attic it. The fact its un-maintained means no longer no one supports it But > its for everyone to agree to this more open way of working.....not just me > and you.An alternative approach if people are concerned these are in broker > is that we have a separate plugin repo that we stick them all, and > release...but this then makes it harder for end users to enable or use...Sent > from my Samsung Galaxy smartphone.> -------- Original message --------From: > Clebert Suconic <clebert.suco...@gmail.com> Date: 18/03/2020 00:57 > (GMT+00:00) To: dev@activemq.apache.org Subject: Re: Re: [DISCUSSION] New > Quorum vote pluggable implementation Just like anything else in apache... > committers/community can ask orvote features out. it doesn't even need to be > a plugin.I don't mean to hijack this discussion away from Quorum.. although > itwould affect the implementation.. .we can keep a plugin's area in > thebroker... and activate them based on the CLI / config.I just want to make > sure we can unlock this going forward.On Tue, Mar 17, 2020 at 8:53 PM > michael.andre.pearce<michael.andre.pea...@me.com.invalid> wrote:>> Well the > point is without us having sorted the issue, then this will keep occuring.I > am for one in favour we relax a little and if someone wants to add a plugin > we simply allow it. Why should there be any filter.Sent from my Samsung > Galaxy smartphone.> -------- Original message --------From: Clebert Suconic > <clebert.suco...@gmail.com> Date: 18/03/2020 00:38 (GMT+00:00) To: > dev@activemq.apache.org Subject: Re: Re: [DISCUSSION] New Quorum vote > pluggable implementation It's on a case by case basis... What we can't do is > transform everynew core functionality (such as monitoring. .it's pretty > standard /core functionality) into a plug-in, and then block its > implementationbased on a past discussion.I say, if there's a plugin, there > should be at least oneimplementation on the broker.. that goes for > Monitoring... we cansupport more.. but this has to have an implementation.You > are a committer, if you want to have your kafka plugin again, justraise a > discussion.. but we can't block every new functionality basedon that > baggage.On Tue, Mar 17, 2020 at 7:35 PM > michael.andre.pearce<michael.andre.pea...@me.com.invalid> wrote:>> I think > the issue with plugins is that whats the rule to include one or not.You know > exactly where im coming from there... re kafka plugin.Im totally for a number > of plugins being added. Including the kafka one.But we have to have a clear > rule of what will be allowed or not and a place to maintain.Sent from my > Samsung Galaxy smartphone.> -------- Original message --------From: Clebert > Suconic <clebert.suco...@gmail.com> Date: 16/03/2020 01:20 (GMT+00:00) To: > dev@activemq.apache.org Subject: Re: Re: [DISCUSSION] New Quorum vote > pluggable implementation We can make pluggable as much as you want, but going > forward I wouldlike us to change how we have been treating plugins.Because of > other discussions, plugins are not really implemented aspart of the > community.We should elect one implementation we can make it and implement > itwell.... and bring it as part of the community. People can add > moreimplementations as part of the open source project.This is probably the > same with Micrometer. Justin implementedMicrometer and the implementation is > not part of activemq. I think weshould change that going forward.This will be > a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, > Mar 13, 2020 at 3:42 AM > michael.andre.pearce<michael.andre.pea...@me.com.invalid> wrote:>> I think a > main thing here is a pluggable api.If someone already has etcd, zookeeper, > consul etc. In their infrastructure they should be able to reuse.Like wise if > you have multiple bulkheaded broker sets it be useful to be able to reuse the > same quorum compontent e.g. in ZK that would be a different base root on the > tree path per broker set.Likewise that said we should have some out the box > support plugins from the community. Obviously what ever we do whilst it maybe > a break change e.g. a 3.0 release there must be a migration solution > (accepted that it may need outage) for existing brokers that works via the > pluggable api not to a specific solution.Also anything from a management api > pov should also be visible in the web management console. Sent from my > Samsung Galaxy smartphone.> -------- Original message --------From: Clebert > Suconic <clebert.suco...@gmail.com> Date: 09/03/2020 14:55 (GMT+00:00) To: > dev@activemq.apache.org Subject: Re: Re: [DISCUSSION] New Quorum vote > pluggable implementation I think we should have a management component, that > runs outside ofthe broker and would manage quorum.That way you can have the > quorum running outside of the broker itself.which would improve the need of > multiple brokers to manage the quorum.You just need Quorum Managers in > distinct places.I had recently worked with a software called Ceph... And Ceph > has theconcept of managers working away from their "broker" (it's not > abroker.. it's a DB, but in a sense it's the same concept here). Ithink we > should do the same.I have talked to Franz and other guys in private.. and it > seemseverybody these days mention raft consensus algorithm. Perhaps weshould > look into it.. and make it pluggable. I believe there's JRaftworking on top > of JGroups already.If we make this pluggable and make it a manager a separate > process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor > <andy.tayl...@gmail.com> wrote:>> Personally I wouldn't use Zookeeper, I > think there are better options. Also> looks like Kafka are replacing it as > well. Saying that, it doesn't really> matter what is used, the main thing is > we need to remove the burden of> providing consensus away from the broker.>> > It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, > KimmKing <kimmk...@apache.org> wrote:>> > Can't agree any more.> > The > HA&Replication is more and more important for a modern messaging> > system.> > > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > > And In my experience this also bring some extra-complexity about> > > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 > 20:33:31, "Martyn Taylor" <m...@martyntaylor.me> wrote:> > >I think this is a > great idea Franz. The HA and replication components> > have> > >been a > source of issues over the years. Two problems I see are that 1)> > >there > isn't a clean separation between the consensus mechanism and the> > > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > > >any standard algorithm or a research paper. Hence, all the issues that> > > >were caught over the years due to various edge cases. Integration with> > > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > > >off to a third party lib). I suspect though, this will be a considerable> > > >amount of work and is likely to introduce new issues, so I'd proceed with> > > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM > nigro_franz <nigro....@gmail.com> wrote:> > >> > >> Hi folks,> > >>> > >> > especially due to the requirements of the current Artemis quorum vote> > >> > algorithm, we've thought to re-implementing it with a different focus in> > > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft > implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the > election phase and cluster member states (ie> > it> > >> should be the > Topology shared between them)> > >> 3) to simplify most common setups in both > amount of configuration and> > >> requirements (eg "witness" nodes could be > implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of > forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to > reduce/eliminate implicit "good practices" in term of> > >> order of actions > to be performed on nodes in "special states" eg proper> > >> restart sequence > after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store > and replication behaviour more> > >> similar:> > >> journal's presence should > be the only difference between the 2s> > >>> > >> A proposal of steps to be > followed to get this:> > >> 1) abstract away the current quorum vote: it > requires extra-care because> > >> the> > >> logic is melted together with the > replication/clustering behaviour> > >> 2) refactor it in order to separate > election phase and cluster member> > >> states> > >> 3) implement a RI > version of such APIs> > >>> > >> Post-actions to help people adopt it, but > need to be thought upfront:> > >> 1) a clean upgrade path for current HA > replication users> > >> 2) deprecate or integrate the current HA replication > into the new> > version> > >>> > >> I've opened this here because many of the > HA replication users are devs> > too> > >> and given that this isn't yet > implemented: we're stull in the> > >> design/proposal phase, so anyone that > want to express their> > >> ideas/opinions/POC on this, is invited to talk > here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> > --> > >> Sent from:> > >> > http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- > Clebert Suconic-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic-- > Clebert Suconic -- Clebert Suconic
