Hi Clebert Seems quite some work bud! Nice...
As This is quite a big feature. To help us understand and go through bits can we organise a call for those interested. It would significantly help me for sure to then be able to review and give feedback, similar to what we did with federation. Best Mike Sent from my iPad > On 9 Oct 2020, at 15:49, Clebert Suconic <[email protected]> wrote: > > I can definitely rename the interface. > > I did not want the interface exposed to users outside of the context > of the mirroring I'm doing here. for other usages users can get what > they want from broker plugins. > > > I will rename this as Mirror > > > and then I can have a MirrorSource and MirrorTarget > > > WDYT? > >> On Thu, Oct 8, 2020 at 8:21 AM Gary Tully <[email protected]> wrote: >> >> Hi Clebert, this is a great piece of work. >> >> One thing popped out for me from the PR, the remoteControl interface. >> >> I know naming is hard :-). That name to me means something is controlling >> from afar. But I think it is really a set of change events that a broker >> can emit. If I understand correctly, a ``mirror`` AMQP outbound connection >> is turning those events/commands into amqp messages on an address that a >> receiver is using to replicate a selective state. >> Another listener could use it for generating notifications or audits? >> >> Should it be called ChangeEvents? or ChangeActions - because there are >> operations for each event. >> >> further on that, if a mirror is making a broker replica, I could imagine a >> change event for a new security role or for a role deletion event. Does >> this sort of thing fit in with your idea of a RemoteControl? >> >> I guess I am getting ahead a little, as in all events that change a broker >> are potentially relevant to a mirror, and if some extra plugins are in >> play, maybe some extra events need to get replicated. >> >> In other words, this is a concrete specific implementation of a more >> general concept, change data capture. To make it extensible would require >> being able to publish events and on the receiver end, find a handler or >> ignore events that are unrecognised. It would need some conventions etc. >> >> In any event, something like that is now possible :-) >> >> /gary >> >> On Wed, 7 Oct 2020 at 23:14, Clebert Suconic <[email protected]> >> wrote: >> >>> sorry for the SPAM: I meant to add the link: >>> >>> https://github.com/apache/activemq-artemis/pull/3294 >>> >>> On Wed, Oct 7, 2020 at 4:17 PM Clebert Suconic >>> <[email protected]> wrote: >>>> >>>> I have posted a Pull Request.. .(for review only at this time) >>>> >>>> I would appreciate reviews. >>>> >>>> I'm fixing some issues, adding docs.. but the overall is ready for >>>> code review already. >>>> >>>> Thank you >>>> >>>> On Thu, Oct 1, 2020 at 9:55 AM Clebert Suconic >>>> <[email protected]> wrote: >>>>> >>>>> I am working on new features in the AMQP Protocol Handler, and I would >>>>> like to explain here what I am doing: >>>>> >>>>> >>>>> I needed to go beyond what we do now with AMQP, and use it connect >>>>> brokers, and eventually with qpid-dispatch for more elaborate >>>>> networking on datacenters. >>>>> >>>>> First part: >>>>> >>>>> The first issue I had while implementing something protocol specific >>>>> (other than Core) was to use the classes within the protocol package. >>>>> As the broker is agnostic to the protocols (exception is core ATM), I >>>>> needed the whole initialization to take place within >>>>> artemis-amqp-protocol. >>>>> >>>>> To get around that I added what I called ProtocolServices. So when you >>>>> implement a protocol service the interface will be called within >>>>> broker start. >>>>> >>>>> >>>>> Second part: >>>>> >>>>> As the broker is now taking action on connecting to other brokers (or >>>>> other AMQP servers), I needed to change out Netty bootstrapping to >>>>> allow outgoing connections with a given protocol. With that I'm >>>>> inverting everything we had for server, and I am abusing the power of >>>>> AMQP here. Since AMQP is totally symmetrical I can now connect brokers >>>>> directly. So, a Consumer on the broker and be connected directly to a >>>>> producer on another broker. Acting like a bridge, but way more >>>>> lightweight. >>>>> >>>>> >>>>> Third part: >>>>> >>>>> I am using that for replicas. At the moment these replicas will not >>>>> sync the client, but they are working quite nicely. and allowing >>>>> multiple replicas. >>>>> >>>>> >>>>> >>>>> The configuration will take part as the following. >>>>> >>>>> You define amqp connections with their URL and their reconnecting >>>>> information, for each connection you can define: >>>>> >>>>> sender <matching addresses>: All addresses matching will have a >>>>> transfer sender. This will act like a push bridge >>>>> receiver < matching addresses>: All matching addresses on this broker >>>>> will create a consumer pulling messages. This will act like a pull >>>>> bridge >>>>> peer <matching addresses>: This will create both ways, but you can >>>>> only use this towards a special server that knows how to handle this. >>>>> (e.g. qpid-dispatch). otherwise you get ping pongs on transfers >>>>> copy <matching addresses> : This will create a replica, without >>>>> sending the acks, (You will be responsible to consume the messages >>>>> yourself at the target broker) >>>>> replica <matching addresses>: Just like copy, but acks are sent and >>>>> messages removed upong ack (or accept) >>>>> >>>>> >>>>> This is an example of the configuration: >>>>> >>>>> <amqp-connections> >>>>> <!-- the connection towards another broker --> >>>>> <amqp-connection uri="tcp://otherNode:5671" name="myconnection" >>>>> retry-interval="333" reconnect-attempts="33"> >>>>> <sender match="TEST-SENDER" /> >>>>> <receiver match="TEST-RECEIVER" /> >>>>> <peer match="TEST-PEER"/> >>>>> <copy match="TEST-COPY"/> >>>>> <replica match="TEST-REPLICA"/> >>>>> </amqp-connection> >>>>> </amqp-connections> >>>>> >>>>> >>>>> so far these are being worked on a my github fork/ but it's not really >>>>> meant for a review at this point as I have some cleanup to do (some >>>>> logging messages and stuff I have to remove). >>>>> >>>>> I am getting ready to send a Pull Request early next week. but you >>>>> have the concepts here already to review them. >>>>> >>>>> >>>>> -- >>>>> Clebert Suconic >>>> >>>> >>>> >>>> -- >>>> Clebert Suconic >>> >>> >>> >>> -- >>> Clebert Suconic >>> > > > > -- > Clebert Suconic
