Looking at the Kafka Connect example. I see that the integration components are all maintained outside of the Kafka project. They're listed here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
How about we do something similar and create a wiki/web page linking the integration projects. I realise our website isn't the most accessible atm but going forward this should be improved. On Thu, Oct 26, 2017 at 5:42 PM, Robbie Gemmell <[email protected]> wrote: > The related PR is https://github.com/apache/activemq-artemis/pull/1607 > for anyone not following along thus far, or later reference. > > My main view there was that such an extension would be better > maintained seperate from the main broker repository+distribution for a > variety of reasons. After that I didn't have a strong view on where it > lives, but I would say that I don't actually see any issue at all with > it being outwith the ActiveMQ project. To me thats just a key thing > enabled by such plugin points existing, and not uncommon with other > projects/components. > > Robbie > > On 26 October 2017 at 16:46, Michael André Pearce > <[email protected]> wrote: > > As noted on the PR part of the PR discussion is should Service > Connectors exist with the broker code. > > > > Service Connectors / Extensions are ActiveMQ Artemis specific. > > > > I think having them within the project space is good so we can grow an > eco system of some, people can just use will aid in having better broker > adoption as tools people look for would be available if contributed. > > > > As stated above it seems the general consensus on the PR is though liked > is the broker code base best for them to live? > > > > If we don’t wish them to exist with broker code so they can have their > own release cycle, and don’t murky the water of the broker, I would like to > discuss if we could make a sub project for these to live, and possibly > propose a sub project be setup called “ActiveMQ Artemis Integrations” ? > > > > If setup I would also propose maybe moving the spring integration there > also. > > > > Cheers > > Mike > > > > > > > > > > > > > > > > Sent from my iPhone > > > >> On 6 Oct 2017, at 12:20, Michael André Pearce < > [email protected]> wrote: > >> > >> Yes and no, it would be same as arguing that core bridge between > artemis clusters could also be separated components. > >> > >> The idea is to effectively have similar to a core bridge but to kafka. > >> > >> Some key parts why are (and probably similar to why core bridge is > implemented within the broker) > >> > >> Latency > >> Extra processes and hops > >> Out the box in broker support > >> > >> Cheers > >> Mike > >> > >> > >> Sent from my iPhone > >> > >>> On 6 Oct 2017, at 12:14, Andy Taylor <[email protected]> wrote: > >>> > >>> I think https://github.com/ppatierno/kafka-connect-amqp is an example > of > >>> this > >>> > >>> On 6 October 2017 at 12:07, Christopher Shannon < > >>> [email protected]> wrote: > >>> > >>>> Why not use Kafka's Connect framework? Seems like building a source > >>>> connector here would make the most sense. This is not to say I'm > against > >>>> the idea I'm just wondering the benefits of doing it from the Artemis > side > >>>> of things when Kafka already has a framework that can be used to > >>>> import/export data. > >>>> > >>>> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce < > >>>> [email protected]> wrote: > >>>> > >>>>> Hi Andy, > >>>>> > >>>>> As you may be aware or not we run both activemq and kafka each has a > >>>>> specific plus point to it, use the right tool for the right job. > >>>>> > >>>>> We use activemq as our feature rich, fully jms compliant and > performant > >>>>> broker. > >>>>> > >>>>> And use kafka more for post processing and data streaming, with > >>>> historical > >>>>> feed also. > >>>>> > >>>>> Idea here is that producers and consumers bind into our activemq > where > >>>> jms > >>>>> api is wanted or needed, lower per message latencies etc. but then > we can > >>>>> bridge messages from a core queue into kafka so that the data can > flow to > >>>>> it as fast as possible and consumed from there also. > >>>>> > >>>>> Idea is to try make a hybrid taking best of both. > >>>>> > >>>>> Cheers > >>>>> Mike > >>>>> > >>>>> Sent from my iPhone > >>>>> > >>>>>> On 6 Oct 2017, at 09:37, Andy Taylor <[email protected]> > wrote: > >>>>>> > >>>>>> Whats the use case here Michael, sounds like a cool idea and > something > >>>>> that > >>>>>> has been mentioned by different people but I am yet to understand > how > >>>>> this > >>>>>> could be used. > >>>>>> > >>>>>> On 6 October 2017 at 07:48, Michael André Pearce < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>>> Hi All, > >>>>>>> > >>>>>>> I am looking to contribute an artemis ServiceConnector that would > >>>>> leverage > >>>>>>> the existing service connector interface to bridge from a queue > into a > >>>>>>> kafka topic similar to bridge between artemis clusters. > >>>>>>> > >>>>>>> Idea is to make it configurable how to transform and encode the > >>>> message > >>>>> in > >>>>>>> kafka but as amqp is becoming very well supported by artemis > provide > >>>> out > >>>>>>> the box that the message be encoded in amqp byte protocol. > >>>>>>> > >>>>>>> It something I’ve been working on privately for a bit, but since > >>>>> becoming > >>>>>>> a committer here is really like to share it. > >>>>>>> > >>>>>>> I hope if I get a positive response here to get a branch ready > soon, > >>>> but > >>>>>>> feedback if this is welcomed would be great. > >>>>>>> > >>>>>>> Cheers > >>>>>>> Mike > >>>>>>> > >>>>>>> > >>>>>>> Sent from my iPhone > >>>>> > >>>> >
