Hi Piergiorgio, thanks for your response. I also agree Kafka is best for this solution.
In fact, we are facing Elastic cluster problems and developers told me it might be possible to persist / distribute messages through kafka instead of directly dealing with Elastic. Also, someone pointed to a logstash solution as ELK does, but you are right, Kafka should be the way. As I understand, logstash should "replace" or "complement" web crawler by just putting "events" into the system that might be transformed into documents in parallel with other repositories. I like a lot a passive connector, some kind of listener that could get individual document requests and incorporate them to the system, not necessary to be tied with Logstash. Best regards! El vie., 7 sept. 2018 a las 14:25, Piergiorgio Lucidi (< [email protected]>) escribió: > Hi Gustavo, > > nice to meet you! > > We actually have a Kafka Repository Connector as mentioned in our latest > compatibility matrix: > > https://manifoldcf.apache.org/release/release-2.10/en_US/included-connectors.html > > Considering Logstash we don't have nothing at the moment but I can imagine > the scenario behind it. > The architecture for this case is quite different because ManifoldCF is > fundamentally a repository crawler and the interaction with logstash should > be inline from logstash to ManifoldCF that is not actually supported for > each single transaction. > > We could think about a passive connector that remains listening to receive > new contents, anyway we should start talking about which could the best > approach for implementing this new behavior. > > Am I missing something? > Hope to receive more comments from other folks in the community :) > > Cheers, > PJ > > > > > Il giorno mar 4 set 2018 alle ore 09:50 Gustavo Beneitez < > [email protected]> ha scritto: > > > Hello Karl, > > > > is there any kind of integration in the roadmap between manifoldCF and > > Kafka or LogStash technologies? > > > > We are trying to expand replications with several technologies in mind. > > > > Thanks! > > > > > -- > Piergiorgio Lucidi > Open Source Evangelist and Digital Transformation Specialist > Member / Mentor / PMC Member / Committer @ The Apache Software Foundation > Community Star / Wiki Gardener / Global Forum Moderator @ Alfresco > Author and Technical Reviewer @ Packt Publishing > Technical Advisory Group Member @ Microsoft > Top Community Contributor @ Crafter > Project Leader / Committer @ JBoss > https://www.open4dev.com >
