Hi Irindu, I think that for this new development we should try to use the latest stable version of MongoDB with its own related drivers and libraries. We only have to check if some dependencies can be in conflict with other libraries currently deployed in ManifoldCF.
Let's try to use the latest version and if it works we will also create an issue for updating the existing MongoDB/GridFS Repository Connector. Finally could you please share all your improvements and questions inside our dev list or in the related issue on JIRA? In this way you could receive update not only from me but also from any other member in the community. Your effort is related to the community so don't be shy to write directly in the dev list :) I would like to create a dedicated branch for your development that starts from the latest release of ManifoldCF, so this means that after the 2.10 release I'll create the branch for you. Cheers, PJ 2018-04-28 4:12 GMT+02:00 Irindu Nugawela <[email protected]>: > Hi Piergiorgio, > I have noticed that you are using mongo-java-driver > > <dependency> > <groupId>org.mongodb</groupId> > <artifactId>mongo-java-driver</artifactId> > <version>2.11.3</version> > </dependency> > > for Gridfs connector but the latest stable is > > <dependency> > <groupId>org.mongodb</groupId> > <artifactId>mongo-java-driver</artifactId> > <version>3.6.3</version> > </dependency> > > there are some API changes between the two which one do you want me to > use. At the moment I cannot use the version 3.6.3 inside the manifoldCF > framework even though I add it as a dependency for my project I encounter > some problems with certain API calls which I suspect because version 2.11.3 > jar is already there. Please guide me with next steps, shall I stick to > version 2.11.3? > > > > On 26 April 2018 at 16:27, Piergiorgio Lucidi <[email protected]> > wrote: > >> Just shared on Twitter: >> https://twitter.com/ApacheManifold/status/989457998715277312 >> >> ^__^ >> >> 2018-04-26 11:39 GMT+02:00 Irindu Nugawela <[email protected]>: >> >>> Hi Piergiorgio, >>> Thank you very much. yes, you may publish about my involvement in our >>> project and please let me know if I can help you in any way possible. Yes, >>> the twitter profile you have listed belongs to me. >>> >>> On 26 April 2018 at 12:23, Piergiorgio Lucidi <[email protected]> >>> wrote: >>> >>>> Hi Irindu, >>>> >>>> no problem, take your time. >>>> >>>> In the meanwhile I'm wondering if you allow me to publish a post about >>>> your involvement in the project on our Twitter profile: >>>> https://twitter.com/apachemanifold >>>> >>>> I noticed that you have a Twitter profile, could you please to confirm >>>> that your profile is the following? >>>> https://twitter.com/IrinduPera >>>> >>>> Please let me know and if you give me your permission I'll write a post. >>>> Thank you and take care. >>>> >>>> Cheers, >>>> PJ >>>> >>>> 2018-04-26 3:33 GMT+02:00 Irindu Nugawela <[email protected]>: >>>> >>>>> Hi Piergiorgio, >>>>> >>>>> Thank you very much for giving me this valuable opportunity, I am >>>>> going to do my best with it. Secondly, I am very sorry that I was not very >>>>> active in the last few days that I have been suffering from a viral fever. >>>>> I am recovering now and will work extra hours in the upcoming weeks to get >>>>> back on track. Please help me with the next steps :-) >>>>> >>>>> On 16 April 2018 at 13:52, Piergiorgio Lucidi <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Irindu, >>>>>> >>>>>> very nice and great job! >>>>>> >>>>>> ^__^ >>>>>> >>>>>> Cheers, >>>>>> PJ >>>>>> >>>>>> >>>>>> 2018-04-16 8:00 GMT+02:00 Irindu Nugawela <[email protected]>: >>>>>> >>>>>>> Hi Piergiorgio , >>>>>>> I have come up with the implementation for edit configuration and >>>>>>> view configuration with the crawler-ui. It's a small step in the long >>>>>>> journey. >>>>>>> Apache ManifoldCF_ View Output Connection Statu... >>>>>>> <https://drive.google.com/file/d/1X3q32ml40yzJ1moTjoXew2K1g5wNk0Gs/view?usp=drive_web> >>>>>>> >>>>>>> >>>>>>> >>>>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> >>>>>>> Virus-free. >>>>>>> www.avast.com >>>>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> >>>>>>> <#m_8005759486091203682_m_-3875022077604017546_m_2871376415022697920_m_-1171787217413983024_m_1069976683875003658_m_645939314292940779_m_2231955365225803757_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>>> >>>>>>> On 15 March 2018 at 00:45, Irindu Nugawela <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> I was able to successfully build ManifoldCF with IntelliJ Idea as >>>>>>>> well. >>>>>>>> >>>>>>>> I have gone through MongoDB - Java >>>>>>>> <https://www.tutorialspoint.com/mongodb/mongodb_java.htm> and >>>>>>>> identified the MongoDB parameters as follows >>>>>>>> >>>>>>>> >>>>>>>> 1. MongoCredential by a useraname and a password >>>>>>>> 2. MongoDatabase >>>>>>>> 3. collection >>>>>>>> >>>>>>>> I am planning to have a single username and a password for a >>>>>>>> Manifold-MongoDB connection >>>>>>>> >>>>>>>> I have identified the operations that we are going to support as >>>>>>>> >>>>>>>> 1. Inserting documents >>>>>>>> 2. Querying documents >>>>>>>> 3. Updating documents >>>>>>>> 4. Deleting documents >>>>>>>> 5. Indexing documents based on a particular field >>>>>>>> >>>>>>>> I have also studied the document format we are going to store >>>>>>>> available here >>>>>>>> <https://github.com/apache/manifoldcf/blob/trunk/framework/agents/src/main/java/org/apache/manifoldcf/agents/interfaces/RepositoryDocument.java> >>>>>>>> which >>>>>>>> MongoDB accepts in BSON format, >>>>>>>> >>>>>>>> I have identified the following fields >>>>>>>> >>>>>>>> 1. binaryFieldData >>>>>>>> 2. binaryLength >>>>>>>> 3. fileName >>>>>>>> 4. contentMimeType >>>>>>>> 5. createdDate >>>>>>>> 6. modifiedDate >>>>>>>> 7. indexingDate >>>>>>>> 8. originalSize >>>>>>>> >>>>>>>> of a Repository Document that I am going to convert to BSON format >>>>>>>> to write to MongoDB >>>>>>>> >>>>>>>> Please tell me am I on the correct path, and correct me if I am >>>>>>>> wrong >>>>>>>> >>>>>>>> On 27 February 2018 at 15:42, Piergiorgio Lucidi < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Irindu, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2018-02-25 13:53 GMT+01:00 Irindu Nugawela <[email protected]>: >>>>>>>>> >>>>>>>>> > Hi All, >>>>>>>>> > I am currently to working on the $subject. I have some issues >>>>>>>>> that I need >>>>>>>>> > to clarify. >>>>>>>>> > First of all, why did we choose Mongo DB as an output target? My >>>>>>>>> first >>>>>>>>> > thought of the project was that it was about writing a >>>>>>>>> repository connector >>>>>>>>> > for MongoDB because MongoDB is a database programme. ( I am >>>>>>>>> aware of its >>>>>>>>> > NoSQL >>>>>>>>> > architecture and its indexing capabilities ). But it is not a >>>>>>>>> search engine >>>>>>>>> > per se. >>>>>>>>> > >>>>>>>>> >>>>>>>>> We want to extend the scenario of ManifoldCF and we would like to >>>>>>>>> add >>>>>>>>> content migration capability. >>>>>>>>> Imagine to have a tool that helps you not only for the indexing >>>>>>>>> process but >>>>>>>>> also in migrating contents. >>>>>>>>> >>>>>>>>> In this way you could migrate contents from a content repository >>>>>>>>> to another >>>>>>>>> one, for example you could migrate contents from Alfresco to >>>>>>>>> MongoDB. >>>>>>>>> >>>>>>>>> >>>>>>>>> > >>>>>>>>> > Then what are the expected operations other than document >>>>>>>>> addition and >>>>>>>>> > deletion? (what activities should be supported?) >>>>>>>>> > >>>>>>>>> >>>>>>>>> The excepted operation are the same but related to migrate >>>>>>>>> contents and not >>>>>>>>> related only to manage the indexing process. >>>>>>>>> >>>>>>>>> >>>>>>>>> > >>>>>>>>> > What capabilities of MongoDB you have already identified that >>>>>>>>> you expect to >>>>>>>>> > be useful for us. (what capabilities of MongoDB you have >>>>>>>>> identified as >>>>>>>>> > useful That you've decided to write an Output connector for it.) >>>>>>>>> > >>>>>>>>> >>>>>>>>> Currently we only have a repository connector for GridFS that >>>>>>>>> allow us to >>>>>>>>> read binaries from MongoDB. >>>>>>>>> We also want to store contents against a MongoDB instance, in this >>>>>>>>> way we >>>>>>>>> can migrate contents on MongoDB from any repository with metadata >>>>>>>>> and >>>>>>>>> binaries. >>>>>>>>> >>>>>>>>> I think it could be very cool :) >>>>>>>>> >>>>>>>>> In the enterprise context is very critical to migrate contents / >>>>>>>>> documents >>>>>>>>> from different kind of repositories and typically it is the first >>>>>>>>> step for >>>>>>>>> a digital transformation project where a company is trying to >>>>>>>>> orchestrate >>>>>>>>> different silos in a smart way. >>>>>>>>> >>>>>>>>> Please let me know if you need more informations. >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> PJ >>>>>>>>> >>>>>>>>> >>>>>>>>> > >>>>>>>>> > I have been through the ManifoldCF Architecture with DaddyWri >>>>>>>>> > <https://github.com/DaddyWri>/manifoldcfinaction Chapter1. >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > Thanks and Regards, >>>>>>>>> > Irindu Nugawela, >>>>>>>>> > Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate, >>>>>>>>> > Faculty of Engineering University of Peradeniya >>>>>>>>> > >>>>>>>>> > <https://www.avast.com/sig-email?utm_medium=email&utm_ >>>>>>>>> > source=link&utm_campaign=sig-email&utm_content=webmail&utm_t >>>>>>>>> erm=icon> >>>>>>>>> > Virus-free. >>>>>>>>> > www.avast.com >>>>>>>>> > <https://www.avast.com/sig-email?utm_medium=email&utm_ >>>>>>>>> > source=link&utm_campaign=sig-email&utm_content=webmail&utm_t >>>>>>>>> erm=link> >>>>>>>>> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Piergiorgio Lucidi >>>>>>>>> https://www.open4dev.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks and Regards, >>>>>>>> Irindu Nugawela, >>>>>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate, >>>>>>>> Faculty of Engineering University of Peradeniya >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks and Regards, >>>>>>> Irindu Nugawela, >>>>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate, >>>>>>> Faculty of Engineering University of Peradeniya >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Piergiorgio Lucidi >>>>>> https://www.open4dev.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> Irindu Nugawela, >>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate, >>>>> Faculty of Engineering University of Peradeniya >>>>> >>>> >>>> >>>> >>>> -- >>>> Piergiorgio Lucidi >>>> https://www.open4dev.com >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> Irindu Nugawela, >>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate, >>> Faculty of Engineering University of Peradeniya >>> >> >> >> >> -- >> Piergiorgio Lucidi >> https://www.open4dev.com >> > > > > -- > Thanks and Regards, > Irindu Nugawela, > Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate, > Faculty of Engineering University of Peradeniya > -- Piergiorgio Lucidi https://www.open4dev.com
