Thank you, that was a great explanation about the project. I'll start working on it and get a proposal ready for review soon. Is there any issues that I can take up to contribute to the OODT project for now? I feel I'll get to know the codebase properly if I contribute some more patches soon.
Thanks, R B Krishna On Sun, Mar 12, 2017 at 7:44 PM, Tom Barber <[email protected]> wrote: > Cool. > > We'll the Kafka idea is pretty simple, although I need to plan the > implementation locations properly. So OODT is made up of a bunch of > components, the most common ones being the Curator, the File Manager and > the Workflow Manager. > > Currently tracking data through OODT isn't as simple as it could or perhaps > should be, so the idea behind OODT-946 is to implement Kafka as a pub sub > system for OODT. For example: > > The curator allows users to ingest data into the OODT platform via the file > manager, so one implementation would be to replace the current > communication between file manager and curator with a Kafka message bus. So > user X curates some data and when they ingest it, instead of calling the > file manager directly the curation message is pushed onto the Kafka message > bus and a number of services could be subscribed to this. They may be the > file manager which would pick it up and ingest the data, but it might also > be an Audit component that tracks the flow of data through OODT, it might > also be the workflow manager etc. > > Similarly when the file manager ingests data, instead of it simply calling > the backend factory for Solr, Lucene or whatever to store the ingested > data, it would be great if it again, pushed the message to Kafka, which in > turn allowed different file manager backends to subscribe to the queue, > perhaps to pick up specific ingestion file types or whatever. But also that > push got optionally audited but the audit component also subscribed to the > mesage queue, so you know when what file got ingested where, along with the > entry into the catalog. > > That type of thing. > > On Sun, Mar 12, 2017 at 1:37 PM, krishna rb <[email protected]> wrote: > > > Yes sir, I am currently looking for issues so that I can understand the > > project and contribute to it. > > Also, I am interested in participating in Gsoc 2017 and I found the > > project: Implement Kafka Messaging System Within OODT > > <https://issues.apache.org/jira/browse/OODT-946> very interesting. > > I am currently learning Apache Kafka. > > It will be immensely helpful if you could explain about the idea. > > > > Thanks, > > R B Krishna > > > > On Mar 12, 2017 6:56 PM, "Tom Barber" <[email protected]> wrote: > > > > > Hi R B, > > > > > > You're a Java developer I assume? > > > > > > Thanks > > > > > > Tom > > > > > > On Sat, Mar 11, 2017 at 10:02 AM, krishna rb <[email protected]> > > wrote: > > > > > > > Hello everyone, > > > > This is R B Krishna, studying 3rd year Computer Science at National > > > > Institute of Technology, Trichy, India. > > > > I am interested in contributing to the project. > > > > > > > > Could someone suggest me some issues that I could take up as a newbie > > and > > > > guide me through it? > > > > > > > > I am currently going through the codebase and trying to get > comfortable > > > > with it. > > > > > > > > Thanks, > > > > R B Krishna. > > > > > > > > > >
