Cool. We'll the Kafka idea is pretty simple, although I need to plan the implementation locations properly. So OODT is made up of a bunch of components, the most common ones being the Curator, the File Manager and the Workflow Manager.
Currently tracking data through OODT isn't as simple as it could or perhaps should be, so the idea behind OODT-946 is to implement Kafka as a pub sub system for OODT. For example: The curator allows users to ingest data into the OODT platform via the file manager, so one implementation would be to replace the current communication between file manager and curator with a Kafka message bus. So user X curates some data and when they ingest it, instead of calling the file manager directly the curation message is pushed onto the Kafka message bus and a number of services could be subscribed to this. They may be the file manager which would pick it up and ingest the data, but it might also be an Audit component that tracks the flow of data through OODT, it might also be the workflow manager etc. Similarly when the file manager ingests data, instead of it simply calling the backend factory for Solr, Lucene or whatever to store the ingested data, it would be great if it again, pushed the message to Kafka, which in turn allowed different file manager backends to subscribe to the queue, perhaps to pick up specific ingestion file types or whatever. But also that push got optionally audited but the audit component also subscribed to the mesage queue, so you know when what file got ingested where, along with the entry into the catalog. That type of thing. On Sun, Mar 12, 2017 at 1:37 PM, krishna rb <[email protected]> wrote: > Yes sir, I am currently looking for issues so that I can understand the > project and contribute to it. > Also, I am interested in participating in Gsoc 2017 and I found the > project: Implement Kafka Messaging System Within OODT > <https://issues.apache.org/jira/browse/OODT-946> very interesting. > I am currently learning Apache Kafka. > It will be immensely helpful if you could explain about the idea. > > Thanks, > R B Krishna > > On Mar 12, 2017 6:56 PM, "Tom Barber" <[email protected]> wrote: > > > Hi R B, > > > > You're a Java developer I assume? > > > > Thanks > > > > Tom > > > > On Sat, Mar 11, 2017 at 10:02 AM, krishna rb <[email protected]> > wrote: > > > > > Hello everyone, > > > This is R B Krishna, studying 3rd year Computer Science at National > > > Institute of Technology, Trichy, India. > > > I am interested in contributing to the project. > > > > > > Could someone suggest me some issues that I could take up as a newbie > and > > > guide me through it? > > > > > > I am currently going through the codebase and trying to get comfortable > > > with it. > > > > > > Thanks, > > > R B Krishna. > > > > > >
