Sumanth, Uli, I think from a framework perspective NiFi is well suited for many ETL use cases. Where I think we fall short today is in the existing set of processors/extensions to help you with it. I write this knowing full well ETL means a lot of different things to a lot of different people. My answer is in the traditional DB to DB type ETL mangling cases (not the broader sense it is sometimes used in). This is coming up quite often though so I expect we'll end up seeing development within the community here pretty soon. Perhaps there are good open source options we can integrate with. Will look more at the list you link to.
Thanks Joe On Wed, Oct 14, 2015 at 4:58 PM, Uli Bethke <[email protected]> wrote: > I would also be interested in the question below. > Anyone? > > > On 14/10/2015 05:17, [email protected] wrote: >> >> I am researching to implement Change Data Capture(CDC) solution by >> observing MySQL binlogs and routing that data through NiFi to detect inserts >> , updates and deletes and finally storing/sync the data into NoSQL(MongoDB) >> and HDFS. >> For this use case the data in source system is normalized (OLTP) and sink >> system is denormalized (OLAP), so I have to accumulate some data in NiFi to >> compose aggregated record and store it into MongoDB. >> >> I would like to know if any NiFi users building CDC solution using NiFi? >> Is NiFi a good choice for this kind of dataflow / real-time ETL? >> >> I found some Java based CDC frameworks here: >> https://github.com/wushujames/mysql-cdc-projects/wiki >> >> Thanks >> Sumanth >> >> > >
