Hello, I can't speak to a direct comparison between NiFi and sqoop, but I can say that sqoop is a specific tool that was built just for database extraction, so it can probably do some things NiFi can't, since NiFi is a general purpose data flow tool.
That being said, NiFi does have the ability to extraction from relation databases... The GenerateTableFetch processor [1] would likely be what you want for more of a bulk-extraction, and QueryDatabaseTable [2] for incremental fetching I believe the "Maximum Value Columns" property on QueryDatabaseTable is how you achieve finding new rows since last execution. Thanks, Bryan [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GenerateTableFetch/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html On Wed, Nov 9, 2016 at 4:37 AM, Provenzano Nicolas < [email protected]> wrote: > Hi all, > > > > I have the following requirements : > > > > · I need to load at day 1 a full SQL table, > > · And then need to incrementally load new data (using capture > data change mechanism). > > > > Initially, I was thinking using Sqoop to do it. > > > > Looking at Nifi and especially the QueryDatabaseTable processor, I’m > wondering if I could use Nifi instead. > > > > Has someone already compared both to do it and what were the outcomes ? > > > > I can’t see however how to configure the QueryDatabaseTable to handle the > new lines (for example, looking at a “lastmodificationdate” field and > taking only the lines for which lastModificationDate > lastRequestDate) ? > > > > Thanks in advance > > > > BR > > > > Nicolas >
