Thanks Bryan. De : Bryan Bende [mailto:[email protected]] Envoyé : jeudi 10 novembre 2016 15:26 À : [email protected] Objet : Re: Nifi vs Sqoop
Hello, I can't speak to a direct comparison between NiFi and sqoop, but I can say that sqoop is a specific tool that was built just for database extraction, so it can probably do some things NiFi can't, since NiFi is a general purpose data flow tool. That being said, NiFi does have the ability to extraction from relation databases... The GenerateTableFetch processor [1] would likely be what you want for more of a bulk-extraction, and QueryDatabaseTable [2] for incremental fetching I believe the "Maximum Value Columns" property on QueryDatabaseTable is how you achieve finding new rows since last execution. Thanks, Bryan [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GenerateTableFetch/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html On Wed, Nov 9, 2016 at 4:37 AM, Provenzano Nicolas <[email protected]<mailto:[email protected]>> wrote: Hi all, I have the following requirements : • I need to load at day 1 a full SQL table, • And then need to incrementally load new data (using capture data change mechanism). Initially, I was thinking using Sqoop to do it. Looking at Nifi and especially the QueryDatabaseTable processor, I’m wondering if I could use Nifi instead. Has someone already compared both to do it and what were the outcomes ? I can’t see however how to configure the QueryDatabaseTable to handle the new lines (for example, looking at a “lastmodificationdate” field and taking only the lines for which lastModificationDate > lastRequestDate) ? Thanks in advance BR Nicolas
