Re: Data Ingestion using NiFi

2019-08-13 Thread Mike Thomsen
One of the easiest ways to trigger events in NiFi is to have a message queue processor set up and listening to a queue where you post an event to trigger the flow. On Tue, Aug 13, 2019 at 11:45 AM Bimal Mehta wrote: > Thanks Mike. > ExecuteSQL looks good and am trying it. > > Also I wanted to

Re: Data Ingestion using NiFi

2019-08-13 Thread Bimal Mehta
Thanks Mike. ExecuteSQL looks good and am trying it. Also I wanted to understand how can we control triggering the NiFi jobs from devops tools like CloudBees/ElectricFlow? On Tue, Aug 13, 2019 at 7:35 AM Mike Thomsen wrote: > Bimal, > > 1. Take a look at ExecuteSQLRecord and see if that works

Re: Data Ingestion using NiFi

2019-08-13 Thread Mike Thomsen
Bimal, 1. Take a look at ExecuteSQLRecord and see if that works for you. I don't use SQL databases that much, but it works like a charm for me and others for querying and getting an inferred avro schema based on the schema of the database table (you can massage it into another format with

Data Ingestion using NiFi

2019-08-12 Thread Bimal Mehta
Hi NiFi users, We had been using the kylo data ingest template to read the data from our Oracle and DB2 databases and move it into HDFS and Hive. The kylo data ingest template also provided some features to validate, profile and split the data based on validation rules. We also built some custom