Process Creation
To whom it may concern, I am trying to create a process that will export data from a MySQL database using a select query. This data will in turn be used to either update or insert records of another MySQL database. At the moment my process in NiFi includes the flowing Process flow: ExecuteSQL - running the select query ConvertAvroToJSON - to clean up the output file which according to the document can be used to create a SQL update or insert query PutFile - used for testing to view output as I'm new to NiFi The other process is: GetFile - obtains the output of the first process ConvertJSONToSQL - using this to take the JSON and make the SQL Insert query PutSQL - Should take the query and run it entering the data into the DB The problem I'm having is once the JSON file hits the ConvertJSONToSQL an error is return about the other table's ID field not being found. This ID filed is auto incrementing and doesn't (or shouldn't in my mind) need to be provided. Could you provide any assistance and/or design advice to get this process up and running correctly would be great. Thanks, Josh
Re: Process Creation
Josh, Happy to help, could you provide some more details around the use case? For example the JSON content (scrubbed if need be of course) and a stack trace or other logging/bulletin text about the error? Regarding PutFile to view output, an alternative is to use a LogAttribute processor with the "Log Payload" property set to true. Then logs/nifi-app.log will contain the attributes and content of each flow file that it is passed. When I start NiFi I usually go right to tailing that log: bin/nifi.sh start && tail -f logs/nifi-app.log Regards, Matt P.S. I see you have a Raytheon email, I was at IDS for 10 years in New England and 2 years at RTSC in Orlando :) On Wed, Mar 9, 2016 at 1:20 PM, Joshua D Rinker < joshua.d.rin...@raytheon.com> wrote: > To whom it may concern, > > I am trying to create a process that will export data from a MySQL > database using a select query. This data will in turn be used to either > update or insert records of another MySQL database. At the moment my > process in NiFi includes the flowing Process flow: > > ExecuteSQL - running the select query > ConvertAvroToJSON - to clean up the output file which according to the > document can be used to create a SQL update or insert query > PutFile - used for testing to view output as I'm new to NiFi > > The other process is: > > GetFile - obtains the output of the first process > ConvertJSONToSQL - using this to take the JSON and make the SQL Insert > query > PutSQL - Should take the query and run it entering the data into the DB > > The problem I'm having is once the JSON file hits the ConvertJSONToSQL an > error is return about the other table's ID field not being found. This ID > filed is auto incrementing and doesn't (or shouldn't in my mind) need to be > provided. > > Could you provide any assistance and/or design advice to get this process > up and running correctly would be great. > > Thanks, > > Josh
Process Creation
To whom it may concern, I am trying to create a process that will export data from a MySQL database using a select query. This data will in turn be used to either update or insert records of another MySQL database. At the moment my process in NiFi includes the flowing Process flow: ExecuteSQL - running the select query ConvertAvroToJSON - to clean up the output file which according to the document can be used to create a SQL update or insert query PutFile - used for testing to view output as I'm new to NiFi The other process is: GetFile - obtains the output of the first process ConvertJSONToSQL - using this to take the JSON and make the SQL Insert query PutSQL - Should take the query and run it entering the data into the DB The problem I'm having is once the JSON file hits the ConvertJSONToSQL an error is return about the other table's ID field not being found. This ID filed is auto incrementing and doesn't (or shouldn't in my mind) need to be provided. Could you provide any assistance and/or design advice to get this process up and running correctly would be great. Thanks, Josh
