Hi James, Thanks for confirming the current feature isn’t there.
I have raised https://issues.apache.org/jira/browse/NIFI-3967 as suggested to capture this use case and converstation. Cheers Mike From: James Wing <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, 24 May 2017 at 02:39 To: "[email protected]" <[email protected]> Subject: Re: JDBC Schema Evolution Mike, I believe you are correct that NiFi does not have built-in support for several of those tasks: * Identifying a schema evolution (separate from a non-conforming flowfile) * Identifying safe schema changes * Automatically applying schema changes to a SQL database However, the new schema registry and record handling components introduced in NiFi 1.2.0 might be useful to build some of these features. I would certainly encourage you to create a JIRA ticket to capture the use case at https://issues.apache.org/jira/browse/NIFI. Thanks, James On Tue, May 23, 2017 at 6:59 AM, Michael Pearce <[email protected]<mailto:[email protected]>> wrote: Hi All, We are evaluating options to migrate from custom systems integrating kafka to our Hadoop and Database stores, to a more generic solution which NiFi at a high level seems to fit the bill….almost. In our system we have kafka with avro data, we want to persist this to our traditional SQL data warehouse staging tables, as just an Extract and Load process similar to Hadoop. To do this though we need the staging to evolve automatically safely (aka don’t remove columns), based on the inbound avro data evolving. We cannot seem to find a way to do this. Does Apache NiFi support automatically evolving SQL database staging table schema’s? Similar in nature to confluent’s kafka connect jdbc connector? If it does not today is there anything in the horizon to support doing this? Thanks in advance, Mike The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
