Mike, I believe you are correct that NiFi does not have built-in support for several of those tasks:
* Identifying a schema evolution (separate from a non-conforming flowfile) * Identifying safe schema changes * Automatically applying schema changes to a SQL database However, the new schema registry and record handling components introduced in NiFi 1.2.0 might be useful to build some of these features. I would certainly encourage you to create a JIRA ticket to capture the use case at https://issues.apache.org/jira/browse/NIFI. Thanks, James On Tue, May 23, 2017 at 6:59 AM, Michael Pearce <[email protected]> wrote: > > > Hi All, > > > > We are evaluating options to migrate from custom systems integrating kafka > to our Hadoop and Database stores, to a more generic solution which NiFi at > a high level seems to fit the bill….almost. > > > > In our system we have kafka with avro data, we want to persist this to our > traditional SQL data warehouse staging tables, as just an Extract and Load > process similar to Hadoop. > > > > To do this though we need the staging to evolve automatically safely (aka > don’t remove columns), based on the inbound avro data evolving. > > > > We cannot seem to find a way to do this. > > > > Does Apache NiFi support automatically evolving SQL database staging table > schema’s? Similar in nature to confluent’s kafka connect jdbc connector? > > If it does not today is there anything in the horizon to support doing > this? > > > > Thanks in advance, > > Mike > > > > > The information contained in this email is strictly confidential and for > the use of the addressee only, unless otherwise indicated. If you are not > the intended recipient, please do not read, copy, use or disclose to others > this message or any attachment. Please also notify the sender by replying > to this email or by telephone (+44(020 7896 0011) and then delete the email > and any copies of it. Opinions, conclusion (etc) that do not relate to the > official business of this company shall be understood as neither given nor > endorsed by it. IG is a trading name of IG Markets Limited (a company > registered in England and Wales, company number 04008957) and IG Index > Limited (a company registered in England and Wales, company number > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG > Index Limited (register number 114059) are authorised and regulated by the > Financial Conduct Authority. >
