Hi All, We are evaluating options to migrate from custom systems integrating kafka to our Hadoop and Database stores, to a more generic solution which NiFi at a high level seems to fit the bill….almost.
In our system we have kafka with avro data, we want to persist this to our traditional SQL data warehouse staging tables, as just an Extract and Load process similar to Hadoop. To do this though we need the staging to evolve automatically safely (aka don’t remove columns), based on the inbound avro data evolving. We cannot seem to find a way to do this. Does Apache NiFi support automatically evolving SQL database staging table schema’s? Similar in nature to confluent’s kafka connect jdbc connector? If it does not today is there anything in the horizon to support doing this? Thanks in advance, Mike The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
