Problem: * Source database with over 200 tables. * Current Nifi ‘system’ we are developing can extract data from those 200 tables into NiFi flows of JSON-formatted data, essentially separate flows for each table with an attribute that indicates the tablename and other useful attributes but NOT the schema. * Do some data transforms, and prepare it for target database load. This is where I am struggling. * Large volume of data so we need to batch load using PutDatabaseRecord. * PutDatabaseRecord record readers such as JsonPathReader need attributes defined for each element in the data – I’d need to define over 200 instances of PutDatabaseRecord and route based on the tablename. Not. * AvroReader seems almost a natural fit, I can InferAvroSchema from the Json, but I’m not finding an easy way to convert the Json to Avro… * CSVReader seems like the only other choice but the manual conversion of formats might also be a pain…
Thoughts on solutions?
