Hey Bob,
The InferAvroSchema processor actually works against JSON and CSV data. It is
designed to infer an Avro Schema
so that you can convert either of those into Avro if you want. So you can send
it JSON data and it will infer the schema for
you and put it in the "inferred.avro.schema" attribute. You can then use the
JsonTreeReader and configure
the "Schema Access Strategy" to "Use Schema Text Property" and set the "Schema
Text" property to "${inferred.avro.schema}"
That should allow you to use the JsonTreeReader with PutDatabaseRecord.
Thanks
-Mark
On Aug 17, 2018, at 9:54 AM, Kuhfahl, Bob
<[email protected]<mailto:[email protected]>> wrote:
Problem:
* Source database with over 200 tables.
* Current Nifi ‘system’ we are developing can extract data from those 200
tables into NiFi flows of JSON-formatted data, essentially separate flows for
each table with an attribute that indicates the tablename and other useful
attributes but NOT the schema.
* Do some data transforms, and prepare it for target database load. This
is where I am struggling.
* Large volume of data so we need to batch load using PutDatabaseRecord.
* PutDatabaseRecord record readers such as JsonPathReader need attributes
defined for each element in the data – I’d need to define over 200 instances of
PutDatabaseRecord and route based on the tablename. Not.
* AvroReader seems almost a natural fit, I can InferAvroSchema from the
Json, but I’m not finding an easy way to convert the Json to Avro…
* CSVReader seems like the only other choice but the manual conversion of
formats might also be a pain…
Thoughts on solutions?
