Re: Record Readers and Writers

2020-04-21 Thread Matt Burgess
Dave, That JSON is actually the schema of the data, not the data itself. Avro Schemas are indeed stored in JSON format, under the hood we have utilities for changing back and forth between Avro schemas and internal object representations of NiFi Record schemas. We don't serialize the Record schema

Re: Record Readers and Writers

2020-04-21 Thread DAVID SMITH
Hi Matt Thanks for your reply, I will certainly take on board everything you and Andy advise and I will look at classes you mentioned and I will also read the links provided. I ran the TestXMLReader as a junit in Eclipse, a sample of the the console output is : 20:15:01.675 [pool-1-thread-1] DEB

Re: Record Readers and Writers

2020-04-21 Thread Andy LoPresto
Hi Dave, The underlying internal “record format” is not JSON. Avro [1] is used to describe schemas across all record formats, but the internal data storage is NiFi specific. You may be interested in these articles by Mark Payne and Bryan Bende [2][3][4] and the potential use of the ScriptedRead

Re: Record Readers and Writers

2020-04-21 Thread Matt Burgess
Dave, Which unit test file(s) for XMLReader are you seeing that the output is JSON? I ran TestXMLReader and TestXMLRecordReader and didn't see any console output. Most unit tests for Readers only test that the internal Record objects have been parsed correctly, they don't usually go an extra step

Record Readers and Writers

2020-04-21 Thread DAVID SMITH
Hi I want to use the ConvertRecord Processor with it's underlying Record Readers and Writers to convert files from XML or JSON to a bespoke format and probably vice versa.I have looked at the Readers/Writers currently provided and decided that I can use the XML/JSON ones provided but I will need