Have you tried using ConvertRecord instead of ConvertAvroSchema? ConvertAvroSchema comes from the Kite bundle which has a few other processors like ConvertCsvToAvro and ConvertJsonToAvro, all of which existed before NiFi's record processing capabilities.
In general the record processors should be used instead. On Fri, Aug 24, 2018 at 10:32 AM, David Gallagher <[email protected]> wrote: > Hi Matt - I tried changing the order of the union in the Output schema, and > also tried using a default value. Nothing worked, I still got an 'unmapped > field' message and could not start the component. But, it looks like it is > possible to add the declaration to the Input schema using any of the > following: > > {"name":"test_field","type":["string"], "default": "Clever"} > {"name":"test_field","type":["string", "null"],"default": "CleverCAD"} > > So, unless I'm missing something it looks like the Input schema absolutely > must map every field in the Output schema and must declare a default as well. > I can do that without having to alter the query, but it would be even better > if I could ignore it completely. > > Thanks, > > Dave > > -----Original Message----- > From: Matt Burgess <[email protected]> > Sent: Thursday, August 23, 2018 2:34 PM > To: [email protected] > Subject: Re: ConvertAvroSchema with unmapped fields > > Dave, > > At one point (perhaps it is still true), in order to have a default value its > type had to be the first type in the array. For optional fields, the default > is null so try inverting your type arrays to match the way they're done in > the input schema; namely, ["null", "string"] rather than ["string","null"]. > Also if you're using a version of NiFi > 1.6.0 or later, you may be able to provide a default value explicitly. > Prior to NiFi 1.6.0 this was a bug [1], please see the Jira for more details > (and an example). > > Regards, > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-4893 > > On Thu, Aug 23, 2018 at 2:29 PM David Gallagher > <[email protected]> wrote: >> >> Hi Andy – this is the schema we’re getting from the database query: >> >> >> >> "type":"record", >> >> "name":"view_Logons", >> >> "namespace":"any.data", >> >> "fields":[ >> >> {"name":"id","type":["null","int"]}, >> >> {"name":"timestamp","type":["null","string"]}, >> >> {"name":"workstation","type":["null","string"]}, >> >> {"name":"username","type":["null","string"]}, >> >> {"name":"transaction_type","type":["null","string"]}, >> >> {"name":"result","type":["null","string"]}, >> >> {"name":"message","type":["null","string"]} >> >> ] >> >> } >> >> >> >> >> >> And this is the schema we’re converting to: >> >> >> >> {"namespace": "com.cleverdevices.cleverreports.avro", >> >> "type": "record", >> >> "name": "AuthorizationTransaction", >> >> "fields": [ >> >> {"name": "product", "type": ["string", "null"]}, >> >> {"name": "sub_product", "type": ["string", "null"]}, >> >> {"name": "id", "type": "long"}, >> >> {"name": "timestamp", "type": "string"}, >> >> {"name": "workstation", "type": ["string", "null"]}, >> >> {"name": "username", "type": ["string", "null"]}, >> >> {"name": "transaction_type", "type": ["string"]}, >> >> {"name": "result", "type": "string"}, >> >> {"name": "message", "type": ["string", "null"]} >> >> ] >> >> } >> >> >> >> Thanks, >> >> >> >> Dave >> >> >> >> From: Andy LoPresto <[email protected]> >> Sent: Thursday, August 23, 2018 1:28 PM >> To: [email protected] >> Subject: Re: ConvertAvroSchema with unmapped fields >> >> >> >> Hi Dave, >> >> >> >> Can you provide the two schemas (redact anything necessary). There is a way >> to specify an “optional” field [1] by setting the type to an array of null >> and the type you support. You can also specify a default value if you wish, >> which will be set for records that do not contain a value there: >> >> >> >> { >> >> "type": "record", >> >> "name": "Address", >> >> "fields" : [ >> >> {"name": "streetNumber", "type": "int"}, >> >> {"name": "aptNumber", "type": ["null", "int"]}, // optional apt >> number >> >> {"name": "country", "type": "string", "default": "US"}, // default >> country >> >> ] >> >> } >> >> >> >> >> >> [1] https://avro.apache.org/docs/1.8.1/spec.html#schema_record >> >> >> >> >> >> Andy LoPresto >> >> [email protected] >> >> [email protected] >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> >> >> On Aug 23, 2018, at 10:07 AM, David Gallagher <[email protected]> >> wrote: >> >> >> >> Hi – I’ve got a scenario where I’m trying to convert the implicit AVRO >> schema associated with a database record to a different AVRO schema. I’m >> trying to use the ConvertAvroSchema processor, but it won’t validate because >> there is an ‘unmapped’ field that exists in the second schema but not the >> first. Now, I could work around this issue by including a constant in the >> database record, but I would prefer to have a null or default value supplied >> by the second schema instead. Is there some setting I’m missing that would >> let me specify the behavior for unmapped fields? >> >> >> >> Thanks, >> >> >> >> Dave >> >>
