Hi Matt - I tried changing the order of the union in the Output schema, and
also tried using a default value. Nothing worked, I still got an 'unmapped
field' message and could not start the component. But, it looks like it is
possible to add the declaration to the Input schema using any of the following:
{"name":"test_field","type":["string"], "default": "Clever"}
{"name":"test_field","type":["string", "null"],"default": "CleverCAD"}
So, unless I'm missing something it looks like the Input schema absolutely must
map every field in the Output schema and must declare a default as well. I can
do that without having to alter the query, but it would be even better if I
could ignore it completely.
Thanks,
Dave
-----Original Message-----
From: Matt Burgess <[email protected]>
Sent: Thursday, August 23, 2018 2:34 PM
To: [email protected]
Subject: Re: ConvertAvroSchema with unmapped fields
Dave,
At one point (perhaps it is still true), in order to have a default value its
type had to be the first type in the array. For optional fields, the default is
null so try inverting your type arrays to match the way they're done in the
input schema; namely, ["null", "string"] rather than ["string","null"]. Also if
you're using a version of NiFi
1.6.0 or later, you may be able to provide a default value explicitly.
Prior to NiFi 1.6.0 this was a bug [1], please see the Jira for more details
(and an example).
Regards,
Matt
[1] https://issues.apache.org/jira/browse/NIFI-4893
On Thu, Aug 23, 2018 at 2:29 PM David Gallagher <[email protected]>
wrote:
>
> Hi Andy – this is the schema we’re getting from the database query:
>
>
>
> "type":"record",
>
> "name":"view_Logons",
>
> "namespace":"any.data",
>
> "fields":[
>
> {"name":"id","type":["null","int"]},
>
> {"name":"timestamp","type":["null","string"]},
>
> {"name":"workstation","type":["null","string"]},
>
> {"name":"username","type":["null","string"]},
>
> {"name":"transaction_type","type":["null","string"]},
>
> {"name":"result","type":["null","string"]},
>
> {"name":"message","type":["null","string"]}
>
> ]
>
> }
>
>
>
>
>
> And this is the schema we’re converting to:
>
>
>
> {"namespace": "com.cleverdevices.cleverreports.avro",
>
> "type": "record",
>
> "name": "AuthorizationTransaction",
>
> "fields": [
>
> {"name": "product", "type": ["string", "null"]},
>
> {"name": "sub_product", "type": ["string", "null"]},
>
> {"name": "id", "type": "long"},
>
> {"name": "timestamp", "type": "string"},
>
> {"name": "workstation", "type": ["string", "null"]},
>
> {"name": "username", "type": ["string", "null"]},
>
> {"name": "transaction_type", "type": ["string"]},
>
> {"name": "result", "type": "string"},
>
> {"name": "message", "type": ["string", "null"]}
>
> ]
>
> }
>
>
>
> Thanks,
>
>
>
> Dave
>
>
>
> From: Andy LoPresto <[email protected]>
> Sent: Thursday, August 23, 2018 1:28 PM
> To: [email protected]
> Subject: Re: ConvertAvroSchema with unmapped fields
>
>
>
> Hi Dave,
>
>
>
> Can you provide the two schemas (redact anything necessary). There is a way
> to specify an “optional” field [1] by setting the type to an array of null
> and the type you support. You can also specify a default value if you wish,
> which will be set for records that do not contain a value there:
>
>
>
> {
>
> "type": "record",
>
> "name": "Address",
>
> "fields" : [
>
> {"name": "streetNumber", "type": "int"},
>
> {"name": "aptNumber", "type": ["null", "int"]}, // optional apt
> number
>
> {"name": "country", "type": "string", "default": "US"}, // default
> country
>
> ]
>
> }
>
>
>
>
>
> [1] https://avro.apache.org/docs/1.8.1/spec.html#schema_record
>
>
>
>
>
> Andy LoPresto
>
> [email protected]
>
> [email protected]
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Aug 23, 2018, at 10:07 AM, David Gallagher <[email protected]>
> wrote:
>
>
>
> Hi – I’ve got a scenario where I’m trying to convert the implicit AVRO schema
> associated with a database record to a different AVRO schema. I’m trying to
> use the ConvertAvroSchema processor, but it won’t validate because there is
> an ‘unmapped’ field that exists in the second schema but not the first. Now,
> I could work around this issue by including a constant in the database
> record, but I would prefer to have a null or default value supplied by the
> second schema instead. Is there some setting I’m missing that would let me
> specify the behavior for unmapped fields?
>
>
>
> Thanks,
>
>
>
> Dave
>
>
