HI,
I seem to have a situation similar to
[NIFI-9903]<https://issues.apache.org/jira/browse/NIFI-9903>, this time in the
JoinEnrichment processor.
I am performing some enrichments, and sometimes the enrichment look up fails as
the item is not there. If the enrichment fails for the first item in an array
of records then the merge fails to properly merge subsequent records in the
same array. The join processor uses an "infer" JSON reader for the original
leg, a schema based reader for the enriched leg, and a "inherit" JSON writer. I
am using the "Insert" join strategy. Here is an example.
If the original record is
[{"transport_protocol_id":17,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]
Then both lookups succeed, and the enrichment record looks like this
[
{"network_transport" : {"name" : "udp", "code" : 17, "alias" : "UDP",
"comment" : "user datagram protocol"}},
{"network_transport" : {"name" : "tcp", "code" : 6, "alias" : "TCP",
"comment" : "transmission control protocol"}
]
And the joined record looks like this.
[
{"transport_protocol_id" : 17,"Enrichment" : {
"network_transport" : {"name" : "udp","code" : 17,"alias" : "UDP","comment"
: "user datagram protocol"}}},
{"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : {"name" : "tcp","code" : 6,"alias" : "TCP","comment"
: "transmission control protocol"}}}
]
However if the first record has a key value that is out of range, such as this
[{"transport_protocol_id":9999,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]
Then the first record in the enriched leg will be null, even if the rest of the
records are correct. However the enrichment is still valid JSON once I have
processed it in the enrichment leg.
[
{"network_transport" : null},
{"network_transport" : {"name" : "tcp", "code" : 6,"alias" : "TCP", "comment"
: "transmission control protocol"}}
]
But the joined record does not properly process the subsequent records, and the
content looks like this.
[
{"transport_protocol_id" : 9999,"Enrichment" : {
"network_transport" : null}},
{"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : "MapRecord[{name=tcp, alias=TCP, comment=transmission
control protocol, code=6}]"}}
]
Is there any step I could use to ensure the join happens as expected? Or is
this the same situation as the JIRA I mentioned above? I am not able to use a
schema based writer as our real case has too many input record types and
enrichment options that the number of combinations, and hence schemas, could
not be managed.
Thanks
Steve Hindmarch
