Chris Norton created NIFI-10192:
-----------------------------------
Summary: LookupRecord attempts first lookup multiple times
Key: NIFI-10192
URL: https://issues.apache.org/jira/browse/NIFI-10192
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.16.3, 1.16.2, 1.16.1
Reporter: Chris Norton
A change was implemented in
[NIFI-9903|https://issues.apache.org/jira/browse/NIFI-9903] which results in
*lookupService.lookup()* being called twice per record ([at least until the
first
match|https://github.com/apache/nifi/blob/rel/nifi-1.16.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupRecord.java#L350]).
For lookup services which are idempotent (CSVRecordLookupService,
DistributedMapCacheLookupService, PropertiesFileLookupService) making lookups
twice won’t affect the result or have undesired side effects. However, the
RestLookupService can make arbitrary HTTP requests for the standard HTTP
methods (GET, POST, PUT, DELETE) and there is no guarantee that these requests
will be idempotent. POST requests in particular are not expected to be
idempotent and may cause undesirable behaviour if invoked multiple times (as in
our case).
As the name suggests, LookupRecord could be expected to be used only to perform
lookups which are idempotent and do not have side effects. [Matt Burgess wrote
an
article|http://funnifi.blogspot.com/2018/08/database-sequence-lookup-with.html]
where it seems the expected behaviour was that *lookupService.lookup()* would
only be called once. The change in behaviour and being called twice would now
cause IDs to be skipped.
It was suggested by Mark Payne in a Slack discussion that lookup results could
be cached up until the first match, which may alleviate the issues we are
seeing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
