Jim Steinebrey created NIFI-14313:
-------------------------------------
Summary: JoinEnrichment processor; failure to close result sets
contributes to excess memory usage
Key: NIFI-14313
URL: https://issues.apache.org/jira/browse/NIFI-14313
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.2.0, 2.1.0, 2.0.0
Reporter: Jim Steinebrey
This ticket involves the JoinEnrichment processor when configured to use a
Join Strategy of SQL (JOIN_SQL in the code). When the JoinEnrichment processor
uses a SqlJoinStrategy, up to 25 result sets are retained in memory in a cache.
In the case of large ResultSets (for example, from record-oriented FlowFiles
with many records), this has the potential to adversely impact memory available
to the NiFi process.
When JoinEnrichment uses the SqlJoinStrategy, the SqlJoinStrategy holds a cache
of SqlJoinCalciteParameters. Each SqlJoinCalciteParameters holds a
PreparedStatement which holds in memory the last result set returned by
executing the PreparedStatement.
The remediation is to close the result set AFTER it is finished being used and
BEFORE it is put into the cache.
This reduces memory usage without worsening performance because a new result
set is always created using executeQuery() each time the PreparedStatement is
reused.
The cache is being used to reuse the PreparedStatement to avoid having to
compile the sql statement over and over for each flow file processed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
