[
https://issues.apache.org/jira/browse/NIFI-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Steinebrey updated NIFI-14313:
----------------------------------
Affects Version/s: 2.3.0
> JoinEnrichment processor; failure to close result sets contributes to excess
> memory usage
> -----------------------------------------------------------------------------------------
>
> Key: NIFI-14313
> URL: https://issues.apache.org/jira/browse/NIFI-14313
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
> Reporter: Jim Steinebrey
> Assignee: Jim Steinebrey
> Priority: Major
> Fix For: 2.4.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This ticket involves the JoinEnrichment processor when configured to use a
> Join Strategy of SQL (JOIN_SQL in the code). When the JoinEnrichment
> processor uses a SqlJoinStrategy, up to 25 result sets are retained in memory
> in a cache. In the case of large ResultSets (for example, from
> record-oriented FlowFiles with many records), this has the potential to
> adversely impact memory available to the NiFi process.
> When JoinEnrichment uses the SqlJoinStrategy, the SqlJoinStrategy holds a
> cache of SqlJoinCalciteParameters. Each SqlJoinCalciteParameters holds a
> PreparedStatement which holds in memory the last result set returned by
> executing the PreparedStatement.
> The remediation is to close the result set AFTER it is finished being used
> and BEFORE it is put into the cache.
> This reduces memory usage without worsening performance because a new result
> set is always created using executeQuery() each time the PreparedStatement is
> reused.
> The cache is being used to reuse the PreparedStatement to avoid having to
> compile the sql statement over and over for each flow file processed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
