Joe Witt created NIFI-11815:
-------------------------------
Summary: Dynamic Controller Service for Database
Connections/Pooling
Key: NIFI-11815
URL: https://issues.apache.org/jira/browse/NIFI-11815
Project: Apache NiFi
Issue Type: New Feature
Reporter: Joe Witt
Consider a user that interacts with a large number of databases/tables/etc..
for various different flows that happen at different times and so on. Could be
thousands of such configurations. Each one being unique means it needs a
different controller service which in turn leads to thousands of these and a
fairly painful configuration.
The current DBCPConnectionPool controller service works great when you know all
those values and they're stable for a given processor/purpose.
A DynamicDBCPConnectionPool could offer a far better configuration experience
for these more ad-hoc and varied cases mentioned above. Consider having a
single ControllerService instance that is configured with a specific user and
password but other values like database name, table name, and such is
configurable via expression language, properties, etc.. that are passed in
potentially on a per flow file basis.
The DynamicDBCPConnectionPool then would create the necessary values from the
expression language evaluation and the controller service settings then do a
lookup for any actual ConnectionPools that match those inputs. If it exists
use it and if it doesn't make a new one. Keep a small cache of 10 or some
configurable number of these. Let them age-off be shut down when no longer
being recently/actively used.
This cuts down from 1000s to a single controller service while still yielding
good performance assuming it is cache/hit friendly periodic usage of a given
combination of inputs. Far better user experience and resource management.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
