Matt Burgess created NIFI-5590:
----------------------------------
Summary: Allow database fetch processors to store state by
database/catalog/schema
Key: NIFI-5590
URL: https://issues.apache.org/jira/browse/NIFI-5590
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Matt Burgess
The database fetch processors (GenerateTableFetch, QueryDatabaseTable, e.g.)
use a scheme of "table.column" to store the state of the maximum values for a
column, as well as a similar entry for the column type.
Now that GenerateTableFetch can accept incoming flow files, and if it uses a
DBCPConnectionPoolLookup service, the database could differ yet contain the
same table/column name combinations (MySQL shards, e.g.). The state values as
currently stored will be shared in this situation, which could certainly lead
to errors.
If possible/prudent, the fully-qualified state name should include
database/catalog/schema information such that the state entries are unique at a
database level. This may prove difficult as different drivers may or may not
make such information available, for example sometimes the database name is the
schema, sometimes it is the catalog, etc. As long as the names are unique at
the database level, there should be no conflicts. This may lead to a more
complicated naming scheme, and backwards-compatibility should be maintained
(unless this is implemented for a major release).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
