[
https://issues.apache.org/jira/browse/NIFI-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848396#comment-15848396
]
ASF GitHub Bot commented on NIFI-2881:
--------------------------------------
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1407#discussion_r98898226
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
---
@@ -212,9 +215,21 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
final Map<String, String> statePropertyMap = new
HashMap<>(stateMap.toMap());
//If an initial max value for column(s) has been specified using
properties, and this column is not in the state manager, sync them to the state
property map
- for(final Map.Entry<String,String> maxProp :
maxValueProperties.entrySet()){
- if
(!statePropertyMap.containsKey(maxProp.getKey().toLowerCase())) {
- statePropertyMap.put(maxProp.getKey().toLowerCase(),
maxProp.getValue());
+ for (final Map.Entry<String, String> maxProp :
maxValueProperties.entrySet()) {
+ String maxPropKey = maxProp.getKey().toLowerCase();
+ String fullyQualifiedMaxPropKey = getStateKey(tableName,
maxPropKey);
+ if (!statePropertyMap.containsKey(fullyQualifiedMaxPropKey)) {
+ String newMaxPropValue;
+ // If we can't find the value at the fully-qualified key
name, it is possible (under a previous scheme)
+ // the value has been stored under a key that is only the
column name. Fall back to check the column name,
+ // but store the new initial max value under the
fully-qualified key.
+ if (statePropertyMap.containsKey(maxPropKey)) {
+ newMaxPropValue = statePropertyMap.get(maxPropKey);
+ } else {
+ newMaxPropValue = maxProp.getValue();
--- End diff --
Agreed, I was faced with the choice of 1) an extra query for
QueryDatabaseTable, which might happen when scheduled (if using static table
names and max-value columns) or once for each table, or 2) not allowing
incoming flow files for QDT. I chose option 2 for the reason you point out,
the other processors offer that functionality in a more robust manner IMHO. It
sounds like we're on the same page, I will add @SeeAlso references to the
processors you mention, along with some more text in the CapabilityDescription
for QDT to refer to the other processors.
> Allow Database Fetch processor(s) to accept incoming flow files and use
> Expression Language
> -------------------------------------------------------------------------------------------
>
> Key: NIFI-2881
> URL: https://issues.apache.org/jira/browse/NIFI-2881
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Matt Burgess
> Assignee: Matt Burgess
>
> The QueryDatabaseTable and GenerateTableFetch processors do not allow
> Expression Language to be used in the properties, mainly because they also do
> not allow incoming connections. This means if the user desires to fetch from
> multiple tables, they currently need one instance of the processor for each
> table, and those table names must be hard-coded.
> To support the same capabilities for multiple tables and more flexible
> configuration via Expression Language, these processors should have
> properties that accept Expression Language, and GenerateTableFetch should
> accept (optional) incoming connections.
> Conversation about the behavior of the processors is welcomed and encouraged.
> For example, if an incoming flow file is available, do we also still run the
> incremental fetch logic for tables that aren't specified by this flow file,
> or do we just do incremental fetching when the processor is scheduled but
> there is no incoming flow file. The latter implies a denial-of-service could
> take place, by flooding the processor with flow files and not letting it do
> its original job of querying the table, keeping track of maximum values, etc.
> This is likely a breaking change to the processors because of how state
> management is implemented. Currently since the table name is hard coded, only
> the column name comprises the key in the state. This would have to be
> extended to have a compound key that represents table name, max-value column
> name, etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
