[
https://issues.apache.org/jira/browse/NIFI-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384205#comment-15384205
]
ASF GitHub Bot commented on NIFI-2156:
--------------------------------------
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/642#discussion_r71347101
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListDatabaseTables.java
---
@@ -260,8 +241,27 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
.filter(segment -> !StringUtils.isEmpty(segment))
.collect(Collectors.joining("."));
- String fqTableName = stateMapProperties.get(fqn);
- if (fqTableName == null) {
+ String lastTimestampForTable = stateMapProperties.get(fqn);
+ boolean refreshTable = true;
+ try {
+ // Refresh state if the interval has elapsed
+ long lastRefreshed = -1;
+ final long currentTime = System.currentTimeMillis();
+ if (!StringUtils.isEmpty(lastTimestampForTable)) {
+ lastRefreshed =
Long.parseLong(lastTimestampForTable);
+ }
+ if (lastRefreshed == -1 || (refreshInterval > 0 &&
currentTime >= (lastRefreshed + refreshInterval))) {
+ stateMapProperties.remove(lastTimestampForTable);
+ } else {
+ refreshTable = false;
+ }
+ } catch (final NumberFormatException nfe) {
+ getLogger().error("Failed to retrieve observed last
table fetches from the State Manager. Will not perform "
--- End diff --
This exception really *shouldn't* happen since we are setting the long
ourselves but if it does it will fail entirely until state is cleared by the
user. In addition to returning and yielding, it should probably clear the
offending state entry (and log in the error message that this is hapenning and
the ramifications). This will at least give the processor a chance to continue
working if it every reaches this state.
> Add ListDatabaseTables processor
> --------------------------------
>
> Key: NIFI-2156
> URL: https://issues.apache.org/jira/browse/NIFI-2156
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: Matt Burgess
> Assignee: Matt Burgess
> Fix For: 1.0.0
>
>
> This processor would use a DatabaseConnectionPool controller service, call
> getTables(), and if the (optional, defaulting-to-false) property "Include Row
> Count" is set, then a "SELECT COUNT(1) from table" would be issued to the
> database. The table catalog, schema, name, type, remarks (and its count if
> specified) will be included as attributes in a zero-content flow file.
> It will also use State Management to only list tables once. If new tables are
> added (and the processor is running), then the new tables' flow files will be
> generated. Changing any property that could affect the list of returned
> tables (such as the DB Connection, catalog, schema pattern, table name
> pattern, or table types) will reset the state and all tables will be fetched
> using the new criteria. The state can also be manually cleared using the
> standard Clear State link on the View State dialog (available on the
> processor's context menu)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
