[ 
https://issues.apache.org/jira/browse/NIFI-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430406#comment-16430406
 ] 

ASF GitHub Bot commented on NIFI-1706:
--------------------------------------

Github user ijokarumawak commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/2162#discussion_r180055717
  
    --- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractDatabaseFetchProcessor.java
 ---
    @@ -291,20 +291,30 @@ public void setup(final ProcessContext context, 
boolean shouldCleanCache, FlowFi
                         if (shouldCleanCache){
                             columnTypeMap.clear();
                         }
    +
    +                    final List<String> maxValueColumnNameList = 
Arrays.asList(maxValueColumnNames.toLowerCase().split(","));
    +                    final List<String> maxValueQualifiedColumnNameList = 
new ArrayList<>();
    +
    +                    for(String maxValueColumn:maxValueColumnNameList){
    +                        String colKey = getStateKey(tableName, 
maxValueColumn.trim());
    +                        maxValueQualifiedColumnNameList.add(colKey);
    +                    }
    +
                         for (int i = 1; i <= numCols; i++) {
                             String colName = 
resultSetMetaData.getColumnName(i).toLowerCase();
                             String colKey = getStateKey(tableName, colName);
    +
    +                        //only include columns that are part of the 
maximum value tracking column list
    +                        
if(!maxValueQualifiedColumnNameList.contains(colKey)){
    +                            continue;
    +                        }
    +
                             int colType = resultSetMetaData.getColumnType(i);
                             columnTypeMap.putIfAbsent(colKey, colType);
                         }
     
    -                    List<String> maxValueColumnNameList = 
Arrays.asList(maxValueColumnNames.split(","));
    -
    -                    for(String maxValueColumn:maxValueColumnNameList){
    -                        String colKey = getStateKey(tableName, 
maxValueColumn.trim().toLowerCase());
    -                        if(!columnTypeMap.containsKey(colKey)){
    -                            throw new ProcessException("Column not found 
in the table/query specified: " + maxValueColumn);
    -                        }
    +                    if(maxValueQualifiedColumnNameList.size() > 0 && 
columnTypeMap.size() != maxValueQualifiedColumnNameList.size()){
    --- End diff --
    
    @patricker This check should be implemented as the previous commit. The 
size of columnTypeMap can be different with GenerateTableFetch when it's 
configured to resolve table and column names dynamically with FlowFile EL and 
deals with multiple tables.


> Extend QueryDatabaseTable to support arbitrary queries
> ------------------------------------------------------
>
>                 Key: NIFI-1706
>                 URL: https://issues.apache.org/jira/browse/NIFI-1706
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 1.4.0
>            Reporter: Paul Bormans
>            Assignee: Peter Wicks
>            Priority: Major
>              Labels: features
>
> The QueryDatabaseTable is able to observe a configured database table for new 
> rows and yield these into the flowfile. The model of an rdbms however is 
> often (if not always) normalized so you would need to join various tables in 
> order to "flatten" the data into useful events for a processing pipeline as 
> can be build with nifi or various tools within the hadoop ecosystem.
> The request is to extend the processor to specify an arbitrary sql query 
> instead of specifying the table name + columns.
> In addition (this may be another issue?) it is desired to limit the number of 
> rows returned per run. Not just because of bandwidth issue's from the nifi 
> pipeline onwards but mainly because huge databases may not be able to return 
> so many records within a reasonable time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to