[
https://issues.apache.org/jira/browse/NIFI-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122656#comment-16122656
]
Joseph Witt commented on NIFI-4283:
-----------------------------------
NIFI-3335 got into a funky state that probably would have been solved by asking
for integrity checker run against NIFI. Instead just cloning and deleting old
ticket. History from NIFI-3335 is here
Back to previous view
[NIFI-3335] GenerateTableFetch should allow you to specify an initial Max Value
Created: 12/Jan/17 Updated: 11/Aug/17
Status: Open
Project: Apache NiFi
Component/s: Extensions
Affects Version/s: None
Fix Version/s: 1.4.0
Type: Improvement Priority: Major
Reporter: Matt Burgess Assignee: Joseph Witt
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Description
NIFI-2583 added the ability (via dynamic properties) to specify initial Max
Values for columns, to enable the user to "pick up where they left off" if
something happened with a flow, a NiFi instance, etc. where the state was
stored but the processing did not complete successfully.
This feature would also be helpful in GenerateTableFetch, which also supports
max-value columns.
Since NIFI-2881 adds incoming flow file support, it's more useful if Initial
max values can be specified via flow file attributes. Because if a table name
is dynamically passed via flow file attribute and Expression Language, user
won't be able to configure dynamic processor attribute in advance for each
possible table.
Add dynamic properties ('initial.maxvalue.<max_value_column>' same as
QueryDatabaseTable) to specify initial max values statically, and also use
incoming flow file attributes named 'initial.maxvalue.<max_value_column>' if
any.
Comments
Comment by Matt Burgess [ 17/Jul/17 ]
Peter Wicks Are you still working on this? If not, do you mind if I unassign
it? Thanks!
Comment by Peter Wicks [ 19/Jul/17 ]
Matt Burgess I'm not working on this. I've unassigned myself.
Comment by ASF GitHub Bot [ 25/Jul/17 ]
GitHub user mattyb149 opened a pull request:
https://github.com/apache/nifi/pull/2039
NIFI-3335: Add initial.maxvalue support to GenerateTableFetch
Moved the common code back to the abstract base class
For all changes:
[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you
are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch
(typically master)?
[x] Is your initial contribution a single, squashed commit?
For code changes:
[x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in
a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
[ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE
file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name
(programmatic access) for each of the new properties?
For documentation related changes:
[x] Have you ensured that format looks appropriate for the output in which it
is rendered?
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mattyb149/nifi NIFI-3335
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2039.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2039
commit 8eea477c02fdb2ca19ec77da9168ceec5e8eae7c
Author: Matt Burgess <mattyb...@apache.org>
Date: 2017-07-25T20:07:52Z
NIFI-3335: Add initial.maxvalue support to GenerateTableFetch
Comment by ASF GitHub Bot [ 26/Jul/17 ]
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2039#discussion_r129494131
— Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
—
@@ -198,6 +201,25 @@ public void onTrigger(final ProcessContext context, final
ProcessSessionFactory
// set as the current state map (after the session has been committed)
final Map<String, String> statePropertyMap = new HashMap<>(stateMap.toMap());
+ // If an initial max value for column(s) has been specified using properties,
and this column is not in the state manager, sync them to the state property map
+ final Map<String,String> maxValueProperties =
getDefaultMaxValueProperties(context.getProperties());
— End diff –
Would it make sense to do it in the ``@OnScheduled`` method of the Abstract
class? You could have the same code in common for QueryDatabaseTable and
GenerateTableFetch. Besides not sure it needs to be done each time the
processor is triggered? Thoughts?
Comment by ASF GitHub Bot [ 26/Jul/17 ]
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2039#discussion_r129648705
— Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
—
@@ -198,6 +201,25 @@ public void onTrigger(final ProcessContext context, final
ProcessSessionFactory
// set as the current state map (after the session has been committed)
final Map<String, String> statePropertyMap = new HashMap<>(stateMap.toMap());
+ // If an initial max value for column(s) has been specified using properties,
and this column is not in the state manager, sync them to the state property map
+ final Map<String,String> maxValueProperties =
getDefaultMaxValueProperties(context.getProperties());
— End diff –
Yeah that's a better idea, should've been done in the original PR but I missed
it during my review and just cut-pasted it back to the abstract class to share.
I won't be able to move all that common code block as it needs the table name
which can come from a flow file. But I can call getDefaultMaxValueProperties()
at schedule-time vs trigger time.
Comment by ASF subversion and git services [ 27/Jul/17 ]
Commit dc4006f423b83e7819981569d17e8d90b9a9d1a4 in nifi's branch
refs/heads/master from Matt Burgess
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=dc4006f ]
NIFI-3335: Add initial.maxvalue support to GenerateTableFetch
Signed-off-by: Pierre Villard <pierre.villard...@gmail.com>
This closes #2039.
Comment by ASF GitHub Bot [ 27/Jul/17 ]
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2039
Comment by Pierre Villard [ 27/Jul/17 ]
OK, don't know why there is an issue with this JIRA specifically... but fact
is: I cannot resolve/close it.
I think that's the same for you Matt Burgess. Could you have a look Joseph Witt?
Comment by Joseph Witt [ 11/Aug/17 ]
i dont see an obvious way to close it either. will try some things...weird
Generated at Fri Aug 11 01:28:09 UTC 2017 by Joseph Witt using JIRA
6.4.14#64029-sha1:ae256fe0fbb912241490ff1cecfb323ea0905ca5.
> GenerateTableFetch should allow you to specify an initial Max Value (clone of
> defunct NIFI-3335)
> ------------------------------------------------------------------------------------------------
>
> Key: NIFI-4283
> URL: https://issues.apache.org/jira/browse/NIFI-4283
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Matt Burgess
> Assignee: Joseph Witt
> Fix For: 1.4.0
>
>
> NIFI-2583 added the ability (via dynamic properties) to specify initial Max
> Values for columns, to enable the user to "pick up where they left off" if
> something happened with a flow, a NiFi instance, etc. where the state was
> stored but the processing did not complete successfully.
> This feature would also be helpful in GenerateTableFetch, which also supports
> max-value columns.
> Since NIFI-2881 adds incoming flow file support, it's more useful if Initial
> max values can be specified via flow file attributes. Because if a table name
> is dynamically passed via flow file attribute and Expression Language, user
> won't be able to configure dynamic processor attribute in advance for each
> possible table.
> Add dynamic properties ('initial.maxvalue.<max_value_column>' same as
> QueryDatabaseTable) to specify initial max values statically, and also use
> incoming flow file attributes named 'initial.maxvalue.<max_value_column>' if
> any.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
