[
https://issues.apache.org/jira/browse/NIFI-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949599#comment-16949599
]
Aram S Openden edited comment on NIFI-6762 at 10/11/19 4:33 PM:
----------------------------------------------------------------
{quote}How do you plan to watch? Scan?
{quote}
Is "Watch" the wrong word perhaps? I was using the word Watch at the
suggestion of [~joewitt], as per his dev mailing list reply on 9/17/2019:
{quote}The purpose of the List* processors is 'once told to look at a thing' it
should generate a listing of what it is there AND it should continue to
look at the thing to see what changes/shows up later. Arguably we should
have called that Watch*.
If your use case is 'once told to look at a thing' it should generate a
listing of what is there at that time and not worry about later updates
happening because you can always redo a listing again later. The
difference there is subtle but important. And for this case/desire the
name 'List*' makes a lot of sense and may be why we've seen folks want it
to work this way rather than how it does.
{quote}
Perhaps I should explain the workflow in a bit more.
The workflow is similar to the existing ListS3 Processor, with the exception
that it also fires its {{on_trigger()}} behavior from an incoming FlowFile
event that tells it to looks for a bucket that matches the input expression in
the "Bucket" input attribute.
So, should we name this component something different. Perhaps this is the case
where {{ListMultipleS3Buckets}} as a name is a better description. Very much
*open to feedback* on that question.
was (Author: aramcodez):
{quote}How do you plan to watch? Scan?
{quote}
Is "Watch" the wrong word perhaps? I was using the word Watch at the
suggestion of [~joewitt], as per his dev mailing list reply on 9/17/2019:
{quote}The purpose of the List* processors is 'once told to look at a thing' it
should generate a listing of what it is there AND it should continue to
look at the thing to see what changes/shows up later. Arguably we should
have called that Watch*.
If your use case is 'once told to look at a thing' it should generate a
listing of what is there at that time and not worry about later updates
happening because you can always redo a listing again later. The
difference there is subtle but important. And for this case/desire the
name 'List*' makes a lot of sense and may be why we've seen folks want it
to work this way rather than how it does.
{quote}
Perhaps I should explain the workflow in a bit more.
The workflow is similar to the existing ListS3 Processor, with the exception
that it also fires its {{on_trigger()}} behavior from an incoming FlowFile
event that tells it to looks for a bucket that matches the input expression in
the "Bucket" input attribute.
So, should we name this component something different. Perhaps this is the case
where ListMultipleS3Buckets as a name is a better description. Very much *open
to feedback* on that question.
> New Processor - Watch Multiple S3 Buckets for content changes
> -------------------------------------------------------------
>
> Key: NIFI-6762
> URL: https://issues.apache.org/jira/browse/NIFI-6762
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Affects Versions: 1.9.2
> Reporter: Aram S Openden
> Priority: Major
> Labels: Processor, S3, watch
> Original Estimate: 384h
> Remaining Estimate: 384h
>
> Proposing New S3 processor that is a "variation" on the existing ListS3
> Processor. For lack of better term, I will call this "Watch Multiple S3
> Buckets Processor" (naming here is open to feedback). This new Processor
> takes an input attribute "Bucket" from the FlowFile, which could use NiFi
> ExpressionLanguage to compute that value. Hence, the Processor could "watch"
> different buckets in an S3 Object Store.
> This processor must handle the State Management logic for multiple S3
> buckets. The simplest solution for this problem add to modify the State
> Management logic to prepend the bucket name to this key/objectName that is
> being used to track state.
> Finally, the Dev should document the maximum number of buckets it will
> monitor at once, maximum amount of state/storage it can have to track the
> various listings in various buckets.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
