[
https://issues.apache.org/jira/browse/NIFI-13372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-13372:
---------------------------
Description:
The existing processor to retrieve a file from a remote system over SFTP,
namely {{FetchSFTP}} and {{{}GetSFTP{}}}, support the removal of the file from
the file system once the content has been copied into the FlowFile.
However, deleting the file from the file system immediately might not be
feasible in certain circumstances.
In cases where the content repository of NiFi does not meet sufficient data
durability guarantees, it might be desired to remove the source file only after
it has been processed successfully and its result transferred to a system that
satisfies those durability constraints.
Additionally, the integrated deletion might fail "silently", that is, the
FlowFile is still sent to the "success" relationship. This results in the file
remaining at the source. Depending on the listing behaviour it may be listed
again (in the worst case over and over). Having the deletion as an extra step
with a failure relationship in case of failure avoids this behaviour.
As of now, there is no built-in solution to achieve such behavior using the
standard NiFi distribution.
Current workarounds involve the usage of a scripted processor or the creation
of a custom processor, that provides the desired functionality.
This issue proposes the addition of a {{DeleteSFTP}} processor to the NiFi
standard-processors bundle, that fills this gap.
It should expect a FlowFile and delete the file at the path derived from the
FlowFile attributes. The default values to determine the file path should be
compatible with the attributes written by the existing {{ListSFTP}} processor.
was:
The existing processor to retrieve a file from a remote system over SFTP,
namely {{FetchSFTP}} and {{{}GetSFTP{}}}, support the removal of the file from
the file system once the content has been copied into the FlowFile.
However, deleting the file from the file system immediately might not be
feasible in certain circumstances.
In cases where the content repository of NiFi does not meet sufficient data
durability guarantees, it might be desired to remove the source file only after
it has been processed successfully and its result transferred to a system that
satisfies those durability constraints.
As of now, there is no built-in solution to achieve such behavior using the
standard NiFi distribution.
Current workarounds involve the usage of a scripted processor or the creation
of a custom processor, that provides the desired functionality.
This issue proposes the addition of a {{DeleteSFTP}} processor to the NiFi
standard-processors bundle, that fills this gap.
It should expect a FlowFile and delete the file at the path derived from the
FlowFile attributes. The default values to determine the file path should be
compatible with the attributes written by the existing {{ListSFTP}} processor.
> Add DeleteSFTP processor
> ------------------------
>
> Key: NIFI-13372
> URL: https://issues.apache.org/jira/browse/NIFI-13372
> Project: Apache NiFi
> Issue Type: New Feature
> Reporter: endzeit
> Assignee: endzeit
> Priority: Major
>
> The existing processor to retrieve a file from a remote system over SFTP,
> namely {{FetchSFTP}} and {{{}GetSFTP{}}}, support the removal of the file
> from the file system once the content has been copied into the FlowFile.
> However, deleting the file from the file system immediately might not be
> feasible in certain circumstances.
> In cases where the content repository of NiFi does not meet sufficient data
> durability guarantees, it might be desired to remove the source file only
> after it has been processed successfully and its result transferred to a
> system that satisfies those durability constraints.
> Additionally, the integrated deletion might fail "silently", that is, the
> FlowFile is still sent to the "success" relationship. This results in the
> file remaining at the source. Depending on the listing behaviour it may be
> listed again (in the worst case over and over). Having the deletion as an
> extra step with a failure relationship in case of failure avoids this
> behaviour.
> As of now, there is no built-in solution to achieve such behavior using the
> standard NiFi distribution.
> Current workarounds involve the usage of a scripted processor or the creation
> of a custom processor, that provides the desired functionality.
> This issue proposes the addition of a {{DeleteSFTP}} processor to the NiFi
> standard-processors bundle, that fills this gap.
> It should expect a FlowFile and delete the file at the path derived from the
> FlowFile attributes. The default values to determine the file path should be
> compatible with the attributes written by the existing {{ListSFTP}} processor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
