Dennis Jaheruddin created NIFI-7971:
---------------------------------------
Summary: Create List-Fetch-Delete pattern
Key: NIFI-7971
URL: https://issues.apache.org/jira/browse/NIFI-7971
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.12.1
Reporter: Dennis Jaheruddin
*Challenge*
There are several processors for getting data. Originally we had GetHDFS,
GetSFTP and these are now being phased out for List Fetch processors.
I think the List-Fetch pattern is a significant improvement, but what I see now
is that people are looking for a List-Fetch-Delete pattern.
*Example usecase:*
Gather data, process and write in destination, delete
There are several ways to achieve this (native but inefficient ones, like 'do a
second Fetch with remove source file and terminate success, or efficient but
scripted ones, like 'use the executestreamprocessor to send the delete
command').
As far as I understand, this would not be needed when using Stateless Nifi, but
that is not relevant for many usecases.
*Proposal*
However, I propose we find a solution that allows people to easily achieve a
List-Fetch-Delete pattern.
Two lines of thought:
# Expand Fetch processors with a 'deleteonly' mode, which would not actually
copy in the data
# Create dedicated processors, like 'DeleteSFTP' or 'ExecuteSFTP'
I have written the example for SFTP, but think this would be a natural
extension for most places where we have the List-Fetch pattern currently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
