[
https://issues.apache.org/jira/browse/NIFI-12900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12900:
---------------------------
Description:
The processor {{PutSFTP}} is based on {{PutFileTransfer}}.
Before an actual upload takes place, potential conflicts (e.g. existing file)
are identified and resolved using {{identifyAndResolveConflictFile(...)}}.
As part of this process, information on the target file is retrieved using
{{FileTransfer.getRemoteFileInfo(...)}}.
In case of {{PutSFTP}} this is implemented by {{SFTPTransfer}}.
The implementation of {{getRemoteFileInfo}} executes {{ls}} on the target
directory path. In case there are a lot of files inside the remote directory,
e.g. >10.000 files, the listing reduces the performance of {{PutSFTP}}
significantly.
Instead of a listing on the directory, file information should be retrieved
using either {{ls}} or {{stat}} on the target file directly.
was:
The processor `PutSFTP` is based on `PutFileTransfer`.
Before an actual upload takes place, potential conflicts (e.g. existing file)
are identified and resolved using `identifyAndResolveConflictFile(...)`.
As part of this process, information on the target file is retrieved using
`FileTransfer.getRemoteFileInfo(...)`.
In case of `PutSFTP` this is implemented by `SFTPTransfer`.
The implementation of `getRemoteFileInfo` executes `ls` on the target directory
path. In case there are a lot of files inside the remote directory, e.g.
>10.000 files, the listing reduces the performance of `PutSFTP` significantly.
Instead of a listing on the directory, file information should be retrieved
using either `ls` or `stat` on the target file directly.
> Avoid unnecessary file listing in PutSFTP
> ------------------------------------------
>
> Key: NIFI-12900
> URL: https://issues.apache.org/jira/browse/NIFI-12900
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: endzeit
> Assignee: endzeit
> Priority: Major
>
> The processor {{PutSFTP}} is based on {{PutFileTransfer}}.
> Before an actual upload takes place, potential conflicts (e.g. existing file)
> are identified and resolved using {{identifyAndResolveConflictFile(...)}}.
> As part of this process, information on the target file is retrieved using
> {{FileTransfer.getRemoteFileInfo(...)}}.
> In case of {{PutSFTP}} this is implemented by {{SFTPTransfer}}.
> The implementation of {{getRemoteFileInfo}} executes {{ls}} on the target
> directory path. In case there are a lot of files inside the remote directory,
> e.g. >10.000 files, the listing reduces the performance of {{PutSFTP}}
> significantly.
> Instead of a listing on the directory, file information should be retrieved
> using either {{ls}} or {{stat}} on the target file directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
