[
https://issues.apache.org/jira/browse/NIFI-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CHANDAN KUMAR updated NIFI-14326:
---------------------------------
Description:
Test case :
=========
sftp location which contains close to 100 files of different timestamp
Run ListSFTP with *Remote Poll Batch Size* to 10
ListSFTP is set run with *Tracking Timestamps*
*ListSFTP just listed 10 files in one go and after that it is not listing
anything*
if I look at the state of ListSFTP it has stored the file which is Not most the
recent one.
Further I keep creating New few files on SFTP but No changes on listing
it Is not listing anything beyond those 10 files.
I did stopped and started back No luck
tried clearing state and started back again same 10 files
As per document *Remote Poll Batch Size*
_The value specifies how many file paths to find in a given directory on the
remote system when doing a file listing. This value in general should not need
to be modified but when polling against a remote system with a tremendous
number of files this value can be critical. Setting this value too high can
result very poor performance and setting it too low can cause the flow to be
slower than normal._
I though it would keep listing new files with each run a batch of 10 files and
it will finish listing all of them but that is not the case.
Seems to be Bug which was introduced with
https://issues.apache.org/jira/browse/NIFI-12772
was:
Test case :
=========
sftp location which contains close to 100 files of different timestamp
Run ListSFTP with *Remote Poll Batch Size* to 10
ListSFTP is set run with *Tracking Timestamps*
*ListSFTP just listed 10 files in one go and after that it is not listing
anything*
if I look at the state of ListSFTP it has stored the file which is Not most the
recent one.
Further I keep creating New few files on SFTP but No changes on listing
it Is not listing anything beyond those 10 files.
I did stopped and started back No luck
tried clearing state and started back again same 10 files
As per document *Remote Poll Batch Size*
_The value specifies how many file paths to find in a given directory on the
remote system when doing a file listing. This value in general should not need
to be modified but when polling against a remote system with a tremendous
number of files this value can be critical. Setting this value too high can
result very poor performance and setting it too low can cause the flow to be
slower than normal._
I though it would keep listing new files with each run a batch of 10 files and
it will finish listing all of them but that is not the case.
Seems to be Bug
> ListSFTP :Remote Poll Batch : does not list files beyond configured value
> --------------------------------------------------------------------------
>
> Key: NIFI-14326
> URL: https://issues.apache.org/jira/browse/NIFI-14326
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.26.0, 1.28.1, 2.2.0
> Reporter: CHANDAN KUMAR
> Priority: Major
>
> Test case :
> =========
> sftp location which contains close to 100 files of different timestamp
> Run ListSFTP with *Remote Poll Batch Size* to 10
> ListSFTP is set run with *Tracking Timestamps*
> *ListSFTP just listed 10 files in one go and after that it is not listing
> anything*
> if I look at the state of ListSFTP it has stored the file which is Not most
> the recent one.
> Further I keep creating New few files on SFTP but No changes on listing
> it Is not listing anything beyond those 10 files.
> I did stopped and started back No luck
> tried clearing state and started back again same 10 files
> As per document *Remote Poll Batch Size*
> _The value specifies how many file paths to find in a given directory on the
> remote system when doing a file listing. This value in general should not
> need to be modified but when polling against a remote system with a
> tremendous number of files this value can be critical. Setting this value too
> high can result very poor performance and setting it too low can cause the
> flow to be slower than normal._
>
> I though it would keep listing new files with each run a batch of 10 files
> and it will finish listing all of them but that is not the case.
> Seems to be Bug which was introduced with
> https://issues.apache.org/jira/browse/NIFI-12772
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
