[
https://issues.apache.org/jira/browse/NIFI-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-13805.
-----------------------------------
Resolution: Feedback Received
Apache NiFi 1.x is no longer maintained and no new release is planned on the
1.x release line. Marking as resolved as part of a cleanup operation. Please
open a new one with an updated description if this is still relevant for NiFi
2.x.
> ListSFTP issue - single file is listed multiple times with incorrect size
> -------------------------------------------------------------------------
>
> Key: NIFI-13805
> URL: https://issues.apache.org/jira/browse/NIFI-13805
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.18.0
> Environment: NiFi - v1.18.0
> OS - RedHat 4.8.5-44
> 3 nodes cluster with external Zookeeper
> Reporter: Artur M.
> Priority: Major
> Attachments: image-2024-09-26-15-34-24-604.png,
> image-2024-09-26-15-35-06-389.png, image-2024-09-26-15-35-43-616.png,
> image-2024-09-26-15-35-56-677.png, image-2024-09-26-15-38-04-850.png,
> image-2024-09-26-15-46-01-227.png, image-2024-09-26-15-48-02-174.png,
> image-2024-09-26-15-48-11-008.png
>
>
> We've noticed that in environment with high data load, from time to time
> files are listed multiple times with incorrect size. File upload to SFTP
> server is completed and confirmed on another VM but after successfull upload
> NiFi's ListSFTP processor see those files as incomplete. It takes few
> ListSFTP iterations for NiFI to see file's full size, in each iteration
> file's size seems to be increasing. It's quite confusing, because as I've
> described files are already fully uploaded, so NiFi should see their final
> size. In the case of another environment with a similar configuration but
> less load we don't observe this issue.
> Our current settings for ListSFTP processor are visible on following
> screenshots. Sensitive data were hide.
> !image-2024-09-26-15-34-24-604.png!
> !image-2024-09-26-15-35-06-389.png!
> !image-2024-09-26-15-35-43-616.png!
> !image-2024-09-26-15-35-56-677.png!
> Current flow is presented on following screenshot (this flow is used in
> multiple processor groups for different data).
> !image-2024-09-26-15-38-04-850.png!
> Files has declared filesize in their name, value from filename is compared in
> "Ensure File Completion" with size returned from ListSFTP processor.
> Following screenshots present files collected by queue "unmatched" with
> incorrect sizes.
> !image-2024-09-26-15-46-01-227.png!
> For files presented above, behaviour of ListSFTP is like on example below.
> File's size changes in each iteration, even though file has been fully
> uploaded and final size of 966 370 266 bytes should be returned in the first
> iteration.
> * 153 248 000 bytes
> * 294 464 000 bytes
> * 460 224 000 bytes
> * 618 592 000 bytes
> * 777 472 000 bytes
> * 959 936 000 bytes
> * 966 370 266 bytes
> In other processor groups which uses ListSFTP, situation is same as it was
> described above.
> !image-2024-09-26-15-48-02-174.png!
> !image-2024-09-26-15-48-11-008.png!
> We believe that described behaviour of ListSFTP is unexpected. Expected
> behaviour is single listing for single file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
