Artur M. created NIFI-13805:
-------------------------------

             Summary: ListSFTP issue - single file is listed multiple times 
with incorrect size
                 Key: NIFI-13805
                 URL: https://issues.apache.org/jira/browse/NIFI-13805
             Project: Apache NiFi
          Issue Type: Bug
    Affects Versions: 1.18.0
         Environment: NiFi - v1.18.0
OS - RedHat 4.8.5-44
3 nodes cluster with external Zookeeper
            Reporter: Artur M.
         Attachments: image-2024-09-26-15-34-24-604.png, 
image-2024-09-26-15-35-06-389.png, image-2024-09-26-15-35-43-616.png, 
image-2024-09-26-15-35-56-677.png, image-2024-09-26-15-38-04-850.png, 
image-2024-09-26-15-46-01-227.png, image-2024-09-26-15-48-02-174.png, 
image-2024-09-26-15-48-11-008.png

We've noticed that in environment with high data load, from time to time files 
are listed multiple times with incorrect size. File upload to SFTP server is 
completed and confirmed on another VM but after successfull upload NiFi's 
ListSFTP processor see those files as incomplete. It takes few ListSFTP 
iterations for NiFI to see file's full size, in each iteration file's size 
seems to be increasing. It's quite confusing, because as I've described files 
are already fully uploaded, so NiFi should see their final size. In the case of 
another environment with a similar configuration but less load we don't observe 
this issue.


Our current settings for ListSFTP processor are visible on following 
screenshots. Sensitive data were hide.

!image-2024-09-26-15-34-24-604.png!

!image-2024-09-26-15-35-06-389.png!

!image-2024-09-26-15-35-43-616.png!

!image-2024-09-26-15-35-56-677.png!

Current flow is presented on following screenshot (this flow is used in 
multiple processor groups for different data).

!image-2024-09-26-15-38-04-850.png!

Files has declared filesize in their name, value from filename is compared in 
"Ensure File Completion" with size returned from ListSFTP processor. 

Following screenshots present files collected by queue "unmatched" with 
incorrect sizes.
!image-2024-09-26-15-46-01-227.png!

For files presented above, behaviour of ListSFTP is like on example below. 
File's size changes in each iteration, even though file has been fully uploaded 
and final size of 966 370 266 bytes should be returned in the first iteration.
 * 153 248 000 bytes
 * 294 464 000 bytes
 * 460 224 000 bytes
 * 618 592 000 bytes
 * 777 472 000 bytes
 * 959 936 000 bytes
 * 966 370 266 bytes


In other processor groups which uses ListSFTP, situation is same as it was 
described above.

!image-2024-09-26-15-48-02-174.png!

!image-2024-09-26-15-48-11-008.png!

We believe that described behaviour of ListSFTP is unexpected. Expected 
behaviour is single listing for single file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to