Jens M Kofoed created NIFI-7276:
-----------------------------------
Summary: PutFTP create alot of received trafic when tx small files
Key: NIFI-7276
URL: https://issues.apache.org/jira/browse/NIFI-7276
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.11.4
Environment: NIFI: Ubuntu 18.04. NIFI 1.11.4
FTP: Ubuntu 18.04 with vsftpd
Reporter: Jens M Kofoed
I don't know if this is a bug, but the result of my investigation is that it
seems like the PutFTP receive between 1,6 - 2,7MB of data per file it transmit.
For me it seems like a lot of RX data.
I'm using nload and nethogs to monitor the network trafic.
I have a lot (100.000+) of small files (<1MB), which I can't combine.
When starting the PutFTP process the nifi server starts receiving data from the
ftp server and it continues with with an avg. of 50 Mbit/s, constantly.
To increasing the throughput I've set the run duration to 2sec. and playing
with the Batch size. With a batch size of 100 and over 5 min. I can see at the
ftp server that nifi is logging in 5 times and start a vsftpd process. within
those 5 min. nifi has tx 700 files and 65.32 MB, but the ftp server has sent
1544MB data back to nifi.
With a batch size of 1000. It took nifi around 8 minutes for transferring 1000
files with a total sum of 64MB, but the ftp server has this time TX 2700MB data
back to nifi.
I tried with larger files and a batch size at 100. NIFI sent 249 files with at
total sum of 32 GB of data within 3½ min. and this time the ftp server sent
400MB data back to nifi.
It seems like the PutFTP receive between 1,6 - 2,7MB of data per file it
transmit. For me it seems like alot of RX data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
