Hi, I have a directory where some other system was sending me tons of files into an SFTP server.
Now ListSFTP works GREAT on every folder everywhere on this server. But this one folder, has I think around 10 million files or something (maybe too many). It takes a long time to run "ls" on this directory like 5-6 minutes for SSH to reply back. It freezes ListSFTP processor on primary node. I end up having to restart the Coordinator node after 30 minutes or hour of waiting. Why does "ls" come back but ListSFTP struggles more and freezes thread? Is there a way to limit the amount of files ListSFTP should pull in? I'd like it if I can divide it up into chunks of 100,000. Maybe I have to run some linux commands to split it apart into 100,000-file folders on the SFTP server? I'm using Nifi 1.3.0 Thanks, <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
