Munagala V. Ramanath created APEXMALHAR-2274:
------------------------------------------------
Summary: AbstractFileInputOperator gets killed when there are a
large number of files.
Key: APEXMALHAR-2274
URL: https://issues.apache.org/jira/browse/APEXMALHAR-2274
Project: Apache Apex Malhar
Issue Type: Bug
Reporter: Munagala V. Ramanath
When there are a large number of files in the monitored directory, the call to
DirectoryScanner.scan() can take a long time since it calls
FileSystem.listStatus() which returns the entire list. Meanwhile, the AppMaster
deems this operator hung and restarts it which again results in the same
problem.
It should use FileSystem.listStatusIterator() to limit the number files
processed in a single call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
