[
https://issues.apache.org/jira/browse/APEXMALHAR-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Zhang reassigned APEXMALHAR-2274:
--------------------------------------
Assignee: Matt Zhang
> AbstractFileInputOperator gets killed when there are a large number of files.
> -----------------------------------------------------------------------------
>
> Key: APEXMALHAR-2274
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2274
> Project: Apache Apex Malhar
> Issue Type: Bug
> Reporter: Munagala V. Ramanath
> Assignee: Matt Zhang
>
> When there are a large number of files in the monitored directory, the call
> to DirectoryScanner.scan() can take a long time since it calls
> FileSystem.listStatus() which returns the entire list. Meanwhile, the
> AppMaster deems this operator hung and restarts it which again results in the
> same problem.
> It should use FileSystem.listStatusIterator() to limit the number files
> processed in a single call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
