[
https://issues.apache.org/jira/browse/FLUME-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173115#comment-14173115
]
Hari Shreedharan commented on FLUME-2502:
-----------------------------------------
I like the fact that you are removing the last element - that is pretty
efficient, since that means you are not moving every element. Using an iterator
won't allow you to remove the last one without traversing the entire list. I'd
keep the current remove(size-1) code, but just do the copy if the list is not
already not an ArrayList.
> Spool source's directory listing is inefficient
> -----------------------------------------------
>
> Key: FLUME-2502
> URL: https://issues.apache.org/jira/browse/FLUME-2502
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.5.0
> Reporter: Prateek Rungta
> Attachments: FLUME-2502-0.patch, FLUME-2502-1.patch,
> FLUME-2502-2.patch
>
>
> As mentioned in
> [FLUME-2309|https://issues.apache.org/jira/browse/FLUME-2309], the directory
> listing can it self become the bottleneck when accessing directories with a
> large number of files (>1M). The fix in that JIRA added in the ability to
> specify `RANDOM` as a Consume-Order to avoid sorting large lists.
> The slowness of the directory listing is still un-addressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
