[ https://issues.apache.org/jira/browse/BEAM-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843658#comment-15843658 ]
Pei He commented on BEAM-1309: ------------------------------ Where is match() is called, and what is the pattern? We can probably use: http://docs.oracle.com/javase/7/docs/api/java/io/File.html#listFiles(java.io.FileFilter) We should also make sure the pattern passed in to match() is not too broad. > FileIOChannelFactory.match() traverses entire parent directory recursively > -------------------------------------------------------------------------- > > Key: BEAM-1309 > URL: https://issues.apache.org/jira/browse/BEAM-1309 > Project: Beam > Issue Type: Bug > Components: sdk-java-core > Reporter: Eugene Kirpichov > Assignee: Pei He > > I was running a pipeline that reads a single file from my local home > directory. > The pipeline got stuck, and upon taking a stack snapshot, I noticed that it > was stuck in FileIOChannelFactory.match(). > The code currently works by traversing the whole parent directory of the > requested filepattern and checking which files match the filepattern. In my > case, that means traversing everything in my home directory, which is *a lot* > (and includes remotely mounted directories). > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java#L109 > This is very wasteful and should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)