[ https://issues.apache.org/jira/browse/BEAM-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092203#comment-16092203 ]
ASF GitHub Bot commented on BEAM-2628: -------------------------------------- GitHub user jkff opened a pull request: https://github.com/apache/beam/pull/3590 [BEAM-2628] Makes AvroSource not open files while splitting https://issues.apache.org/jira/browse/BEAM-2628 AvroSource would open each file while splitting, in createForSubrangeOfFile, in order to fetch the file's codec, sync marker and writer schema. This is extremely slow with a large number of files, and unnecessary - none of this information is needed for splitting, only for reading. This PR moves that code into the reader: now, when we open an AvroReader, we first read the file metadata. This effectively moves the overhead of opening every file from {sequentially during splitting} to {in parallel inside every work item}, which is much better and gets rid of the sequential bottleneck in splitting. There's a few more minor simplifications (e.g. removing unnecessary lazy initialization and such) and immaterial renames to AvroSource in this PR. AvroSource is not a public API surface class (is marked Experimental), so this is fine. R: @reuvenlax CC: @matthewlang You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam avro-split-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3590.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3590 ---- commit 4973caaa57b9959944d503de1f33bd7d7c64b027 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2017-07-18T20:40:52Z Minor changes to AvroSource in preparation for refactoring commit 2bfb3d50503627308d431909bac0fe768ee8d688 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2017-07-18T21:09:03Z Gets rid of opening Avro files in createForSubrangeOfFile codepath ---- > AvroSource.split() sequentially opens every matched file > -------------------------------------------------------- > > Key: BEAM-2628 > URL: https://issues.apache.org/jira/browse/BEAM-2628 > Project: Beam > Issue Type: Bug > Components: sdk-java-core > Reporter: Eugene Kirpichov > Assignee: Eugene Kirpichov > > When you do AvroIO.read().from(filepattern), during splitting of AvroSource > the filepattern gets expanded into N files, and then for each of the N files > we do this: > https://github.com/apache/beam/blob/v2.0.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java#L259 > This is very slow. E.g. one job was reading 15,000 files, and it took almost > 2 hours to split the source because opening each file and reading schema was > taking about 0.5s. > I'm not quite sure why we need the file metadata while splitting... -- This message was sent by Atlassian JIRA (v6.4.14#64029)