lukecwik commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3 URL: https://github.com/apache/beam/pull/11037#issuecomment-596669808 I understand what your saying and how your solution resolves the problem. I'm trying to say that the problem shouldn't occur in the first place because [AvroIO.read](https://github.com/apache/beam/blob/a4ab76881d843a17906d866f0d6225def82ea6d0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L1030) should expand to [MatchAll](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L622) that contains a [Reshuffle](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L641) followed by [ReadMatches](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L690) Reshuffle should ensure that there is either a repartition between MatchAll and ReadMatches, is it missing? If it isn't missing, they why is the following stage only executing on a single machine?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
