ihji commented on a change in pull request #11205: [BEAM-9578] Enumerating
artifacts is too expensive in Java
URL: https://github.com/apache/beam/pull/11205#discussion_r404411951
##########
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##########
@@ -214,24 +220,90 @@ public static Environment createProcessEnvironment(
pathsToStage.addAll(stagingFiles);
}
- ImmutableList.Builder<ArtifactInformation> filesToStage =
ImmutableList.builder();
+ ImmutableList.Builder<Supplier<ArtifactInformation>> lazyArtifactsBuilder =
+ ImmutableList.builder();
for (String path : pathsToStage) {
Review comment:
This for loop is fairly cheap and from stream of Suppliers we can easily get
additional performance benefits by creating parallelStream. When we consider
parallelizing expensive computations, some boilerplate codes are needed anyway
in `getNonDeferredArtifacts()`. I think building a stream is a nice way to
abstract them out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
