[
https://issues.apache.org/jira/browse/BEAM-9056?focusedWorklogId=393774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-393774
]
ASF GitHub Bot logged work on BEAM-9056:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Feb/20 21:59
Start Date: 26/Feb/20 21:59
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #10621:
[BEAM-9056] Staging artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r384675886
##########
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##########
@@ -175,6 +197,90 @@ public static Environment createProcessEnvironment(
}
}
+ public static Collection<ArtifactInformation> getArtifacts(PipelineOptions
options) {
+ Set<String> pathsToStage = Sets.newHashSet();
+ // TODO(heejong): remove jar_packages experimental flag when
cross-language dependency
+ // management is implemented for all runners.
+ List<String> experiments =
options.as(ExperimentalOptions.class).getExperiments();
+ if (experiments != null) {
+ Optional<String> jarPackages =
+ experiments.stream()
+ .filter((String flag) -> flag.startsWith("jar_packages="))
+ .findFirst();
+ jarPackages.ifPresent(
+ s ->
pathsToStage.addAll(Arrays.asList(s.replaceFirst("jar_packages=",
"").split(","))));
+ }
+ List<String> stagingFiles =
options.as(PortablePipelineOptions.class).getFilesToStage();
+ if (stagingFiles == null) {
+ pathsToStage.addAll(
+ detectClassPathResourcesToStage(Environments.class.getClassLoader(),
options));
+ if (pathsToStage.isEmpty()) {
+ throw new IllegalArgumentException("No classpath elements found.");
+ }
+ LOG.debug(
+ "PortablePipelineOptions.filesToStage was not specified. "
+ + "Defaulting to files from the classpath: {}",
+ pathsToStage.size());
+ } else {
+ pathsToStage.addAll(stagingFiles);
+ }
+
+ ImmutableList.Builder<ArtifactInformation> filesToStage =
ImmutableList.builder();
+ for (String path : pathsToStage) {
+ File file = new File(path);
+ if (new File(path).exists()) {
+ // Spurious items get added to the classpath. Filter by just those
that exist.
+ if (file.isDirectory()) {
+ // Zip up directories so we can upload them to the artifact service.
Review comment:
Is this the behavior expected by Beam runners (zipping up directories) ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 393774)
Time Spent: 3h (was: 2h 50m)
> Staging artifacts from environment
> ----------------------------------
>
> Key: BEAM-9056
> URL: https://issues.apache.org/jira/browse/BEAM-9056
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
> Reporter: Heejong Lee
> Assignee: Heejong Lee
> Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> staging artifacts from artifact information embedded in environment proto.
> detail:
> https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
