[
https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=316846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316846
]
ASF GitHub Bot logged work on BEAM-8240:
----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Sep/19 17:22
Start Date: 23/Sep/19 17:22
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9629: [BEAM-8240]
Sets workerHarnessContaienrImage in the default Environment of DataflowRunner
URL: https://github.com/apache/beam/pull/9629#discussion_r327213316
##########
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
##########
@@ -1319,6 +1322,60 @@ public void testTemplateRunnerFullCompletion() throws
Exception {
expectedLogs.verifyInfo("Template successfully created");
}
+ /**
+ * Tests that when {@link
DataflowPipelineOptions#setWorkerHarnessContainerImage(String)} pipeline
+ * option is set, {@link DataflowRunner} sets that value as the {@link
+ * DockerPayload#getContainerImage()} of the default {@link Environment}
used when generating the
+ * model pipeline proto.
+ */
+ @Test
+ public void testSetWorkerHarnessContainerImageInPipelineProto() throws
Exception {
+ File existingFile = tmpFolder.newFile();
+ DataflowPipelineOptions options =
PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+
+ String containerImage = "gcr.io/IMAGE/foo";
+
options.as(DataflowPipelineOptions.class).setWorkerHarnessContainerImage(containerImage);
+ options.setJobName("TestJobName");
+ options.setGcpCredential(new TestCredential());
+ options.setPathValidatorClass(NoopPathValidator.class);
+ options.setProject("test-project");
+ options.setRunner(DataflowRunner.class);
+
+ // Setting a template location so that pipeline does not actually try to
execute on Dataflow
+ // service.
+ options.setTemplateLocation(existingFile.getPath());
+
+ options.setTempLocation(tmpFolder.getRoot().getPath());
+ Pipeline p = Pipeline.create(options);
+
+ p.run();
+
+ File stagingDirectory =
+ new
File(options.as(DataflowPipelineOptions.class).getStagingLocation());
+ assertTrue(stagingDirectory.isDirectory());
+ File[] files = stagingDirectory.listFiles();
+
+ File protoFile = null;
+ for (File file : files) {
+ String filePath = file.getAbsolutePath();
+ if (filePath.contains("pipeline") && filePath.endsWith("pb")) {
+ assertTrue("Cannot uniquely identify the proto pipeline.", (protoFile
== null));
+ protoFile = file;
+ }
+ }
+
+ byte[] protoBytes = Files.readAllBytes(protoFile.toPath());
+ RunnerApi.Pipeline pipeline = RunnerApi.Pipeline.parseFrom(protoBytes);
+
+ assertEquals(1, pipeline.getComponents().getEnvironmentsCount());
+ assertEquals(1, pipeline.getComponents().getEnvironmentsCount());
+ Environment defaultEnvironment =
+
pipeline.getComponents().getEnvironmentsMap().values().iterator().next();
Review comment:
```suggestion
Iterables.getOnlyElement(pipeline.getComponents().getEnvironmentsMap().values());
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 316846)
Time Spent: 4h 10m (was: 4h)
> Fix pipeline proto to contain worker_harness_container_image override
> ---------------------------------------------------------------------
>
> Key: BEAM-8240
> URL: https://issues.apache.org/jira/browse/BEAM-8240
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
> Reporter: Luke Cwik
> Assignee: Luke Cwik
> Priority: Minor
> Fix For: 2.17.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> SDK harness incorrectly identifies itself when using custom SDK container
> within environment field when building pipeline proto.
>
> Passing in the experiment *worker_harness_container_image=YYY* doesn't
> override the pipeline proto environment field and it is still being populated
> with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802*
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
