[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=435024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-435024 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 15:39 Start Date: 19/May/20 15:39 Worklog Time Spent: 10m Work Description: chamikaramj merged pull request #11039: URL: https://github.com/apache/beam/pull/11039 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 435024) Time Spent: 12h (was: 11h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434982 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 14:45 Start Date: 19/May/20 14:45 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630867766 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434982) Time Spent: 11h 50m (was: 11h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 11h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434981 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 14:44 Start Date: 19/May/20 14:44 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630867540 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434981) Time Spent: 11h 40m (was: 11.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 11h 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434820 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 07:46 Start Date: 19/May/20 07:46 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630647249 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434820) Time Spent: 11.5h (was: 11h 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 11.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434819 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 07:44 Start Date: 19/May/20 07:44 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630646157 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434819) Time Spent: 11h 20m (was: 11h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 11h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434808 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 07:17 Start Date: 19/May/20 07:17 Worklog Time Spent: 10m Work Description: ihji commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630632776 Rebased and fixed conflict. Please rerun the test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434808) Time Spent: 11h 10m (was: 11h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 11h 10m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434805 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 07:16 Start Date: 19/May/20 07:16 Worklog Time Spent: 10m Work Description: ihji commented on a change in pull request #11039: URL: https://github.com/apache/beam/pull/11039#discussion_r427079698 ## File path: runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java ## @@ -53,9 +53,12 @@ import org.apache.beam.model.pipeline.v1.RunnerApi.DockerPayload; Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434805) Time Spent: 11h (was: 10h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 11h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434791 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 05:13 Start Date: 19/May/20 05:13 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630584744 Heejong, can you please resolve conflicts and push an update to rerun tests ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434791) Time Spent: 10h 50m (was: 10h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 10h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434725=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434725 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/May/20 01:12 Start Date: 19/May/20 01:12 Worklog Time Spent: 10m Work Description: lukecwik commented on a change in pull request #11039: URL: https://github.com/apache/beam/pull/11039#discussion_r426974548 ## File path: runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java ## @@ -53,9 +53,12 @@ import org.apache.beam.model.pipeline.v1.RunnerApi.DockerPayload; Review comment: Shouldn't we have a test to show the artifacts were properly set? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434725) Time Spent: 10h 40m (was: 10.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434722=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434722
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 19/May/20 01:11
Start Date: 19/May/20 01:11
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r426973945
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -784,7 +877,25 @@ public DataflowPipelineJob run(Pipeline pipeline) {
"Executing pipeline on the Dataflow Service, which will have billing
implications "
+ "related to Google Compute Engine usage and other Google Cloud
Services.");
-List packages = options.getStager().stageDefaultFiles();
+// Capture the sdkComponents for look up during step translations
+SdkComponents sdkComponents = SdkComponents.create();
+
+DataflowPipelineOptions dataflowOptions =
options.as(DataflowPipelineOptions.class);
+String workerHarnessContainerImageURL =
DataflowRunner.getContainerImageForJob(dataflowOptions);
+RunnerApi.Environment defaultEnvironmentForDataflow =
+Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+
+sdkComponents.registerEnvironment(
+defaultEnvironmentForDataflow
+.toBuilder()
+.addAllDependencies(getDefaultArtifacts())
Review comment:
We also need to make sure we have the capabilities set, I have this PR:
https://github.com/apache/beam/pull/11748 since it was missing before.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434722)
Time Spent: 10.5h (was: 10h 20m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: P0
> Fix For: 2.22.0
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434721
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 19/May/20 01:10
Start Date: 19/May/20 01:10
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r426973945
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -784,7 +877,25 @@ public DataflowPipelineJob run(Pipeline pipeline) {
"Executing pipeline on the Dataflow Service, which will have billing
implications "
+ "related to Google Compute Engine usage and other Google Cloud
Services.");
-List packages = options.getStager().stageDefaultFiles();
+// Capture the sdkComponents for look up during step translations
+SdkComponents sdkComponents = SdkComponents.create();
+
+DataflowPipelineOptions dataflowOptions =
options.as(DataflowPipelineOptions.class);
+String workerHarnessContainerImageURL =
DataflowRunner.getContainerImageForJob(dataflowOptions);
+RunnerApi.Environment defaultEnvironmentForDataflow =
+Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+
+sdkComponents.registerEnvironment(
+defaultEnvironmentForDataflow
+.toBuilder()
+.addAllDependencies(getDefaultArtifacts())
Review comment:
We also need to make sure we have the capabilities set, I have this PR:
https://github.com/apache/beam/pull/11748
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434721)
Time Spent: 10h 20m (was: 10h 10m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: P0
> Fix For: 2.22.0
>
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434717
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 19/May/20 01:03
Start Date: 19/May/20 01:03
Worklog Time Spent: 10m
Work Description: ihji commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r426971999
##
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/PackageUtilTest.java
##
@@ -195,7 +187,7 @@ public void testFileWithExtensionPackageNamingAndSize()
throws Exception {
PackageAttributes attr = makePackageAttributes(tmpFile, null);
DataflowPackage target = attr.getDestination();
-assertThat(target.getName(), RegexMatcher.matches("file-" + HASH_PATTERN +
".txt"));
+assertThat(target.getName(), RegexMatcher.matches(UUID_PATTERN + ".txt"));
Review comment:
Hmm. You're right. UUID is implementation detail behind the uniqueness
guarantee. I changed the tests to only check whether it keeps the same
extension.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434717)
Time Spent: 10h 10m (was: 10h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: P0
> Fix For: 2.22.0
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434669 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 18/May/20 23:28 Start Date: 18/May/20 23:28 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630487229 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434669) Time Spent: 10h (was: 9h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 10h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434646=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434646
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 18/May/20 22:29
Start Date: 18/May/20 22:29
Worklog Time Spent: 10m
Work Description: robertwb commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r426926720
##
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/PackageUtilTest.java
##
@@ -195,7 +187,7 @@ public void testFileWithExtensionPackageNamingAndSize()
throws Exception {
PackageAttributes attr = makePackageAttributes(tmpFile, null);
DataflowPackage target = attr.getDestination();
-assertThat(target.getName(), RegexMatcher.matches("file-" + HASH_PATTERN +
".txt"));
+assertThat(target.getName(), RegexMatcher.matches(UUID_PATTERN + ".txt"));
Review comment:
Is it important, for the purposes of this test (or Dataflow in general)
that the staged file name is in the form of a UUID? If not (and I don't think
it is) it's better not to test for it. (A test that might be good to add is to
see if two same-named files in different directories actually get staged to
different places, which is the underlying, important intent.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434646)
Time Spent: 9h 50m (was: 9h 40m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: P0
> Fix For: 2.22.0
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434644
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 18/May/20 22:24
Start Date: 18/May/20 22:24
Worklog Time Spent: 10m
Work Description: ihji commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r426924859
##
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/PackageUtilTest.java
##
@@ -195,7 +187,7 @@ public void testFileWithExtensionPackageNamingAndSize()
throws Exception {
PackageAttributes attr = makePackageAttributes(tmpFile, null);
DataflowPackage target = attr.getDestination();
-assertThat(target.getName(), RegexMatcher.matches("file-" + HASH_PATTERN +
".txt"));
+assertThat(target.getName(), RegexMatcher.matches(UUID_PATTERN + ".txt"));
Review comment:
but it only checks whether the staged file name has the same extension
(vs. checks whether the staged file name is in the form of UUID with the same
extension)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434644)
Time Spent: 9h 40m (was: 9.5h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: P0
> Fix For: 2.22.0
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434642=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434642 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 18/May/20 22:18 Start Date: 18/May/20 22:18 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630465362 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434642) Time Spent: 9h 20m (was: 9h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 9h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434643 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 18/May/20 22:18 Start Date: 18/May/20 22:18 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-630465449 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434643) Time Spent: 9.5h (was: 9h 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=434607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434607
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 18/May/20 20:55
Start Date: 18/May/20 20:55
Worklog Time Spent: 10m
Work Description: robertwb commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r426876833
##
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/PackageUtilTest.java
##
@@ -195,7 +187,7 @@ public void testFileWithExtensionPackageNamingAndSize()
throws Exception {
PackageAttributes attr = makePackageAttributes(tmpFile, null);
DataflowPackage target = attr.getDestination();
-assertThat(target.getName(), RegexMatcher.matches("file-" + HASH_PATTERN +
".txt"));
+assertThat(target.getName(), RegexMatcher.matches(UUID_PATTERN + ".txt"));
Review comment:
You could just match `".*.txt"` here, rather than hard-coding the uuid
format. (Same blow.)
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -784,7 +877,25 @@ public DataflowPipelineJob run(Pipeline pipeline) {
"Executing pipeline on the Dataflow Service, which will have billing
implications "
+ "related to Google Compute Engine usage and other Google Cloud
Services.");
-List packages = options.getStager().stageDefaultFiles();
+// Capture the sdkComponents for look up during step translations
+SdkComponents sdkComponents = SdkComponents.create();
+
+DataflowPipelineOptions dataflowOptions =
options.as(DataflowPipelineOptions.class);
+String workerHarnessContainerImageURL =
DataflowRunner.getContainerImageForJob(dataflowOptions);
+RunnerApi.Environment defaultEnvironmentForDataflow =
+Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+
+sdkComponents.registerEnvironment(
+defaultEnvironmentForDataflow
+.toBuilder()
+.addAllDependencies(getDefaultArtifacts())
+.build());
+
+RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline,
sdkComponents, true);
+
+LOG.debug("Portable pipeline proto:\n{}",
TextFormat.printToString(pipelineProto));
Review comment:
OK, we don't have to change this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434607)
Time Spent: 9h 10m (was: 9h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: P0
> Fix For: 2.22.0
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=433951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433951 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 15/May/20 22:05 Start Date: 15/May/20 22:05 Worklog Time Spent: 10m Work Description: chamikaramj edited a comment on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-629519752 Yes, I think we need this. Heejong, can you make the JIRA a blocker for beam 2.22 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 433951) Time Spent: 9h (was: 8h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=433950=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433950 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 15/May/20 22:04 Start Date: 15/May/20 22:04 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-629519752 Yes, I think we need this. Heejong, can you make the jar a blocker for beam 2.22 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 433950) Time Spent: 8h 50m (was: 8h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=433806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433806 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 15/May/20 17:49 Start Date: 15/May/20 17:49 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-629394803 @ihji Just thinking about timing, were we hoping to get this into 2.22? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 433806) Time Spent: 8h 40m (was: 8.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=431931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-431931 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 07/May/20 23:24 Start Date: 07/May/20 23:24 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-625546290 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 431931) Time Spent: 8.5h (was: 8h 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=431876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-431876 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 07/May/20 20:01 Start Date: 07/May/20 20:01 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-625467377 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 431876) Time Spent: 8h 20m (was: 8h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=431582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-431582
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 07/May/20 06:38
Start Date: 07/May/20 06:38
Worklog Time Spent: 10m
Work Description: ihji commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r421271157
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
##
@@ -442,45 +448,56 @@ public static StagingResult uploaded(PackageAttributes
attributes) {
/** Holds the metadata necessary to stage a file or confirm that a staged
file has not changed. */
@AutoValue
abstract static class PackageAttributes {
-
-public static PackageAttributes forFileToStage(File source, String
stagingPath)
+public static PackageAttributes forFileToStage(File file, String
stagingPath)
throws IOException {
+ return forFileToStage(file.getPath(), null, stagingPath);
+}
+public static PackageAttributes forFileToStage(
Review comment:
I think we can remove those hashing and zipping. It's redundant except
test cases. Will update the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 431582)
Time Spent: 8h 10m (was: 8h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=430499=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430499 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 05/May/20 03:11 Start Date: 05/May/20 03:11 Worklog Time Spent: 10m Work Description: ihji commented on a change in pull request #11039: URL: https://github.com/apache/beam/pull/11039#discussion_r419841344 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -462,7 +462,8 @@ def run_pipeline(self, pipeline, options): use_fnapi = apiclient._use_fnapi(options) from apache_beam.transforms import environments default_environment = environments.DockerEnvironment.from_container_image( -apiclient.get_container_image_from_options(options)) +apiclient.get_container_image_from_options(options), +artifacts=environments.python_sdk_dependencies(options)) Review comment: We need pipeline option to populate artifacts. I think we could either use `from_options` instead and override container image or just leave as is. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 430499) Time Spent: 8h (was: 7h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=430498=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430498 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 05/May/20 03:04 Start Date: 05/May/20 03:04 Worklog Time Spent: 10m Work Description: ihji commented on a change in pull request #11039: URL: https://github.com/apache/beam/pull/11039#discussion_r419839927 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py ## @@ -102,7 +102,8 @@ def setUp(self): '--staging_location=ignored', '--temp_location=/dev/null', '--no_auth', -'--dry_run=True' +'--dry_run=True', +'--sdk_location=container' Review comment: The test tries to download a dev version of apache-beam dependency (which indeed does not exist in pypi) when it constructs the environment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 430498) Time Spent: 7h 50m (was: 7h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=430497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430497
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 05/May/20 02:57
Start Date: 05/May/20 02:57
Worklog Time Spent: 10m
Work Description: ihji commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r419838456
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
##
@@ -336,25 +323,26 @@ public DataflowPackage stageToFile(
final AtomicInteger numCached = new AtomicInteger(0);
List> destinationPackages = new
ArrayList<>();
-for (String classpathElement : classpathElements) {
- DataflowPackage sourcePackage = new DataflowPackage();
- if (classpathElement.contains("=")) {
-String[] components = classpathElement.split("=", 2);
-sourcePackage.setName(components[0]);
-sourcePackage.setLocation(components[1]);
- } else {
-sourcePackage.setName(null);
-sourcePackage.setLocation(classpathElement);
+for (StagedFile classpathElement : classpathElements) {
+ DataflowPackage targetPackage = classpathElement.getStagedPackage();
+ String source = classpathElement.getSource();
+ if (source.contains("=")) {
Review comment:
removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 430497)
Time Spent: 7h 40m (was: 7.5h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=430496=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430496
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 05/May/20 02:56
Start Date: 05/May/20 02:56
Worklog Time Spent: 10m
Work Description: ihji commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r419838363
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1271,6 +1271,11 @@ message DeferredArtifactPayload {
message ArtifactStagingToRolePayload {
// A generated staged name (relative path under staging directory).
string staged_name = 1;
+
+ // (Optional) An artifact name when a runner supports it.
+ // For example, DataflowRunner requires predefined names for some artifacts
+ // such as "dataflow-worker.jar", "windmill_main".
+ string alias_name = 2;
Review comment:
This is Dataflow specific requirement. `DataflowPackage` model has two
separate fields for `location` and `name`. `staged_name` and `alias_name`
correspond to `location` and `name` respectively.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 430496)
Time Spent: 7.5h (was: 7h 20m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=430494=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430494
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 05/May/20 02:27
Start Date: 05/May/20 02:27
Worklog Time Spent: 10m
Work Description: ihji commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r419832339
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -772,6 +783,88 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
+ImmutableList.Builder filesToStageBuilder =
ImmutableList.builder();
+for (Map.Entry entry :
+pipeline.getComponents().getEnvironmentsMap().entrySet()) {
+ for (RunnerApi.ArtifactInformation info :
entry.getValue().getDependenciesList()) {
+if
(!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn()))
{
+ throw new RuntimeException(
+ String.format("unsupported artifact type %s",
info.getTypeUrn()));
+}
+RunnerApi.ArtifactFilePayload filePayload;
+try {
+ filePayload =
RunnerApi.ArtifactFilePayload.parseFrom(info.getTypePayload());
+} catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException("Error parsing artifact file payload.",
e);
+}
+if (!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Roles.STAGING_TO)
+.equals(info.getRoleUrn())) {
+ throw new RuntimeException(
+ String.format("unsupported artifact role %s",
info.getRoleUrn()));
+}
+RunnerApi.ArtifactStagingToRolePayload stagingPayload;
+try {
+ stagingPayload =
RunnerApi.ArtifactStagingToRolePayload.parseFrom(info.getRolePayload());
+} catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException("Error parsing artifact staging_to role
payload.", e);
+}
+DataflowPackage target = new DataflowPackage();
+target.setLocation(stagingPayload.getStagedName());
+if (!Strings.isNullOrEmpty(stagingPayload.getAliasName())) {
+ target.setName(stagingPayload.getAliasName());
+}
+filesToStageBuilder.add(StagedFile.of(filePayload.getPath(), target));
+ }
+}
+return options.getStager().stageFiles(filesToStageBuilder.build());
+ }
+
+ private List getDefaultArtifacts() {
+ImmutableList.Builder pathsToStageBuilder =
ImmutableList.builder();
+ImmutableMap.Builder aliasMapBuilder =
ImmutableMap.builder();
+String windmillBinary =
+
options.as(DataflowPipelineDebugOptions.class).getOverrideWindmillBinary();
+String dataflowWorkerJar = options.getDataflowWorkerJar();
+if (dataflowWorkerJar != null && !dataflowWorkerJar.isEmpty()) {
+ // Put the user specified worker jar at the start of the classpath, to
be consistent with the
+ // built in worker order.
+ pathsToStageBuilder.add(dataflowWorkerJar);
+ aliasMapBuilder.put(dataflowWorkerJar, "dataflow-worker.jar");
+}
+for (String path : options.getFilesToStage()) {
+ if (path.contains("=")) {
Review comment:
Yes. This syntax is only supported in Dataflow runner. `DataflowPackage`
has a separate field `name` in addition to `location` and "=" separator allows
to prefix `name` to the location of the source e.g.
"dataflow.jar=/tmp/foo.jar". I could remove this special syntax but I decided
to keep it since it's already exposed to users via `--filesToStage` option so
removing it may cause backward compatibility issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 430494)
Time Spent: 7h 20m (was: 7h 10m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=430491=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430491
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 05/May/20 02:16
Start Date: 05/May/20 02:16
Worklog Time Spent: 10m
Work Description: ihji commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r419829971
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -784,7 +877,25 @@ public DataflowPipelineJob run(Pipeline pipeline) {
"Executing pipeline on the Dataflow Service, which will have billing
implications "
+ "related to Google Compute Engine usage and other Google Cloud
Services.");
-List packages = options.getStager().stageDefaultFiles();
+// Capture the sdkComponents for look up during step translations
+SdkComponents sdkComponents = SdkComponents.create();
+
+DataflowPipelineOptions dataflowOptions =
options.as(DataflowPipelineOptions.class);
+String workerHarnessContainerImageURL =
DataflowRunner.getContainerImageForJob(dataflowOptions);
+RunnerApi.Environment defaultEnvironmentForDataflow =
+Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+
+sdkComponents.registerEnvironment(
+defaultEnvironmentForDataflow
+.toBuilder()
+.addAllDependencies(getDefaultArtifacts())
+.build());
+
+RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline,
sdkComponents, true);
+
+LOG.debug("Portable pipeline proto:\n{}",
TextFormat.printToString(pipelineProto));
Review comment:
This debug log is not new. It's just relocated. Do you think it would be
better to remove this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 430491)
Time Spent: 7h 10m (was: 7h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=430490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430490
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 05/May/20 02:14
Start Date: 05/May/20 02:14
Worklog Time Spent: 10m
Work Description: ihji commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r419829659
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -210,56 +209,55 @@ public static Environment createProcessEnvironment(
}
}
- private static List getArtifacts(List
stagingFiles) {
-Set pathsToStage = Sets.newHashSet(stagingFiles);
+ public static List getArtifacts(
+ List stagingFiles, StagingFileNameGenerator generator) {
ImmutableList.Builder artifactsBuilder =
ImmutableList.builder();
-for (String path : pathsToStage) {
+for (String path : ImmutableSet.copyOf(stagingFiles)) {
Review comment:
`ImmutableSet` preserves the order but I think we don't need to make a
copy here. Will use `LinkedHashSet` instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 430490)
Time Spent: 7h (was: 6h 50m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=429592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-429592
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 01/May/20 18:16
Start Date: 01/May/20 18:16
Worklog Time Spent: 10m
Work Description: robertwb commented on a change in pull request #11039:
URL: https://github.com/apache/beam/pull/11039#discussion_r418651280
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1271,6 +1271,11 @@ message DeferredArtifactPayload {
message ArtifactStagingToRolePayload {
// A generated staged name (relative path under staging directory).
string staged_name = 1;
+
+ // (Optional) An artifact name when a runner supports it.
+ // For example, DataflowRunner requires predefined names for some artifacts
+ // such as "dataflow-worker.jar", "windmill_main".
+ string alias_name = 2;
Review comment:
Why does this have to be distinct from staged_name? (Also, eventually we
hope that designated roles can remove the need for magic names.)
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -210,56 +209,55 @@ public static Environment createProcessEnvironment(
}
}
- private static List getArtifacts(List
stagingFiles) {
-Set pathsToStage = Sets.newHashSet(stagingFiles);
+ public static List getArtifacts(
+ List stagingFiles, StagingFileNameGenerator generator) {
ImmutableList.Builder artifactsBuilder =
ImmutableList.builder();
-for (String path : pathsToStage) {
+for (String path : ImmutableSet.copyOf(stagingFiles)) {
Review comment:
Isn't order important to preserve? (Also, why do we need to make a copy?)
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -784,7 +877,25 @@ public DataflowPipelineJob run(Pipeline pipeline) {
"Executing pipeline on the Dataflow Service, which will have billing
implications "
+ "related to Google Compute Engine usage and other Google Cloud
Services.");
-List packages = options.getStager().stageDefaultFiles();
+// Capture the sdkComponents for look up during step translations
+SdkComponents sdkComponents = SdkComponents.create();
+
+DataflowPipelineOptions dataflowOptions =
options.as(DataflowPipelineOptions.class);
+String workerHarnessContainerImageURL =
DataflowRunner.getContainerImageForJob(dataflowOptions);
+RunnerApi.Environment defaultEnvironmentForDataflow =
+Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+
+sdkComponents.registerEnvironment(
+defaultEnvironmentForDataflow
+.toBuilder()
+.addAllDependencies(getDefaultArtifacts())
Review comment:
How does this get invoked for cross-language pipelines?
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
##
@@ -336,25 +323,26 @@ public DataflowPackage stageToFile(
final AtomicInteger numCached = new AtomicInteger(0);
List> destinationPackages = new
ArrayList<>();
-for (String classpathElement : classpathElements) {
- DataflowPackage sourcePackage = new DataflowPackage();
- if (classpathElement.contains("=")) {
-String[] components = classpathElement.split("=", 2);
-sourcePackage.setName(components[0]);
-sourcePackage.setLocation(components[1]);
- } else {
-sourcePackage.setName(null);
-sourcePackage.setLocation(classpathElement);
+for (StagedFile classpathElement : classpathElements) {
+ DataflowPackage targetPackage = classpathElement.getStagedPackage();
+ String source = classpathElement.getSource();
+ if (source.contains("=")) {
Review comment:
Why do we have to handle this here and above?
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -772,6 +783,88 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
+ImmutableList.Builder filesToStageBuilder =
ImmutableList.builder();
+for (Map.Entry entry :
+pipeline.getComponents().getEnvironmentsMap().entrySet()) {
+ for (RunnerApi.ArtifactInformation info :
entry.getValue().getDependenciesList()) {
+if
(!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn()))
{
+ throw new RuntimeException(
+ String.format("unsupported artifact type %s",
info.getTypeUrn()));
+}
+RunnerApi.ArtifactFilePayload filePayload;
+
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=428794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428794 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 29/Apr/20 23:31 Start Date: 29/Apr/20 23:31 Worklog Time Spent: 10m Work Description: ihji commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-621523014 @chamikaramj @robertwb : PTAL. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 428794) Time Spent: 6h 40m (was: 6.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=428389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428389 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 28/Apr/20 21:28 Start Date: 28/Apr/20 21:28 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-620864937 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 428389) Time Spent: 6.5h (was: 6h 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=428387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428387 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 28/Apr/20 21:27 Start Date: 28/Apr/20 21:27 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-620864665 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 428387) Time Spent: 6h 10m (was: 6h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=428388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428388 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 28/Apr/20 21:27 Start Date: 28/Apr/20 21:27 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-620864801 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 428388) Time Spent: 6h 20m (was: 6h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=427138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-427138 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 25/Apr/20 00:19 Start Date: 25/Apr/20 00:19 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-619289162 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 427138) Time Spent: 6h (was: 5h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=427097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-427097 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 24/Apr/20 21:04 Start Date: 24/Apr/20 21:04 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-619238029 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 427097) Time Spent: 5h 50m (was: 5h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=425393=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-425393 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 20/Apr/20 17:36 Start Date: 20/Apr/20 17:36 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11039: URL: https://github.com/apache/beam/pull/11039#issuecomment-616705346 Looks like this needs a rebase. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 425393) Time Spent: 5h 40m (was: 5.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=412355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-412355 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 30/Mar/20 14:02 Start Date: 30/Mar/20 14:02 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-606018509 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 412355) Time Spent: 5.5h (was: 5h 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=412354=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-412354 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 30/Mar/20 14:02 Start Date: 30/Mar/20 14:02 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-606018431 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 412354) Time Spent: 5h 20m (was: 5h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=411823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411823 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 29/Mar/20 00:07 Start Date: 29/Mar/20 00:07 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605536418 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 411823) Time Spent: 5h 10m (was: 5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=411751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411751 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 28/Mar/20 16:24 Start Date: 28/Mar/20 16:24 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605475407 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 411751) Time Spent: 5h (was: 4h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=411750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411750 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 28/Mar/20 16:23 Start Date: 28/Mar/20 16:23 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605473827 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 411750) Time Spent: 4h 50m (was: 4h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=411748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411748 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 28/Mar/20 16:22 Start Date: 28/Mar/20 16:22 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605472960 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 411748) Time Spent: 4.5h (was: 4h 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=411749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411749 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 28/Mar/20 16:22 Start Date: 28/Mar/20 16:22 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605473440 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 411749) Time Spent: 4h 40m (was: 4.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=411351=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411351 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 27/Mar/20 20:26 Start Date: 27/Mar/20 20:26 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-60538 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 411351) Time Spent: 4h 20m (was: 4h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=411346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411346 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 27/Mar/20 20:23 Start Date: 27/Mar/20 20:23 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605298889 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 411346) Time Spent: 4h 10m (was: 4h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=411345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411345 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 27/Mar/20 20:21 Start Date: 27/Mar/20 20:21 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605298244 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 411345) Time Spent: 4h (was: 3h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=47=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-47 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 27/Mar/20 14:30 Start Date: 27/Mar/20 14:30 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605031722 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 47) Time Spent: 3.5h (was: 3h 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=49=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-49 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 27/Mar/20 14:30 Start Date: 27/Mar/20 14:30 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605032008 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 49) Time Spent: 3h 50m (was: 3h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=48=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-48 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 27/Mar/20 14:30 Start Date: 27/Mar/20 14:30 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605031824 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 48) Time Spent: 3h 40m (was: 3.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=45=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-45 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 27/Mar/20 14:29 Start Date: 27/Mar/20 14:29 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605031496 Run Java PreCommi This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 45) Time Spent: 3h 10m (was: 3h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=46=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-46 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 27/Mar/20 14:29 Start Date: 27/Mar/20 14:29 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-605031622 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 46) Time Spent: 3h 20m (was: 3h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=410571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410571 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 26/Mar/20 20:01 Start Date: 26/Mar/20 20:01 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-604655018 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 410571) Time Spent: 3h (was: 2h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=409802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409802 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 25/Mar/20 20:34 Start Date: 25/Mar/20 20:34 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-604072538 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409802) Time Spent: 2h 50m (was: 2h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=408787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408787 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 24/Mar/20 14:15 Start Date: 24/Mar/20 14:15 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-603263845 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408787) Time Spent: 2.5h (was: 2h 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=408788=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408788 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 24/Mar/20 14:15 Start Date: 24/Mar/20 14:15 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-603263934 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408788) Time Spent: 2h 40m (was: 2.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=408318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408318 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 23/Mar/20 21:05 Start Date: 23/Mar/20 21:05 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-602855911 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408318) Time Spent: 2h 20m (was: 2h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=408221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408221 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 23/Mar/20 19:55 Start Date: 23/Mar/20 19:55 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-602822540 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408221) Time Spent: 2h 10m (was: 2h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=407181=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407181 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 20/Mar/20 19:01 Start Date: 20/Mar/20 19:01 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-601864064 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407181) Time Spent: 2h (was: 1h 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=405868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405868
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:35
Start Date: 19/Mar/20 00:35
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #11039: [BEAM-9383]
Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r394722832
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
+ImmutableList.Builder filesToStageBuilder =
ImmutableList.builder();
+for (Map.Entry entry :
+pipeline.getComponents().getEnvironmentsMap().entrySet()) {
+ for (RunnerApi.ArtifactInformation info :
entry.getValue().getDependenciesList()) {
+if
(!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn()))
{
Review comment:
Currently, DataflowRunner auto-generates staging names based on source file
names. So ROLE (STAGING_TO) is ignored anyway. We could change this behavior
later in BEAM-9455.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405868)
Time Spent: 1h 40m (was: 1.5h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=405869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405869 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/Mar/20 00:35 Start Date: 19/Mar/20 00:35 Worklog Time Spent: 10m Work Description: ihji commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-600925930 @chamikaramj PTAL and rerun the tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405869) Time Spent: 1h 50m (was: 1h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=405867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405867
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:31
Start Date: 19/Mar/20 00:31
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #11039: [BEAM-9383]
Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r394721868
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
Review comment:
It's private method so we can't directly test it. Existing test (such as
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java#L758)
should cover this method too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405867)
Time Spent: 1.5h (was: 1h 20m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=404888=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404888
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 17/Mar/20 18:18
Start Date: 17/Mar/20 18:18
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #11039: [BEAM-9383]
Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r393879915
##
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##
@@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path):
with open(from_path, 'rb') as f:
self.stage_file(to_folder, to_name, f, total_size=total_size)
- def _stage_resources(self, options):
+ def _stage_resources(self, pipeline, options):
google_cloud_options = options.view_as(GoogleCloudOptions)
if google_cloud_options.staging_location is None:
raise RuntimeError('The --staging_location option must be specified.')
if google_cloud_options.temp_location is None:
raise RuntimeError('The --temp_location option must be specified.')
+resources = []
+for _, env in pipeline.components.environments.items():
+ for dep in env.dependencies:
+if dep.type_urn != common_urns.artifact_types.FILE.urn:
+ raise RuntimeError('unsupported artifact type %s' % dep.type_urn)
+if dep.role_urn != common_urns.artifact_roles.STAGING_TO.urn:
+ raise RuntimeError('unsupported role type %s' % dep.role_urn)
+type_payload = beam_runner_api_pb2.ArtifactFilePayload.FromString(
+dep.type_payload)
+role_payload = \
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404888)
Time Spent: 1h 10m (was: 1h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=404889=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404889 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 17/Mar/20 18:18 Start Date: 17/Mar/20 18:18 Worklog Time Spent: 10m Work Description: ihji commented on pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#discussion_r393879970 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path): with open(from_path, 'rb') as f: self.stage_file(to_folder, to_name, f, total_size=total_size) - def _stage_resources(self, options): + def _stage_resources(self, pipeline, options): Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404889) Time Spent: 1h 20m (was: 1h 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=403278=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-403278
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 13/Mar/20 23:49
Start Date: 13/Mar/20 23:49
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530455
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
+ImmutableList.Builder filesToStageBuilder =
ImmutableList.builder();
+for (Map.Entry entry :
+pipeline.getComponents().getEnvironmentsMap().entrySet()) {
+ for (RunnerApi.ArtifactInformation info :
entry.getValue().getDependenciesList()) {
+if
(!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn()))
{
Review comment:
Check the ROLE as well ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 403278)
Time Spent: 50m (was: 40m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=403281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-403281
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 13/Mar/20 23:49
Start Date: 13/Mar/20 23:49
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530711
##
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##
@@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path):
with open(from_path, 'rb') as f:
self.stage_file(to_folder, to_name, f, total_size=total_size)
- def _stage_resources(self, options):
+ def _stage_resources(self, pipeline, options):
google_cloud_options = options.view_as(GoogleCloudOptions)
if google_cloud_options.staging_location is None:
raise RuntimeError('The --staging_location option must be specified.')
if google_cloud_options.temp_location is None:
raise RuntimeError('The --temp_location option must be specified.')
+resources = []
+for _, env in pipeline.components.environments.items():
+ for dep in env.dependencies:
+if dep.type_urn != common_urns.artifact_types.FILE.urn:
+ raise RuntimeError('unsupported artifact type %s' % dep.type_urn)
+if dep.role_urn != common_urns.artifact_roles.STAGING_TO.urn:
+ raise RuntimeError('unsupported role type %s' % dep.role_urn)
+type_payload = beam_runner_api_pb2.ArtifactFilePayload.FromString(
+dep.type_payload)
+role_payload = \
Review comment:
Nit: Pls use ( instead of \ for formatting.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 403281)
Time Spent: 1h (was: 50m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=403279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-403279
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 13/Mar/20 23:49
Start Date: 13/Mar/20 23:49
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530472
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
Review comment:
Please add a unit test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 403279)
Time Spent: 1h (was: 50m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=403280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-403280 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 13/Mar/20 23:49 Start Date: 13/Mar/20 23:49 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#discussion_r392530581 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path): with open(from_path, 'rb') as f: self.stage_file(to_folder, to_name, f, total_size=total_size) - def _stage_resources(self, options): + def _stage_resources(self, pipeline, options): Review comment: Please add a unit test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 403280) Time Spent: 1h (was: 50m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=402529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-402529 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 12/Mar/20 22:22 Start Date: 12/Mar/20 22:22 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-598458492 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 402529) Time Spent: 40m (was: 0.5h) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=402516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-402516 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 12/Mar/20 21:55 Start Date: 12/Mar/20 21:55 Worklog Time Spent: 10m Work Description: ihji commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-598435272 R: @robertwb, @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 402516) Time Spent: 0.5h (was: 20m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=397968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397968 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 04/Mar/20 23:30 Start Date: 04/Mar/20 23:30 Worklog Time Spent: 10m Work Description: ihji commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-594932376 This PR depends on #10621. needs to be rebased before merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397968) Time Spent: 20m (was: 10m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=397190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397190 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 04/Mar/20 02:30 Start Date: 04/Mar/20 02:30 Worklog Time Spent: 10m Work Description: ihji commented on pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [