[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=434776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434776 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/May/20 03:12 Start Date: 19/May/20 03:12 Worklog Time Spent: 10m Work Description: lukecwik commented on a change in pull request #11748: URL: https://github.com/apache/beam/pull/11748#discussion_r427005013 ## File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java ## @@ -183,7 +183,10 @@ public JobSpecification translate( String workerHarnessContainerImageURL = DataflowRunner.getContainerImageForJob(options.as(DataflowPipelineOptions.class)); RunnerApi.Environment defaultEnvironmentForDataflow = -Environments.createDockerEnvironment(workerHarnessContainerImageURL); +Environments.createDockerEnvironment(workerHarnessContainerImageURL) +.toBuilder() +.addAllCapabilities(Environments.getJavaCapabilities()) Review comment: Requirements is on the pipeline and that works. Since Dataflow is creating its own environment it is difficult to have common code because of the differences in the artifact staging and job APIs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434776) Time Spent: 8h 10m (was: 8h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=434777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434777 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/May/20 03:12 Start Date: 19/May/20 03:12 Worklog Time Spent: 10m Work Description: lukecwik merged pull request #11748: URL: https://github.com/apache/beam/pull/11748 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434777) Time Spent: 8h 20m (was: 8h 10m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=434724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434724 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/May/20 01:12 Start Date: 19/May/20 01:12 Worklog Time Spent: 10m Work Description: robertwb commented on a change in pull request #11748: URL: https://github.com/apache/beam/pull/11748#discussion_r426974286 ## File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java ## @@ -183,7 +183,10 @@ public JobSpecification translate( String workerHarnessContainerImageURL = DataflowRunner.getContainerImageForJob(options.as(DataflowPipelineOptions.class)); RunnerApi.Environment defaultEnvironmentForDataflow = -Environments.createDockerEnvironment(workerHarnessContainerImageURL); +Environments.createDockerEnvironment(workerHarnessContainerImageURL) +.toBuilder() +.addAllCapabilities(Environments.getJavaCapabilities()) Review comment: Is there some common utility we should be using here (rather than duplicating the code that's used in all the other portable runners)? What about requirements? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434724) Time Spent: 8h (was: 7h 50m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=434718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434718 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/May/20 01:03 Start Date: 19/May/20 01:03 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11748: URL: https://github.com/apache/beam/pull/11748#issuecomment-630514195 CC: @ananvay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434718) Time Spent: 7h 50m (was: 7h 40m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 7h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=434714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434714 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/May/20 01:03 Start Date: 19/May/20 01:03 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11748: URL: https://github.com/apache/beam/pull/11748#issuecomment-630514089 R: @y1chi @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434714) Time Spent: 7.5h (was: 7h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=434715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434715 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/May/20 01:03 Start Date: 19/May/20 01:03 Worklog Time Spent: 10m Work Description: lukecwik edited a comment on pull request #11748: URL: https://github.com/apache/beam/pull/11748#issuecomment-630514089 R: @ihji @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434715) Time Spent: 7h 40m (was: 7.5h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 7h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=434713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434713 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/May/20 01:02 Start Date: 19/May/20 01:02 Worklog Time Spent: 10m Work Description: lukecwik opened a new pull request #11748: URL: https://github.com/apache/beam/pull/11748 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=407176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407176 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Mar/20 18:55 Start Date: 20/Mar/20 18:55 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407176) Time Spent: 7h 10m (was: 7h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406544 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/Mar/20 20:14 Start Date: 19/Mar/20 20:14 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-601394987 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 406544) Time Spent: 7h (was: 6h 50m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406531
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 19/Mar/20 19:29
Start Date: 19/Mar/20 19:29
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11162: [BEAM-9339,
BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn
requirement to Python SDK.
URL: https://github.com/apache/beam/pull/11162#discussion_r395269363
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -457,10 +457,9 @@ message ParDoPayload {
// be placed in the pipeline requirements.
map timer_family_specs = 9;
- // Whether the DoFn is splittable
- bool splittable = 6;
-
- // (Required if splittable == true) Id of the restriction coder.
+ // (Optional) Only set when this ParDo contains a splittable DoFn.
Review comment:
Because it's not as obvious. But I can see the pros and cons.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406531)
Time Spent: 6h 40m (was: 6.5h)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406532=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406532
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 19/Mar/20 19:29
Start Date: 19/Mar/20 19:29
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11162: [BEAM-9339,
BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn
requirement to Python SDK.
URL: https://github.com/apache/beam/pull/11162#discussion_r395269373
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1355,6 +1355,10 @@ message StandardRequirements {
// This requirement indicates the requires_time_sorted_input field of ParDo
// transform payloads must be inspected.
REQUIRES_TIME_SORTED_INPUT = 3 [(beam_urn) =
"beam:requirement:pardo:time_sorted_input:v1"];
+
+// This requirement indicates the restriction_coder_id field of ParDo
+// transform payloads must be inspected.
+REQUIRES_SPLITTABLE_DOFN = 4 [(beam_urn) =
"beam:requirement:pardo:splittable_dofn:v1"];
Review comment:
Fair point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406532)
Time Spent: 6h 50m (was: 6h 40m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406451 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/Mar/20 18:03 Start Date: 19/Mar/20 18:03 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-601331123 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 406451) Time Spent: 6.5h (was: 6h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406450
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 19/Mar/20 18:03
Start Date: 19/Mar/20 18:03
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11162: [BEAM-9339,
BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn
requirement to Python SDK.
URL: https://github.com/apache/beam/pull/11162#discussion_r395220391
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -457,10 +457,9 @@ message ParDoPayload {
// be placed in the pipeline requirements.
map timer_family_specs = 9;
- // Whether the DoFn is splittable
- bool splittable = 6;
-
- // (Required if splittable == true) Id of the restriction coder.
+ // (Optional) Only set when this ParDo contains a splittable DoFn.
Review comment:
`splittable` is invalid without the other field being set so why have it?
It would allow for bugs where one is set and the other is not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406450)
Time Spent: 6h 20m (was: 6h 10m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406444=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406444
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 19/Mar/20 17:56
Start Date: 19/Mar/20 17:56
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11162: [BEAM-9339,
BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn
requirement to Python SDK.
URL: https://github.com/apache/beam/pull/11162#discussion_r395216164
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1355,6 +1355,10 @@ message StandardRequirements {
// This requirement indicates the requires_time_sorted_input field of ParDo
// transform payloads must be inspected.
REQUIRES_TIME_SORTED_INPUT = 3 [(beam_urn) =
"beam:requirement:pardo:time_sorted_input:v1"];
+
+// This requirement indicates the restriction_coder_id field of ParDo
+// transform payloads must be inspected.
+REQUIRES_SPLITTABLE_DOFN = 4 [(beam_urn) =
"beam:requirement:pardo:splittable_dofn:v1"];
Review comment:
If its unbounded, a runner will never be able to execute it without
expansion which requires the runner to at least inspect whether its capable of
doing it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406444)
Time Spent: 6h 10m (was: 6h)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406440=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406440
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 19/Mar/20 17:52
Start Date: 19/Mar/20 17:52
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11162: [BEAM-9339,
BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn
requirement to Python SDK.
URL: https://github.com/apache/beam/pull/11162#discussion_r395213529
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1355,6 +1355,10 @@ message StandardRequirements {
// This requirement indicates the requires_time_sorted_input field of ParDo
// transform payloads must be inspected.
REQUIRES_TIME_SORTED_INPUT = 3 [(beam_urn) =
"beam:requirement:pardo:time_sorted_input:v1"];
+
+// This requirement indicates the restriction_coder_id field of ParDo
+// transform payloads must be inspected.
+REQUIRES_SPLITTABLE_DOFN = 4 [(beam_urn) =
"beam:requirement:pardo:splittable_dofn:v1"];
Review comment:
Is this a requirement, given that the pipeline would still execute correctly
were expansion not performed? (Maybe the performance is bad enough to merit
this, but then again, maybe SDF is so primitive that all runners should do
something do support sources.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406440)
Time Spent: 6h (was: 5h 50m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 6h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406439
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 19/Mar/20 17:52
Start Date: 19/Mar/20 17:52
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11162: [BEAM-9339,
BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn
requirement to Python SDK.
URL: https://github.com/apache/beam/pull/11162#discussion_r395212786
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -457,10 +457,9 @@ message ParDoPayload {
// be placed in the pipeline requirements.
map timer_family_specs = 9;
- // Whether the DoFn is splittable
- bool splittable = 6;
-
- // (Required if splittable == true) Id of the restriction coder.
+ // (Optional) Only set when this ParDo contains a splittable DoFn.
Review comment:
I'm not convinced that using the presence of a restriction coder is better
than explicitly marking things as splittable, but could possibly be persuaded.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406439)
Time Spent: 5h 50m (was: 5h 40m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406394 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/Mar/20 16:50 Start Date: 19/Mar/20 16:50 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-601292658 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 406394) Time Spent: 5h 40m (was: 5.5h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406367 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/Mar/20 16:22 Start Date: 19/Mar/20 16:22 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11160: [BEAM-9339] Declare capabilities in the Java SDK. URL: https://github.com/apache/beam/pull/11160 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 406367) Time Spent: 5.5h (was: 5h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=406334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406334 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/Mar/20 15:37 Start Date: 19/Mar/20 15:37 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-601248628 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 406334) Time Spent: 5h 20m (was: 5h 10m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405864
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:26
Start Date: 19/Mar/20 00:26
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11160: [BEAM-9339]
Declare capabilities in the Java SDK.
URL: https://github.com/apache/beam/pull/11160#discussion_r394720559
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -228,6 +232,10 @@ public static Environment createProcessEnvironment(
return filesToStage.build();
}
+ public static Set getJavaCapabilities() {
+return ModelCoders.urns();
Review comment:
Yes but only some metrics are being reported.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405864)
Time Spent: 5h 10m (was: 5h)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405852 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/Mar/20 00:04 Start Date: 19/Mar/20 00:04 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-600917167 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405852) Time Spent: 5h (was: 4h 50m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405828 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 23:38 Start Date: 18/Mar/20 23:38 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11160: [BEAM-9339] Declare capabilities in the Java SDK. URL: https://github.com/apache/beam/pull/11160#issuecomment-600910480 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405828) Time Spent: 4h 50m (was: 4h 40m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405827 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 23:37 Start Date: 18/Mar/20 23:37 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-600910385 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405827) Time Spent: 4h 40m (was: 4.5h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405814 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 23:03 Start Date: 18/Mar/20 23:03 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-600901398 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405814) Time Spent: 4.5h (was: 4h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405798
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:39
Start Date: 18/Mar/20 22:39
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11160: [BEAM-9339]
Declare capabilities in the Java SDK.
URL: https://github.com/apache/beam/pull/11160#discussion_r394679769
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -228,6 +232,10 @@ public static Environment createProcessEnvironment(
return filesToStage.build();
}
+ public static Set getJavaCapabilities() {
+return ModelCoders.urns();
Review comment:
(I suppose that's just the new metrics PCollection element counts, right?)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405798)
Time Spent: 4h 20m (was: 4h 10m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405797
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:38
Start Date: 18/Mar/20 22:38
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11160: [BEAM-9339]
Declare capabilities in the Java SDK.
URL: https://github.com/apache/beam/pull/11160#discussion_r394679580
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -228,6 +232,10 @@ public static Environment createProcessEnvironment(
return filesToStage.build();
}
+ public static Set getJavaCapabilities() {
+return ModelCoders.urns();
Review comment:
Good call on multi-core. Is progress reporting actually done now?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405797)
Time Spent: 4h 10m (was: 4h)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405733 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 20:39 Start Date: 18/Mar/20 20:39 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11160: [BEAM-9339] Declare capabilities in the Java SDK. URL: https://github.com/apache/beam/pull/11160#issuecomment-600847451 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405733) Time Spent: 4h (was: 3h 50m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405711 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 19:38 Start Date: 18/Mar/20 19:38 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405712 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 19:38 Start Date: 18/Mar/20 19:38 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-600821917 R: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405712) Time Spent: 3h 50m (was: 3h 40m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405674 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 18:38 Start Date: 18/Mar/20 18:38 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11160: [BEAM-9339] Declare capabilities in the Java SDK. URL: https://github.com/apache/beam/pull/11160#issuecomment-600796683 Whats your strategy for adding all the pardo requirements? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405674) Time Spent: 3.5h (was: 3h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405656
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 18/Mar/20 18:14
Start Date: 18/Mar/20 18:14
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11160: [BEAM-9339]
Declare capabilities in the Java SDK.
URL: https://github.com/apache/beam/pull/11160#discussion_r394546436
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -228,6 +232,10 @@ public static Environment createProcessEnvironment(
return filesToStage.build();
}
+ public static Set getJavaCapabilities() {
+return ModelCoders.urns();
Review comment:
Add:
MULTI_CORE_BUNDLE_PROCESSING
PROGRESS_REPORTING
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405656)
Time Spent: 3h 20m (was: 3h 10m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405641 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 18:00 Start Date: 18/Mar/20 18:00 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11160: [BEAM-9339] Declare capabilities in the Java SDK. URL: https://github.com/apache/beam/pull/11160#issuecomment-600779129 R: @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405641) Time Spent: 3h 10m (was: 3h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405639=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405639 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 17:59 Start Date: 18/Mar/20 17:59 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11160: [BEAM-9339] Declare capabilities in the Java SDK. URL: https://github.com/apache/beam/pull/11160 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390912 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 21/Feb/20 21:26 Start Date: 21/Feb/20 21:26 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10908: [BEAM-9339] Declare capabilities for Python SDK. URL: https://github.com/apache/beam/pull/10908 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390912) Time Spent: 2h 50m (was: 2h 40m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390805 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 21/Feb/20 17:33 Start Date: 21/Feb/20 17:33 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390805) Time Spent: 2h 40m (was: 2.5h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390311 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 23:14 Start Date: 20/Feb/20 23:14 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#discussion_r382311690 ## File path: sdks/go/pkg/beam/core/runtime/graphx/translate.go ## @@ -26,6 +27,7 @@ import ( "github.com/apache/beam/sdks/go/pkg/beam/core/util/protox" "github.com/apache/beam/sdks/go/pkg/beam/internal/errors" pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1" + "github.com/apache/beam/sdks/go/pkg/beam/options/jobopts" Review comment: OK, done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390311) Time Spent: 2.5h (was: 2h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390291
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 22:16
Start Date: 20/Feb/20 22:16
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #10911: [BEAM-9339]
Declare capabilities for Go SDK.
URL: https://github.com/apache/beam/pull/10911#discussion_r382286309
##
File path: sdks/go/pkg/beam/core/runtime/graphx/coder.go
##
@@ -50,6 +50,22 @@ const (
urnCoGBKList = "beam:go:coder:cogbklist:v1" // CoGBK representation.
Not a coder.
)
+func KnownStandardCoders() []string {
Review comment:
Since this lives in the same package as the caller, we can make the function
un-exported (lowercase k for knownStandardCoders() instead).
It's reasonably to have this function live here, due to locality with the
coder URNs
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390291)
Time Spent: 2h 20m (was: 2h 10m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390292 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 22:16 Start Date: 20/Feb/20 22:16 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#discussion_r382287004 ## File path: sdks/go/pkg/beam/core/runtime/graphx/translate.go ## @@ -26,6 +27,7 @@ import ( "github.com/apache/beam/sdks/go/pkg/beam/core/util/protox" "github.com/apache/beam/sdks/go/pkg/beam/internal/errors" pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1" + "github.com/apache/beam/sdks/go/pkg/beam/options/jobopts" Review comment: I don't like that we're importing this package here. jobopts creates flags, which could be unexpected or unnecessary for certain runners. Could we instead just pass the environment urn in as a string? (the only thing we actually use from the package). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390292) Time Spent: 2h 20m (was: 2h 10m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390283 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 22:04 Start Date: 20/Feb/20 22:04 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#issuecomment-589361208 OK, we do need this for dataflow. I refactored the code to be shared in translate.go. PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390283) Time Spent: 2h 10m (was: 2h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390218
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 20:09
Start Date: 20/Feb/20 20:09
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #10911: [BEAM-9339]
Declare capabilities for Go SDK.
URL: https://github.com/apache/beam/pull/10911#discussion_r382230929
##
File path: sdks/go/pkg/beam/runners/universal/universal.go
##
@@ -78,6 +78,20 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
return err
}
+const (
+ urnLegacyProgressReporting = "beam:protocol:progress_reporting:v0"
+ urnMultiCore =
"beam:protocol:multi_core_bundle_processing:v1"
+)
+
+func goCapabilities() []string {
+ capabilities := []string{
+ urnLegacyProgressReporting,
+ urnLegacyProgressReporting,
+ }
+ capabilities = append(capabilities, graphx.KnownStandardCoders()...)
+ return capabilities
+}
+
Review comment:
Fair enough.
Technically append always returns a new "slice" but the backing array for
the data may be reused. It's possible to have aliasing problems if done poorly,
but that's not a risk here.
It mostly comes down to the difference between the used length, and the
capacity of the slice. If there's enough capacity, it will simply mutate the
backing slice.
To avoid the aliasing, as a rule, one should avoid using an slice one didn't
allocate as the base for append.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390218)
Time Spent: 2h (was: 1h 50m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390201 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 19:39 Start Date: 20/Feb/20 19:39 Worklog Time Spent: 10m Work Description: lostluck commented on issue #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#issuecomment-589271526 [Dataflow creates it's own environment](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/dataflow/dataflow.go#L219) and doesn't use the universal runner. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390201) Time Spent: 1h 50m (was: 1h 40m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390191
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 19:29
Start Date: 20/Feb/20 19:29
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10908: [BEAM-9339]
Declare capabilities for Python SDK.
URL: https://github.com/apache/beam/pull/10908#discussion_r382210720
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1209,6 +1209,30 @@ message ExternalPayload {
map params = 2; // Arbitrary extra parameters to pass
}
+// These URNs are used to indicate capabilities of environments that cannot
+// simply be expressed as a component (such as a Coder or PTransform) that this
+// environment understands.
+message StandardProtocols {
+ enum Enum {
+// Indicates suport for progress reporting via the legacy metric APIs.
+LEGACY_PROGRESS_REPORTING = 0 [(beam_urn) =
"beam:protocol:progress_reporting:v0"];
+
+// Indicates suport for progress reporting via the new metric APIs.
Review comment:
Good point. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390191)
Time Spent: 1h 40m (was: 1.5h)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390189 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 19:23 Start Date: 20/Feb/20 19:23 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#issuecomment-589264262 I'll defer further refactoring, but could you clarify your comment about Dataflow? Are we not creating/setting an environment at all in that case? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390189) Time Spent: 1.5h (was: 1h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390187
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 19:21
Start Date: 20/Feb/20 19:21
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10911: [BEAM-9339]
Declare capabilities for Go SDK.
URL: https://github.com/apache/beam/pull/10911#discussion_r382206635
##
File path: sdks/go/pkg/beam/runners/universal/universal.go
##
@@ -78,6 +78,20 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
return err
}
+const (
+ urnLegacyProgressReporting = "beam:protocol:progress_reporting:v0"
+ urnMultiCore =
"beam:protocol:multi_core_bundle_processing:v1"
+)
+
+func goCapabilities() []string {
+ capabilities := []string{
+ urnLegacyProgressReporting,
+ urnLegacyProgressReporting,
+ }
+ capabilities = append(capabilities, graphx.KnownStandardCoders()...)
+ return capabilities
+}
+
Review comment:
Done. I wasn't sure what the convention was--the semantics of append (may
modify the first argument or return a new value) seemed odd to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390187)
Time Spent: 1h 20m (was: 1h 10m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390186
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 19:20
Start Date: 20/Feb/20 19:20
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10911: [BEAM-9339]
Declare capabilities for Go SDK.
URL: https://github.com/apache/beam/pull/10911#discussion_r382206178
##
File path: sdks/go/pkg/beam/runners/universal/universal.go
##
@@ -78,6 +78,20 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
return err
}
+const (
+ urnLegacyProgressReporting = "beam:protocol:progress_reporting:v0"
+ urnMultiCore =
"beam:protocol:multi_core_bundle_processing:v1"
+)
+
+func goCapabilities() []string {
+ capabilities := []string{
+ urnLegacyProgressReporting,
+ urnLegacyProgressReporting,
Review comment:
Oh, yes. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390186)
Time Spent: 1h 10m (was: 1h)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=389762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389762
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 02:47
Start Date: 20/Feb/20 02:47
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #10908:
[BEAM-9339] Declare capabilities for Python SDK.
URL: https://github.com/apache/beam/pull/10908#discussion_r381688154
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1209,6 +1209,30 @@ message ExternalPayload {
map params = 2; // Arbitrary extra parameters to pass
}
+// These URNs are used to indicate capabilities of environments that cannot
+// simply be expressed as a component (such as a Coder or PTransform) that this
+// environment understands.
+message StandardProtocols {
+ enum Enum {
+// Indicates suport for progress reporting via the legacy metric APIs.
+LEGACY_PROGRESS_REPORTING = 0 [(beam_urn) =
"beam:protocol:progress_reporting:v0"];
+
+// Indicates suport for progress reporting via the new metric APIs.
Review comment:
nit: should we add a reference to clarify what old/new metrics are ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389762)
Time Spent: 1h (was: 50m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=389763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389763 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 02:47 Start Date: 20/Feb/20 02:47 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #10908: [BEAM-9339] Declare capabilities for Python SDK. URL: https://github.com/apache/beam/pull/10908#discussion_r381690351 ## File path: sdks/python/apache_beam/transforms/environments_test.py ## @@ -53,10 +63,14 @@ def test_environment_encoding(self): state_cache_size=0, data_buffer_time_limit_ms=0), SubprocessSDKEnvironment(command_string=u'foö')): context = pipeline_context.PipelineContext() - self.assertEqual( - environment, - Environment.from_runner_api( - environment.to_runner_api(context), context)) + proto = environment.to_runner_api(context) + reconstructed = Environment.from_runner_api(proto, context) + self.assertEqual(environment, reconstructed) + self.assertEqual(proto, reconstructed.to_runner_api(context)) + + def test_sdk_capabilities(self): +sdk_capabilities = environments.python_sdk_capabilities() +self.assertIn(common_urns.coders.LENGTH_PREFIX.urn, sdk_capabilities) Review comment: Probably test for every expected capability here ? And also force failure if the test has not been updated to reflect a new capability ? The test can be updated as the list of capabilities are updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 389763) Time Spent: 1h (was: 50m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=389745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389745
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 01:56
Start Date: 20/Feb/20 01:56
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #10911: [BEAM-9339]
Declare capabilities for Go SDK.
URL: https://github.com/apache/beam/pull/10911#discussion_r381659754
##
File path: sdks/go/pkg/beam/runners/universal/universal.go
##
@@ -78,6 +78,20 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
return err
}
+const (
+ urnLegacyProgressReporting = "beam:protocol:progress_reporting:v0"
+ urnMultiCore =
"beam:protocol:multi_core_bundle_processing:v1"
+)
+
+func goCapabilities() []string {
+ capabilities := []string{
+ urnLegacyProgressReporting,
+ urnLegacyProgressReporting,
+ }
+ capabilities = append(capabilities, graphx.KnownStandardCoders()...)
+ return capabilities
+}
+
Review comment:
You can return straight from the append and save a line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389745)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=389744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389744
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 01:56
Start Date: 20/Feb/20 01:56
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #10911: [BEAM-9339]
Declare capabilities for Go SDK.
URL: https://github.com/apache/beam/pull/10911#discussion_r381658325
##
File path: sdks/go/pkg/beam/runners/universal/universal.go
##
@@ -78,6 +78,20 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
return err
}
+const (
+ urnLegacyProgressReporting = "beam:protocol:progress_reporting:v0"
+ urnMultiCore =
"beam:protocol:multi_core_bundle_processing:v1"
+)
+
+func goCapabilities() []string {
+ capabilities := []string{
+ urnLegacyProgressReporting,
+ urnLegacyProgressReporting,
Review comment:
Accidentally duped.
urnMultiCore?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389744)
Time Spent: 50m (was: 40m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=389733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389733 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 01:22 Start Date: 20/Feb/20 01:22 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#issuecomment-588561432 R: @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 389733) Time Spent: 40m (was: 0.5h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=389731=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389731 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 01:21 Start Date: 20/Feb/20 01:21 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=389700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389700 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 00:23 Start Date: 20/Feb/20 00:23 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10908: [BEAM-9339] Declare capabilities for Python SDK. URL: https://github.com/apache/beam/pull/10908#issuecomment-588544811 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 389700) Time Spent: 20m (was: 10m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=389698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389698 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 00:22 Start Date: 20/Feb/20 00:22 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10908: [BEAM-9339] Declare capabilities for Python SDK. URL: https://github.com/apache/beam/pull/10908 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[