[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=414300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414300 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 01/Apr/20 21:37 Start Date: 01/Apr/20 21:37 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11277: [BEAM-9340] Populate requirement for timer families. URL: https://github.com/apache/beam/pull/11277 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 414300) Time Spent: 5.5h (was: 5h 20m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=413516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413516 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 31/Mar/20 21:04 Start Date: 31/Mar/20 21:04 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11277: [BEAM-9340] Populate requirement for timer families. URL: https://github.com/apache/beam/pull/11277#issuecomment-606874607 CC: @boyuanzz This looks like a version that covers more then #11266 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 413516) Time Spent: 5h 20m (was: 5h 10m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=413511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413511 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 31/Mar/20 20:52 Start Date: 31/Mar/20 20:52 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11277: [BEAM-9340] Populate requirement for timer families. URL: https://github.com/apache/beam/pull/11277#issuecomment-606869063 We should consider consolidating timers and timer families at a lower level, at lower priority than getting the protos in shape. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 413511) Time Spent: 5h 10m (was: 5h) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=413509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413509 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 31/Mar/20 20:50 Start Date: 31/Mar/20 20:50 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11277: [BEAM-9340] Populate requirement for timer families. URL: https://github.com/apache/beam/pull/11277 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python |
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=410444=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410444 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 26/Mar/20 17:49 Start Date: 26/Mar/20 17:49 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11224: [BEAM-9340] Validate pipeline requirements in PipelineValidator. URL: https://github.com/apache/beam/pull/11224 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 410444) Time Spent: 4h 50m (was: 4h 40m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[
https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=410443=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410443
]
ASF GitHub Bot logged work on BEAM-9340:
Author: ASF GitHub Bot
Created on: 26/Mar/20 17:49
Start Date: 26/Mar/20 17:49
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11224: [BEAM-9340]
Validate pipeline requirements in PipelineValidator.
URL: https://github.com/apache/beam/pull/11224#discussion_r398771741
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java
##
@@ -221,23 +229,39 @@ private static void validateParDo(String id, PTransform
transform, Components co
id,
sideInputId);
}
-// TODO: Validate state_specs and timer_specs
+if (payload.getStateSpecsCount() > 0 || payload.getTimerSpecsCount() > 0) {
Review comment:
We can merge now and you will need to swap from TimerSpecs to
TimerFamilySpecs here as part of your PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410443)
Time Spent: 4h 40m (was: 4.5h)
> Properly populate pipeline proto requirements.
> --
>
> Key: BEAM-9340
> URL: https://issues.apache.org/jira/browse/BEAM-9340
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[
https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=410406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410406
]
ASF GitHub Bot logged work on BEAM-9340:
Author: ASF GitHub Bot
Created on: 26/Mar/20 17:07
Start Date: 26/Mar/20 17:07
Worklog Time Spent: 10m
Work Description: boyuanzz commented on pull request #11224: [BEAM-9340]
Validate pipeline requirements in PipelineValidator.
URL: https://github.com/apache/beam/pull/11224#discussion_r398741485
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java
##
@@ -221,23 +229,39 @@ private static void validateParDo(String id, PTransform
transform, Components co
id,
sideInputId);
}
-// TODO: Validate state_specs and timer_specs
+if (payload.getStateSpecsCount() > 0 || payload.getTimerSpecsCount() > 0) {
Review comment:
Ack. can we merge it as soon as possible? Then I can rebase my changes on
top of this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410406)
Time Spent: 4.5h (was: 4h 20m)
> Properly populate pipeline proto requirements.
> --
>
> Key: BEAM-9340
> URL: https://issues.apache.org/jira/browse/BEAM-9340
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[
https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=410372=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410372
]
ASF GitHub Bot logged work on BEAM-9340:
Author: ASF GitHub Bot
Created on: 26/Mar/20 16:10
Start Date: 26/Mar/20 16:10
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11224: [BEAM-9340]
Validate pipeline requirements in PipelineValidator.
URL: https://github.com/apache/beam/pull/11224#discussion_r398695607
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java
##
@@ -221,23 +229,39 @@ private static void validateParDo(String id, PTransform
transform, Components co
id,
sideInputId);
}
-// TODO: Validate state_specs and timer_specs
+if (payload.getStateSpecsCount() > 0 || payload.getTimerSpecsCount() > 0) {
Review comment:
This should also have timer family specs.
@boyuanzz Once this code is merged, this is another place where TimerSpecs
will need to be removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410372)
Time Spent: 4h 20m (was: 4h 10m)
> Properly populate pipeline proto requirements.
> --
>
> Key: BEAM-9340
> URL: https://issues.apache.org/jira/browse/BEAM-9340
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409969 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 23:37 Start Date: 25/Mar/20 23:37 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11224: [BEAM-9340] Validate pipeline requirements in PipelineValidator. URL: https://github.com/apache/beam/pull/11224#issuecomment-604145091 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409969) Time Spent: 4h 10m (was: 4h) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409967 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 23:35 Start Date: 25/Mar/20 23:35 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11224: [BEAM-9340] Validate pipeline requirements in PipelineValidator. URL: https://github.com/apache/beam/pull/11224#issuecomment-604144495 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409967) Time Spent: 4h (was: 3h 50m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409713 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 18:38 Start Date: 25/Mar/20 18:38 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11224: [BEAM-9340] Validate pipeline requirements in PipelineValidator. URL: https://github.com/apache/beam/pull/11224#issuecomment-604014835 Follow-up to https://github.com/apache/beam/pull/11165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409713) Time Spent: 3h 50m (was: 3h 40m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409700 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 18:35 Start Date: 25/Mar/20 18:35 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11224: [BEAM-9340] Validate pipeline requirements in PipelineValidator. URL: https://github.com/apache/beam/pull/11224 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409647 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 17:02 Start Date: 25/Mar/20 17:02 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-603962869 Now org.apache.beam.sdk.io.cassandra.CassandraIOTest.classMethod is failing. These are unrelated and didn't fail before. Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409647) Time Spent: 3h 20m (was: 3h 10m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409648 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 17:02 Start Date: 25/Mar/20 17:02 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409648) Time Spent: 3.5h (was: 3h 20m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409119 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 24/Mar/20 22:00 Start Date: 24/Mar/20 22:00 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-603527163 Good point about the validator. I'll do that in a follow-up PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409119) Time Spent: 3h 10m (was: 3h) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409117 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 24/Mar/20 21:59 Start Date: 24/Mar/20 21:59 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-603527045 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409117) Time Spent: 3h (was: 2h 50m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=408033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408033 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 23/Mar/20 16:26 Start Date: 23/Mar/20 16:26 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-602707990 It would be good to extend [PipelineValidator](https://github.com/apache/beam/blob/33845727ec635eb07f95db12809e6ce217af8b33/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java#L42) to make sure that SDKs are submitting their jobs correctly including the set of requirements by inspecting the ParDoPayload. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408033) Time Spent: 2h 50m (was: 2h 40m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=408032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408032 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 23/Mar/20 16:24 Start Date: 23/Mar/20 16:24 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-602707990 It would be good to extend [PipelineValidator](https://github.com/apache/beam/blob/33845727ec635eb07f95db12809e6ce217af8b33/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java#L42) to make sure that SDKs are submitting their jobs correctly including the set of requirements. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408032) Time Spent: 2h 40m (was: 2.5h) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=408029=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408029 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 23/Mar/20 16:20 Start Date: 23/Mar/20 16:20 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-602705842 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408029) Time Spent: 2.5h (was: 2h 20m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=407315=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407315 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 20/Mar/20 23:06 Start Date: 20/Mar/20 23:06 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-601947065 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407315) Time Spent: 2h 20m (was: 2h 10m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=407311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407311 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 20/Mar/20 23:02 Start Date: 20/Mar/20 23:02 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-601946285 This passes locally. org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInFinishBundleStateful seems unrelated. Retrying again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407311) Time Spent: 2h (was: 1h 50m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=407312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407312 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 20/Mar/20 23:02 Start Date: 20/Mar/20 23:02 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-601946329 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407312) Time Spent: 2h 10m (was: 2h) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=407272=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407272 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 20/Mar/20 21:37 Start Date: 20/Mar/20 21:37 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-601921071 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407272) Time Spent: 1h 50m (was: 1h 40m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=407190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407190 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 20/Mar/20 19:22 Start Date: 20/Mar/20 19:22 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-601872470 I rebased on top of your PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407190) Time Spent: 1h 40m (was: 1.5h) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=407189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407189 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 20/Mar/20 19:22 Start Date: 20/Mar/20 19:22 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#discussion_r395845584 ## File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java ## @@ -124,6 +124,15 @@ public static final String SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN = "beam:transform:sdf_process_sized_element_and_restrictions:v1"; + public static final String REQUIRES_STATEFUL_PROCESSING_URN = + getUrn(RunnerApi.StandardRequirements.Enum.REQUIRES_STATEFUL_PROCESSING); Review comment: That's unfortunate, but I see the pattern. Have to be vigilant to prevent bugs. (Unlikely that these'll be used in switch statements, but consistency is good.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407189) Time Spent: 1.5h (was: 1h 20m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[
https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=406568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406568
]
ASF GitHub Bot logged work on BEAM-9340:
Author: ASF GitHub Bot
Created on: 19/Mar/20 20:49
Start Date: 19/Mar/20 20:49
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11165: [BEAM-9340]
Populate requirements for Java.
URL: https://github.com/apache/beam/pull/11165#discussion_r395310246
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
##
@@ -86,25 +91,30 @@ public static SdkComponents create(RunnerApi.Components
components) {
}
public static SdkComponents create(PipelineOptions options) {
-SdkComponents sdkComponents = new
SdkComponents(RunnerApi.Components.getDefaultInstance(), "");
+SdkComponents sdkComponents =
+new SdkComponents(RunnerApi.Components.getDefaultInstance(), null, "");
PortablePipelineOptions portablePipelineOptions =
options.as(PortablePipelineOptions.class);
sdkComponents.registerEnvironment(
Environments.createOrGetDefaultEnvironment(portablePipelineOptions));
return sdkComponents;
}
- private SdkComponents(RunnerApi.Components components, String newIdPrefix) {
+ private SdkComponents(
+ Components components, Collection requirements, String
newIdPrefix) {
this.newIdPrefix = newIdPrefix;
+this.requirements = new HashSet<>();
if (components == null) {
- return;
+ if (requirements != null) {
+this.requirements.addAll(requirements);
+ }
+} else {
+ mergeFrom(components, requirements);
}
-
-mergeFrom(components);
}
/** Merge Components proto into this SdkComponents instance. */
- public void mergeFrom(RunnerApi.Components components) {
+ public void mergeFrom(RunnerApi.Components components, Collection
requirements) {
Review comment:
```suggestion
public void mergeFrom(RunnerApi.Components components, @Nullable
Collection requirements) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406568)
Time Spent: 1h 20m (was: 1h 10m)
> Properly populate pipeline proto requirements.
> --
>
> Key: BEAM-9340
> URL: https://issues.apache.org/jira/browse/BEAM-9340
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[
https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=406567=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406567
]
ASF GitHub Bot logged work on BEAM-9340:
Author: ASF GitHub Bot
Created on: 19/Mar/20 20:49
Start Date: 19/Mar/20 20:49
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11165: [BEAM-9340]
Populate requirements for Java.
URL: https://github.com/apache/beam/pull/11165#discussion_r395308799
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/EnvironmentsTest.java
##
@@ -115,7 +115,7 @@ public void getEnvironmentPTransform() throws IOException {
ParDo.of(
new DoFn() {
@ProcessElement
- public void process(ProcessContext ctxt) {}
+ public void process(ProcessContext ctxt)
{}CreateExecutableStageNodeFunction.java
Review comment:
```suggestion
public void process(ProcessContext ctxt) {}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406567)
Time Spent: 1h 10m (was: 1h)
> Properly populate pipeline proto requirements.
> --
>
> Key: BEAM-9340
> URL: https://issues.apache.org/jira/browse/BEAM-9340
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[
https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=406569=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406569
]
ASF GitHub Bot logged work on BEAM-9340:
Author: ASF GitHub Bot
Created on: 19/Mar/20 20:49
Start Date: 19/Mar/20 20:49
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11165: [BEAM-9340]
Populate requirements for Java.
URL: https://github.com/apache/beam/pull/11165#discussion_r395310152
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
##
@@ -86,25 +91,30 @@ public static SdkComponents create(RunnerApi.Components
components) {
}
public static SdkComponents create(PipelineOptions options) {
-SdkComponents sdkComponents = new
SdkComponents(RunnerApi.Components.getDefaultInstance(), "");
+SdkComponents sdkComponents =
+new SdkComponents(RunnerApi.Components.getDefaultInstance(), null, "");
PortablePipelineOptions portablePipelineOptions =
options.as(PortablePipelineOptions.class);
sdkComponents.registerEnvironment(
Environments.createOrGetDefaultEnvironment(portablePipelineOptions));
return sdkComponents;
}
- private SdkComponents(RunnerApi.Components components, String newIdPrefix) {
+ private SdkComponents(
+ Components components, Collection requirements, String
newIdPrefix) {
Review comment:
```suggestion
Components components, @Nullable Collection requirements,
String newIdPrefix) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406569)
Time Spent: 1h 20m (was: 1h 10m)
> Properly populate pipeline proto requirements.
> --
>
> Key: BEAM-9340
> URL: https://issues.apache.org/jira/browse/BEAM-9340
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=406566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406566 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 19/Mar/20 20:49 Start Date: 19/Mar/20 20:49 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#discussion_r395306659 ## File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java ## @@ -124,6 +124,15 @@ public static final String SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN = "beam:transform:sdf_process_sized_element_and_restrictions:v1"; + public static final String REQUIRES_STATEFUL_PROCESSING_URN = + getUrn(RunnerApi.StandardRequirements.Enum.REQUIRES_STATEFUL_PROCESSING); Review comment: Please define string constants and then validate that they match in the static initializer. This allows for the string constants to be used in switch/case statements since they are compile time constants. I have pr/11162 which duplicates this effort, which ever goes in second needs to get updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 406566) Time Spent: 1h (was: 50m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=405820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405820 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 18/Mar/20 23:22 Start Date: 18/Mar/20 23:22 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=391804=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391804 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 24/Feb/20 16:24 Start Date: 24/Feb/20 16:24 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10909: [BEAM-9340] Populate requirements for Python DoFn properties. URL: https://github.com/apache/beam/pull/10909 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391804) Time Spent: 40m (was: 0.5h) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=391802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391802 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 24/Feb/20 16:23 Start Date: 24/Feb/20 16:23 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10909: [BEAM-9340] Populate requirements for Python DoFn properties. URL: https://github.com/apache/beam/pull/10909#issuecomment-590415489 Rebased to deal with merge conflicts. Merging now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391802) Time Spent: 0.5h (was: 20m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=390179=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390179 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 20/Feb/20 19:18 Start Date: 20/Feb/20 19:18 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10909: [BEAM-9340] Populate requirements for Python DoFn properties. URL: https://github.com/apache/beam/pull/10909#issuecomment-589261538 Thanks for the quick review. Responded about (a) on the list for more visibility. I think (c) is the way to go, runners can become more sophisticated about conditions in which they can satisfy requirements and SDKs can become more sophisticated about conditions in which they require them based on the pipeline context. (BTW, I did think about (b) but wanted to try to keep the requirements as local as possible, and make it possible to declare requirements that are not explicit in the pipeline proto itself.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390179) Time Spent: 20m (was: 10m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=389705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389705 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 20/Feb/20 00:33 Start Date: 20/Feb/20 00:33 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10909: [BEAM-9340] Populate requirements for Python DoFn properties. URL: https://github.com/apache/beam/pull/10909 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[