[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=306884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306884 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 05/Sep/19 04:27 Start Date: 05/Sep/19 04:27 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9478: [BEAM-7909] fix python container image name URL: https://github.com/apache/beam/pull/9478 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 306884) Time Spent: 12h (was: 11h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.16.0 > > Time Spent: 12h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=306847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306847 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 05/Sep/19 02:42 Start Date: 05/Sep/19 02:42 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9478: [BEAM-7909] fix python container image name URL: https://github.com/apache/beam/pull/9478#issuecomment-528171699 R: @aaltay I found a typo when running post commit tests. Is it possible to review and merge? Thanks, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 306847) Time Spent: 11h 50m (was: 11h 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.16.0 > > Time Spent: 11h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=306776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306776 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 04/Sep/19 23:19 Start Date: 04/Sep/19 23:19 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9478: [BEAM-7909] fix python container image name URL: https://github.com/apache/beam/pull/9478#issuecomment-528128877 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 306776) Time Spent: 11h 40m (was: 11.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.16.0 > > Time Spent: 11h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=306731=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306731 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 04/Sep/19 21:25 Start Date: 04/Sep/19 21:25 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9478: [BEAM-7909] fix python container image name URL: https://github.com/apache/beam/pull/9478#issuecomment-528093491 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 306731) Time Spent: 11.5h (was: 11h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.16.0 > > Time Spent: 11.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=306728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306728 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 04/Sep/19 21:20 Start Date: 04/Sep/19 21:20 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9478: [BEAM-7909] fix python container image name URL: https://github.com/apache/beam/pull/9478 Fix Python container image name variable. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=306580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306580 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 04/Sep/19 17:52 Start Date: 04/Sep/19 17:52 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 306580) Time Spent: 11h 10m (was: 11h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 11h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=305846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305846 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 03/Sep/19 20:07 Start Date: 03/Sep/19 20:07 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-527618100 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305846) Time Spent: 11h (was: 10h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=305828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305828 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 03/Sep/19 19:53 Start Date: 03/Sep/19 19:53 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-527612779 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305828) Time Spent: 10h 50m (was: 10h 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=305815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305815 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 03/Sep/19 19:20 Start Date: 03/Sep/19 19:20 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-527601559 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305815) Time Spent: 10h 40m (was: 10.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=304915=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-304915 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 31/Aug/19 22:46 Start Date: 31/Aug/19 22:46 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526870615 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 304915) Time Spent: 10.5h (was: 10h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 10.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=304908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-304908 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 31/Aug/19 22:10 Start Date: 31/Aug/19 22:10 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526868870 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 304908) Time Spent: 10h 10m (was: 10h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=304907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-304907 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 31/Aug/19 22:10 Start Date: 31/Aug/19 22:10 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526868850 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 304907) Time Spent: 10h (was: 9h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 10h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=304880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-304880 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 31/Aug/19 20:41 Start Date: 31/Aug/19 20:41 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526863771 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 304880) Time Spent: 9h 40m (was: 9.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=304881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-304881 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 31/Aug/19 20:41 Start Date: 31/Aug/19 20:41 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526863788 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 304881) Time Spent: 9h 50m (was: 9h 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 9h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=304019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-304019 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 30/Aug/19 03:23 Start Date: 30/Aug/19 03:23 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526440933 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 304019) Time Spent: 9.5h (was: 9h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=303982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303982 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 30/Aug/19 00:46 Start Date: 30/Aug/19 00:46 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526411707 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 303982) Time Spent: 9h 20m (was: 9h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=303963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303963 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 30/Aug/19 00:06 Start Date: 30/Aug/19 00:06 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526404789 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 303963) Time Spent: 9h 10m (was: 9h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=303956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303956 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 29/Aug/19 23:53 Start Date: 29/Aug/19 23:53 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526402467 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 303956) Time Spent: 9h (was: 8h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=303330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303330 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 29/Aug/19 03:18 Start Date: 29/Aug/19 03:18 Worklog Time Spent: 10m Work Description: aaltay commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-526004103 @markflyhigh @ibzib could you help with the test failure here? What can we do? Can we disable some tests? Could we break up various python versions so that we do not stack up flakiness? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 303330) Time Spent: 8h 50m (was: 8h 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=302172=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302172 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 27/Aug/19 16:28 Start Date: 27/Aug/19 16:28 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-525379907 > And python portable failed on streaming wordcount. please fix. This test is blocked by https://issues.apache.org/jira/browse/BEAM-7993. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302172) Time Spent: 8h 40m (was: 8.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=302169=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302169 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 27/Aug/19 16:27 Start Date: 27/Aug/19 16:27 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-525379575 Run JavaBeamZetaSQL PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302169) Time Spent: 8h 20m (was: 8h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=302170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302170 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 27/Aug/19 16:27 Start Date: 27/Aug/19 16:27 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] Python3 docker containers URL: https://github.com/apache/beam/pull/9351#issuecomment-525379640 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302170) Time Spent: 8.5h (was: 8h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=301482=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-301482 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 26/Aug/19 20:43 Start Date: 26/Aug/19 20:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#issuecomment-525024490 > Thank you! > > @Hannah-Jiang could you also file another JIRA issue for finding a process to keep this list in sync with setup.py. Thank you Ahmet for reviewing and merging it. I created a ticket: https://issues.apache.org/jira/browse/BEAM-8094 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 301482) Time Spent: 8h 10m (was: 8h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=301394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-301394 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 26/Aug/19 17:58 Start Date: 26/Aug/19 17:58 Worklog Time Spent: 10m Work Description: robertwb commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-524962193 Would this PR better be named "Python 3 containers" or similar? I don't see customized container specific code here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 301394) Time Spent: 8h (was: 7h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=301348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-301348 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 26/Aug/19 16:58 Start Date: 26/Aug/19 16:58 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-524939709 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 301348) Time Spent: 7h 50m (was: 7h 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300611=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300611 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 23:14 Start Date: 23/Aug/19 23:14 Worklog Time Spent: 10m Work Description: yifanzou commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-524489675 And python portable failed on streaming wordcount. please fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300611) Time Spent: 7h 40m (was: 7.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300610 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 23:11 Start Date: 23/Aug/19 23:11 Worklog Time Spent: 10m Work Description: yifanzou commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-524489235 LGTM. Thanks, Hannah. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300610) Time Spent: 7.5h (was: 7h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300485 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 20:18 Start Date: 23/Aug/19 20:18 Worklog Time Spent: 10m Work Description: aaltay commented on issue #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#issuecomment-524448436 Thank you! @Hannah-Jiang could you also file another JIRA issue for finding a process to keep this list in sync with setup.py. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300485) Time Spent: 7h 20m (was: 7h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300484 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 20:17 Start Date: 23/Aug/19 20:17 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300484) Time Spent: 7h 10m (was: 7h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300421=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300421 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 18:01 Start Date: 23/Aug/19 18:01 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#discussion_r317242610 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -28,46 +28,46 @@ avro-python3==1.8.2;python_version>="3.4" fastavro==0.21.4 crcmod==1.7 dill==0.2.9 -future==0.16.0 +future==0.17.1 futures==3.2.0;python_version<"3.0" -grpcio==1.15.0 -hdfs==2.1.0 -httplib2==0.9.2 +grpcio==1.21.1 +hdfs==2.5.6 +httplib2==0.12.0 mock==2.0.0 oauth2client==3.0.0 -protobuf==3.6.1 -pyarrow==0.11.1 -pydot==1.2.4 -pyparsing==2.3.1 -pytz==2018.4 +protobuf==3.8.0 +pyarrow==0.13.0 +pydot==1.4.1 +pyparsing==2.4.0 Review comment: Yes. Thanks for more details. I created a ticket https://issues.apache.org/jira/browse/BEAM-8084. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300421) Time Spent: 6h 50m (was: 6h 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300422=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300422 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 18:01 Start Date: 23/Aug/19 18:01 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#issuecomment-524408227 All tests passed, can we merge it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300422) Time Spent: 7h (was: 6h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300405=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300405 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 17:35 Start Date: 23/Aug/19 17:35 Worklog Time Spent: 10m Work Description: aaltay commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-524399949 @yifanzou could you PTAL? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300405) Time Spent: 6h 40m (was: 6.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300403 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 17:34 Start Date: 23/Aug/19 17:34 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#discussion_r317232320 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -28,46 +28,46 @@ avro-python3==1.8.2;python_version>="3.4" fastavro==0.21.4 crcmod==1.7 dill==0.2.9 -future==0.16.0 +future==0.17.1 futures==3.2.0;python_version<"3.0" -grpcio==1.15.0 -hdfs==2.1.0 -httplib2==0.9.2 +grpcio==1.21.1 +hdfs==2.5.6 +httplib2==0.12.0 mock==2.0.0 oauth2client==3.0.0 -protobuf==3.6.1 -pyarrow==0.11.1 -pydot==1.2.4 -pyparsing==2.3.1 -pytz==2018.4 +protobuf==3.8.0 +pyarrow==0.13.0 +pydot==1.4.1 +pyparsing==2.4.0 Review comment: Beam has a list of dependencies (listed in setup.py). These are Beam's direct dependency. Each of these direct dependency will have their own list of dependencies. These are transitive dependencies of Beam. Beam depends on them indirectly and in setup.py Beam does not list them. In a worker environment it is beneficial to lock version of all Beam dependencies including the indirect ones. Does it answer your question? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300403) Time Spent: 6.5h (was: 6h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=300380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300380 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 23/Aug/19 16:57 Start Date: 23/Aug/19 16:57 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#discussion_r317219578 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -28,46 +28,46 @@ avro-python3==1.8.2;python_version>="3.4" fastavro==0.21.4 crcmod==1.7 dill==0.2.9 -future==0.16.0 +future==0.17.1 futures==3.2.0;python_version<"3.0" -grpcio==1.15.0 -hdfs==2.1.0 -httplib2==0.9.2 +grpcio==1.21.1 +hdfs==2.5.6 +httplib2==0.12.0 mock==2.0.0 oauth2client==3.0.0 -protobuf==3.6.1 -pyarrow==0.11.1 -pydot==1.2.4 -pyparsing==2.3.1 -pytz==2018.4 +protobuf==3.8.0 +pyarrow==0.13.0 +pydot==1.4.1 +pyparsing==2.4.0 Review comment: I removed `pyparsing` from the list. I haven't found a ticket for listing all transitive dependencies. Could you please explain what it is? I will create a ticket when I have more details. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300380) Time Spent: 6h 20m (was: 6h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298896 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 21/Aug/19 18:53 Start Date: 21/Aug/19 18:53 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#discussion_r316349043 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -28,46 +28,46 @@ avro-python3==1.8.2;python_version>="3.4" fastavro==0.21.4 crcmod==1.7 dill==0.2.9 -future==0.16.0 +future==0.17.1 futures==3.2.0;python_version<"3.0" -grpcio==1.15.0 -hdfs==2.1.0 -httplib2==0.9.2 +grpcio==1.21.1 +hdfs==2.5.6 +httplib2==0.12.0 mock==2.0.0 oauth2client==3.0.0 -protobuf==3.6.1 -pyarrow==0.11.1 -pydot==1.2.4 -pyparsing==2.3.1 -pytz==2018.4 +protobuf==3.8.0 +pyarrow==0.13.0 +pydot==1.4.1 +pyparsing==2.4.0 Review comment: We already have pydot in the list. And we do not list other transitive dependencies. Should we remove this from here? Separately, it is actually a good idea to list all transitive dependencies. There was a mailing list discussion on this. I do not know if there is a JIRA issue about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298896) Time Spent: 6h 10m (was: 6h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298856 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 21/Aug/19 16:59 Start Date: 21/Aug/19 16:59 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-523551551 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298856) Time Spent: 6h (was: 5h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298833 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 21/Aug/19 16:30 Start Date: 21/Aug/19 16:30 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r315945392 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -67,7 +67,7 @@ pandas==0.23.4 protorpc==0.11.1 python-gflags==3.0.6 setuptools<=39.1.0 # requirement for Tensorflow. -tensorflow==1.11.0 +tensorflow==1.14.0 Review comment: Keep `tensorflow` and `pyyaml` changes here, because previous versions are not compatible with py37 and py37 tests cannot pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298833) Time Spent: 5h 50m (was: 5h 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298318 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 23:51 Start Date: 20/Aug/19 23:51 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#issuecomment-523238910 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298318) Time Spent: 5h 40m (was: 5.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298309 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 23:29 Start Date: 20/Aug/19 23:29 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r315945392 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -67,7 +67,7 @@ pandas==0.23.4 protorpc==0.11.1 python-gflags==3.0.6 setuptools<=39.1.0 # requirement for Tensorflow. -tensorflow==1.11.0 +tensorflow==1.14.0 Review comment: Keep these two changes here, because previous versions are not compatible with py37 and py37 tests cannot pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298309) Time Spent: 5.5h (was: 5h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298308 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 23:23 Start Date: 20/Aug/19 23:23 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r315943981 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -28,46 +28,46 @@ avro-python3==1.8.2;python_version>="3.4" fastavro==0.21.4 crcmod==1.7 dill==0.2.9 -future==0.16.0 +future==0.17.1 Review comment: I created a separate PR(#9388) for version upgrade and rolled back here. Thanks for bringing it up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298308) Time Spent: 5h 20m (was: 5h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298304=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298304 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 23:21 Start Date: 20/Aug/19 23:21 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388#issuecomment-523232590 R: @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298304) Time Spent: 5h 10m (was: 5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298303 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 23:21 Start Date: 20/Aug/19 23:21 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9388: [BEAM-7909] upgrade python lib versions to match to dataflow worker URL: https://github.com/apache/beam/pull/9388 This PR is upgrading python library versions to match versions installed at dataflow workers. (https://cloud.google.com/dataflow/docs/concepts/sdk-worker-dependencies) In order to make it easy to track down in case anything is broken, I created a separate PR for this change. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298079 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 18:06 Start Date: 20/Aug/19 18:06 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r315826000 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -28,46 +28,46 @@ avro-python3==1.8.2;python_version>="3.4" fastavro==0.21.4 crcmod==1.7 dill==0.2.9 -future==0.16.0 +future==0.17.1 Review comment: There are enough changes here that it feels like this should be a different PR or at least commit (especially in tracking down if this breaks anything). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298079) Time Spent: 4h 40m (was: 4.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298071=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298071 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 17:50 Start Date: 20/Aug/19 17:50 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-523124200 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298071) Time Spent: 4.5h (was: 4h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298069=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298069 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 17:49 Start Date: 20/Aug/19 17:49 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r315822376 ## File path: sdks/python/setup.py ## @@ -125,15 +125,15 @@ def get_version(): 'pytz>=2018.3', # [BEAM-5628] Beam VCF IO is not supported in Python 3. 'pyvcf>=0.6.8,<0.7.0; python_version < "3.0"', -'pyyaml>=3.12,<4.0.0', +'pyyaml>=3.13,<4.0.0', Review comment: oh, this was by mistake, 3.13 is in the range, so no need to change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298069) Time Spent: 4h 20m (was: 4h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298053
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 20/Aug/19 17:26
Start Date: 20/Aug/19 17:26
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9351:
[BEAM-7909] support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315812097
##
File path: sdks/python/container/build.gradle
##
@@ -60,15 +51,13 @@ golang {
}
}
-docker {
- name containerImageName(name: "python")
- files "./build"
+task buildDocker {
Review comment:
I decided to go with `buildAll`. `build` is already used, so cannot use it
here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 298053)
Time Spent: 4h 10m (was: 4h)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=298003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298003
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 20/Aug/19 16:17
Start Date: 20/Aug/19 16:17
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9351:
[BEAM-7909] support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315782138
##
File path: sdks/python/container/py35/build.gradle
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'base'
+ id 'org.apache.beam.module'
+}
+applyDockerNature()
+
+description = "Apache Beam :: SDKs :: Python :: Container :: Python 35
Container"
+
+configurations {
+ sdkSourceTarball
+ sdkHarnessLauncher
+}
+
+dependencies {
+ sdkSourceTarball project(path: ":sdks:python", configuration: "distTarBall")
+ sdkHarnessLauncher project(path: ":sdks:python:container", configuration:
"sdkHarnessLauncher")
+}
+
+task copyDockerfileDependencies(type: Copy) {
Review comment:
We are sharing the same files under `sdks/python/container`. However, we
need to copy the files to a temp directory when build images, this task is
doing this part.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 298003)
Time Spent: 4h (was: 3h 50m)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297623
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 20/Aug/19 05:29
Start Date: 20/Aug/19 05:29
Worklog Time Spent: 10m
Work Description: yifanzou commented on pull request #9351: [BEAM-7909]
support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315511451
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -84,16 +84,10 @@ def __init__(self):
@staticmethod
def default_docker_image():
if 'USER' in os.environ:
- if sys.version_info[0] == 2:
-version_suffix = ''
- elif sys.version_info[0:2] == (3, 5):
-version_suffix = '3'
- else:
-version_suffix = '3'
-# TODO(BEAM-7474): Use an image which has correct Python minor version.
-logging.warning('Make sure that locally built Python SDK docker image '
-'has Python %d.%d interpreter. See also: BEAM-7474.' %
(
-sys.version_info[0], sys.version_info[1]))
+ version_suffix = ''.join([str(i) for i in sys.version_info[0:2]])
Review comment:
+1 with Ahmet for not skipping these tests. But, failing a test which
because of the user didn't build images beforehand does not make sense to me.
Could we add warnings in the tests if the default images are missing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297623)
Time Spent: 3h 50m (was: 3h 40m)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297626
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 20/Aug/19 05:29
Start Date: 20/Aug/19 05:29
Worklog Time Spent: 10m
Work Description: yifanzou commented on pull request #9351: [BEAM-7909]
support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315507170
##
File path: sdks/python/test-suites/portable/py2/build.gradle
##
@@ -29,7 +29,7 @@ addPortableWordCountTasks()
task preCommitPy2() {
dependsOn ':runners:flink:1.5:job-server-container:docker'
- dependsOn ':sdks:python:container:docker'
+ dependsOn ':sdks:python:container:buildDocker'
Review comment:
Could we only depend on ':sdks:python:container:py2:docker rather than build
all images? Same below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297626)
Time Spent: 3h 50m (was: 3h 40m)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297625
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 20/Aug/19 05:29
Start Date: 20/Aug/19 05:29
Worklog Time Spent: 10m
Work Description: yifanzou commented on pull request #9351: [BEAM-7909]
support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315506168
##
File path: sdks/python/container/build.gradle
##
@@ -60,15 +51,13 @@ golang {
}
}
-docker {
- name containerImageName(name: "python")
- files "./build"
+task buildDocker {
Review comment:
We talked offline and come out with this approach to build IMAGES WITH ALL
PY VERSIONS in a singe task. `build` is good to me, but I am not sure if that
is a keyword in gradle dsl. Alternatively, we could call it `buildAll`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297625)
Time Spent: 3h 50m (was: 3h 40m)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297624=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297624
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 20/Aug/19 05:29
Start Date: 20/Aug/19 05:29
Worklog Time Spent: 10m
Work Description: yifanzou commented on pull request #9351: [BEAM-7909]
support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315507918
##
File path: sdks/python/container/py35/build.gradle
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'base'
+ id 'org.apache.beam.module'
+}
+applyDockerNature()
+
+description = "Apache Beam :: SDKs :: Python :: Container :: Python 35
Container"
+
+configurations {
+ sdkSourceTarball
+ sdkHarnessLauncher
+}
+
+dependencies {
+ sdkSourceTarball project(path: ":sdks:python", configuration: "distTarBall")
+ sdkHarnessLauncher project(path: ":sdks:python:container", configuration:
"sdkHarnessLauncher")
+}
+
+task copyDockerfileDependencies(type: Copy) {
Review comment:
Do we have to write separate `copyDockerfileDependencies` and
`copyDockerfileDependencies` in py2|35|36|37? Could we try to extract them into
the parent build.gradle to make future maintenance easier?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297624)
Time Spent: 3h 50m (was: 3h 40m)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297531
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 20/Aug/19 01:19
Start Date: 20/Aug/19 01:19
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9351: [BEAM-7909]
support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315472687
##
File path: sdks/python/container/build.gradle
##
@@ -60,15 +51,13 @@ golang {
}
}
-docker {
- name containerImageName(name: "python")
- files "./build"
+task buildDocker {
Review comment:
This change sounds good.
We could also rename it to just `build` . It would make sense to call it
like `./gradlew -p sdks/python/container build`
/cc @yifanzou on gradle related questions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297531)
Time Spent: 3h 40m (was: 3.5h)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297529
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 20/Aug/19 01:19
Start Date: 20/Aug/19 01:19
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9351: [BEAM-7909]
support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315472495
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -84,16 +84,10 @@ def __init__(self):
@staticmethod
def default_docker_image():
if 'USER' in os.environ:
- if sys.version_info[0] == 2:
-version_suffix = ''
- elif sys.version_info[0:2] == (3, 5):
-version_suffix = '3'
- else:
-version_suffix = '3'
-# TODO(BEAM-7474): Use an image which has correct Python minor version.
-logging.warning('Make sure that locally built Python SDK docker image '
-'has Python %d.%d interpreter. See also: BEAM-7474.' %
(
-sys.version_info[0], sys.version_info[1]))
+ version_suffix = ''.join([str(i) for i in sys.version_info[0:2]])
Review comment:
Could you chat with @alanmyrvold or @yifanzou on what would be the best
testing practices here?
I think we should not skip tests if possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297529)
Time Spent: 3.5h (was: 3h 20m)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297530 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 20/Aug/19 01:19 Start Date: 20/Aug/19 01:19 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r315472846 ## File path: sdks/python/setup.py ## @@ -125,15 +125,15 @@ def get_version(): 'pytz>=2018.3', # [BEAM-5628] Beam VCF IO is not supported in Python 3. 'pyvcf>=0.6.8,<0.7.0; python_version < "3.0"', -'pyyaml>=3.12,<4.0.0', +'pyyaml>=3.13,<4.0.0', Review comment: Why do we need this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297530) Time Spent: 3.5h (was: 3h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297503=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297503 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 19/Aug/19 23:58 Start Date: 19/Aug/19 23:58 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-522798308 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297503) Time Spent: 3h 20m (was: 3h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297500=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297500 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 19/Aug/19 23:53 Start Date: 19/Aug/19 23:53 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-522797346 @aaltay , I made some changes, can you please take a look when you have time? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297500) Time Spent: 3h (was: 2h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297489
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 19/Aug/19 23:23
Start Date: 19/Aug/19 23:23
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9351:
[BEAM-7909] support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r315451829
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -84,16 +84,10 @@ def __init__(self):
@staticmethod
def default_docker_image():
if 'USER' in os.environ:
- if sys.version_info[0] == 2:
-version_suffix = ''
- elif sys.version_info[0:2] == (3, 5):
-version_suffix = '3'
- else:
-version_suffix = '3'
-# TODO(BEAM-7474): Use an image which has correct Python minor version.
-logging.warning('Make sure that locally built Python SDK docker image '
-'has Python %d.%d interpreter. See also: BEAM-7474.' %
(
-sys.version_info[0], sys.version_info[1]))
+ version_suffix = ''.join([str(i) for i in sys.version_info[0:2]])
Review comment:
`PortableRunnerTestWithLocalDocker` class at `portable_runner_test.py` will
test this function. However, in order to run these tests, users should build
docker images first, which I don't think all users need to do. Do we want to
skip these tests at local? Is there any good practice I can refer?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297489)
Time Spent: 2h 40m (was: 2.5h)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297487=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297487 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 19/Aug/19 23:19 Start Date: 19/Aug/19 23:19 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r315451101 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -67,7 +67,7 @@ pandas==0.23.4 protorpc==0.11.1 python-gflags==3.0.6 setuptools<=39.1.0 # requirement for Tensorflow. -tensorflow==1.11.0 +tensorflow==1.13.1 Review comment: I compared with libraries installed by default at dataflow workers and matched versions to dataflow workers. For libraries whose versions are not in the range of `setup.py`, I updated version range accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297487) Time Spent: 2.5h (was: 2h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=297328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297328 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 19/Aug/19 17:44 Start Date: 19/Aug/19 17:44 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-522682235 R: @yifanzou This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297328) Time Spent: 2h 20m (was: 2h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[
https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=296668=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296668
]
ASF GitHub Bot logged work on BEAM-7909:
Author: ASF GitHub Bot
Created on: 16/Aug/19 23:45
Start Date: 16/Aug/19 23:45
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9351: [BEAM-7909]
support customized container for Python
URL: https://github.com/apache/beam/pull/9351#discussion_r314920513
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -84,16 +84,10 @@ def __init__(self):
@staticmethod
def default_docker_image():
if 'USER' in os.environ:
- if sys.version_info[0] == 2:
-version_suffix = ''
- elif sys.version_info[0:2] == (3, 5):
-version_suffix = '3'
- else:
-version_suffix = '3'
-# TODO(BEAM-7474): Use an image which has correct Python minor version.
-logging.warning('Make sure that locally built Python SDK docker image '
-'has Python %d.%d interpreter. See also: BEAM-7474.' %
(
-sys.version_info[0], sys.version_info[1]))
+ version_suffix = ''.join([str(i) for i in sys.version_info[0:2]])
Review comment:
This changes the behavior for python 2. I agree with the change, let's make
sure that we test it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296668)
Time Spent: 2h 10m (was: 2h)
> Write integration tests to test customized containers
> -
>
> Key: BEAM-7909
> URL: https://issues.apache.org/jira/browse/BEAM-7909
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=296667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296667 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 16/Aug/19 23:45 Start Date: 16/Aug/19 23:45 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r314921003 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -67,7 +67,7 @@ pandas==0.23.4 protorpc==0.11.1 python-gflags==3.0.6 setuptools<=39.1.0 # requirement for Tensorflow. -tensorflow==1.11.0 +tensorflow==1.13.1 Review comment: Could you upgrade these dependencies to match (a) setup.py (latest in the range if possible) and (b) https://cloud.google.com/dataflow/docs/concepts/sdk-worker-dependencies We should probably try to figure out a way to keep version in sync. This was recently discussed in the mailing list. For ideas you can look at that (https://lists.apache.org/thread.html/e174967bab6c224db40f66af74e7d07466812cc6b78189e55da88cb0@%3Cdev.beam.apache.org%3E) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296667) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=29=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-29 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 16/Aug/19 23:45 Start Date: 16/Aug/19 23:45 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r314920538 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -84,16 +84,10 @@ def __init__(self): @staticmethod def default_docker_image(): if 'USER' in os.environ: - if sys.version_info[0] == 2: -version_suffix = '' - elif sys.version_info[0:2] == (3, 5): -version_suffix = '3' - else: -version_suffix = '3' -# TODO(BEAM-7474): Use an image which has correct Python minor version. Review comment: You can probably close this issue as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 29) Time Spent: 2h (was: 1h 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=296484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296484 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 16/Aug/19 18:13 Start Date: 16/Aug/19 18:13 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#discussion_r314834051 ## File path: sdks/python/container/base_image_requirements.txt ## @@ -67,7 +67,7 @@ pandas==0.23.4 protorpc==0.11.1 python-gflags==3.0.6 setuptools<=39.1.0 # requirement for Tensorflow. -tensorflow==1.11.0 +tensorflow==1.13.1 Review comment: `pyyaml` and `tensorflow` version upgrade is because of py3.7. Previous versions don't work for py3.7. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296484) Time Spent: 1h 50m (was: 1h 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295877 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 23:25 Start Date: 15/Aug/19 23:25 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-521831467 R: @aaltay Hi, Ahmet, I created a PR, can we review it and make sure I am on a correct track? Thanks, Hannah This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295877) Time Spent: 1h 40m (was: 1.5h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295857 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 23:07 Start Date: 15/Aug/19 23:07 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [WIP][BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-521794846 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295857) Time Spent: 1h 20m (was: 1h 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295858 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 23:07 Start Date: 15/Aug/19 23:07 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [WIP][BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-521822580 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295858) Time Spent: 1.5h (was: 1h 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295811 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 22:41 Start Date: 15/Aug/19 22:41 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [WIP][BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-521822580 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295811) Time Spent: 1h 10m (was: 1h) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295757 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 20:57 Start Date: 15/Aug/19 20:57 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9351: [WIP][BEAM-7909] support customized container for Python URL: https://github.com/apache/beam/pull/9351#issuecomment-521794846 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295757) Time Spent: 1h (was: 50m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295662 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 18:43 Start Date: 15/Aug/19 18:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9335: [WIP][BEAM-7909] customized containers for python URL: https://github.com/apache/beam/pull/9335 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295662) Time Spent: 50m (was: 40m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295661 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 18:43 Start Date: 15/Aug/19 18:43 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9351: [WIP][BEAM-7909] support customized container URL: https://github.com/apache/beam/pull/9351 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295525=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295525 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 16:17 Start Date: 15/Aug/19 16:17 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9335: [WIP][BEAM-7909] customized containers for python URL: https://github.com/apache/beam/pull/9335#issuecomment-521465832 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295525) Time Spent: 0.5h (was: 20m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=295094=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295094 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 15/Aug/19 00:16 Start Date: 15/Aug/19 00:16 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #9335: [WIP][BEAM-7909] customized containers for python URL: https://github.com/apache/beam/pull/9335#issuecomment-521465832 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295094) Time Spent: 20m (was: 10m) > Write integration tests to test customized containers > - > > Key: BEAM-7909 > URL: https://issues.apache.org/jira/browse/BEAM-7909 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7909) Write integration tests to test customized containers
[ https://issues.apache.org/jira/browse/BEAM-7909?focusedWorklogId=294287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294287 ] ASF GitHub Bot logged work on BEAM-7909: Author: ASF GitHub Bot Created on: 13/Aug/19 23:54 Start Date: 13/Aug/19 23:54 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #9335: [WIP][BEAM-7909] customized containers for python URL: https://github.com/apache/beam/pull/9335 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[