[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=456349&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-456349 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 08/Jul/20 21:41 Start Date: 08/Jul/20 21:41 Worklog Time Spent: 10m Work Description: lostluck merged pull request #12193: URL: https://github.com/apache/beam/pull/12193 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 456349) Time Spent: 4.5h (was: 4h 20m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, sdk-go >Reporter: Kyle Weaver >Priority: P3 > Time Spent: 4.5h > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] > Update 11/12: this is complete for Python and Java, Go remains. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=456279&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-456279 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 08/Jul/20 18:18 Start Date: 08/Jul/20 18:18 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #12193: URL: https://github.com/apache/beam/pull/12193#issuecomment-655678835 Thanks for the pr! Cheers. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 456279) Time Spent: 4h 20m (was: 4h 10m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, sdk-go >Reporter: Kyle Weaver >Priority: P3 > Time Spent: 4h 20m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] > Update 11/12: this is complete for Python and Java, Go remains. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=456215&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-456215 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 08/Jul/20 15:45 Start Date: 08/Jul/20 15:45 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #12193: URL: https://github.com/apache/beam/pull/12193#issuecomment-655600276 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 456215) Time Spent: 4h 10m (was: 4h) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, sdk-go >Reporter: Kyle Weaver >Priority: P3 > Time Spent: 4h 10m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] > Update 11/12: this is complete for Python and Java, Go remains. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=455813&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-455813 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 08/Jul/20 02:22 Start Date: 08/Jul/20 02:22 Worklog Time Spent: 10m Work Description: bamnet commented on pull request #12193: URL: https://github.com/apache/beam/pull/12193#issuecomment-655241403 R: @lostluck /fyi @ibzib This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 455813) Time Spent: 4h (was: 3h 50m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, sdk-go >Reporter: Kyle Weaver >Priority: P3 > Time Spent: 4h > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] > Update 11/12: this is complete for Python and Java, Go remains. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=455811&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-455811 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 08/Jul/20 02:21 Start Date: 08/Jul/20 02:21 Worklog Time Spent: 10m Work Description: bamnet opened a new pull request #12193: URL: https://github.com/apache/beam/pull/12193 If the region flag is unset, attempt to extract a region from $CLOUDSDK_COMPUTE_REGION and the gcloud SDK. This mirrors the [Java](https://github.com/apache/beam/blob/64e69d68f85903d94170fa2efc5a1633d82da589/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java#L218) and [Python](https://github.com/apache/beam/blob/72d9744affc6ff4a63816f737e18c4abe14f245f/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L1437) implementation. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [X] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [X] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 --- | --- | --- | --- | --- | --- | --- Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | --- Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build St
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=341437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341437 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 11/Nov/19 19:08 Start Date: 11/Nov/19 19:08 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10048: [BEAM-8472] test Java default GCP region URL: https://github.com/apache/beam/pull/10048#issuecomment-552573076 Going forward, it is much easier to review if you don't flatten the commits so I can see the usually trivial diff. I can flatten the commits when merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341437) Time Spent: 3h 40m (was: 3.5h) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=341436&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341436 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 11/Nov/19 19:08 Start Date: 11/Nov/19 19:08 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10048: [BEAM-8472] test Java default GCP region URL: https://github.com/apache/beam/pull/10048 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341436) Time Spent: 3.5h (was: 3h 20m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=341399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341399 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 11/Nov/19 18:09 Start Date: 11/Nov/19 18:09 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10048: [BEAM-8472] test Java default GCP region URL: https://github.com/apache/beam/pull/10048#issuecomment-552551911 > It looks like there is a checkstyle and spotbugs violation, please fix. Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341399) Time Spent: 3h 20m (was: 3h 10m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[
https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=340823&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-340823
]
ASF GitHub Bot logged work on BEAM-8472:
Author: ASF GitHub Bot
Created on: 08/Nov/19 23:43
Start Date: 08/Nov/19 23:43
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9974: [BEAM-8472] Get
default GCP region from gcloud (Java)
URL: https://github.com/apache/beam/pull/9974#discussion_r344407980
##
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
##
@@ -199,4 +199,10 @@ public void testDefaultStagingLocationUnset() {
thrown.expectMessage("Error constructing default value for
stagingLocation");
options.getStagingLocation();
}
+
+ @Test
+ public void testDefaultGcpRegion() {
+DataflowPipelineOptions options =
PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+assertEquals("us-central1", options.getRegion());
Review comment:
Thanks for catching this Luke. Filed #10048 to improve this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 340823)
Time Spent: 3h 10m (was: 3h)
> Get default GCP region from gcloud
> --
>
> Key: BEAM-8472
> URL: https://issues.apache.org/jira/browse/BEAM-8472
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Currently, we default to us-central1 if --region flag is not set. The Google
> Cloud SDK generally tries to get a default value in this case for
> convenience, which we should follow.
> [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=340821&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-340821 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 08/Nov/19 23:40 Start Date: 08/Nov/19 23:40 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10048: [BEAM-8472] test Java default GCP region URL: https://github.com/apache/beam/pull/10048 Refactor and add tests for `DefaultGcpRegionFactory`. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)[ Get default GCP region from gcloud
[
https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=340793&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-340793
]
ASF GitHub Bot logged work on BEAM-8472:
Author: ASF GitHub Bot
Created on: 08/Nov/19 22:50
Start Date: 08/Nov/19 22:50
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9974: [BEAM-8472]
Get default GCP region from gcloud (Java)
URL: https://github.com/apache/beam/pull/9974#discussion_r344397166
##
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
##
@@ -199,4 +199,10 @@ public void testDefaultStagingLocationUnset() {
thrown.expectMessage("Error constructing default value for
stagingLocation");
options.getStagingLocation();
}
+
+ @Test
+ public void testDefaultGcpRegion() {
+DataflowPipelineOptions options =
PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+assertEquals("us-central1", options.getRegion());
Review comment:
This test will fail on any machine where the `CLOUDSDK_COMPUTE_REGION` is
defined or `gcloud config compute/region` is set.
You can test your code by instantiating the DefaultGcpRegionFactory directly
and ensuring that it is factored in such a way where you can pass in the
environment map directly to it and similarly for testing the process execution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 340793)
Time Spent: 2h 50m (was: 2h 40m)
> Get default GCP region from gcloud
> --
>
> Key: BEAM-8472
> URL: https://issues.apache.org/jira/browse/BEAM-8472
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently, we default to us-central1 if --region flag is not set. The Google
> Cloud SDK generally tries to get a default value in this case for
> convenience, which we should follow.
> [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=340784&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-340784 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 08/Nov/19 22:24 Start Date: 08/Nov/19 22:24 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9974: [BEAM-8472] Get default GCP region from gcloud (Java) URL: https://github.com/apache/beam/pull/9974 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 340784) Time Spent: 2h 40m (was: 2.5h) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[
https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=338937&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338937
]
ASF GitHub Bot logged work on BEAM-8472:
Author: ASF GitHub Bot
Created on: 05/Nov/19 18:53
Start Date: 05/Nov/19 18:53
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9974: [BEAM-8472] Get
default GCP region from gcloud (Java)
URL: https://github.com/apache/beam/pull/9974#discussion_r342736635
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -357,6 +357,49 @@ public static DataflowRunner fromOptions(PipelineOptions
options) {
return new DataflowRunner(dataflowOptions);
}
+ /**
+ * Get a default value for Google Cloud region according to
+ * https://cloud.google.com/compute/docs/gcloud-compute/#default-properties.
If no other default
+ * can be found, returns "us-central1".
+ */
+ static String getDefaultGcpRegion() {
Review comment:
Made a default instance factory.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 338937)
Time Spent: 2.5h (was: 2h 20m)
> Get default GCP region from gcloud
> --
>
> Key: BEAM-8472
> URL: https://issues.apache.org/jira/browse/BEAM-8472
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Currently, we default to us-central1 if --region flag is not set. The Google
> Cloud SDK generally tries to get a default value in this case for
> convenience, which we should follow.
> [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[
https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=338842&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338842
]
ASF GitHub Bot logged work on BEAM-8472:
Author: ASF GitHub Bot
Created on: 05/Nov/19 16:25
Start Date: 05/Nov/19 16:25
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9974: [BEAM-8472]
Get default GCP region from gcloud (Java)
URL: https://github.com/apache/beam/pull/9974#discussion_r342650592
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -357,6 +357,49 @@ public static DataflowRunner fromOptions(PipelineOptions
options) {
return new DataflowRunner(dataflowOptions);
}
+ /**
+ * Get a default value for Google Cloud region according to
+ * https://cloud.google.com/compute/docs/gcloud-compute/#default-properties.
If no other default
+ * can be found, returns "us-central1".
+ */
+ static String getDefaultGcpRegion() {
Review comment:
Is there a reason why we aren't using the default instance factory to
configure the region instead of detecting and setting it here. This would mean
that all users of the pipeline option would get it and not just Dataflow.
See
[DefaultProjectFactory](https://github.com/apache/beam/blob/1c56a7627648c4f271dcfb35f0a6f2cfe303a18c/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java#L180)
for an example.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 338842)
Time Spent: 2h 20m (was: 2h 10m)
> Get default GCP region from gcloud
> --
>
> Key: BEAM-8472
> URL: https://issues.apache.org/jira/browse/BEAM-8472
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently, we default to us-central1 if --region flag is not set. The Google
> Cloud SDK generally tries to get a default value in this case for
> convenience, which we should follow.
> [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=338431&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338431 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 05/Nov/19 01:06 Start Date: 05/Nov/19 01:06 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9974: [BEAM-8472] Get default GCP region from gcloud (Java) URL: https://github.com/apache/beam/pull/9974#issuecomment-549617602 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 338431) Time Spent: 2h 10m (was: 2h) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[
https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=338410&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338410
]
ASF GitHub Bot logged work on BEAM-8472:
Author: ASF GitHub Bot
Created on: 04/Nov/19 23:42
Start Date: 04/Nov/19 23:42
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9974: [BEAM-8472] Get
default GCP region from gcloud (Java)
URL: https://github.com/apache/beam/pull/9974#discussion_r342324597
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -357,6 +357,50 @@ public static DataflowRunner fromOptions(PipelineOptions
options) {
return new DataflowRunner(dataflowOptions);
}
+ /**
+ * Get a default value for Google Cloud region according to
+ * https://cloud.google.com/compute/docs/gcloud-compute/#default-properties.
If no other default
+ * can be found, returns "us-central1".
+ */
+ static String getDefaultGcpRegion() {
+String environmentRegion = System.getenv("CLOUDSDK_COMPUTE_REGION");
+if (environmentRegion != null && !environmentRegion.isEmpty()) {
+ LOG.info("Using default GCP region {} from $CLOUDSDK_COMPUTE_REGION",
environmentRegion);
+ return environmentRegion;
+}
+try {
+ ProcessBuilder pb =
+ new ProcessBuilder(Arrays.asList("gcloud", "config", "get-value",
"compute/region"));
+ Process process = pb.start();
+ BufferedReader reader =
+ new BufferedReader(
+ new InputStreamReader(process.getInputStream(),
StandardCharsets.UTF_8));
+ BufferedReader errorReader =
+ new BufferedReader(
+ new InputStreamReader(process.getErrorStream(),
StandardCharsets.UTF_8));
+ process.waitFor(1, TimeUnit.SECONDS);
Review comment:
> Is 1 second enough?
Tests on my machine (in debug mode) indicated it took on average around 0.25
seconds, including reading the output stream after. I have bumped the timeout
to 2s just to make double sure.
> Should we check the return value of this call?
Not sure it matters, since I think it should interrupt the process after the
time limit has passed, but it couldn't hurt to check anyway.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 338410)
Time Spent: 2h (was: 1h 50m)
> Get default GCP region from gcloud
> --
>
> Key: BEAM-8472
> URL: https://issues.apache.org/jira/browse/BEAM-8472
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently, we default to us-central1 if --region flag is not set. The Google
> Cloud SDK generally tries to get a default value in this case for
> convenience, which we should follow.
> [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[
https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=337628&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337628
]
ASF GitHub Bot logged work on BEAM-8472:
Author: ASF GitHub Bot
Created on: 02/Nov/19 00:34
Start Date: 02/Nov/19 00:34
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9974: [BEAM-8472] Get
default GCP region from gcloud (Java)
URL: https://github.com/apache/beam/pull/9974#discussion_r341790105
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -357,6 +357,50 @@ public static DataflowRunner fromOptions(PipelineOptions
options) {
return new DataflowRunner(dataflowOptions);
}
+ /**
+ * Get a default value for Google Cloud region according to
+ * https://cloud.google.com/compute/docs/gcloud-compute/#default-properties.
If no other default
+ * can be found, returns "us-central1".
+ */
+ static String getDefaultGcpRegion() {
+String environmentRegion = System.getenv("CLOUDSDK_COMPUTE_REGION");
+if (environmentRegion != null && !environmentRegion.isEmpty()) {
+ LOG.info("Using default GCP region {} from $CLOUDSDK_COMPUTE_REGION",
environmentRegion);
+ return environmentRegion;
+}
+try {
+ ProcessBuilder pb =
+ new ProcessBuilder(Arrays.asList("gcloud", "config", "get-value",
"compute/region"));
+ Process process = pb.start();
+ BufferedReader reader =
+ new BufferedReader(
+ new InputStreamReader(process.getInputStream(),
StandardCharsets.UTF_8));
+ BufferedReader errorReader =
+ new BufferedReader(
+ new InputStreamReader(process.getErrorStream(),
StandardCharsets.UTF_8));
+ process.waitFor(1, TimeUnit.SECONDS);
Review comment:
- Is 1 second enough?
- Should we check the return value of this call? (IIUC it returns false for
timeout.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337628)
Time Spent: 1h 50m (was: 1h 40m)
> Get default GCP region from gcloud
> --
>
> Key: BEAM-8472
> URL: https://issues.apache.org/jira/browse/BEAM-8472
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Currently, we default to us-central1 if --region flag is not set. The Google
> Cloud SDK generally tries to get a default value in this case for
> convenience, which we should follow.
> [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=337623&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337623 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 01/Nov/19 23:54 Start Date: 01/Nov/19 23:54 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9974: [BEAM-8472] Get default GCP region from gcloud (Java) URL: https://github.com/apache/beam/pull/9974 Same as #9868 but more verbose :smile: As with the Python version, I've verified this works locally. This doesn't fit into a unit test because it requires special environment configuration and/or calling an external process, but nor does it seem to warrant an integration test. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [ Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=337105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337105 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 31/Oct/19 23:59 Start Date: 31/Oct/19 23:59 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9868: [BEAM-8472] get default GCP region option (Python) URL: https://github.com/apache/beam/pull/9868 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337105) Time Spent: 1.5h (was: 1h 20m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=337060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337060 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 31/Oct/19 21:19 Start Date: 31/Oct/19 21:19 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9868: [BEAM-8472] get default GCP region option (Python) URL: https://github.com/apache/beam/pull/9868#issuecomment-548573257 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337060) Time Spent: 1h 20m (was: 1h 10m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=336331&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336331 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 30/Oct/19 17:31 Start Date: 30/Oct/19 17:31 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9868: [BEAM-8472] get default GCP region option (Python) URL: https://github.com/apache/beam/pull/9868#issuecomment-548024197 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336331) Time Spent: 1h 10m (was: 1h) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=335791&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335791 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 29/Oct/19 22:10 Start Date: 29/Oct/19 22:10 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9868: [BEAM-8472] get default GCP region option (Python) URL: https://github.com/apache/beam/pull/9868#discussion_r340350119 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -22,6 +22,7 @@ import argparse import json import logging +import subprocess Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 335791) Time Spent: 1h (was: 50m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=334286&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334286 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 25/Oct/19 18:35 Start Date: 25/Oct/19 18:35 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9868: [BEAM-8472] get default GCP region option (Python) URL: https://github.com/apache/beam/pull/9868#discussion_r339186799 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -22,6 +22,7 @@ import argparse import json import logging +import subprocess Review comment: You may want to use https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/processes.py instead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 334286) Time Spent: 50m (was: 40m) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=333928&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-333928 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 25/Oct/19 07:14 Start Date: 25/Oct/19 07:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9868: [BEAM-8472] get default GCP region option (Python) URL: https://github.com/apache/beam/pull/9868#issuecomment-546233180 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 333928) Time Spent: 40m (was: 0.5h) > Get default GCP region from gcloud > -- > > Key: BEAM-8472 > URL: https://issues.apache.org/jira/browse/BEAM-8472 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Currently, we default to us-central1 if --region flag is not set. The Google > Cloud SDK generally tries to get a default value in this case for > convenience, which we should follow. > [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[
https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=333631&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-333631
]
ASF GitHub Bot logged work on BEAM-8472:
Author: ASF GitHub Bot
Created on: 24/Oct/19 18:14
Start Date: 24/Oct/19 18:14
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9868: [BEAM-8472] get
default GCP region option (Python)
URL: https://github.com/apache/beam/pull/9868#discussion_r338720101
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -500,6 +501,32 @@ def _add_argparse_args(cls, parser):
choices=['COST_OPTIMIZED', 'SPEED_OPTIMIZED'],
help='Set the Flexible Resource Scheduling mode')
+ def _get_default_gcp_region():
+"""Get a default value for Google Cloud region according to
+https://cloud.google.com/compute/docs/gcloud-compute/#default-properties.
+If no other default can be found, returns 'us-central1'.
+"""
+environment_region = os.environ.get('CLOUDSDK_COMPUTE_REGION')
+if environment_region:
+ logging.info('Using default GCP region %s from $CLOUDSDK_COMPUTE_REGION',
+ environment_region)
+ return environment_region
+try:
+ cmd = ['gcloud', 'config', 'get-value', 'compute/region']
+ output = subprocess.check_output(cmd).decode('utf-8').strip()
+ if '(unset)' not in output:
+logging.info('Using default GCP region %s from `%s`',
+output, ' '.join(cmd))
+return output
+except subprocess.CalledProcessError:
+ pass
+logging.warning(
+'--region not set; will default to us-central1. Future releases of '
+'Beam will require the user to set --region explicitly, or else have a
'
+'default set via the gcloud tool. '
+'https://cloud.google.com/compute/docs/regions-zones')
+return 'us-central1'
Review comment:
Actually it makes sense with the warning above. Yes, we can default for now
and will change in the future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 333631)
Time Spent: 0.5h (was: 20m)
> Get default GCP region from gcloud
> --
>
> Key: BEAM-8472
> URL: https://issues.apache.org/jira/browse/BEAM-8472
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, we default to us-central1 if --region flag is not set. The Google
> Cloud SDK generally tries to get a default value in this case for
> convenience, which we should follow.
> [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[
https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=333629&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-333629
]
ASF GitHub Bot logged work on BEAM-8472:
Author: ASF GitHub Bot
Created on: 24/Oct/19 18:13
Start Date: 24/Oct/19 18:13
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9868: [BEAM-8472] get
default GCP region option (Python)
URL: https://github.com/apache/beam/pull/9868#discussion_r338719413
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -500,6 +501,32 @@ def _add_argparse_args(cls, parser):
choices=['COST_OPTIMIZED', 'SPEED_OPTIMIZED'],
help='Set the Flexible Resource Scheduling mode')
+ def _get_default_gcp_region():
+"""Get a default value for Google Cloud region according to
+https://cloud.google.com/compute/docs/gcloud-compute/#default-properties.
+If no other default can be found, returns 'us-central1'.
+"""
+environment_region = os.environ.get('CLOUDSDK_COMPUTE_REGION')
+if environment_region:
+ logging.info('Using default GCP region %s from $CLOUDSDK_COMPUTE_REGION',
+ environment_region)
+ return environment_region
+try:
+ cmd = ['gcloud', 'config', 'get-value', 'compute/region']
+ output = subprocess.check_output(cmd).decode('utf-8').strip()
+ if '(unset)' not in output:
+logging.info('Using default GCP region %s from `%s`',
+output, ' '.join(cmd))
+return output
+except subprocess.CalledProcessError:
+ pass
+logging.warning(
+'--region not set; will default to us-central1. Future releases of '
+'Beam will require the user to set --region explicitly, or else have a
'
+'default set via the gcloud tool. '
+'https://cloud.google.com/compute/docs/regions-zones')
+return 'us-central1'
Review comment:
We should not default us-central1. If we cannot get this information, the
flag should be required.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 333629)
Time Spent: 20m (was: 10m)
> Get default GCP region from gcloud
> --
>
> Key: BEAM-8472
> URL: https://issues.apache.org/jira/browse/BEAM-8472
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, we default to us-central1 if --region flag is not set. The Google
> Cloud SDK generally tries to get a default value in this case for
> convenience, which we should follow.
> [https://cloud.google.com/compute/docs/gcloud-compute/#order_of_precedence_for_default_properties]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8472) Get default GCP region from gcloud
[ https://issues.apache.org/jira/browse/BEAM-8472?focusedWorklogId=333462&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-333462 ] ASF GitHub Bot logged work on BEAM-8472: Author: ASF GitHub Bot Created on: 24/Oct/19 14:35 Start Date: 24/Oct/19 14:35 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9868: [BEAM-8472] get default GCP region option (Python) URL: https://github.com/apache/beam/pull/9868 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)[
