[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[
https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87767&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87767
]
ASF GitHub Bot logged work on BEAM-3774:
Author: ASF GitHub Bot
Created on: 04/Apr/18 21:20
Start Date: 04/Apr/18 21:20
Worklog Time Spent: 10m
Work Description: chamikaramj closed pull request #5001: [BEAM-3774] Adds
support for reading from/writing to more BQ geographical locations
URL: https://github.com/apache/beam/pull/5001
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
index 96a06229713..29b405bf368 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
@@ -20,6 +20,7 @@
import static com.google.common.base.Preconditions.checkState;
+import com.google.api.services.bigquery.model.Dataset;
import com.google.api.services.bigquery.model.Job;
import com.google.api.services.bigquery.model.JobStatus;
import com.google.api.services.bigquery.model.TableReference;
@@ -203,12 +204,28 @@ static void verifyDatasetPresence(DatasetService
datasetService, TableReference
} else {
throw new RuntimeException(
String.format(
-UNABLE_TO_CONFIRM_PRESENCE_OF_RESOURCE_ERROR, "dataset",
toTableSpec(table)),
-e);
+UNABLE_TO_CONFIRM_PRESENCE_OF_RESOURCE_ERROR, "dataset",
toTableSpec(table)), e);
}
}
}
+ static String getDatasetLocation(
+ DatasetService datasetService, String projectId, String datasetId) {
+Dataset dataset;
+try {
+ dataset = datasetService.getDataset(projectId, datasetId);
+} catch (Exception e) {
+ if (e instanceof InterruptedException) {
+Thread.currentThread().interrupt();
+ }
+ throw new RuntimeException(
+ String.format(
+ "unable to obtain dataset for dataset %s in project %s",
datasetId, projectId),
+ e);
+}
+return dataset.getLocation();
+ }
+
static void verifyTablePresence(DatasetService datasetService,
TableReference table) {
try {
datasetService.getTable(table);
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 88de9b4e505..fab238cb788 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -170,6 +170,13 @@
* .fromQuery("SELECT year, mean_temp FROM [samples.weather_stations]"));
* }
*
+ * Users can optionally specify a query priority using {@link
TypedRead#withQueryPriority(
+ * TypedRead.QueryPriority)} and a geographic location where the query will be
executed using {@link
+ * TypedRead#withQueryLocation(String)}. Query location must be specified for
jobs that are not
+ * executed in US or EU. See https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query";>BigQuery
Jobs:
+ * query.
+ *
* Writing
*
* To write to a BigQuery table, apply a {@link BigQueryIO.Write}
transformation. This consumes a
@@ -549,6 +556,7 @@ public Read withTemplateCompatibility() {
abstract Builder setWithTemplateCompatibility(Boolean
useTemplateCompatibility);
abstract Builder setBigQueryServices(BigQueryServices
bigQueryServices);
abstract Builder setQueryPriority(QueryPriority priority);
+ abstract Builder setQueryLocation(String location);
abstract TypedRead build();
abstract Builder setParseFn(
@@ -570,6 +578,8 @@ public Read withTemplateCompatibility() {
@Nullable abstract QueryPriority getQueryPriority();
+@Nullable abstract String getQueryLocation();
+
@Nullable abstract Coder getCoder();
/**
@@ -632,7 +642,8 @@ public Read withTemplateCompatibility() {
getBigQueryServices(),
coder,
getParseFn(),
-MoreObjects.firstNonNull(getQueryPriority(),
QueryPriority.BATCH));
+MoreObjects.firstNonNull(getQueryPriority(),
QueryPriority.BATCH),
+getQueryLocation());
}
return source;
}
@@ -68
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[ https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87683&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87683 ] ASF GitHub Bot logged work on BEAM-3774: Author: ASF GitHub Bot Created on: 04/Apr/18 17:54 Start Date: 04/Apr/18 17:54 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #5001: [BEAM-3774] Adds support for reading from/writing to more BQ geographical locations URL: https://github.com/apache/beam/pull/5001#issuecomment-378688813 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87683) Time Spent: 1h 40m (was: 1.5h) > Update BigQuery jobs to explicitly specify the region > - > > Key: BEAM-3774 > URL: https://issues.apache.org/jira/browse/BEAM-3774 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > This is needed to support BQ regions other than US and EU. Region can be > obtained by a Dataset.get() request so no need to update the user API. > Both Python and Java SDKs have to be updated. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[ https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87433&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87433 ] ASF GitHub Bot logged work on BEAM-3774: Author: ASF GitHub Bot Created on: 04/Apr/18 06:14 Start Date: 04/Apr/18 06:14 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #5001: [BEAM-3774] Adds support for reading from/writing to more BQ geographical locations URL: https://github.com/apache/beam/pull/5001#issuecomment-378491897 Run Java Gradle PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87433) Time Spent: 1.5h (was: 1h 20m) > Update BigQuery jobs to explicitly specify the region > - > > Key: BEAM-3774 > URL: https://issues.apache.org/jira/browse/BEAM-3774 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > This is needed to support BQ regions other than US and EU. Region can be > obtained by a Dataset.get() request so no need to update the user API. > Both Python and Java SDKs have to be updated. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[ https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87361&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87361 ] ASF GitHub Bot logged work on BEAM-3774: Author: ASF GitHub Bot Created on: 04/Apr/18 00:03 Start Date: 04/Apr/18 00:03 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #5001: [BEAM-3774] Adds support for reading from/writing to more BQ geographical locations URL: https://github.com/apache/beam/pull/5001#issuecomment-378437356 Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87361) Time Spent: 1h 20m (was: 1h 10m) > Update BigQuery jobs to explicitly specify the region > - > > Key: BEAM-3774 > URL: https://issues.apache.org/jira/browse/BEAM-3774 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > This is needed to support BQ regions other than US and EU. Region can be > obtained by a Dataset.get() request so no need to update the user API. > Both Python and Java SDKs have to be updated. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[
https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87359&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87359
]
ASF GitHub Bot logged work on BEAM-3774:
Author: ASF GitHub Bot
Created on: 04/Apr/18 00:03
Start Date: 04/Apr/18 00:03
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#5001: [BEAM-3774] Adds support for reading from/writing to more BQ
geographical locations
URL: https://github.com/apache/beam/pull/5001#discussion_r178995621
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
##
@@ -97,13 +109,17 @@ public long getEstimatedSizeBytes(PipelineOptions
options) throws Exception {
protected TableReference getTableToExtract(BigQueryOptions bqOptions)
throws IOException, InterruptedException {
// 1. Find the location of the query.
-String location = null;
-List referencedTables =
-dryRunQueryIfNeeded(bqOptions).getQuery().getReferencedTables();
+String location = this.location;
DatasetService tableService = bqServices.getDatasetService(bqOptions);
-if (referencedTables != null && !referencedTables.isEmpty()) {
- TableReference queryTable = referencedTables.get(0);
- location = tableService.getTable(queryTable).getLocation();
+if (location == null) {
+ // If location was not provided we try to determine it from the tables
referenced by the
+ // Query. This will only work for BQ locations US and EU.
+ List referencedTables =
+ dryRunQueryIfNeeded(bqOptions).getQuery().getReferencedTables();
+ if (referencedTables != null && !referencedTables.isEmpty()) {
+TableReference queryTable = referencedTables.get(0);
Review comment:
Yes. You can run queries over multiple tables/datasets but they have to be
in the same geographical location.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87359)
Time Spent: 1h 10m (was: 1h)
> Update BigQuery jobs to explicitly specify the region
> -
>
> Key: BEAM-3774
> URL: https://issues.apache.org/jira/browse/BEAM-3774
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> This is needed to support BQ regions other than US and EU. Region can be
> obtained by a Dataset.get() request so no need to update the user API.
> Both Python and Java SDKs have to be updated.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[
https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87345
]
ASF GitHub Bot logged work on BEAM-3774:
Author: ASF GitHub Bot
Created on: 03/Apr/18 23:46
Start Date: 03/Apr/18 23:46
Worklog Time Spent: 10m
Work Description: tgroh commented on a change in pull request #5001:
[BEAM-3774] Adds support for reading from/writing to more BQ geographical
locations
URL: https://github.com/apache/beam/pull/5001#discussion_r178992977
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
##
@@ -97,13 +109,17 @@ public long getEstimatedSizeBytes(PipelineOptions
options) throws Exception {
protected TableReference getTableToExtract(BigQueryOptions bqOptions)
throws IOException, InterruptedException {
// 1. Find the location of the query.
-String location = null;
-List referencedTables =
-dryRunQueryIfNeeded(bqOptions).getQuery().getReferencedTables();
+String location = this.location;
DatasetService tableService = bqServices.getDatasetService(bqOptions);
-if (referencedTables != null && !referencedTables.isEmpty()) {
- TableReference queryTable = referencedTables.get(0);
- location = tableService.getTable(queryTable).getLocation();
+if (location == null) {
+ // If location was not provided we try to determine it from the tables
referenced by the
+ // Query. This will only work for BQ locations US and EU.
+ List referencedTables =
+ dryRunQueryIfNeeded(bqOptions).getQuery().getReferencedTables();
+ if (referencedTables != null && !referencedTables.isEmpty()) {
+TableReference queryTable = referencedTables.get(0);
Review comment:
Queries can run over multiple tables?
It's assumed that a query running over tables in multiple locations is
forbidden, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87345)
Time Spent: 1h (was: 50m)
> Update BigQuery jobs to explicitly specify the region
> -
>
> Key: BEAM-3774
> URL: https://issues.apache.org/jira/browse/BEAM-3774
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This is needed to support BQ regions other than US and EU. Region can be
> obtained by a Dataset.get() request so no need to update the user API.
> Both Python and Java SDKs have to be updated.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[ https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87337&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87337 ] ASF GitHub Bot logged work on BEAM-3774: Author: ASF GitHub Bot Created on: 03/Apr/18 23:26 Start Date: 03/Apr/18 23:26 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #5001: [BEAM-3774] Adds support for reading from/writing to more BQ geographical locations URL: https://github.com/apache/beam/pull/5001#issuecomment-378431075 Thanks. PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87337) Time Spent: 50m (was: 40m) > Update BigQuery jobs to explicitly specify the region > - > > Key: BEAM-3774 > URL: https://issues.apache.org/jira/browse/BEAM-3774 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > This is needed to support BQ regions other than US and EU. Region can be > obtained by a Dataset.get() request so no need to update the user API. > Both Python and Java SDKs have to be updated. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[
https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87335
]
ASF GitHub Bot logged work on BEAM-3774:
Author: ASF GitHub Bot
Created on: 03/Apr/18 23:26
Start Date: 03/Apr/18 23:26
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#5001: [BEAM-3774] Adds support for reading from/writing to more BQ
geographical locations
URL: https://github.com/apache/beam/pull/5001#discussion_r178990101
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
##
@@ -170,6 +170,11 @@
* .fromQuery("SELECT year, mean_temp FROM [samples.weather_stations]"));
* }
*
+ * Users can optionally specify a query priority using {@link
TypedRead#withQueryPriority(
+ * TypedRead.QueryPriority)} and a geographic location where the query will be
executed using
+ * {@link TypedRead#withQueryLocation(String)}. Query location must be
specified for jobs that are
+ * not executed in US or EU.
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87335)
Time Spent: 40m (was: 0.5h)
> Update BigQuery jobs to explicitly specify the region
> -
>
> Key: BEAM-3774
> URL: https://issues.apache.org/jira/browse/BEAM-3774
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This is needed to support BQ regions other than US and EU. Region can be
> obtained by a Dataset.get() request so no need to update the user API.
> Both Python and Java SDKs have to be updated.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[
https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87336&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87336
]
ASF GitHub Bot logged work on BEAM-3774:
Author: ASF GitHub Bot
Created on: 03/Apr/18 23:26
Start Date: 03/Apr/18 23:26
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#5001: [BEAM-3774] Adds support for reading from/writing to more BQ
geographical locations
URL: https://github.com/apache/beam/pull/5001#discussion_r178990118
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
##
@@ -687,7 +696,7 @@ public void validate(PipelineOptions options) {
new JobConfigurationQuery()
.setQuery(getQuery().get())
.setFlattenResults(getFlattenResults())
-.setUseLegacySql(getUseLegacySql()));
+.setUseLegacySql(getUseLegacySql()), getQueryLocation());
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87336)
> Update BigQuery jobs to explicitly specify the region
> -
>
> Key: BEAM-3774
> URL: https://issues.apache.org/jira/browse/BEAM-3774
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This is needed to support BQ regions other than US and EU. Region can be
> obtained by a Dataset.get() request so no need to update the user API.
> Both Python and Java SDKs have to be updated.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[
https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87156&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87156
]
ASF GitHub Bot logged work on BEAM-3774:
Author: ASF GitHub Bot
Created on: 03/Apr/18 16:41
Start Date: 03/Apr/18 16:41
Worklog Time Spent: 10m
Work Description: tgroh commented on a change in pull request #5001:
[BEAM-3774] Adds support for reading from/writing to more BQ geographical
locations
URL: https://github.com/apache/beam/pull/5001#discussion_r178887647
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
##
@@ -170,6 +170,11 @@
* .fromQuery("SELECT year, mean_temp FROM [samples.weather_stations]"));
* }
*
+ * Users can optionally specify a query priority using {@link
TypedRead#withQueryPriority(
+ * TypedRead.QueryPriority)} and a geographic location where the query will be
executed using
+ * {@link TypedRead#withQueryLocation(String)}. Query location must be
specified for jobs that are
+ * not executed in US or EU.
Review comment:
Is there a specific piece of BigQuery documentation that we can try to link
to?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87156)
> Update BigQuery jobs to explicitly specify the region
> -
>
> Key: BEAM-3774
> URL: https://issues.apache.org/jira/browse/BEAM-3774
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This is needed to support BQ regions other than US and EU. Region can be
> obtained by a Dataset.get() request so no need to update the user API.
> Both Python and Java SDKs have to be updated.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[
https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87155&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87155
]
ASF GitHub Bot logged work on BEAM-3774:
Author: ASF GitHub Bot
Created on: 03/Apr/18 16:41
Start Date: 03/Apr/18 16:41
Worklog Time Spent: 10m
Work Description: tgroh commented on a change in pull request #5001:
[BEAM-3774] Adds support for reading from/writing to more BQ geographical
locations
URL: https://github.com/apache/beam/pull/5001#discussion_r178887991
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
##
@@ -687,7 +696,7 @@ public void validate(PipelineOptions options) {
new JobConfigurationQuery()
.setQuery(getQuery().get())
.setFlattenResults(getFlattenResults())
-.setUseLegacySql(getUseLegacySql()));
+.setUseLegacySql(getUseLegacySql()), getQueryLocation());
Review comment:
Newline between the `JobConfigurationQuery` and the location, as it reads as
though it's part of that configuration (which I don't think it is)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87155)
Time Spent: 0.5h (was: 20m)
> Update BigQuery jobs to explicitly specify the region
> -
>
> Key: BEAM-3774
> URL: https://issues.apache.org/jira/browse/BEAM-3774
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This is needed to support BQ regions other than US and EU. Region can be
> obtained by a Dataset.get() request so no need to update the user API.
> Both Python and Java SDKs have to be updated.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[ https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=86982&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86982 ] ASF GitHub Bot logged work on BEAM-3774: Author: ASF GitHub Bot Created on: 03/Apr/18 07:08 Start Date: 03/Apr/18 07:08 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #5001: [BEAM-3774] Adds support for reading from/writing to more BQ geographical locations URL: https://github.com/apache/beam/pull/5001#issuecomment-378150335 R: @tgroh This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 86982) Time Spent: 20m (was: 10m) > Update BigQuery jobs to explicitly specify the region > - > > Key: BEAM-3774 > URL: https://issues.apache.org/jira/browse/BEAM-3774 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > This is needed to support BQ regions other than US and EU. Region can be > obtained by a Dataset.get() request so no need to update the user API. > Both Python and Java SDKs have to be updated. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3774) Update BigQuery jobs to explicitly specify the region
[ https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=86937&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86937 ] ASF GitHub Bot logged work on BEAM-3774: Author: ASF GitHub Bot Created on: 03/Apr/18 04:43 Start Date: 03/Apr/18 04:43 Worklog Time Spent: 10m Work Description: chamikaramj opened a new pull request #5001: [BEAM-3774] Adds support for reading from/writing to BQ datasets not in US or EU. URL: https://github.com/apache/beam/pull/5001 DESCRIPTION HERE Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand: - [ ] What the pull request does - [ ] Why it does it - [ ] How it does it - [ ] Why this approach - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 86937) Time Spent: 10m Remaining Estimate: 0h > Update BigQuery jobs to explicitly specify the region > - > > Key: BEAM-3774 > URL: https://issues.apache.org/jira/browse/BEAM-3774 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This is needed to support BQ regions other than US and EU. Region can be > obtained by a Dataset.get() request so no need to update the user API. > Both Python and Java SDKs have to be updated. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
