[
https://issues.apache.org/jira/browse/BEAM-3774?focusedWorklogId=87359&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87359
]
ASF GitHub Bot logged work on BEAM-3774:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Apr/18 00:03
Start Date: 04/Apr/18 00:03
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#5001: [BEAM-3774] Adds support for reading from/writing to more BQ
geographical locations
URL: https://github.com/apache/beam/pull/5001#discussion_r178995621
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
##########
@@ -97,13 +109,17 @@ public long getEstimatedSizeBytes(PipelineOptions
options) throws Exception {
protected TableReference getTableToExtract(BigQueryOptions bqOptions)
throws IOException, InterruptedException {
// 1. Find the location of the query.
- String location = null;
- List<TableReference> referencedTables =
- dryRunQueryIfNeeded(bqOptions).getQuery().getReferencedTables();
+ String location = this.location;
DatasetService tableService = bqServices.getDatasetService(bqOptions);
- if (referencedTables != null && !referencedTables.isEmpty()) {
- TableReference queryTable = referencedTables.get(0);
- location = tableService.getTable(queryTable).getLocation();
+ if (location == null) {
+ // If location was not provided we try to determine it from the tables
referenced by the
+ // Query. This will only work for BQ locations US and EU.
+ List<TableReference> referencedTables =
+ dryRunQueryIfNeeded(bqOptions).getQuery().getReferencedTables();
+ if (referencedTables != null && !referencedTables.isEmpty()) {
+ TableReference queryTable = referencedTables.get(0);
Review comment:
Yes. You can run queries over multiple tables/datasets but they have to be
in the same geographical location.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 87359)
Time Spent: 1h 10m (was: 1h)
> Update BigQuery jobs to explicitly specify the region
> -----------------------------------------------------
>
> Key: BEAM-3774
> URL: https://issues.apache.org/jira/browse/BEAM-3774
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Chamikara Jayalath
> Assignee: Chamikara Jayalath
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> This is needed to support BQ regions other than US and EU. Region can be
> obtained by a Dataset.get() request so no need to update the user API.
> Both Python and Java SDKs have to be updated.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
