[ 
https://issues.apache.org/jira/browse/BEAM-5457?focusedWorklogId=151328&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151328
 ]

ASF GitHub Bot logged work on BEAM-5457:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 04/Oct/18 18:43
            Start Date: 04/Oct/18 18:43
    Worklog Time Spent: 10m 
      Work Description: udim commented on a change in pull request #6463: 
[BEAM-5457][BEAM-1909] Make BigQuerySource work for the EU
URL: https://github.com/apache/beam/pull/6463#discussion_r222784985
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery.py
 ##########
 @@ -659,27 +659,41 @@ def __init__(self, source, test_bigquery_client=None, 
use_legacy_sql=True,
     else:
       self.query = self.source.query
 
-  def _get_source_table_location(self):
-    tr = self.source.table_reference
-    if tr is None:
-      # TODO: implement location retrieval for query sources
-      return
+  def _get_source_location(self):
+    """
+    Get the source location (e.g. ``"EU"`` or ``"US"``) from either
 
-    if tr.projectId is None:
-      source_project_id = self.executing_project
-    else:
-      source_project_id = tr.projectId
+    - :data:`source.table_reference`
+      or
+    - The first referenced table in :data:`source.query`
+
+    See Also:
+      - :meth:`BigQueryWrapper.get_query_location`
+      - :meth:`BigQueryWrapper.get_table_location`
 
-    source_dataset_id = tr.datasetId
-    source_table_id = tr.tableId
-    source_location = self.client.get_table_location(
-        source_project_id, source_dataset_id, source_table_id)
-    return source_location
+    Returns:
+      Optional[str]: The source location, if any.
+    """
+    if self.source.table_reference is not None:
+      tr = self.source.table_reference
+      return self.client.get_table_location(
+          tr.projectId if tr.projectId is not None else self.executing_project,
+          tr.datasetId, tr.tableId)
+    elif self.source.query is not None:
+      return self.client.get_query_location(
+          self.executing_project,
+          self.source.query,
+          self.source.use_legacy_sql)
+    else:
+      logging.error(
+          "Unable to get source location for %r as it has neither "
+          "table nor query.", self.source)
+      return None
 
 Review comment:
   Perhaps this check is better placed in `BigQueryReader.__init__`, since it 
already makes the same assumption that either a table reference or query is set:
   
https://github.com/apache/beam/blob/5aacd6bcbf78d0b70ff3d289ad7fbc0a156a54c8/sdks/python/apache_beam/io/gcp/bigquery.py#L649-L660

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 151328)
    Time Spent: 3h 40m  (was: 3.5h)

> BigQuerySource(query=...) in DirectRunner creates temp dataset in the wrong 
> location
> ------------------------------------------------------------------------------------
>
>                 Key: BEAM-5457
>                 URL: https://issues.apache.org/jira/browse/BEAM-5457
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>    Affects Versions: 2.6.0
>            Reporter: Joar Wandborg
>            Assignee: Chamikara Jayalath
>            Priority: Major
>          Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> I'm in the EU, if I have a
>  
> {code:java}
> BigQuerySource(
>     query="SELECT x, y FROM `my-other-project.mydataset.my_european_table`",
>     project="myproject",
>     use_standard_sql=True
> ){code}
> And then run the Pipeline through the DirectRunner I get the following 
> warning and error:
> {noformat}
> 2018-09-21 11:39:52,620 WARNING root create_temporary_dataset
> Dataset myproject:temp_dataset_0bbb28f014a24225b668a67341f4f71e does not 
> exist so we will create it as temporary with location=None {noformat}
> {noformat}
> HttpBadRequestError: HttpError accessing 
> <https://www.googleapis.com/bigquery/v2/projects/myproject/queries/xyz123?alt=json&maxResults=10000>:
>  response: <{'status': '400', 'content-length': '354', 'x-xss-protection': 
> '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 
> 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', 
> '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Fri, 21 Sep 
> 2018 09:39:55 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443"; 
> ma=2592000; v="44,43,39,35"', 'content-type': 'application/json; 
> charset=UTF-8'}>, content <{
>   "error": {
>     "code": 400,
>     "message": "Cannot read and write in different locations: source: EU, 
> destination: US",
>     "errors": [
>       {
>         "message": "Cannot read and write in different locations: source: EU, 
> destination: US",
>         "domain": "global",
>         "reason": "invalid"
>       }
>     ],
>     "status": "INVALID_ARGUMENT"
>   }
> {noformat}
> There's a TODO in the code that looks very related: 
> [https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665|https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665,]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to