[ 
https://issues.apache.org/jira/browse/BEAM-3991?focusedWorklogId=90234&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90234
 ]

ASF GitHub Bot logged work on BEAM-3991:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 12/Apr/18 00:00
            Start Date: 12/Apr/18 00:00
    Worklog Time Spent: 10m 
      Work Description: chamikaramj closed pull request #5105: [BEAM-3991] 
Updates gcsio to use a API specific batch endpoint.
URL: https://github.com/apache/beam/pull/5105
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/gcp/gcsio.py 
b/sdks/python/apache_beam/io/gcp/gcsio.py
index f687686fd64..e428e3b1401 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio.py
@@ -91,6 +91,17 @@
 # GcsIO.delete_batch().
 MAX_BATCH_OPERATION_SIZE = 100
 
+# Batch endpoint URL for GCS.
+# We have to specify an API specific endpoint here since Google APIs global
+# batch endpoints will be deprecated on 03/25/2019.
+# See 
https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html.
  # pylint: disable=line-too-long
+# Currently apitools library uses a global batch endpoint by default:
+# https://github.com/google/apitools/blob/master/apitools/base/py/batch.py#L152
+# TODO: remove this constant and it's usage after apitools move to using an API
+# specific batch endpoint or after Beam gcsio module start using a GCS client
+# library that does not use global batch endpoints.
+GCS_BATCH_ENDPOINT = 'https://www.googleapis.com/batch/storage/v1'
+
 
 def proxy_info_from_environment_var(proxy_env_var):
   """Reads proxy info from the environment and converts to httplib2.ProxyInfo.
@@ -274,6 +285,7 @@ def delete_batch(self, paths):
     if not paths:
       return []
     batch_request = BatchApiRequest(
+        batch_url=GCS_BATCH_ENDPOINT,
         retryable_codes=retry.SERVER_ERROR_OR_TIMEOUT_CODES)
     for path in paths:
       bucket, object_path = parse_gcs_path(path)
@@ -328,6 +340,7 @@ def copy_batch(self, src_dest_pairs):
     if not src_dest_pairs:
       return []
     batch_request = BatchApiRequest(
+        batch_url=GCS_BATCH_ENDPOINT,
         retryable_codes=retry.SERVER_ERROR_OR_TIMEOUT_CODES)
     for src, dest in src_dest_pairs:
       src_bucket, src_path = parse_gcs_path(src)


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 90234)
    Time Spent: 50m  (was: 40m)

> Update dependency 'google-api-services-storage' to latest version
> -----------------------------------------------------------------
>
>                 Key: BEAM-3991
>                 URL: https://issues.apache.org/jira/browse/BEAM-3991
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-gcp, sdk-py-core
>            Reporter: Chamikara Jayalath
>            Assignee: Chamikara Jayalath
>            Priority: Blocker
>             Fix For: 2.5.0
>
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two 
> years old.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to