[
https://issues.apache.org/jira/browse/BEAM-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Webb updated BEAM-10636:
-----------------------------
Status: Open (was: Triage Needed)
> Dataflow deployments with FnAPI runner fail on python3.8 due to pycrypto
> -------------------------------------------------------------------------
>
> Key: BEAM-10636
> URL: https://issues.apache.org/jira/browse/BEAM-10636
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp, runner-dataflow, sdk-py-harness
> Affects Versions: 2.23.0
> Reporter: David Rubinstein
> Priority: P3
>
> PyCrypto [uses|[https://github.com/pycrypto/pycrypto/issues/308]]
> `time.clock()` which was
> [deprecated|[https://docs.python.org/3/whatsnew/3.8.html#api-and-feature-removals]]
> in Python3.8
> Upon further look this has to do with google-apitools depending on
> oauth2client (which is archived) which depends on pycrypto (which is not
> maintained anymore). Not really sure what to do to fix the dependencies, but
> deployments in Python3.8 with FnAPI on Dataflow are broken for me until then.
>
> Relevant stacktrace.
> {noformat}
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apache_beam/pipeline.py", line 534,
> in run
> [2020-08-04T18:03:22.978Z] return self.runner.run_pipeline(self,
> self._options)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apache_beam/runners/dataflow/dataflow_runner.py",
> line 558, in run_pipeline
> [2020-08-04T18:03:22.978Z] self.job = apiclient.Job(options,
> self.proto_pipeline)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apache_beam/runners/dataflow/internal/apiclient.py",
> line 495, in __init__
> [2020-08-04T18:03:22.978Z] self.proto.replaceJobId =
> self.job_id_for_name(self.proto.name)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apache_beam/runners/dataflow/internal/apiclient.py",
> line 518, in job_id_for_name
> [2020-08-04T18:03:22.978Z] return DataflowApplicationClient(
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apache_beam/runners/dataflow/internal/apiclient.py",
> line 924, in job_id_for_name
> [2020-08-04T18:03:22.978Z] response =
> self._client.projects_locations_jobs.List(request)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
> line 763, in List
> [2020-08-04T18:03:22.978Z] return self._RunMethod(
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apitools/base/py/base_api.py", line
> 728, in _RunMethod
> [2020-08-04T18:03:22.978Z] http_response = http_wrapper.MakeRequest(
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apitools/base/py/http_wrapper.py",
> line 359, in MakeRequest
> [2020-08-04T18:03:22.978Z] retry_func(ExceptionRetryArgs(http,
> http_request, e, retry,
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apitools/base/py/http_wrapper.py",
> line 304, in HandleExceptionsAndRebuildHttpConnections
> [2020-08-04T18:03:22.978Z] raise retry_args.exc
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apitools/base/py/http_wrapper.py",
> line 348, in MakeRequest
> [2020-08-04T18:03:22.978Z] return _MakeRequestNoRetry(
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/apitools/base/py/http_wrapper.py",
> line 397, in _MakeRequestNoRetry
> [2020-08-04T18:03:22.978Z] info, content = http.request(
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 153,
> in new_request
> [2020-08-04T18:03:22.978Z] credentials._refresh(orig_request_method)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/oauth2client/client.py", line 765, in
> _refresh
> [2020-08-04T18:03:22.978Z] self._do_refresh_request(http_request)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/oauth2client/client.py", line 792, in
> _do_refresh_request
> [2020-08-04T18:03:22.978Z] body = self._generate_refresh_request_body()
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/oauth2client/client.py", line 1501,
> in _generate_refresh_request_body
> [2020-08-04T18:03:22.978Z] assertion = self._generate_assertion()
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/oauth2client/service_account.py",
> line 385, in _generate_assertion
> [2020-08-04T18:03:22.978Z] return crypt.make_signed_jwt(self._signer,
> payload,
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/oauth2client/crypt.py", line 97, in
> make_signed_jwt
> [2020-08-04T18:03:22.978Z] signature = signer.sign(signing_input)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/oauth2client/_pycrypto_crypt.py",
> line 99, in sign
> [2020-08-04T18:03:22.978Z] return
> PKCS1_v1_5.new(self._key).sign(SHA256.new(message))
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/Signature/PKCS1_v1_5.py", line
> 112, in sign
> [2020-08-04T18:03:22.978Z] m = self._key.decrypt(em)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/PublicKey/RSA.py", line 174,
> in decrypt
> [2020-08-04T18:03:22.978Z] return pubkey.pubkey.decrypt(self, ciphertext)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/PublicKey/pubkey.py", line 93,
> in decrypt
> [2020-08-04T18:03:22.978Z] plaintext=self._decrypt(ciphertext)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/PublicKey/RSA.py", line 235,
> in _decrypt
> [2020-08-04T18:03:22.978Z] r = getRandomRange(1, self.key.n-1,
> randfunc=self._randfunc)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/Util/number.py", line 123, in
> getRandomRange
> [2020-08-04T18:03:22.978Z] value = getRandomInteger(bits, randfunc)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/Util/number.py", line 104, in
> getRandomInteger
> [2020-08-04T18:03:22.978Z] S = randfunc(N>>3)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/Random/_UserFriendlyRNG.py",
> line 202, in read
> [2020-08-04T18:03:22.978Z] return self._singleton.read(bytes)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/Random/_UserFriendlyRNG.py",
> line 178, in read
> [2020-08-04T18:03:22.978Z] return _UserFriendlyRNG.read(self, bytes)
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/Random/_UserFriendlyRNG.py",
> line 129, in read
> [2020-08-04T18:03:22.978Z] self._ec.collect()
> [2020-08-04T18:03:22.978Z] File
> "/usr/local/lib/python3.8/site-packages/Crypto/Random/_UserFriendlyRNG.py",
> line 77, in collect
> [2020-08-04T18:03:22.978Z] t = time.clock()
> [2020-08-04T18:03:22.978Z] AttributeError: module 'time' has no attribute
> 'clock'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
