[
https://issues.apache.org/jira/browse/BEAM-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670829#comment-16670829
]
Mark Liu commented on BEAM-5788:
--------------------------------
I wrote few lines to reproduce the problem in isolate environment. Codes are:
{code:python}
import unittest
from storage_v1_client import *
from storage_v1_messages import *
class GetTest(unittest.TestCase):
def testGet(self):
_BUCKET = 'apache-beam-samples' # public bucket
_OBJECT = 'apache/LICENSE' # file name
request = StorageObjectsGetRequest(bucket=_BUCKET,
object=_OBJECT)
response = StorageV1().objects.Get(request)
print('Finish Get. Response: %s' % response)
{code}
storage_v1_client and storage_v1_messages are provided in
https://github.com/google/apitools/tree/master/samples/storage_sample/storage_v1.
In a virtual environment, I installed latest google-apitools library (0.5.24)
and run this test. In Python 2 environment, test passed and response is
printed. In Python 3.5 environment, test failed at same type error.
> wordcount_fnapi_it failed on TestDataflowRunner because of JSON string
> decoding error
> -------------------------------------------------------------------------------------
>
> Key: BEAM-5788
> URL: https://issues.apache.org/jira/browse/BEAM-5788
> Project: Beam
> Issue Type: Sub-task
> Components: test-failures
> Reporter: Mark Liu
> Assignee: Valentyn Tymofieiev
> Priority: Major
>
> Similar to BEAM-5785, wordcount_fnapi_it failed on Python 3 when running with
> TestDataflowRunner. Got TypeError: the JSON object must be str, not 'bytes'.
> This error cause infinite retry before job could submitted to service.
> More details about my env and test:
> Python version: 3.5.3
> Test:
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_fnapi_it
> Command:
> {code}
> python setup.py nosetests \
> --tests
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_fnapi_it \
> --nocapture \
> --nologcapture \
> --test-pipeline-options=" \
> --runner=TestDataflowRunner \
> --project=<my_project> \
> --staging_location=<my_staging> \
> --temp_location=<my_temp> \
> --output=<my_output> \
>
>
> --sdk_location=.../beam/sdks/python/dist/apache-beam-2.9.0.dev0.tar.gz \
> --num_workers=1"
> {code}
> Stacktrace:
> {code}
> WARNING:root:Retry with exponential backoff: waiting for 7.661876827680761
> seconds before retrying exists because we caught exception: TypeError: the
> JSON object must be str, not 'bytes'
> Traceback for above exception (most recent call last):
> File ".../beam/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
> return fun(*args, **kwargs)
> File ".../beam/sdks/python/apache_beam/io/gcp/gcsio.py", line 375, in exists
> self.client.objects.Get(request) # metadata
> File
> ".../beam/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
> line 955, in Get
> download=download)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py",
> line 722, in _RunMethod
> return self.ProcessHttpResponse(method_config, http_response, request)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py",
> line 728, in ProcessHttpResponse
> self.__ProcessHttpResponse(method_config, http_response, request))
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py",
> line 611, in __ProcessHttpResponse
> response_type, http_response.content)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py",
> line 442, in DeserializeMessage
> message = encoding.JsonToMessage(response_type, data)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/encoding.py",
> line 104, in JsonToMessage
> return _ProtoJsonApiTools.Get().decode_message(message_type, message)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/encoding.py",
> line 290, in decode_message
> message_type, result)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/protorpclite/protojson.py",
> line 210, in decode_message
> dictionary = json.loads(encoded_message)
> File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
> s.__class__.__name__))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
