Marappa Ganesan created BEAM-7935:
-------------------------------------
Summary: TypeError: can't pickle _cffi_backend.CDataGCP objects
Key: BEAM-7935
URL: https://issues.apache.org/jira/browse/BEAM-7935
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Affects Versions: 2.13.0
Environment: Linux(Debian GNU/Linux 9) ( SMP x86_64 GNU/Linux)
Python 2.7.13
Content of requirements.txt file
------------------------------------------
apache-beam==2.13.0
apache-beam[gcp]
google-cloud-core==1.0.3
google-cloud-iam
google-cloud-bigquery==1.6.1
PyYAML==3.13
six==1.12.0
pyOpenSSL
httplib2==0.12.0
google-apitools
google-api-python-client
google-cloud-storage==1.17.0
Reporter: Marappa Ganesan
Python Dataflow runner failed with folrowing error
*-----------------------------------------------------------------------------------------------------------------------*
*p.run().wait_until_finish()*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 416, in run*
*pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
line 274, in dump_session*
*dill.dump_session(file_path)*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py",
line 393, in dump_session*
*pickler.dump(main)*
*File "/usr/lib/python2.7/pickle.py", line 224, in dump*
*self.save(obj)*
*File "/usr/lib/python2.7/pickle.py", line 286, in save*
*f(self, obj) # Call unbound method with explicit self*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
line 149, in save_module*
*return old_save_module(pickler, obj)*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py",
line 1269, in save_module*
*state=_main_dict)*
*File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce*
*save(state)*
*File "/usr/lib/python2.7/pickle.py", line 286, in save*
*f(self, obj) # Call unbound method with explicit self*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
line 198, in new_save_module_dict*
*return old_save_module_dict(pickler, obj)*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py",
line 902, in save_module_dict*
*StockPickler.save_dict(pickler, obj)*
*File "/usr/lib/python2.7/pickle.py", line 655, in save_dict*
*self._batch_setitems(obj.iteritems())*
*File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems*
*save(v)*
*File "/usr/lib/python2.7/pickle.py", line 331, in save*
*self.save_reduce(obj=obj, *rv)*
*File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce*
*save(state)*
*File "/usr/lib/python2.7/pickle.py", line 286, in save*
*f(self, obj) # Call unbound method with explicit self*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
line 198, in new_save_module_dict*
*return old_save_module_dict(pickler, obj)*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py",
line 902, in save_module_dict*
*StockPickler.save_dict(pickler, obj)*
*File "/usr/lib/python2.7/pickle.py", line 655, in save_dict*
*self._batch_setitems(obj.iteritems())*
*File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems*
*save(v)*
*File "/usr/lib/python2.7/pickle.py", line 331, in save*
*self.save_reduce(obj=obj, *rv)*
*File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce*
*save(state)*
*File "/usr/lib/python2.7/pickle.py", line 286, in save*
*f(self, obj) # Call unbound method with explicit self*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
line 198, in new_save_module_dict*
*return old_save_module_dict(pickler, obj)*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py",
line 902, in save_module_dict*
*StockPickler.save_dict(pickler, obj)*
*File "/usr/lib/python2.7/pickle.py", line 655, in save_dict*
*self._batch_setitems(obj.iteritems())*
*File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems*
*save(v)*
*File "/usr/lib/python2.7/pickle.py", line 331, in save*
*self.save_reduce(obj=obj, *rv)*
*File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce*
*save(state)*
*File "/usr/lib/python2.7/pickle.py", line 286, in save*
*f(self, obj) # Call unbound method with explicit self*
*File
"/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
line 198, in new_save_module_dict*
*return old_save_module_dict(pickler, obj)*
*File "/usr/lib/python2.7/pickle.py", line 655, in save_dict*
*self._batch_setitems(obj.iteritems())*
*File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems*
*save(v)*
*File "/usr/lib/python2.7/pickle.py", line 306, in save*
*rv = reduce(self.proto)*
*TypeError: can't pickle _cffi_backend.CDataGCP objects*
*-----------------------------------------------------------------------------------------------------------------------*
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
