damccorm opened a new issue, #20733:
URL: https://github.com/apache/beam/issues/20733
Hi, I'm getting the following error when using apache-beam 2.27.0 in a
Python3 virtualenv:
```
airflow@airflow-worker-7fb797d459-nf8gh:~$
/tmp/dataflow-venvtflya9ij/bin/python
/home/airflow/gcs/dags/advisor/create_dataset/beam_pipelines/dummy.py
Traceback
(most recent call last):
File
"/home/airflow/gcs/dags/advisor/create_dataset/beam_pipelines/dummy.py",
line 1, in <module>
import apache_beam
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/__init__.py",
line 95, in <module>
from apache_beam import coders
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/__init__.py",
line 19, in <module>
from apache_beam.coders.coders import *
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/coders.py",
line 43, in <module>
from future.moves import pickle
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/future/moves/__init__.py",
line 8, in <module>
import_top_level_modules()
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/future/standard_library/__init__.py",
line 810, in import_top_level_modules
with exclude_local_folder_imports(*TOP_LEVEL_MODULES):
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/future/standard_library/__init__.py",
line 781, in __enter__
module = __import__(m, level=0)
File
"/home/airflow/gcs/dags/advisor/create_dataset/beam_pipelines/test.py",
line 3, in <module>
from apache_beam.options.pipeline_options import PipelineOptions
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/options/pipeline_options.py",
line 41, in <module>
from apache_beam.transforms.display import HasDisplayData
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/transforms/__init__.py",
line 23, in <module>
from apache_beam.transforms import combiners
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/transforms/combiners.py",
line 45, in <module>
from apache_beam.transforms import core
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/transforms/core.py",
line 40, in <module>
from apache_beam.coders import typecoders
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/typecoders.py",
line 198, in <module>
registry = CoderRegistry()
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/typecoders.py",
line 91, in __init__
self.register_standard_coders(fallback_coder)
File
"/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/typecoders.py",
line 95, in register_standard_coders
self._register_coder_internal(int, coders.VarIntCoder)
AttributeError:
module 'apache_beam.coders.coders' has no attribute 'VarIntCoder'
```
My `dummy.py` file consists of only:
```
import apache_beam
if __name__ == '__main__': print('MAIN')
```
Strangely to me, I do not get the error when running the venv python
interactively and executing the `import apache_beam` statement inside:
```
airflow@airflow-worker-7fb797d459-nf8gh:~$
/tmp/dataflow-venvtflya9ij/bin/python
Python 3.6.10 (default,
Feb 1 2021, 12:07:35)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more
information.
>>> import apache_beam
>>>
```
Note: In both cases the `sys.path` packages dirs are exactly the same.
Equally strangely to me, the script runs in both scenarios (script &
interactive) when I use the previous version `apache-beam\==0.26.0`.
\---
A bit of context.
I got to this error cause Im using a Google Cloud Composer (Airflow)
operator
(`airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator`)
that launches the creation of a virtualenv & installs apache-beam, and
executes the beam script in order the trigger run the Beam pipeline on Google
Cloud DataFlow.
I'm being able to reproduce these issue by connecting to the Kubernetes pod
running this operator and manually executing the steps.
—
In order to reproduce this issue, you can:
1. create a python3 virtualenv: `virtualenv /tmp/venv \--python=python3`.
2. create the dummy.py file with the following code inside:
```
import apache_beam
```
3. install apache-beam 2.27.0: `/tmp/venv/bin/pip install
apache-beam\==2.27.0`.
4. run the script: `/tmp/venv/bin/python dummy.py`.
5. check it does not happen with `apache-beam\==2.26.0`.
—
Also reported in:
[https://stackoverflow.com/questions/66243327/inconsistent-behaviour-when-importing-a-package-interactively-vs-running-as-scri](https://stackoverflow.com/questions/66243327/inconsistent-behaviour-when-importing-a-package-interactively-vs-running-as-scri)
Imported from Jira
[BEAM-11826](https://issues.apache.org/jira/browse/BEAM-11826). Original Jira
may contain additional context.
Reported by: lqrz.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
