Chris C GUO created BEAM-12779:
----------------------------------
Summary: apache beam is not able to find its library when running
via DataflowRunner
Key: BEAM-12779
URL: https://issues.apache.org/jira/browse/BEAM-12779
Project: Beam
Issue Type: Bug
Components: beam-model
Affects Versions: 2.31.0
Environment: (venv) λ python --version
Python 3.6.1 :: Anaconda 4.4.0 (64-bit)
(venv) λ pip show apache-beam
Name: apache-beam
Version: 2.31.0
Summary: Apache Beam SDK for Python
Home-page: https://beam.apache.org
Author: Apache Software Foundation
Author-email: [email protected]
License: Apache License, Version 2.0
Location: venv\lib\site-packages
Requires: pydot, grpcio, avro-python3, protobuf, dill, python-dateutil, hdfs,
httplib2, requests, typing-extensions, future, numpy, pytz, pyarrow,
dataclasses, pymongo, oauth2client, crcmod, fastavro
Required-by: chris-dataflow
Reporter: Chris C GUO
INFO:apache_beam.runners.portability.stager:Downloading source distribution of
the SDK from PyPi
INFO:apache_beam.runners.portability.stager:Executing command: ['python.exe',
'-m', 'pip', 'download', '--dest', 'AppData\\Local\\Temp\\tmptb7pff18',
'apache-beam==2.31.0', '--no-deps', '--no-binary', ':all:']
ERROR: Could not find a version that satisfies the requirement
apache-beam==2.31.0 (from versions: none)
ERROR: No matching distribution found for apache-beam==2.31.0
Traceback (most recent call last):
File "XXXXX\lib\site-packages\apache_beam\utils\processes.py", line 89, in
check_output
out = subprocess.check_output(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['XXXXX\\venv\\Scripts\\python.exe',
'-m', 'pip', 'download', '--dest', 'XXXXX\\AppData\\Local\\Temp\\tmptb7pff18',
'apache-beam==2.31.0', '--no-deps', '--no-binary', ':all:']' returned non-zero
exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "rdl_test.py", line 118, in <module>
pipeline_args,
File "rdl_test.py", line 82, in run
| "Write to GCS" >> ParDo(WriteToGCS(output_path))
File "XXXXX\lib\site-packages\apache_beam\pipeline.py", line 585, in __exit__
self.result = self.run()
File "XXXXX\lib\site-packages\apache_beam\pipeline.py", line 540, in run
self._options).run(False)
File "XXXXX\lib\site-packages\apache_beam\pipeline.py", line 564, in run
return self.runner.run_pipeline(self, self._options)
File "XXXXX\lib\site-packages\apache_beam\runners\dataflow\dataflow_runner.py",
line 454, in run_pipeline
artifacts=environments.python_sdk_dependencies(options),
File "XXXXX\lib\site-packages\apache_beam\transforms\environments.py", line
806, in python_sdk_dependencies
skip_prestaged_dependencies=skip_prestaged_dependencies)
File "XXXXX\lib\site-packages\apache_beam\runners\portability\stager.py", line
285, in create_job_resources
Stager._create_beam_sdk(sdk_remote_location, temp_dir))
File "XXXXX\lib\site-packages\apache_beam\runners\portability\stager.py", line
712, in _create_beam_sdk
sdk_local_file = Stager._download_pypi_sdk_package(temp_dir)
File "XXXXX\lib\site-packages\apache_beam\runners\portability\stager.py", line
826, in _download_pypi_sdk_package
processes.check_output(cmd_args)
File "XXXXX\lib\site-packages\apache_beam\utils\processes.py", line 97, in
check_output
.format(traceback.format_exc(), args[0][6], error.output))
RuntimeError: Full traceback: Traceback (most recent call last):
File "XXXXX\lib\site-packages\apache_beam\utils\processes.py", line 89, in
check_output
out = subprocess.check_output(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['XXXXX\\venv\\Scripts\\python.exe',
'-m', 'pip', 'download', '--dest', 'XXXXX\\AppData\\Local\\Temp\\tmptb7pff18',
'apache-beam==2.31.0', '--no-deps', '--no-binary', ':all:']' returned non-zero
exit status 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
