jackwangcs created FLINK-24663:
----------------------------------
Summary: PyFlink failed to get the site packege path because of no
quote in shell command
Key: FLINK-24663
URL: https://issues.apache.org/jira/browse/FLINK-24663
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.12.3
Reporter: jackwangcs
Flink throws an exception when it tries to install 3rd party dependencies:
{code:java}
Caused by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsCaused
by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsoutput:
File "<string>", line 1 import sys;from distutils.dist import
Distribution;install_obj = Distribution().get_command_obj('install',
create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
^SyntaxError: invalid syntax
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
at
org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
at
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
....{code}
This can be reproduced by running the python script in a bash shell:
{code:java}
python3 -c import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1]) /tmp/requirements
-bash: syntax error near unexpected token `(' {code}
The solution is to quota all argements to execute:
{code:java}
python3 -c "import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])" "/tmp/requirements"
/tmp/requirements
/tmp/requirements/lib/python3.6/site-packages {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
