[
https://issues.apache.org/jira/browse/FLINK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436601#comment-17436601
]
Dian Fu commented on FLINK-24663:
---------------------------------
[~jackwangcs] Thanks for reporting this issue. Would you like to fix it?
> PyFlink failed to get the site packege path because of SyntaxError in shell
> command
> -----------------------------------------------------------------------------------
>
> Key: FLINK-24663
> URL: https://issues.apache.org/jira/browse/FLINK-24663
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Affects Versions: 1.12.3
> Reporter: jackwangcs
> Priority: Major
>
> Flink throws an exception when it tries to install 3rd party dependencies:
> {code:java}
> Caused by: java.io.IOException: Failed to execute the command: python -c
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
> /mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirements
> output: File "<string>", line 1
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
>
>
>
>
> ^SyntaxError: invalid syntax
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
> at
> org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
> ....{code}
> This can be reproduced by running the python script in a bash shell:
> {code:java}
> python3 -c import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1]) /tmp/requirements
> -bash: syntax error near unexpected token `(' {code}
> The solution is to quote all argements to execute:
> {code:java}
> python3 "-c" "import sys;from distutils.dist import Distribution;install_obj
> = Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])" "/tmp/requirements"
> /tmp/requirements
> /tmp/requirements/lib/python3.6/site-packages{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
