[
https://issues.apache.org/jira/browse/FLINK-32136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
William Ashley updated FLINK-32136:
-----------------------------------
Description:
On distros where python's {{purelib}} is different than {{platlib}} (e.g.
Amazon Linux 2, but from my research it's all of the Redhat-based ones), you
wind up with components of packages being installed across two different
locations (e.g. {{/usr/local/lib/python3.7/site-packages/pyflink}} and
{{{}/usr/local/lib64/python3.7/site-packages/pyflink{}}}).
{{_find_flink_home}}
[handles|https://github.com/apache/flink/blob/06688f345f6793a8964ec00002175f44cda13c33/flink-python/pyflink/find_flink_home.py#L58C63-L60]
this, and in flink releases <= 1.13.2 its setting of the {{FLINK_LIB_DIR}}
environment variable was the one being used. However, from 1.13.3, a
refactoring of {{launch_gateway_server_process}}
([1.13.2,|https://github.com/apache/flink/blob/release-1.13.2/flink-python/pyflink/pyflink_gateway_server.py#L200]
[1.13.3|https://github.com/apache/flink/blob/release-1.13.3/flink-python/pyflink/pyflink_gateway_server.py#L280])
re-ordered some method calls. {{{}prepare_environment_variable{}}}'s
non-awareness of multiple homes and setting of {{FLINK_LIB_DIR}} now is the one
that matters, and it is the incorrect location.
I've confirmed this problem on Amazon Linux 2 and 2023. The problem does not
exist on, for example, Ubuntu 20 and 22 (for which {{platlib}} ==
{{{}purelib{}}}).
Repro steps on Amazon Linux 2
{quote}{{yum -y install python3 java-11}}
{{pip3 install apache-flink==1.13.3}}
{{python3 -c 'from pyflink.table import EnvironmentSettings ;
EnvironmentSettings.new_instance()'}}
{quote}
The resulting error is
{quote}{{The flink-python jar is not found in the opt folder of the FLINK_HOME:
/usr/local/lib64/python3.7/site-packages/pyflink}}
{{Error: Could not find or load main class
org.apache.flink.client.python.PythonGatewayServer}}
{{Caused by: java.lang.ClassNotFoundException:
org.apache.flink.client.python.PythonGatewayServer}}
{{Traceback (most recent call last):}}
{{ File "<string>", line 1, in <module>}}
{{ File
"/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
line 214, in new_instance}}
{{ return EnvironmentSettings.Builder()}}
{{ File
"/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
line 48, in _{_}init{_}_}}
{{ gateway = get_gateway()}}
{{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
line 62, in get_gateway}}
{{ _gateway = launch_gateway()}}
{{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
line 112, in launch_gateway}}
{{ raise Exception("Java gateway process exited before sending its port
number")}}
{{Exception: Java gateway process exited before sending its port number}}
{quote}
The flink home under /lib64/ does not contain the jar, but it is in the /lib/
location
{quote}{{bash-4.2# find /usr/local/lib64/python3.7/site-packages/pyflink -name
"flink-python*.jar"}}
{{bash-4.2# find /usr/local/lib/python3.7/site-packages/pyflink -name
"flink-python*.jar"}}
{{/usr/local/lib/python3.7/site-packages/pyflink/opt/flink-python_2.11-1.13.3.jar}}
{quote}
was:
On distros where python's {{purelib}} is different than {{platlib}} (e.g.
Amazon Linux 2, but from my research it's all of the Redhat-based ones), you
wind up with components of packages being installed across two different
locations (e.g. {{/usr/local/lib/python3.7/site-packages/pyflink}} and
{{{}/usr/local/lib64/python3.7/site-packages/pyflink{}}}).
{{_find_flink_home}}
[handles|https://github.com/apache/flink/blob/06688f345f6793a8964ec00002175f44cda13c33/flink-python/pyflink/find_flink_home.py#L58C63-L60]
this, and in flink releases <= 1.13.2 its setting of the {{FLINK_LIB_DIR}}
environment variable was the one being used. However, from 1.13.3, a
refactoring of launch_gateway_server_process
([1.13.2,|https://github.com/apache/flink/blob/release-1.13.2/flink-python/pyflink/pyflink_gateway_server.py#L200]
[1.13.3|https://github.com/apache/flink/blob/release-1.13.3/flink-python/pyflink/pyflink_gateway_server.py#L280])
re-ordered some method calls. prepare_environment_variable's non-awareness of
multiple homes and setting of {{FLINK_LIB_DIR}} now is the one that matters,
and it is the incorrect location.
I've confirmed this problem on Amazon Linux 2 and 2023. The problem does not
exist on, for example, Ubuntu 20 and 22 (for which {{platlib}} ==
{{{}purelib{}}}).
Repro steps on Amazon Linux 2
{quote}{{yum -y install python3 java-11}}
{{pip3 install apache-flink==1.13.3}}
{{python3 -c 'from pyflink.table import EnvironmentSettings ;
EnvironmentSettings.new_instance()'}}
{quote}
The resulting error is
{quote}{{The flink-python jar is not found in the opt folder of the FLINK_HOME:
/usr/local/lib64/python3.7/site-packages/pyflink}}
{{Error: Could not find or load main class
org.apache.flink.client.python.PythonGatewayServer}}
{{Caused by: java.lang.ClassNotFoundException:
org.apache.flink.client.python.PythonGatewayServer}}
{{Traceback (most recent call last):}}
{{ File "<string>", line 1, in <module>}}
{{ File
"/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
line 214, in new_instance}}
{{ return EnvironmentSettings.Builder()}}
{{ File
"/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
line 48, in __init__}}
{{ gateway = get_gateway()}}
{{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
line 62, in get_gateway}}
{{ _gateway = launch_gateway()}}
{{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
line 112, in launch_gateway}}
{{ raise Exception("Java gateway process exited before sending its port
number")}}
{{Exception: Java gateway process exited before sending its port number}}
{quote}
The flink home under /lib64/ does not contain the jar, but it is in the /lib/
location
{quote}{{bash-4.2# find /usr/local/lib64/python3.7/site-packages/pyflink -name
"flink-python*.jar"}}
{{bash-4.2# find /usr/local/lib/python3.7/site-packages/pyflink -name
"flink-python*.jar"}}
{{/usr/local/lib/python3.7/site-packages/pyflink/opt/flink-python_2.11-1.13.3.jar}}
{quote}
> Pyflink gateway server launch fails when purelib != platlib
> -----------------------------------------------------------
>
> Key: FLINK-32136
> URL: https://issues.apache.org/jira/browse/FLINK-32136
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Affects Versions: 1.13.3
> Reporter: William Ashley
> Priority: Major
>
> On distros where python's {{purelib}} is different than {{platlib}} (e.g.
> Amazon Linux 2, but from my research it's all of the Redhat-based ones), you
> wind up with components of packages being installed across two different
> locations (e.g. {{/usr/local/lib/python3.7/site-packages/pyflink}} and
> {{{}/usr/local/lib64/python3.7/site-packages/pyflink{}}}).
> {{_find_flink_home}}
> [handles|https://github.com/apache/flink/blob/06688f345f6793a8964ec00002175f44cda13c33/flink-python/pyflink/find_flink_home.py#L58C63-L60]
> this, and in flink releases <= 1.13.2 its setting of the {{FLINK_LIB_DIR}}
> environment variable was the one being used. However, from 1.13.3, a
> refactoring of {{launch_gateway_server_process}}
> ([1.13.2,|https://github.com/apache/flink/blob/release-1.13.2/flink-python/pyflink/pyflink_gateway_server.py#L200]
>
> [1.13.3|https://github.com/apache/flink/blob/release-1.13.3/flink-python/pyflink/pyflink_gateway_server.py#L280])
> re-ordered some method calls. {{{}prepare_environment_variable{}}}'s
> non-awareness of multiple homes and setting of {{FLINK_LIB_DIR}} now is the
> one that matters, and it is the incorrect location.
> I've confirmed this problem on Amazon Linux 2 and 2023. The problem does not
> exist on, for example, Ubuntu 20 and 22 (for which {{platlib}} ==
> {{{}purelib{}}}).
> Repro steps on Amazon Linux 2
> {quote}{{yum -y install python3 java-11}}
> {{pip3 install apache-flink==1.13.3}}
> {{python3 -c 'from pyflink.table import EnvironmentSettings ;
> EnvironmentSettings.new_instance()'}}
> {quote}
> The resulting error is
> {quote}{{The flink-python jar is not found in the opt folder of the
> FLINK_HOME: /usr/local/lib64/python3.7/site-packages/pyflink}}
> {{Error: Could not find or load main class
> org.apache.flink.client.python.PythonGatewayServer}}
> {{Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.client.python.PythonGatewayServer}}
> {{Traceback (most recent call last):}}
> {{ File "<string>", line 1, in <module>}}
> {{ File
> "/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
> line 214, in new_instance}}
> {{ return EnvironmentSettings.Builder()}}
> {{ File
> "/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
> line 48, in _{_}init{_}_}}
> {{ gateway = get_gateway()}}
> {{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
> line 62, in get_gateway}}
> {{ _gateway = launch_gateway()}}
> {{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
> line 112, in launch_gateway}}
> {{ raise Exception("Java gateway process exited before sending its port
> number")}}
> {{Exception: Java gateway process exited before sending its port number}}
> {quote}
> The flink home under /lib64/ does not contain the jar, but it is in the /lib/
> location
> {quote}{{bash-4.2# find /usr/local/lib64/python3.7/site-packages/pyflink
> -name "flink-python*.jar"}}
> {{bash-4.2# find /usr/local/lib/python3.7/site-packages/pyflink -name
> "flink-python*.jar"}}
> {{/usr/local/lib/python3.7/site-packages/pyflink/opt/flink-python_2.11-1.13.3.jar}}
> {quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
