Mehmet Aktas created FLINK-33294:
------------------------------------
Summary: PyFlink minicluster is shut down after every job
completion
Key: FLINK-33294
URL: https://issues.apache.org/jira/browse/FLINK-33294
Project: Flink
Issue Type: Bug
Reporter: Mehmet Aktas
Attachments: pyflink.log
*What*
A new PyFlink minicluster is created for each job. This significantly slows
down execution.
We tried to configure the minicluster to prevent it from shutting down after
every job completion, as shown
[here|https://github.com/mfatihaktas/flink-playground/blob/main/tests/test_pyflink_mini_cluster.py#L17].
This, however, did not seem to make any change.
*Why*
Our project relies on testing many PyFlink jobs on Github Actions. This issue
is causing our testing process to take an unfeasibly long time.
*How to reproduce*
* Clone [https://github.com/mfatihaktas/flink-playground]
* {{make env; make install}} (dependencies: direnv, make)
* Run {{run_tests.sh}}
* Find the directory in which PyFlink writes the logs
*
{code:python}
python -c "import pyflink;import
os;print(os.path.dirname(os.path.abspath(pyflink.__file__))+'/log')" {code}
* Search in `flink-***.log` for “minicluster shutting down/created”. An
example log is attached here for convenience: [^pyflink.log]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
