[
https://issues.apache.org/jira/browse/FLINK-33294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-33294:
-----------------------------------
Priority: Major (was: Blocker)
> PyFlink minicluster is shut down after every job completion
> -----------------------------------------------------------
>
> Key: FLINK-33294
> URL: https://issues.apache.org/jira/browse/FLINK-33294
> Project: Flink
> Issue Type: Bug
> Reporter: Mehmet Aktas
> Priority: Major
> Attachments: pyflink.log
>
>
> *What*
> A new PyFlink minicluster is created for each job. This significantly slows
> down execution.
> We tried to configure the minicluster to prevent it from shutting down after
> every job completion, as shown
> [here|https://github.com/mfatihaktas/flink-playground/blob/main/tests/test_pyflink_mini_cluster.py#L17].
> This, however, did not seem to make any change.
> *Why*
> Our project relies on testing many PyFlink jobs on Github Actions. This issue
> is causing our testing process to take an unfeasibly long time.
> *How to reproduce*
> * Clone [https://github.com/mfatihaktas/flink-playground]
> * {{make env; make install}} (dependencies: direnv, make)
> * Run {{run_tests.sh}}
> * Find the directory in which PyFlink writes the logs
> *
> {code:python}
> python -c "import pyflink;import
> os;print(os.path.dirname(os.path.abspath(pyflink.__file__))+'/log')" {code}
> * Search in `flink-***.log` for “minicluster shutting down/created”. An
> example log is attached here for convenience: [^pyflink.log]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
