[
https://issues.apache.org/jira/browse/FLINK-40628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40628:
-----------------------------------
Labels: pull-request-available (was: )
> PyFlink TTL initialization can hang task threads and leave jobs stuck in
> INITIALIZING
> -------------------------------------------------------------------------------------
>
> Key: FLINK-40628
> URL: https://issues.apache.org/jira/browse/FLINK-40628
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Affects Versions: 1.19.3
> Reporter: Bowen Li
> Assignee: Bowen Li
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0
>
>
> *User impact*
> Python operators can remain stuck in INITIALIZING, preventing the job from
> processing data and completing checkpoints. The TaskManager process can
> remain alive while its task threads are blocked, so process-liveness checks
> may not detect the failure. Restarting the application can recover it, but
> the problem can recur.
> *Cause*
> With default TTL background cleanup, embedded state initialization follows:
> RuntimeContext.get_state()
> → default RocksDB cleanup strategy
> → Duration.of_days(30)
> → get_gateway()
> → subprocess.Popen(preexec_fn=...)
> This unexpectedly launches a separate gateway process from the multithreaded
> TaskManager. The child process can deadlock before exec, leaving the calling
> task thread waiting and the job unable to finish initialization. Python
> documents this deadlock risk.
> *Verification*
> The unexpected gateway request was reproduced through a real
> JVM/Pemja/RocksDB job using a Flink 1.19.1-derived distribution, Python
> 3.12.9, and Pemja 0.5.5. The same code path remains in upstream 1.19.3,
> 1.20.5, 2.3.0, and inspected master.
> The production initialization hang was observed; its exact blocked lock was
> not captured.
> *Proposed fix*
> Avoid constructing a Py4J-backed Duration during embedded TTL initialization.
> Keep the default duration as a Python value and convert it through Pemja. Add
> a regression test ensuring RuntimeContext.get_state() with default TTL never
> requests a gateway.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
