bgeng777 opened a new pull request, #29678: URL: https://github.com/apache/flink/pull/29678
## What is the purpose of the change [FLINK-41206](https://issues.apache.org/jira/browse/FLINK-41206) When a process-mode Python worker dies, the task may report only a gRPC error or "Failed to close remote bundle". Default TaskManager logs do not clearly identify the worker's exit. This change records the worker ID, PID and exit code at INFO level. Exit code 137 helps users investigate SIGKILL, including possible OOM kills; it does not prove OOM. ## Brief change log - Use `exec` in the Unix launcher so Java manages Python directly, preserving its exit status when cleanup races with worker failure. Boot output is still copied through a background `tee`. - Copy Beam 2.75.0's `ProcessManager` and add an asynchronous exit callback with one neutral INFO record per exit. Exclude the dependency's copy from the shaded JAR. - Include PID and worker ID in the Python startup log and add real-process regression tests. No new runtime waits or polling are added. The task's final exception is unchanged. Background `tee` may finish writing the boot log after Python exits, and its failures no longer determine the launcher's exit code. Cleanup now sends SIGTERM directly to Python. Windows launcher behavior is unchanged. ## Verifying this change - Seven launcher tests cover exit status, boot output, direct process ownership, and SIGKILL followed by immediate cleanup. Five failed against the previous launcher; all pass with the change. - Three boot tests pass, including startup failure. - `./mvnw -o -pl flink-python verify` passes: 370 unit tests and one integration test. The packaged JAR contains the local `ProcessManager` exactly once. - A standalone-cluster prototype of the same launcher/callback behavior logged the Python PID and exit code 137 in two runtime SIGKILL runs at default INFO level. Normal completion, cancellation and missing-dependency startup failure were also checked. The final packaged JAR has not been rerun in that cluster. - Full-repository `clean verify` has not been run. Offline verification skipped CycloneDX; the previous-version artifact was unavailable for the API compatibility comparison. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no; it overrides one existing Beam class. - The public API: no. - The serializers: no. - The runtime per-record code paths: no. - Anything that affects deployment or recovery: yes, Python worker process startup and cleanup. - The S3 file system connector: no. ## Documentation - Does this pull request introduce a new feature? no; this is a process-exit diagnostic fix. - If yes, how is the feature documented? not applicable. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Codex (gpt-6-astra) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
