Bharath Reddy Gunapati created FLINK-41297:
----------------------------------------------
Summary: PackagedProgramApplication never closes the
PackagedProgram submitted via REST run-application
Key: FLINK-41297
URL: https://issues.apache.org/jira/browse/FLINK-41297
Project: Flink
Issue Type: Bug
Components: Client / Job Submission, Runtime / REST
Affects Versions: 2.3.0
Reporter: Bharath Reddy Gunapati
Split out of FLINK-35376 / PR
[#29473|https://github.com/apache/flink/pull/29473] at the reviewer's request:
FLINK-35376 covers the duplicate {{PackagedProgram}} built by
{{JarHandlerContext#applyToConfiguration}} and affects all release branches,
while this ticket covers the program ownership of the {{run-application}} REST
endpoint, which is new in 2.3.
h3. Problem
{{JarRunApplicationHandler}} builds a {{PackagedProgram}} from the uploaded
jar, wraps it in a {{PackagedProgramApplication}} and submits that to the
Dispatcher. Nothing ever closes that program:
* {{PackagedProgramApplication#dispose()}} only cancels the execution task and
completion future; the {{program}} field is never closed, neither when the
application reaches a terminal state nor when it is disposed.
* If {{DispatcherGateway#submitApplication}} fails (application rejected, RPC
timeout) the handler does not close the program either, and the Dispatcher does
not dispose a rejected application.
* After a recovery {{runApplicationEntryPoint}} reconstructs the program from
the {{PackagedProgramDescriptor}} ({{program == null}} path); that program is
owned by the application and is not closed either.
{{PackagedProgram#close()}} is what deletes the nested {{lib/}} jars that were
extracted into {{java.io.tmpdir}} and closes the user-code classloader. Because
it is never called, every {{run-application}} request leaves its extracted
{{lib/}} jars in tmp until the JobManager process exits ({{deleteOnExit}}), and
the classloader stays open for as long as the application object is referenced.
h3. Constraint found during review of PR 29473
In Application Mode {{ApplicationDispatcherGatewayServiceFactory#create}} hands
the *same* {{PackagedProgram}} instance to a new {{PackagedProgramApplication}}
on every leadership grant. Closing the program when that application finishes
or is disposed makes the next leader run {{main()}} with a closed classloader
and deleted {{lib/}} jars (reproduced locally by the reviewer). The fix
therefore has to distinguish a program the application owns (REST
{{run-application}}, where the handler built it, and the
descriptor-reconstructed program) from a program shared by the entrypoint
factory, for example with an ownership flag, or by letting the
handler/Dispatcher close it once the application is removed.
h3. Proposed scope
* Close the program exactly once when an owned {{PackagedProgramApplication}}
finishes or is disposed, without touching the shared program used by
{{ApplicationDispatcherGatewayServiceFactory}}.
* Dispose (and thereby close the program of) an application that the Dispatcher
rejects, and close the program in {{JarRunApplicationHandler}} when submission
fails before the application was created.
* Regression test: submit a jar with a nested {{lib/}} jar via
{{run-application}} and assert that {{java.io.tmpdir}} is clean once the
application is finished; plus a test that a {{PackagedProgramApplication}}
created by the factory can run twice (leadership regained) with the same
program.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
