wenjin272 opened a new issue, #1291: URL: https://github.com/apache/flink-agents/issues/1291
### Search before asking - [x] I searched existing issues. This tracks the broader cross-job lifecycle work deferred from #943 and #1281. ### Description Support multiple Flink Agents jobs that use the Python runtime reusing the same TaskManager JVM, either sequentially or concurrently. This includes Python jobs and Java jobs with Python components. The scope includes: - Multiple jobs submitted to a Session cluster. - Multiple jobs launched by the same application in Application Mode. The relevant boundary is reuse of the same TaskManager process. Separate applications with dedicated TaskManagers do not share this process state. ### Background and known gaps #943 addresses dependency-generation changes during failover of the same JobID. It explicitly leaves cross-job reuse and cleanup of process-global Python state outside its scope. #1281 reports consecutive-job startup failures, and #1282 documents the limitation and recommends restarting the Session cluster before submitting another job that uses the Python runtime. The broader support work needs to address: 1. **Pemja classloading and native-library ownership.** #1281 reproduced startup failures in successive Session jobs when the distribution JAR is not in `lib`, including `NoSuchFieldException: loadedLibraryNames` and `PyObject cannot be cast to PyObject`. Loading Pemja parent-first avoided those failures in the reported tests, but does not establish cross-job support. 2. **Python imports and dependency lifecycle.** Job completion does not deactivate the job's dependency generation. A subsequent job importing the same package name can reuse an old module from `sys.modules`, potentially executing stale code or accessing resources in a deleted dependency directory. 3. **Function-cache isolation.** The Python function cache is keyed by `(module, qualname)`, without job or dependency-generation identity, so different jobs can reuse the same cached function. 4. **Coordination between active jobs.** A dependency-generation refresh clears the process-wide function cache. As discussed in #943, this can interfere with another job's cache lookup and introduces a possible `KeyError` race. Cleanup must also preserve modules and paths still needed by active jobs. The Session startup failures were reproduced in #1281. The concurrent-job race and Application Mode scenarios still need end-to-end validation; we should not assume the same classloading failure occurs in every deployment layout. ### Expected outcome Define and validate the supported cross-job lifecycle, including sequential reuse, concurrent execution, and recovery of one job while another remains active. Jobs should load their intended code and resources without stale imports, cross-job cache reuse, or cleanup interfering with other jobs. Update the documented support boundary once these guarantees are established. This issue tracks the overall support gap; it does not prescribe a particular implementation or recommend a parent-first configuration as a complete workaround. ### Priority **Minor.** We have not yet seen a concrete user requirement for this capability. The report in #1281 came from local Quickstart validation, with no production use case reported. We therefore consider this a low-priority follow-up and can revisit its priority when concrete user scenarios emerge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
