sangkyoonnam opened a new issue, #1281: URL: https://github.com/apache/flink-agents/issues/1281
### Search before asking - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description On a standalone session cluster set up with Setup's manual PyFlink installation, the first Python agent job runs, but later ones on the same TaskManager fail in `ActionExecutionOperator.open` until the cluster restarts. That installation leaves the flink-agents distribution JAR out of `$FLINK_HOME/lib`, as #818 intends. I propose documenting `classloader.parent-first-patterns.additional: pemja.` for this layout, which keeps #818's point: the flink-agents JARs stay in `pipeline.jars` and their integration dependencies stay off the system class path. I reproduced this on main (`a50a8a18`) on macOS arm64 and Linux aarch64, and with `flink-agents==0.3.1` from PyPI on macOS arm64. In each three-job sequence I ran (Linux with pemja 0.5.7, macOS with pemja 0.6.3, and macOS with flink-agents 0.3.1), the first job produced output and the second and third failed. Restarting the cluster let a first job run again. From main with stock Flink 2.3.0 (pemja 0.5.7) on JDK 17: ``` java.lang.RuntimeException: java.lang.NoSuchFieldException: loadedLibraryNames at pemja.utils.CommonUtils.loadLibrary(CommonUtils.java:117) at pemja.utils.CommonUtils.loadPython(CommonUtils.java:50) at pemja.core.PythonInterpreter$MainInterpreter.initialize(PythonInterpreter.java:365) at pemja.core.PythonInterpreter.initialize(PythonInterpreter.java:144) at pemja.core.PythonInterpreter.<init>(PythonInterpreter.java:45) at org.apache.flink.agents.runtime.env.EmbeddedPythonEnvironment.getInterpreter(EmbeddedPythonEnvironment.java:45) at org.apache.flink.agents.runtime.operator.PythonBridgeManager.open(PythonBridgeManager.java:176) ``` On macOS with JDK 11, and with JDK 17 after swapping pemja 0.6.3 into the Flink 2.3.0 JARs and the Python environment (FLINK-38500 bumps Flink to 0.6.3), the interpreter starts and the job fails one step later: ``` java.lang.ClassCastException: class pemja.core.object.PyObject cannot be cast to class pemja.core.object.PyObject (pemja.core.object.PyObject is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @33f922f3; pemja.core.object.PyObject is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @1efbbcf8) at org.apache.flink.agents.runtime.python.utils.PythonResourceAdapterImpl.open(PythonResourceAdapterImpl.java:96) ``` The `ClassCastException` shows `pemja.core.object.PyObject` loaded by two different job class loaders. `flink run -py` adds the flink-python JAR from `opt` to the job's libraries (`PackagedProgram.getJobJarAndDependencies`), and the default parent-first patterns include `org.apache.flink.` but not `pemja.`. The Python entry point registers the flink-agents JARs through `pipeline.jars`, so when they aren't in `lib`, each job's `ChildFirstClassLoader` defines the runtime classes and resolves their pemja references from that job's libraries. I suspect pemja keeps native references to classes from the first job's loader; I haven't traced its native code. PyFlink's own embedded operators come from the flink-python JAR that Setup copies into `lib` and resolve pemja through the parent loader. As a control, a plain PyFlink thread-mode job (`from_collection` → `map` → `print`) finished twice in a row on one cluster with each pemja version. I tested two ways to avoid it: - `classloader.parent-first-patterns.additional: pemja.` in `config.yaml` before the cluster starts. Three consecutive jobs produced output on Linux with pemja 0.5.7 and on macOS with pemja 0.6.3. On macOS with pemja 0.5.7, four consecutive jobs all started, and three produced output before I cancelled them. - The dist JAR in `$FLINK_HOME/lib` before the cluster starts, which the normal `tools/install.sh` path does whether or not PyFlink is enabled. Three consecutive jobs produced output on macOS. I didn't test this on Linux. The second departs from #818's layout; the first doesn't, since pemja comes from the flink-python JAR already in `lib`. I'll send a PR with the Setup step. The step would be unnecessary if Flink's `classloader.parent-first-patterns.default` included `pemja.`, since pemja's native library can belong to only one class loader at a time. I think that's the right place for the actual fix, and I'd like to take it to a FLINK ticket as a follow-up. Does that work for you, or would you rather handle it in the flink-agents runtime? ### How to reproduce Follow Setup's manual installation for a PyFlink job: Flink 2.3.0 with `flink-python` copied from `opt` to `lib`, a venv with `apache-flink==2.3.0` and `flink-agents` (`pip install flink-agents==0.3.1`, or the wheel built by `./tools/build.sh`), `PYTHONPATH` set, and no dist JAR in `lib`. Start the cluster, and Ollama with `qwen3:8b` as in the Quickstart. The examples are under `$PYTHONPATH/flink_agents/examples/quickstart`. 1. Submit `workflow_single_agent_example.py` with `$FLINK_HOME/bin/flink run -py`, and wait for output in the TaskManager `.out`. 2. Cancel the job and wait until its slot is free. 3. Submit `yaml_workflow_agent_example.py`, then `workflow_single_agent_example.py` again, without restarting the cluster. 4. Both fail while the operator opens, with one of the errors above. ### Version and environment - flink-agents main at `a50a8a18`, and 0.3.1 from PyPI - Flink 2.3.0 standalone session cluster, one TaskManager - macOS 15.3.1 on Apple M4: JDK 17.0.14 and 11.0.32.1, Python 3.11.15 - Debian 12 aarch64 in Docker: OpenJDK 17.0.20.1, Python 3.11 I haven't tested application mode, Kubernetes, YARN, x86_64, or a full `install.sh` run. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
