wenjin272 opened a new pull request, #1198: URL: https://github.com/apache/flink-agents/pull/1198
Linked issue: #1121 (part of #1055) ### Purpose of change #### Outcome and intent Java callers declare Python resources with `PythonResourceDescriptor`; Python callers declare Java resources with `JavaResourceDescriptor`. Native resources retain `ResourceDescriptor`. Callers name the real implementation class instead of an internal wrapper and a `pythonClazz` / `java_clazz` argument. This draft moves cross-language implementation details out of API and aligns the declaration model in Java, Python, and YAML. It expands the wrapper relocation into declaration cleanup so API no longer needs wrapper shells or wrapper-name constants. The API and compatibility choices remain open for discussion. #### Runtime flow 1. An annotation, decorator, programmatic registration, or YAML section supplies the resource type. The descriptor carries implementation language, target module/class, and initialization arguments. 2. Plan selects the language-specific provider. Java targets use an empty module plus the full class name; Python targets use a module plus class name. Plan JSON preserves these fields in both directions. 3. Providers instantiate native resources or select the concrete cross-language wrapper by resource type. Runtime supplies the bridge adapter; Python runtime also unwraps Flink metric groups before forwarding them to Java. 4. Existing resource-cache ownership and wrapper lifecycle paths remain responsible for resource cleanup. #### Key decisions - Concrete wrappers and adapter contracts live in Plan on both sides; interpreter/bridge implementations and runtime metric handling stay in Runtime. `ResourceProvider` stays in Plan, without a factory redesign. - API retains resource abstractions and descriptors, not per-resource bridge shells. Java API no longer depends on Pemja. - No compatibility shims for the removed wrapper declarations or imports. Preserving them would retain the API-to-implementation coupling this change removes. ### Behavioral Semantics #### Interaction decisions | Declaration site | Implementation | Public declaration | Provider | |---|---|---|---| | Java | Java | `ResourceDescriptor` | Java | | Java | Python | `PythonResourceDescriptor` | Python | | Python | Python | `ResourceDescriptor` | Python | | Python | Java | `JavaResourceDescriptor` | Java | | YAML on either host | `type: java` / `type: python` | Actual target class and language | Selected implementation language | The general cross-language descriptor routes cover chat connections/setups, embedding connections/setups, and vector stores. Existing specialized MCP handling remains; this does not add arbitrary cross-language resource support. #### Behavioral contracts - Foreign resource declarations can build a Plan without importing/loading the foreign target class in the declaring language. - Language, actual target, and business arguments survive Plan serialization across Java and Python; wrapper metadata is not injected into arguments. - Decorated Python resource declaration methods run once during provider extraction. - YAML aliases continue resolving to the selected language's implementation, with internal wrapper selection deferred to Plan. - Metric binding continues forwarding Java metric groups through Python wrappers. Moved Java wrappers retain their handle-ownership and close behavior. #### Failure behavior Incomplete Python module/class declarations and conflicting provider/descriptor languages raise validation errors. Unsupported cross-language YAML types are rejected; unmapped provider types and missing adapters fail rather than falling back to another language. Class loading, construction, and bridge-call failures propagate through the existing provider/wrapper paths; this adds no retry or fallback policy. Python runtime metric forwarding accepts `FlinkMetricGroup` or `None` and rejects other metric-group implementations. ### Tests | Contract | Evidence | |---|---| | Declare without loading foreign target; preserve target and arguments | `ResourceDescriptorTest`, `test_resource_declarations.py`, bidirectional `test_agent_plan_compatibility.sh` with nonexistent foreign target classes | | Reject invalid targets and conflicting language | Descriptor tests and `test_provider_rejects_conflicting_wire_language` | | Invoke declaration once and select provider | `test_decorator_routes_without_importing_java_and_runs_once`, `AgentPlanTest` | | Preserve YAML resolution | Java YAML loader/parity tests and Python YAML tests | | Forward metrics and reject invalid groups | `test_cross_language_metric_group.py` | | Preserve handle cleanup and cache lifecycle | Moved wrapper tests, `PythonObjectScopeTest`, `PythonResourceProviderTest`, resource-cache tests | Local verification on this implementation: Java API/Plan/Runtime plus MCP reactor tests: **1,773 passed, 13 skipped**. Python API/YAML/Plan/Runtime and the output-schema declaration check: **748 passed, 13 skipped**. Bidirectional Plan JSON compatibility, cross-language example compilation, ResourceName consistency, Ruff, Spotless, and whitespace checks passed. Not verified: live external-model/vector-store end-to-end execution, distribution/class-loading across every supported Flink version, and restoration of old checkpoints or old serialized plans. Skipped tests are not counted as verified. <details> <summary>Implementation boundaries and build evidence</summary> Wrappers and adapter contracts are under Java `plan.resource.python` and Python `plan.resource.java`. Python conversion helpers move into Plan; runtime entry points continue exposing the functions invoked by the Java bridge. Runtime adapters retain interpreter and metric implementation dependencies. The Python Java chat connection's close method now uses its stored `_j_resource` handle. Verification used targeted Maven reactor builds and the existing Python environment with this branch's source first on `PYTHONPATH`. The compatibility JAR was rebuilt with `-Dmaven.jar.forceCreation=true` before the successful bidirectional script run, avoiding reuse of an already-shaded artifact. Tests were run before publication, not against a rebased version of latest main. </details> ### API Breaking change: remove API wrapper classes, wrapper-name constants, and Python's `java_resource` marker decorator. Migrate Java declarations to `PythonResourceDescriptor.Builder.newBuilder(pythonFqn)` and Python declarations to `JavaResourceDescriptor(clazz=javaFqn)`; pass business arguments as before. Internal bridge imports move to Plan. The wire descriptor now includes `language` and the actual implementation target. Old wrapper-based plans are not a supported compatibility format; regenerate plans. Public native resource abstractions, native declaration entry points, and YAML syntax remain. No checkpoint migration guarantee is made. ### Documentation - [ ] `doc-needed` - [ ] `doc-not-needed` - [x] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: Codex 0.153.4 (GPT-5; exact model version unavailable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
