wenjin272 opened a new pull request, #1376:
URL: https://github.com/apache/flink-agents/pull/1376
### Purpose of change
Nested and dotted YAML entries that resolve to the same agent option now use
the later entry in file order in both Java and Python, matching Flink.
Previously Python always favored dotted entries, while Java could lose
declaration order before expansion.
#### Runtime flow
Environment creation reads `FLINK_CONF_DIR` → the runtime loader selects the
file using the running Flink version → parses standard YAML or legacy lines →
expands keys in document order and extracts `agent.*` options → constructs
`AgentConfiguration` for storage and typed access.
#### Key decisions
Keep file selection, parsing, expansion and extraction together in each
runtime loader. Move legacy parsing into those loaders and remove the second
expansion from Java's configuration constructor. This preserves map-valued
options supplied programmatically and avoids having file-loading
responsibilities in the Python plan layer.
### Behavioral Semantics
#### Interaction decisions
| Runtime/configuration | Result |
| --- | --- |
| No configuration directory | Empty file-derived configuration |
| Flink 1.20 or unknown version; legacy file present | Read
`flink-conf.yaml`, even if `config.yaml` also exists |
| Flink 1.20 or unknown version; legacy file absent | Read `config.yaml` |
| Flink 2.x | Read only `config.yaml`; warn if a legacy file is present |
| Standard YAML; nested and dotted entries collide | Later entry wins, in
either declaration order |
| Legacy file | Only flattened `agent.*` keys; values remain strings |
#### Behavioral contracts
- Standard YAML accepts nested and dotted agent keys, including collisions
within the `agent` mapping.
- Legacy parsing retains Flink's line-based rules: strip comments, split the
first `": "`, skip invalid pairs, and let repeated keys overwrite earlier
values.
- Configuration objects store full option keys without expanding map-valued
options; Trace Log target lists and nested scope values survive plan JSON round
trips.
- File selection, typed reads and programmatic overrides retain their
existing behavior.
#### Failure behavior
A missing or unreadable selected file, malformed YAML, or
configuration-construction error raises a directory-context runtime exception
with its cause retained. There is no retry or fallback after selecting a file.
Invalid legacy lines are warned about and skipped. Python retains its
validation that the YAML document and non-null `agent` section are mappings.
### Tests
| Contract | Coverage |
| --- | --- |
| Later nested/dotted entry wins in either order | Java
`testConfigYamlUsesFileOrder*`; Python `test_config_yaml_uses_file_order*` and
`test_agent_keys_follow_file_order` |
| Nested, dotted and empty YAML inputs | Loader tests in both languages |
| Version-dependent selection and missing-file behavior | Java `testFlink1*`
/ `testFlink2*`; Python `test_flink_1*` / `test_flink_2*`, unknown-version and
no-directory tests |
| Legacy comments, invalid lines, duplicates and string conversion | Legacy
parser and loader cases consolidated into both loader test suites |
| Map-valued options and Trace Log structures survive plan JSON |
`AgentConfigurationTest`, `test_configuration.py`, and loader YAML-to-plan
tests |
| Typed reads and environment configuration overrides | Configuration tests
and both remote-environment test suites |
| Malformed YAML fails with context | Both loaders' unparseable/invalid-YAML
tests |
Focused verification: **Java 71 passed, 4 skipped; Python 197 passed, 1
skipped**. Skips are snapshot-regeneration helpers. Ruff and Spotless passed.
Not verified: the full test suites, a real Flink 1.20 runtime, and every
file-permission/I/O failure. Version selection for 1.20 is covered through
explicit version inputs/fixtures; local runtime verification used Flink 2.3.0.
<details>
<summary>Verification commands and implementation details</summary>
Java (reactor rebuilds the affected modules):
```sh
mvn -pl runtime -am --batch-mode --no-transfer-progress test \
-Dtest=AgentConfigurationTest,AgentPlanCrossLanguageTest,AgentConfigLoaderTest,RemoteExecutionEnvironmentTest
\
-Dsurefire.failIfNoSpecifiedTests=false
```
Python, from `python/`, with `PYTHONPATH` set to the active environment's
purelib:
```sh
.venv/bin/python -m pytest \
flink_agents/plan/tests/test_configuration.py \
flink_agents/plan/tests/test_agent_plan.py \
flink_agents/plan/tests/test_agent_plan_cross_language.py \
flink_agents/runtime/tests/test_agent_config_loader.py \
flink_agents/runtime/tests/test_remote_execution_environment.py -q
--tb=short
```
Parser tests move from the plan/configuration and standalone legacy-parser
suites to the runtime loader suites. Lists are retained as values during
expansion; configuration construction continues to normalize Trace Log targets.
Python configuration loading does not introduce a Java gateway.
</details>
### API
Java `AgentConfiguration(Map)` now expects full option keys and preserves
map values rather than recursively expanding them. File-based nested YAML
remains supported by the loader.
Remove Java `RemoteExecutionEnvironment.loadAgentConfiguration` and its
filename constants, Python `AgentConfiguration.load_from_file`, and the
standalone legacy-parser helpers. Direct callers must migrate to
environment-based loading or supply full keys to configuration objects.
Programmatic setters and the plan JSON format are unchanged.
### Documentation
- [ ] `doc-needed`
- [ ] `doc-not-needed`
- [x] `doc-included`
### Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
Generated-by: Codex CLI 0.153.4 (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
