semlanghi opened a new pull request, #1337:
URL: https://github.com/apache/flink-agents/pull/1337
Linked issue: #1336
### Purpose of change
The Java `RemoteExecutionEnvironment` loads agent configuration only from
`$FLINK_CONF_DIR/config.yaml`.
On Flink 1.20 distributions that still ship only the legacy
`flink-conf.yaml`, every Java agent job
fails at submission with `FileNotFoundException: .../config.yaml`.
#296 / #298 added the legacy-file fallback to the Python runtime only. This
PR does the same for Java.
### Behavioral Semantics
- **File selection:** `config.yaml` is used when it exists. Otherwise
`flink-conf.yaml` is used and a
WARN is logged. This is the order Flink 1.19/1.20's `GlobalConfiguration`
uses. The Python runtime
checks the legacy file first. That only matters when both files are
present, and I'm happy to align
either way.
- **Key shapes:** agent options are read from a nested `agent:` map, as
before, and also from flat
top-level `agent.<key>` entries with the prefix removed. The legacy format
is flat key/value, and
`config.yaml` also allows flattened keys. Defining the same key in both
shapes is not supported. The result is unspecified, as with duplicate
keys in Flink's own config.
- **Failure behavior:** unchanged. If neither file exists, or the file
cannot be parsed, loading still
throws `RuntimeException("Failed to load Flink Agents configuration from
<dir>")`. An unset
`FLINK_CONF_DIR` still gives an empty configuration.
### Tests
New cases in `RemoteExecutionEnvironmentTest`:
- `testLoadAgentConfigurationFromLegacyFile`: only `flink-conf.yaml`, with
flat `agent.*` keys and a
non-agent key that is ignored.
- `testLoadAgentConfigurationFromLegacyFileWithoutAgentOptions`: a typical
distribution
`flink-conf.yaml` with no agent options. This is the reported failure; it
now gives an empty
configuration.
- `testLoadAgentConfigurationFromLegacyFileReadsTypedOptions`: real
`AgentConfigOptions` set as flat
legacy keys, read back with `AgentConfiguration#get(ConfigOption)`.
- `testLoadAgentConfigurationWithFlattenedKeys`: `config.yaml` with a nested
map and a flat key mixed.
- `testLoadAgentConfigurationPrefersConfigYamlOverLegacyFile`: both files
present.
- `testLoadAgentConfigurationFailIfLegacyFileUnparseable`: an unparseable
legacy file still fails,
with the conf dir in the message.
Against the code before this change, the four legacy and flattened-key cases
fail. The two
behaviour-guard cases (preference order, unparseable file) pass on both.
The existing tests are unchanged and pass, including
`testLoadAgentConfigurationFailIfNotLoaded`.
`mvn -pl runtime clean verify` passes.
### API
Adds the public constant
`RemoteExecutionEnvironment.LEGACY_FLINK_CONF_FILENAME`, next to the existing
`FLINK_CONF_FILENAME`. No other API changes.
### Documentation
`docs/operations/configuration.md`: the Java file-selection order, and the
flattened `agent.<key>`
form (Java only, needed with `flink-conf.yaml`).
- [ ] `doc-needed`
- [ ] `doc-not-needed`
- [x] `doc-included`
### Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
Generated-by: Claude Code 2.1.312 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
