morningman opened a new pull request, #66347:
URL: https://github.com/apache/doris/pull/66347
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Giving one connector a new deployment-level setting currently costs two
edits in
the engine: an `@ConfField` in fe-common's `Config`, and a line in fe-core's
`DefaultConnectorContext.buildEnvironment()` that forwards it. A connector
plugin
cannot read `Config` itself — it loads child-first, so its own bundled copy
shadows the engine's and every field reads back as a code default — so the
engine
has to carry each key by name. The result is that `fe-core` knows the config
key
names of connectors it is otherwise entirely agnostic about, and that list
only
grows.
**This PR gives a connector plugin a configuration file of its own.** The
engine
reads `<pluginDir>/<name>.conf` (where `<name>` is
`ConnectorProvider.name()`),
parses it generically, and serves it back through the new
`ConnectorContext.getConnectorConfig()`. No key name of any connector reaches
`fe-core`, and **a new connector needs no engine change at all** to add a
deployment-level setting.
Connectors read a setting through one entry point:
```java
ConnectorConf.get(context, "drivers_dir", "jdbc_drivers_dir", null)
// ^ key in <name>.conf ^ the fe.conf key it used
to live under
```
Resolution is **plugin conf → fe.conf → default**, with blank treated as "not
set" at each step (an operator who writes `key=` has not configured it, and
reading it as a set empty string would let one stray line mask the fe.conf
value
actually in effect).
Six settings across five connectors are moved onto the new channel. **Every
`@ConfField` is kept** and stays the fallback, so an existing deployment
upgrades
with nothing to edit and behaves exactly as before:
| connector (`name()`) | conf file | key | fe.conf fallback |
| --- | --- | --- | --- |
| `trino-connector` | `trino-connector.conf` | `plugin_dir` |
`trino_connector_plugin_dir` |
| `hms` (hive) | `hms.conf` | `default_file_format` |
`hive_default_file_format` |
| `hms` (hive) | `hms.conf` | `enable_create_bucket_table` |
`enable_create_hive_bucket_table` |
| `jdbc` | `jdbc.conf` | `drivers_dir` | `jdbc_drivers_dir` |
| `jdbc` | `jdbc.conf` | `force_sqlserver_encrypt_false` |
`force_sqlserver_jdbc_encrypt_false` |
| `iceberg` / `paimon` | `iceberg.conf` / `paimon.conf` | `drivers_dir` |
`jdbc_drivers_dir` |
| `iceberg` / `paimon` | `iceberg.conf` / `paimon.conf` |
`metastore_client_timeout_second` | `hive_metastore_client_timeout_second` |
Two of these are shared by several connectors at the fe.conf end (one
`jdbc_drivers_dir` serves jdbc, iceberg and paimon). A per-plugin file cannot
express that, so a deployment that moves them sets the value in each plugin's
conf; leaving them commented out keeps the single shared fe.conf value,
which is
what every existing deployment gets. Both templates say so.
`doris_home` and `doris_version` stay in `getEnvironment()` — they are not
connector settings. `jdbc_driver_secure_path` is dropped from it: no
connector
ever read it (the JDBC allow-list is enforced in fe-core by `JdbcResource`,
which
reads `Config` directly), so it was a dead key that read like a connector
setting. The `Config` field itself is unchanged.
**Packaging.** A plugin ships `src/main/resources/<name>.conf.template`;
`build.sh`
seeds the live `<name>.conf` from it with `cp -n`, globbing
`*.conf.template` with
no connector named, so a new connector needs no `build.sh` change either.
The live
`.conf` is deliberately **not** in the plugin zip, so the ordinary upgrade —
unzipping a newer plugin build over the deployed directory — refreshes the
jars
and the template but never the administrator's file.
Note the conf file is named after `ConnectorProvider.name()`, **not** after
the
plugin directory: `plugins/connector/hive/` holds `hms.conf` and
`plugins/connector/trino/` holds `trino-connector.conf`. The directory name
is the
deployer's choice and cannot be what the engine keys on. Each connector
carries a
test asserting its shipped template name still tracks `name()`.
`connector.plugin.api.version` stays at **1.0**. The added
`ConnectorContext.getConnectorConfig()` is a `default` method, nothing
outside
fe-connector-spi implements `ConnectorContext`, and the only decorators of it
extend the parent-first `ForwardingConnectorContext` — loaded from the FE's
own
classpath, so they inherit the new forward without being rebuilt. A connector
plugin built before this change loads and behaves exactly as it did; it
simply
never reads the new map.
### Release note
Connector plugins can now carry their own deployment-level configuration
file,
`<DORIS_HOME>/plugins/connector/<dir>/<name>.conf`, seeded from a template
shipped
with each plugin. Settings there take precedence over the corresponding
`fe.conf`
keys, which keep working unchanged — no action is required when upgrading.
The
file must be maintained on every FE node and takes effect after an FE
restart.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
**Unit tests** — new: `ConnectorConfFileTest`, `ConnectorConfTest` (spi),
`ConnectorPluginConfTest` (fe-core, drives `loadPlugins` + `createConnector`
on real
plugin directories), `IcebergConnectorConfTest`, `PaimonConnectorConfTest`,
plus
per-connector cases and a template-name guard in `TrinoBootstrapTest`,
`HiveConnectorMetadataDdlTest`, `JdbcUrlNormalizerTest`. Existing
`ForwardingConnectorContextTest` covers the new forward by reflection.
Each was mutation-checked (precedence reversed, forward removed, surface
baseline
reverted, conf not keyed per provider, conf never attached, template
renamed) and
confirmed to fail.
**Manual test** — `sh build.sh --fe`, then verified in
`output/fe/plugins/connector/`:
- `hive/hms.conf`, `trino/trino-connector.conf`, `iceberg/iceberg.conf`,
`jdbc/jdbc.conf`, `paimon/paimon.conf` are created and byte-identical to
their
templates; `es/`, `hudi/`, `maxcompute/` ship no template and get no file,
with
no error.
- Every seeded file has **0 active settings** (all commented out), so a fresh
deployment behaves exactly as before.
- Upgrade safety: hand-edited `hms.conf`, replayed the deploy step (`unzip
-o` +
the `cp -n` loop) from the real plugin zip — the edit survives and only the
template is refreshed.
- Behavior changed:
- [x] Yes.
Two, both narrow:
1. A deployment-level setting now resolves from the plugin's `<name>.conf`
before
`fe.conf`. For a deployment that does not edit the seeded (all-commented)
file,
nothing changes.
2. A blank `hive_default_file_format` in fe.conf now falls through to `orc`
instead
of reaching the metastore create as an empty format string. An empty file
format
was never a working configuration.
- Does this need documentation?
- [x] Yes. <!-- doris-website PR not filed yet -->
Each shipped `.conf.template` documents its own keys inline, and
`fe/fe-connector/README.md` plus `fe-connector-api/package-info.java` (Rule
7) are
updated for connector authors. A doris-website page for operators is still
to be
written.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
