junjunclub opened a new pull request, #11186:
URL: https://github.com/apache/seatunnel/pull/11186
### Purpose of this pull request
This PR adds a `connect` mode (Layer 1) to the `--dry-run` option of the
SeaTunnel (Zeta) CLI, following up on the merged Layer 0 work (#10763) and
implementing the Layer 1 design described in #10681:
```shell
sh bin/seatunnel.sh --config <config> --dry-run connect
```
It is opened as a **draft / RFC**. The implementation direction (a
connector-level `validateConnection`-style SPI) is the one proposed by the
maintainer in #10681 ("Layer 1 requires a `validateConnection()` SPI method").
What is **not** yet decided anywhere is the **connector rollout scope** — that
is the main thing I'd like to agree on before connectors start implementing the
hooks. Please see "Open questions" below.
#### Relationship to #10681 / #10763
- #10681 (feature design) defines Layer 1 = connectivity + schema inference
+ source/sink schema compatibility, and explicitly lists `All connectors (Layer
1 requires a validateConnection() SPI method)` under *Affected Modules*.
- #10763 merged Layer 0 (`static`). Reviewers asked to keep future modes
(`connect/sample/shadow`) **out of the public path until runtime + docs + tests
are ready together**, and to keep PR scope tight. This PR tries to honor both.
| Layer | Mode | What it checks | Status |
|-------|------|----------------|--------|
| 0 | `static` | Offline config / plugin / DAG validation, no network
required | merged (#10763) |
| 1 | `connect` | Schema inference + connectivity / schema-compatibility
validation, **no data movement** | **this PR** |
| 2 | `sample` | Read a small sample through source/transform without
writing | later |
| 3 | `shadow` | Non-destructive shadow run | later |
#### What this PR implements (Layer 1 = `connect`)
`connect` walks the whole source → transform → sink DAG and validates it
**without creating any runtime objects**. It does **not** create source
readers, sink writers, committers, or save-mode handlers; it does not submit a
job, read source records, execute save-mode logic, or write target data.
**1. New opt-in factory SPI hooks** (all `default`, so fully backward
compatible — this is the `validateConnection()` SPI from #10681):
- `TableSourceFactory#inferSchemaForDryRun(context)` — infer source schemas
without creating readers (default delegates to `discoverTableSchemas`).
- `TableSourceFactory#validateConnectionForDryRun(context, catalogTables)` —
metadata-level check: credentials, permissions, source table/topic/path
existence (default no-op).
- `TableSinkFactory#validateConnectionForDryRun(context)` — sink
connectivity + schema compatibility, target existence/createability (default
no-op).
**2. `DryRunConnectValidator`** (new) orchestrates the DAG:
- Source: discover factory → infer schema → validate connection; propagate
inferred schemas downstream.
- Transform: resolve `plugin_input`/`plugin_output` wiring
(dependency-ordered), build the transform to compute produced schemas, and
check input schema-type consistency.
- Sink: resolve input vertices (incl. multi-table rules) →
`validateConnectionForDryRun`.
- Reuses the same connector-jar resolution / child-first classloader split
(`JobPluginClasspathHelper`) introduced in #10763, so dry-run and runtime stay
aligned.
**3.** `FakeSourceFactory` implements `inferSchemaForDryRun` as a reference
example. CLI routing updated so `--dry-run connect` is accepted;
`sample`/`shadow` still error as not implemented (keeping the #10763 discipline
of not exposing unfinished modes).
#### Scope: connectors are intentionally NOT in this PR
Today only `FakeSource` implements a hook, so for most connectors
`validateConnectionForDryRun` is a no-op and `connect` effectively performs
**schema inference + DAG wiring validation** only. Implementing real
connectivity checks for the ~57 source / ~64 sink connectors is **explicitly
out of scope here** and is meant to land as focused follow-up PRs (one
connector or a small batch at a time), so this PR stays reviewable.
#### Open questions for maintainers (please weigh in)
1. **SPI shape**: Do the proposed factory hooks (`inferSchemaForDryRun`,
`validateConnectionForDryRun`) match the `validateConnection()` SPI intent from
#10681? Any preferred name / placement (factory vs source/sink instance vs
reuse of the existing `Catalog` SPI)?
2. **Public API surface**: Is adding 3 new `default` methods to
`TableSourceFactory` / `TableSinkFactory` acceptable, given the
cross-module-scope concern raised on #10763?
3. **Rollout / pilot connector**: Should this PR stay framework-only (+
`FakeSource` reference) and add connectors in follow-up PRs? If yes, which
connector do you want as the first real implementation (I'd suggest **JDBC**,
since it already has a `Catalog`)?
4. **Unsupported connectors**: When a connector does not implement the hook,
should `connect` stay a silent no-op (current), or explicitly report `SKIPPED /
not verified` per connector so users aren't misled?
5. **Boundary**: Confirm the hard boundary — no runtime instances, no
save-mode execution, no DDL/DML, no data read/write.
6. **Command / contract**: `connect` runs as a validate command (like
`static`) and stays CLI-only (REST keeps rejecting dry-run, consistent with
#10763). Should `connect` always run `static` first (cumulative)?
### Does this PR introduce _any_ user-facing change?
Yes.
- `-d` / `--dry-run` now accepts `connect` in addition to `static`. Existing
`static` / `--check` behavior is unchanged.
- New public default methods on `TableSourceFactory` / `TableSinkFactory`
(backward compatible — connectors are not required to implement them).
- Docs updated: `docs/en/engines/zeta/user-command.md`,
`docs/zh/engines/zeta/user-command.md`.
- No change to existing job execution behavior.
### How was this patch tested?
- Unit tests for CLI parsing of `--dry-run connect`
(`ClientCommandArgsTest`).
- Unit tests for the connect validation flow in
`SeaTunnelConfValidateCommandTest` (schema inference + DAG wiring;
positive/negative cases) using `FakeSource`.
- `<!-- TODO before un-drafting: run and confirm -->` `./mvnw
spotless:apply` and `./mvnw -q -DskipTests verify` pass locally.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice — N/A.
* [x] Documentation updated for the new feature (`docs/en` + `docs/zh`).
* [ ] `incompatible-changes.md` — N/A (backward compatible default methods).
* [ ] Connector code checklist — N/A (no new connector; connector hook
implementations are follow-up PRs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
