zhangshenghang opened a new pull request, #12865:
URL: https://github.com/apache/seatunnel/pull/12865
## Purpose of this PR
Yes, I am happy to contribute to Apache SeaTunnel. This PR fixes a batch of
real documentation inaccuracies found by auditing connector docs line-by-line
against the Java source code on this branch. It covers 58 documentation files
(29 English + 29 Chinese mirrors).
## What documentation issues were found
All issues below were verified directly against the source code before
fixing:
### Feature checkboxes that contradicted the code
1. **Easysearch source** (`docs/{en,zh}/connectors/source/Easysearch.md`):
`parallelism` was unchecked, but `EasysearchSource` implements
`SupportParallelism` and its split enumerator distributes splits round-robin
across readers → checked.
2. **Qdrant sink** (`docs/{en,zh}/connectors/sink/Qdrant.md`): `support
multiple table write` was unchecked, but `QdrantSink implements
SupportMultiTableSink` (and the doc itself documents
`multi_table_sink_replica`, which is in the factory OptionRule) → checked.
3. **Milvus sink** (`docs/{en,zh}/connectors/sink/Milvus.md`): `support
multiple table write` was checked, but `MilvusSink` implements only
`SeaTunnelSink, SupportSaveMode` (no `SupportMultiTableSink` anywhere in
connector-milvus) → unchecked.
4. **AmazonDynamoDB source**
(`docs/{en,zh}/connectors/source/AmazonDynamoDB.md`): `column projection` was
unchecked, but `AmazonDynamoDBSource implements SupportColumnProjection` →
checked.
5. **MongoDB source** (`docs/{en,zh}/connectors/source/MongoDB.md`):
`stream` was checked and the description claimed "supports both batch and
streaming jobs" / "in streaming mode … resumes from the last committed cursor",
but `MongodbSource.getBoundedness()` returns `Boundedness.BOUNDED`
unconditionally and `MongodbReader` signals no-more-element once splits are
exhausted → unchecked, and the prose now describes it as a batch (bounded)
source with checkpoint-based split resume.
6. **Greenplum / Redshift sinks**
(`docs/{en,zh}/connectors/sink/{Greenplum,Redshift}.md`): `timer flush` was
unchecked, but both connectors are the shared JDBC sink, whose
`JdbcSinkFactory` OptionRule includes `JdbcSinkOptions.BATCH_INTERVAL_MS`; all
sibling JDBC sink docs (Mysql, Oracle, PostgreSql, DB2, Vertica, …) mark this
checked → checked.
### Option table values that contradicted the code
7. **`multi_table_sink_replica` default** in AmazonDynamoDB / Http / GraphQL
sink docs (`docs/{en,zh}/connectors/sink/{AmazonDynamoDB,Http,GraphQL}.md`):
default shown as `-`, but `SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA`
declares `.defaultValue(1)` and the option is part of each factory's OptionRule
→ default `1` (consistent with Hudi/Iceberg/Prometheus/Redis/Qdrant docs).
8. **Elasticsearch `auth_type` type**
(`docs/{en,zh}/connectors/{source,sink}/Elasticsearch.md`): table row said
`string`, but `ElasticsearchBaseOptions` declares
`Options.key("auth_type").enumType(AuthTypeEnum.class)`; the docs' own section
headings say `[enum]`. The sink row description also omitted the supported
`api_key_encoded` value from `AuthTypeEnum` → type `enum`, description now
lists all three values.
### Documented functionality that does not exist in the code
9. **DataHub sink `${table}` placeholder**
(`docs/{en,zh}/connectors/sink/Datahub.md`): the doc claimed `topic` supports
`${table}` placeholders (and a deprecated `${table_name}` alias) for per-table
routing in multi-table jobs, with a matching example `topic = "${table}"`. No
placeholder substitution exists anywhere in connector-datahub (`DataHubSink`
passes `TOPIC` verbatim to `DataHubWriter`, which uses it literally) — a copied
example would look up a topic literally named `${table}` and fail. Removed the
placeholder claims and FAQ entry; the multi-table example now uses a literal
topic with an explanation that all input tables share the single configured
topic.
10. **Druid sink `${table_name}` placeholder**
(`docs/{en,zh}/connectors/sink/Druid.md`): same problem —
`DruidSink.createWriter` passes `DATASOURCE` verbatim to `DruidWriter`; no
substitution exists in connector-druid. Removed the placeholder claims and
fixed the multi-table example to use a literal datasource name.
11. **FtpFile sink "Writing via SFTP" section**
(`docs/{en,zh}/connectors/sink/FtpFile.md`): the section was fabricated — the
FTP sink always builds an `ftp://` URI from `host`/`port` (`FtpConf.SCHEMA =
"ftp"`), `fs.defaultFS` is not part of the factory OptionRule, `ftp_properties`
does not exist anywhere in the Java codebase, and `fs.sftp.knownHosts` /
`fs.sftp.port` are not real keys of `SFTPFileSystem` (the actual key is
`fs.sftp.host.port`). Replaced the section with a note that SFTP is served by
the `SftpFile` sink.
12. **HdfsFile `read_columns` JSON projection**
(`docs/{en,zh}/connectors/source/HdfsFile.md`): the option claimed column
projection is supported for `[text,json,csv,orc,parquet,excel,xml]`, but
`JsonReadStrategy` never reads `readColumns` (projection is implemented only in
Text/Csv/Orc/Parquet/Excel/Xml strategies) → removed `json` from the list and
noted that projection is not applied to JSON files.
### RocketMQ source
13. **False inheritance claim**
(`docs/{en,zh}/connectors/source/RocketMQ.md`): the doc said items in
`tables_configs` "inherit the top-level defaults", but `RocketMqSourceConfig`
parses each item standalone (`ReadonlyConfig.fromMap(tableConfig)`) —
`format`/`tags`/`schema`/`ignore_parse_errors` fall back to the option defaults
(`json`/`false`), not the job-level values; only `start.mode` inherits the
top-level value (`RocketMqSourceSplitEnumerator.getEffectiveStartMode`).
Reworded accordingly.
14. **FAQ omits two supported start modes**: the FAQ listed only
FIRST/LAST/TIMESTAMP, but `StartMode` (and the doc's own options table) also
includes `CONSUME_FROM_GROUP_OFFSETS` (the default) and
`CONSUME_FROM_SPECIFIC_OFFSETS` → FAQ now lists all five.
### JDBC source docs: type mappings, option types, defaults, examples
All verified against the per-dialect converters in connector-jdbc:
15. **`partition_lower_bound` / `partition_upper_bound` declared as
`BigDecimal` (or `Long`)** in Mysql, Oracle, PostgreSQL, SqlServer, OceanBase,
Redshift, Vertica, HiveJdbc source docs, but `JdbcSourceOptions` declares both
as `.stringType()` → `String`. Matching `[BigDecimal]` section headings fixed
too.
16. **`partition_num` default "job parallelism"** in Mysql, PostgreSQL,
SqlServer, OceanBase, Redshift, Vertica source docs, but `PARTITION_NUM` has no
declared default and `JdbcSourceTableConfig.DEFAULT_PARTITION_NUMBER = 10` is
substituted at runtime → default `10`.
17. **Timestamp-with-timezone types mapped to plain `TIMESTAMP`**, while the
converters return `LocalTimeType.OFFSET_DATE_TIME_TYPE` (`TIMESTAMP_TZ`): MySQL
`TIMESTAMP` (`MySqlTypeConverter`), Oracle `TIMESTAMP WITH (LOCAL) TIME ZONE`,
PostgreSQL `timestamptz`, SQL Server `DATETIMEOFFSET`, OceanBase MySQL-mode
`TIMESTAMP` and Oracle-mode LTZ, Redshift `TIMESTAMPTZ` → separate
`TIMESTAMP_TZ` rows.
18. **Oracle NUMBER rows** (`docs/{en,zh}/connectors/source/Oracle.md`):
`NUMBER(scale != 0) → DECIMAL(38,18)` and `NUMBER(18 < p, scale == 0) →
DECIMAL(38,0)` contradicted `OracleTypeConverter` (scale ≤ 18 keeps the
declared precision/scale, scale > 18 caps scale at 18; scale-0 with 19 ≤ p < 38
keeps the precision, p ≥ 38 becomes `DECIMAL(38,0)`).
19. **PostgreSQL**: `driver` description pointed to the MySQL driver class
in the PostgreSQL doc; `query` was marked required although
`JdbcSourceFactory.optionRule()` lists `QUERY` as optional; the `table_list`
example contained invalid SQL (`select * id, name from …`); the `_BYTEA →
ARRAY<TINYINT>` mapping has no `_bytea` case in `PostgresTypeConverter`
(removed the row).
20. **DB2 source**: `DECIMAL/DEC/NUMERIC/NUM → DECIMAL(38,18)` contradicted
`DB2TypeConverter` (returns `DecimalType(precision, scale)` as declared);
`query` was marked required.
21. **OceanBase source**: MySQL-mode row `TINYINT / TINYINT UNSIGNED →
SMALLINT` self-contradicted the previous `TINYINT → BYTE` row
(`OceanBaseMySqlTypeConverter` maps only `TINYINT UNSIGNED`/`SMALLINT` to
SMALLINT); Oracle-mode `Number(p), p > 18 → DECIMAL(38,18)` contradicted
`OracleTypeConverter`.
22. **Redshift source**: `DECIMAL(size+1, scale)` formula never applied by
`RedshiftTypeConverter` → `DECIMAL(precision, scale)`.
23. **Vertica source**: `BIT(n)` listed in the BYTES row although
`VerticaTypeMapper` maps all `BIT` to BOOLEAN (and the doc's own first row
already said so) → removed from the BYTES row.
24. **HiveJdbc**: the ">38 → DECIMAL(38,18)" capping does not exist in
`HiveTypeMapper` (any defined precision keeps `(precision, scale)`; `(38,18)`
only when precision is missing) → rows reworded.
25. **`query` marked required** in DB2, Greenplum, Vertica, HiveJdbc,
PostgreSQL source docs although `JdbcSourceFactory.optionRule()` only requires
`url`/`driver` → `No`, with the "required when neither `table_path` nor
`table_list` is configured" note.
### Misc
26. **DingTalk sink description**
(`docs/{en,zh}/connectors/sink/DingTalk.md`): claimed the request is signed
"for each row", but `DingTalkWriter` builds the signed client once (when
`client == null`) and reuses it; the description now matches the code (and the
same file's own options section).
27. **Assert sink `comment` rule type**
(`docs/{en,zh}/connectors/sink/Assert.md`): the Type column said `comment` (not
a type); the rule is read as a string via `AssertConfig.COLUMN_COMMENT` →
`string`.
## Which areas/files were updated (58 files: 29 EN + 29 ZH mirrors)
- `docs/{en,zh}/connectors/source/`: Easysearch, Elasticsearch,
AmazonDynamoDB, MongoDB, HdfsFile, RocketMQ, Mysql, Oracle, PostgreSQL,
SqlServer, DB2, OceanBase, Greenplum, Redshift, Vertica, HiveJdbc
- `docs/{en,zh}/connectors/sink/`: Elasticsearch, Qdrant, Milvus,
AmazonDynamoDB, Datahub, Druid, FtpFile, Http, GraphQL, Greenplum, Redshift,
DingTalk, Assert
## Were English and Chinese docs both checked?
Yes. Every issue above was fixed in both `docs/en` and `docs/zh` wherever
both files contain the same content; the two file trees were edited in lockstep
and re-checked together.
## How duplicate PRs from the last 7 days were checked
I listed all PRs (open and closed) from the last 7 days before starting via
`gh pr list --repo apache/seatunnel`, then extracted the full changed-file
lists of every docs PR in that window (my own #12813, #12771, #12758, #12730,
#12709 and merged #12833, plus #12859 Copy transform and #12784 CLI docs from
other authors — 159 doc files in total) and explicitly excluded all of them
from this batch. Every file touched here is absent from those lists; several
overlapping candidates (e.g. Redis source, Kafka, Paimon, copy transform) were
dropped from consideration for that reason. The
Qdrant/Milvus/ES/Datahub/Druid/FtpFile/HdfsFile and the RDBMS source files
above were also re-audited and found not to be covered by any open PR.
## How was this verified?
- Every changed table row was checked against the corresponding `Option` /
`OptionRule` definition, converter/mapper class, or factory identifier in the
Java source (file:line evidence per item above).
- Post-edit automated checks over all 58 changed files: every Markdown table
row opens/closes with `|` and pipe counts match each table's header row (the
only "mismatches" reported are pre-existing escaped pipes like
`string\|ConfigMap` in Assert.md, which are valid markdown), and all relative
`.md` links resolve to existing files.
- Verified that the remaining `${table_name}` occurrences in FtpFile path
examples are a real file-sink feature (`LocalFile.md`/`S3File.md` document the
same CatalogTable path injection implemented in the file base module), so they
were intentionally kept.
- Docs-only change: no Java/build code is affected, so the full Maven build
was not run; the checks above are the appropriate verification for this change.
(`ConnectorOptionCheckTest` in seatunnel-ci-tools only validates that connector
classes have matching `*Options` Java classes and is unrelated to doc content.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
