zhangshenghang opened a new pull request, #12709:
URL: https://github.com/apache/seatunnel/pull/12709
## Purpose of this PR
This PR fixes a batch of real documentation inaccuracies found by comparing
docs against the actual source code on `dev`. It covers connector docs (Kafka,
Hive, Kudu, FtpFile, HdfsFile, FakeSource, Paimon, Iceberg, StarRocks, Jdbc,
Doris, MongoDB), transform docs (LLM, Embedding, Python, Calcite) and the docs
sidebar, in both English and Chinese.
## What was found and changed
Every fix below was verified against the corresponding Java source before
editing.
**Kafka (source + sink, en + zh)**
- Removed the false claim "By default we use 2pc to guarantee exactly once":
`KafkaSinkOptions.SEMANTICS` defaults to `KafkaSemantics.NON`, which provides
no guarantees. The note now explains how to actually get exactly-once
(`semantics = EXACTLY_ONCE` + checkpoint).
- Rewrote the SASL/Kerberos FAQ for source and sink: the previous examples
put `kafka.security.protocol`, `kafka.sasl.*` as top-level keys. The connector
only reads the `kafka.config` map (`KafkaSinkWriter.getKafkaProperties` /
`KafkaSourceConfig.createKafkaProperties`), so those flat `kafka.*` keys are
silently ignored and authentication would never take effect. Examples now use
`kafka.config = { ... }`.
- Sink FAQ format list now includes `maxwell_json` and
`compatible_debezium_json` (present in `MessageFormat` / the options table);
source `format` row now includes `compatible_kafka_connect_json`.
- Documented that `partition` and `partition_key_fields` are mutually
exclusive (`KafkaSinkSerializer` throws a validation error when both are set).
- zh sink: aligned the outdated "configure kafka_headers_fields carefully"
note with the code — NATIVE + `kafka_headers_fields` throws
`OPERATION_NOT_SUPPORTED` at serializer init, so the job fails to start.
- Fixed `kafka.max.poll.records` → `max.poll.records` inside `kafka.config`,
and the `[formats](../formats)` directory link → `../formats/overview.md`.
**Hive (source + sink, en + zh)**
- Source: removed claims of CSV/JSON/markdown support.
`HiveTableUtils.parseFileFormat` only supports text/parquet/orc and throws
"Hive connector only support [text parquet orc] table now"; the whole
fabricated markdown "document-element" schema section was removed.
- Source: fixed inverted deprecation guidance. Code prefers `table_list` and
marks the `tables_configs` path `@Deprecated` ("hive is structured, should use
table_list"), while the doc told users to migrate to `tables_configs`. Options
table, sections and the multi-table example now point to `table_list`.
- Sink: removed CSV/JSON from the write-format lists (write path only
resolves ORC/TEXT/PARQUET, format comes from the target table `STORED AS`),
rewrote the format and partition FAQs (the documented
`file_format_type`/`partition_by` user config is derived/overridden from table
metadata, not user-set), and removed a stray copy-pasted line under `krb5_path`.
- Source: `compress_codec` no longer lists json/csv rows.
**Kudu (source + sink, en + zh)**
- Fixed wrong type mappings: source INT8/INT16/INT32 were collapsed to
`INT`; now INT8→TINYINT, INT16→SMALLINT, INT32→INT per `KuduTypeMapper`. Sink
table now maps TINYINT→INT8, SMALLINT→INT16, INT→INT32, BIGINT→INT64.
**FtpFile (source, en + zh)**
- Removed the "Reading via SFTP" example that used a non-existent
`ftp_properties` option and `fs.defaultFS` (ignored by `FtpConf`, which builds
`ftp://host:port` from `host`/`port`). The connector cannot read SFTP at all;
the section now explains this and points to the dedicated SftpFile connector.
Added parquet/orc to the format list.
**HdfsFile (sink, en + zh)**
- `canal_json`/`debezium_json`/`maxwell_json` were listed under "compress
codec" in the English doc; they are `FileFormat` entries, now listed under file
format types (matching the zh doc). zh list also gained the missing XML/binary
entries.
**FakeSource (en + zh)**
- Removed the `table-names` example: FakeSource never reads `table-names`
(`MultipleTableFakeSourceConfig` only handles `tables_configs`/`schema`), so
the example silently produced one table; a correct `tables_configs` example
already exists in the same doc.
- Fixed `double.fake.mode` description that said "generating float data"
(copy-paste).
**Paimon (source, en + zh)**
- FAQ used a non-existent `paimon.catalog.type` option; the real key is
`catalog_type` (`PaimonBaseOptions`).
**Iceberg (source, zh)**
- Three Kerberos rows had English `no` in the required column of an
otherwise-Chinese table; unified to 否.
**StarRocks (sink, en + zh)**
- `batch_max_rows` is `intType` and `batch_max_bytes` is `longType` in
`StarRocksSinkOptions`; the doc had the types swapped. Also documented
`multi_table_sink_replica` (default 1).
**Jdbc (source + sink, en + zh)**
- Documented the `schema` option (`JdbcCommonOptions.SCHEMA`, "for databases
that support the schema parameter, give it priority") — the en sink example
already used `schema = "dbo"` without the option being documented.
- Documented `multi_table_sink_replica` (default 1).
- zh sink: added the SqlServer CDC multi-table example that only existed in
en, and fixed the mistranslated heading "示例可选" → "方言列表" (en: "dialect list").
**Doris (source + sink, en + zh)**
- Sink: documented `default-database` (default `information_schema`) and
`multi_table_sink_replica`.
- Source: added the top-level `database`/`table` rows to the base options
table (the note and all examples require them, but they were only documented
inside the `table_list` sub-table).
**MongoDB (sink, en + zh)**
- Documented `multi_table_sink_replica` (default 1).
**Transforms**
- `docs/sidebars.js`: registered `transforms/encrypt` — the page existed in
en+zh but was unreachable from navigation.
- llm (en + zh): added `QIANFAN` to `model_provider` options
(`ModelProvider.QIANFAN`), fixed `api_key` required=yes contradiction (it is
conditional per `LLMTransformFactory.optionRule`; not used by CUSTOM),
documented `secret_key`, `oauth_path` and `process_batch_size` (default 100,
legacy `inference_batch_size` fallback).
- embedding (en + zh): documented `process_batch_size` and added `ZHIPU` to
the provider list.
- python (en + zh): noted that `ROUTE_TO_TABLE` passes config validation but
is rejected at runtime (only FAIL/SKIP supported).
- calcite (en + zh): added the missing built-in `GET_JSON_OBJECT` UDF
(`GetJsonObjectFunction`), since the text claims only Calcite standard
functions exist beyond the listed UDFs.
## Areas updated
- `docs/en|zh/connectors/source|sink/`: Kafka, Hive, Kudu, FtpFile,
HdfsFile, FakeSource, Paimon, Iceberg, StarRocks, Jdbc, Doris, MongoDB
- `docs/en|zh/transforms/`: llm, embedding, python, calcite
- `docs/sidebars.js`
42 files changed (275 insertions, 215 deletions).
## Were both English and Chinese docs checked?
Yes. Every fix above was applied to both `docs/en` and `docs/zh` wherever
both versions exist. The en/zh parity issues found (Hive format lists, zh Kafka
NATIVE note, zh Jdbc missing SqlServer CDC example, zh Iceberg table language
mix, zh HdfsFile missing XML/binary) were aligned in this PR.
## Duplicate check (last 7 days)
I listed all PRs created against apache/seatunnel in the last week before
submitting:
- #12693 (zhangshenghang, Oct 8) covers getting-started / introduction
concepts / a few transforms (sql-udf, rowkind-extractor, replace) — no file
overlap with this PR.
- #12679 (zhangshenghang, Oct 7) covers a different connector set (Airtable,
Assert, Datahub, DuckDB, Elasticsearch, GraphQL, Greenplum, Vertica, DB2,
Mysql, OceanBase, OneSignal, PostgreSQL, Pulsar, Redshift, RocketMQ, SqlServer,
InfluxDB, ObsFile) — no file overlap.
- #12641 (zhangshenghang, Oct 5) covers `docs/**/engines/**` — no overlap.
- #12675 (merged) fixed the Mqtt `field_delimiter` table only — untouched
here.
This PR deliberately avoids every file touched by those PRs.
## Verification
- Every documentation claim changed here was checked against the
corresponding source file (Options classes, factories, type mappers,
serializers) — file/line evidence listed above.
- `node --check docs/sidebars.js` passes.
- A script validated all 245 relative markdown links across the 42 changed
files resolve to existing files; markdown table column consistency checked for
all changed tables.
- The CI dead-link job (`markdown-link-check`) requires network access and
is `continue-on-error` upstream; it was not run locally. No Java build is
needed for docs-only changes, so `spotless:apply` / `mvnw verify` were not run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
