zhengxiang378928908-code opened a new issue, #12754: URL: https://github.com/apache/seatunnel/issues/12754
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened Several Doris column types fail when the Doris catalog loads a table, so a job reading such a table stops before any data is read. Most of them were added in Doris 2.1+/4.0 and are not handled by `AbstractDorisTypeConverter#convert`. I created one table per type on **Doris 4.0.6** and loaded each one with `DorisCatalog#getTable` on current `dev`: | Doris type | `information_schema.columns.DATA_TYPE` | Result on `dev` | |---|---|---| | `IPV4` (2.1+) | `ipv4` | **COMMON-20** | | `IPV6` (2.1+) | `ipv6` | **COMMON-20** | | `STRUCT<a:INT, b:VARCHAR(10)>` | `struct` | **COMMON-20** | | `TIMESTAMPTZ` (4.0) | `timestamp` | **COMMON-20** (reported as `TIMESTAMP`) | | `BITMAP` / `HLL` / `QUANTILE_STATE` | `bitmap` / `hll` / `unknown` | COMMON-20 | | `JSON`, `VARIANT`, `DATETIME(6)`, `DECIMALV3(38,10)`, `ARRAY<INT>`, `MAP<STRING,INT>`, `LARGEINT` | — | OK | Notes: - `IPV4` and `IPV6` are common in log and network data, and could map to `STRING`. - `STRUCT` is the only complex type left: `ARRAY` and `MAP` are supported. It could map to SeaTunnel's `SeaTunnelRowType`. - `TIMESTAMPTZ` (Doris 4.0) is reported by `information_schema` as plain `timestamp`, so the converter sees `TIMESTAMP`, which it doesn't recognise. It probably belongs with `TIMESTAMP_TZ`, but Doris's time-zone semantics for this type need checking first. - `BITMAP`, `HLL` and `QUANTILE_STATE` are aggregate states that cannot be read as plain values. Failing on them is arguably expected, though a clearer error, or the option to skip those columns, would help users. I'd be happy to start with `IPV4`/`IPV6`, which have a straightforward mapping and are the most commonly used, and then follow up with `STRUCT` and `TIMESTAMPTZ`, if that split works for the maintainers. ### SeaTunnel Version dev (b0f6feddf) ### SeaTunnel Config ```sql -- Doris 4.0.6 CREATE TABLE t_ipv4 (id INT, v IPV4) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES('replication_num'='1'); CREATE TABLE t_struct (id INT, v STRUCT<a:INT, b:VARCHAR(10)>) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES('replication_num'='1'); CREATE TABLE t_tstz (id INT, v TIMESTAMPTZ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES('replication_num'='1'); ``` ```conf source { Doris { fenodes = "doris-fe:8030" query-port = 9030 username = "root" password = "" database = "seatunnel_probe_tmp" table = "t_ipv4" } } ``` ### Running Command ```shell # DorisCatalog#getTable(TablePath.of("seatunnel_probe_tmp", "<table>")) against Doris 4.0.6 ./bin/seatunnel.sh --config doris_source.conf -m local ``` ### Error Exception ```log ErrorCode:[COMMON-20], ErrorDescription:['doris' table 'seatunnel_probe_tmp.t1' unsupported get catalog table with field data types '{"v":"IPV4"}'] ErrorCode:[COMMON-20], ErrorDescription:['doris' table 'seatunnel_probe_tmp.t2' unsupported get catalog table with field data types '{"v":"IPV6"}'] ErrorCode:[COMMON-20], ErrorDescription:['doris' table 'seatunnel_probe_tmp.t3' unsupported get catalog table with field data types '{"v":"STRUCT"}'] ErrorCode:[COMMON-20], ErrorDescription:['doris' table 'seatunnel_probe_tmp.t14' unsupported get catalog table with field data types '{"v":"TIMESTAMP"}'] ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version Java 8 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
