Doris-Breakwater commented on issue #67485: URL: https://github.com/apache/doris/issues/67485#issuecomment-5523487227
Breakwater-GitHub-Analysis-Slot: slot_bc7e689d1154 ## Initial assessment **Triage:** Confirmed code-level defect in the Doris 4.1.3 PostgreSQL CDC Streaming Job auto-table-creation path. The issue currently has no labels, assignee, or linked development work. Confidence is high from the 4.1.3 source path, although an end-to-end PostgreSQL 16/PostGIS reproduction was not run because the report does not include the actual exception or FE stack trace. ### Verified facts 1. In 4.1.3, [`JdbcPostgreSQLClient.jdbcTypeToDoris`](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java#L118-L199) has no `geometry` mapping. Its default branch returns `Type.UNSUPPORTED`. 2. During `CREATE JOB`, [`generateCreateTableCmds`](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java#L404-L423) calls `getColumns(...)` before parsing and applying `table.<table>.exclude_columns`. 3. [`getColumns(...)`](https://github.com/apache/doris/blob/4.1.3/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java#L475-L506) rejects every `UNSUPPORTED` column immediately. Therefore `table.building.exclude_columns = "geometry"` cannot be reached in time to suppress the failure. The code implies the error should contain `Unsupported column type, table:[building], column:[geometry]`; the reporter's exact error is still needed to confirm that this is the observed exception. 4. The post-creation CDC paths already use the desired ordering: snapshot/incremental row serialization checks the exclusion before value conversion ([code](https://github.com/apache/doris/blob/4.1.3/fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/deserialize/DebeziumJsonDeserializer.java#L154-L174)), and PostgreSQL schema-change handling skips an excluded added column before Doris type generation ([code](https://github.com/apache/doris/blob/4.1.3/fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/deserialize/PostgresDebeziumJsonDeserializer.java#L204-L238)). Static tracing therefore localizes this report to initialization/target-schema generation; runtime behavior should still be covered by an integration test. 5. Current master contains pre-created target-table support from [#66950](https://github.com/apache/doris/pull/66950), which bypasses source-to-Doris type conversion when the target table already exists. That change postdates 4.1.3 and is not present in the inspected `branch-4.1` ref, so it is not a verified workaround for the reported release. The non-pre-created auto-creation path on master still validates unsupported types before filtering exclusions. ### Recommended maintainer action Prefer a targeted ordering fix rather than mapping every unknown PostgreSQL type to `STRING`: - Parse `exclude_columns` before `getColumns(...)` performs unsupported-type validation. - Fetch the source column metadata, validate excluded names against the complete name set, and continue rejecting excluded primary-key columns. - Remove the explicitly excluded columns, then validate/map only the remaining columns to Doris types. - Keep the existing failure for an unsupported column that is not excluded. Globally defaulting all unrecognized extension types to `STRING` would change JDBC behavior beyond CDC and could silently misrepresent types whose wire/runtime representation is not safely string-compatible. Suggested regression coverage: - PostgreSQL/PostGIS table with `id` as the primary key and `geometry geometry(GEOMETRYCOLLECTION, 4326)` excluded: `CREATE JOB` succeeds and the Doris table omits `geometry`. - The same source table without the exclusion still fails with a clear unsupported-column error. - Existing validation for nonexistent and primary-key exclusions remains intact. - Initial snapshot plus incremental INSERT/UPDATE/DELETE succeed while the PostGIS column is present. - DROP/re-ADD of the excluded PostGIS column does not emit Doris DDL and does not stop the job. A 4.1 backport should be considered after the fix is verified. ### Information still needed from the reporter - The complete SQL client error returned by `CREATE JOB` and the corresponding FE log stack trace. - The full Doris build string (`SELECT VERSION()`), exact PostgreSQL JDBC driver version/artifact, PostgreSQL server version, and PostGIS version (`SELECT postgis_full_version()`). - Confirmation of the actual source database: the job example uses `test`, while the exported DDL header names `tp_szt`. - A minimal inserted row containing a non-null `GEOMETRYCOLLECTION` and whether the failure occurs synchronously during `CREATE JOB` or only after the job starts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
