goutamadwant opened a new issue, #12560: URL: https://github.com/apache/seatunnel/issues/12560
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened pgoutput does not stream generated columns (none before PostgreSQL 18; on 18 only STORED ones when the publication has `publish_generated_columns = stored`, VIRTUAL never). Postgres-CDC still keeps them in the table schema, so the snapshot writes the real value and every streamed change writes NULL. With an upsert sink, an UPDATE overwrites the correct value with NULL. With `schema-changes.enabled = true`, the first streamed change on the table fails the job with `SE-07 ... supports only ADD COLUMN`, because the RELATION message has fewer columns than the cached schema. Columns left out by a publication column list behave the same. **Steps** 1. `CREATE TABLE public.g (id INT PRIMARY KEY, price NUMERIC(10,2), px2 NUMERIC(10,2) GENERATED ALWAYS AS (price * 2) STORED); ALTER TABLE public.g REPLICA IDENTITY FULL; INSERT INTO public.g (id, price) VALUES (1, 1), (2, 2);` 2. Start the job below and wait for the snapshot (sink `px2` for id 1 = `2.00`). 3. `UPDATE public.g SET price = 7 WHERE id = 1; INSERT INTO public.g (id, price) VALUES (100, 5);` **Before / After** (dev 4c874e2a4, Zeta, pgjdbc 42.4.3; sink `px2` after step 3) | Case | dev | with fix | |---|---|---| | PG 14.2 / 17.9, schema-changes off | id 1 overwritten with NULL, id 100 NULL | `px2` not in CDC schema, other columns correct | | PG 14.2 / 17.9, schema-changes on | job FAILED (SE-07) | job running, data correct | | PG 18.6 STORED or VIRTUAL, off / on | NULL / job FAILED | excluded / job running, data correct | | PG 18.6 with `publish_generated_columns = stored` | `14.00`, `10.00` | `14.00`, `10.00` (column kept) | ### SeaTunnel Version dev (4c874e2a4), 3.0.0 ### SeaTunnel Config ```conf env { parallelism = 1, job.mode = "STREAMING", checkpoint.interval = 5000 } source { Postgres-CDC { url = "jdbc:postgresql://localhost:5432/db", username = "postgres", password = "***", database-names = ["db"], schema-names = ["public"], table-names = ["db.public.g"], slot.name = "st", schema-changes.enabled = true } } sink { Jdbc { url = "jdbc:postgresql://localhost:5432/sink", driver = "org.postgresql.Driver", username = "postgres", password = "***", generate_sink_sql = true, database = "sink", table = "public.${table_name}", schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST", data_save_mode = "APPEND_DATA" } } ``` ### Running Command ```shell bin/seatunnel.sh --config pg_cdc.conf ``` ### Error Exception ```log SchemaValidationException: ErrorCode:[SE-07], ErrorDescription:[Schema change type is not supported] - PostgreSQL CDC currently supports only ADD COLUMN relation changes. ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version Java 8, Java 11 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
