tkaymak commented on PR #40702: URL: https://github.com/apache/beam/pull/40702#issuecomment-6335433917
Thanks @Abacn and @ilyasahsan123, great feedback! I reworked the PR: * The extra metadata is now opt-in via `DebeziumIO.readAsJson().withFullMetadata(true)` and off by default, so the default output is unchanged and the existing tests in all SDKs stay as they are. The full mode is covered by unit tests and one new MySQL IT case. * Instead of a nested `metadata.source` that repeats `connector`, `version`, `name`, `db` and `table`, the additional source fields are now added directly to `metadata`, and only the ones that aren't there yet (e.g. MySQL `file`, `pos`, `row`, `gtid`, `ts_ms`; PostgreSQL `lsn`, `txId`, `ts_ms`). For MySQL I kept `file` explicitly, since the binlog file currently shows up under `schema`. * @ilyasahsan123 `op` is included as `metadata.op` behind the same option. * The second commit exposes the option in Python (`full_metadata=True`) and Go (`debeziumio.FullMetadata()`), and touches the PostCommit trigger files so the xlang tests run again. @Abacn regarding the "table" field: `metadata.table` is already part of the default output. Did you mean something else, maybe `op`? Happy to adjust. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
