ilyasahsan123 commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-6105370007
Hi @tkaymak
We should enable the `pos` and `row` attributes within the `metadata` to
support data duplication (MySQL) in the data warehouse.
Additionally, We also need source.ts_ms to identify the database commit
time, which is useful for debugging.
Currently, Apache Beam supports only these following attributes, and we
could not deduplicate the data correctly.
Apache Beam:
```
metadata={
"connector": "mysql",
"version": "3.5.2.Final",
"name": "beam-debezium-connector",
"database": "inventory",
"schema": "mysql-bin.000003",
"table": "products_on_hand"
}
```
However, here is the complete metadata generated by Debezium:
Debezium
```
"source": {
"version": "3.5.2.Final",
"connector": "mysql",
"name": "tutorial",
"ts_ms": 1791689147000,
"snapshot": "false",
"db": "inventory",
"sequence": null,
"ts_us": 1791689147000000,
"ts_ns": 1791689147000000000,
"table": "customers",
"server_id": 223344,
"gtid": null,
"file": "mysql-bin.000003",
"pos": 796,
"row": 0,
"thread": 13,
"query": null
}
```
Is there any explanation about this?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
