junmuz commented on PR #9822: URL: https://github.com/apache/paimon/pull/9822#issuecomment-5780164392
> Given the significant nature of the changes, I need specific use cases to justify their validity; please provide a detailed explanation of how this would be implemented in a real-world production environment. @JingsongLi Let me explain this in detail below. Yeah I agree the changes are more complex that's why I initially went with [minimal change approach](https://github.com/apache/paimon/pull/9691). ### Use case and motivation Paimon’s lookup changelog producer reconstructs UPDATE_BEFORE and DELETE records by looking up the previous table state. As a result, regular columns in those records describe the old row, not necessarily the event that caused the change. For example: UPDATE_BEFORE: event_ts = 50, __internal__event_ts = 100 UPDATE_AFTER: event_ts = 100, __internal__event_ts = 100 Downstream CDC consumers often need the event timestamp or source sequence to apply changes deterministically. #### Concurrent Sinks to a Single Cassandra table Multiple sources can fan into the same Cassandra table while being processed by independent jobs. A common example is migrating from Kafka to Paimon: Kafka retains the original event payload for retract events, whereas Paimon’s lookup-generated changelog may represent UPDATE_BEFORE and DELETE records using the previous row image. This makes it difficult to replace a Kafka-based job with a Paimon-based job without changing downstream conflict-resolution behavior. Exposing selected fields as changelog metadata preserves the event-level values and gives the sink a consistent, globally comparable event version for conditional writes and tombstones, preventing stale updates from overwriting newer changes or resurrecting deleted records. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
