tchivs commented on PR #4202: URL: https://github.com/apache/flink-cdc/pull/4202#issuecomment-3696253123
> Thanks for @tchivs' contribution. > > I wonder if we need individual metadata columns for database, schema, and table, since they're always available in [Transform expressions](https://github.com/apache/flink-cdc/blob/master/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/metadata/MetadataColumns.java) (only after [FLINK-38840](https://issues.apache.org/jira/browse/FLINK-38840) got closed). Thanks for the review @yuxiqian! You raise an important point about the overlap with Transform metadata fields. You're right that __namespace_name__, __schema_name__, and __table_name__ are already available in Transform expressions. Let me clarify the design rationale: 1. op_ts is essential and non-redundant: - There's no equivalent in Transform metadata fields (MetadataColumns.java only defines __namespace_name__, __schema_name__, __table_name__, __data_event_type__) - op_ts can only be obtained via metadata.list from the source connector - This is consistent with MySQL connector's implementation 2. For database_name, schema_name, table_name - there is overlap: I see two perspectives here: Argument for keeping them: - Sink persistence: Users can pass these to downstream sinks without writing transform rules - Consistency: MySQL connector has op_ts via metadata.list, so having all metadata follow the same pattern is intuitive - Simplicity: Direct configuration is simpler than transform expressions for basic use cases Argument for removing them: - Redundancy: Transform already provides __namespace_name__, __schema_name__, __table_name__ - Maintenance: Less code to maintain if we rely on Transform metadata My suggestion: - Keep op_ts (essential, no alternative) - For database_name/schema_name/table_name, I'm open to either approach: - Option A: Keep them for consistency and ease of use - Option B: Remove them and document the Transform approach in the docs What's your preference? I'm happy to adjust the PR based on the team's direction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
