jsingh-yelp opened a new pull request, #8473: URL: https://github.com/apache/paimon/pull/8473
### Purpose Flink catalog names used in `tableIdentifier.asSummaryString()` are arbitrary per-job the same physical table can appear under different lineage dataset names depending on how the job registers its catalog. This becomes especially problematic in organizations with multiple teams, where different teams can end up using different catalog names for the same Paimon table, resulting in fragmented lineage graphs. This PR adds `lineage-use-catalog-key-as-identifier`, a Flink catalog option that allows users to opt-in to using `catalog-key` as the catalog identifier in lineage dataset names. `catalog-key` is a more stable cross-job identifier that users explicitly configure. This is particularly valuable when the same database and table exist across different JDBC catalogs the `catalog-key` disambiguates them consistently across all jobs regardless of how each job names its catalog. **Note**: This PR change depends on a Flink change: https://issues.apache.org/jira/browse/FLINK-39935 (PR: https://github.com/apache/flink/pull/28439) with out this Flink will continue using catalog name. ### Tests - Added unit tests for this - Manually tested this scenario e2e and it works as expected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
