hemanthboyina opened a new pull request, #59137: URL: https://github.com/apache/spark/pull/59137
**What changes were proposed in this pull request?** This PR propagates trusted DataSource V2 primary keys as Catalyst distinct keys. It introduces a way for logical plan nodes to provide known distinct keys and makes DataSourceV2Relation expose trusted top-level primary keys. Safe DataSourceV2ScanRelation instances preserve these keys when the complete key remains in the output. A primary key is trusted when it is either validated or unvalidated with RELY. Invalid and unvalidated NORELY constraints are ignored. The propagation is conservative for malformed metadata, nested key references, incomplete composite keys, nested schema pruning, sampling with replacement, and specialized scan paths that may not preserve row uniqueness. **Why are the changes needed?** Data Source V2 tables can expose primary keys through Table.constraints(), but this metadata is not currently connected to Catalyst's distinctKeys. As a result, existing optimizer rules cannot use trusted primary keys to remove redundant operations. For example, Spark retains the grouping-only aggregate for: ```sql SELECT DISTINCT id FROM t; ``` even when `id` is a trusted primary key. This PR connects DSv2 primary-key metadata to the existing distinct-key propagation framework so that existing optimizer rules can remove such redundant operations safely. **Does this PR introduce any user facing change?** Yes. Spark can produce a more efficient optimized plan by removing redundant duplicate-elimination operations when a Data Source V2 table reports a trusted primary key. Query results and public APIs are unchanged. **How was this patch tested?** Ran all test suite. **Was this patch authored or co-authored using generative AI tooling?** Yes, used gpt 5.6 sol -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
