rdblue opened a new pull request #28027: [SPARK-31255][SQL] Add 
SupportsMetadataColumns to DSv2 (WIP)
URL: https://github.com/apache/spark/pull/28027
 
 
   ### What changes were proposed in this pull request?
   
   This adds support for metadata columns to DataSourceV2. If a source 
implements `SupportsMetadataColumns` it must also implement 
`SupportsPushDownRequiredColumns` to support projecting those columns.
   
   The analyzer is updated to resolve metadata columns from 
`LogicalPlan.metadataOutput`, and this adds a rule that will add metadata 
columns to the output of `DataSourceV2Relation` if one is used.
   
   ### Why are the changes needed?
   
   This is the solution discussed for exposing additional data in the Kafka 
source. It is also needed for a generic `MERGE INTO` plan.
   
   ### Does this PR introduce any user-facing change?
   
   Yes. Users can project additional columns from sources that implement the 
new API. This also updates `DescribeTableExec` to show metadata columns.
   
   ### How was this patch tested?
   
   Will include new unit tests.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to