ulysses-you commented on code in PR #39870:
URL: https://github.com/apache/spark/pull/39870#discussion_r1098098734
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -54,15 +54,18 @@ case class DataSourceV2Relation(
override lazy val metadataOutput: Seq[AttributeReference] = table match {
case hasMeta: SupportsMetadataColumns =>
- val resolve = conf.resolver
- val outputNames = outputSet.map(_.name)
- def isOutputColumn(col: MetadataColumn): Boolean = {
- outputNames.exists(name => resolve(col.name, name))
+ getTagValue(CACHED_META_OUTPUT_TAG).getOrElse {
Review Comment:
It's acutally the different issue, my test cases still fail with your pr.
After `addMetadataCol`, the newNode is always different with original
node(metadata output merges into output), so an extra project can not be avoid.
(Your pr fixs a special case of `hiddenOutputTag` with NaturalJoin)
Then, the only way to resolve rest metadata columns is using metadata
output. But before this pr, the metadata output is lost once call
`addMetadataCol`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
