fbocse commented on a change in pull request #28027: [SPARK-31255][SQL] Add
SupportsMetadataColumns to DSv2 (WIP)
URL: https://github.com/apache/spark/pull/28027#discussion_r398901106
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
##########
@@ -48,6 +48,15 @@ case class DataSourceV2Relation(
import DataSourceV2Implicits._
+ override lazy val metadataOutput: Seq[AttributeReference] = table match {
+ case hasMeta: SupportsMetadataColumns =>
+ val attrs = hasMeta.metadataColumns
+ val outputNames = outputSet.map(_.name).toSet
+ attrs.filterNot(col => outputNames.contains(col.name)).toAttributes
Review comment:
Can this possibly be buggy because of case-sensitivity playing a trick when
filtering by Set.contains or column names are "normalized" wrt to case?
```suggestion
val outputNames = outputSet.map(_.name.toLowerCase()).toSet
attrs.filterNot(col =>
outputNames.contains(col.name.toLowerCase())).toAttributes
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
