anuragmantri commented on code in PR #55518:
URL: https://github.com/apache/spark/pull/55518#discussion_r3198406083
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/RowLevelOperationRuntimeGroupFiltering.scala:
##########
@@ -139,17 +138,13 @@ class
RowLevelOperationRuntimeGroupFiltering(optimizeSubqueries: Rule[LogicalPla
tableAttrs: Seq[Attribute],
scanAttrs: Seq[Attribute]): AttributeMap[Attribute] = {
- val attrMapping = tableAttrs.map { tableAttr =>
+ // The scan may be narrowed to exclude columns not needed by the connector.
+ // Attributes absent from the scan are skipped here; the caller must ensure
+ // that any attribute referenced in the condition is present in the scan.
+ val attrMapping = tableAttrs.flatMap { tableAttr =>
scanAttrs
.find(scanAttr => conf.resolver(scanAttr.name, tableAttr.name))
.map(scanAttr => tableAttr -> scanAttr)
- .getOrElse {
- throw new AnalysisException(
- errorClass = "_LEGACY_ERROR_TEMP_3075",
- messageParameters = Map(
- "tableAttr" -> tableAttr.toString,
- "scanAttrs" -> scanAttrs.mkString(",")))
- }
}
Review Comment:
I believe this is safe because condition-referenced columns are guaranteed
to be in the scan. Please correct me if I'm wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
