sabbasani commented on code in PR #12726:
URL: https://github.com/apache/gluten/pull/12726#discussion_r4167514807
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PushDownInputFileExpression.scala:
##########
@@ -104,15 +103,44 @@ object PushDownInputFileExpression {
ProjectExec(f.output, FilterExec(newCondition, newChild))
}
+ /**
+ * Returns true when any of the injected metadata attribute names (all
lowercase) matches a
+ * column already present in the scan output at the Velox/case-insensitive
level.
+ *
+ * Velox is case-insensitive. If the scan has a user data column named
e.g. "Input_File_Name"
+ * its lowercase form "input_file_name" collides with the metadata
function of the same name.
+ * Velox rejects a TableScan that maps the same lowercase column name to
both a Regular handle
+ * (data column) and a PartitionKey/metadata handle (file-path metadata).
When this conflict is
+ * detected the scan must fall back to Vanilla so that Spark's own
FilePartitionReader sets the
+ * InputFileBlockHolder thread-local and input_file_name() returns the
correct file path.
+ */
+ private def hasVeloxColumnNameConflict(
+ scanOutput: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ replacedExprs: mutable.Map[String, Alias]): Boolean = {
+ val scanOutputLowerNames =
scanOutput.map(_.name.toLowerCase(java.util.Locale.ROOT)).toSet
Review Comment:
@infvg
Thanks for checking this. I’ve pushed the changes along with the regression
tests for the case-sensitive handling.
Could you please pull the latest changes and try the reproduction locally
once? Please let me know if you still see the issue with
`spark.sql.caseSensitive=false`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
