sabbasani commented on code in PR #12726:
URL: https://github.com/apache/gluten/pull/12726#discussion_r4141812500
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala:
##########
@@ -123,7 +123,9 @@ trait BasicScanExecTransformer extends LeafTransformSupport
with BaseDataSource
InputFileBlockLength().prettyName)
val neededInputFileRelatedMetadataKeys =
- inputFileRelatedMetadataKeys.filter(k => output.exists(_.name == k))
+ inputFileRelatedMetadataKeys.filter {
+ k => output.exists(a => ConverterUtils.normalizeColName(a.name) == k)
+ }
Review Comment:
Thanks for raising this. We investigated
ConverterUtils.normalizeColName(a.name) == k. Since
inputFileRelatedMetadataKeys are all-lowercase canonical names
(input_file_name, etc.), normalizeColName preserves casing under
spark.sql.caseSensitive=true (so Input_File_Name does not match) and lowercases
under false (so case variants match). This follows Spark's case-sensitivity
semantics and is covered by regression tests in FallbackSuite and IcebergSuite.
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala:
##########
@@ -123,7 +123,9 @@ trait BasicScanExecTransformer extends LeafTransformSupport
with BaseDataSource
InputFileBlockLength().prettyName)
val neededInputFileRelatedMetadataKeys =
- inputFileRelatedMetadataKeys.filter(k => output.exists(_.name == k))
+ inputFileRelatedMetadataKeys.filter {
+ k => output.exists(a => ConverterUtils.normalizeColName(a.name) == k)
+ }
Review Comment:
I investigated ConverterUtils.normalizeColName(a.name) == k. Since
inputFileRelatedMetadataKeys are all-lowercase canonical names
(input_file_name, etc.), normalizeColName preserves casing under
spark.sql.caseSensitive=true (so Input_File_Name does not match) and lowercases
under false (so case variants match). This follows Spark's case-sensitivity
semantics and is covered by regression tests in FallbackSuite and IcebergSuite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
