viirya commented on code in PR #36332:
URL: https://github.com/apache/spark/pull/36332#discussion_r857281246
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVScan.scala:
##########
@@ -70,8 +70,8 @@ case class CSVScan(
ExprUtils.verifyColumnNameOfCorruptRecord(dataSchema,
parsedOptions.columnNameOfCorruptRecord)
// Don't push any filter which refers to the "virtual" column which cannot
present in the input.
// Such filters will be applied later on the upper layer.
- val actualFilters =
-
pushedFilters.filterNot(_.references.contains(parsedOptions.columnNameOfCorruptRecord))
+ val actualFilters = pushedFilters.map(_.toV1)
+
.filterNot(_.references.contains(parsedOptions.columnNameOfCorruptRecord))
Review Comment:
Hmm, why we cannot work on v2 here and other places directly? I feel it is
verbose and redundant to see `toV1`, `toV2` there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
