Zouxxyy commented on code in PR #11153:
URL:
https://github.com/apache/incubator-gluten/pull/11153#discussion_r2567878049
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BatchScanExecTransformer.scala:
##########
@@ -102,30 +109,13 @@ abstract class BatchScanExecTransformerBase(
postDriverMetrics()
}
- // Similar to the problem encountered in
https://github.com/oap-project/gluten/pull/3184,
- // we cannot add member variables to BatchScanExecTransformerBase, which
inherits from case
- // class. Otherwise, we will encounter an issue where makeCopy cannot find a
constructor
- // with the corresponding number of parameters.
- // The workaround is to add a mutable list to pass in pushdownFilters.
- protected var pushdownFilters: Seq[Expression] = scan match {
- case fileScan: FileScan =>
- fileScan.dataFilters.filter {
- expr =>
- ExpressionConverter.canReplaceWithExpressionTransformer(
- ExpressionConverter.replaceAttributeReference(expr),
- output)
- }
+ override def scanFilters: Seq[Expression] = scan match {
+ case fileScan: FileScan => fileScan.dataFilters
case _ =>
- logInfo(s"${scan.getClass.toString} does not support push down filters")
+ // todo: support DSv2 scan
Review Comment:
Thanks for the review, I've made some minor adjustments.
I'd like to comment here is that we could support other DSv2 scans in the
future. For example, if IcebergScan or PaimonScan retains Spark's Catalyst
Filters and provides interface to expose them, then we could either match more
case or just inherit this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
