weiqingy commented on code in PR #2379:
URL: https://github.com/apache/auron/pull/2379#discussion_r3526454131
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala:
##########
@@ -82,35 +85,54 @@ object IcebergScanSupport extends Logging {
}
}
- def plan(exec: BatchScanExec): Option[IcebergScanPlan] = {
- exec.getTagValue(scanPlanTag) match {
+ def plan(exec: BatchScanExec, useRuntimeFilters: Boolean = false):
Option[IcebergScanPlan] = {
+ val tag =
+ if (useRuntimeFilters && exec.runtimeFilters.nonEmpty) {
+ runtimeFilteredScanPlanTag
+ } else {
+ scanPlanTag
+ }
+ exec.getTagValue(tag) match {
case Some(cached) => cached
case None =>
- val planned = planUncached(exec)
- exec.setTagValue(scanPlanTag, planned)
+ val planned = planUncached(exec, useRuntimeFilters)
+ exec.setTagValue(tag, planned)
planned
}
}
- private def planUncached(exec: BatchScanExec): Option[IcebergScanPlan] = {
+ def withRuntimeFilters(
+ exec: BatchScanExec,
+ runtimeFilters: Seq[SparkExpression]): BatchScanExec = {
+ if (exec.runtimeFilters == runtimeFilters) {
Review Comment:
Makes sense — keeping the native scan's runtime filters independent from
`basedScan.runtimeFilters` is a reasonable boundary to hold, and the added
comment makes that intent explicit so the copy branch won't read as dead code
to the next person. Agreed that exercising the differing-filters path would
need a native rewrite rule that's broader than this PR. Thanks for checking the
4.1 shims profile compiles too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
