lyne7-sc commented on code in PR #2379:
URL: https://github.com/apache/auron/pull/2379#discussion_r3551319435
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala:
##########
@@ -382,6 +418,32 @@ object IcebergScanSupport extends Logging {
}
}
+ private def runtimeFilteredPartitions(exec: BatchScanExec):
Option[Seq[InputPartition]] = {
+ if (exec.runtimeFilters.isEmpty) {
+ return None
+ }
+
+ exec.prepare()
+ MethodUtils.invokeMethod(exec, true, "waitForSubqueries")
+ invokeDeclaredMethod(exec, "filteredPartitions") match {
+ case Some(seq: scala.collection.Seq[_]) =>
+ Some(flattenPartitions(seq))
+ case _ =>
+ None
+ }
+ }
Review Comment:
Thanks. I am intentionally not wrapping `prepare()` / `waitForSubqueries` in
`NonFatal` here.
Those calls match Spark `BatchScanExec` execution semantics: a failure while
preparing or waiting for runtime-filter subqueries is a real query failure and
should remain visible, not silently become an unfiltered full scan. This is the
same error-semantics point from the earlier review thread.
The graceful fallback is only for the `filteredPartitions` hook being
unavailable, in which case `runtimeFilteredPartitions` returns `None` and the
caller falls back with a warning. I added an inline comment to make this
boundary explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
