cloud-fan commented on code in PR #42318:
URL: https://github.com/apache/spark/pull/42318#discussion_r1283276807
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -249,12 +249,18 @@ trait FileSourceScanLike extends DataSourceScanExec {
private def isDynamicPruningFilter(e: Expression): Boolean =
e.exists(_.isInstanceOf[PlanExpression[_]])
+
+ // This field will be accessed during planning (e.g., `outputPartitioning`
relies on it), and can
+ // only use static filters.
@transient lazy val selectedPartitions: Array[PartitionDirectory] = {
Review Comment:
This is a public field of `FileSourceScanLike`. Some external Spark
extensions may access it during planning. So I make it safe to be executed
during planning, instead of ensuring it to be only executed after planning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
