yyanyy commented on code in PR #59662:
URL: https://github.com/apache/spark/pull/59662#discussion_r4365390038
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:
##########
@@ -48,9 +48,12 @@ import org.apache.spark.util.Utils
object V2ScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper {
import DataSourceV2Implicits._
- def apply(plan: LogicalPlan): LogicalPlan = {
+ def apply(plan: LogicalPlan): LogicalPlan =
+ applyWithContext(plan, isSubquery = plan.isInstanceOf[Subquery])
Review Comment:
Yes, these scans can later become join inputs. The intended signal is the
planning scope when Spark creates the builder, allowing the connector to choose
how much planning/statistics work to do at that point. Later rewrites may move
or reuse the scan. I’ve clarified this creation-time meaning in the Javadoc;
the true expectations for scans initially constructed during subquery
optimization are intentional.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:
##########
@@ -48,9 +48,12 @@ import org.apache.spark.util.Utils
object V2ScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper {
import DataSourceV2Implicits._
- def apply(plan: LogicalPlan): LogicalPlan = {
+ def apply(plan: LogicalPlan): LogicalPlan =
+ applyWithContext(plan, isSubquery = plan.isInstanceOf[Subquery])
+
+ private def applyWithContext(plan: LogicalPlan, isSubquery: Boolean):
LogicalPlan = {
Review Comment:
(similar to above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
