HyukjinKwon commented on code in PR #40520:
URL: https://github.com/apache/spark/pull/40520#discussion_r1144800754
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/MapInPandasExec.scala:
##########
@@ -28,7 +28,8 @@ import org.apache.spark.sql.execution.SparkPlan
case class MapInPandasExec(
func: Expression,
output: Seq[Attribute],
- child: SparkPlan)
+ child: SparkPlan,
+ override val isBarrier: Boolean)
Review Comment:
It would require an implementation much complicated than this actually. The
current implementation might work for now because these specific physical plans
don't move around much for now but the implementation is flaky because the
physical plans can change via Catalyst Optimizer (e.g., predicate pushdown) but
the barrier execution mode requires that the barrier is created exactly the
location where it's invoked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
