jerrypeng commented on code in PR #59523:
URL: https://github.com/apache/spark/pull/59523#discussion_r4429443703
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/PipelinedShuffleEligibility.scala:
##########
@@ -49,6 +50,9 @@ private[sql] object PipelinedShuffleEligibility extends
Logging {
def isUnsupportedConsumer(plan: SparkPlan): Boolean = plan match {
case _: CoalesceExec | _: CartesianProductExec | _:
DeserializeToObjectExec |
_: CollectLimitExec | _: CollectTailExec | _:
TakeOrderedAndProjectExec => true
+ // Caching can persist a shuffle reader before zipWithIndex submits its
counting job.
Review Comment:
Updated the shared predicate's Scaladoc, the rule's introductory
explanation, and its inline fallback comment to explicitly include operators
that can persist a channel reader. This covers cached
`AttachDistributedSequenceExec`, where cache hits can skip channel readers. The
eligibility behavior is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
