Ngone51 commented on code in PR #52213:
URL: https://github.com/apache/spark/pull/52213#discussion_r2425279745
##########
sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala:
##########
@@ -205,6 +206,9 @@ class InjectRuntimeFilterSuite extends QueryTest with
SQLTestUtils with SharedSp
sql("analyze table bf5part compute statistics for columns a5, b5, c5, d5,
e5, f5")
sql("analyze table bf5filtered compute statistics for columns a5, b5, c5,
d5, e5, f5")
+ // Tests depend on intermediate results that would otherwise be cleaned up
when
Review Comment:
[MapOutputTracker.unregisterMapOutput()](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L865C7-L865C26)
throws `SparkException` due to the `shuffleId` doesn't exist in
`shuffleStatuses`. And the `shuffleId` can only be cleaned up by
[MapOutputTracker.unregisterShuffle()](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L946),
which is only invoked by `ContextCleaner`
[here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ContextCleaner.scala#L242).
`ContextCleaner` invokes both `MapOutputTracker.unregisterShuffle()` and
`shuffleDriverComponents.removeShuffle()` to clean up shuffle metadata/data
files on both driver and executor separately.
However, when[ the main query
completes](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala#L200-L207),
it only calls `shuffleDriverComponents.removeShuffle()` but not
`MapOutputTracker.unregisterShuffle()`. So it should not be the culprit who
cleans up the `shuffleId` in `shuffleStatuses`.
One posibility I can think of now is - that `shuffleId` maybe hasn't been
even registered through `MapOutputTracker.registerShuffle()` when the main
query completes. But not sure if that's possible that subquery can start even
if the required shuffle is not ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
