Hi,
Sorry. A false alarm. Got mistaken with what IDEA calls "unused" may
not really be unused. It is (re)assigned in StageInfo.fromStage for
a ShuffleMapStage [1] and then caught in ExecutorMonitor [2] (since it's a
SparkListener).
[1]
https://github.com/apache/spark/blob/094563384478a402c36415ed
Hi,
It's been a while. Glad to be back Sparkians!
I've been exploring ExecutorMonitor.onJobStart in 3.0.1 and noticed that it
uses StageInfo.shuffleDepId [1] that is None by default and moreover never
"written to" according to IntelliJ IDEA.
Is this the case and intentional?
I'm wondering how m
