Re: [3.0.1] ExecutorMonitor.onJobStart and StageInfo.shuffleDepId that's never used?

2020-12-30 Thread Jacek Laskowski
Hi, Sorry. A false alarm. Got mistaken with what IDEA calls "unused" may not really be unused. It is (re)assigned in StageInfo.fromStage for a ShuffleMapStage [1] and then caught in ExecutorMonitor [2] (since it's a SparkListener). [1]

[3.0.1] ExecutorMonitor.onJobStart and StageInfo.shuffleDepId that's never used?

2020-12-30 Thread Jacek Laskowski
Hi, It's been a while. Glad to be back Sparkians! I've been exploring ExecutorMonitor.onJobStart in 3.0.1 and noticed that it uses StageInfo.shuffleDepId [1] that is None by default and moreover never "written to" according to IntelliJ IDEA. Is this the case and intentional? I'm wondering how