Hi, It's been a while. Glad to be back Sparkians!
I've been exploring ExecutorMonitor.onJobStart in 3.0.1 and noticed that it uses StageInfo.shuffleDepId [1] that is None by default and moreover never "written to" according to IntelliJ IDEA. Is this the case and intentional? I'm wondering how much IDEA knows about codegen and that's where it's used (?) I've just stumbled upon it and before I spend more time on this I thought I'd ask (perhaps it's going to change in 3.1?). Help appreciated. [1] https://github.com/apache/spark/blob/78df2caec8c94c31e5c9ddc30ed8acb424084181/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L179 Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski>
