Re: DAGScheduler.handleJobCancellation uses jobIdToStageIds for verification while jobIdToActiveJob for lookup?

2016-10-13 Thread Mark Hamstra
There were at least a couple of ideas behind the original thinking on using both of those Maps: 1) We needed the ability to efficiently get from jobId to both ActiveJob objects and to sets of associated Stages, and using both Maps here was an opportunity to do a little sanity checking to make sure

Re: DAGScheduler.handleJobCancellation uses jobIdToStageIds for verification while jobIdToActiveJob for lookup?

2016-10-13 Thread Jacek Laskowski
Thanks Imran! Not only did the response come so promptly, but also it's something I could work on (and have another Spark contributor badge unlocked)! Thanks. Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark Follo

Re: DAGScheduler.handleJobCancellation uses jobIdToStageIds for verification while jobIdToActiveJob for lookup?

2016-10-13 Thread Imran Rashid
Hi Jacek, doesn't look like there is any good reason -- Mark Hamstra might know this best. Feel free to open a jira & pr for it, you can ping Mark, Kay Ousterhout, and me (@squito) for review. Imran On Thu, Oct 13, 2016 at 7:56 AM, Jacek Laskowski wrote: > Hi, > > Is there a reason why DAGSch