Re: Latency due to driver fetching sizes of output statuses

Do all of those thousands of Stages end up being actual Stages that need to be computed, or are the vast majority of them eventually "skipped" Stages? If the latter, then there is the potential to modify the DAGScheduler to avoid much of this behavior:

Re: Latency due to driver fetching sizes of output statuses

yea, they're all skipped, here's a gif scrolling through the DAG viz. Thanks for the JIRA pointer, I'll keep an eye on that one! On Sat, Jan 23, 2016 at 4:53 PM Mark Hamstra

Latency due to driver fetching sizes of output statuses

I have a recursive algorithm that performs a few jobs on successively smaller RDDs, and then a few more jobs on successively larger RDDs as the recursion unwinds, resulting in a somewhat deeply-nested (a few dozen levels) RDD lineage. I am observing significant delays starting jobs while the