Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-63244719
Hey @davies - I played with this a bit and I actually found the behavior
around concurrent stages might not be great. The reason is that the set of
active stages will change as stages complete, and then it will suddenly change
the slider significantly once one stage completes. Here is an example workload:
```
> ./bin/spark-shell --conf spark.scheduler.mode=FAIR
scala> val a = sc.makeRDD(1 to 1000, 10000).map(x => (x, x)).reduceByKey(_
+ _)
scala> val b = sc.makeRDD(1 to 1000, 10000).map(x => (x, x)).reduceByKey(_
+ _)
scala> a.union(b).count()
```
Probably what we want in the longer term is to have a slider for the entire
job rather than stages. But anyways, I'd prefer either the "flip flop" behavior
or have multiple stacked progress bars. @kayousterhout didn't like the "flip
flop" but I find it more understandable than what is here now. Since this is an
opt-in feature I think it's fine to have some version that can go in now and
then refine it later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
