Thanks all for the comments. Just a clarification: On Tue, Aug 1, 2017 at 2:18 AM, Sean Owen <so...@cloudera.com> wrote: > Is 'spark-ui' too broad? doesn't sound like this module would actually house > all the UIs. spark-shs-ui or something? > Good that this can be implemented in parallel to the existing mechanism for > the initial milestones. > So the "Mx" milestones are essentially required, in your view? and the "SMx" > are optional and stand-alone?
The spec attached to the bug was written before I had a chance to work on most of the implementation, so it doesn't accurately reflect the current status of the code. The major changes are all there, but I changed some details when I ran into issues during implementation that I thought would end up being blockers. The two main changes from the spec are: - I gave up separating the UI into a separate module for now. It would probably break someone's workflow, and it made it really hard to track UI changes made concurrently. With the current approach I know I'm not missing anything that was added to the UI while I was writing that code. - While doing some perf testing I decided to make some optional milestones required; mainly, having an in-memory implementation, and having that be the default. This also keeps the current behavior as the default, and people have to opt-in to using the new code. - In spite of calling it out as a non-goal, I found that at least the current stage page became really slow with a LevelDB backend with large jobs (it's already slow now, but it got worse), so I included SPARK-20657 in the things I think are needed for the first implementation. That should be all. -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
