Hey community Spark UI with the plan visualisation is an excellent resource for finding out crucial information about how your application is doing and what parts of the execution can still be optimized to fulfill time/resource constraints.
The graph in its current form is sufficient for simpler applications. It starts to show limitations, however, when we have applications with complex logic and resultant humongous plan. The limitations are in the form of trying to navigate the complex graph further complicated by what can fit in your screen/viewport. To make it user friendly, we could try to make it interactive allowing users to decide on what area of the plan they want focus on. There are a couple of inspirations/examples for this. 1. https://tensorboard.dev/experiment/rldGbR8rRHeCEbkK61SWTQ/#graphs 2. https://cs.brown.edu/people/jcmace/d3/graph.html?id=small.json If we could achieve this then it will be an even more powerful resource for spark users. Curious what you think Santosh
