Hi, I've noticed my spark app (on ec2) gets slower and slower as I repeatedly execute it.
With a fresh ec2 cluster, it is snappy and takes about 15 mins to complete, after running the same app 4 times it gets slower and takes to 40 mins and more. While the cluster gets slower, the monitoring metrics show less and less activities (almost no cpu, or io). When it gets slow, sometimes the number of running tasks (light blue in web ui progress bar) is zero, and only the number of completed tasks (dark blue) increments. Is this a known spark issue? Do I need to restart spark master and workers in between running apps?
