Thanks for the response Mayur. I was seeing the webui of 0.9.0 spark. I see
lots of detailed statistics in the newer 1.0.0-snapshot version. The only thing
I found missing was the actual code that I had typed in at the spark-shell
prompt, but I can always get it from the shell history.
On
Is there a way to see the resource usage of each spark-shell command — say time
taken and memory used?
I checked the WebUI of spark-shell and of the master and I don’t see any such
breakdown. I see the time taken in the INFO logs but nothing about memory
usage. It would also be nice to track
