Re: Environment tab meaning

Hi, I'm not surprised to see Hadoop jars on the driver (yet I couldn't explain exactly why they need to be there). I can't find a way now to display the classpath for executors. Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark

Re: Environment tab meaning

Thank you Jacek. In case of YARN, I see that hadoop jars are present in system classpath for Driver. Will it be the same for all executors? On Tue, Jun 7, 2016 at 11:22 AM, Jacek Laskowski wrote: > Ouch, I made a mistake :( Sorry. > > You've asked about spark **history**

Re: Environment tab meaning

Ouch, I made a mistake :( Sorry. You've asked about spark **history** server. It's pretty much the same. HistoryServer is a web interface for completed and running (aka incomplete) Spark applications. It uses EventLoggingListener to collect events as JSON using org.apache.spark.util.JsonProtocol

Re: Environment tab meaning

Hi, It is the driver - see the port. Is this 4040 or similar? It's started when SparkContext starts and is controlled by spark.ui.enabled. spark.ui.enabled (default: true) = controls whether the web UI is started or not. It's through JobProgressListener which is the SparkListener for web UI

Environment tab meaning

Hi, In spark history server, we see environment tab. Is it show environment of Driver or Executor or both? - Jobs - Stages - Storage