Re: Total Shuffle Read and Write Size of Spark workload

While the SparkListener method is likely all around better, if you just need this quickly you should be able to do a SSH local port redirection over putty. In the putty configuration: - Go to Connection: SSH: Tunnels - In the Source port field, enter 4040 (or another unused port on your machine)

Re: Total Shuffle Read and Write Size of Spark workload

On Mon, Sep 19, 2016 at 11:36 AM, Mich Talebzadeh wrote: > Spark UI on port 4040 by default That's exactly *a* SparkListener + web UI :) Jacek - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Total Shuffle Read and Write Size of Spark workload

Hi Cristina, http://blog.jaceklaskowski.pl/spark-workshop/slides/08_Monitoring_using_SparkListeners.html http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.scheduler.SparkListener Let me know if you've got more questions. Pozdrawiam, Jacek Laskowski https://medium.co

Re: Total Shuffle Read and Write Size of Spark workload

I Mich, I do not have access to UI as I am running jobs on remote system and I can access it using putty only so only console or logs files are available to me. Thanks On Mon, Sep 19, 2016 at 11:36 AM, Mich Talebzadeh wrote: > Spark UI on port 4040 by default > > HTH > > Dr Mich Talebzadeh > >

Re: Total Shuffle Read and Write Size of Spark workload

Spark UI on port 4040 by default HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it

Re: Total Shuffle Read and Write Size of Spark workload

Could you please explain a little bit? On Sun, Sep 18, 2016 at 10:19 PM, Jacek Laskowski wrote: > SparkListener perhaps? > > Jacek > > On 15 Sep 2016 1:41 p.m., "Cristina Rozee" > wrote: > >> Hello, >> >> I am running a spark application and I would like to know the total >> amount of shuffle d

Re: Total Shuffle Read and Write Size of Spark workload

SparkListener perhaps? Jacek On 15 Sep 2016 1:41 p.m., "Cristina Rozee" wrote: > Hello, > > I am running a spark application and I would like to know the total amount > of shuffle data (read + write ) so could anyone let me know how to get this > information? > > Thank you > Cristina. >

Total Shuffle Read and Write Size of Spark workload

Hello, I am running a spark application and I would like to know the total amount of shuffle data (read + write ) so could anyone let me know how to get this information? Thank you Cristina.