Re: SPARK BENCHMARK TEST
Hi Please can someone advice on this. On Wed, Sep 17, 2014 at 6:59 PM, VJ Shalish wrote: > I am trying to benchmark spark in a hadoop cluster. > I need to design a sample spark job to test the CPU utilization, RAM > usage, Input throughput, Output throughput and Duration of execution in the > cluster. > > I need to test the state of the cluster for :- > > A spark job which uses high CPU > A spark job which uses high RAM > A spark job which uses high Input throughput > A spark job which uses high Output throughput > A spark job which takes long time. > > These have to be tested individually and a combination of these scenarios > would also be used. > > Please help me in understanding the factors of a Spark Job which would > contribute to CPU utilization, RAM usage, Input throughput, Output > throughput, Duration of execution in the cluster. So that I can design > spark jobs that could be used for testing. > > > > Thanks > Shalish. > > >
SPARK BENCHMARK TEST
I am trying to benchmark spark in a hadoop cluster. I need to design a sample spark job to test the CPU utilization, RAM usage, Input throughput, Output throughput and Duration of execution in the cluster. I need to test the state of the cluster for :- A spark job which uses high CPU A spark job which uses high RAM A spark job which uses high Input throughput A spark job which uses high Output throughput A spark job which takes long time. These have to be tested individually and a combination of these scenarios would also be used. Please help me in understanding the factors of a Spark Job which would contribute to CPU utilization, RAM usage, Input throughput, Output throughput, Duration of execution in the cluster. So that I can design spark jobs that could be used for testing. Thanks Shalish.
Re: CPU RAM
Hi I need the same through Java. Doesn't the SPark API support this? On Wed, Sep 17, 2014 at 2:48 AM, Akhil Das wrote: > Ganglia does give you a cluster wide and per machine utilization of > resources, but i don't think it gives your per Spark Job. If you want to > build something from scratch then you can follow up like : > > 1. Login to the machine > 2. Get the PIDs > 3. For network IO per process, you can have a look at > http://nethogs.sourceforge.net/ > 4. You can make use of the information in /proc/[pid]/stat and /proc/stat > to estimate CPU usage and all > > > Similarly you can get any metric of process once you have the PID. > > > Thanks > Best Regards > > On Wed, Sep 17, 2014 at 8:59 AM, VJ Shalish wrote: > >> Sorry for the confusion Team. >> My requirement is to measure the CPU utilisation, RAM usage, Network IO >> and other metrics of a SPARK JOB using Java program. >> Please help on the same. >> >> On Tue, Sep 16, 2014 at 11:23 PM, Amit wrote: >> >>> Not particularly related to Spark, but you can check out SIGAR API. It >>> let's you get CPU, Memory, Network, Filesystem and process based metrics. >>> >>> Amit >>> On Sep 16, 2014, at 20:14, VJ Shalish wrote: >>> >>> > Hi >>> > >>> > I need to get the CPU utilisation, RAM usage, Network IO and other >>> metrics using Java program. Can anyone help me on this? >>> > >>> > Thanks >>> > Shalish. >>> >> >> >
Re: CPU RAM
Sorry for the confusion Team. My requirement is to measure the CPU utilisation, RAM usage, Network IO and other metrics of a SPARK JOB using Java program. Please help on the same. On Tue, Sep 16, 2014 at 11:23 PM, Amit wrote: > Not particularly related to Spark, but you can check out SIGAR API. It > let's you get CPU, Memory, Network, Filesystem and process based metrics. > > Amit > On Sep 16, 2014, at 20:14, VJ Shalish wrote: > > > Hi > > > > I need to get the CPU utilisation, RAM usage, Network IO and other > metrics using Java program. Can anyone help me on this? > > > > Thanks > > Shalish. >
Re: CPU RAM
Thank u for the response Amit. So is it that, we cannot measure the CPU consumption, RAM usage of a spark job through a Java program? On Tue, Sep 16, 2014 at 11:23 PM, Amit wrote: > Not particularly related to Spark, but you can check out SIGAR API. It > let's you get CPU, Memory, Network, Filesystem and process based metrics. > > Amit > On Sep 16, 2014, at 20:14, VJ Shalish wrote: > > > Hi > > > > I need to get the CPU utilisation, RAM usage, Network IO and other > metrics using Java program. Can anyone help me on this? > > > > Thanks > > Shalish. >
CPU RAM
Hi I need to get the CPU utilisation, RAM usage, Network IO and other metrics using Java program. Can anyone help me on this? Thanks Shalish.
