Re: Spark master takes more time with local[8] than local[1]

[nsalian]

Hi,

Thanks for the question.
Is it possible for you to elaborate on your application?
The flow of the application will help to understand what could potentially
cause things to slow down?

Do logs give you any idea what goes on? Have you had a chance to look?

Thank you.



-
Neelesh S. Salian
Cloudera
--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-master-takes-more-time-with-local-8-than-local-1-tp26052p26061.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark master takes more time with local[8] than local[1]

[Ted Yu]

bq. I'm reading a 4.3 GB file

The contents of the file can be held in one executor.

Can you try files with much larger size ?

Cheers

On Sun, Jan 24, 2016 at 12:11 PM, jimitkr  wrote:

> Hi All,
>
> I have a machine with the following configuration:
> 32 GB RAM
> 500 GB HDD
> 8 CPUs
>
> Following are the parameters i'm starting my Spark context with:
>
> val conf = new
>
> SparkConf().setAppName("MasterApp").setMaster("local[1]").set("spark.executor.memory",
> "20g")
>
> I'm reading a 4.3 GB file and counting the no. of characters in it.
>
> When i run my program with:
> local[1], the count is returned in 1.8 minutes
> local[8], the count is returned in 4.2 minutes
>
> Both the times, Spark uses Storage Memory of 3.7 GB.
>
> Why would my program take more time with local[8]?
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-master-takes-more-time-with-local-8-than-local-1-tp26052.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>