Re: how to set spark.executor.memory and heap size
Hi All, This is a bit late, but I found it helpful. Piggy-backing on Wang Hao's comment, spark will ignore the "spark.executor.memory" setting if you add it to SparkConf via: conf.set("spark.executor.memory", "1g") What you actually should do depends on how you run spark. I found some "official" documentation for this in a bug report here: https://issues.apache.org/jira/browse/SPARK-1264 Alex On Fri, Jun 13, 2014 at 10:40 AM, Hao Wang wrote: > Hi, Laurent > > You could set Spark.executor.memory and heap size by following methods: > > 1. in you conf/spark-env.sh: > *export SPARK_WORKER_MEMORY=38g* > *export SPARK_JAVA_OPTS="-XX:-UseGCOverheadLimit > -XX:+UseConcMarkSweepGC -Xmx2g -XX:MaxPermSize=256m"* > > 2. you could also add modification for executor memory and java opts in > *spark-submit > *parameters. > > Check the Spark *configure *and *tuning *docs, you could find full > answers there. > > > Regards, > Wang Hao(王灏) > > CloudTeam | School of Software Engineering > Shanghai Jiao Tong University > Address:800 Dongchuan Road, Minhang District, Shanghai, 200240 > Email:wh.s...@gmail.com > > > On Thu, Jun 12, 2014 at 6:29 PM, Laurent T > wrote: > >> Hi, >> >> Can you give us a little more insight on how you used that file to solve >> your problem ? >> We're having the same OOM as you were and haven't been able to solve it >> yet. >> >> Thanks >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p7469.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >
Re: how to set spark.executor.memory and heap size
Hi, Laurent You could set Spark.executor.memory and heap size by following methods: 1. in you conf/spark-env.sh: *export SPARK_WORKER_MEMORY=38g* *export SPARK_JAVA_OPTS="-XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -Xmx2g -XX:MaxPermSize=256m"* 2. you could also add modification for executor memory and java opts in *spark-submit *parameters. Check the Spark *configure *and *tuning *docs, you could find full answers there. Regards, Wang Hao(王灏) CloudTeam | School of Software Engineering Shanghai Jiao Tong University Address:800 Dongchuan Road, Minhang District, Shanghai, 200240 Email:wh.s...@gmail.com On Thu, Jun 12, 2014 at 6:29 PM, Laurent T wrote: > Hi, > > Can you give us a little more insight on how you used that file to solve > your problem ? > We're having the same OOM as you were and haven't been able to solve it > yet. > > Thanks > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p7469.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
Re: how to set spark.executor.memory and heap size
Hi, Can you give us a little more insight on how you used that file to solve your problem ? We're having the same OOM as you were and haven't been able to solve it yet. Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p7469.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
Hi, finally, i solve this problem by using the SPARK_HOME/bin/run-example script to run my application, and it works. i guess the error is due to lack of some classpath -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4872.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
Hi I am also curious about this question. The textFile function was supposed to read a hdfs file? In this case ,It is on local filesystem that the file was taken in.There are any recognization ways to identify the local filesystem and the hdfs in the textFile function? Beside, the OOM exeception is really strange. Keeping eyes on this. 2014-04-25 13:10 GMT+08:00 Sean Owen : > On Fri, Apr 25, 2014 at 2:20 AM, wxhsdp wrote: > > 14/04/25 08:38:36 WARN util.NativeCodeLoader: Unable to load > native-hadoop > > library for your platform... using builtin-java classes where applicable > > 14/04/25 08:38:36 WARN snappy.LoadSnappy: Snappy native library not > loaded > > Since this comes up regularly -- these warnings from Hadoop are > entirely safe to ignore for development and testing. >
Re: how to set spark.executor.memory and heap size
On Fri, Apr 25, 2014 at 2:20 AM, wxhsdp wrote: > 14/04/25 08:38:36 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/04/25 08:38:36 WARN snappy.LoadSnappy: Snappy native library not loaded Since this comes up regularly -- these warnings from Hadoop are entirely safe to ignore for development and testing.
Re: how to set spark.executor.memory and heap size
i noticed that error occurs at org.apache.hadoop.io.WritableUtils.readCompressedStringArray(WritableUtils.java:183) at org.apache.hadoop.conf.Configuration.readFields(Configuration.java:2378) at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285) at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77) at org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) is it related to the warning below? 14/04/25 08:38:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/04/25 08:38:36 WARN snappy.LoadSnappy: Snappy native library not loaded -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4798.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
anyone knows the reason? i've googled a bit, and found some guys had the same problem, but with no replies... -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4796.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
it seems that it's nothing about settings, i tried take action, and find it's ok, but error occurs when i tried count and collect val a = sc.textFile("any file") a.take(n).foreach(println) //ok a.count() //failed a.collect()//failed val b = sc.parallelize((Array(1,2,3,4)) b.take(n).foreach(println) //ok b.count() //ok b.collect()//ok it's so weird Arpit Tak-2 wrote > Okk fine, > > try like this , i tried and it works.. > specify spark path also in constructor... > and also > export SPARK_JAVA_OPTS="-Xms300m -Xmx512m -XX:MaxPermSize=1g" > > import org.apache.spark.SparkContext > import org.apache.spark.SparkContext._ > object SimpleApp { >def main(args: Array[String]) { > val logFile = "/var/log/auth.log" // read any file. > val sc = new SparkContext("spark://localhost:7077", "Simple > App", > "/home/ubuntu/spark-0.9.1-incubating/", > List("target/scala-2.10/simple-project_2.10-2.0.jar")) > val tr = sc.textFile(logFile).cache > tr.take(100).foreach(println) > >} > } > > This will work > > > On Thu, Apr 24, 2014 at 3:00 PM, wxhsdp < > wxhsdp@ > > wrote: > >> hi arpit, >> on spark shell, i can read local file properly, >> but when i use sbt run, error occurs. >> the sbt error message is in the beginning of the thread >> >> >> Arpit Tak-2 wrote >> > Hi, >> > >> > You should be able to read it, file://or file:/// not even required for >> > reading locally , just path is enough.. >> > what error message you getting on spark-shell while reading... >> > for local: >> > >> > >> > Also read the same from hdfs file also ... >> > put your README file there and read , it works in both ways.. >> > val a= sc.textFile("hdfs://localhost:54310/t/README.md") >> > >> > also, print stack message of your spark-shell... >> > >> > >> > On Thu, Apr 24, 2014 at 2:25 PM, wxhsdp < >> >> > wxhsdp@ >> >> > > wrote: >> > >> >> thanks for your reply, adnan, i tried >> >> val logFile = "file:///home/wxhsdp/spark/example/standalone/README.md" >> >> i think there needs three left slash behind file: >> >> >> >> it's just the same as val logFile = >> >> "home/wxhsdp/spark/example/standalone/README.md" >> >> the error remains:( >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4743.html >> >> Sent from the Apache Spark User List mailing list archive at >> Nabble.com. >> >> >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4745.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4752.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
Okk fine, try like this , i tried and it works.. specify spark path also in constructor... and also export SPARK_JAVA_OPTS="-Xms300m -Xmx512m -XX:MaxPermSize=1g" import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ object SimpleApp { def main(args: Array[String]) { val logFile = "/var/log/auth.log" // read any file. val sc = new SparkContext("spark://localhost:7077", "Simple App", "/home/ubuntu/spark-0.9.1-incubating/", List("target/scala-2.10/simple-project_2.10-2.0.jar")) val tr = sc.textFile(logFile).cache tr.take(100).foreach(println) } } This will work On Thu, Apr 24, 2014 at 3:00 PM, wxhsdp wrote: > hi arpit, > on spark shell, i can read local file properly, > but when i use sbt run, error occurs. > the sbt error message is in the beginning of the thread > > > Arpit Tak-2 wrote > > Hi, > > > > You should be able to read it, file://or file:/// not even required for > > reading locally , just path is enough.. > > what error message you getting on spark-shell while reading... > > for local: > > > > > > Also read the same from hdfs file also ... > > put your README file there and read , it works in both ways.. > > val a= sc.textFile("hdfs://localhost:54310/t/README.md") > > > > also, print stack message of your spark-shell... > > > > > > On Thu, Apr 24, 2014 at 2:25 PM, wxhsdp < > > > wxhsdp@ > > > > wrote: > > > >> thanks for your reply, adnan, i tried > >> val logFile = "file:///home/wxhsdp/spark/example/standalone/README.md" > >> i think there needs three left slash behind file: > >> > >> it's just the same as val logFile = > >> "home/wxhsdp/spark/example/standalone/README.md" > >> the error remains:( > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4743.html > >> Sent from the Apache Spark User List mailing list archive at Nabble.com. > >> > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4745.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
Re: how to set spark.executor.memory and heap size
hi arpit, on spark shell, i can read local file properly, but when i use sbt run, error occurs. the sbt error message is in the beginning of the thread Arpit Tak-2 wrote > Hi, > > You should be able to read it, file://or file:/// not even required for > reading locally , just path is enough.. > what error message you getting on spark-shell while reading... > for local: > > > Also read the same from hdfs file also ... > put your README file there and read , it works in both ways.. > val a= sc.textFile("hdfs://localhost:54310/t/README.md") > > also, print stack message of your spark-shell... > > > On Thu, Apr 24, 2014 at 2:25 PM, wxhsdp < > wxhsdp@ > > wrote: > >> thanks for your reply, adnan, i tried >> val logFile = "file:///home/wxhsdp/spark/example/standalone/README.md" >> i think there needs three left slash behind file: >> >> it's just the same as val logFile = >> "home/wxhsdp/spark/example/standalone/README.md" >> the error remains:( >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4743.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4745.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
Hi, You should be able to read it, file://or file:/// not even required for reading locally , just path is enough.. what error message you getting on spark-shell while reading... for local: Also read the same from hdfs file also ... put your README file there and read , it works in both ways.. val a= sc.textFile("hdfs://localhost:54310/t/README.md") also, print stack message of your spark-shell... On Thu, Apr 24, 2014 at 2:25 PM, wxhsdp wrote: > thanks for your reply, adnan, i tried > val logFile = "file:///home/wxhsdp/spark/example/standalone/README.md" > i think there needs three left slash behind file: > > it's just the same as val logFile = > "home/wxhsdp/spark/example/standalone/README.md" > the error remains:( > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4743.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
Re: how to set spark.executor.memory and heap size
thanks for your reply, adnan, i tried val logFile = "file:///home/wxhsdp/spark/example/standalone/README.md" i think there needs three left slash behind file: it's just the same as val logFile = "home/wxhsdp/spark/example/standalone/README.md" the error remains:( -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4743.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
Sorry wrong format: file:///home/wxhsdp/spark/example/standalone/README.md An extra / is needed at the start. On Thu, Apr 24, 2014 at 1:46 PM, Adnan Yaqoob wrote: > You need to use proper url format: > > file://home/wxhsdp/spark/example/standalone/README.md > > > On Thu, Apr 24, 2014 at 1:29 PM, wxhsdp wrote: > >> i think maybe it's the problem of read local file >> >> val logFile = "/home/wxhsdp/spark/example/standalone/README.md" >> val logData = sc.textFile(logFile).cache() >> >> if i replace the above code with >> >> val logData = sc.parallelize(Array(1,2,3,4)).cache() >> >> the job can complete successfully >> >> can't i read a file located at local file system? anyone knows the reason? >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4740.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >
Re: how to set spark.executor.memory and heap size
You need to use proper url format: file://home/wxhsdp/spark/example/standalone/README.md On Thu, Apr 24, 2014 at 1:29 PM, wxhsdp wrote: > i think maybe it's the problem of read local file > > val logFile = "/home/wxhsdp/spark/example/standalone/README.md" > val logData = sc.textFile(logFile).cache() > > if i replace the above code with > > val logData = sc.parallelize(Array(1,2,3,4)).cache() > > the job can complete successfully > > can't i read a file located at local file system? anyone knows the reason? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4740.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
Re: how to set spark.executor.memory and heap size
i think maybe it's the problem of read local file val logFile = "/home/wxhsdp/spark/example/standalone/README.md" val logData = sc.textFile(logFile).cache() if i replace the above code with val logData = sc.parallelize(Array(1,2,3,4)).cache() the job can complete successfully can't i read a file located at local file system? anyone knows the reason? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4740.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Re: how to set spark.executor.memory and heap size
i tried, but no effect Qin Wei wrote > try the complete path > > > qinwei > From: wxhsdpDate: 2014-04-24 14:21To: userSubject: Re: how to set > spark.executor.memory and heap sizethank you, i add setJars, but nothing > changes > > val conf = new SparkConf() > .setMaster("spark://127.0.0.1:7077") > .setAppName("Simple App") > .set("spark.executor.memory", "1g") > .setJars(Seq("target/scala-2.10/simple-project_2.10-1.0.jar")) > val sc = new SparkContext(conf) > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4732.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4736.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Re: how to set spark.executor.memory and heap size
try the complete path qinwei From: wxhsdpDate: 2014-04-24 14:21To: userSubject: Re: how to set spark.executor.memory and heap sizethank you, i add setJars, but nothing changes val conf = new SparkConf() .setMaster("spark://127.0.0.1:7077") .setAppName("Simple App") .set("spark.executor.memory", "1g") .setJars(Seq("target/scala-2.10/simple-project_2.10-1.0.jar")) val sc = new SparkContext(conf) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4732.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
thank you, i add setJars, but nothing changes val conf = new SparkConf() .setMaster("spark://127.0.0.1:7077") .setAppName("Simple App") .set("spark.executor.memory", "1g") .setJars(Seq("target/scala-2.10/simple-project_2.10-1.0.jar")) val sc = new SparkContext(conf) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4732.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: how to set spark.executor.memory and heap size
When I was testing spark, I faced this issue, this issue is not related to memory shortage, It is because your configurations are not correct. Try to pass you current Jar to to the SparkContext with SparkConf's setJars function and try again. On Thu, Apr 24, 2014 at 8:38 AM, wxhsdp wrote: > by the way, codes run ok in spark shell > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4720.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
Re: how to set spark.executor.memory and heap size
by the way, codes run ok in spark shell -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4720.html Sent from the Apache Spark User List mailing list archive at Nabble.com.