[jira] [Comment Edited] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container
[ https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341498#comment-14341498 ] Lianhui Wang edited comment on SPARK-6056 at 2/28/15 12:36 PM: --- [~carlmartin] what is your executor's memory? that is ok when i run your test using 1g. in addition, i use spark 1.2.0 release version. [~adav] yes, thanks. was (Author: lianhuiwang): [~carlmartin] what is your executor's memory? that is ok when i run your test using 1g. in addition, i use spark 1.2.0 release version. Unlimit offHeap memory use cause RM killing the container - Key: SPARK-6056 URL: https://issues.apache.org/jira/browse/SPARK-6056 Project: Spark Issue Type: Bug Components: Shuffle, Spark Core Affects Versions: 1.2.1 Reporter: SaintBacchus No matter set the `preferDirectBufs` or limit the number of thread or not ,spark can not limit the use of offheap memory. At line 269 of the class 'AbstractNioByteChannel' in netty-4.0.23.Final, Netty had allocated a offheap memory buffer with the same size in heap. So how many buffer you want to transfor, the same size offheap memory will be allocated. But once the allocated memory size reach the capacity of the overhead momery set in yarn, this executor will be killed. I wrote a simple code to test it: ```scala val bufferRdd = sc.makeRDD(0 to 10, 10).map(x=new Array[Byte](10*1024*1024)).persist bufferRdd.count val part = bufferRdd.partitions(0) val sparkEnv = SparkEnv.get val blockMgr = sparkEnv.blockManager val blockOption = blockMgr.get(RDDBlockId(bufferRdd.id, part.index)) val resultIt = blockOption.get.data.asInstanceOf[Iterator[Array[Byte]]] val len = resultIt.map(_.length).sum ``` If use multi-thread to get len, the physical memery will soon exceed the limit set by spark.yarn.executor.memoryOverhead -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container
[
https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341957#comment-14341957
]
SaintBacchus edited comment on SPARK-6056 at 3/1/15 6:01 AM:
-
[~adav] Thx for comment. I can't understand what you say clearly. Do you mean
if 'spark.shuffle.io.preferDirectBufs ' was set to be false, would netty only
allocat the heap memory? Is it right?
I test it again.And I had update the test program in the Description.
I had set the _spark.shuffle.io.preferDirectBufs_ and when I type
*test_driver(20,100)(test)* , the result is this:
_ 76602 root 20 0 1597m 339m 25m S0 0.1 0:04.89 java _
_76602 root 20 0 {color:red} 1777m {color} 1.0g 26m S 99 0.3
0:07.88 java _
_ 76602 root 20 0 1597m 880m 26m S4 0.3 0:07.99 java_
The red num is visual memory and it had raised about 180mb in the moment and
total transfor 200mb data (20 * 10MB) from executor to driver.
I think it's a big problem. If I use 40 threads to get the result, it will need
near 400mb momery and so exceed the limit of yarn fanally killed by yarn.
If there is a way to limit the peek use of memory, it will be fine. In addtion,
I though the user side number of remote fetch block threads is uncontrollable,
it's better to be controlled in spark.
[~lianhuiwang] I use the recent spark in github and I also tested the 1.2.0
release version.
In my test case, I use the default memory: 1G executor and 384 overhead. But in
the real case, momery is much more.
was (Author: carlmartin):
[~adav] Thx for comment. I can't understand what you say clearly. Do you mean
if 'spark.shuffle.io.preferDirectBufs ' was set to be false, would netty only
allocat the heap memory? Is it right?
I test it again.And I had update the test program in the Description.
I had set the _spark.shuffle.io.preferDirectBufs_ and when I type
*test_driver(20,100)(test)* , the result is this:
_ 76602 root 20 0 1597m 339m 25m S0 0.1 0:04.89 java _
_76602 root 20 0 {color:red} 1777m {color} 1.0g 26m S 99 0.3
0:07.88 java _
_ 76602 root 20 0 1597m 880m 26m S4 0.3 0:07.99 java_
The red num had raised about 180mb in the moment and total transfor 200mb data
(20 * 10MB) from executor to driver.
I think it's a big problem. If I use 40 threads to get the result, it will need
near 400mb momery and so exceed the limit of yarn fanally killed by yarn.
If there is a way to limit the peek use of memory, it will be fine. In addtion,
I though the user side number of remote fetch block threads is uncontrollable,
it's better to be controlled in spark.
[~lianhuiwang] I use the recent spark in github and I also tested the 1.2.0
release version.
In my test case, I use the default memory: 1G executor and 384 overhead. But in
the real case, momery is much more.
Unlimit offHeap memory use cause RM killing the container
-
Key: SPARK-6056
URL: https://issues.apache.org/jira/browse/SPARK-6056
Project: Spark
Issue Type: Bug
Components: Shuffle, Spark Core
Affects Versions: 1.2.1
Reporter: SaintBacchus
No matter set the `preferDirectBufs` or limit the number of thread or not
,spark can not limit the use of offheap memory.
At line 269 of the class 'AbstractNioByteChannel' in netty-4.0.23.Final,
Netty had allocated a offheap memory buffer with the same size in heap.
So how many buffer you want to transfor, the same size offheap memory will be
allocated.
But once the allocated memory size reach the capacity of the overhead momery
set in yarn, this executor will be killed.
I wrote a simple code to test it:
{code:title=test.scala|borderStyle=solid}
import org.apache.spark.storage._
import org.apache.spark._
val bufferRdd = sc.makeRDD(0 to 10, 10).map(x=new
Array[Byte](10*1024*1024)).persist
bufferRdd.count
val part = bufferRdd.partitions(0)
val sparkEnv = SparkEnv.get
val blockMgr = sparkEnv.blockManager
def test = {
val blockOption = blockMgr.get(RDDBlockId(bufferRdd.id, part.index))
val resultIt =
blockOption.get.data.asInstanceOf[Iterator[Array[Byte]]]
val len = resultIt.map(_.length).sum
println(s[${Thread.currentThread.getId}] get block length = $len)
}
def test_driver(count:Int, parallel:Int)(f: = Unit) = {
val tpool = new scala.concurrent.forkjoin.ForkJoinPool(parallel)
val taskSupport = new
scala.collection.parallel.ForkJoinTaskSupport(tpool)
val parseq = (1 to
[jira] [Comment Edited] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container
[
https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341957#comment-14341957
]
SaintBacchus edited comment on SPARK-6056 at 3/1/15 6:03 AM:
-
[~adav] Thx for comment. I can't understand what you say clearly. Do you mean
if 'spark.shuffle.io.preferDirectBufs ' was set to be false, would netty only
allocat the heap memory? Is it right?
I test it again.And I had update the test program in the Description.
I had set the _spark.shuffle.io.preferDirectBufs_ and when I type
*test_driver(20,100)(test)* , the result is this:
_ 76602 root 20 0 1597m 339m 25m S0 0.1 0:04.89 java _
_76602 root 20 0 {color:red} 1777m {color} 1.0g 26m S 99 0.3
0:07.88 java _
_ 76602 root 20 0 1597m 880m 26m S4 0.3 0:07.99 java_
The red num is visual memory and it had raised about 180mb in the moment and
total transfor 200mb data (20 * 10MB) from executor to driver.
I think it's a problem. If I use 40 threads to get the result, it will need
near 400mb momery and soon exceed the limit of yarn, fanally killed by yarn.
If there is a way to limit the peek use of memory, it will be fine. In addtion,
I though the number of remote fetch block threads in user side is
uncontrollable, it's better to be controlled in spark.
[~lianhuiwang] I use the recent spark in github and I also tested the 1.2.0
release version.
In my test case, I use the default memory: 1G executor and 384 overhead. But in
the real case, momery is much more.
was (Author: carlmartin):
[~adav] Thx for comment. I can't understand what you say clearly. Do you mean
if 'spark.shuffle.io.preferDirectBufs ' was set to be false, would netty only
allocat the heap memory? Is it right?
I test it again.And I had update the test program in the Description.
I had set the _spark.shuffle.io.preferDirectBufs_ and when I type
*test_driver(20,100)(test)* , the result is this:
_ 76602 root 20 0 1597m 339m 25m S0 0.1 0:04.89 java _
_76602 root 20 0 {color:red} 1777m {color} 1.0g 26m S 99 0.3
0:07.88 java _
_ 76602 root 20 0 1597m 880m 26m S4 0.3 0:07.99 java_
The red num is visual memory and it had raised about 180mb in the moment and
total transfor 200mb data (20 * 10MB) from executor to driver.
I think it's a big problem. If I use 40 threads to get the result, it will need
near 400mb momery and so exceed the limit of yarn fanally killed by yarn.
If there is a way to limit the peek use of memory, it will be fine. In addtion,
I though the user side number of remote fetch block threads is uncontrollable,
it's better to be controlled in spark.
[~lianhuiwang] I use the recent spark in github and I also tested the 1.2.0
release version.
In my test case, I use the default memory: 1G executor and 384 overhead. But in
the real case, momery is much more.
Unlimit offHeap memory use cause RM killing the container
-
Key: SPARK-6056
URL: https://issues.apache.org/jira/browse/SPARK-6056
Project: Spark
Issue Type: Bug
Components: Shuffle, Spark Core
Affects Versions: 1.2.1
Reporter: SaintBacchus
No matter set the `preferDirectBufs` or limit the number of thread or not
,spark can not limit the use of offheap memory.
At line 269 of the class 'AbstractNioByteChannel' in netty-4.0.23.Final,
Netty had allocated a offheap memory buffer with the same size in heap.
So how many buffer you want to transfor, the same size offheap memory will be
allocated.
But once the allocated memory size reach the capacity of the overhead momery
set in yarn, this executor will be killed.
I wrote a simple code to test it:
{code:title=test.scala|borderStyle=solid}
import org.apache.spark.storage._
import org.apache.spark._
val bufferRdd = sc.makeRDD(0 to 10, 10).map(x=new
Array[Byte](10*1024*1024)).persist
bufferRdd.count
val part = bufferRdd.partitions(0)
val sparkEnv = SparkEnv.get
val blockMgr = sparkEnv.blockManager
def test = {
val blockOption = blockMgr.get(RDDBlockId(bufferRdd.id, part.index))
val resultIt =
blockOption.get.data.asInstanceOf[Iterator[Array[Byte]]]
val len = resultIt.map(_.length).sum
println(s[${Thread.currentThread.getId}] get block length = $len)
}
def test_driver(count:Int, parallel:Int)(f: = Unit) = {
val tpool = new scala.concurrent.forkjoin.ForkJoinPool(parallel)
val taskSupport = new
scala.collection.parallel.ForkJoinTaskSupport(tpool)
[jira] [Comment Edited] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container
[
https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341957#comment-14341957
]
SaintBacchus edited comment on SPARK-6056 at 3/1/15 6:04 AM:
-
[~adav] Thx for comment. I can't understand what you say clearly. Do you mean
if 'spark.shuffle.io.preferDirectBufs ' was set to be false, would netty only
allocat the heap memory? Is it right?
I test it again.And I had update the test program in the Description.
I had set the _spark.shuffle.io.preferDirectBufs_ and when I type
*test_driver(20,100)(test)* , the result is this:
_ 76602 root 20 0 1597m 339m 25m S0 0.1 0:04.89 java _
_76602 root 20 0 {color:red} 1777m {color} 1.0g 26m S 99 0.3
0:07.88 java _
_ 76602 root 20 0 1597m 880m 26m S4 0.3 0:07.99 java_
The red num is visual memory and it had raised about 180mb in the moment and
the test case total transfor 200mb data (20 * 10MB) from executor to driver.
I think it's a problem. If I use 40 threads to get the result, it will need
near 400mb momery and soon exceed the limit of yarn, fanally killed by yarn.
If there is a way to limit the peek use of memory, it will be fine. In addtion,
I though the number of remote fetch block threads in user side is
uncontrollable, it's better to be controlled in spark.
[~lianhuiwang] I use the recent spark in github and I also tested the 1.2.0
release version.
In my test case, I use the default memory: 1G executor and 384 overhead. But in
the real case, momery is much more.
was (Author: carlmartin):
[~adav] Thx for comment. I can't understand what you say clearly. Do you mean
if 'spark.shuffle.io.preferDirectBufs ' was set to be false, would netty only
allocat the heap memory? Is it right?
I test it again.And I had update the test program in the Description.
I had set the _spark.shuffle.io.preferDirectBufs_ and when I type
*test_driver(20,100)(test)* , the result is this:
_ 76602 root 20 0 1597m 339m 25m S0 0.1 0:04.89 java _
_76602 root 20 0 {color:red} 1777m {color} 1.0g 26m S 99 0.3
0:07.88 java _
_ 76602 root 20 0 1597m 880m 26m S4 0.3 0:07.99 java_
The red num is visual memory and it had raised about 180mb in the moment and
total transfor 200mb data (20 * 10MB) from executor to driver.
I think it's a problem. If I use 40 threads to get the result, it will need
near 400mb momery and soon exceed the limit of yarn, fanally killed by yarn.
If there is a way to limit the peek use of memory, it will be fine. In addtion,
I though the number of remote fetch block threads in user side is
uncontrollable, it's better to be controlled in spark.
[~lianhuiwang] I use the recent spark in github and I also tested the 1.2.0
release version.
In my test case, I use the default memory: 1G executor and 384 overhead. But in
the real case, momery is much more.
Unlimit offHeap memory use cause RM killing the container
-
Key: SPARK-6056
URL: https://issues.apache.org/jira/browse/SPARK-6056
Project: Spark
Issue Type: Bug
Components: Shuffle, Spark Core
Affects Versions: 1.2.1
Reporter: SaintBacchus
No matter set the `preferDirectBufs` or limit the number of thread or not
,spark can not limit the use of offheap memory.
At line 269 of the class 'AbstractNioByteChannel' in netty-4.0.23.Final,
Netty had allocated a offheap memory buffer with the same size in heap.
So how many buffer you want to transfor, the same size offheap memory will be
allocated.
But once the allocated memory size reach the capacity of the overhead momery
set in yarn, this executor will be killed.
I wrote a simple code to test it:
{code:title=test.scala|borderStyle=solid}
import org.apache.spark.storage._
import org.apache.spark._
val bufferRdd = sc.makeRDD(0 to 10, 10).map(x=new
Array[Byte](10*1024*1024)).persist
bufferRdd.count
val part = bufferRdd.partitions(0)
val sparkEnv = SparkEnv.get
val blockMgr = sparkEnv.blockManager
def test = {
val blockOption = blockMgr.get(RDDBlockId(bufferRdd.id, part.index))
val resultIt =
blockOption.get.data.asInstanceOf[Iterator[Array[Byte]]]
val len = resultIt.map(_.length).sum
println(s[${Thread.currentThread.getId}] get block length = $len)
}
def test_driver(count:Int, parallel:Int)(f: = Unit) = {
val tpool = new scala.concurrent.forkjoin.ForkJoinPool(parallel)
val taskSupport = new
[jira] [Comment Edited] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container
[ https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339861#comment-14339861 ] SaintBacchus edited comment on SPARK-6056 at 2/27/15 8:13 AM: -- Hi,[~adav] [~lianhuiwang][~zzcclp] I had read your discuss at https://issues.apache.org/jira/browse/SPARK-2468. I meet the similiar problem again. No matter set the `preferDirectBufs` or limit the number of thread or not, spark can not limit the use of offheap memory. At line 269 of the class 'AbstractNioByteChannel' in netty-4.0.23.Final, Netty had allocated a offheap memory buffer with the same size in heap. So how many buffer you want to transfor, the same size offheap memory will be allocated. But once the allocated memory size reach the capacity of the overhead momery set in yarn, this executor will be killed. was (Author: carlmartin): Hi,[~adav] [~lianhuiwang][~zzcclp] I had read your discuss at https://issues.apache.org/jira/browse/SPARK-2468. I meet the similiar problem again. No matter set the `preferDirectBufs` or limit the number of thread or not can not limit the use of offheap memory. At line 269 of the class 'AbstractNioByteChannel' in netty-4.0.23.Final, Netty had allocated a offheap memory buffer with the same size in heap. So how many buffer you want to transfor, the same size offheap memory will be allocated. But once the allocated memory size reach the capacity of the overhead momery set in yarn, this executor will be killed. Unlimit offHeap memory use cause RM killing the container - Key: SPARK-6056 URL: https://issues.apache.org/jira/browse/SPARK-6056 Project: Spark Issue Type: Bug Components: Shuffle, Spark Core Affects Versions: 1.2.1 Reporter: SaintBacchus -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
