[ https://issues.apache.org/jira/browse/SPARK-11101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-11101. ------------------------------- Resolution: Invalid If it's a question, you should ask as u...@spark.apache.org, not make a JIRA. It may have nothing to do with your process, though you do need to verify how much it uses. There is little margin in the YARN allocation for off-heap memory, so you probably have to increase this value, yes. > pipe() operation OOM > -------------------- > > Key: SPARK-11101 > URL: https://issues.apache.org/jira/browse/SPARK-11101 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.4.1 > Environment: spark on yarn > Reporter: hotdog > Original Estimate: 72h > Remaining Estimate: 72h > > when using pipe() operation with large data(10TB), the pipe() operation > always OOM. > I use pipe() to calling a external c++ process. I'm sure the c++ program only > use little memory(about 1MB). > my parameters: > executor-memory 16g > executor-cores 4 > num-executors 400 > "spark.yarn.executor.memoryOverhead", "8192" > partition number: 60000 > does pipe() operation use many off-heap memory? > the log is : > killed by YARN for exceeding memory limits. 24.4 GB of 24 GB physical memory > used. Consider boosting spark.yarn.executor.memoryOverhead. > should I continue boosting spark.yarn.executor.memoryOverhead? Or there are > some bugs in the pipe() operation? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org