Re: Unable to use more than 1 executor for spark streaming application with YARN
How can I get more information regarding this exception? On Wed, Jun 17, 2015 at 1:17 AM, Saiph Kappa wrote: > Hi, > > I am running a simple spark streaming application on hadoop 2.7.0/YARN > (master: yarn-client) with 2 executors in different machines. However, > while the app is running, I can see on the app web UI (tab executors) that > only 1 executor keeps completing tasks over time, the other executor only > works and completes tasks for some seconds. From the logs I can see an > exception arising, though it is not clear what went wrong. > > Here is the yarn-nodemanager log: > « > 2015-06-17 00:29:50,967 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Starting resource-monitoring for container_1434391147618_0007_01_03 > 2015-06-17 00:29:50,977 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 286.5 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:29:53,991 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 463.7 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:29:57,009 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 465.7 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:30:00,024 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 467.6 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:30:03,032 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 474.0 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:30:06,041 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 480.2 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:30:09,053 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 540.9 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:30:12,068 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 550.9 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:30:15,075 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 551.1 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:30:18,090 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30553 for container-id > container_1434391147618_0007_01_03: 558.7 MB of 3 GB physical memory > used; 2.7 GB of 6.3 GB virtual memory used > 2015-06-17 00:30:20,157 WARN > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit > code from container container_1434391147618_0007_01_03 is : 1 > 2015-06-17 00:30:20,157 WARN > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1434391147618_0007_01_03 and exit code: 1 > ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) > at org.apache.hadoop.util.Shell.run(Shell.java:456) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecut
Unable to use more than 1 executor for spark streaming application with YARN
Hi, I am running a simple spark streaming application on hadoop 2.7.0/YARN (master: yarn-client) with 2 executors in different machines. However, while the app is running, I can see on the app web UI (tab executors) that only 1 executor keeps completing tasks over time, the other executor only works and completes tasks for some seconds. From the logs I can see an exception arising, though it is not clear what went wrong. Here is the yarn-nodemanager log: « 2015-06-17 00:29:50,967 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1434391147618_0007_01_03 2015-06-17 00:29:50,977 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 286.5 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:29:53,991 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 463.7 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:29:57,009 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 465.7 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:00,024 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 467.6 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:03,032 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 474.0 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:06,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 480.2 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:09,053 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 540.9 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:12,068 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 550.9 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:15,075 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 551.1 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:18,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_03: 558.7 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:20,157 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1434391147618_0007_01_03 is : 1 2015-06-17 00:30:20,157 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1434391147618_0007_01_03 and exit code: 1 ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanage
