Thanks for information this fixed the issue. Issue was in spark-master memory when I specify manually 1G for master. it start working
On 30 July 2015 at 14:26, Shao, Saisai <saisai.s...@intel.com> wrote: > You’d better also check the log of nodemanager, sometimes because your > memory usage exceeds the limit of Yarn container’s configuration. > > > > I’ve met similar problem before, here is the warning log in nodemanager: > > > > 2015-07-07 17:06:07,141 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Container [pid=17385,containerID=container_1436259427993_0001_02_000001] is > running beyond virtual memory limits. Current usage: 318.1 MB of 1 GB > physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing > container. > > > > The default pmem-vmem ratio is 2.1, but seems executor requires more vmem > when started, so nodemanager will kill it. If you met similar problem, you > could increase this configuration “yarn.nodemanager.vmem-pmem-ratio”. > > > > Thanks > > Jerry > > > > *From:* Jeff Zhang [mailto:zjf...@gmail.com] > *Sent:* Thursday, July 30, 2015 4:36 PM > *To:* Jeetendra Gangele > *Cc:* user > *Subject:* Re: Spark on YARN > > > > >> 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: > SIGTERM > > > > AM is killed somehow, may due to preemption. Does it always happen ? > Resource manager log would be helpful. > > > > > > > > On Thu, Jul 30, 2015 at 4:17 PM, Jeetendra Gangele <gangele...@gmail.com> > wrote: > > I can't see the application logs here. All the logs are going into > stderr. can anybody help here? > > > > On 30 July 2015 at 12:21, Jeetendra Gangele <gangele...@gmail.com> wrote: > > I am running below command this is default spark PI program but this is > not running all the log are going in stderr but at the terminal job is > succeeding .I guess there are con issue job it not at all launching > > > > /bin/spark-submit --class org.apache.spark.examples.SparkPi --master > yarn-cluster lib/spark-examples-1.4.1-hadoop2.6.0.jar 10 > > > > > > Complete log > > > > SLF4J: Class path contains multiple SLF4J bindings. > > SLF4J: Found binding in > [jar:file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/23/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: Found binding in > [jar:file:/opt/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > 15/07/30 12:13:31 INFO yarn.ApplicationMaster: Registered signal handlers for > [TERM, HUP, INT] > > 15/07/30 12:13:32 INFO yarn.ApplicationMaster: ApplicationAttemptId: > appattempt_1438090734187_0010_000001 > > 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop > > 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop > > 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(hadoop); users > with modify permissions: Set(hadoop) > > 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Starting the user application > in a separate Thread > > 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context > initialization > > 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context > initialization ... > > 15/07/30 12:13:33 INFO spark.SparkContext: Running Spark version 1.4.1 > > 15/07/30 12:13:33 WARN spark.SparkConf: > > SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411'). > > This is deprecated in Spark 1.0+. > > > > Please instead use: > > - ./spark-submit with conf/spark-defaults.conf to set defaults for an > application > > - ./spark-submit with --driver-java-options to set -X options for a driver > > - spark.executor.extraJavaOptions to set -X options for executors > > - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master > or worker) > > > > 15/07/30 12:13:33 WARN spark.SparkConf: Setting > 'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a > work-around. > > 15/07/30 12:13:33 WARN spark.SparkConf: Setting > 'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a > work-around. > > 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop > > 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop > > 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(hadoop); users > with modify permissions: Set(hadoop) > > 15/07/30 12:13:33 INFO slf4j.Slf4jLogger: Slf4jLogger started > > 15/07/30 12:13:33 INFO Remoting: Starting remoting > > 15/07/30 12:13:34 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriver@10.21.1.77:53411] > > 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'sparkDriver' > on port 53411. > > 15/07/30 12:13:34 INFO spark.SparkEnv: Registering MapOutputTracker > > 15/07/30 12:13:34 INFO spark.SparkEnv: Registering BlockManagerMaster > > 15/07/30 12:13:34 INFO storage.DiskBlockManager: Created local directory at > /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/blockmgr-2166bbd9-b1ed-41d1-bc95-92c6a7fbd36f > > 15/07/30 12:13:34 INFO storage.MemoryStore: MemoryStore started with capacity > 246.0 MB > > 15/07/30 12:13:34 INFO spark.HttpFileServer: HTTP File server directory is > /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c > > 15/07/30 12:13:34 INFO spark.HttpServer: Starting HTTP Server > > 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/07/30 12:13:34 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:52507 > > 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'HTTP file > server' on port 52507. > > 15/07/30 12:13:34 INFO spark.SparkEnv: Registering OutputCommitCoordinator > > 15/07/30 12:13:34 INFO ui.JettyUtils: Adding filter: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter > > 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/07/30 12:13:34 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:59596 > > 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'SparkUI' on > port 59596. > > 15/07/30 12:13:34 INFO ui.SparkUI: Started SparkUI at http://10.21.1.77:59596 > > 15/07/30 12:13:34 INFO cluster.YarnClusterScheduler: Created > YarnClusterScheduler > > 15/07/30 12:13:34 INFO util.Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43354. > > 15/07/30 12:13:34 INFO netty.NettyBlockTransferService: Server created on > 43354 > > 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Trying to register > BlockManager > > 15/07/30 12:13:34 INFO storage.BlockManagerMasterEndpoint: Registering block > manager 10.21.1.77:43354 with 246.0 MB RAM, BlockManagerId(driver, > 10.21.1.77, 43354) > > 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Registered BlockManager > > 15/07/30 12:13:34 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster registered as > AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#-603094240]) > > 15/07/30 12:13:34 INFO client.RMProxy: Connecting to ResourceManager at > hadoop-1/10.21.1.77:8030 > > 15/07/30 12:13:34 INFO yarn.YarnRMClient: Registering the ApplicationMaster > > 15/07/30 12:13:34 INFO yarn.YarnAllocator: Will request 2 executor > containers, each with 1 cores and 1408 MB memory including 384 MB overhead > > 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, > capability: <memory:1408, vCores:1>) > > 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, > capability: <memory:1408, vCores:1>) > > 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Started progress reporter > thread - sleep time : 5000 > > 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM > > 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, > exitCode: 0, (reason: Shutdown hook called before final status was reported.) > > 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Unregistering > ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before > final status was reported.) > > 15/07/30 12:13:35 INFO impl.AMRMClientImpl: Waiting for application to be > successfully unregistered. > > 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Deleting staging directory > .sparkStaging/application_1438090734187_0010 > > 15/07/30 12:13:35 INFO storage.DiskBlockManager: Shutdown hook called > > 15/07/30 12:13:35 INFO util.Utils: Shutdown hook called > > 15/07/30 12:13:35 INFO util.Utils: Deleting directory > /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/userFiles-337c9be5-569f-43ff-ba1f-ec24daab9ea5 > > 15/07/30 12:13:35 INFO util.Utils: Deleting directory > /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c > > > > > > > > -- > > Best Regards > > Jeff Zhang >
