I can't see the application logs here. All the logs are going into stderr. can anybody help here?
On 30 July 2015 at 12:21, Jeetendra Gangele <gangele...@gmail.com> wrote: > I am running below command this is default spark PI program but this is > not running all the log are going in stderr but at the terminal job is > succeeding .I guess there are con issue job it not at all launching > > /bin/spark-submit --class org.apache.spark.examples.SparkPi --master > yarn-cluster lib/spark-examples-1.4.1-hadoop2.6.0.jar 10 > > > Complete log > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/23/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 15/07/30 12:13:31 INFO yarn.ApplicationMaster: Registered signal handlers for > [TERM, HUP, INT] > 15/07/30 12:13:32 INFO yarn.ApplicationMaster: ApplicationAttemptId: > appattempt_1438090734187_0010_000001 > 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop > 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop > 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(hadoop); users > with modify permissions: Set(hadoop) > 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Starting the user application > in a separate Thread > 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context > initialization > 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context > initialization ... > 15/07/30 12:13:33 INFO spark.SparkContext: Running Spark version 1.4.1 > 15/07/30 12:13:33 WARN spark.SparkConf: > SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411'). > This is deprecated in Spark 1.0+. > > Please instead use: > - ./spark-submit with conf/spark-defaults.conf to set defaults for an > application > - ./spark-submit with --driver-java-options to set -X options for a driver > - spark.executor.extraJavaOptions to set -X options for executors > - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master > or worker) > > 15/07/30 12:13:33 WARN spark.SparkConf: Setting > 'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a > work-around. > 15/07/30 12:13:33 WARN spark.SparkConf: Setting > 'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a > work-around. > 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop > 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop > 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(hadoop); users > with modify permissions: Set(hadoop) > 15/07/30 12:13:33 INFO slf4j.Slf4jLogger: Slf4jLogger started > 15/07/30 12:13:33 INFO Remoting: Starting remoting > 15/07/30 12:13:34 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriver@10.21.1.77:53411] > 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'sparkDriver' > on port 53411. > 15/07/30 12:13:34 INFO spark.SparkEnv: Registering MapOutputTracker > 15/07/30 12:13:34 INFO spark.SparkEnv: Registering BlockManagerMaster > 15/07/30 12:13:34 INFO storage.DiskBlockManager: Created local directory at > /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/blockmgr-2166bbd9-b1ed-41d1-bc95-92c6a7fbd36f > 15/07/30 12:13:34 INFO storage.MemoryStore: MemoryStore started with capacity > 246.0 MB > 15/07/30 12:13:34 INFO spark.HttpFileServer: HTTP File server directory is > /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c > 15/07/30 12:13:34 INFO spark.HttpServer: Starting HTTP Server > 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT > 15/07/30 12:13:34 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:52507 > 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'HTTP file > server' on port 52507. > 15/07/30 12:13:34 INFO spark.SparkEnv: Registering OutputCommitCoordinator > 15/07/30 12:13:34 INFO ui.JettyUtils: Adding filter: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter > 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT > 15/07/30 12:13:34 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:59596 > 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'SparkUI' on > port 59596. > 15/07/30 12:13:34 INFO ui.SparkUI: Started SparkUI at http://10.21.1.77:59596 > 15/07/30 12:13:34 INFO cluster.YarnClusterScheduler: Created > YarnClusterScheduler > 15/07/30 12:13:34 INFO util.Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43354. > 15/07/30 12:13:34 INFO netty.NettyBlockTransferService: Server created on > 43354 > 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Trying to register > BlockManager > 15/07/30 12:13:34 INFO storage.BlockManagerMasterEndpoint: Registering block > manager 10.21.1.77:43354 with 246.0 MB RAM, BlockManagerId(driver, > 10.21.1.77, 43354) > 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Registered BlockManager > 15/07/30 12:13:34 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster registered as > AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#-603094240]) > 15/07/30 12:13:34 INFO client.RMProxy: Connecting to ResourceManager at > hadoop-1/10.21.1.77:8030 > 15/07/30 12:13:34 INFO yarn.YarnRMClient: Registering the ApplicationMaster > 15/07/30 12:13:34 INFO yarn.YarnAllocator: Will request 2 executor > containers, each with 1 cores and 1408 MB memory including 384 MB overhead > 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, > capability: <memory:1408, vCores:1>) > 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, > capability: <memory:1408, vCores:1>) > 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Started progress reporter > thread - sleep time : 5000 > 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM > 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, > exitCode: 0, (reason: Shutdown hook called before final status was reported.) > 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Unregistering > ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before > final status was reported.) > 15/07/30 12:13:35 INFO impl.AMRMClientImpl: Waiting for application to be > successfully unregistered. > 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Deleting staging directory > .sparkStaging/application_1438090734187_0010 > 15/07/30 12:13:35 INFO storage.DiskBlockManager: Shutdown hook called > 15/07/30 12:13:35 INFO util.Utils: Shutdown hook called > 15/07/30 12:13:35 INFO util.Utils: Deleting directory > /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/userFiles-337c9be5-569f-43ff-ba1f-ec24daab9ea5 > 15/07/30 12:13:35 INFO util.Utils: Deleting directory > /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c > >
