Github user piaozhexiu commented on the pull request:
https://github.com/apache/spark/pull/5362#issuecomment-89728738
@srowen I'd like to turn down pretty much every INFO message from YARN
client except the AM url. (See below.) As can be seen, none of these is useful
for end users except for the AM url. Unfortunately, I can't selectively turn
down other messages since they're all in the same package.
How about if I print the tracking url to stdout and leave the INFO log as
is? Then, I can turn off INFO in YARN client.
-----
15/04/05 06:36:29 INFO Configuration.deprecation:
mapred.input.dir.recursive is deprecated. Instead, use
mapreduce.input.fileinputformat.input.dir.recursive
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.max.split.size is
deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.min.split.size is
deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
15/04/05 06:36:29 INFO Configuration.deprecation:
mapred.min.split.size.per.rack is deprecated. Instead, use
mapreduce.input.fileinputformat.split.minsize.per.rack
15/04/05 06:36:29 INFO Configuration.deprecation:
mapred.min.split.size.per.node is deprecated. Instead, use
mapreduce.input.fileinputformat.split.minsize.per.node
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.reduce.tasks is
deprecated. Instead, use mapreduce.job.reduces
15/04/05 06:36:29 INFO Configuration.deprecation:
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
15/04/05 06:36:29 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/04/05 06:36:29 INFO spark.SparkContext: Running Spark version 1.3.0
15/04/05 06:36:30 INFO spark.SecurityManager: Changing view acls to:
cheolsoop
15/04/05 06:36:30 INFO spark.SecurityManager: Changing modify acls to:
cheolsoop
15/04/05 06:36:30 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(cheolsoop); users with modify permissions: Set(cheolsoop)
15/04/05 06:36:30 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/04/05 06:36:30 INFO Remoting: Starting remoting
15/04/05 06:36:30 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://[email protected]:64877]
15/04/05 06:36:30 INFO util.Utils: Successfully started service
'sparkDriver' on port 64877.
15/04/05 06:36:30 INFO spark.SparkEnv: Registering MapOutputTracker
15/04/05 06:36:30 INFO spark.SparkEnv: Registering BlockManagerMaster
15/04/05 06:36:30 INFO storage.DiskBlockManager: Created local directory at
/mnt/spark_tmp/spark-e95cf4af-ec65-469a-ad1a-827d1149eeab/blockmgr-ec922fd7-58c5-497a-a952-247fcb3ab779
15/04/05 06:36:30 INFO storage.MemoryStore: MemoryStore started with
capacity 265.4 MB
15/04/05 06:36:31 INFO spark.HttpFileServer: HTTP File server directory is
/mnt/spark_tmp/spark-84f23ed2-ebf3-4022-93e1-fbb31325ab3f/httpd-128e4efa-c666-4311-b1ee-c0868eaca4bc
15/04/05 06:36:31 INFO spark.HttpServer: Starting HTTP Server
15/04/05 06:36:31 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/04/05 06:36:31 INFO server.AbstractConnector: Started
[email protected]:36543
15/04/05 06:36:31 INFO util.Utils: Successfully started service 'HTTP file
server' on port 36543.
15/04/05 06:36:31 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/04/05 06:36:31 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/04/05 06:36:31 INFO server.AbstractConnector: Started
[email protected]:47936
15/04/05 06:36:31 INFO util.Utils: Successfully started service 'SparkUI'
on port 47936.
15/04/05 06:36:31 INFO ui.SparkUI: Started SparkUI at
http://ip-10-99-146-254.ec2.internal:47936
15/04/05 06:36:31 INFO client.RMProxy: Connecting to ResourceManager at
/10.171.119.231:9022
15/04/05 06:36:31 INFO yarn.Client: Requesting a new application from
cluster with 1300 NodeManagers
15/04/05 06:36:31 INFO yarn.Client: Verifying our application has not
requested more than the maximum memory capability of the cluster (10240 MB per
container)
15/04/05 06:36:31 INFO yarn.Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
15/04/05 06:36:31 INFO yarn.Client: Setting up container launch context for
our AM
15/04/05 06:36:31 INFO yarn.Client: Preparing resources for our AM container
15/04/05 06:36:32 INFO yarn.Client: Uploading resource
file:/mnt/tmp/bdp-clients/cheolsoop/20150405_063623.027801.prodsparkshell13/jars/spark-1.3.0/lib/spark-assembly-1.3.1-SNAPSHOT-hadoop2.4.0.jar
->
hdfs://10.171.119.231:9000/user/cheolsoop/.sparkStaging/application_1426271585556_249126/spark-assembly-1.3.1-SNAPSHOT-hadoop2.4.0.jar
15/04/05 06:36:35 INFO yarn.Client: Setting up the launch environment for
our AM container
15/04/05 06:36:35 INFO spark.SecurityManager: Changing view acls to:
cheolsoop
15/04/05 06:36:35 INFO spark.SecurityManager: Changing modify acls to:
cheolsoop
15/04/05 06:36:35 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(cheolsoop); users with modify permissions: Set(cheolsoop)
15/04/05 06:36:35 INFO yarn.Client: Submitting application 249126 to
ResourceManager
15/04/05 06:36:35 INFO impl.YarnClientImpl: Submitted application
application_1426271585556_249126
15/04/05 06:36:36 INFO yarn.Client: Application report for
application_1426271585556_249126 (state: ACCEPTED)
15/04/05 06:36:36 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.adhoc.cheolsoop
start time: 1428215795227
final status: UNDEFINED
tracking URL:
http://10.171.119.231:9046/proxy/application_1426271585556_249126/
user: cheolsoop
15/04/05 06:36:37 INFO yarn.Client: Application report for
application_1426271585556_249126 (state: ACCEPTED)
-----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
