[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 Also, mind adding `[SPARK-24873]` in the PR title since the JIRA happened to be open anyway. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 @wangyum, mind adding Closes #21784 here? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 @vanzin WDYT about this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 OK, but you can set sc.setLogLevel in the shell. For instance, if I run `spark.range(10).show()` with INFO, I got something like this: ``` scala> 18/07/18 07:58:47 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) spark.r18/07/18 07:58:48 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) ange(10)18/07/18 07:58:49 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) .show()18/07/18 07:58:50 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:51 INFO internal.SharedState: loading hive config file: file:/home/spark/spark/conf/hive-site.xml 18/07/18 07:58:51 INFO internal.SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/spark/spark/spark-warehouse'). 18/07/18 07:58:51 INFO internal.SharedState: Warehouse path is 'file:/home/spark/spark/spark-warehouse'. 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7a68818c{/SQL,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5f745970{/SQL/json,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2afd8972{/SQL/execution,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5784f6b9{/SQL/execution/json,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql. 18/07/18 07:58:51 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@6ccf06f1{/static/sql,null,AVAILABLE,@Spark} 18/07/18 07:58:51 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:52 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint 18/07/18 07:58:52 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 254.142542 ms 18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 65.397101 ms 18/07/18 07:58:53 INFO spark.SparkContext: Starting job: show at :24 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Got job 0 (show at :24) with 1 output partitions 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (show at :24) 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Parents of final stage: List() 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Missing parents: List() 18/07/18 07:58:53 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4] at show at :24), which has no missing parents 18/07/18 07:58:53 INFO yarn.Client: Application report for application_1531383843352_0013 (state: RUNNING) 18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 8.1 KB, free 408.9 MB) 18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 4.0 KB, free 408.9 MB) 18/07/18 07:58:53 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ctr-e138-1518143905142-411342-01-02.hwx.site:41073 (size: 4.0 KB, free: 408.9 MB) 18/07/18 07:58:53 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1039 18/07/18 07:58:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at show at :24) (first 15 tasks are for partitions Vector(0)) 18/07/18 07:58:54 INFO cluster.YarnScheduler: Adding task set 0.0 with 1 tasks 18/07/18 07:58:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, ctr-e138-1518143905142-411342-01-08.hwx.site, executor 2, partition 0, PROCESS_LOCAL, 7864 bytes) [Stage 0:> (0 + 1) / 1]18/07/18 07:58:54 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ctr-e138-1518143905142-411342-01-08.hwx.site:37671 (size: 4.0 KB, free: 366.3 MB) 18/07/18 07:58:54 INFO yarn.Client: Application report for appl
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21784 It's noisy when type something: ![spark-24128](https://user-images.githubusercontent.com/5399861/42857022-8afee254-8a7a-11e8-8ee0-47a19af99fa7.gif) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 Hm, yea. I don't think find this super noisy though to be honest. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21784 cc @vanzin, @jerryshao --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21784 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93111/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21784 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21784 **[Test build #93111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93111/testReport)** for PR 21784 at commit [`d6563ec`](https://github.com/apache/spark/commit/d6563ecd4708088b5e87fa62629c3c5048c91b6c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21784 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21784 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1012/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21784 **[Test build #93111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93111/testReport)** for PR 21784 at commit [`d6563ec`](https://github.com/apache/spark/commit/d6563ecd4708088b5e87fa62629c3c5048c91b6c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org