Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21784
OK, but you can set sc.setLogLevel in the shell. For instance, if I run
`spark.range(10).show()` with INFO, I got something like this:
```
scala> 18/07/18 07:58:47 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
spark.r18/07/18 07:58:48 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
ange(10)18/07/18 07:58:49 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
.show()18/07/18 07:58:50 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:51 INFO internal.SharedState: loading hive config file:
file:/home/spark/spark/conf/hive-site.xml
18/07/18 07:58:51 INFO internal.SharedState: Setting
hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir
('file:/home/spark/spark/spark-warehouse').
18/07/18 07:58:51 INFO internal.SharedState: Warehouse path is
'file:/home/spark/spark/spark-warehouse'.
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@7a68818c{/SQL,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@5f745970{/SQL/json,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@2afd8972{/SQL/execution,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to
/SQL/execution/json.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@5784f6b9{/SQL/execution/json,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@6ccf06f1{/static/sql,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:52 INFO state.StateStoreCoordinatorRef: Registered
StateStoreCoordinator endpoint
18/07/18 07:58:52 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 254.142542
ms
18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 65.397101 ms
18/07/18 07:58:53 INFO spark.SparkContext: Starting job: show at
<console>:24
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Got job 0 (show at
<console>:24) with 1 output partitions
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 0
(show at <console>:24)
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Parents of final stage:
List()
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Missing parents: List()
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Submitting ResultStage 0
(MapPartitionsRDD[4] at show at <console>:24), which has no missing parents
18/07/18 07:58:53 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0 stored as
values in memory (estimated size 8.1 KB, free 408.9 MB)
18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0_piece0 stored
as bytes in memory (estimated size 4.0 KB, free 408.9 MB)
18/07/18 07:58:53 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
in memory on ctr-e138-1518143905142-411342-01-000002.hwx.site:41073 (size: 4.0
KB, free: 408.9 MB)
18/07/18 07:58:53 INFO spark.SparkContext: Created broadcast 0 from
broadcast at DAGScheduler.scala:1039
18/07/18 07:58:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks
from ResultStage 0 (MapPartitionsRDD[4] at show at <console>:24) (first 15
tasks are for partitions Vector(0))
18/07/18 07:58:54 INFO cluster.YarnScheduler: Adding task set 0.0 with 1
tasks
18/07/18 07:58:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage
0.0 (TID 0, ctr-e138-1518143905142-411342-01-000008.hwx.site, executor 2,
partition 0, PROCESS_LOCAL, 7864 bytes)
[Stage 0:> (0 + 1)
/ 1]18/07/18 07:58:54 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
in memory on ctr-e138-1518143905142-411342-01-000008.hwx.site:37671 (size: 4.0
KB, free: 366.3 MB)
18/07/18 07:58:54 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage
0.0 (TID 0) in 1115 ms on ctr-e138-1518143905142-411342-01-000008.hwx.site
(executor 2) (1/1)
18/07/18 07:58:55 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose
tasks have all completed, from pool
18/07/18 07:58:55 INFO scheduler.DAGScheduler: ResultStage 0 (show at
<console>:24) finished in 1.609 s
18/07/18 07:58:55 INFO scheduler.DAGScheduler: Job 0 finished: show at
<console>:24, took 1.673348 s
18/07/18 07:58:55 INFO spark.SparkContext: Starting job: show at
<console>:24
18/07/18 07:58:55 INFO scheduler.DAGScheduler: Got job 1 (show at
<console>:24) with 1 output partitions
18/07/18 07:58:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 1
(show at <console>:24)
18/07/18 07:58:55 INFO scheduler.DAGScheduler: Parents of final stage:
List()
18/07/18 07:58:55 INFO scheduler.DAGScheduler: Missing parents: List()
18/07/18 07:58:55 INFO scheduler.DAGScheduler: Submitting ResultStage 1
(MapPartitionsRDD[4] at show at <console>:24), which has no missing parents
18/07/18 07:58:55 INFO memory.MemoryStore: Block broadcast_1 stored as
values in memory (estimated size 8.1 KB, free 408.9 MB)
18/07/18 07:58:55 INFO memory.MemoryStore: Block broadcast_1_piece0 stored
as bytes in memory (estimated size 4.0 KB, free 408.9 MB)
18/07/18 07:58:55 INFO storage.BlockManagerInfo: Added broadcast_1_piece0
in memory on ctr-e138-1518143905142-411342-01-000002.hwx.site:41073 (size: 4.0
KB, free: 408.9 MB)
18/07/18 07:58:55 INFO spark.SparkContext: Created broadcast 1 from
broadcast at DAGScheduler.scala:1039
18/07/18 07:58:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks
from ResultStage 1 (MapPartitionsRDD[4] at show at <console>:24) (first 15
tasks are for partitions Vector(1))
18/07/18 07:58:55 INFO cluster.YarnScheduler: Adding task set 1.0 with 1
tasks
18/07/18 07:58:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage
1.0 (TID 1, ctr-e138-1518143905142-411342-01-000007.hwx.site, executor 1,
partition 1, PROCESS_LOCAL, 7864 bytes)
18/07/18 07:58:55 INFO storage.BlockManagerInfo: Added broadcast_1_piece0
in memory on ctr-e138-1518143905142-411342-01-000007.hwx.site:33073 (size: 4.0
KB, free: 366.3 MB)
18/07/18 07:58:55 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
[Stage 1:> (0 + 1)
/ 1]18/07/18 07:58:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage
1.0 (TID 1) in 948 ms on ctr-e138-1518143905142-411342-01-000007.hwx.site
(executor 1) (1/1)
18/07/18 07:58:56 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose
tasks have all completed, from pool
18/07/18 07:58:56 INFO scheduler.DAGScheduler: ResultStage 1 (show at
<console>:24) finished in 0.956 s
18/07/18 07:58:56 INFO scheduler.DAGScheduler: Job 1 finished: show at
<console>:24, took 0.959303 s
+---+
| id|
+---+
| 0|
| 1|
| 2|
| 3|
| 4|
| 5|
| 6|
| 7|
| 8|
| 9|
+---+
```
Considering it's info level, I was thinking it's not super noisy. I am okay
with disabling though. I got your point. Let me defer this to @vanzin and
@jerryshao.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
