vinodkc commented on PR #41746:
URL: https://github.com/apache/spark/pull/41746#issuecomment-1630101969
### Manual test steps where the feature is tested at least with 3 log
levels. With new executor creation after sc.setLogLevel() call.
1) initial one: WARN (this can be from the log4j config)
`rootLogger.level = warn`
```
> ./bin/spark-shell --master yarn --conf
spark.executor.sync.log.level=true
```
2) first update to INFO (sc.setLogLevel("INFO"))
**a) In the driver**
```
scala> sc.setLogLevel("INFO")
23/07/10 20:59:42 INFO [dispatcher-CoarseGrainedScheduler]
YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to refresh the
log level to INFO
scala> sc.parallelize(1 to 10).count
23/07/10 21:00:07 INFO [main] SparkContext: Starting job: count at
<console>:24
23/07/10 21:00:07 INFO [dag-scheduler-event-loop] DAGScheduler: Got job 0
(count at <console>:24) with 2 output partitions
23/07/10 21:00:07 INFO [dag-scheduler-event-loop] DAGScheduler: Final stage:
ResultStage 0 (count at <console>:24)
```
**b) In the executor**
```
Container: container_1689037381071_0016_01_000002 on 10.0.0.223_50550
LogAggregationType: AGGREGATED
=====================================================================
LogType:stderr
LogLastModifiedTime:Mon Jul 10 21:01:40 -0700 2023
LogLength:6827
LogContents:
23/07/10 20:59:37 WARN [main] NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
23/07/10 21:00:07 INFO [dispatcher-Executor]
YarnCoarseGrainedExecutorBackend: Got assigned task 0
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] Executor: Running task 0.0 in stage 0.0 (TID 0)
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] TorrentBroadcast: Started reading broadcast variable 0 with 1
pieces (est
imated total size 4.0 MiB)
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] TransportClientFactory: Successfully created connection to
/10.0.0.223:55
625 after 1 ms (0 ms spent in bootstraps)
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] MemoryStore: Block broadcast_0_piece0 stored as bytes in memory
(estimate
d size 1782.0 B, free 93.3 MiB)
```
3) second update to DEBUG (sc.setLogLevel("DEBUG"),to check whether the
config update really works the 2nd time too)
```
sc.setLogLevel("DEBUG")
sc.parallelize(1 to 10).count
```
Driver printed DEBUG logs
In the executor [container_1689037381071_0016_01_000002] log I could see
DEBUG log.
_**This confirms that log Level was propagated to Executor**_
```
Container: container_1689037381071_0016_01_000002 on 10.0.0.223_50550
LogAggregationType: AGGREGATED
=====================================================================
LogType:stderr
LogLastModifiedTime:Mon Jul 10 21:01:40 -0700 2023
LogLength:6827
LogContents:
23/07/10 20:59:37 WARN [main] NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
23/07/10 21:00:07 INFO [dispatcher-Executor]
YarnCoarseGrainedExecutorBackend: Got assigned task 0
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] Executor: Running task 0.0 in stage 0.0 (TID 0)
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] TorrentBroadcast: Started reading broadcast variable 0 with 1
pieces (estimated total size 4.0 MiB)
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] TransportClientFactory: Successfully created connection to /****:55
625 after 1 ms (0 ms spent in bootstraps)
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] MemoryStore: Block broadcast_0_piece0 stored as bytes in memory
(estimate
d size 1782.0 B, free 93.3 MiB)
23/07/10 21:00:07 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] TorrentBroadcast: Reading broadcast variable 0 took 128 ms
23/07/10 21:00:08 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] MemoryStore: Block broadcast_0 stored as values in memory
(estimated size
2.8 KiB, free 93.3 MiB)
23/07/10 21:00:08 INFO [Executor task launch worker for task 0.0 in stage
0.0 (TID 0)] Executor: Finished task 0.0 in stage 0.0 (TID 0). 973 bytes result
sent t
o driver
23/07/10 21:00:08 INFO [dispatcher-Executor]
YarnCoarseGrainedExecutorBackend: Got assigned task 1
23/07/10 21:00:08 INFO [Executor task launch worker for task 1.0 in stage
0.0 (TID 1)] Executor: Running task 1.0 in stage 0.0 (TID 1)
23/07/10 21:00:08 INFO [Executor task launch worker for task 1.0 in stage
0.0 (TID 1)] Executor: Finished task 1.0 in stage 0.0 (TID 1). 887 bytes result
sent t
o driver
23/07/10 21:00:40 INFO [dispatcher-Executor]
YarnCoarseGrainedExecutorBackend: Got assigned task 2
23/07/10 21:00:40 INFO [Executor task launch worker for task 0.0 in stage
1.0 (TID 2)] Executor: Running task 0.0 in stage 1.0 (TID 2)
23/07/10 21:00:40 DEBUG [Executor task launch worker for task 0.0 in stage
1.0 (TID 2)] Executor: task 0.0 in stage 1.0 (TID 2)'s epoch is 0
```
4) Scale down the executor by killing container
container_1689037381071_0016_01_000002. (Instead of scaling up, did scale down
to zero executors).In that scenario also new executors will be created to run
the next jobs
Killed current executor container_1689037381071_0016_01_000002
```
>yarn container -signal container_1689037381071_0016_01_000002
GRACEFUL_SHUTDOWN
```
On spark-shell run a new job
```
sc.parallelize(1 to 10).count
```
AM created a new Container: `container_1689037381071_0016_01_000003`
In the **new executor** [container_1689037381071_0016_01_000003] log, we
can see DEBUG log. _**This confirms that newly create Executor also pulled the
latest loglevel from Driver**_
```
Container: container_1689037381071_0016_01_000003 on 10.0.0.223_50550
LogAggregationType: AGGREGATED
=====================================================================
LogType:stderr
LogLastModifiedTime:Mon Jul 10 21:01:40 -0700 2023
LogLength:10269
LogContents:
23/07/10 21:01:12 WARN [main] NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
23/07/10 21:01:13 INFO [main] SecurityManager: Changing view acls to: vkc
23/07/10 21:01:13 INFO [main] SecurityManager: Changing modify acls to: vkc
23/07/10 21:01:13 INFO [main] SecurityManager: Changing view acls groups to:
23/07/10 21:01:13 INFO [main] SecurityManager: Changing modify acls groups
to:
.......
23/07/10 21:01:13 DEBUG [main] DiskBlockManager: Adding shutdown hook
23/07/10 21:01:13 DEBUG [main] ShutdownHookManager: Adding shutdown hook
23/07/10 21:01:13 INFO [main] MemoryStore: MemoryStore started with capacity
93.3 MiB
23/07/10 21:01:13 INFO [dispatcher-Executor]
YarnCoarseGrainedExecutorBackend: Connecting to driver:
spark://CoarseGrainedScheduler@******:55599
23/07/10 21:01:13 DEBUG [dispatcher-Executor]
YarnCoarseGrainedExecutorBackend: Resource profile id is: 0
23/07/10 21:01:13 INFO [dispatcher-Executor] ResourceUtils:
==============================================================
23/07/10 21:01:13 INFO [dispatcher-Executor] ResourceUtils: No custom
resources configured for spark.executor.
23/07/10 21:01:13 INFO [dispatcher-Executor] ResourceUtils:
==============================================================
23/07/10 21:01:13 DEBUG [rpc-client-3-1] YarnContainerInfoHelper: Base URL
for logs:
http://10.0.0.223:8042/node/containerlogs/container_1689037381071_0016_01_000003/***
23/07/10 21:01:13 INFO [dispatcher-Executor]
YarnCoarseGrainedExecutorBackend: Successfully registered with driver
23/07/10 21:01:13 INFO [dispatcher-Executor] Executor: Starting executor ID
2 on host *****
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
