Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36783358
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -110,6 +114,7 @@ private[spark] abstract class YarnSchedulerBackend(
addWebUIFilter(filterName, filterParams, proxyBase)
case RemoveExecutor(executorId, reason) =>
+ logWarning(reason)
--- End diff --
This isn't actually. That's really the line that shows us that the
Container was exited due to YARN killing the container. If you go debug, you
can see all the Akka communication and find it out that way, but there's
nothing info or harsher that logs this.
I thought about warning being a little too much here but the only WARN
messages being logged here, as far as I can tell are legitimate kills by YARN
(including a snippet below) so unless you disagree, I am inclined to keep it at
WARN.
````
Marks-MacBook-Pro-2:new mgrover$ grep -ri "WARN
YarnSchedulerBackend$YarnSchedulerEndpoint" c.out
c.out:15/08/09 08:24:39 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Container killed by YARN for exceeding memory limits. 1.4 GB of 1.4 GB physical
memory used. Consider boosting spark.yarn.executor.memoryOverhead.
c.out:15/08/09 08:24:39 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Yarn deallocated the executor 1 (container
container_1438872994881_0029_01_000002). Reason: Container
[pid=11659,containerID=container_1438872994881_0029_01_000002] is running
beyond physical memory limits. Current usage: 1.4 GB of 1.4 GB physical memory
used; 2.5 GB of 2.9 GB virtual memory used. Killing container.
c.out:15/08/09 08:24:42 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Container killed by YARN for exceeding memory limits. 1.6 GB of 1.4 GB physical
memory used. Consider boosting spark.yarn.executor.memoryOverhead.
c.out:15/08/09 08:24:42 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Yarn deallocated the executor 2 (container
container_1438872994881_0029_01_000003). Reason: Container
[pid=29045,containerID=container_1438872994881_0029_01_000003] is running
beyond physical memory limits. Current usage: 1.6 GB of 1.4 GB physical memory
used; 2.6 GB of 2.9 GB virtual memory used. Killing container.
c.out:15/08/09 08:24:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Container killed by YARN for exceeding memory limits. 1.6 GB of 1.4 GB physical
memory used. Consider boosting spark.yarn.executor.memoryOverhead.
c.out:15/08/09 08:24:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Yarn deallocated the executor 4 (container
container_1438872994881_0029_01_000005). Reason: Container
[pid=29121,containerID=container_1438872994881_0029_01_000005] is running
beyond physical memory limits. Current usage: 1.6 GB of 1.4 GB physical memory
used; 2.7 GB of 2.9 GB virtual memory used. Killing container.
````
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
