[
https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343317#comment-16343317
]
Bang Xiao commented on SPARK-23252:
-----------------------------------
[~srowen] it seems the job waits for the results of those tasks that have
failed， but it will never get those results since the failed tasks have not be
resubmited.
When NodeManager and CoarseGrainedExecutorBackend processes killed
simultaneously, the following log appears on the driver end：
{code:java}
18/01/29 14:32:35 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling
executor 24.
18/01/29 14:32:35 INFO DAGScheduler: Executor lost: 24 (epoch 1)
18/01/29 14:32:35 ERROR TransportClient: Failed to send RPC 8557067791977911361
to /10.142.103.168:14733: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/01/29 14:32:35 ERROR TransportClient: Failed to send RPC 7460664886971675621
to /10.142.103.168:14751: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/01/29 14:32:35 ERROR TransportClient: Failed to send RPC 5802441956021450458
to /10.142.103.168:14750: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/01/29 14:32:35 ERROR TransportClient: Failed to send RPC 9203205043102726551
to /10.142.103.168:14739: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/01/29 14:32:35 ERROR TransportClient: Failed to send RPC 5217847872442409416
to /10.142.103.168:14744: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/01/29 14:32:35 INFO BlockManagerMasterEndpoint: Trying to remove executor 24
from BlockManagerMaster.
18/01/29 14:32:35 INFO BlockManagerMasterEndpoint: Removing block manager
BlockManagerId(24, rsync.slave06.jupiter.zw.ted, 46509, None){code}
> When NodeManager and CoarseGrainedExecutorBackend processes are killed, the
> job will be blocked
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-23252
> URL: https://issues.apache.org/jira/browse/SPARK-23252
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: Bang Xiao
> Priority: Major
>
> This happens when 'spark.dynamicAllocation.enabled' is set to be 'true'. We
> use Yarn as our resource manager.
> 1，spark-submit "JavaWordCount" application in yarn-client mode
> 2, Kill NodeManager and CoarseGrainedExecutorBackend processes in one node
> when the job is in stage 0
> if we just kill all CoarseGrainedExecutorBackend in that node, TaskSetManager
> will pending the failure task to resubmit. but if the NodeManager and
> CoarseGrainedExecutorBackend processes killed simultaneously，the whole job
> will be blocked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
