[
https://issues.apache.org/jira/browse/SPARK-22424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235379#comment-16235379
]
chengning edited comment on SPARK-22424 at 11/2/17 8:25 AM:
------------------------------------------------------------
I have another picture shows clearly
!1.png|thumbnail!
executor
17/11/01 16:40:55 INFO executor.CoarseGrainedExecutorBackend: Got assigned task
640603
17/11/01 16:40:55 INFO executor.Executor: Running task 3.0 in stage 8218.0 (TID
640603)
17/11/01 16:40:55 INFO spark.MapOutputTrackerWorker: Updating epoch to 2319 and
clearing cache
17/11/01 16:40:55 INFO broadcast.TorrentBroadcast: Started reading broadcast
variable 8218
17/11/01 16:40:55 INFO memory.MemoryStore: Block broadcast_8218_piece0 stored
as bytes in memory (estimated size 15.2 KB, free 2.2 GB)
17/11/01 16:40:55 INFO broadcast.TorrentBroadcast: Reading broadcast variable
8218 took 6 ms
17/11/01 16:40:55 INFO memory.MemoryStore: Block broadcast_8218 stored as
values in memory (estimated size 31.5 KB, free 2.2 GB)
17/11/01 16:40:55 INFO spark.MapOutputTrackerWorker: Don't have map outputs for
shuffle 2318, fetching them
17/11/01 16:40:55 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker
endpoint = NettyRpcEndpointRef(spark://[email protected]:33084)
17/11/01 16:40:55 INFO spark.MapOutputTrackerWorker: Got the output locations
17/11/01 16:40:55 INFO storage.ShuffleBlockFetcherIterator: Getting 28
non-empty blocks out of 30 blocks
17/11/01 16:40:55 INFO storage.ShuffleBlockFetcherIterator: Started 27 remote
fetches in 3 ms
17/11/01 16:40:55 INFO codegen.CodeGenerator: Code generated in 21.652093 ms
17/11/01 16:40:55 INFO executor.Executor: Finished task 3.0 in stage 8218.0
(TID 640603). 3554 bytes result sent to
driver
17/11/01 16:40:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage
8218.0 (TID 640603, Letv6CU621YYPS, executor 12, partition 3, PROCESS_LOCAL,
6324 bytes)
17/11/01 16:40:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage
8218.0 (TID 640603) in 167 ms on Letv6CU621YYPS (executor 12) (16/200)
17/11/01 16:40:55 ERROR scheduler.LiveListenerBus: Dropping SparkListenerEvent
because no remaining room in event queue. This likely means one of the
SparkListeners is too slow and cannot keep up with the rate at which tasks are
being started by the scheduler.
was (Author: chengning):
I have another picture shows clearly
!1.png|thumbnail!
executor
17/11/01 16:40:55 INFO executor.CoarseGrainedExecutorBackend: Got assigned task
640603
17/11/01 16:40:55 INFO executor.Executor: Running task 3.0 in stage 8218.0 (TID
640603)
17/11/01 16:40:55 INFO spark.MapOutputTrackerWorker: Updating epoch to 2319 and
clearing cache
17/11/01 16:40:55 INFO broadcast.TorrentBroadcast: Started reading broadcast
variable 8218
17/11/01 16:40:55 INFO memory.MemoryStore: Block broadcast_8218_piece0 stored
as bytes in memory (estimated size 15.2 KB, free 2.2 GB)
17/11/01 16:40:55 INFO broadcast.TorrentBroadcast: Reading broadcast variable
8218 took 6 ms
17/11/01 16:40:55 INFO memory.MemoryStore: Block broadcast_8218 stored as
values in memory (estimated size 31.5 KB, free 2.2 GB)
17/11/01 16:40:55 INFO spark.MapOutputTrackerWorker: Don't have map outputs for
shuffle 2318, fetching them
17/11/01 16:40:55 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker
endpoint = NettyRpcEndpointRef(spark://[email protected]:33084)
17/11/01 16:40:55 INFO spark.MapOutputTrackerWorker: Got the output locations
17/11/01 16:40:55 INFO storage.ShuffleBlockFetcherIterator: Getting 28
non-empty blocks out of 30 blocks
17/11/01 16:40:55 INFO storage.ShuffleBlockFetcherIterator: Started 27 remote
fetches in 3 ms
17/11/01 16:40:55 INFO codegen.CodeGenerator: Code generated in 21.652093 ms
17/11/01 16:40:55 INFO executor.Executor: Finished task 3.0 in stage 8218.0
(TID 640603). 3554 bytes result sent to
driver
17/11/01 16:40:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage
8218.0 (TID 640603, Letv6CU621YYPS, executor 12, partition 3, PROCESS_LOCAL,
6324 bytes)
17/11/01 16:40:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage
8218.0 (TID 640603) in 167 ms on Letv6CU621YYPS (executor 12) (16/200)
17/11/01 16:40:55 ERROR scheduler.LiveListenerBus: Dropping SparkListenerEvent
because no remaining room in event queue. This likely means one of the
SparkListeners is too slow and cannot keep up with the rate at which tasks are
being started by the scheduler.
> Task not finished for a long time in monitor UI, but I found it finished in
> logs
> --------------------------------------------------------------------------------
>
> Key: SPARK-22424
> URL: https://issues.apache.org/jira/browse/SPARK-22424
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: chengning
> Priority: Blocking
> Attachments: 1.png, C33oL.jpg
>
>
> Task not finished for a long time in monitor UI, but I found it finished in
> logs
> Thanks a lot.
> !https://i.stack.imgur.com/C33oL.jpg!
> !C33oL.jpg|thumbnail!
> executor log:
> 17/09/29 17:32:28 INFO executor.CoarseGrainedExecutorBackend: Got assigned
> task 213492
> 17/09/29 17:32:28 INFO executor.Executor: Running task 52.0 in stage 2468.0
> (TID 213492)
> 17/09/29 17:32:28 INFO storage.ShuffleBlockFetcherIterator: Getting 30
> non-empty blocks out of 30 blocks
> 17/09/29 17:32:28 INFO storage.ShuffleBlockFetcherIterator: Started 29 remote
> fetches in 1 ms
> 17:32:28.447: tcPartition=7 ms
> 17/09/29 17:32:28 INFO executor.Executor: Finished task 52.0 in stage 2468.0
> (TID 213492). 2755 bytes result sent to driver
> driver logļ¼:
> 17/09/29 17:32:28 INFO scheduler.TaskSetManager: Starting task 52.0 in stage
> 2468.0 (TID 213492, HMGQXD2, executor 1, partition 52, PROCESS_LOCAL, 6386
> bytes)
> 17/09/29 17:32:28 INFO scheduler.TaskSetManager: Finished task 52.0 in stage
> 2468.0 (TID 213492) in 24 ms on HMGQXD2 (executor 1) (53/200)
> 17/09/29 17:32:28 INFO cluster.YarnScheduler: Removed TaskSet 2468.0, whose
> tasks have all completed, from pool
> 17/09/29 17:32:28 INFO scheduler.DAGScheduler: ResultStage 2468
> (foreachPartition at Counter2.java:152) finished in 0.255 s
> 17/09/29 17:32:28 INFO scheduler.DAGScheduler: Job 1647 finished:
> foreachPartition at Counter2.java:152, took 0.415256 s
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
