[
https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927305#comment-16927305
]
feiwang edited comment on SPARK-29043 at 9/11/19 7:15 AM:
----------------------------------------------------------
[~kabhwan] Other replay threads are waiting for the straggler replay thread.
!image-2019-09-11-15-10-25-326.png!
!image-2019-09-11-15-09-22-912.png!
I think we could set a status for replaying log into application listing, such
as Processing and Completed.
When we checking logs to replay, we could filter the logs which are processing
to prevent been replayed repeatedly.
was (Author: hzfeiwang):
[~kabhwan] Other replay threads are waiting for the straggler replay thread.
!image-2019-09-11-15-10-25-326.png!
!image-2019-09-11-15-09-22-912.png!
I think we could set a status for replaying log to application listing, such as
Processing and Completed.
When we checking logs to replay, we could filter the logs which are processing
to prevent been replayed repeatedly.
> [History Server]Only one replay thread of FsHistoryProvider work because of
> straggler
> -------------------------------------------------------------------------------------
>
> Key: SPARK-29043
> URL: https://issues.apache.org/jira/browse/SPARK-29043
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4
> Reporter: feiwang
> Priority: Major
> Attachments: image-2019-09-11-15-09-22-912.png,
> image-2019-09-11-15-10-25-326.png, screenshot-1.png
>
>
> As shown in the attachment, we set spark.history.fs.numReplayThreads=30 for
> spark history server.
> However, there is only one replay thread work because of straggler.
> Let's check the code.
> https://github.com/apache/spark/blob/7f36cd2aa5e066a807d498b8c51645b136f08a75/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L509-L547
> There is a synchronous operation for all replay tasks.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
