[
https://issues.apache.org/jira/browse/SPARK-39620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-39620.
----------------------------------
Fix Version/s: 3.4.0
Resolution: Fixed
Issue resolved by pull request 37008
[https://github.com/apache/spark/pull/37008]
> History server page and API are using inconsistent conditions to filter
> running applications
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-39620
> URL: https://issues.apache.org/jira/browse/SPARK-39620
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.2.1, 3.3.0
> Reporter: wy
> Assignee: wy
> Priority: Major
> Fix For: 3.4.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> When opening summary page, history server follows this logic:
> * If there's completed/incomplete application, page will add script in
> response, using AJAX to call the REST API to get the filtered list.
> * If there's no such application, page will only return a message telling
> nothing found.
> Issue is that page and REST API are using different conditions to filter
> applications. In HistoryPage, an application is considered as completed as
> long as the last attempt is completed. But in ApplicationListResource, all
> attempts should be completed. This brings inconsistency and will cause issue
> in a corner case.
> In driver, event queues have capacity to protect memory. When there's too
> many events, some of them will be dropped and the event log file will be
> incomplete. For an application with multiple attempts, there's possibility
> that the last attempt is completed, but the previous attempts is considered
> as incomplete due to loss of application end event.
> For this type of application, page thinks it is completed, but the API thinks
> it is still running. When opening summary page:
> * When checking completed applications, page will call script, but API
> returns nothing.
> * When checking incomplete applications, page returns nothing.
> So the user won't be able to see this app in history server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
