[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications
[
https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15539115#comment-15539115
]
Apache Spark commented on SPARK-17671:
--
User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/15321
> Spark 2.0 history server summary page is slow even set
> spark.history.ui.maxApplications
> ---
>
> Key: SPARK-17671
> URL: https://issues.apache.org/jira/browse/SPARK-17671
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0
>Reporter: Gang Wu
>
> This is a subsequent task of
> [SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243]. After the
> fix of SPARK-17243 (limit the number of applications in the JSON string
> transferred from history server backend to web UI frontend), the history
> server does display the target number of history summaries.
> However, when there are more than 10k application history, it still gets
> slower and slower. The problem is in the following code:
> {code:title=ApplicationListResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class ApplicationListResource(uiRoot: UIRoot) {
> @GET
> def appList(
> @QueryParam("status") status: JList[ApplicationStatus],
> @DefaultValue("2010-01-01") @QueryParam("minDate") minDate:
> SimpleDateParam,
> @DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate:
> SimpleDateParam,
> @QueryParam("limit") limit: Integer)
> : Iterator[ApplicationInfo] = {
> // although there is a limit operation in the end
> // the following line still does a transformation for all history
> // in the list
> val allApps = uiRoot.getApplicationInfoList
>
> // ...
> // irrelevant code is omitted
> // ...
> if (limit != null) {
> appList.take(limit)
> } else {
> appList
> }
> }
> }
> {code}
> What the code **uiRoot.getApplicationInfoList** does is to transform every
> application history from class ApplicationHistoryInfo to class
> ApplicationInfo. So if there are 10k applications, 10k transformations will
> be done even we have limited 5000 jobs here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications
[
https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534034#comment-15534034
]
Apache Spark commented on SPARK-17671:
--
User 'wgtmac' has created a pull request for this issue:
https://github.com/apache/spark/pull/15303
> Spark 2.0 history server summary page is slow even set
> spark.history.ui.maxApplications
> ---
>
> Key: SPARK-17671
> URL: https://issues.apache.org/jira/browse/SPARK-17671
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0
>Reporter: Gang Wu
>
> This is a subsequent task of
> [SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243]. After the
> fix of SPARK-17243 (limit the number of applications in the JSON string
> transferred from history server backend to web UI frontend), the history
> server does display the target number of history summaries.
> However, when there are more than 10k application history, it still gets
> slower and slower. The problem is in the following code:
> {code:title=ApplicationListResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class ApplicationListResource(uiRoot: UIRoot) {
> @GET
> def appList(
> @QueryParam("status") status: JList[ApplicationStatus],
> @DefaultValue("2010-01-01") @QueryParam("minDate") minDate:
> SimpleDateParam,
> @DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate:
> SimpleDateParam,
> @QueryParam("limit") limit: Integer)
> : Iterator[ApplicationInfo] = {
> // although there is a limit operation in the end
> // the following line still does a transformation for all history
> // in the list
> val allApps = uiRoot.getApplicationInfoList
>
> // ...
> // irrelevant code is omitted
> // ...
> if (limit != null) {
> appList.take(limit)
> } else {
> appList
> }
> }
> }
> {code}
> What the code **uiRoot.getApplicationInfoList** does is to transform every
> application history from class ApplicationHistoryInfo to class
> ApplicationInfo. So if there are 10k applications, 10k transformations will
> be done even we have limited 5000 jobs here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications
[
https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524595#comment-15524595
]
Gang Wu commented on SPARK-17671:
-
Hi [~ajbozarth], can you take a look at the PR? Thanks!
> Spark 2.0 history server summary page is slow even set
> spark.history.ui.maxApplications
> ---
>
> Key: SPARK-17671
> URL: https://issues.apache.org/jira/browse/SPARK-17671
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0
>Reporter: Gang Wu
>
> This is a subsequent task of
> [SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243]. After the
> fix of SPARK-17243 (limit the number of applications in the JSON string
> transferred from history server backend to web UI frontend), the history
> server does display the target number of history summaries.
> However, when there are more than 10k application history, it still gets
> slower and slower. The problem is in the following code:
> {code:title=ApplicationListResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class ApplicationListResource(uiRoot: UIRoot) {
> @GET
> def appList(
> @QueryParam("status") status: JList[ApplicationStatus],
> @DefaultValue("2010-01-01") @QueryParam("minDate") minDate:
> SimpleDateParam,
> @DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate:
> SimpleDateParam,
> @QueryParam("limit") limit: Integer)
> : Iterator[ApplicationInfo] = {
> // although there is a limit operation in the end
> // the following line still does a transformation for all history
> // in the list
> val allApps = uiRoot.getApplicationInfoList
>
> // ...
> // irrelevant code is omitted
> // ...
> if (limit != null) {
> appList.take(limit)
> } else {
> appList
> }
> }
> }
> {code}
> What the code **uiRoot.getApplicationInfoList** does is to transform every
> application history from class ApplicationHistoryInfo to class
> ApplicationInfo. So if there are 10k applications, 10k transformations will
> be done even we have limited 5000 jobs here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications
[
https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524587#comment-15524587
]
Apache Spark commented on SPARK-17671:
--
User 'wgtmac' has created a pull request for this issue:
https://github.com/apache/spark/pull/15248
> Spark 2.0 history server summary page is slow even set
> spark.history.ui.maxApplications
> ---
>
> Key: SPARK-17671
> URL: https://issues.apache.org/jira/browse/SPARK-17671
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0
>Reporter: Gang Wu
>
> This is a subsequent task of
> [SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243]. After the
> fix of SPARK-17243 (limit the number of applications in the JSON string
> transferred from history server backend to web UI frontend), the history
> server does display the target number of history summaries.
> However, when there are more than 10k application history, it still gets
> slower and slower. The problem is in the following code:
> {code:title=ApplicationListResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class ApplicationListResource(uiRoot: UIRoot) {
> @GET
> def appList(
> @QueryParam("status") status: JList[ApplicationStatus],
> @DefaultValue("2010-01-01") @QueryParam("minDate") minDate:
> SimpleDateParam,
> @DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate:
> SimpleDateParam,
> @QueryParam("limit") limit: Integer)
> : Iterator[ApplicationInfo] = {
> // although there is a limit operation in the end
> // the following line still does a transformation for all history
> // in the list
> val allApps = uiRoot.getApplicationInfoList
>
> // ...
> // irrelevant code is omitted
> // ...
> if (limit != null) {
> appList.take(limit)
> } else {
> appList
> }
> }
> }
> {code}
> What the code **uiRoot.getApplicationInfoList** does is to transform every
> application history from class ApplicationHistoryInfo to class
> ApplicationInfo. So if there are 10k applications, 10k transformations will
> be done even we have limited 5000 jobs here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications
[
https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524173#comment-15524173
]
Gang Wu commented on SPARK-17671:
-
I'm working on this and will send a pull request soon.
> Spark 2.0 history server summary page is slow even set
> spark.history.ui.maxApplications
> ---
>
> Key: SPARK-17671
> URL: https://issues.apache.org/jira/browse/SPARK-17671
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0
>Reporter: Gang Wu
>
> This is a subsequent task of
> [SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243]. After the
> fix of SPARK-17243 (limit the number of applications in the JSON string
> transferred from history server backend to web UI frontend), the history
> server does display the target number of history summaries.
> However, when there are more than 10k application history, it still gets
> slower and slower. The problem is in the following code:
> {code:title=ApplicationListResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class ApplicationListResource(uiRoot: UIRoot) {
> @GET
> def appList(
> @QueryParam("status") status: JList[ApplicationStatus],
> @DefaultValue("2010-01-01") @QueryParam("minDate") minDate:
> SimpleDateParam,
> @DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate:
> SimpleDateParam,
> @QueryParam("limit") limit: Integer)
> : Iterator[ApplicationInfo] = {
> // although there is a limit operation in the end
> // the following line still does a transformation for all history
> // in the list
> val allApps = uiRoot.getApplicationInfoList
>
> // ...
> // irrelevant code is omitted
> // ...
> if (limit != null) {
> appList.take(limit)
> } else {
> appList
> }
> }
> }
> {code}
> What the code **uiRoot.getApplicationInfoList** does is to transform every
> application history from class ApplicationHistoryInfo to class
> ApplicationInfo. So if there are 10k applications, 10k transformations will
> be done even we have limited 5000 jobs here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
