[jira] [Updated] (SPARK-34898) Send ExecutorMetricsUpdate EventLog appropriately
[
https://issues.apache.org/jira/browse/SPARK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34898:
---
Description:
In current EventLoggingListener, we won't write
SparkListenerExecutorMetricsUpdate message at all
{code:java}
override def onExecutorMetricsUpdate(event:
SparkListenerExecutorMetricsUpdate): Unit = {
if (shouldLogStageExecutorMetrics) {
event.executorUpdates.foreach { case (stageKey1, newPeaks) =>
liveStageExecutorMetrics.foreach { case (stageKey2, metricsPerExecutor) =>
// If the update came from the driver, stageKey1 will be the dummy key
(-1, -1),
// so record those peaks for all active stages.
// Otherwise, record the peaks for the matching stage.
if (stageKey1 == DRIVER_STAGE_KEY || stageKey1 == stageKey2) {
val metrics = metricsPerExecutor.getOrElseUpdate(
event.execId, new ExecutorMetrics())
metrics.compareAndUpdatePeakValues(newPeaks)
}
}
}
}
}
{code}
It causes this effect that we can't get driver peakMemoryMetrics in SHS. We
can get executor's since it will update with TaskEnd events.
was:
In current EventLoggingListener, we won't write
SparkListenerExecutorMetricsUpdate message at all
{code:java}
override def onExecutorMetricsUpdate(event:
SparkListenerExecutorMetricsUpdate): Unit = {
if (shouldLogStageExecutorMetrics) {
event.executorUpdates.foreach { case (stageKey1, newPeaks) =>
liveStageExecutorMetrics.foreach { case (stageKey2, metricsPerExecutor) =>
// If the update came from the driver, stageKey1 will be the dummy key
(-1, -1),
// so record those peaks for all active stages.
// Otherwise, record the peaks for the matching stage.
if (stageKey1 == DRIVER_STAGE_KEY || stageKey1 == stageKey2) {
val metrics = metricsPerExecutor.getOrElseUpdate(
event.execId, new ExecutorMetrics())
metrics.compareAndUpdatePeakValues(newPeaks)
}
}
}
}
}
{code}
It cause we can't get driver executor peakMemoryMetrics in SHS. We can got
executor's since it will update with TaskEnd events.
> Send ExecutorMetricsUpdate EventLog appropriately
> -
>
> Key: SPARK-34898
> URL: https://issues.apache.org/jira/browse/SPARK-34898
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.2.0
>Reporter: angerszhu
>Assignee: Apache Spark
>Priority: Major
>
> In current EventLoggingListener, we won't write
> SparkListenerExecutorMetricsUpdate message at all
> {code:java}
> override def onExecutorMetricsUpdate(event:
> SparkListenerExecutorMetricsUpdate): Unit = {
> if (shouldLogStageExecutorMetrics) {
> event.executorUpdates.foreach { case (stageKey1, newPeaks) =>
> liveStageExecutorMetrics.foreach { case (stageKey2, metricsPerExecutor)
> =>
> // If the update came from the driver, stageKey1 will be the dummy
> key (-1, -1),
> // so record those peaks for all active stages.
> // Otherwise, record the peaks for the matching stage.
> if (stageKey1 == DRIVER_STAGE_KEY || stageKey1 == stageKey2) {
> val metrics = metricsPerExecutor.getOrElseUpdate(
> event.execId, new ExecutorMetrics())
> metrics.compareAndUpdatePeakValues(newPeaks)
> }
> }
> }
> }
> }
> {code}
> It causes this effect that we can't get driver peakMemoryMetrics in SHS. We
> can get executor's since it will update with TaskEnd events.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Comment: was deleted
(was: Hi [~Baohe Zhang] , This ticket proposes a new REST API:
http://:18080/api/v1/applicationsexecutorSummary.
It means to display the percentile distribution of peak memory metrics among
the executors used in a given stage. It can help Spark users debug/monitor a
bottleneck of a stage.
In the ticket https://issues.apache.org/jira/browse/SPARK-32446, it proposed to
add a REST API, which can display the percentile distribution of peak memory
metrics for all executors used in an application. The REST API is:
http://:18080/api/v1/applications///executorSummary
Hence this ticket displays executorSummary for a given stage inside an
application. SPARK-32446 wants to display executorSummary for the entire
application. They are different.
)
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> executorMetricsSummary.json, lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png, taskMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> overall stages, i.e. endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task metrics summary
> information in percentile distribution (see example of a single stage in
> [^taskMetricsDistributions.json] ) and executor metrics summary information
> in percentile distribution (see example of a single stage in
> [^executorMetricsDistributions.json] ). The default value is
> withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
> The output is an aggregate of all stages meeting the filtering conditions for
> a given application.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34092) support filtering by task status in REST API call for a specific stage
[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Description: When we want to get a specific stage's detail info with task information, it will return all tasks. The content is huge for a stage with thousands of tasks. And we often just want to know some failed tasks/running tasks with whole stage info to judge whether or not a task has some problem. Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST API calls: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need to set details=true in order to drill down to the task level within a specified stage. was: When we want to get a specific stage's detail info with task information, it will return all tasks. The content is huge for a stage with hundreds of tasks. And we often just want to know some failed tasks/running tasks with whole stage info to judge whether or not a task has some problem. Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST API calls: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need to set details=true in order to drill down to the task level within a specified stage. > support filtering by task status in REST API call for a specific stage > -- > > Key: SPARK-34092 > URL: https://issues.apache.org/jira/browse/SPARK-34092 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > When we want to get a specific stage's detail info with task information, it > will return all tasks. The content is huge for a stage with thousands of > tasks. And we often just want to know some failed tasks/running tasks with > whole stage info to judge whether or not a task has some problem. Query > parameter taskStatus can be used to filter the tasks meeting a specific > status in the REST API call for a given stage. We want to support the > following REST API calls: > applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > Need to set details=true in order to drill down to the task level within a > specified stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308305#comment-17308305
]
Ron Hu commented on SPARK-34488:
Hi [~srowen], Thanks for merging this PR. [~angerszhuuu] and I worked closely
on this JIRA. Angers presented the PR for this ticket. Please put
[~angerszhuuu] into the assignee field in this ticket. He deserves the credit.
Thanks.
> Support task Metrics Distributions and executor Metrics Distributions in the
> REST API call for a specified stage
>
>
> Key: SPARK-34488
> URL: https://issues.apache.org/jira/browse/SPARK-34488
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.0.2
>Reporter: Ron Hu
>Assignee: Ron Hu
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: executorMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> For a specific stage, it is useful to show the task metrics in percentile
> distribution. This information can help users know whether or not there is a
> skew/bottleneck among tasks in a given stage. We list an example in
> [^taskMetricsDistributions.json]
> Similarly, it is useful to show the executor metrics in percentile
> distribution for a specific stage. This information can show whether or not
> there is a skewed load on some executors. We list an example in
> [^executorMetricsDistributions.json]
>
> We define withSummaries query parameter in the REST API for a specific stage
> as:
> applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
> * When withSummaries=true, both task metrics in percentile distribution and
> executor metrics in percentile distribution are included in the REST API
> output. The default value of withSummaries is false, i.e. no metrics
> percentile distribution will be included in the REST API output.
> * Query parameter quantiles define the quantiles we use to calculate metrics
> distributions. It takes effect only when {{withSummaries=true.}} Its
> default value is {{0.0,0.25,0.5,0.75,1.0.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34092) support filtering by task status in REST API call for a specific stage
[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Description: When we want to get a specific stage's detail info with task information, it will return all tasks. The content is huge for a stage with hundreds of tasks. And we often just want to know some failed tasks/running tasks with whole stage info to judge whether or not a task has some problem. Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST API calls: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need to set details=true in order to drill down to the task level within a specified stage. was: When we want to get a specific stage's detail info with task information, it will return all tasks. The content is huge for a stage with hundreds of tasks. And we often just want to know some failed tasks/running tasks with whole stage info to judge whether or not a task has some problem. Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST APIc calls: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need to set details=true in order to drill down to the task level within a specified stage. > support filtering by task status in REST API call for a specific stage > -- > > Key: SPARK-34092 > URL: https://issues.apache.org/jira/browse/SPARK-34092 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > When we want to get a specific stage's detail info with task information, it > will return all tasks. The content is huge for a stage with hundreds of > tasks. And we often just want to know some failed tasks/running tasks with > whole stage info to judge whether or not a task has some problem. Query > parameter taskStatus can be used to filter the tasks meeting a specific > status in the REST API call for a given stage. We want to support the > following REST API calls: > applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > Need to set details=true in order to drill down to the task level within a > specified stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Description:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. This information can show whether or not there is a
skewed load on some executors. We list an example in
[^executorMetricsDistributions.json]
We define withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
* When withSummaries=true, both task metrics in percentile distribution and
executor metrics in percentile distribution are included in the REST API
output. The default value of withSummaries is false, i.e. no metrics
percentile distribution will be included in the REST API output.
* Query parameter quantiles define the quantiles we use to calculate metrics
distributions. It takes effect only when {{withSummaries=true.}} Its default
value is {{0.0,0.25,0.5,0.75,1.0.}}
was:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. This information can show whether or not there is a
skewed load on some executors. We list an example in
[^executorMetricsDistributions.json]
We define withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
* When withSummaries=true, both task metrics in percentile distribution and
executor metrics in percentile distribution are included in the REST API
output. The default value of withSummaries is false, i.e. no metrics
percentile distribution will be included in the REST API output.
* Query parameter quantiles define the quantiles we use to calculate metrics
distributions. It takes effect only when {{withSummaries=true.}} Its default
is {{0.0,0.25,0.5,0.75,1.0.}}
> Support task Metrics Distributions and executor Metrics Distributions in the
> REST API call for a specified stage
>
>
> Key: SPARK-34488
> URL: https://issues.apache.org/jira/browse/SPARK-34488
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.0.2
>Reporter: Ron Hu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> For a specific stage, it is useful to show the task metrics in percentile
> distribution. This information can help users know whether or not there is a
> skew/bottleneck among tasks in a given stage. We list an example in
> [^taskMetricsDistributions.json]
> Similarly, it is useful to show the executor metrics in percentile
> distribution for a specific stage. This information can show whether or not
> there is a skewed load on some executors. We list an example in
> [^executorMetricsDistributions.json]
>
> We define withSummaries query parameter in the REST API for a specific stage
> as:
> applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
> * When withSummaries=true, both task metrics in percentile distribution and
> executor metrics in percentile distribution are included in the REST API
> output. The default value of withSummaries is false, i.e. no metrics
> percentile distribution will be included in the REST API output.
> * Query parameter quantiles define the quantiles we use to calculate metrics
> distributions. It takes effect only when {{withSummaries=true.}} Its
> default value is {{0.0,0.25,0.5,0.75,1.0.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Description:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. This information can show whether or not there is a
skewed load on some executors. We list an example in
[^executorMetricsDistributions.json]
We define withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
* When withSummaries=true, both task metrics in percentile distribution and
executor metrics in percentile distribution are included in the REST API
output. The default value of withSummaries is false, i.e. no metrics
percentile distribution will be included in the REST API output.
* Query parameter quantiles define the quantiles we use to calculate metrics
distributions. It takes effect only when {{withSummaries=true.}} Its default
is {{0.0,0.25,0.5,0.75,1.0.}}
was:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. This information can show whether or not there is a
skewed load on some executors. We list an example in
[^executorMetricsDistributions.json]
We define withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
* When withSummaries=true, both task metrics in percentile distribution and
executor metrics in percentile distribution are included in the REST API
output. The default value of withSummaries is false, i.e. no metrics
percentile distribution will be included in the REST API output.
* Query parameter quantiles define the quantiles we use when calculating
metrics distributions. It takes effect only when {{withSummaries=true.}} Its
default is {{0.0,0.25,0.5,0.75,1.0.}}
> Support task Metrics Distributions and executor Metrics Distributions in the
> REST API call for a specified stage
>
>
> Key: SPARK-34488
> URL: https://issues.apache.org/jira/browse/SPARK-34488
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.0.2
>Reporter: Ron Hu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> For a specific stage, it is useful to show the task metrics in percentile
> distribution. This information can help users know whether or not there is a
> skew/bottleneck among tasks in a given stage. We list an example in
> [^taskMetricsDistributions.json]
> Similarly, it is useful to show the executor metrics in percentile
> distribution for a specific stage. This information can show whether or not
> there is a skewed load on some executors. We list an example in
> [^executorMetricsDistributions.json]
>
> We define withSummaries query parameter in the REST API for a specific stage
> as:
> applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
> * When withSummaries=true, both task metrics in percentile distribution and
> executor metrics in percentile distribution are included in the REST API
> output. The default value of withSummaries is false, i.e. no metrics
> percentile distribution will be included in the REST API output.
> * Query parameter quantiles define the quantiles we use to calculate metrics
> distributions. It takes effect only when {{withSummaries=true.}} Its
> default is {{0.0,0.25,0.5,0.75,1.0.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Description:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. This information can show whether or not there is a
skewed load on some executors. We list an example in
[^executorMetricsDistributions.json]
We define withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
* When withSummaries=true, both task metrics in percentile distribution and
executor metrics in percentile distribution are included in the REST API
output. The default value of withSummaries is false, i.e. no metrics
percentile distribution will be included in the REST API output.
* Query parameter quantiles define the quantiles we use when calculating
metrics distributions. It takes effect only when {{withSummaries=true.}} Its
default is {{0.0,0.25,0.5,0.75,1.0.}}
was:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. This information can show whether or not there is a
skewed load on some executors. We list an example in
[^executorMetricsDistributions.json]
We define withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]
When withSummaries=true, both task metrics in percentile distribution and
executor metrics in percentile distribution are included in the REST API
output. The default value of withSummaries is false, i.e. no metrics
percentile distribution will be included in the REST API output.
> Support task Metrics Distributions and executor Metrics Distributions in the
> REST API call for a specified stage
>
>
> Key: SPARK-34488
> URL: https://issues.apache.org/jira/browse/SPARK-34488
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.0.2
>Reporter: Ron Hu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> For a specific stage, it is useful to show the task metrics in percentile
> distribution. This information can help users know whether or not there is a
> skew/bottleneck among tasks in a given stage. We list an example in
> [^taskMetricsDistributions.json]
> Similarly, it is useful to show the executor metrics in percentile
> distribution for a specific stage. This information can show whether or not
> there is a skewed load on some executors. We list an example in
> [^executorMetricsDistributions.json]
>
> We define withSummaries query parameter in the REST API for a specific stage
> as:
> applications///?withSummaries=[true|false]=0.0,0.25,0.5,0.75,1.0
> * When withSummaries=true, both task metrics in percentile distribution and
> executor metrics in percentile distribution are included in the REST API
> output. The default value of withSummaries is false, i.e. no metrics
> percentile distribution will be included in the REST API output.
> * Query parameter quantiles define the quantiles we use when calculating
> metrics distributions. It takes effect only when {{withSummaries=true.}}
> Its default is {{0.0,0.25,0.5,0.75,1.0.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34092) support filtering by task status in REST API call for a specific stage
[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291067#comment-17291067 ] Ron Hu commented on SPARK-34092: We clarified the need for query parameter taskStatus for a specific stage. > support filtering by task status in REST API call for a specific stage > -- > > Key: SPARK-34092 > URL: https://issues.apache.org/jira/browse/SPARK-34092 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > When we want to get a specific stage's detail info with task information, it > will return all tasks. The content is huge for a stage with hundreds of > tasks. And we often just want to know some failed tasks/running tasks with > whole stage info to judge whether or not a task has some problem. Query > parameter taskStatus can be used to filter the tasks meeting a specific > status in the REST API call for a given stage. We want to support the > following REST APIc calls: > applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > Need to set details=true in order to drill down to the task level within a > specified stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34092) support filtering by task status in REST API call for a specific stage
[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Description: When we want to get a specific stage's detail info with task information, it will return all tasks. The content is huge for a stage with hundreds of tasks. And we often just want to know some failed tasks/running tasks with whole stage info to judge whether or not a task has some problem. Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST APIc calls: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need to set details=true in order to drill down to the task level within a specified stage. was: Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST API calls: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need to set details=true in order to drill down to the task level within a specified stage. > support filtering by task status in REST API call for a specific stage > -- > > Key: SPARK-34092 > URL: https://issues.apache.org/jira/browse/SPARK-34092 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > When we want to get a specific stage's detail info with task information, it > will return all tasks. The content is huge for a stage with hundreds of > tasks. And we often just want to know some failed tasks/running tasks with > whole stage info to judge whether or not a task has some problem. Query > parameter taskStatus can be used to filter the tasks meeting a specific > status in the REST API call for a given stage. We want to support the > following REST APIc calls: > applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > Need to set details=true in order to drill down to the task level within a > specified stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289646#comment-17289646 ] Ron Hu commented on SPARK-34488: It should be noted that this Jira addresses query parameter withSummaries for a specific stage. In another Jira https://issues.apache.org/jira/browse/SPARK-26399, we address query parameter withSummaries for overall stages. > Support task Metrics Distributions and executor Metrics Distributions in the > REST API call for a specified stage > > > Key: SPARK-34488 > URL: https://issues.apache.org/jira/browse/SPARK-34488 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.2 >Reporter: Ron Hu >Priority: Major > Attachments: executorMetricsDistributions.json, > taskMetricsDistributions.json > > > For a specific stage, it is useful to show the task metrics in percentile > distribution. This information can help users know whether or not there is a > skew/bottleneck among tasks in a given stage. We list an example in > [^taskMetricsDistributions.json] > Similarly, it is useful to show the executor metrics in percentile > distribution for a specific stage. This information can show whether or not > there is a skewed load on some executors. We list an example in > [^executorMetricsDistributions.json] > > We define withSummaries query parameter in the REST API for a specific stage > as: > applications///?withSummaries=[true|false] > When withSummaries=true, both task metrics in percentile distribution and > executor metrics in percentile distribution are included in the REST API > output. The default value of withSummaries is false, i.e. no metrics > percentile distribution will be included in the REST API output. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34092) support filtering by task status in REST API call for a specific stage
[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289644#comment-17289644 ] Ron Hu commented on SPARK-34092: It should be noted that this Jira addresses query parameter taskStatus for a specific stage. In another Jira https://issues.apache.org/jira/browse/SPARK-26399, we address query parameter taskStatus for overall stages. > support filtering by task status in REST API call for a specific stage > -- > > Key: SPARK-34092 > URL: https://issues.apache.org/jira/browse/SPARK-34092 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > Query parameter taskStatus can be used to filter the tasks meeting a specific > status in the REST API call for a given stage. We want to support the > following REST API calls: > applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > Need to set details=true in order to drill down to the task level within a > specified stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for overall
stages, i.e. endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task metrics summary
information in percentile distribution (see example of a single stage in
[^taskMetricsDistributions.json] ) and executor metrics summary information in
percentile distribution (see example of a single stage in
[^executorMetricsDistributions.json] ). The default value is
withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
The output is an aggregate of all stages meeting the filtering conditions for a
given application.
was:
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for overall
stages, i.e. endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task metrics summary
information in percentile distribution (see example of a single stage in
[^taskMetricsDistributions.json] ) and executor metrics summary information in
percentile distribution (see example of a single stage in
[^executorMetricsDistributions.json] ). The default value is
withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
The output is an aggregate of all stages meeting the filtering conditions.
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> executorMetricsSummary.json, lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png, taskMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> overall stages, i.e. endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task metrics summary
> information in percentile distribution (see example of a single stage in
> [^taskMetricsDistributions.json] ) and executor metrics summary information
> in percentile distribution (see example of a single stage in
> [^executorMetricsDistributions.json] ). The default value is
> withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
> The output is an aggregate of all stages meeting the filtering conditions for
> a given application.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for overall
stages, i.e. endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task metrics summary
information in percentile distribution (see example of a single stage in
[^taskMetricsDistributions.json] ) and executor metrics summary information in
percentile distribution (see example of a single stage in
[^executorMetricsDistributions.json] ). The default value is
withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
The output is an aggregate of all stages meeting the filtering conditions.
was:
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task metrics summary
information in percentile distribution (see example in
[^taskMetricsDistributions.json] ) and executor metrics summary information in
percentile distribution (see example in [^executorMetricsDistributions.json]
). The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> executorMetricsSummary.json, lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png, taskMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> overall stages, i.e. endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task metrics summary
> information in percentile distribution (see example of a single stage in
> [^taskMetricsDistributions.json] ) and executor metrics summary information
> in percentile distribution (see example of a single stage in
> [^executorMetricsDistributions.json] ). The default value is
> withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
> The output is an aggregate of all stages meeting the filtering conditions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task metrics summary
information in percentile distribution (see example in
[^taskMetricsDistributions.json] ) and executor metrics summary information in
percentile distribution (see example in [^executorMetricsDistributions.json]
). The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
was:
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> executorMetricsSummary.json, lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png, taskMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task metrics summary
> information in percentile distribution (see example in
> [^taskMetricsDistributions.json] ) and executor metrics summary information
> in percentile distribution (see example in
> [^executorMetricsDistributions.json] ). The default value is
> withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288026#comment-17288026
]
Ron Hu commented on SPARK-26399:
Add an example of task Metrics percentile Distributions.
[^taskMetricsDistributions.json]
Add an example of executor Metrics percentile Distributions.
[^executorMetricsDistributions.json]
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> executorMetricsSummary.json, lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png, taskMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Attachment: executorMetricsDistributions.json
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> executorMetricsSummary.json, lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png, taskMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Attachment: taskMetricsDistributions.json
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> executorMetricsSummary.json, lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png, taskMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Attachment: taskMetricsDistributions.json
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png, taskMetricsDistributions.json
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Summary: Define query parameters to support various filtering conditions in
REST API for overall stages (was: Add new stage-level REST APIs and parameters)
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Define query parameters to support various filtering conditions in REST API for overall stages
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
> Define query parameters to support various filtering conditions in REST API
> for overall stages
> --
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> [~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
> endpoint /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter can be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Summary: Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage (was: Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage) > Support task Metrics Distributions and executor Metrics Distributions in the > REST API call for a specified stage > > > Key: SPARK-34488 > URL: https://issues.apache.org/jira/browse/SPARK-34488 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.2 >Reporter: Ron Hu >Priority: Major > Attachments: executorMetricsDistributions.json, > taskMetricsDistributions.json > > > For a specific stage, it is useful to show the task metrics in percentile > distribution. This information can help users know whether or not there is a > skew/bottleneck among tasks in a given stage. We list an example in > [^taskMetricsDistributions.json] > Similarly, it is useful to show the executor metrics in percentile > distribution for a specific stage. This information can show whether or not > there is a skewed load on some executors. We list an example in > [^executorMetricsDistributions.json] > > We define withSummaries query parameter in the REST API for a specific stage > as: > applications///?withSummaries=[true|false] > When withSummaries=true, both task metrics in percentile distribution and > executor metrics in percentile distribution are included in the REST API > output. The default value of withSummaries is false, i.e. no metrics > percentile distribution will be included in the REST API output. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34488: --- Description: For a specific stage, it is useful to show the task metrics in percentile distribution. This information can help users know whether or not there is a skew/bottleneck among tasks in a given stage. We list an example in [^taskMetricsDistributions.json] Similarly, it is useful to show the executor metrics in percentile distribution for a specific stage. This information can show whether or not there is a skewed load on some executors. We list an example in [^executorMetricsDistributions.json] We define withSummaries query parameter in the REST API for a specific stage as: applications///?withSummaries=[true|false] When withSummaries=true, both task metrics in percentile distribution and executor metrics in percentile distribution are included in the REST API output. The default value of withSummaries is false, i.e. no metrics percentile distribution will be included in the REST API output. was: For a specific stage, it is useful to show the task metrics in percentile distribution. This information can help users know whether or not there is a skew/bottleneck among tasks in a given stage. We list an example in [^taskMetricsDistributions.json] Similarly, it is useful to show the executor metrics in percentile distribution for a specific stage. We list an example in [^executorMetricsDistributions.json] We use withSummaries query parameter in the REST API for a specific stage as: applications///?withSummaries=[true|false] > Support task Metrics Distributions and executor Metrics Distributions in the > REST API entry for a specified stage > - > > Key: SPARK-34488 > URL: https://issues.apache.org/jira/browse/SPARK-34488 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.2 >Reporter: Ron Hu >Priority: Major > Attachments: executorMetricsDistributions.json, > taskMetricsDistributions.json > > > For a specific stage, it is useful to show the task metrics in percentile > distribution. This information can help users know whether or not there is a > skew/bottleneck among tasks in a given stage. We list an example in > [^taskMetricsDistributions.json] > Similarly, it is useful to show the executor metrics in percentile > distribution for a specific stage. This information can show whether or not > there is a skewed load on some executors. We list an example in > [^executorMetricsDistributions.json] > > We define withSummaries query parameter in the REST API for a specific stage > as: > applications///?withSummaries=[true|false] > When withSummaries=true, both task metrics in percentile distribution and > executor metrics in percentile distribution are included in the REST API > output. The default value of withSummaries is false, i.e. no metrics > percentile distribution will be included in the REST API output. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Description:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. We list an example in
[^executorMetricsDistributions.json]
We use withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]
was:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
Here is an example,
"taskMetricsDistributions" :
{ "quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0, 1195000.0,
1195000.0 ], "executorRunTime" : [ 3.0, 3.0, 3.0, 3.0, 3.0 ],
"executorCpuTime" : [ 321.0, 321.0, 321.0, 321.0, 321.0 ],
"resultSize" : [ 2648.0, 2648.0, 2648.0, 2648.0, 2648.0 ], "jvmGcTime" :
[ 0.0, 0.0, 0.0, 0.0, 0.0 ], "resultSerializationTime" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ], "gettingResultTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"schedulerDelay" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ], "peakExecutionMemory" : [
0.0, 0.0, 0.0, 0.0, 0.0 ], "memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0
], "diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "inputMetrics" :
{"bytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "recordsRead" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ] }
,
"outputMetrics" : \{"bytesWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"recordsWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] },
"shuffleReadMetrics" : \{"readBytes" : [ 236.0, 236.0, 236.0, 236.0, 236.0
], "readRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"remoteBlocksFetched" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "localBlocksFetched"
: [ 4.0, 4.0, 4.0, 4.0, 4.0 ], "fetchWaitTime" : [ 0.0, 0.0, 0.0, 0.0,
0.0 ], "remoteBytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"remoteBytesReadToDisk" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"totalBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ] },
"shuffleWriteMetrics" : \{"writeBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"writeRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "writeTime" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ] }
}
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage.
Here is an example:
"executorMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"taskTime" : [ 6.0, 6.0, 6.0, 6.0, 6.0 ],
"failedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"succeededTasks" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"killedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleRead" : [ 236.0, 236.0, 236.0, 236.0, 236.0 ],
"shuffleReadRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"shuffleWrite" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleWriteRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakMemoryMetrics" :
{ "JVMHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "JVMOffHeapMemory"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "OnHeapExecutionMemory" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ], "OffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "DirectPoolMemory"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MappedPoolMemory" : [ 0.0, 0.0, 0.0, 0.0,
0.0 ], "ProcessTreeJVMVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeJVMRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCCount"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0
], "MajorGCCount" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MajorGCTime" : [
0.0, 0.0, 0.0, 0.0, 0.0 ] }
}
We use withSummaries query parameter in the REST API for a
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Description:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
Here is an example,
"taskMetricsDistributions" :
{ "quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0, 1195000.0,
1195000.0 ], "executorRunTime" : [ 3.0, 3.0, 3.0, 3.0, 3.0 ],
"executorCpuTime" : [ 321.0, 321.0, 321.0, 321.0, 321.0 ],
"resultSize" : [ 2648.0, 2648.0, 2648.0, 2648.0, 2648.0 ], "jvmGcTime" :
[ 0.0, 0.0, 0.0, 0.0, 0.0 ], "resultSerializationTime" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ], "gettingResultTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"schedulerDelay" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ], "peakExecutionMemory" : [
0.0, 0.0, 0.0, 0.0, 0.0 ], "memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0
], "diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "inputMetrics" :
{"bytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "recordsRead" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ] }
,
"outputMetrics" : \{"bytesWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"recordsWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] },
"shuffleReadMetrics" : \{"readBytes" : [ 236.0, 236.0, 236.0, 236.0, 236.0
], "readRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"remoteBlocksFetched" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "localBlocksFetched"
: [ 4.0, 4.0, 4.0, 4.0, 4.0 ], "fetchWaitTime" : [ 0.0, 0.0, 0.0, 0.0,
0.0 ], "remoteBytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"remoteBytesReadToDisk" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"totalBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ] },
"shuffleWriteMetrics" : \{"writeBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"writeRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "writeTime" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ] }
}
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage.
Here is an example:
"executorMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"taskTime" : [ 6.0, 6.0, 6.0, 6.0, 6.0 ],
"failedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"succeededTasks" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"killedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleRead" : [ 236.0, 236.0, 236.0, 236.0, 236.0 ],
"shuffleReadRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"shuffleWrite" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleWriteRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakMemoryMetrics" :
{ "JVMHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "JVMOffHeapMemory"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "OnHeapExecutionMemory" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ], "OffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "DirectPoolMemory"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MappedPoolMemory" : [ 0.0, 0.0, 0.0, 0.0,
0.0 ], "ProcessTreeJVMVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeJVMRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCCount"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0
], "MajorGCCount" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MajorGCTime" : [
0.0, 0.0, 0.0, 0.0, 0.0 ] }
}
We use withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]
was:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
Here is an example,
"taskMetricsDistributions" :
{ "quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0, 1195000.0,
1195000.0 ],
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Attachment: executorMetricsDistributions.json
> Support task Metrics Distributions and executor Metrics Distributions in the
> REST API entry for a specified stage
> -
>
> Key: SPARK-34488
> URL: https://issues.apache.org/jira/browse/SPARK-34488
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.0.2
>Reporter: Ron Hu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> For a specific stage, it is useful to show the task metrics in percentile
> distribution. This information can help users know whether or not there is a
> skew/bottleneck among tasks in a given stage. We list an example in
> Here is an example,
>
> "taskMetricsDistributions" :
> { "quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
> "executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
> "executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0, 1195000.0,
> 1195000.0 ], "executorRunTime" : [ 3.0, 3.0, 3.0, 3.0, 3.0 ],
> "executorCpuTime" : [ 321.0, 321.0, 321.0, 321.0, 321.0
> ], "resultSize" : [ 2648.0, 2648.0, 2648.0, 2648.0, 2648.0 ],
> "jvmGcTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "resultSerializationTime" : [
> 0.0, 0.0, 0.0, 0.0, 0.0 ], "gettingResultTime" : [ 0.0, 0.0, 0.0, 0.0,
> 0.0 ], "schedulerDelay" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ],
> "peakExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "memoryBytesSpilled"
> : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0,
> 0.0 ], "inputMetrics" : \\{"bytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "recordsRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] }
> ,
> "outputMetrics" : \{"bytesWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "recordsWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] },
> "shuffleReadMetrics" : \{"readBytes" : [ 236.0, 236.0, 236.0, 236.0,
> 236.0 ], "readRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
> "remoteBlocksFetched" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "localBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ], "fetchWaitTime" : [
> 0.0, 0.0, 0.0, 0.0, 0.0 ], "remoteBytesRead" : [ 0.0, 0.0, 0.0, 0.0,
> 0.0 ], "remoteBytesReadToDisk" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "totalBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ] },
> "shuffleWriteMetrics" : \{"writeBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "writeRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "writeTime" : [ 0.0, 0.0, 0.0,
> 0.0, 0.0 ] }
> }
>
> Similarly, it is useful to show the executor metrics in percentile
> distribution for a specific stage.
> Here is an example:
>
> "executorMetricsDistributions" : {
> "quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
> "taskTime" : [ 6.0, 6.0, 6.0, 6.0, 6.0 ],
> "failedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "succeededTasks" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
> "killedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "shuffleRead" : [ 236.0, 236.0, 236.0, 236.0, 236.0 ],
> "shuffleReadRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
> "shuffleWrite" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "shuffleWriteRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "peakMemoryMetrics" :
> { "JVMHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "JVMOffHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OnHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OnHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OnHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "DirectPoolMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MappedPoolMemory" :
> [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "ProcessTreeJVMVMemory" : [ 0.0, 0.0, 0.0,
> 0.0, 0.0 ], "ProcessTreeJVMRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "ProcessTreePythonVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "ProcessTreePythonRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "ProcessTreeOtherVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "ProcessTreeOtherRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "MinorGCCount" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCTime" : [
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Attachment: taskMetricsDistributions.json
> Support task Metrics Distributions and executor Metrics Distributions in the
> REST API entry for a specified stage
> -
>
> Key: SPARK-34488
> URL: https://issues.apache.org/jira/browse/SPARK-34488
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.0.2
>Reporter: Ron Hu
>Priority: Major
> Attachments: executorMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> For a specific stage, it is useful to show the task metrics in percentile
> distribution. This information can help users know whether or not there is a
> skew/bottleneck among tasks in a given stage. We list an example in
> Here is an example,
>
> "taskMetricsDistributions" :
> { "quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
> "executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
> "executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0, 1195000.0,
> 1195000.0 ], "executorRunTime" : [ 3.0, 3.0, 3.0, 3.0, 3.0 ],
> "executorCpuTime" : [ 321.0, 321.0, 321.0, 321.0, 321.0
> ], "resultSize" : [ 2648.0, 2648.0, 2648.0, 2648.0, 2648.0 ],
> "jvmGcTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "resultSerializationTime" : [
> 0.0, 0.0, 0.0, 0.0, 0.0 ], "gettingResultTime" : [ 0.0, 0.0, 0.0, 0.0,
> 0.0 ], "schedulerDelay" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ],
> "peakExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "memoryBytesSpilled"
> : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0,
> 0.0 ], "inputMetrics" : \\{"bytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "recordsRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] }
> ,
> "outputMetrics" : \{"bytesWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "recordsWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] },
> "shuffleReadMetrics" : \{"readBytes" : [ 236.0, 236.0, 236.0, 236.0,
> 236.0 ], "readRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
> "remoteBlocksFetched" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "localBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ], "fetchWaitTime" : [
> 0.0, 0.0, 0.0, 0.0, 0.0 ], "remoteBytesRead" : [ 0.0, 0.0, 0.0, 0.0,
> 0.0 ], "remoteBytesReadToDisk" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "totalBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ] },
> "shuffleWriteMetrics" : \{"writeBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "writeRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "writeTime" : [ 0.0, 0.0, 0.0,
> 0.0, 0.0 ] }
> }
>
> Similarly, it is useful to show the executor metrics in percentile
> distribution for a specific stage.
> Here is an example:
>
> "executorMetricsDistributions" : {
> "quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
> "taskTime" : [ 6.0, 6.0, 6.0, 6.0, 6.0 ],
> "failedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "succeededTasks" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
> "killedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "shuffleRead" : [ 236.0, 236.0, 236.0, 236.0, 236.0 ],
> "shuffleReadRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
> "shuffleWrite" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "shuffleWriteRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "peakMemoryMetrics" :
> { "JVMHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "JVMOffHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OnHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OnHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OnHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "OffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "DirectPoolMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MappedPoolMemory" :
> [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "ProcessTreeJVMVMemory" : [ 0.0, 0.0, 0.0,
> 0.0, 0.0 ], "ProcessTreeJVMRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "ProcessTreePythonVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "ProcessTreePythonRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "ProcessTreeOtherVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "ProcessTreeOtherRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
> "MinorGCCount" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCTime" : [ 0.0,
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Description:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
Here is an example,
"taskMetricsDistributions" :
{ "quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0, 1195000.0,
1195000.0 ], "executorRunTime" : [ 3.0, 3.0, 3.0, 3.0, 3.0 ],
"executorCpuTime" : [ 321.0, 321.0, 321.0, 321.0, 321.0 ],
"resultSize" : [ 2648.0, 2648.0, 2648.0, 2648.0, 2648.0 ], "jvmGcTime" :
[ 0.0, 0.0, 0.0, 0.0, 0.0 ], "resultSerializationTime" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ], "gettingResultTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"schedulerDelay" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ], "peakExecutionMemory" : [
0.0, 0.0, 0.0, 0.0, 0.0 ], "memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0
], "diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "inputMetrics" :
\\{"bytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "recordsRead" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ] }
,
"outputMetrics" : \{"bytesWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"recordsWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] },
"shuffleReadMetrics" : \{"readBytes" : [ 236.0, 236.0, 236.0, 236.0, 236.0
], "readRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"remoteBlocksFetched" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "localBlocksFetched"
: [ 4.0, 4.0, 4.0, 4.0, 4.0 ], "fetchWaitTime" : [ 0.0, 0.0, 0.0, 0.0,
0.0 ], "remoteBytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"remoteBytesReadToDisk" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"totalBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ] },
"shuffleWriteMetrics" : \{"writeBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"writeRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "writeTime" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ] }
}
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage.
Here is an example:
"executorMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"taskTime" : [ 6.0, 6.0, 6.0, 6.0, 6.0 ],
"failedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"succeededTasks" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"killedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleRead" : [ 236.0, 236.0, 236.0, 236.0, 236.0 ],
"shuffleReadRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"shuffleWrite" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleWriteRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakMemoryMetrics" :
{ "JVMHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "JVMOffHeapMemory"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "OnHeapExecutionMemory" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ], "OffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "DirectPoolMemory"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MappedPoolMemory" : [ 0.0, 0.0, 0.0, 0.0,
0.0 ], "ProcessTreeJVMVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeJVMRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCCount"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0
], "MajorGCCount" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MajorGCTime" : [
0.0, 0.0, 0.0, 0.0, 0.0 ] }
}
We use withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]
was:
For a specific stage, it is useful to show the task metrics in percentile
distribution.
This information can help users know whether or not there is a skew/bottleneck
among tasks in a given stage.
Here is an example,
"taskMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0,
1195000.0, 1195000.0 ],
"executorRunTime" :
[jira] [Updated] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage
[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---
Description:
For a specific stage, it is useful to show the task metrics in percentile
distribution.
This information can help users know whether or not there is a skew/bottleneck
among tasks in a given stage.
Here is an example,
"taskMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0,
1195000.0, 1195000.0 ],
"executorRunTime" : [ 3.0, 3.0, 3.0, 3.0, 3.0 ],
"executorCpuTime" : [ 321.0, 321.0, 321.0, 321.0, 321.0
],
"resultSize" : [ 2648.0, 2648.0, 2648.0, 2648.0, 2648.0 ],
"jvmGcTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"resultSerializationTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"gettingResultTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"schedulerDelay" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ],
"peakExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputMetrics" : \{"bytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"recordsRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] },
"outputMetrics" : \{"bytesWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"recordsWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ] },
"shuffleReadMetrics" : \{"readBytes" : [ 236.0, 236.0, 236.0, 236.0, 236.0
], "readRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"remoteBlocksFetched" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "localBlocksFetched"
: [ 4.0, 4.0, 4.0, 4.0, 4.0 ], "fetchWaitTime" : [ 0.0, 0.0, 0.0, 0.0,
0.0 ], "remoteBytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"remoteBytesReadToDisk" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"totalBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ] },
"shuffleWriteMetrics" : \{"writeBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"writeRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "writeTime" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ] }
}
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage.
Here is an example:
"executorMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"taskTime" : [ 6.0, 6.0, 6.0, 6.0, 6.0 ],
"failedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"succeededTasks" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"killedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleRead" : [ 236.0, 236.0, 236.0, 236.0, 236.0 ],
"shuffleReadRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"shuffleWrite" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleWriteRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakMemoryMetrics" :
{ "JVMHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "JVMOffHeapMemory"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "OnHeapExecutionMemory" : [ 0.0, 0.0, 0.0,
0.0, 0.0 ], "OffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "DirectPoolMemory"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MappedPoolMemory" : [ 0.0, 0.0, 0.0, 0.0,
0.0 ], "ProcessTreeJVMVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeJVMRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCCount"
: [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MinorGCTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0
], "MajorGCCount" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "MajorGCTime" : [
0.0, 0.0, 0.0, 0.0, 0.0 ] }
}
We use withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]
was:
For a specific stage, it is useful to show the task metrics in percentile
distribution.
This information can help users know whether or not there is a skew/bottleneck
among tasks in a given stage.
Here is an example,
"taskMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0,
1195000.0, 1195000.0 ],
"executorRunTime" : [ 3.0, 3.0, 3.0, 3.0,
[jira] [Created] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API entry for a specified stage
Ron Hu created SPARK-34488:
--
Summary: Support task Metrics Distributions and executor Metrics
Distributions in the REST API entry for a specified stage
Key: SPARK-34488
URL: https://issues.apache.org/jira/browse/SPARK-34488
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 3.0.2
Reporter: Ron Hu
For a specific stage, it is useful to show the task metrics in percentile
distribution.
This information can help users know whether or not there is a skew/bottleneck
among tasks in a given stage.
Here is an example,
"taskMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"executorDeserializeTime" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"executorDeserializeCpuTime" : [ 1195000.0, 1195000.0, 1195000.0,
1195000.0, 1195000.0 ],
"executorRunTime" : [ 3.0, 3.0, 3.0, 3.0, 3.0 ],
"executorCpuTime" : [ 321.0, 321.0, 321.0, 321.0, 321.0
],
"resultSize" : [ 2648.0, 2648.0, 2648.0, 2648.0, 2648.0 ],
"jvmGcTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"resultSerializationTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"gettingResultTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"schedulerDelay" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ],
"peakExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputMetrics" : {
"bytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"recordsRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ]
},
"outputMetrics" : {
"bytesWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"recordsWritten" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ]
},
"shuffleReadMetrics" : {
"readBytes" : [ 236.0, 236.0, 236.0, 236.0, 236.0 ],
"readRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"remoteBlocksFetched" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"localBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"fetchWaitTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"remoteBytesRead" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"remoteBytesReadToDisk" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"totalBlocksFetched" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ]
},
"shuffleWriteMetrics" : {
"writeBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"writeRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"writeTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ]
}
}
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage.
Here is an example:
"executorMetricsDistributions" : {
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"taskTime" : [ 6.0, 6.0, 6.0, 6.0, 6.0 ],
"failedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"succeededTasks" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"killedTasks" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleRead" : [ 236.0, 236.0, 236.0, 236.0, 236.0 ],
"shuffleReadRecords" : [ 4.0, 4.0, 4.0, 4.0, 4.0 ],
"shuffleWrite" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleWriteRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakMemoryMetrics" : {
"JVMHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"JVMOffHeapMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OnHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"OffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"DirectPoolMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"MappedPoolMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeJVMVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeJVMRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreePythonRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherVMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"ProcessTreeOtherRSSMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"MinorGCCount" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"MinorGCTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"MajorGCCount" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"MajorGCTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ]
}
}
We use withSummaries query parameter in the REST API for a specific stage as:
applications///?withSummaries=[true|false]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For
[jira] [Updated] (SPARK-34092) support filtering by task status in REST API call for a specific stage
[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Summary: support filtering by task status in REST API call for a specific stage (was: Stage level RestFul API support filter by task status) > support filtering by task status in REST API call for a specific stage > -- > > Key: SPARK-34092 > URL: https://issues.apache.org/jira/browse/SPARK-34092 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > Query parameter taskStatus can be used to filter the tasks meeting a specific > status in the REST API call for a given stage. We want to support the > following REST API calls: > applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > Need to set details=true in order to drill down to the task level within a > specified stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34092) Stage level RestFul API support filter by task status
[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Description: Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST API calls: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need to set details=true in order to drill down to the task level within a specified stage. was: Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST API call: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need > Stage level RestFul API support filter by task status > - > > Key: SPARK-34092 > URL: https://issues.apache.org/jira/browse/SPARK-34092 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > Query parameter taskStatus can be used to filter the tasks meeting a specific > status in the REST API call for a given stage. We want to support the > following REST API calls: > applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > Need to set details=true in order to drill down to the task level within a > specified stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34092) Stage level RestFul API support filter by task status
[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-34092: --- Description: Query parameter taskStatus can be used to filter the tasks meeting a specific status in the REST API call for a given stage. We want to support the following REST API call: applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] Need was:Support filter Task by taskstatus when details is true > Stage level RestFul API support filter by task status > - > > Key: SPARK-34092 > URL: https://issues.apache.org/jira/browse/SPARK-34092 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > Query parameter taskStatus can be used to filter the tasks meeting a specific > status in the REST API call for a given stage. We want to support the > following REST API call: > applications///stages/?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > applications///stages//?details=true=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] > Need -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set when
details=true (i.e. this parameter will be ignored when details=false).
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter will be set when
details=true (i.e. this parameter will be ignored when details=false).
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261948#comment-17261948
]
Ron Hu edited comment on SPARK-26399 at 1/23/21, 3:39 AM:
--
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter is meaningful only
when details=true (i.e. this parameter will be ignored when details=false).
was (Author: ron8hu):
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter is meaningful only
when details=true (i.e. this parameter will be ignored when details=false).
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter will be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter will be set when
details=true (i.e. this parameter will be ignored when details=false).
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter will be set when
details=true (i.e. this parameter will be ignored when details=false).
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]
> where
> * query parameter details=true is
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261948#comment-17261948
]
Ron Hu edited comment on SPARK-26399 at 1/10/21, 6:22 PM:
--
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter is meaningful when
details=true (i.e. this parameter will be ignored when details=false).
was (Author: ron8hu):
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter will be set when
details=true (i.e. this parameter will be ignored when details=false).
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter will be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261948#comment-17261948
]
Ron Hu edited comment on SPARK-26399 at 1/10/21, 6:22 PM:
--
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter is meaningful only
when details=true (i.e. this parameter will be ignored when details=false).
was (Author: ron8hu):
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter is meaningful when
details=true (i.e. this parameter will be ignored when details=false).
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter will be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261948#comment-17261948
]
Ron Hu edited comment on SPARK-26399 at 1/10/21, 6:20 PM:
--
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter will be set when
details=true (i.e. this parameter will be ignored when details=false).
was (Author: ron8hu):
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set only when
details=true.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages. This parameter will be set when
> details=true (i.e. this parameter will be ignored when details=false).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter will be set when
details=true (i.e. this parameter will be ignored when details=false).
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261948#comment-17261948
]
Ron Hu edited comment on SPARK-26399 at 1/10/21, 6:18 PM:
--
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages. This parameter can be set only when
details=true.
was (Author: ron8hu):
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status;
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status.
When status parameter is not specified, a list of all stages are generated.
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages.
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status;
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status.
> When status parameter is not specified, a list of all stages are generated.
> * query parameter withSummaries=true is to
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261948#comment-17261948
]
Ron Hu edited comment on SPARK-26399 at 1/9/21, 9:25 PM:
-
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status;
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages.
was (Author: ron8hu):
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where query parameter details=true is to show the detailed task information
within each stage. The default value is details=false;
query parameter status can select those stages with the specified status;
query parameter withSummaries=true is to show both task summary information in
percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
taskStatus is to show only those tasks with the specified status within their
corresponding stages.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status;
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * query parameter taskStatus is to show only those tasks with the specified
> status within their corresponding stages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status;
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* query parameter taskStatus is to show only those tasks with the specified
status within their corresponding stages.
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status;
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* taskStatus is to show only those tasks with the specified status within
their corresponding stages.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status;
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> *
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where
* query parameter details=true is to show the detailed task information within
each stage. The default value is details=false;
* query parameter status can select those stages with the specified status;
* query parameter withSummaries=true is to show both task summary information
in percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
* taskStatus is to show only those tasks with the specified status within
their corresponding stages.
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where query parameter details=true is to show the detailed task information
within each stage. The default value is details=false;
query parameter status can select those stages with the specified status;
query parameter withSummaries=true is to show both task summary information in
percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
taskStatus is to show only those tasks with the specified status within their
corresponding stages.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where
> * query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> * query parameter status can select those stages with the specified status;
> * query parameter withSummaries=true is to show both task summary
> information in percentile distribution and executor summary information in
> percentile distribution. The default value is withSummaries=false.
> * taskStatus is to show
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
*. *. *
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where query parameter details=true is to show the detailed task information
within each stage. The default value is details=false;
query parameter status can select those stages with the specified status;
query parameter withSummaries=true is to show both task summary information in
percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
taskStatus is to show only those tasks with the specified status within their
corresponding stages.
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where query parameter details=true is to show the detailed task information
within each stage. The default value is details=false;
query parameter status can select those stages with the specified status;
query parameter withSummaries=true is to show both task summary information in
percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
taskStatus is to show only those tasks with the specified status within their
corresponding stages.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> *. *. *
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> query parameter status can select those stages with the specified status;
> query parameter withSummaries=true is to show both task summary information
> in percentile distribution and executor summary information in percentile
> distribution. The default value is withSummaries=false.
> taskStatus is to show only those tasks with the specified status
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
Note that the above description is too brief to be clear. [~angerszhuuu] and
[~ron8hu] discussed a generic and consistent way for endpoint
/application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where query parameter details=true is to show the detailed task information
within each stage. The default value is details=false;
query parameter status can select those stages with the specified status;
query parameter withSummaries=true is to show both task summary information in
percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
taskStatus is to show only those tasks with the specified status within their
corresponding stages.
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
Note that the above description is too brief to be clear. Ron Hu added the
additional details to explain the use cases from the downstream products. See
the comments dated 1/07/2021 with a couple of sample json files.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> Note that the above description is too brief to be clear. [~angerszhuuu] and
> [~ron8hu] discussed a generic and consistent way for endpoint
> /application/\{app-id}/stages. It can be:
> /application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
> where query parameter details=true is to show the detailed task information
> within each stage. The default value is details=false;
> query parameter status can select those stages with the specified status;
> query parameter withSummaries=true is to show both task summary information
> in percentile distribution and executor summary information in percentile
> distribution. The default value is withSummaries=false.
> taskStatus is to show only those tasks with the specified status within their
> corresponding stages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261948#comment-17261948
]
Ron Hu commented on SPARK-26399:
[~angerszhuuu] and [~ron8hu] discussed a generic and consistent way for
endpoint /application/\{app-id}/stages. It can be:
/application/\{app-id}/stages?details=[true|false]=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]=[true|false]=[RUNNING|SUCCESS|FAILED|PENDING]
where query parameter details=true is to show the detailed task information
within each stage. The default value is details=false;
query parameter status can select those stages with the specified status;
query parameter withSummaries=true is to show both task summary information in
percentile distribution and executor summary information in percentile
distribution. The default value is withSummaries=false.
taskStatus is to show only those tasks with the specified status within their
corresponding stages.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> Note that the above description is too brief to be clear. Ron Hu added the
> additional details to explain the use cases from the downstream products.
> See the comments dated 1/07/2021 with a couple of sample json files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260856#comment-17260856
]
Ron Hu edited comment on SPARK-26399 at 1/8/21, 5:18 AM:
-
The initial description of this jira has this statement: "filtering for task
status, and returning tasks that match (for example, FAILED tasks)"
To achieve the above statement, we need an new endpoint like this:
/applications/[app-id]/stages?taskstatus=[FAILED|KILLED|SUCCESS]
If a user specifies /applications/[app-id]/stages?taskstatus=KILLED, then we
generate a json file to contain all the killed task information from all the
stages. This way can help users monitor all the killed tasks. For example,
when a Spark user enables speculation, he needs the information of all the
killed tasks so that he can monitor the benefit/cost brought by speculation.
I attach a sample json file [^lispark230_restapi_ex2_stages_failedTasks.json]
which contains the failed tasks and the corresponding stages for reference.
was (Author: ron8hu):
The initial description of this jira has this statement: "filtering for task
status, and returning tasks that match (for example, FAILED tasks)"
To achieve the above statement, we need an new endpoint like this:
/applications/[app-id]/stages?taskstatus=[FAILED|KILLED|SUCCESS]
If a user specifies /applications/[app-id]/stages?taskstatus=KILLED, then we
generate a json file to contain all the killed task information from all the
stages. This way can help users monitor all the killed tasks. For example, a
Spark user enables speculation, he needs the information of all the killed
tasks so that he can monitor the benefit/cost brought by speculation.
I attach a sample json file [^lispark230_restapi_ex2_stages_failedTasks.json]
which contains the failed tasks and the corresponding stages for reference.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> Note that the above description is too brief to be clear. Ron Hu added the
> additional details to explain the use cases from the downstream products.
> See the comments dated 1/07/2021 with a couple of sample json files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
Note that the above description is too brief to be clear. Ron Hu added the
additional details to explain the use cases from the downstream products. See
the comments dated 1/07/2021 with a couple of sample json files.
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
> Note that the above description is too brief to be clear. Ron Hu added the
> additional details to explain the use cases from the downstream products.
> See the comments dated 1/07/2021 with a couple of sample json files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260856#comment-17260856
]
Ron Hu commented on SPARK-26399:
The initial description of this jira has this statement: "filtering for task
status, and returning tasks that match (for example, FAILED tasks)"
To achieve the above statement, we need an new endpoint like this:
/applications/[app-id]/stages?taskstatus=[FAILED|KILLED|SUCCESS]
If a user specifies /applications/[app-id]/stages?taskstatus=KILLED, then we
generate a json file to contain all the killed task information from all the
stages. This way can help users monitor all the killed tasks. For example, a
Spark user enables speculation, he needs the information of all the killed
tasks so that he can monitor the benefit/cost brought by speculation.
I attach a sample json file [^lispark230_restapi_ex2_stages_failedTasks.json]
which contains the failed tasks and the corresponding stages for reference.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Attachment: lispark230_restapi_ex2_stages_failedTasks.json
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260854#comment-17260854
]
Ron Hu commented on SPARK-26399:
[^lispark230_restapi_ex2_stages_failedTasks.json]
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_failedTasks.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260805#comment-17260805
]
Ron Hu edited comment on SPARK-26399 at 1/7/21, 10:23 PM:
--
Dr. Elephant ([https://github.com/linkedin/dr-elephant]) is a downstream open
source product that utilizes Spark monitoring information so that it can advise
Spark users where to optimize their configuration parameters ranging from
memory usage, number of cores, etc. Because the initial description of this
ticket is too brief to be clear. Let me explain the use cases for Dr. Elephant
here.
REST API /applications/[app-id]/stages: This useful endpoint generate a json
file containing all stages for a given application. The current Spark version
already provides it.
In order to debug if there exists a skew issue, a downstream product also needs:
- taskMetricsSummary: It includes task metric information such as
executorRunTime, inputMetrics, outputMetrics, shuffleReadMetrics, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the tasks in a
given stage. The same information shows up in Web UI for a specified stage.
- executorMetricsSummary: It includes executor metrics information such as
number of tasks, input bytes, peak JVM memory, peak execution memory, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the executors used
in a given stage. This information has been developed by [~angerszhuuu] in the
PR he submitted.
We can add the above information to the the json file generated by
/applications/[app-id]/stages. It may double the size of the stages endpoints
file. It should be fine because the current stages json file is not that big.
Here is one sample json file for stages endpoint.
[^lispark230_restapi_ex2_stages_withSummaries.json]
An alternative approach is to add a new REST API such as
"/applications/[app-id]/stages/withSummaries". But it may need a little bit
more code for a new endpoint.
was (Author: ron8hu):
Dr. Elephant ([https://github.com/linkedin/dr-elephant]) is a downstream open
source product that utilizes Spark monitoring information so that it can advise
Spark users where to optimize their configuration parameters ranging from
memory usage, number of cores, etc. Because the initial description of this
ticket is too brief to be clear. Let me explain the use cases for Dr. Elephant
here.
REST API /applications/[app-id]/stages: This useful endpoint provides a list of
all stages for a given application. The current Spark version already provides
it.
In order to debug if there exists a skew issue, a downstream product also needs:
- taskMetricsSummary: It includes task metric information such as
executorRunTime, inputMetrics, outputMetrics, shuffleReadMetrics, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the tasks in a
given stage. The same information shows up in Web UI for a specified stage.
- executorMetricsSummary: It includes executor metrics information such as
number of tasks, input bytes, peak JVM memory, peak execution memory, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the executors used
in a given stage. This information has been developed by [~angerszhuuu] in the
PR he submitted.
We can add the above information to the the json file generated by
/applications/[app-id]/stages. It may double the size of the stages endpoints
file. It should be fine because the current stages json file is not that big.
Here is one sample json file for stages endpoint.
[^lispark230_restapi_ex2_stages_withSummaries.json]
An alternative approach is to add a new REST API such as
"/applications/[app-id]/stages/withSummaries". But it may need a little bit
more code for a new endpoint.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260805#comment-17260805
]
Ron Hu edited comment on SPARK-26399 at 1/7/21, 8:41 PM:
-
Dr. Elephant ([https://github.com/linkedin/dr-elephant]) is a downstream open
source product that utilizes Spark monitoring information so that it can advise
Spark users where to optimize their configuration parameters ranging from
memory usage, number of cores, etc. Because the initial description of this
ticket is too brief to be clear. Let me explain the use cases for Dr. Elephant
here.
REST API /applications/[app-id]/stages: This useful endpoint provides a list of
all stages for a given application. The current Spark version already provides
it.
In order to debug if there exists a skew issue, a downstream product also needs:
- taskMetricsSummary: It includes task metric information such as
executorRunTime, inputMetrics, outputMetrics, shuffleReadMetrics, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the tasks in a
given stage. The same information shows up in Web UI for a specified stage.
- executorMetricsSummary: It includes executor metrics information such as
number of tasks, input bytes, peak JVM memory, peak execution memory, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the executors used
in a given stage. This information has been developed by [~angerszhuuu] in the
PR he submitted.
We can add the above information to the the json file generated by
/applications/[app-id]/stages. It may double the size of the stages endpoints
file. It should be fine because the current stages json file is not that big.
Here is one sample json file for stages endpoint.
[^lispark230_restapi_ex2_stages_withSummaries.json]
An alternative approach is to add a new REST API such as
"/applications/[app-id]/stages/withSummaries". But it may need a little bit
more code for a new endpoint.
was (Author: ron8hu):
Dr. Elephant ([https://github.com/linkedin/dr-elephant]) is a downstream open
source product that utilizes Spark monitoring information so that it can advise
Spark users where to optimize their configuration parameters ranging from
memory usage, number of cores, etc. Because the initial description of this
ticket is too brief to be clear. Let me explain the use cases for Dr. Elephant
here.
REST API /applications/[app-id]/stages: This useful endpoint provides a list of
all stages for a given application. The current Spark version already provides
it.
In order to debug if there exists a skew issue, a downstream product also needs:
- taskMetricsSummary: It includes task metric information such as
executorRunTime, inputMetrics, outputMetrics, shuffleReadMetrics, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the tasks in a
given stage. The same information shows up in Web UI for a specified stage.
- executorMetricsSummary: It includes executor metrics information such as
number of tasks, input bytes, peak JVM memory, peak execution memory, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the executors used
in a given stage. This information has been developed by [~angerszhuuu] in the
PR he submitted.
We can add the above information to the the json file generated by
/applications/[app-id]/stages. It may double the size of the stages endpoints
file. It should be fine because the current stages json file is not that big.
Here is one sample json file "lispark230_restapi_ex2_stages_withSummaries.json"
for stages endpoint. [^lispark230_restapi_ex2_stages_withSummaries.json]
An alternative approach is to add a new REST API such as
"/applications/[app-id]/stages/withSummaries". But it may need a little bit
more code for a new endpoint.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260805#comment-17260805
]
Ron Hu edited comment on SPARK-26399 at 1/7/21, 8:40 PM:
-
Dr. Elephant ([https://github.com/linkedin/dr-elephant]) is a downstream open
source product that utilizes Spark monitoring information so that it can advise
Spark users where to optimize their configuration parameters ranging from
memory usage, number of cores, etc. Because the initial description of this
ticket is too brief to be clear. Let me explain the use cases for Dr. Elephant
here.
REST API /applications/[app-id]/stages: This useful endpoint provides a list of
all stages for a given application. The current Spark version already provides
it.
In order to debug if there exists a skew issue, a downstream product also needs:
- taskMetricsSummary: It includes task metric information such as
executorRunTime, inputMetrics, outputMetrics, shuffleReadMetrics, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the tasks in a
given stage. The same information shows up in Web UI for a specified stage.
- executorMetricsSummary: It includes executor metrics information such as
number of tasks, input bytes, peak JVM memory, peak execution memory, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the executors used
in a given stage. This information has been developed by [~angerszhuuu] in the
PR he submitted.
We can add the above information to the the json file generated by
/applications/[app-id]/stages. It may double the size of the stages endpoints
file. It should be fine because the current stages json file is not that big.
Here is one sample json file "lispark230_restapi_ex2_stages_withSummaries.json"
for stages endpoint. [^lispark230_restapi_ex2_stages_withSummaries.json]
An alternative approach is to add a new REST API such as
"/applications/[app-id]/stages/withSummaries". But it may need a little bit
more code for a new endpoint.
was (Author: ron8hu):
Dr. Elephant (https://github.com/linkedin/dr-elephant) is a downstream open
source product that utilizes Spark monitoring information so that it can advise
Spark users where to optimize their configuration parameters ranging from
memory usage, number of cores, etc. Because the initial description of this
ticket is too brief to be clear. Let me explain the use cases for Dr. Elephant
here.
REST API /applications/[app-id]/stages: This useful endpoint provides a list of
all stages for a given application. The current Spark version already provides
it.
In order to debug if there exists a skew issue, a downstream product also needs:
- taskMetricsSummary: It includes task metric information such as
executorRunTime, inputMetrics, outputMetrics, shuffleReadMetrics, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the tasks in a
given stage. The same information shows up in Web UI for a specified stage.
- executorMetricsSummary: It includes executor metrics information such as
number of tasks, input bytes, peak JVM memory, peak execution memory, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the executors used
in a given stage. This information has been developed by [~angerszhuuu] in the
PR he submitted.
We can add the above information to the the json file generated by
/applications/[app-id]/stages. It may double the size of the stages endpoints
file. It should be fine because the current stages json file is not that big.
Here is one sample json file "lispark230_restapi_ex2_stages_withSummaries.json"
for stages endpoint.
An alternative approach is to add a new REST API such as
"/applications/[app-id]/stages/withSummaries". But it may need a little bit
more code for a new endpoint.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Attachment: lispark230_restapi_ex2_stages_withSummaries.json
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> lispark230_restapi_ex2_stages_withSummaries.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260808#comment-17260808
]
Ron Hu commented on SPARK-26399:
[^lispark230_restapi_ex2_stages_withSummaries.json]
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260805#comment-17260805
]
Ron Hu commented on SPARK-26399:
Dr. Elephant (https://github.com/linkedin/dr-elephant) is a downstream open
source product that utilizes Spark monitoring information so that it can advise
Spark users where to optimize their configuration parameters ranging from
memory usage, number of cores, etc. Because the initial description of this
ticket is too brief to be clear. Let me explain the use cases for Dr. Elephant
here.
REST API /applications/[app-id]/stages: This useful endpoint provides a list of
all stages for a given application. The current Spark version already provides
it.
In order to debug if there exists a skew issue, a downstream product also needs:
- taskMetricsSummary: It includes task metric information such as
executorRunTime, inputMetrics, outputMetrics, shuffleReadMetrics, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the tasks in a
given stage. The same information shows up in Web UI for a specified stage.
- executorMetricsSummary: It includes executor metrics information such as
number of tasks, input bytes, peak JVM memory, peak execution memory, etc. All
in quantile distribution (0.0, 0.25, 0.5, 0.75, 1.0) for all the executors used
in a given stage. This information has been developed by [~angerszhuuu] in the
PR he submitted.
We can add the above information to the the json file generated by
/applications/[app-id]/stages. It may double the size of the stages endpoints
file. It should be fine because the current stages json file is not that big.
Here is one sample json file "lispark230_restapi_ex2_stages_withSummaries.json"
for stages endpoint.
An alternative approach is to add a new REST API such as
"/applications/[app-id]/stages/withSummaries". But it may need a little bit
more code for a new endpoint.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257844#comment-17257844
]
Ron Hu edited comment on SPARK-26399 at 1/3/21, 9:00 PM:
-
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached web UI file !stage_executorSummary_image1.png! , you can see a
sample image file of the "Summery Metrics for Executors" for a stage. Its
corresponding REST API output can be something like:
attach a json file here . [^executorMetricsSummary.json]
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
was (Author: ron8hu):
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached web UI file !stage_executorSummary_image1.png! , you can see a
sample image file of the "Summery Metrics for Executors' for a stage. Its
corresponding REST API output can be something like:
attach a json file here . [^executorMetricsSummary.json]
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorMetricsSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
was:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSummary REST API, which will return executor summary metrics for a
specified stage:
{code:java}
curl http://:18080/api/v1/applicationsexecutorSummary{code}
Add parameters to the stages REST API to specify:
* filtering for task status, and returning tasks that match (for example,
FAILED tasks).
* task metric quantiles, add adding the task summary if specified
* executor metric quantiles, and adding the executor summary if specified
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http:// server>:18080/api/v1/applicationsexecutorMetricsSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257844#comment-17257844
]
Ron Hu edited comment on SPARK-26399 at 1/3/21, 7:46 PM:
-
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached web UI file !stage_executorSummary_image1.png! , you can see a
sample image file of the "Summery Metrics for Executors' for a stage. Its
corresponding REST API output can be something like:
attach a json file here . [^executorMetricsSummary.json]
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
was (Author: ron8hu):
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached file, you can see a sample image file of the "Summery Metrics
for Executors' for a stage. Its corresponding REST API output can be something
like:
attach a json file here . [^executorMetricsSummary.json]
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257844#comment-17257844
]
Ron Hu edited comment on SPARK-26399 at 1/3/21, 7:45 PM:
-
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached file, you can see a sample image file of the "Summery Metrics
for Executors' for a stage. Its corresponding REST API output can be something
like:
attach a json file here . [^executorMetricsSummary.json]
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
was (Author: ron8hu):
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached file, you can see a sample image file of the "Summery Metrics
for Executors' for a stage. Its corresponding REST API output can be something
like:
attach a json file here .
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257844#comment-17257844
]
Ron Hu edited comment on SPARK-26399 at 1/3/21, 7:44 PM:
-
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached file, you can see a sample image file of the "Summery Metrics
for Executors' for a stage. Its corresponding REST API output can be something
like:
attach a json file here .
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
was (Author: ron8hu):
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached file, you can see a sample image file of the "Summery Metrics
for Executors' for a stage. Its corresponding REST API output can be something
like:
{
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"numTasks" : [ 1.0, 1.0, 3.0, 3.0, 4.0 ],
"inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleRead" : [ 0.0, 2.50967876E8, 7.50516665E8, 7.51114124E8,
1.001617709E9 ],
"shuffleReadRecords" : [ 0.0, 740880.0, 2215608.0, 2217351.0, 2957194.0 ],
"shuffleWrite" : [ 0.0, 2.3658701E8, 7.07482405E8, 7.08012783E8, 9.44322243E8
],
"shuffleWriteRecords" : [ 0.0, 726968.0, 2174281.0, 2176014.0, 2902184.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakJVMHeapMemory" : [ 2.09883992E8, 4.6213568E8, 7.5947948E8, 9.8473656E8,
9.8473656E8 ],
"peakJVMOffHeapMemory" : [ 6.0829472E7, 6.1343616E7, 6.271752E7, 9.1926448E7,
9.1926448E7 ],
"peakOnHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakOffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakOnHeapStorageMemory" : [ 7023.0, 12537.0, 19560.0, 19560.0, 19560.0 ],
"peakOffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakOnHeapUnifiedMemory" : [ 7023.0, 12537.0, 19560.0, 19560.0, 19560.0 ],
"peakOffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakDirectPoolMemory" : [ 10742.0, 10865.0, 12781.0, 157182.0, 157182.0 ],
"peakMappedPoolMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakProcessTreeJVMVMemory" : [ 8.296026112E9, 9.678606336E9, 9.684373504E9,
9.691553792E9, 9.691553792E9 ],
"peakProcessTreeJVMRSSMemory" : [ 5.26491648E8, 7.03639552E8, 9.64222976E8,
1.210867712E9, 1.210867712E9 ]
}
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
>
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Attachment: executorMetricsSummary.json
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257846#comment-17257846
]
Ron Hu commented on SPARK-26399:
[^executorMetricsSummary.json]
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: executorMetricsSummary.json,
> stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-26399:
---
Attachment: stage_executorSummary_image1.png
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257845#comment-17257845
]
Ron Hu commented on SPARK-26399:
!stage_executorSummary_image1.png!
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
> Attachments: stage_executorSummary_image1.png
>
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257844#comment-17257844
]
Ron Hu commented on SPARK-26399:
[~angerszhuuu] found that the "executorSummary" field already exists in stage
REST API output. In the existing stage json file, the "executorSummary" field
contains a list of executor metrics for all executors used for a given stage.
In addition to the detailed metrics information for each executor, we also need
the percentile distribution among the executors. This is because we need the
percentile information in order to find out how bad a skew problem is. For
example, we compute the ratio of maximal value over median value and maximal
value over 75th percentile value. If the ratio of max-over-median is equal to
5, there is a skew issue. If the ratio of max-over-75th-percentile is equal to
5, then there is a really bad skew issue.
In the attached file, you can see a sample image file of the "Summery Metrics
for Executors' for a stage. Its corresponding REST API output can be something
like:
{
"quantiles" : [ 0.0, 0.25, 0.5, 0.75, 1.0 ],
"numTasks" : [ 1.0, 1.0, 3.0, 3.0, 4.0 ],
"inputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"inputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputBytes" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"outputRecords" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"shuffleRead" : [ 0.0, 2.50967876E8, 7.50516665E8, 7.51114124E8,
1.001617709E9 ],
"shuffleReadRecords" : [ 0.0, 740880.0, 2215608.0, 2217351.0, 2957194.0 ],
"shuffleWrite" : [ 0.0, 2.3658701E8, 7.07482405E8, 7.08012783E8, 9.44322243E8
],
"shuffleWriteRecords" : [ 0.0, 726968.0, 2174281.0, 2176014.0, 2902184.0 ],
"memoryBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"diskBytesSpilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakJVMHeapMemory" : [ 2.09883992E8, 4.6213568E8, 7.5947948E8, 9.8473656E8,
9.8473656E8 ],
"peakJVMOffHeapMemory" : [ 6.0829472E7, 6.1343616E7, 6.271752E7, 9.1926448E7,
9.1926448E7 ],
"peakOnHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakOffHeapExecutionMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakOnHeapStorageMemory" : [ 7023.0, 12537.0, 19560.0, 19560.0, 19560.0 ],
"peakOffHeapStorageMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakOnHeapUnifiedMemory" : [ 7023.0, 12537.0, 19560.0, 19560.0, 19560.0 ],
"peakOffHeapUnifiedMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakDirectPoolMemory" : [ 10742.0, 10865.0, 12781.0, 157182.0, 157182.0 ],
"peakMappedPoolMemory" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],
"peakProcessTreeJVMVMemory" : [ 8.296026112E9, 9.678606336E9, 9.684373504E9,
9.691553792E9, 9.691553792E9 ],
"peakProcessTreeJVMRSSMemory" : [ 5.26491648E8, 7.03639552E8, 9.64222976E8,
1.210867712E9, 1.210867712E9 ]
}
Since the field name "executorSummary" already exists, we should change this
REST API endpoint name. We may change it to "executorMetricsSummary". The new
REST API can be:
http://:18080/api/v1/applicationsexecutorMetricsSummary
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257639#comment-17257639
]
Ron Hu commented on SPARK-26399:
Hi [~Baohe Zhang] , This ticket proposes a new REST API:
http://:18080/api/v1/applicationsexecutorSummary.
It means to display the percentile distribution of peak memory metrics among
the executors used in a given stage. It can help Spark users debug/monitor a
bottleneck of a stage.
In the ticket https://issues.apache.org/jira/browse/SPARK-32446, it proposed to
add a REST API, which can display the percentile distribution of peak memory
metrics for all executors used in an application. The REST API is:
http://:18080/api/v1/applications///executorSummary
Hence this ticket displays executorSummary for a given stage inside an
application. SPARK-32446 wants to display executorSummary for the entire
application. They are different.
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26341) Expose executor memory metrics at the stage level, in the Stages tab
[ https://issues.apache.org/jira/browse/SPARK-26341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-26341: --- Affects Version/s: (was: 3.1.0) 3.0.1 > Expose executor memory metrics at the stage level, in the Stages tab > > > Key: SPARK-26341 > URL: https://issues.apache.org/jira/browse/SPARK-26341 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, Web UI >Affects Versions: 3.0.1 >Reporter: Edward Lu >Priority: Major > > Sub-task SPARK-23431 will add stage level executor memory metrics (peak > values for each stage, and peak values for each executor for the stage). This > information should also be exposed the the web UI, so that users can see > which stages are memory intensive. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21322) support histogram in filter cardinality estimation
[ https://issues.apache.org/jira/browse/SPARK-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258601#comment-16258601 ] Ron Hu edited comment on SPARK-21322 at 11/21/17 1:43 AM: -- Pull request 19357 was created while there were several dependencies that had not been merged yet at that time. As a result, it has caused many conflicts. I decided to close pull request 19357 and start a clean one. Please review the new pull request 19783 instead. Thanks. was (Author: ron8hu): Pull request 19357 was created while there were several dependencies that had not been defined yet at that time. As a result, it has caused many conflicts. I decided to close pull request 19357 and start a clean one. Please review the new pull request 19783 instead. Thanks. > support histogram in filter cardinality estimation > -- > > Key: SPARK-21322 > URL: https://issues.apache.org/jira/browse/SPARK-21322 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ron Hu > > Histogram is effective in dealing with skewed distribution. After we > generate histogram information for column statistics, we need to adjust > filter estimation based on histogram data structure. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21322) support histogram in filter cardinality estimation
[ https://issues.apache.org/jira/browse/SPARK-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258601#comment-16258601 ] Ron Hu commented on SPARK-21322: Pull request 19357 was created while there were several dependencies that had not been defined yet at that time. As a result, it has caused many conflicts. I decided to close pull request 19357 and start a clean one. Please review the new pull request 19783 instead. Thanks. > support histogram in filter cardinality estimation > -- > > Key: SPARK-21322 > URL: https://issues.apache.org/jira/browse/SPARK-21322 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ron Hu > > Histogram is effective in dealing with skewed distribution. After we > generate histogram information for column statistics, we need to adjust > filter estimation based on histogram data structure. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-22164) support histogram in estimating the cardinality of aggregate (or group-by) operator
Ron Hu created SPARK-22164: -- Summary: support histogram in estimating the cardinality of aggregate (or group-by) operator Key: SPARK-22164 URL: https://issues.apache.org/jira/browse/SPARK-22164 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 2.2.0 Reporter: Ron Hu Histogram is effective in dealing with skewed distribution. After we generate histogram information for column statistics, we need to adjust aggregate (or group-by) cardinality estimation based on equi-height histogram information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17075) Cardinality Estimation of Predicate Expressions
[ https://issues.apache.org/jira/browse/SPARK-17075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181793#comment-16181793 ] Ron Hu commented on SPARK-17075: This jira has been resolved. Pull request 19357 was added here by accident. Pull request 19357 is fore [SPARK-21322]. > Cardinality Estimation of Predicate Expressions > --- > > Key: SPARK-17075 > URL: https://issues.apache.org/jira/browse/SPARK-17075 > Project: Spark > Issue Type: Sub-task > Components: Optimizer >Affects Versions: 2.0.0 >Reporter: Ron Hu >Assignee: Ron Hu > Fix For: 2.2.0 > > > A filter condition is the predicate expression specified in the WHERE clause > of a SQL select statement. A predicate can be a compound logical expression > with logical AND, OR, NOT operators combining multiple single conditions. A > single condition usually has comparison operators such as =, <, <=, >, >=, > ‘like’, etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21322) support histogram in filter cardinality estimation
Ron Hu created SPARK-21322: -- Summary: support histogram in filter cardinality estimation Key: SPARK-21322 URL: https://issues.apache.org/jira/browse/SPARK-21322 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 2.1.0 Reporter: Ron Hu Histogram is effective in dealing with skewed distribution. After we generate histogram information for column statistics, we need to adjust filter estimation based on histogram data structure. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19408) cardinality estimation involving two columns of the same table
[ https://issues.apache.org/jira/browse/SPARK-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-19408: --- Description: In SPARK-17075, we estimate cardinality of predicate expression "column (op) literal", where op is =, <, <=, >, or >=. In SQL queries, we also see predicate expressions involving two columns such as "column-1 (op) column-2" where column-1 and column-2 belong to same table. Note that, if column-1 and column-2 belong to different tables, then it is a join operator's work, NOT a filter operator's work. In this jira, we want to estimate the filter factor of predicate expressions involving two columns of same table. For example, multiple tpc-h queries have this kind of predicate "WHERE l_commitdate < l_receiptdate". was: In SPARK-17075, we estimate cardinality of predicate expression "column (op) literal", where op is =, <, <=, >, or >=. In SQL queries, we also see predicate expressions involving two columns such as "column-1 (op) column-2" where column-1 and column-2 belong to same table. Note that, if column-1 and column-2 belong to different tables, then it is a join operator's work, NOT a filter operator's work. In this jira, we want to estimate the filter factor of predicate expressions involving two columns of same table. > cardinality estimation involving two columns of the same table > -- > > Key: SPARK-19408 > URL: https://issues.apache.org/jira/browse/SPARK-19408 > Project: Spark > Issue Type: Sub-task > Components: Optimizer >Affects Versions: 2.1.0 >Reporter: Ron Hu > > In SPARK-17075, we estimate cardinality of predicate expression "column (op) > literal", where op is =, <, <=, >, or >=. In SQL queries, we also see > predicate expressions involving two columns such as "column-1 (op) column-2" > where column-1 and column-2 belong to same table. Note that, if column-1 and > column-2 belong to different tables, then it is a join operator's work, NOT a > filter operator's work. > In this jira, we want to estimate the filter factor of predicate expressions > involving two columns of same table. For example, multiple tpc-h queries > have this kind of predicate "WHERE l_commitdate < l_receiptdate". -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19408) cardinality estimation involving two columns of the same table
[ https://issues.apache.org/jira/browse/SPARK-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-19408: --- Target Version/s: 2.2.0 (was: 2.3.0) > cardinality estimation involving two columns of the same table > -- > > Key: SPARK-19408 > URL: https://issues.apache.org/jira/browse/SPARK-19408 > Project: Spark > Issue Type: Sub-task > Components: Optimizer >Affects Versions: 2.1.0 >Reporter: Ron Hu > > In SPARK-17075, we estimate cardinality of predicate expression "column (op) > literal", where op is =, <, <=, >, or >=. In SQL queries, we also see > predicate expressions involving two columns such as "column-1 (op) column-2" > where column-1 and column-2 belong to same table. Note that, if column-1 and > column-2 belong to different tables, then it is a join operator's work, NOT a > filter operator's work. > In this jira, we want to estimate the filter factor of predicate expressions > involving two columns of same table. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19408) cardinality estimation involving two columns of the same table
[ https://issues.apache.org/jira/browse/SPARK-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-19408: --- Target Version/s: 2.3.0 (was: 2.2.0) > cardinality estimation involving two columns of the same table > -- > > Key: SPARK-19408 > URL: https://issues.apache.org/jira/browse/SPARK-19408 > Project: Spark > Issue Type: Sub-task > Components: Optimizer >Affects Versions: 2.1.0 >Reporter: Ron Hu > > In SPARK-17075, we estimate cardinality of predicate expression "column (op) > literal", where op is =, <, <=, >, or >=. In SQL queries, we also see > predicate expressions involving two columns such as "column-1 (op) column-2" > where column-1 and column-2 belong to same table. Note that, if column-1 and > column-2 belong to different tables, then it is a join operator's work, NOT a > filter operator's work. > In this jira, we want to estimate the filter factor of predicate expressions > involving two columns of same table. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] (SPARK-19408) cardinality estimation involving two columns of the same table
Title: Message Title Ron Hu created an issue Spark / SPARK-19408 cardinality estimation involving two columns of the same table Issue Type: Sub-task Affects Versions: 2.1.0 Assignee: Unassigned Components: Optimizer Created: 31/Jan/17 01:44 Priority: Major Reporter: Ron Hu In SPARK-17075, we estimate cardinality of predicate _expression_ "column (op) literal", where op is =, <, <=, >, or >=. In SQL queries, we also see predicate expressions involving two columns such as "column-1 (op) column-2" where column-1 and column-2 belong to same table. Note that, if column-1 and column-2 belong to different tables, then it is a join operator's work, NOT a filter operator's work. In this jira, we want to estimate the filter factor of predicate expressions involving two columns of same table.
[jira] [Issue Comment Deleted] (SPARK-17076) Cardinality estimation of join operator
[ https://issues.apache.org/jira/browse/SPARK-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-17076: --- Comment: was deleted (was: Hi, I am out of office 11 /15 through 11/18 with very limited Internet access. My reply to your email will be delayed. Thanks. Best, Ron Hu ) > Cardinality estimation of join operator > --- > > Key: SPARK-17076 > URL: https://issues.apache.org/jira/browse/SPARK-17076 > Project: Spark > Issue Type: Sub-task > Components: Optimizer >Affects Versions: 2.0.0 >Reporter: Ron Hu > > support cardinality estimates for equi-join, Cartesian product join, and > outer join, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710667#comment-15710667 ] Ron Hu commented on SPARK-16026: Hi Reynold, I previously worked on filter cardinality estimation using the old statistics structure. Now I need to refactor my code using the new basic statistics structure we agreed on. As I am traveling on a business trip now, I will resume my work on Monday after I return to Bay Area. Zhenhua is currently busy with some customer tasks this week. He will return to work on CBO soon. > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Labels: releasenotes > Attachments: Spark_CBO_Design_Spec.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17076) Cardinality estimation of join operator
[ https://issues.apache.org/jira/browse/SPARK-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665664#comment-15665664 ] Ron Hu commented on SPARK-17076: Hi, I am out of office 11 /15 through 11/18 with very limited Internet access. My reply to your email will be delayed. Thanks. Best, Ron Hu > Cardinality estimation of join operator > --- > > Key: SPARK-17076 > URL: https://issues.apache.org/jira/browse/SPARK-17076 > Project: Spark > Issue Type: Sub-task > Components: Optimizer >Affects Versions: 2.0.0 >Reporter: Ron Hu > > support cardinality estimates for equi-join, Cartesian product join, and > outer join, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Comment: was deleted (was: Design Specification of Spark Cost-Based Optimization is now posted.) > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17791) Join reordering using star schema detection
[ https://issues.apache.org/jira/browse/SPARK-17791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15616590#comment-15616590 ] Ron Hu commented on SPARK-17791: This JIRA is indeed complementary to Cost-Based Optimizer (or CBO) project. But we need to be more careful, and as a result we should do this together with CBO. Let me explain it below. Previously I commented in SPARK-17626 , we need detailed statistics such as number of distinct values for a join column and number of records of a table in order to decide fact tables and dimension tables. Today we are collecting statistics to reliably predict fact tables and dimension tables in CBO project. In addition, we are estimating selectivity for every relational algebra operator in CBO so that we can give reliable plan cardinality. Without CBO's support, the author currently is using SQL hint to provide predicate selectivity. Note that SQL hint does not work well as it is not automated and it just shows the current weakness of Spark SQL optimizer. The author also uses initial table size to predict fact/dimension tables. As we know, after applying predicate, the relevant records to participate a join may be a small subset of the initial table. Hence initial table size is not a reliable way to decide a star schema. While this PR can show promising performance gain on most tpc-ds benchmark queries as tpc-ds has a well-know star schema, but can this approach still work in real world applications which do not clearly define a star schema? Therefore, I suggest that [SPARK-17791] and [SPARK-17626] should be tightly integrated with CBO project. We are set to release CBO in Spark 2.2. With the progress made so far, we can begin integrating star join reordering in December. The same comment also applies to [SPARK-17626]. My two cents. Thanks. > Join reordering using star schema detection > --- > > Key: SPARK-17791 > URL: https://issues.apache.org/jira/browse/SPARK-17791 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.1.0 >Reporter: Ioana Delaney >Priority: Critical > Attachments: StarJoinReordering1005.doc > > > This JIRA is a sub-task of SPARK-17626. > The objective is to provide a consistent performance improvement for star > schema queries. Star schema consists of one or more fact tables referencing a > number of dimension tables. In general, queries against star schema are > expected to run fast because of the established RI constraints among the > tables. This design proposes a join reordering based on natural, generally > accepted heuristics for star schema queries: > * Finds the star join with the largest fact table and places it on the > driving arm of the left-deep join. This plan avoids large tables on the > inner, and thus favors hash joins. > * Applies the most selective dimensions early in the plan to reduce the > amount of data flow. > The design description is included in the below attached document. > \\ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17626) TPC-DS performance improvements using star-schema heuristics
[
https://issues.apache.org/jira/browse/SPARK-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559044#comment-15559044
]
Ron Hu commented on SPARK-17626:
In the CBO design spec we posted in
https://issues.apache.org/jira/browse/SPARK-16026,
we illustrated a Multi-way Join Ordering Optimization algorithm using dynamic
programming technique. This algorithm should be able to pick up the best join
re-ordering plan. It is possible that the search space is big. We need some
heuristics to reduce the search space.
As Zhenhua pointed out, we can identify all the primary-key/foreign-key joins
as we collect number of distinct values to infer whether or not a join column
is a primary key. If a join relation has primary key join column, then it is a
dimension table. If a join relation has foreign key columns, then it is a fact
table. Once a fact table is identified, we form a star schema by finding out
all the dimension tables that have join conditions with the given fact table.
As for the selectivity hint, we do not need selectivity hint to deal with
comparison expression like:
column_nameconstant_value
where a comparison operator is =, <, <=, >, >=, etc.
This is because, with the histogram we are implementing now in CBO, we can find
the filtering selectivity properly. However, for the following cases, a
selectivity hint will be helpful.
Case 1:
WHERE o_comment not like '%special%request%' /* TPC-H Q13 */
Histogram cannot provide such detailed statistics information for a string
pattern which can a complex regular expression.
Case 2:
WHERE l_commitdate < l_receiptdate /* TPC-H Q4 */
Today we define one-dimensional histogram to keep track the data distribution
of a single column. We do not handle the non-equal relationship between two
columns.
> TPC-DS performance improvements using star-schema heuristics
>
>
> Key: SPARK-17626
> URL: https://issues.apache.org/jira/browse/SPARK-17626
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Ioana Delaney
>Priority: Critical
> Attachments: StarSchemaJoinReordering.pptx
>
>
> *TPC-DS performance improvements using star-schema heuristics*
> \\
> \\
> TPC-DS consists of multiple snowflake schema, which are multiple star schema
> with dimensions linking to dimensions. A star schema consists of a fact table
> referencing a number of dimension tables. Fact table holds the main data
> about a business. Dimension table, a usually smaller table, describes data
> reflecting the dimension/attribute of a business.
> \\
> \\
> As part of the benchmark performance investigation, we observed a pattern of
> sub-optimal execution plans of large fact tables joins. Manual rewrite of
> some of the queries into selective fact-dimensions joins resulted in
> significant performance improvement. This prompted us to develop a simple
> join reordering algorithm based on star schema detection. The performance
> testing using *1TB TPC-DS workload* shows an overall improvement of *19%*.
> \\
> \\
> *Summary of the results:*
> {code}
> Passed 99
> Failed 0
> Total q time (s) 14,962
> Max time1,467
> Min time3
> Mean time 145
> Geomean44
> {code}
> *Compared to baseline* (Negative = improvement; Positive = Degradation):
> {code}
> End to end improved (%) -19%
> Mean time improved (%) -19%
> Geomean improved (%) -24%
> End to end improved (seconds) -3,603
> Number of queries improved (>10%) 45
> Number of queries degraded (>10%) 6
> Number of queries unchanged48
> Top 10 queries improved (%) -20%
> {code}
> Cluster: 20-node cluster with each node having:
> * 10 2TB hard disks in a JBOD configuration, 2 Intel(R) Xeon(R) CPU E5-2680
> v2 @ 2.80GHz processors, 128 GB RAM, 10Gigabit Ethernet.
> * Total memory for the cluster: 2.5TB
> * Total storage: 400TB
> * Total CPU cores: 480
> Hadoop stack: IBM Open Platform with Apache Hadoop v4.2. Apache Spark 2.0 GA
> Database info:
> * Schema: TPCDS
> * Scale factor: 1TB total space
> * Storage format: Parquet with Snappy compression
> Our investigation and results are included in the attached document.
> There are two parts to this improvement:
> # Join reordering using star schema detection
> # New selectivity hint to specify the selectivity of the predicates over base
> tables. Selectivity hint is optional and it was not used in the above TPC-DS
> tests.
> \\
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional
[jira] [Commented] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472000#comment-15472000 ] Ron Hu commented on SPARK-16026: Hi Srinath, Thank you for your comments. Let me answer them one by one. First, we should consider the data shuffle cost. Yes, this is part of phase 2 cost functions in our plan. As we already implemented the phase 1 cost function, we want to contribute our existing development work to Spark community ASAP. We will expand to phase 2 CBO work soon. In phase 2, we will develop cost function for each execution operator. The EXCHANGE operator is one we need to define its cost function. Your suggestion is quite reasonable. Second, we define two statements: (1) ANALYZE TABLE table_name COMPUTE STATISTICS; (2) ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS column-name1, column-name2, …. As you know, the ANALYZE TABLE command collects the auxiliary statistics information. A good DBA needs to monitor the status of the statistics information. I mean there always exists an issue whether or not the statistics data is stale. Hence, we do not want to use the transaction criteria to view statistics data. On the other hand, we may do a little better to make them consistent. One way is to refresh table level statistics when we execute the command "ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS column-name1, column-name2, …. " to collect column level statistics. Third, we do not have default selectivity assumed. In the design spec, we defined how to estimate the cardinality for logical AND operator in section 6. In the future, we may use either 2-dimensional histogram and/or SQL hint to handle the correlation among multiple correlated columns. > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Comment: was deleted (was: Hi Srinath, Thank you for your comments. Let me answer them one by one. First, we should consider the data shuffle cost. Yes, this is part of phase 2 cost functions in our plan. As we already implemented the phase 1 cost function, we want to contribute our existing development work to Spark community ASAP. We will expand to phase 2 CBO work soon. In phase 2, we will develop cost function for each execution operator. The EXCHANGE operator is one we need to define its cost function. Your suggestion is quite reasonable. Second, we define two statements: (1) ANALYZE TABLE table_name COMPUTE STATISTICS; (2) ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS column-name1, column-name2, …. As you know, the ANALYZE TABLE command collects the auxiliary statistics information. A good DBA needs to monitor the status of the statistics information. I mean there always exists an issue whether or not the statistics data is stale. Hence, we do not want to use the transaction criteria to view statistics data. On the other hand, we may do a little better to make them consistent. One way is to refresh table level statistics when we execute the command "ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS column-name1, column-name2, …. " to collect column level statistics. Third, we do not have default selectivity assumed. In the design spec, we defined how to estimate the cardinality for logical AND operator in section 6. In the future, we may use either 2-dimensional histogram and/or SQL hint to handle the correlation among multiple correlated columns. ) > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15471986#comment-15471986 ] Ron Hu commented on SPARK-16026: Hi Srinath, Thank you for your comments. Let me answer them one by one. First, we should consider the data shuffle cost. Yes, this is part of phase 2 cost functions in our plan. As we already implemented the phase 1 cost function, we want to contribute our existing development work to Spark community ASAP. We will expand to phase 2 CBO work soon. In phase 2, we will develop cost function for each execution operator. The EXCHANGE operator is one we need to define its cost function. Your suggestion is quite reasonable. Second, we define two statements: (1) ANALYZE TABLE table_name COMPUTE STATISTICS; (2) ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS column-name1, column-name2, …. As you know, the ANALYZE TABLE command collects the auxiliary statistics information. A good DBA needs to monitor the status of the statistics information. I mean there always exists an issue whether or not the statistics data is stale. Hence, we do not want to use the transaction criteria to view statistics data. On the other hand, we may do a little better to make them consistent. One way is to refresh table level statistics when we execute the command "ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS column-name1, column-name2, …. " to collect column level statistics. Third, we do not have default selectivity assumed. In the design spec, we defined how to estimate the cardinality for logical AND operator in section 6. In the future, we may use either 2-dimensional histogram and/or SQL hint to handle the correlation among multiple correlated columns. > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Attachment: Spark_CBO_Design_Spec.pdf > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Attachment: (was: Spark_CBO_Design_Spec_v_1_2.pdf) > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Attachment: Spark_CBO_Design_Spec_v_1_2.pdf > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec_v_1_2.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Attachment: (was: Spark_CBO_Design_Spec_v_1_2.pdf) > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec_v_1_2.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422375#comment-15422375 ] Ron Hu commented on SPARK-16026: Design Specification of Spark Cost-Based Optimization is now posted. > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec_v_1_2.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Comment: was deleted (was: I will post a pdf version of the design spec soon.) > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec_v_1_2.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Comment: was deleted (was: Design Specification of Spark Cost-Based Optimization) > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec_v_1_2.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-16026: --- Attachment: Spark_CBO_Design_Spec_v_1_2.pdf Design Specification of Spark Cost-Based Optimization > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > Attachments: Spark_CBO_Design_Spec_v_1_2.pdf > > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422346#comment-15422346 ] Ron Hu commented on SPARK-16026: I will post a pdf version of the design spec soon. > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16026) Cost-based Optimizer framework
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422332#comment-15422332 ] Ron Hu commented on SPARK-16026: Sean, I wrote "Design Specification of Spark Cost-Based Optimization" last month and it has been reviewed by Databricks folks like Reynold Xin, Michael Armbrust, Herman, etc. It is a Google document managed by Reynold. I will post a pdf version of this design document soon. I create subtasks for this jira because we will begin checking in code in the coming month. > Cost-based Optimizer framework > -- > > Key: SPARK-16026 > URL: https://issues.apache.org/jira/browse/SPARK-16026 > Project: Spark > Issue Type: New Feature > Components: SQL >Reporter: Reynold Xin > > This is an umbrella ticket to implement a cost-based optimizer framework > beyond broadcast join selection. This framework can be used to implement some > useful optimizations such as join reordering. > The design should discuss how to break the work down into multiple, smaller > logical units. For example, changes to statistics class, system catalog, cost > estimation/propagation in expressions, cost estimation/propagation in > operators can be done in decoupled pull requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-16990) Define the data structure to hold the statistics for CBO
[ https://issues.apache.org/jira/browse/SPARK-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu closed SPARK-16990. -- Resolution: Duplicate > Define the data structure to hold the statistics for CBO > > > Key: SPARK-16990 > URL: https://issues.apache.org/jira/browse/SPARK-16990 > Project: Spark > Issue Type: Sub-task > Components: Optimizer >Affects Versions: 2.1.0 >Reporter: Ron Hu > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
