[
https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Hu updated SPARK-34488:
---------------------------
Description:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. This information can show whether or not there is a
skewed load on some executors. We list an example in
[^executorMetricsDistributions.json]
We define withSummaries query parameter in the REST API for a specific stage as:
applications/<application_id>/<application_attempt/stages/<stage_id>/<stage_attempt>?withSummaries=[true|false]&quantiles=0.0,0.25,0.5,0.75,1.0
* When withSummaries=true, both task metrics in percentile distribution and
executor metrics in percentile distribution are included in the REST API
output. The default value of withSummaries is false, i.e. no metrics
percentile distribution will be included in the REST API output.
* Query parameter quantiles define the quantiles we use when calculating
metrics distributions. It takes effect only when {{withSummaries=true.}} Its
default is {{0.0,0.25,0.5,0.75,1.0.}}
was:
For a specific stage, it is useful to show the task metrics in percentile
distribution. This information can help users know whether or not there is a
skew/bottleneck among tasks in a given stage. We list an example in
[^taskMetricsDistributions.json]
Similarly, it is useful to show the executor metrics in percentile distribution
for a specific stage. This information can show whether or not there is a
skewed load on some executors. We list an example in
[^executorMetricsDistributions.json]
We define withSummaries query parameter in the REST API for a specific stage as:
applications/<application_id>/<application_attempt/stages/<stage_id>/<stage_attempt>?withSummaries=[true|false]
When withSummaries=true, both task metrics in percentile distribution and
executor metrics in percentile distribution are included in the REST API
output. The default value of withSummaries is false, i.e. no metrics
percentile distribution will be included in the REST API output.
> Support task Metrics Distributions and executor Metrics Distributions in the
> REST API call for a specified stage
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-34488
> URL: https://issues.apache.org/jira/browse/SPARK-34488
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.0.2
> Reporter: Ron Hu
> Priority: Major
> Attachments: executorMetricsDistributions.json,
> taskMetricsDistributions.json
>
>
> For a specific stage, it is useful to show the task metrics in percentile
> distribution. This information can help users know whether or not there is a
> skew/bottleneck among tasks in a given stage. We list an example in
> [^taskMetricsDistributions.json]
> Similarly, it is useful to show the executor metrics in percentile
> distribution for a specific stage. This information can show whether or not
> there is a skewed load on some executors. We list an example in
> [^executorMetricsDistributions.json]
>
> We define withSummaries query parameter in the REST API for a specific stage
> as:
> applications/<application_id>/<application_attempt/stages/<stage_id>/<stage_attempt>?withSummaries=[true|false]&quantiles=0.0,0.25,0.5,0.75,1.0
> * When withSummaries=true, both task metrics in percentile distribution and
> executor metrics in percentile distribution are included in the REST API
> output. The default value of withSummaries is false, i.e. no metrics
> percentile distribution will be included in the REST API output.
> * Query parameter quantiles define the quantiles we use when calculating
> metrics distributions. It takes effect only when {{withSummaries=true.}}
> Its default is {{0.0,0.25,0.5,0.75,1.0.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
