Eren Avsarogullari created SPARK-38222:
------------------------------------------
Summary: Expose nodeDesc attribute in SQL Rest API
Key: SPARK-38222
URL: https://issues.apache.org/jira/browse/SPARK-38222
Project: Spark
Issue Type: Task
Components: SQL
Affects Versions: 3.2.0
Reporter: Eren Avsarogullari
Currently, SQL public Rest API does not expose node description and it is
useful to have nodeDesc attribute at query level to have more details such as:
{code:java}
- Join Operators(BHJ, SMJ, SHJ) => when correlating join operator with join
type and which leg is built for BHJ.
- HashAggregate => aggregated keys and agg functions
- List can be extended for other physical operators.{code}
*Current Sample Json Result:*
{code:java}
{
"nodeId" : 14,
"nodeName" : "BroadcastHashJoin",
"wholeStageCodegenId" : 3,
"stageIds" : [ 5 ],
"metrics" : [ {
"name" : "number of output rows",
"value" : {
"amount" : "2"
}
}
},
...
{
"nodeId" : 8,
"nodeName" : "HashAggregate",
"wholeStageCodegenId" : 4,
"stageIds" : [ 8 ],
"metrics" : [ {
"name" : "spill size",
"value" : {
"amount" : "0.0"
}
}
} {code}
*New* {*}Sample Json Result{*}{*}:{*}
{code:java}
{
"nodeId" : 14,
"nodeName" : "BroadcastHashJoin",
"nodeDesc" : "BroadcastHashJoin [id#4], [id#24], Inner, BuildLeft, false",
"wholeStageCodegenId" : 3,
"stageIds" : [ 5 ],
"metrics" : [ {
"name" : "number of output rows",
"value" : {
"amount" : "2"
}
}
},
...
{
"nodeId" : 8,
"nodeName" : "HashAggregate",
"nodeDesc" : "HashAggregate(keys=[name#5, age#6, salary#18],
functions=[avg(cast(age#6 as bigint)), avg(salary#18)])",
"wholeStageCodegenId" : 4,
"stageIds" : [ 8 ],
"metrics" : [ {
"name" : "spill size",
"value" : {
"amount" : "0.0"
}
}
} {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
