erenavsarogullari opened a new pull request #35536:
URL: https://github.com/apache/spark/pull/35536
### What changes were proposed in this pull request?
Currently, SQL public Rest API does not expose `node description` and it is
useful to have more details at query level such as:
```
- Join Operators(BHJ, SMJ, SHJ) => when correlating join operator with join
type and which leg is built for BHJ.
- HashAggregate => aggregated keys and agg functions
- List can be extended for other physical operators.
```
Current Sample Json Result:
```
{
"nodeId" : 14,
"nodeName" : "BroadcastHashJoin",
"wholeStageCodegenId" : 3,
"stageIds" : [ 5 ],
"metrics" : [ {
"name" : "number of output rows",
"value" : {
"amount" : "2"
}
}
},
...
{
"nodeId" : 8,
"nodeName" : "HashAggregate",
"wholeStageCodegenId" : 4,
"stageIds" : [ 8 ],
"metrics" : [ {
"name" : "spill size",
"value" : {
"amount" : "0.0"
}
}
}
```
New Sample Json Result:
```
{
"nodeId" : 14,
"nodeName" : "BroadcastHashJoin",
"nodeDesc" : "BroadcastHashJoin [id#4], [id#24], Inner, BuildLeft,
false",
"wholeStageCodegenId" : 3,
"stageIds" : [ 5 ],
"metrics" : [ {
"name" : "number of output rows",
"value" : {
"amount" : "2"
}
}
},
...
{
"nodeId" : 8,
"nodeName" : "HashAggregate",
"nodeDesc" : "HashAggregate(keys=[name#5, age#6, salary#18],
functions=[avg(cast(age#6 as bigint)), avg(salary#18)])",
"wholeStageCodegenId" : 4,
"stageIds" : [ 8 ],
"metrics" : [ {
"name" : "spill size",
"value" : {
"amount" : "0.0"
}
}
}
```
### Why are the changes needed?
It is useful to have more details at query level such as join type, which
leg is built for BHJ, aggregated keys, agg functions etc.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add more coverage to existing UTs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
