GitHub user jianqiao opened a pull request:
https://github.com/apache/incubator-quickstep/pull/76
Visualize execution plan DAGs annotated with profiling stats
This PR adds support for visualization of execution plan DAGs. Each
relational operator in the DAG is also annotated with the execution profiling
stats of its work orders. The main changes are
- Added a `getName()` method for each relational operator.
- Implemented visualization code in `utility/ExecutionDAGVisualizer.*`.
- Slightly refactored work order profiling code to record both start time
and end time of each work order.
To enable this feature, set the gflag `visualize_execution_dag` to true,
e.g.
```
quickstep_cli_shell -visualize_execution_dag=true
```
The execution dag will be printed via `stderr` in DOT format. Each
relational operator node contains the following information:
- Relational operator ID, Relational operator name.
- The input relation's name (if it is a stored relation).
- Total execution time of all the work orders of this relational operator.
- Execution time span (start time of the first work order and finish time
of the last work order) of this relational operator.
- Degree of concurrency (total time / span) of this relational operator.
Here's an example output, for SSB query 05:
```
digraph g {
rankdir=BT
node [penwidth=2]
edge [fontsize=16 fontcolor=gray penwidth=2]
0 [ label="[0] SelectOperator input stored relation
[supplier] span: [0ms, 6ms] (0.74%) total: 38ms (0.13%) effective
concurrency: 6.29" style=filled fillcolor="0.004717 0.004717 1.0" ]
1 [ label="[1] SelectOperator input stored relation [part] span:
[0ms, 10ms] (1.25%) total: 273ms (0.90%) effective concurrency: 26.26"
style=filled fillcolor="0.011673 0.011673 1.0" ]
2 [ label="[2] BuildHashOperator span: [10ms, 12ms]
(0.21%) total: 19ms (0.06%) effective concurrency: 11.20" style=filled
fillcolor="0.001494 0.001494 1.0" ]
3 [ label="[3] HashJoinOperator probe side stored relation
[lineorder] span: [17ms, 752ms] (88.04%) total: 29059ms
(95.43%) effective concurrency: 39.53" style=filled fillcolor="1.000000
1.000000 1.0" ]
5 [ label="[5] BuildHashOperator span: [6ms, 9ms] (0.39%) total:
6ms (0.02%) effective concurrency: 1.97" style=filled fillcolor="0.002248
0.002248 1.0" ]
6 [ label="[6] HashJoinOperator span: [748ms, 773ms]
(3.04%) total: 438ms (1.44%) effective concurrency: 17.28" style=filled
fillcolor="0.024394 0.024394 1.0" ]
8 [ label="[8] BuildHashOperator input stored relation
[ddate] span: [8ms, 8ms] (0.05%) total: 0ms (0.00%) effective
concurrency: 1.00" style=filled fillcolor="0.000274 0.000274 1.0" ]
9 [ label="[9] HashJoinOperator span: [767ms, 806ms]
(4.65%) total: 372ms (1.22%) effective concurrency: 9.59" style=filled
fillcolor="0.032002 0.032002 1.0" ]
11 [ label="[11] AggregationOperator span: [792ms, 832ms]
(4.80%) total: 223ms (0.73%) effective concurrency: 5.58" style=filled
fillcolor="0.030163 0.030163 1.0" ]
12 [ label="[12] FinalizeAggregationOperator span: [832ms, 833ms]
(0.02%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled
fillcolor="0.000139 0.000139 1.0" ]
13 [ label="[13] SortRunGenerationOperator span: [834ms, 834ms]
(0.01%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled
fillcolor="0.000030 0.000030 1.0" ]
14 [ label="[14] SortMergeRunOperator span: [834ms, 834ms]
(0.01%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled
fillcolor="0.000043 0.000043 1.0" ]
16 [ label="[16] SelectOperator span: [834ms, 834ms]
(0.00%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled
fillcolor="0.000011 0.000011 1.0" ]
0 -> 6 [ style=dashed ]
0 -> 5 [ ]
1 -> 3 [ style=dashed ]
1 -> 2 [ ]
2 -> 3 [ style=dashed ]
3 -> 6 [ ]
5 -> 6 [ style=dashed ]
6 -> 9 [ ]
8 -> 9 [ style=dashed ]
9 -> 11 [ ]
11 -> 12 [ style=dashed ]
12 -> 13 [ ]
13 -> 14 [ ]
14 -> 16 [ ]
}
```
To convert the DOT description into an image. A quick way is to use online
visualization tools (e.g. http://www.webgraphviz.com, just copy and paste the
graph description into the webpage's textbox and click _Generate Graph!_.
The output can also be redirected to a file and then converted to an image
with the dot tool, e.g.
```
quickstep_cli_shell -visualize_execution_dag=true < some_query.sql 2>
some_graph.dot
dot -Gsize="8,10.5" -Teps some_graph.dot -o some_graph.eps
```
Here eps can be `pdf`, `png`, etc.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-quickstep
execution-dag-visualizer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quickstep/pull/76.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #76
----
commit 4b944328afde90c961122d547c6fa4a8d0230c1c
Author: Jianqiao Zhu <[email protected]>
Date: 2016-08-02T21:57:47Z
Add visualization for execution plan DAGs combined with profiling stats
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
