GitHub user carsonwang opened a pull request:
https://github.com/apache/spark/pull/10061
[SPARK-11206] Support SQL UI on the history server (resubmit)
On the live web UI, there is a SQL tab which provides valuable information
for the SQL query. But once the workload is finished, we won't see the SQL tab
on the history server. It will be helpful if we support SQL UI on the history
server so we can analyze it even after its execution.
To support SQL UI on the history server:
1. I added an onOtherEvent method to the SparkListener trait and post all
SQL related events to the same event bus.
2. Two SQL events SparkListenerSQLExecutionStart and
SparkListenerSQLExecutionEnd are defined in the sql module.
3. The new SQL events are written to event log using Jackson.
4. A new trait SparkHistoryListenerFactory is added to allow the history
server to feed events to the SQL history listener. The SQL implementation is
loaded at runtime using java.util.ServiceLoader.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/carsonwang/spark SqlHistoryUI
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10061.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10061
----
commit fdf9d28362fe991a1df0d5392c9021db78fa7541
Author: Carson Wang <[email protected]>
Date: 2015-10-16T07:00:11Z
Update SparkListener to handle other events
commit ff4075d89b480d99d0320390d00a5b10aadc9a93
Author: Carson Wang <[email protected]>
Date: 2015-10-20T06:06:07Z
Write sql events to event log
commit b9870e6579a59628c82470e55f2cb6c4ec8fa2a7
Author: Carson Wang <[email protected]>
Date: 2015-10-21T06:29:37Z
Move sql UI classes to core
commit 3833055b7c94247b700e6fd0565629e71611d307
Author: Carson Wang <[email protected]>
Date: 2015-10-21T06:32:28Z
rename SqlMetricInfo class name
commit c0abfc6b7432750812a47790fe51b35dacba7429
Author: Carson Wang <[email protected]>
Date: 2015-10-22T02:22:51Z
Update sql metric param
commit a5b1cf42a4847b2c9c9674ce0d6aa4d332498ca2
Author: Carson Wang <[email protected]>
Date: 2015-10-26T02:09:32Z
handle accumulator updates in sql history UI
commit 7b30bc736f09600b25772e30636f8a5c19c6db5e
Author: Carson Wang <[email protected]>
Date: 2015-10-26T06:51:25Z
Use a single SQL Tab for all SparkContext
commit d52288bb2e18a5f0e898110893a091919e13ea84
Author: Carson Wang <[email protected]>
Date: 2015-10-27T08:49:30Z
update style
commit 7a2acedfc524e5c5887bd783e6fbbe289313306a
Author: Carson Wang <[email protected]>
Date: 2015-10-28T05:08:23Z
Throws exception for unknown events
commit caab0bab0299d4eb985b2e5e68cc5813faac6dfb
Author: Carson Wang <[email protected]>
Date: 2015-10-28T05:39:07Z
Fix build error
commit 0af5afeafe894614e7c1cb83f343db0a0869ad77
Author: Carson Wang <[email protected]>
Date: 2015-10-28T06:07:28Z
Fix style
commit 8d565f24ab365482333d4ff2331c72b01d02358a
Author: Carson Wang <[email protected]>
Date: 2015-11-06T02:46:35Z
Avoid moving the sql classes
commit 1954d71d674dc35659cba6f9e3b56d42a756778e
Author: Carson Wang <[email protected]>
Date: 2015-11-06T06:15:59Z
code clean
commit 927bae84244f900724c3865aac969e4639594760
Author: Carson Wang <[email protected]>
Date: 2015-11-06T07:17:45Z
code clean
commit b03d98bcb6854a01b65fd5e43368580dec192482
Author: Carson Wang <[email protected]>
Date: 2015-11-09T05:24:04Z
minor fix
commit 51f913bed91dd95029b404092e8cd7c25b02cad6
Author: Carson Wang <[email protected]>
Date: 2015-11-10T02:04:09Z
Fix unit tests
commit bca3f5ffca9a9e6f5c56633812432892af66b4e7
Author: Carson Wang <[email protected]>
Date: 2015-11-12T06:07:36Z
Address comments
commit 60033f8da8bd564c01811c25dc9dbfdf897896b7
Author: Carson Wang <[email protected]>
Date: 2015-11-12T06:28:31Z
Fix RAT test
commit fe5c16529d821098658cb61ca0f0a21d2a568270
Author: Carson Wang <[email protected]>
Date: 2015-11-12T06:36:20Z
Fix style
commit 8d94707d4e44741d5e3c24a146d2bf526cf884ce
Author: Carson Wang <[email protected]>
Date: 2015-11-19T03:24:50Z
Address vanzin's comments
commit 56f24bafb3f5b9306bf393220d018e0620f1296d
Author: Carson Wang <[email protected]>
Date: 2015-11-19T07:37:23Z
Address hao's comments
commit 690277e37c76b98ba57bc4bd5b0ae2c87669578d
Author: Carson Wang <[email protected]>
Date: 2015-11-20T02:39:40Z
Remove one empty line
commit 5270209504a337b50cdbafe8cc3e8f83c6ea9745
Author: Carson Wang <[email protected]>
Date: 2015-12-01T06:04:40Z
clear sqlListener
commit 8222a0c61a1b05acde18c9d1825528805dedd16f
Author: Carson Wang <[email protected]>
Date: 2015-12-01T06:39:02Z
Merge branch 'master' into SqlHistoryUI
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
