GitHub user carsonwang opened a pull request: https://github.com/apache/spark/pull/9297
[SPARK-11206] Support SQL UI on the history server On the live web UI, there is a SQL tab which provides valuable information for the SQL query. But once the workload is finished, we won't see the SQL tab on the history server. It will be helpful if we support SQL UI on the history server so we can analyze it even after its execution. To support SQL UI on the history server, I add an `onOtherEvent` method to the `SparkListener` trait and post all SQL related events to the same event bus. Two SQL events `SparkListenerSQLExecutionStart` and `SparkListenerSQLExecutionEnd` are added. The history server, the standalone Master that rebuilds the web UI, and the event log listener which writes events to the storage are all in the core module. These components have to reference some necessary SQL classes like the SQL events, the SQLTab, etc. It will be best if we can make these components downstream of sql in the future. Currently I have to move these classes to core. This change also make a single SQLTab for all `SQLContext` because they can now share a single `SQLListener`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/carsonwang/spark SqlHistoryUI Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9297.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9297 ---- commit fdf9d28362fe991a1df0d5392c9021db78fa7541 Author: Carson Wang <carson.w...@intel.com> Date: 2015-10-16T07:00:11Z Update SparkListener to handle other events commit ff4075d89b480d99d0320390d00a5b10aadc9a93 Author: Carson Wang <carson.w...@intel.com> Date: 2015-10-20T06:06:07Z Write sql events to event log commit b9870e6579a59628c82470e55f2cb6c4ec8fa2a7 Author: Carson Wang <carson.w...@intel.com> Date: 2015-10-21T06:29:37Z Move sql UI classes to core commit 3833055b7c94247b700e6fd0565629e71611d307 Author: Carson Wang <carson.w...@intel.com> Date: 2015-10-21T06:32:28Z rename SqlMetricInfo class name commit c0abfc6b7432750812a47790fe51b35dacba7429 Author: Carson Wang <carson.w...@intel.com> Date: 2015-10-22T02:22:51Z Update sql metric param commit a5b1cf42a4847b2c9c9674ce0d6aa4d332498ca2 Author: Carson Wang <carson.w...@intel.com> Date: 2015-10-26T02:09:32Z handle accumulator updates in sql history UI commit 7b30bc736f09600b25772e30636f8a5c19c6db5e Author: Carson Wang <carson.w...@intel.com> Date: 2015-10-26T06:51:25Z Use a single SQL Tab for all SparkContext commit d52288bb2e18a5f0e898110893a091919e13ea84 Author: Carson Wang <carson.w...@intel.com> Date: 2015-10-27T08:49:30Z update style ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org