GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/17747
[SPARK-11373] [CORE] Add metrics to the FsHistoryProvider
## What changes were proposed in this pull request?
This adds metrics to the history server, with the `FsHistoryProvider`
metering its operation count and time spent in activities.
The `HistoryServer` sets up the codahale metrics for the Web under metrics/
with metrics/metrics behind metrics, metrics/health any health probes and
metrics/threads a thread dump. There's currently no attempt to hook up JMX,
etc. The Web servlets are the ones tests can easily hit and don't need
infrastructure, so are the good initial first step.
It then instantiated the configured `ApplicationHistoryProvider`
implementation, now invoking the `start(): Option[Source]` method. If this
method
returns an instance of `org.apache.spark.metrics.source.Source`, then it
will
```
def start(): Option[Source] = {
None
}
```
Part of the patch for `HistoryServerSuite` removes the call to
`HistoryServer.initialize()` in the `before` clause. That was a duplicate call,
one which hit the re-entrancy tests on the provider and registry. As well as
cutting it, `HistoryServer.initialize()` has been made idempotent. That should
not be needed -but it will eliminate the problem arising again.
## How was this patch tested?
- New test suite `HistoryMetricSourceSuite` to verify low level metrics, in
particular ability to wire up a lambda expression to a Coda Hale probe.
- The test suite `FsHistoryProviderSuite` queries the FsHistoryProvider
metrics within the existing tests, alongside the existing probes of the
`ApplicationCache` metrics. this not only adds coverage of the metrics, it
allows the metrics to be used within assertions on the behaviour of the History
provider.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/steveloughran/spark
history/SPARK-11373-minimal-hs-metrics
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17747.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17747
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
