*TL;DR: I'm unit-testing Prometheus metrics: server's requests count and
latencies. The server is written in Go. Latencies are non-determinstic and
there seems to be no good way to fake clock in Prometheus or Go. Does
Prometheus provide a method similar to CollectAndCompare that lets
validating a value (e.g. latency) against a range, not exact expectation?
If not, would it be a good feature to add to Prometheus? Alternatively, is
there a way to fake clock in Prometheus, so
e.g. InstrumentHandlerDuration would propagate the pre-defined fake
duration? Or, is there a good way to fake clock in Go?*
I'm measuring server's requests count (Counter) and latencies (Summary),
and test both with unit tests. I observe the metrics via
InstrumentHandlerCounter and InstrumentHandlerDuration respectively.
For testing requests count, I hardcode the expectation in a string
constant, and use CollectAndCompare to perform an exact match validation:
// Make some test requests.
..
// Validate them
expectation := strings.NewReader(`
# HELP my_metric_requests_total My help.
# TYPE my_metric_requests_total counter
my_metric_requests_total{code="200"} 2
my_metric_requests_total{code="304"} 1
my_metric_requests_total{code="502"} 1
my_metric_requests_total{code="503"} 1
`)
this.Require().NoError(promtest.CollectAndCompare(myMetricRequestsTotal,
expectation, "my_metric_requests_total"))
I couldn't find a way to do the same for latencies, because they are
non-deterministic. So instead of the one-liner check above I dig into the
internals of the gathered metrics:
// Make some test requests.
hintPrefix := "My test."
...
// Validate them
type codeLabelPair string
type scenarioExpectedSampleCountMap map[codeLabelPair]uint64
expectedSampleCountMap := scenarioExpectedSampleCountMap{
`name:"code" value:"200" `: 3,
`name:"code" value:"304" `: 1,
`name:"code" value:"502" `: 2,
}
reg := prometheus.NewPedanticRegistry()
if err := reg.Register(promRequestsLatency); err != nil {
this.T().Errorf(hintPrefix+" - registering collector failed: %s", err)
}
actualMetricFamilyArr, err := reg.Gather()
if err != nil {
this.T().Errorf(hintPrefix+" - gathering metrics failed: %s", err)
}
assert.Equal(this.T(), 1, len(actualMetricFamilyArr),
hintPrefix+" expects exactly one metric family.")
assert.Equal(this.T(), "request_latencies_in_seconds",
*actualMetricFamilyArr[0].Name,
hintPrefix+" expects the right metric name.")
assert.Equal(this.T(), len(expectedSampleCountMap),
len(actualMetricFamilyArr[0].Metric),
hintPrefix+" expects the right amount of metrics collected and gathered.")
for _, actualMetric := range actualMetricFamilyArr[0].Metric {
// Expect the right sample count.
code := actualMetric.Label[0].String()
expectedSampleCount := expectedSampleCountMap[codeLabelPair(code)]
actualSampleCount := actualMetric.Summary.GetSampleCount()
assert.Equal(this.T(), expectedSampleCount, actualSampleCount, hintPrefix+"
expects the right sample count for "+code)
// Test quantiles.
expectedQuantileKeys := []float64{0.5, 0.9, 0.99}
// Expect the right number of quantiles.
assert.Equal(this.T(), len(expectedQuantileKeys),
len(actualMetric.Summary.Quantile), hintPrefix+" expects the right number
of quantiles.")
// Expect the right quantiles.
// Expect positive quantile values, because latencies are non-zero.
// Don't check the exact values, because latencies are non-deterministic.
for i, quantile := range actualMetric.Summary.Quantile {
assert.Equal(this.T(), expectedQuantileKeys[i], quantile.GetQuantile(),
hintPrefix+" expects the right quantile.")
assert.True(this.T(), quantile.GetValue() > .0, hintPrefix+" expects
non-zero quantile value (latency).")
}
}
This seems to be more complex than it should be. Is there a one-liner way,
similar to the CollectAndCompare call I'm making above to validate requests
count?
Alternatively, is there a way to fake clock in Prometheus, so e.g.
InstrumentHandlerDuration would propagate the pre-defined fake duration?
Or, is there a good way to fake clock in Go? There is an option that
doesn't look safe enough:
https://www.reddit.com/r/golang/comments/30try1/monkey_patching_in_go/
https://news.ycombinator.com/item?id=22442170.
Thanks.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/b440c496-cfb1-4b5f-8fb5-11ca47485a3cn%40googlegroups.com.
