yashisrani opened a new issue, #3628:
URL: https://github.com/apache/dubbo-go/issues/3628
## Description
The observability integration test (`integrate_test.sh`) intermittently
fails with:
```
failed: Prometheus scrape targets are up
observability semantic verification failed for: Prometheus scrape targets
are up
```
This check verifies that all Prometheus scrape targets report a healthy `up`
status. However, it frequently fails even when all other observability checks
pass successfully.
## Evidence of Flakiness
In recent CI runs, the following pattern is observed:
- ✅ Prometheus has scraped `dubbo_provider_requests_succeed_total`
- ✅ Jaeger holds a consumer/provider cross-service trace
- ✅ Grafana data source and dashboard are provisioned
- ❌ **Prometheus scrape targets are up** (FAILED)
The fact that metrics are being scraped and traces are recorded proves that
services are running and exposing metrics. The failure is a
**timing/synchronization issue**, not an actual functional problem.
## Root Cause
Prometheus starts scraping before all instrumented services have fully
registered their `/metrics` endpoints. The verification script checks scrape
target health immediately after service startup, but:
1. Dubbo-go services need time to initialize the metrics exporter
(`metrics/prometheus/registry.go:153-171`)
2. The Prometheus HTTP server binds asynchronously
(`http.Server.ListenAndServe`)
3. The verification script does not wait for all targets to transition to
`up` state
## Affected Files
- `integrate_test.sh` — triggers the integration test suite
- `dubbo-go-samples/observability/` — contains the semantic verification
logic
- `metrics/prometheus/registry.go` — metrics exporter initialization
## Proposed Fix
1. **Add a readiness wait** in the verification script before checking
scrape target status:
```bash
# Wait for all expected targets to report "up" with retry
wait_for_scrape_targets() {
local max_retries=10
local interval=5
for i in $(seq 1 $max_retries); do
if all_targets_up; then
return 0
fi
sleep $interval
done
return 1
}
```
2. **Increase the scrape timeout** in the Prometheus configuration to
accommodate slower service startup:
```yaml
scrape_configs:
- job_name: 'dubbo'
scrape_interval: 5s
scrape_timeout: 10s
```
3. **Add a health check endpoint** to the metrics exporter so Prometheus can
verify readiness before scraping.
## Impact
- Causes false-negative CI failures on otherwise correct PRs
- Requires manual re-runs, wasting CI resources and reviewer time
- Reduces confidence in the observability test suite
## Reproduction
Run the integration test multiple times:
```bash
./integrate_test.sh apache/dubbo-go <sha> main
```
The failure rate is approximately 20-30% based on recent CI history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
