Trust but verify.
We monitor and alert on:
- Every API connection we rely on for production
- Every SMSC connection
- Call Metrics
- Inbound and Outbound SMS Volume against trailing 3 weeks of data on
the same day and time to be alerted of spikes or unexpected drops
- Test inbound and outbound SMS regularly on every DID when there has
not been recent activity
- Test inbound calling to every DID when there has not been recent
activity
- Test outbound carrier calling endpoints to ensure they are responding
and reachable
- automatedly review logs for problem indicators, spikes in errors, etc
- and other stuff!
Successes:
- Detected a situation where a tier 0 carrier was completely unaware
that there would be 20-40 minute drops in a bind to their inbound
short code carrier/partner that we highlighted and detected based on
our trailing 3 weeks of comparative data based on a small volume.
- Alerted staff to periods of API unavailability that their own
monitoring missed, due to a Web Application Firewall bug
- Caught countless situations where a number ported away, or an NNID
assignment was "lost" and our carrier did not detect nor notify us of
these actions
I'm fascinated by the automation of call quality and packet loss in calls,
and look forward to hearing more from other people about how and what they
are monitoring, and stories about how that proactive testing has found
issues and improved customer experience.
:-)
Beckman, operationally nerdy
On Wed, 4 Nov 2026, Mike Hammett via VoiceOps wrote:
Hi all,
We just chased two weeks of intermittent inbound RTP loss (carrier toward our
SBC, business hours only) that nothing we had could see. Our SBC's counters and
VoIPmonitor agreed on what arrived; the carrier's egress probe showed zero
loss, so the loss was somewhere on the public path between them, and the
carrier called it clean. We are also mid-migration from Metaswitch to
NetSapiens, so whatever we set up has to make sense on both.
Before this, we had VoIPmonitor on a mirror of the SBC traffic and Smokeping
and LibreNMS watching our own routers, which told us nothing about the carrier
path. What we are putting in now: MOS and loss alerts in VoIPmonitor, Smokeping
with call-shaped pings (1 per second, 172 bytes, EF) to every hop toward each
carrier that answers, and SIP OPTIONS every minute to the carriers' SBCs (they
answer with a 500, which is fine; it is a timed reply over the real path). The
gap is anything that measures inbound traffic to us continuously.
I am after persistent monitoring people have left running for months, not
one-off tests. A few things I would like to hear about:
- Scheduled test calls through the carrier, scored by your passive monitor. Who
places them, how often, and did it ever catch something before customers did?
- Carriers that give you ongoing quality data: RTCP from their side, a looking
glass, per-trunk stats. Does anyone get this without a fight?
- Measuring one-way loss toward you. A VM near the carrier running irtt or
similar? Something a NetSapiens shop already does out of the box?
- OPTIONS to carrier SBCs as a standing probe. Any carrier pushback?
The 2023 "troubleshooting RTP loss" thread covered the one-off side (mtr, sized
pings, Telchemy). I will summarize the replies and add them to the list.
Thanks,
Mike
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
---------------------------------------------------------------------------
Peter Beckman Internet Guy
[email protected] https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list -- [email protected]
https://lists.voiceops.org/postorius/lists/voiceops.voiceops.org/
To unsubscribe send an email to [email protected]