Hi Malte,
Thank you for the long email, I will try to answer you as best I can.
On 15/11/2021 08:32, Malte Appel wrote:
Hi everyone,
sorry in advance for the long mail.
tl;dr: Anchor API and UI give inconsistent results. Some anchor mesh
measurements could be fixed, some might target non-anchors. Not sure
what to do
about it.
I am currently working a lot with RIPE Atlas and recently wanted to
use the
anchors and their mesh measurements in particular. I wanted to answer
two simple
queries:
1. Get a list of all active anchors
2. Get a list of all active anchor mesh and probe measurements
(traceroute,
for my particular use case)
However, while trying to answer these queries, I stumbled upon quite some
inconsistencies, depending on which interface / API is used. As far as
I can
tell there are four ways in which one could technically answer query 1:
Going forward, the API is going to be the absolute source of truth. In
the (near) future the web content will be all based off the data from
the API. We are aware of inconsistencies between the web content and the
API but these will most likely not be fixed until we switch the pages
over to using the API instead.
1. Look at the website: https://atlas.ripe.net/anchors/list/
-> 840 results
2. Query the /anchors API with attribute is_disabled = false:
https://atlas.ripe.net/api/v2/anchors/?is_disabled=false
-> 844 results
3. Query the /probes API with attributes is_anchor = true and status
= 1
(Connected):
https://atlas.ripe.net/api/v2/probes/?is_anchor=True&status=1
-> 729 results
4. Query the /probes API with attributes tags = system-anchor and
status = 1
(Connected):
https://atlas.ripe.net/api/v2/probes/?tags=system-anchor&status=1
-> 729 results
Methods 3 and 4 are actually consistent!
The anchor tag is based on the value of is_anchor so these should indeed
be identical.
The main discrepancy is between the /anchors and /probes API: 116
anchors that
are listed on the webpage and/or /anchors API as ‘active’ are inactive
(97
abandoned; 19 disconnected at time of writing) according to their
respective
/probes entry.
Active and Inactive are anchor data points while connected/abandoned are
probe data points.
This means in short that anchors can be active while the probe is
considered abandoned.
We are rethinking the whole connected/disconnected and abandoned
terminology but for now these will be different concepts.
I understand that disconnects might be temporary, but some anchors
seem to be
inactive for years (at least according to their status) and are still
listed as
active.
It does make sense to mark an anchor as inactive when the probe becomes
abandoned so I will implement that soon. This will make the results a
little more predictable and logical.
I have attached a text file with some notes that go deeper into the
differences,
but might be hard to read.
For query 2, I faced a similar situation:
1. Look at the website: https://atlas.ripe.net/anchors/list/full/
Anchoring Mesh IPv4: 840
Anchoring Mesh IPv6: 739
Anchoring Probes IPv4: 840
Anchoring Probes IPv6: 705
Total: 3124
2. Query the /anchor-measurements API
Anchoring Mesh IPv4: 849
Anchoring Mesh IPv6: 743
Anchoring Probes IPv4: 902
Anchoring Probes IPv6: 753
Total: 3247
3. Query the /measurements API with attributes status = 2 (Ongoing),
type = traceroute and corresponding attributes:
Anchoring Mesh IPv4: af = 4; tags=anchoring,mesh
Anchoring Mesh IPv6: af = 6; tags=anchoring,mesh
Anchoring Probes IPv4: af = 4; tags=anchoring,probes
Anchoring Probes IPv6: af = 6; tags=anchoring,probes
For example:
https://atlas.ripe.net/api/v2/measurements/?status=2&type=traceroute&af=6&tags=anchoring,probes
Anchoring Mesh IPv4: 1026
Anchoring Mesh IPv6: 902
Anchoring Probes IPv4: 668
Anchoring Probes IPv6: 619
Total: 3215
These results are even more mixed:
- Tags can be inconsistent: Some measurements have none, some have the
‘probes’ or ’mesh’ tag, but miss the ’anchoring’ tag.
The anchoring tag is removed when an anchor becomes decommissioned so
it's correct that some measurements do not have this tag as only the
'active' set of measurements has the anchoring tag applied.
- Some anchors have multiple measurements (especially probes
measurements), of
which most actually are run by the same set of probes, i.e., they are
duplicates.
This is a known problem and I realised that the script that cleans this
up hasn't been run for a while. I will do this soon so the duplicates
will disappear.
- Which measurements are contained in which of the three result sets
is very
mixed, maybe I should draw a Venn diagram :)
Finally, I looked at the consistency of the IP addresses of the
/anchors API
(ip_v4 and ip_v6), the /probes API (address_v4, address_v6), the DNS
result for
the FQDN of the anchors, and the target IP of the mesh/probes
measurements.
I noticed some problems, since our lab (IIJ) also operates an anchor
(probe 6425
[0]) and we updated the IP address some time ago, but are actually not
reached
by the mesh measurement, because the measurement still targets the old
IP.
When an anchor changes IP addresses after it's been deployed, the
current set of anchoring measurements needs to be stopped and a new set
needs to be created. When checking this, I saw that this hasn't happened
yet for all anchors. I will fix this soon after the RIPE meeting.
This should fix the vast majority of cases where you saw this issue
assuming we are aware of all IP addresses changes.
I attached a CSV that includes the raw data (of measurements with some
form of
problem), but basically there are 93 measurements from connected
anchors that
fail, and out of which 68 (from 29 anchors) could work, if the
measurement would
target the correct IP. These measurements have matching anchor/probe
IPs and DNS
records, so I do not know why the measurement target is stale. There
are some
additional measurements that could work, but it is unclear what the
intended
‘correct’ IP is.
On that note, there are 48 measurements that ‘work‘, i.e., they get a
response
from the target, but it is not clear if the target is the intended
receiver:
- 8 target abandoned anchors
- 18 have different probe and anchor IPs and target one of them
- 21 have the same probe and anchor IP but target something else
Thank you for this list, I will compare it to the dataset I have to make
sure I am correcting all known problems.
Again, I am sorry for this long mail. I understand that RIPE Atlas is
a huge
project that has grown over time so it might be hard to keep some things
synchronized, and some other things might not be easily decidable
(e.g., when to
mark an anchor is inactive).
However, I think especially the IP address of an anchor in the
/anchors and
/probes APIs, in the DNS entry, and the target of the mesh/probes
measurements
need to be consistent. Currently some mesh measurements might target
an entirely
different machine.
Since the introduction of the VM anchors, the network has grown much
faster and unfortunately some data quality checks have not been run or
implemented fully. We are working on this however and we expect to have
most of these issues fixed before the end of the year.
I wanted to bring some attention to this, but not sure what else I can
do as a
user. I don‘t want to complain too much :)
For now I will just use all data sources as input and apply some
sanity checks.
Thank you again for bringing this to our attention. We're working hard
to fix the issues. If you feel I have not addressed some of your issues,
please let me know.
Apologies for these bugs, hopefully they will be permanently fixed in
the near future.
Kind regards,
Johan ter Beest
RIPE Atlas Engineer
Best,
Malte
P.S.: Some feedback on how we can bring the measurement of our anchor
to target
our anchor would be nice though.
See answer inline above on stopping and recreating the active anchor
measurements set
[0] https://atlas.ripe.net/probes/6425
