Hi Shane, (We can take the detailed sw engineering part to a private conversation :-))
This is an interesting aspect. Basically you're arguing that the system should figure out how to provide "stable" pagination, depending on what the original query was. I suspect there are corner cases for this, eg. the interaction between sorting asked for (say, by probe ID descending) and whether that affects this algorithm. Nevertheless, if the underlying data can change while you're paginating (and it can indeed!), the results may be inconsistent anyway. It would be useful for you to use a consistent snapshot (I really mean the daily dumps) if your question is not sensitive to the probes' real-time connected/disconnected status. Cheers, Robert On Thu, Sep 4, 2025 at 1:31 PM Shane Kerr <[email protected]> wrote: > Hello, > > The REST API query to get probes is documented here: > > https://atlas.ripe.net/docs/apis/rest-api-reference/probes > > If you do a lookup which would return too many results, it sets the > "next" field in the return, like this: > > -------------------------------------------------------------------- > { > "count": 55125, > "next": "https://atlas.ripe.net/api/v2/probes/?page=2";, > "previous": null, > ... > -------------------------------------------------------------------- > > The idea is the REST API client will then make a call to the "next" URL > and get the next batch of data; this is basic paging. > > There is no kind of transaction or anything, so the status of the probes > can change between each call. This is fine since the status of the > probes changes independently and you cannot count on their status really. > > One problem is that using "page" for this means it is possible to get > the same probe twice. > > For example, if you are looking for "Connected" probes, and returning > the probes in order of probe ID, and a probe that you have already > returned becomes "Connected", then the answers will all shift up by one, > and you will see the last probe twice. > > This can be avoided by instead of using "page" for the "next" field > using whatever the sorting is for the returned values in the query. I > think by default this is probe ID, so the "next" field could be: > > -------------------------------------------------------------------- > { > "count": 55125, > "next": "https://atlas.ripe.net/api/v2/probes/?id__gt=101";, > "previous": null, > ... > -------------------------------------------------------------------- > > I suspect this would be faster on the back-end too. > > I ended up doing this logic on the client side and it works fine, > although that is for a specific sorting order. It would be more > complicated to do this for all orderings on the server side. > > Cheers, > > -- > Shane > ----- > To unsubscribe from this mailing list or change your subscription options, > please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ > As we have migrated to Mailman 3, you will need to create an account with > the email matching your subscription before you can change your settings. > More details at: https://www.ripe.net/membership/mail/mailman-3-migration/ >
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
