andrewmusselman opened a new issue, #48:
URL: https://github.com/apache/tooling-gofannon/issues/48
## Background
`CouchDBService.find()` (`services/database_service/couchdb.py`)
historically defaulted to `limit=10000`. Callers that span a whole namespace —
`list_keys`, `get_all`, `list_namespaces` in `data_store_service.py` — called
it without an explicit limit and were silently truncated at 10,000 rows.
Consequences on any namespace with >10k docs:
- `clear_namespace` deleted only the first 10,000 docs and left the rest as
orphans.
- Every `list_keys`-based read (ASVS discovery file scope, component
detection, audit file reads) saw at most 10,000 files.
This affects any repository with more than 10,000 files. It surfaced when an
OpenOffice audit logged `Cleared 10000 existing files` — a suspiciously round
number that turned out to be the cap, not a real count.
## Fix applied
- `couchdb.find()`: `limit=None` now means "return all matching rows,"
fetched via CouchDB Mango **bookmark pagination** (10,000-row pages until a
short page or a repeated/absent bookmark). Explicit integer limits are
unchanged — they still issue a single bounded request (e.g. `run_registry`'s
`limit=100`).
- `data_store_service.py`: `list_keys`, `get_all`, `list_namespaces` now
pass `limit=None`.
- `base.find()`: accepts `limit=None` as unbounded, so non-CouchDB backends
(memory, etc.) are safe.
Tested in isolation: pagination returns all rows at 44,850 and 130,000 with
no duplicates.
## Why this issue exists (the open risk)
The bookmark pagination reads the continuation token off the query result
object as `result.bookmark`. This matches python-cloudant's documented
behavior, but **has not yet been verified against the actual deployed CouchDB
client on a real large namespace.** If the client surfaces the bookmark under a
different attribute (or not at all), the pagination loop stops after the first
page and degrades to the old capped behavior — no worse than before, but not
fixed. The code is defensive: it stops rather than looping forever if the
bookmark is missing or unchanged.
## Verification owed (do this on the first >10k-file run)
OpenOffice (9,938 files) is under the cap and **cannot** exercise this. Use
a repository known to exceed 10,000 files.
Either:
1. Run a normal audit and watch the log for the file/cleared counts, **or**
2. Direct check from the gofannon environment:
```python
ns = data_store.use_namespace("files:apache/<big-repo>")
print(len(ns.list_keys()))
```
**Pass:** the count reflects the true number (e.g. `47213`).
**Fail:** the count freezes at exactly `10000` for a repo known to be larger.
## Fallback if verification fails
Switch `couchdb.find()`'s `limit=None` path from bookmark pagination to
**skip-based pagination** (`"skip": offset`, incrementing by page size until a
short page). Skip is universally supported by all CouchDB clients, so it cannot
have the surfacing problem. Trade-off: slightly slower on very large namespaces
(the server scans past skipped rows each page), but correct everywhere. This is
a localized change to the one pagination loop.
## Cleanup note (operational, not code)
This fix prevents future orphans but does not retroactively remove orphans
left by past truncated clears. Any namespace previously cleared while holding
>10k docs still has stale documents. After deploying, those namespaces should
be cleaned once — either a manual CouchDB wipe of the affected `files:*`
namespaces, or a `clearCache=true` run (now that `clear` paginates, it will
delete all of them).
## Deploy paths
- `services/database_service/couchdb.py`
- `services/database_service/base.py`
- `services/data_store_service.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
