andrewmusselman opened a new issue, #33:
URL: https://github.com/apache/tooling-agents/issues/33
## Summary
The bundle and single-section audit agents apply `include_files` glob
patterns to **every** namespace they load, including supplemental namespaces
threaded in via the orchestrator's `supplementalData` input. Discovery
generates source-code-shaped patterns (e.g., `airflow-core/src/**`) that don't
match files in supplemental namespaces (e.g., `AGENTS.md`), so guidance docs
intended to provide audit context are silently filtered out before reaching
Opus.
## Symptom
A user stores project guidance documents (`AGENTS.md`, `security_model.rst`,
threat-model notes, etc.) in a CouchDB namespace like `audit_guidance:airflow`
with keys like `AGENTS.md`, then passes `supplementalData:
audit_guidance:airflow` to the orchestrator expecting that file to be in scope
for every audit Opus call.
In practice, the file only loads for ASVS sections whose pass has an empty
`include_files` list — `chapter_general` passes for unassigned sections, or all
passes when running with `discover=false`. For sections in domain-specific
discovery passes (the majority on a real codebase at L3), the supplemental key
doesn't match the source-code globs and gets filtered out silently.
The result is inconsistent calibration across the same audit run: some
sections see the guidance and produce findings that respect the project's
documented security model; others audit blind and flag things the project
explicitly disclaims as non-vulnerabilities (e.g., DFP/Triggerer database
access, Scheduler-not-executing-user-code design, multi-team isolation gaps
that are tracked as known limitations).
This is silent — no error, no log entry showing how many supplemental files
survived the filter, no indication in the consolidated.md that the model was
working without context.
## Root Cause
`asvs_bundle.py:538-556`:
```python
all_files = {}
for ns in namespaces:
ns_store = data_store.use_namespace(ns)
keys = ns_store.list_keys()
if include_files: # <- applies uniformly to all
namespaces
keys = [k for k in keys if any(
fnmatch.fnmatch(k, pattern) for pattern in include_files
)]
file_contents = ns_store.get_many(keys) if keys else {}
...
```
`asvs_audit.py:464-485` has the same shape.
The orchestrator builds `namespaces` deterministically:
```python
namespaces = [code_namespace]
if supplemental_data:
for ns in supplemental_data.split(","):
...
namespaces.append(ns)
```
The first entry is always the primary source-code namespace; the rest are
supplemental context. The `include_files` filter is meaningful only for the
primary — it's how discovery scopes audit attention to a domain. Supplemental
namespaces are *additional context* by intent and should not be filtered with
patterns that were never designed to apply to them.
## Proposed Fix
Split the load loop. The first namespace (primary) is filtered as today.
Subsequent namespaces (supplemental) load without filtering.
Codify the convention in code comments:
> The first namespace in `namespaces` is the primary source-code namespace
and is subject to `include_files` filtering. Subsequent namespaces are
supplemental context (from `supplementalData`) and load fully — they're meant
to provide additional documentation, threat-model notes, related-repo overlays,
and similar inputs that should always be in scope regardless of what file scope
the discovery pass produced.
The "No files found" guard should check the primary file count specifically,
not the combined `all_files` size — otherwise an audit could proceed with only
guidance docs and no source code.
Add a per-namespace log line indicating primary vs supplemental and the file
count loaded, so this becomes visible in operator logs.
## Implementation
Two files need the same conceptual change:
### `asvs_bundle.py:538-556`
Replace the existing `for ns in namespaces` loop with:
```python
# The orchestrator's contract: namespaces[0] is the primary source-code
# namespace and is subject to include_files filtering. Subsequent
# namespaces (from supplementalData) are supplemental context and load
# without filtering — guidance docs, threat models, related-repo
# overlays, etc. that should always be in scope.
all_files = {}
primary_file_count = 0
for idx, ns in enumerate(namespaces):
is_primary = (idx == 0)
ns_store = data_store.use_namespace(ns)
keys = ns_store.list_keys()
if is_primary and include_files:
keys = [k for k in keys if any(
fnmatch.fnmatch(k, pattern) for pattern in include_files
)]
print(f" [bundle] namespace '{ns}' (primary): "
f"{len(keys)} keys after include_files filter", flush=True)
else:
scope = "primary" if is_primary else "supplemental"
print(f" [bundle] namespace '{ns}' ({scope}): "
f"{len(keys)} keys (no filter)", flush=True)
file_contents = ns_store.get_many(keys) if keys else {}
for k, v in file_contents.items():
if v is not None:
content = v if isinstance(v, str) else json.dumps(v,
default=str) if v else ""
all_files[k] = content
if is_primary:
primary_file_count += 1
if primary_file_count == 0:
return {"outputText": json.dumps({
"error": f"No files found in namespaces {namespaces}"
})}
```
The "No files found" message text is preserved verbatim so existing
log/inspection tooling that pattern-matches on it (notably
`inspect_audit_findings.py`) keeps working.
### `asvs_audit.py:464-485`
Apply the same change. Note that `asvs_audit.py` returns a plain-string
error rather than JSON, so the error return shape stays as `f"Error: No files
found in namespaces {namespaces}"`.
## Validation
1. Store `AGENTS.md` to `audit_guidance:airflow` via `asvs_guidance_upload`.
2. Run `apache/airflow/airflow-core` audit at L3 with `supplementalData:
audit_guidance:airflow` and `discover=true` (the case that previously dropped
the guidance silently).
3. In the audit logs, every bundle should show:
`[bundle] namespace 'audit_guidance:airflow' (supplemental): 1 keys (no
filter)`
This confirms the supplemental file made it past loading.
4. Spot-check the resulting consolidated report for findings that previously
contradicted `AGENTS.md`'s "What is NOT considered a security vulnerability"
section — particularly findings about Dag File Processor / Triggerer database
access, Scheduler isolation, multi-team task-level isolation. Those should
disappear (or change to "documented design decision, not a vulnerability") in
the new run.
5. As a coarse signal, total findings count should drop relative to a
pre-fix run on the same source — the model now has reason to disclaim a class
of findings rather than report them.
## Side benefits
- Per-namespace logging makes future supplementalData usage debuggable.
Today there is no way to know whether a supplemental file actually reached the
prompt without instrumenting the bundle agent yourself.
- The convention "primary = filtered, supplemental = full" makes
`supplementalData` finally do what its name implies. Future supplemental
sources (related-repo overlays, vendor-specific threat models,
organization-wide guidance) can be added without per-source workarounds.
- Unblocks the `asvs_guidance_upload` workflow in its intended form —
per-repo guidance namespaces consumed by their respective audit runs without
`discover=false` workarounds.
## Out of scope (separate issues)
- Filtering supplemental namespaces by some *other* pattern (e.g., loading
only certain guidance keys for certain ASVS sections). For now, all keys in a
supplemental namespace load for every audit call. Tracked separately if it ever
matters.
- Bundle/audit prompts surfacing supplemental docs explicitly to the model
("the following are project-authored guidance documents; treat them as
authoritative for distinguishing real vulnerabilities from documented design
decisions"). The current prompt just includes them as more files. A prompt edit
to call them out is probably worth it but lives in its own issue.
## Related
- `asvs_guidance_upload` agent (the uploader designed to produce per-repo
`audit_guidance:{repo}` namespaces). This issue unblocks that agent's intended
use case.
- The empty-bundle stubs issue is adjacent but independent — that's about
`asvs_bundle` writing error stubs to CouchDB; this is about its filter behavior
with supplemental namespaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
