This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-steward.git
The following commit(s) were added to refs/heads/main by this push:
new 976620f feat(security-issue-sync): per-CVE-change pause + 6th
anonymise hygiene gate (#374)
976620f is described below
commit 976620f29e4205ef04781479da472a9fc330f54d
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sat May 30 00:40:09 2026 +0200
feat(security-issue-sync): per-CVE-change pause + 6th anonymise hygiene
gate (#374)
Two behavioural additions to the bulk-mode sync pass that emerged
from a 24-tracker sync session where many CVE-affecting body-field
rewrites would have landed in one yes/no.
1. Per-CVE-change pause (bulk-mode orchestration)
Bulk mode now buckets trackers into two:
- CVE-affecting — any tracker whose proposal touches a body
field that lands in the regenerated CVE JSON (Title, Short
public summary for publish, CWE, Severity, Affected versions,
Reporter credited as, Remediation developer, PR with the fix,
Public advisory URL). Each one walks individually: own
proposal, own confirm, own apply, ascending tracker-number
order. No --bundled override.
- Non-CVE-affecting — label flips, milestone touches,
assignee swaps, board column moves, status-rollup entries,
reporter Gmail drafts, RM hand-off comments. Bundled into
one combined proposal as before.
The five pre-push hygiene gates catch mechanical drift; the
per-tracker walk catches judgment drift (threat-model framing,
credit-line choice, CWE fit). Both are needed because the
record ships to cve.org and stays there.
2. Sixth pre-push hygiene gate — anonymise private-scanner +
internal-finder names
When the tracker's source is a private scanner (internal
SAST, partner-shared scan, unpublished bug-bounty pipeline),
the regenerated CVE JSON's public-facing fields (descriptions,
credits) must NOT carry the scanner product name or the
individual finder's name. The audit-trail surfaces (Security
mailing list thread field, status-rollup comment, Gmail
thread) keep the original references for security-team
auditing. Exempt cases — public HackerOne/huntr.dev URL,
self-credit on a security@ thread, org-disclosed channel —
keep the named credit. New scanner-products.md project-config
template declares the per-project private-scanner token list
and the finder-anonymise policy.
Eval coverage: new step-bulk-orchestration suite with three
cases — all label-only (one bundled), mixed buckets (split),
all CVE-affecting (all walked).
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
---
.claude/skills/security-issue-sync/SKILL.md | 127 +++++++++++++++++--
projects/_template/scanner-products.md | 138 +++++++++++++++++++++
.../evals/security-issue-sync/README.md | 5 +-
.../fixtures/case-1-all-label-only/expected.json | 5 +
.../fixtures/case-1-all-label-only/report.md | 91 ++++++++++++++
.../fixtures/case-2-mixed-buckets/expected.json | 5 +
.../fixtures/case-2-mixed-buckets/report.md | 124 ++++++++++++++++++
.../case-3-all-cve-affecting/expected.json | 5 +
.../fixtures/case-3-all-cve-affecting/report.md | 98 +++++++++++++++
.../fixtures/output-spec.md | 47 +++++++
.../fixtures/step-config.json | 4 +
.../fixtures/user-prompt-template.md | 7 ++
12 files changed, 643 insertions(+), 13 deletions(-)
diff --git a/.claude/skills/security-issue-sync/SKILL.md
b/.claude/skills/security-issue-sync/SKILL.md
index 3083508..6bcbaeb 100644
--- a/.claude/skills/security-issue-sync/SKILL.md
+++ b/.claude/skills/security-issue-sync/SKILL.md
@@ -244,17 +244,90 @@ concurrently, which is exactly what the sync needs.
- Return a **compact structured report** — not a freeform
narrative. The exact shape is below.
-3. **Aggregate and present one combined proposal.** Once all
- subagents return, fold their reports into one table / numbered
- proposal covering every issue, grouped so the user can confirm
- with `all`, `NN:all`, `NN:1,3`, or per-issue subsets (see the
- existing apply-loop conventions). Only after the user confirms
- does the orchestrator apply changes.
-
-4. **Apply sequentially, not in parallel.** Even though assessment
- ran in parallel, the apply phase must be sequential so
- `gh`-rate-limit surprises, partial failures, and user interrupts
- stay legible. Do not spawn subagents for the apply phase.
+3. **Bucket trackers by CVE-record impact.** A tracker's proposed
+ changes fall into one of two buckets:
+
+ - **CVE-affecting** — any proposal that changes a body field
+ whose value lands in the regenerated CVE JSON pushed to
+ Vulnogram. Concretely: *Title* (issue title; ships into
+ `containers.cna.title`), *Short public summary for publish*,
+ *CWE*, *Severity*, *Affected versions*, *Reporter credited
+ as*, *Remediation developer*, *PR with the fix*, *Public
+ advisory URL*. Also: any change to the issue title itself
+ (the generator reads it verbatim into `title`). The
+ [pre-push hygiene gates in Step 5b 1b](#decision-flow) all
+ scan fields in this bucket; the bucket exists for the same
+ reason the gates do — these are the values that ship to
+ `cve.org` and stay there.
+ - **Non-CVE-affecting** — label flips, milestone touches,
+ assignee swaps, project-board column moves, status-rollup
+ entries, reporter Gmail drafts, RM hand-off comments
+ (template-bodied, no per-tracker CVE content). These
+ change tracker state but do not alter the published CVE
+ record.
+
+4. **Walk CVE-affecting trackers individually; bundle the rest.**
+ The two buckets are presented to the user differently:
+
+ - **Non-CVE-affecting bucket** — fold into one combined
+ proposal, same shape as before this change. The user
+ confirms once with `all`, `NN:all`, `NN:1,3`, or per-issue
+ subsets, and the orchestrator applies them sequentially.
+ This bucket is bundled because the actions are reversible,
+ low-blast-radius, and do not leak into public CVE surfaces.
+ - **CVE-affecting bucket** — walk **one tracker at a time**,
+ even when many trackers are in the bulk run. For each
+ CVE-affecting tracker the orchestrator:
+ 1. Presents a self-contained per-tracker proposal —
+ CVE ID, gate-failure summary, every proposed body-field
+ update with old / new value side by side, the planned
+ regen+push action, any deferral conditions. The proposal
+ is the only context the user needs to read to confirm.
+ 2. Waits for explicit user confirmation (`OK`, item subset,
+ free-form edits, or `skip`). A confirmed bundle from the
+ previous tracker does **not** carry to the next.
+ 3. Applies the confirmed items + Step 5/5b regen+push for
+ that one tracker, then moves to the next.
+
+ **Why the per-tracker walk for CVE-affecting changes.** A
+ bundled confirmation across N CVE-affecting trackers
+ compresses N independent CVE records into one yes/no, which
+ makes it too easy to nod through a body-field rewrite that
+ ships to `cve.org` (where it stays — `cve.org` records are
+ public-by-default and edits land as new revisions, not silent
+ overwrites). The five pre-push hygiene gates in
+ [Step 5b 1b](#decision-flow) catch *mechanical* drift (bare
+ CWE, missing upgrade target, etc.) but they cannot catch
+ *judgment* drift — whether the team agrees the summary's
+ threat model is right, whether the credit line names the
+ right person, whether the CWE choice fits the patch's
+ nature. Those decisions belong with the user, one record at
+ a time. The non-CVE-affecting bucket does not share this
+ property (a wrong label is reverted with a single `gh issue
+ edit`; a wrong public CVE summary requires a record
+ revision that downstream feeds re-pick-up).
+
+ **No `--bundled` override.** This is the default and only
+ mode for the CVE-affecting bucket. The user can still
+ `skip` a tracker, or interrupt mid-walk and ask the
+ orchestrator to abort the rest; what they cannot do is
+ collapse multiple CVE pushes into one confirmation. The
+ round-trip cost (one extra confirmation prompt per CVE) is
+ the point: it forces a re-read of the per-tracker proposal
+ right before the push lands on `cve.org`.
+
+ **Walk order.** Within the CVE-affecting bucket, walk
+ trackers in **ascending tracker-number order** unless the
+ user names a different order at confirmation. Deterministic
+ order makes the walk predictable across reruns and lets the
+ user say *"do #232 last, I want to think about it"* without
+ the orchestrator inventing a heuristic.
+
+5. **Apply sequentially, not in parallel.** Even though
+ assessment ran in parallel, the apply phase must be
+ sequential so `gh`-rate-limit surprises, partial failures,
+ and user interrupts stay legible. Do not spawn subagents for
+ the apply phase.
### Subagent report shape
@@ -765,6 +838,7 @@ update, label change, or next-step recommendation in Step 2:
| The *"Short public summary for publish"* body field is populated but does
**not** state the triggering conditions — the rendered text describes the bug
mechanism without identifying (a) the attacker role / capability, (b) the
deployment configuration that has to be active, OR (c) the action the attacker
takes against which surface. Detector heuristic: scan the summary for any of
these phrases — *"an authenticated [\\w ]+ user"*, *"a Dag author"*, *"an
attacker with"*, *"a user able to" [...]
| The tracker is an **incomplete-fix follow-up to another CVE** — detected by
any of: the rollup or body mentions *"incomplete fix for `CVE-YYYY-NNNNN`"* /
*"follow-up to `CVE-YYYY-NNNNN`"* / *"sibling tracker"*; the title contains a
*"(incomplete fix for `CVE-YYYY-NNNNN`)"* parenthetical; the `affected[]` array
names a different `packageName` than the referenced prior CVE; OR the tracker
was opened as a split from a closed-`announced` tracker whose CVE is already
PUBLISHED — **AND** the [...]
| The *"CWE"* body field is populated with a bare `CWE-NNN` token (no
description text) — e.g. `CWE-22` or `CWE-502` alone, without the canonical
short description that follows in the format `CWE-NNN: <Title>` | Propose
expanding the field to `CWE-NNN: <Canonical Title>` per the MITRE CWE catalog
(e.g. `CWE-22: Improper Limitation of a Pathname to a Restricted Directory
('Path Traversal')`, `CWE-502: Deserialization of Untrusted Data`, `CWE-601:
URL Redirection to Untrusted Site ('Open R [...]
+| The tracker's *Security mailing list thread* body field references a
**private scanner product** (declared in
[`<project-config>/scanner-products.md`](../../../<project-config>/scanner-products.md)
— e.g. internal SAST, partner-shared scan, unpublished bug-bounty pipeline)
**AND** the *Reporter credited as* body field names a person rather than
`anonymous` / a public handle, **AND** there is no signal the finder consented
to public credit (no inbound `security@` message from them under [...]
| The **issue title** contains adopter-specific or internal noise that would
otherwise ship to the public CVE record — leading or trailing project-name
tokens (e.g. ``Apache Airflow:`` / ``in Apache Airflow`` / ``(Apache Airflow
X.Y)``), internal split markers (``(split from #NNN)`` / ``(split for scope
clarity from #NNN)``), report-form classifiers (``[ Security Report ]`` /
``[Security Issue]``), external-tracker IDs in parentheses or brackets
(``[GHSA-xxxx-xxxx-xxxx]``, ``(ZDRES-NNNNN [...]
| A release carrying the fix has shipped. Detection is **scope-dependent** —
different scope labels on a project can ride different release trains, each
with its own *"is it released?"* signal (which artifact registry to consult,
what to query, how to map a tracker's milestone to that registry,
partial-release edge cases). The per-scope detection recipe lives in
[`<project-config>/scope-labels.md` — *Detecting that a fix release has
shipped*](../../../<project-config>/scope-labels.md#det [...]
| GHSA state transition (opened, accepted, published, rejected) in a
GHSA-forwarded email | If the GHSA is closed as "not accepted" but the security
team accepted the report on `security@`, flag the divergence in the status
comment so it is not lost. |
@@ -2899,7 +2973,7 @@ Step 6 below describes how to verify the state advance
landed
CVE worthy). There is no record to push to.
1b. **Pre-push hygiene-gate scan.** Before any push call, re-scan
- the JSON about to be pushed for the five pre-push gates that
+ the JSON about to be pushed for the six pre-push gates that
make the published CVE record user-facing:
- **Title strip cascade** — `containers.cna.title` must have
@@ -2927,6 +3001,35 @@ Step 6 below describes how to verify the state advance
landed
- **CWE field has the long-form description** —
`problemTypes[0].descriptions[0].description` must be in
the `CWE-NNN: <Title>` shape, not a bare `CWE-NNN` token.
+ - **Anonymise private-scanner and internal-finder names**
+ — when the tracker's source is a private scanner, an
+ internal-partner-shared scan, or an unpublished bug-bounty
+ pipeline (signal: the *Security mailing list thread* body
+ field references a scanner product name or names an
+ individual reporter who arrived through a private channel
+ rather than `security@`), the regenerated JSON must NOT
+ carry the scanner product name or the individual finder's
+ name in any public-facing field. Scan
+ `containers.cna.descriptions[].value` (the public summary)
+ and `containers.cna.credits[].value` for: known
+ scanner-product tokens declared in
+
[`<project-config>/scanner-products.md`](../../../<project-config>/scanner-products.md)
+ (e.g. `Mythos`, `<vendor> SAST`, `<scanner-tool>`); and
+ for `credits[].value` entries that match a person-name
+ pattern (`<First> <Last>` shape) when the
+ `discovery-channel` signal is private. On match: propose
+ replacing the credit value with `anonymous` and stripping
+ the scanner product name from the summary text. The
+ tracker's *Security mailing list thread* body field stays
+ unchanged (it is the private audit trail); only the
+ CVE-record JSON gets the anonymise scrub. **Public
+ bug-bounty submissions and named ASF-community reporters
+ are exempt** (the scrubber must not anonymise a credit
+ that was already public elsewhere — HackerOne report URL,
+ huntr.dev public report, the reporter's own self-disclosure
+ on `security@` with their real name). Rationale, examples,
+ and the full opt-in / opt-out matrix live in
+
[`<project-config>/scanner-products.md`](../../../<project-config>/scanner-products.md).
When any gate fails the JSON the regen just produced, the
right recovery is **not** to push — fix the underlying body
diff --git a/projects/_template/scanner-products.md
b/projects/_template/scanner-products.md
new file mode 100644
index 0000000..19c78c7
--- /dev/null
+++ b/projects/_template/scanner-products.md
@@ -0,0 +1,138 @@
+<!-- START doctoc generated TOC please keep comment here to allow auto update
-->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with
[DocToc](https://github.com/thlorenz/doctoc)*
+
+- [TODO: `<Project Name>` — private scanner products + finder
anonymisation](#todo-project-name--private-scanner-products--finder-anonymisation)
+ - [Private scanner products to
anonymise](#private-scanner-products-to-anonymise)
+ - [Finder anonymisation policy](#finder-anonymisation-policy)
+ - [Exempt cases (keep the named
credit)](#exempt-cases-keep-the-named-credit)
+ - [Per-finder override](#per-finder-override)
+ - [Detection signal — how the skill recognises a private-scanner
thread](#detection-signal--how-the-skill-recognises-a-private-scanner-thread)
+ - [Related rules](#related-rules)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+# TODO: `<Project Name>` — private scanner products + finder anonymisation
+
+This file lists the **private scanner products** whose name must
+not appear in any public CVE surface for this project, and
+declares the **finder-anonymisation policy** the
+[`security-issue-sync`](../../.claude/skills/security-issue-sync/SKILL.md)
+skill applies during sync.
+
+The CVE-JSON generator reads three body fields verbatim into
+public surfaces:
+
+- *Short public summary for publish* → `containers.cna.descriptions[].value`
+- *Reporter credited as* → `containers.cna.credits[].value` (type `finder`)
+- *Security mailing list thread* → **private audit trail**,
+ not pushed to the CVE record
+
+The sync skill's anonymise gate (Step 1d signal row +
+Step 5b 1b sixth pre-push gate) scrubs the first two of those
+when the report came in through a private scanner channel. The
+*Security mailing list thread* field stays untouched (it is the
+team's audit trail).
+
+## Private scanner products to anonymise
+
+Each row lists one scanner product whose name must be stripped
+from public CVE surfaces when its findings flow into trackers
+on this project. Add a row per scanner the project's security
+team works with privately.
+
+| Product name (token) | Discovery channel | Notes |
+|---|---|---|
+| TODO: `<scanner-product>` | TODO: `<channel — internal SAST, partner-shared,
unpublished bug-bounty pipeline, vendor private disclosure>` | TODO: `<one-line
context — why the name is sensitive, contract clause if any>` |
+
+**Token format.** The scanner-product token is the literal
+string the scrubber matches against the *Short public summary*
+text and the *Security mailing list thread* field. Match is
+case-insensitive substring. Add aliases on separate rows when
+a product has multiple names.
+
+**When this list is empty.** If the project's security team
+never receives findings from private scanners (every report
+arrives via `security@<project>.apache.org` from a named
+individual), leave this table empty with a note to that
+effect. The sync skill's anonymise gate becomes a no-op.
+
+## Finder anonymisation policy
+
+The anonymise gate applies the following rule when the *Security
+mailing list thread* field references one of the products
+above:
+
+1. **Default**: rewrite the *Reporter credited as* field to
+ `anonymous` and strip the scanner-product name from the
+ *Short public summary for publish* body field text.
+2. **Audit trail stays**: the *Security mailing list thread*
+ field, the status-rollup comment, and the Gmail / PonyMail
+ thread keep the original scanner-product + person-name
+ references for security-team auditing.
+
+### Exempt cases (keep the named credit)
+
+The scrubber must **not** anonymise a credit that was already
+public elsewhere:
+
+- The finder has a public **HackerOne** or **huntr.dev** report
+ URL on the thread.
+- The finder self-credited under their real name in their own
+ inbound `security@` message (i.e. they sent the report
+ themselves, named themselves, and asked to be credited).
+- The finder's organisation publicly disclosed the discovery
+ channel (a vendor blog post, an ASF community announcement,
+ a public CVE record that already names them).
+
+In any of these cases the credit was public-by-the-finder's-
+choice before this CVE shipped; the scrubber is for the
+opposite scenario (no explicit public consent).
+
+### Per-finder override
+
+A finder can opt back **in** to public credit by sending an
+explicit *"please credit me as `<name>`"* line on the thread.
+Record the consent in the tracker's status-rollup comment, set
+the *Reporter credited as* field to the consented form, and
+the gate stops firing for that tracker.
+
+## Detection signal — how the skill recognises a private-scanner thread
+
+The Step 1d signal row fires when **all** of the following
+are true:
+
+1. The *Security mailing list thread* body field's text
+ contains one of the scanner-product tokens declared above
+ (case-insensitive substring).
+2. The *Reporter credited as* body field names a person
+ (`<First> <Last>` pattern) rather than `anonymous` /
+ a known public handle.
+3. None of the exempt-case signals above is present on the
+ tracker (no public HackerOne/huntr.dev URL; no inbound
+ self-naming `security@` message; no public org-disclosed
+ channel).
+
+When the signal fires, sync proposes the rewrite as a
+numbered Step 2 item; the user confirms or skips per the
+normal proposal flow.
+
+## Related rules
+
+- The anonymise rule is one of **six pre-push hygiene gates**
+ in [Step 5b
1b](../../.claude/skills/security-issue-sync/SKILL.md#decision-flow)
+ of the `security-issue-sync` skill. The other five are
+ title cleanup, upgrade-target version, trigger conditions,
+ incomplete-fix cross-CVE clause, and CWE long-form
+ description.
+- The CVE-record-bound surfaces also follow the broader
+ "Confidentiality of `<tracker>`" rules in
+ [`AGENTS.md`](../../AGENTS.md#confidentiality-of-the-tracker-repository).
+- The same anonymisation rule indirectly applies to advisory
+ mail drafted on the `users@<project>.apache.org` thread —
+ the draft is composed from the same *Short public summary*
+ the sync skill polished, so the anonymise scrub is a
+ one-time fix that propagates downstream.
diff --git a/tools/skill-evals/evals/security-issue-sync/README.md
b/tools/skill-evals/evals/security-issue-sync/README.md
index 5f91d84..b97f2e0 100644
--- a/tools/skill-evals/evals/security-issue-sync/README.md
+++ b/tools/skill-evals/evals/security-issue-sync/README.md
@@ -1,6 +1,6 @@
# security-issue-sync eval suite
-Behavioral evals for the `security-issue-sync` skill. Seven steps are
+Behavioral evals for the `security-issue-sync` skill. Eight steps are
covered; steps 0 (pre-flight), 1a–1e (data gathering), 1g (cve.org API
check), 4 (shell apply), and 5/5b/5c (CVE artifact regeneration) are
skipped — all low-signal for structured-output evals.
@@ -16,6 +16,7 @@ skipped — all low-signal for structured-output evals.
| Guardrails | Guardrail violation detection | 3 | CVSS propagation, ASF
project naming, clean pass |
| 3 | Confirm with user | 3 | apply-all, selective, cancel |
| 6 | Recap | 2 | Structural assertions; with and without CVE/draft |
+| Bulk orchestration | Bucket-and-walk decision in bulk mode | 3 | All
label-only (one bundled), mixed buckets (split), all CVE-affecting (all walked)
|
## Hard rules exercised
@@ -27,3 +28,5 @@ skipped — all low-signal for structured-output evals.
- **CVE allocate skill explicitly named and linked**: Step-6 next-step must
name and link `security-cve-allocate` (2c case-2).
- **No-action parking**: Step-11 produces `has_concrete_action: false` (2c
case-3).
- **Golden rule 2 — no bare issue numbers**: `has_bare_issue_numbers` must
always be false.
+- **CVE-affecting trackers walked individually**: in bulk mode, any tracker
whose `proposed_body_field_updates` names a CVE-publication field (Title, Short
public summary for publish, CWE, Severity, Affected versions, Reporter credited
as, Remediation developer, PR with the fix, Public advisory URL) must end up in
`cve_affecting` and the `walk_order` is ascending by tracker number
(bulk-orchestration cases 2 + 3).
+- **Non-CVE-affecting trackers bundled**: trackers whose proposals are limited
to label flips, milestone touches, assignee swaps, project-board moves,
status-rollup entries, reporter Gmail drafts, or RM hand-off comments end up in
`non_cve_affecting` (bulk-orchestration case 1 + the two label-only trackers in
case 2).
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-1-all-label-only/expected.json
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-1-all-label-only/expected.json
new file mode 100644
index 0000000..08ce493
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-1-all-label-only/expected.json
@@ -0,0 +1,5 @@
+{
+ "cve_affecting": [],
+ "non_cve_affecting": [318, 319, 320],
+ "walk_order": []
+}
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-1-all-label-only/report.md
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-1-all-label-only/report.md
new file mode 100644
index 0000000..d55d1a8
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-1-all-label-only/report.md
@@ -0,0 +1,91 @@
+Three subagent reports came back from parallel Step-1 assessment of a `sync
all open` selector that resolved to 3 trackers. Each report lists what the
subagent observed and what the orchestrator should propose.
+
+---
+
+**Tracker airflow-s/airflow-s#318**
+
+```yaml
+issue: 318
+title: Cross-team Connection visibility regression
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr created, security issue]
+current_milestone: 3.2.2
+current_assignees: [alice]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/68001
+ state: merged
+ merged_at: 2026-05-28T10:00:00Z
+ milestone: 3.2.2
+release_shipped: false
+cve_id: CVE-2026-50001
+process_step: 11
+proposed_label_add: [pr merged]
+proposed_label_remove: [pr created]
+proposed_milestone: null
+proposed_assignees_add: []
+proposed_body_field_updates: []
+proposed_status_comment: "Step 11 — PR merged, awaiting release"
+proposed_reporter_email: null
+blockers: []
+notes: ""
+```
+
+---
+
+**Tracker airflow-s/airflow-s#319**
+
+```yaml
+issue: 319
+title: Worker JWT scope tightening
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr created, security issue]
+current_milestone: 3.2.2
+current_assignees: [bob]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/68002
+ state: merged
+ merged_at: 2026-05-28T11:00:00Z
+ milestone: 3.2.2
+release_shipped: false
+cve_id: CVE-2026-50002
+process_step: 11
+proposed_label_add: [pr merged]
+proposed_label_remove: [pr created]
+proposed_milestone: null
+proposed_assignees_add: []
+proposed_body_field_updates: []
+proposed_status_comment: "Step 11 — PR merged, awaiting release"
+proposed_reporter_email: null
+blockers: []
+notes: ""
+```
+
+---
+
+**Tracker airflow-s/airflow-s#320**
+
+```yaml
+issue: 320
+title: HITL Detail visibility on shared DAG
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr created, security issue]
+current_milestone: 3.2.2
+current_assignees: [carol]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/68003
+ state: merged
+ merged_at: 2026-05-28T12:00:00Z
+ milestone: 3.2.2
+release_shipped: false
+cve_id: CVE-2026-50003
+process_step: 11
+proposed_label_add: [pr merged]
+proposed_label_remove: [pr created]
+proposed_milestone: null
+proposed_assignees_add: []
+proposed_body_field_updates: []
+proposed_status_comment: "Step 11 — PR merged, awaiting release"
+proposed_reporter_email: null
+blockers: []
+notes: ""
+```
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-2-mixed-buckets/expected.json
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-2-mixed-buckets/expected.json
new file mode 100644
index 0000000..557de8f
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-2-mixed-buckets/expected.json
@@ -0,0 +1,5 @@
+{
+ "cve_affecting": [411, 412],
+ "non_cve_affecting": [410, 413],
+ "walk_order": [411, 412]
+}
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-2-mixed-buckets/report.md
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-2-mixed-buckets/report.md
new file mode 100644
index 0000000..d3bf164
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-2-mixed-buckets/report.md
@@ -0,0 +1,124 @@
+Four subagent reports came back from a `sync all open` selector. Two trackers
carry body-field updates (CWE, summary); two are pure label flips.
+
+---
+
+**Tracker airflow-s/airflow-s#410**
+
+```yaml
+issue: 410
+title: HITL Detail visibility on shared DAG
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr merged, security issue, rc voting]
+current_milestone: 3.2.2
+current_assignees: [potiuk]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/67800
+ state: merged
+ merged_at: 2026-05-15T09:00:00Z
+ milestone: 3.2.2
+release_shipped: true
+cve_id: CVE-2026-50010
+process_step: 12
+proposed_label_add: [fix released]
+proposed_label_remove: [pr merged, rc voting]
+proposed_milestone: null
+proposed_assignees_add: [vatsrahul1001]
+proposed_body_field_updates: []
+proposed_status_comment: "Step 12 — fix released; RM hand-off to
@vatsrahul1001"
+proposed_reporter_email: "Release shipped notification"
+blockers: []
+notes: ""
+```
+
+---
+
+**Tracker airflow-s/airflow-s#411**
+
+```yaml
+issue: 411
+title: Path traversal in custom operator deserialization
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr merged, security issue]
+current_milestone: 3.2.2
+current_assignees: [bob]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/67801
+ state: merged
+ merged_at: 2026-05-16T09:00:00Z
+ milestone: 3.2.2
+release_shipped: true
+cve_id: CVE-2026-50011
+process_step: 12
+proposed_label_add: [fix released]
+proposed_label_remove: [pr merged]
+proposed_milestone: null
+proposed_assignees_add: [vatsrahul1001]
+proposed_body_field_updates:
+ - "Short public summary for publish — add upgrade target (3.2.2 or later) +
WHO/WHEN/ACTION trigger conditions"
+ - "CWE — rewrite bare `CWE-22` to long form `CWE-22: Improper Limitation of
a Pathname to a Restricted Directory ('Path Traversal')`"
+proposed_status_comment: "Step 12 — fix released; body cleanup for gates #1,
#2, #5"
+proposed_reporter_email: "Release shipped notification"
+blockers: []
+notes: ""
+```
+
+---
+
+**Tracker airflow-s/airflow-s#412**
+
+```yaml
+issue: 412
+title: Bulk task instance API authorization bypass
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr merged, security issue]
+current_milestone: 3.2.2
+current_assignees: [carol]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/67802
+ state: merged
+ merged_at: 2026-05-17T09:00:00Z
+ milestone: 3.2.2
+release_shipped: true
+cve_id: CVE-2026-50012
+process_step: 12
+proposed_label_add: [fix released]
+proposed_label_remove: [pr merged]
+proposed_milestone: null
+proposed_assignees_add: [vatsrahul1001]
+proposed_body_field_updates:
+ - "Reporter credited as — anonymise from `Jane Doe (internal scan)` to
`anonymous` (private-scanner source)"
+proposed_status_comment: "Step 12 — fix released; anonymise gate fired"
+proposed_reporter_email: null
+blockers: []
+notes: ""
+```
+
+---
+
+**Tracker airflow-s/airflow-s#413**
+
+```yaml
+issue: 413
+title: Stale connection cache after rotation
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr merged, security issue]
+current_milestone: 3.2.2
+current_assignees: [dave]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/67803
+ state: merged
+ merged_at: 2026-05-18T09:00:00Z
+ milestone: 3.2.2
+release_shipped: true
+cve_id: CVE-2026-50013
+process_step: 12
+proposed_label_add: [fix released]
+proposed_label_remove: [pr merged]
+proposed_milestone: null
+proposed_assignees_add: [vatsrahul1001]
+proposed_body_field_updates: []
+proposed_status_comment: "Step 12 — fix released"
+proposed_reporter_email: "Release shipped notification"
+blockers: []
+notes: ""
+```
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-3-all-cve-affecting/expected.json
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-3-all-cve-affecting/expected.json
new file mode 100644
index 0000000..36b87e8
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-3-all-cve-affecting/expected.json
@@ -0,0 +1,5 @@
+{
+ "cve_affecting": [450, 451, 452],
+ "non_cve_affecting": [],
+ "walk_order": [450, 451, 452]
+}
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-3-all-cve-affecting/report.md
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-3-all-cve-affecting/report.md
new file mode 100644
index 0000000..c8d33d6
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/case-3-all-cve-affecting/report.md
@@ -0,0 +1,98 @@
+Three subagent reports came back. Every tracker has at least one body-field
update that lands in the regenerated CVE JSON — title cleanup, summary rewrite,
affected-versions normalisation.
+
+---
+
+**Tracker airflow-s/airflow-s#450**
+
+```yaml
+issue: 450
+title: "[ Security Report ] XSS in DAG description rendering"
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr created, security issue]
+current_milestone: null
+current_assignees: [author1]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/68100
+ state: open
+ merged_at: null
+ milestone: null
+release_shipped: false
+cve_id: CVE-2026-50050
+process_step: 10
+proposed_label_add: []
+proposed_label_remove: []
+proposed_milestone: null
+proposed_assignees_add: []
+proposed_body_field_updates:
+ - "Title — strip `[ Security Report ]` prefix"
+ - "Affected versions — wrap value in backticks (currently bare `>= 3.0.0`)"
+proposed_status_comment: "Pre-merge cleanup — gates #4 + Affected versions
wrap"
+proposed_reporter_email: null
+blockers: []
+notes: ""
+```
+
+---
+
+**Tracker airflow-s/airflow-s#451**
+
+```yaml
+issue: 451
+title: SSRF via webhook URL parameter
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr created, security issue]
+current_milestone: null
+current_assignees: [author2]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/68101
+ state: open
+ merged_at: null
+ milestone: null
+release_shipped: false
+cve_id: CVE-2026-50051
+process_step: 10
+proposed_label_add: []
+proposed_label_remove: []
+proposed_milestone: null
+proposed_assignees_add: []
+proposed_body_field_updates:
+ - "Short public summary for publish — write summary from `_No response_`
with WHO/WHEN/ACTION + upgrade target"
+ - "CWE — set bare `_No response_` to `CWE-918: Server-Side Request Forgery
(SSRF)`"
+ - "Severity — set `Unknown` to `Medium`"
+proposed_status_comment: "Pre-merge cleanup — gates #1, #2, #5 + severity"
+proposed_reporter_email: null
+blockers: []
+notes: ""
+```
+
+---
+
+**Tracker airflow-s/airflow-s#452**
+
+```yaml
+issue: 452
+title: "Missing scope check on /api/v2/connections/{id}/test"
+scope_label: airflow
+current_labels: [airflow, cve allocated, pr merged, security issue]
+current_milestone: 3.2.3
+current_assignees: [author3]
+fix_pr:
+ url: https://github.com/apache/airflow/pull/68102
+ state: merged
+ merged_at: 2026-05-26T09:00:00Z
+ milestone: 3.2.3
+release_shipped: false
+cve_id: CVE-2026-50052
+process_step: 11
+proposed_label_add: []
+proposed_label_remove: []
+proposed_milestone: null
+proposed_assignees_add: []
+proposed_body_field_updates:
+ - "Remediation developer — set `_No response_` to `Jarek Potiuk` (PR author)"
+ - "PR with the fix — populate with apache/airflow#68102"
+proposed_status_comment: "Step 11 — PR merged; fill remediation developer"
+proposed_reporter_email: null
+blockers: []
+notes: ""
+```
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/output-spec.md
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/output-spec.md
new file mode 100644
index 0000000..2fef39d
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/output-spec.md
@@ -0,0 +1,47 @@
+## Eval task
+
+You are evaluating the **bulk-mode orchestration** step of the
+`security-issue-sync` skill — specifically the bucket-and-walk
+decision the orchestrator makes after parallel subagents return
+their per-tracker reports.
+
+The skill's bulk mode splits trackers into two buckets after
+assessment:
+
+- **CVE-affecting** — any tracker whose proposal includes a body-field
+ update that lands in the regenerated CVE JSON
+ (Title, Short public summary for publish, CWE, Severity,
+ Affected versions, Reporter credited as, Remediation developer,
+ PR with the fix, Public advisory URL). Each CVE-affecting tracker
+ is walked **individually**: own proposal, own confirmation, own
+ apply, in ascending tracker-number order.
+- **Non-CVE-affecting** — only label flips, milestone touches,
+ assignee swaps, project-board column moves, status-rollup
+ entries, reporter Gmail drafts, RM hand-off comments. Bundled
+ into one combined proposal.
+
+Given the per-tracker subagent reports in the user turn, return
+ONLY valid JSON with this shape:
+
+```json
+{
+ "cve_affecting": [<issue_number>, ...],
+ "non_cve_affecting": [<issue_number>, ...],
+ "walk_order": [<issue_number>, ...]
+}
+```
+
+Field rules:
+
+- `cve_affecting`: every tracker whose `proposed_body_field_updates`
+ list contains at least one entry that names a CVE-publication
+ field (Title, Short public summary for publish, CWE, Severity,
+ Affected versions, Reporter credited as, Remediation developer,
+ PR with the fix, Public advisory URL). Order is not significant
+ in this list.
+- `non_cve_affecting`: every tracker that is NOT in `cve_affecting`.
+ Order is not significant in this list.
+- `walk_order`: ascending tracker-number order over `cve_affecting`.
+ Empty list when `cve_affecting` is empty.
+
+Do not include any text outside the JSON object.
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/step-config.json
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/step-config.json
new file mode 100644
index 0000000..bf54718
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/step-config.json
@@ -0,0 +1,4 @@
+{
+ "skill_md": ".claude/skills/security-issue-sync/SKILL.md",
+ "step_heading": "## Bulk mode — syncing many issues in parallel"
+}
diff --git
a/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/user-prompt-template.md
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/user-prompt-template.md
new file mode 100644
index 0000000..03e551b
--- /dev/null
+++
b/tools/skill-evals/evals/security-issue-sync/step-bulk-orchestration/fixtures/user-prompt-template.md
@@ -0,0 +1,7 @@
+## Subagent reports
+
+{report}
+
+Bucket the trackers into `cve_affecting` vs `non_cve_affecting`
+per the rule above, and produce `walk_order` in ascending
+tracker-number order over `cve_affecting`. Return JSON only.
