This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-steward.git
The following commit(s) were added to refs/heads/main by this push:
new 07a94a6d feat(spec-loop): scope update runs incrementally from a
.last-sync marker (#467)
07a94a6d is described below
commit 07a94a6dca81f5775a8a6459d7a9cba17090032a
Author: Justin Mclean <[email protected]>
AuthorDate: Thu Jun 11 22:54:35 2026 +1000
feat(spec-loop): scope update runs incrementally from a .last-sync marker
(#467)
* improved loop update
* fix minor bug
* make the branch name unique per run
---
tools/spec-loop/.last-sync | 2 +-
tools/spec-loop/PROMPT_update.md | 46 +++++++++++++-----
tools/spec-loop/loop.sh | 102 ++++++++++++++++++++++++++++++++++++++-
3 files changed, 134 insertions(+), 16 deletions(-)
diff --git a/tools/spec-loop/.last-sync b/tools/spec-loop/.last-sync
index 52b6b76e..a65a14fc 100644
--- a/tools/spec-loop/.last-sync
+++ b/tools/spec-loop/.last-sync
@@ -1 +1 @@
-043b48d5e56e30ab84f83da92c50566f471e18fe
+e4f79fc8d2470f2120ee5b6df1215d0f8159874f
diff --git a/tools/spec-loop/PROMPT_update.md b/tools/spec-loop/PROMPT_update.md
index be5c2174..5ea7d317 100644
--- a/tools/spec-loop/PROMPT_update.md
+++ b/tools/spec-loop/PROMPT_update.md
@@ -16,14 +16,26 @@ Context to load first:
Steps:
-1. **Create the sync branch off the integration base**, then switch to
- it: `git checkout -b sync-specs`. (One reviewable PR for the
- sync.) Never commit the sync to the integration branch.
-2. Inventory the code with parallel subagents:
+1. **Check the `## Incremental scope` section appended below by the
+ runner.** If it names a previous sync commit, run the `git diff
+ --name-only` command it provides and treat that file list as the
+ *only* surface to re-audit — everything else is already in sync as of
+ that commit. If the diff is empty, exit without creating a branch or
+ commit (print "specs already in sync as of <SHA>"). If no previous
+ sync commit is recorded, fall through to a full inventory.
+2. **Create a uniquely-named sync branch off the integration base**, then
+ switch to it: `git checkout -b "sync-specs-$(date +%Y%m%d-%H%M%S)"`. A
+ fresh branch every run keeps each sync as its own reviewable PR and
+ never collides with or commits on top of a previous `sync-specs*`
+ branch. Note the exact name you created — you will print it in the
+ human-run commands below. Never commit the sync to the integration
+ branch.
+3. Inventory the code with parallel subagents (full inventory only if
+ step 1 did not narrow the surface):
- every `.claude/skills/*/SKILL.md` (name, mode, what it does);
- every `tools/*` project (what it does, its tests);
- the mode/status table in `docs/modes.md`.
-3. Diff that inventory against `tools/spec-loop/specs/`:
+4. Diff that inventory against `tools/spec-loop/specs/`:
- **New functionality with no spec** → author a new topic-named spec
(no number prefix) following the format in
[`specs/README.md`](specs/README.md), grounded in the real code it
@@ -35,25 +47,33 @@ Steps:
are reflected).
- **Removed functionality** → mark the spec or move it to a `Known
gaps`/retired note; do not silently delete history.
-4. Update `specs/overview.md` and `specs/README.md` indexes if areas were
+5. Update `specs/overview.md` and `specs/README.md` indexes if areas were
added or renamed.
-5. `git add -A` then `git commit` with subject
+6. `git add -A` then `git commit` with subject
`docs(spec-loop): sync specs with contributed functionality` and a
- `Generated-by: Claude (Opus 4.7)` trailer.
+ `Generated-by: Claude (Opus 4.7)` trailer. **Do NOT touch
+ `tools/spec-loop/.last-sync` yourself** — `loop.sh` amends the marker
+ into this commit after you finish, so the next `update` run knows to
+ scope from `$BASE_HEAD`. Leaving it alone avoids merge conflicts with
+ that amendment.
Then STOP. Do NOT push, do NOT open a PR. Print the human-run commands:
+(substitute `<sync-branch>` with the exact branch name you created in
+step 2)
+
```text
-git push -u origin sync-specs
-gh pr create --web --base <integration-base> --head sync-specs \
+git push -u origin <sync-branch>
+gh pr create --web --base <integration-base> --head <sync-branch> \
--title "Sync specs with contributed functionality" --body-file <body>
```
Rules:
-- **Edit specs only.** This beat changes `tools/spec-loop/specs/` (and
- the indexes). It must NOT change any skill, tool, or doc outside the
- spec directory — it documents reality, it does not alter it.
+- **Edit specs only.** This beat changes `tools/spec-loop/specs/` and
+ the indexes. It must NOT change any skill, tool, or doc outside the
+ spec directory — it documents reality, it does not alter it. The
+ marker file `.last-sync` is owned by `loop.sh`; do not touch it.
- Confirm with a code search before recording something as present or
absent. Do not invent behaviour the code does not have.
- Keep the RFCs untouched — they are a separate governance layer.
diff --git a/tools/spec-loop/loop.sh b/tools/spec-loop/loop.sh
index 966c5eef..aa855bb5 100755
--- a/tools/spec-loop/loop.sh
+++ b/tools/spec-loop/loop.sh
@@ -87,6 +87,10 @@ TOOLING_REF="${TOOLING_REF:-HEAD}"
AGENT="${SPEC_LOOP_AGENT:-claude}"
MODEL="${SPEC_LOOP_MODEL:-sonnet}"
PR_LIMIT="${SPEC_LOOP_PR_LIMIT:-100}"
+# Agent output format. Default `text` is what the spinner expects; switch to
+# `stream-json` (SPEC_LOOP_OUTPUT_FORMAT=stream-json) to see live tool-call
+# events when debugging a slow or wedged run.
+OUTPUT_FORMAT="${SPEC_LOOP_OUTPUT_FORMAT:-text}"
# Plan length that triggers ONE consolidation round before building. The
# consolidate beat preserves every planned work item, so a plan that is long
# because of *pending work* (not stale history) cannot shrink below this —
@@ -215,6 +219,50 @@ open_pr_context() {
# work is what stops the agent re-picking the same top-priority plan item and
# rebuilding it on a new branch every iteration. Reads refs only, so it is
# correct regardless of which branch is currently checked out.
+# Incremental scope for `update`: read the saved sync marker and tell the
+# agent to only re-inspect paths that changed since then. Without this the
+# update beat re-audits every skill, tool, and modes.md row on every run,
+# which is the bulk of the streaming volume on a slow link.
+#
+# The marker is `tools/spec-loop/.last-sync` — a plain file containing the
+# BASE SHA the specs were last synced against. Read from the working tree
+# (so a half-finished sync counts); fall back to the control branch via
+# `git show` for the case where BASE is already checked out. The update
+# prompt overwrites this file at the end of every successful sync.
+update_scope_context() {
+ echo ""
+ echo "## Incremental scope — only re-inspect what changed since the last
sync"
+ echo ""
+ local marker="tools/spec-loop/.last-sync"
+ local prev
+ if [ -f "$marker" ]; then
+ prev="$(tr -d '[:space:]' < "$marker")"
+ else
+ prev="$(git show "$TOOLING_REF:$marker" 2>/dev/null | tr -d
'[:space:]')"
+ fi
+ if [ -z "$prev" ]; then
+ echo "No \`$marker\` recorded — do a full inventory, then write the
new"
+ echo "BASE SHA to that file as part of the sync commit."
+ return 0
+ fi
+ echo "The last sync marker (\`$marker\`) is \`$prev\`. The current BASE
HEAD"
+ echo "is \`$BASE_HEAD\`."
+ echo ""
+ echo "Scope your inventory to paths touched in that range:"
+ echo ""
+ echo '```'
+ echo "git diff --name-only $prev..$BASE_HEAD -- .claude/skills tools
docs/modes.md"
+ echo '```'
+ echo ""
+ echo "Skills, tools, and \`docs/modes.md\` rows untouched in that range
are still"
+ echo "in sync as of the previous run — skip them. Only re-inspect specs
whose"
+ echo "subjects appear in the diff. If the diff is empty there is nothing
to"
+ echo "sync; exit without creating a branch or commit."
+ echo ""
+ echo "When you finish the sync, overwrite \`$marker\` with \`$BASE_HEAD\`
and"
+ echo "include it in the sync commit so the next run picks up from here."
+}
+
local_branch_context() {
echo ""
echo "## Local work-item branches"
@@ -311,7 +359,11 @@ while true; do
echo "Error: could not read '$ACTIVE_PROMPT' from the working tree or
control branch '$TOOLING_REF'." >&2
rm -f "$PROMPT_WITH_CONTEXT"; break
fi
- open_pr_context >> "$PROMPT_WITH_CONTEXT"
+ # Update mode just diffs code against specs; it doesn't pick a work item,
so
+ # the open-PR list (a network round-trip via gh) buys nothing. Skip it
there.
+ if [ "$MODE" != "update" ]; then
+ open_pr_context >> "$PROMPT_WITH_CONTEXT"
+ fi
local_branch_context >> "$PROMPT_WITH_CONTEXT"
if [ "$BUILD_ITERATION" = true ]; then
@@ -358,6 +410,13 @@ while true; do
fi
fi
BASE_HEAD="$(git rev-parse HEAD)"
+
+ # Update-only: append incremental scope now that BASE is checked out
+ # and BASE_HEAD is known. The agent will diff $prev..$BASE_HEAD on the
+ # listed paths and skip anything not touched in that range.
+ if [ "$MODE" = "update" ]; then
+ update_scope_context >> "$PROMPT_WITH_CONTEXT"
+ fi
fi
# Run one iteration with a fresh context.
@@ -369,10 +428,15 @@ while true; do
# --disallowedTools … defense-in-depth: hard-deny push and
# gh so a stray call cannot reach the
# remote even with permissions skipped.
+ # Claude CLI requires --verbose with -p when output-format is stream-json;
+ # add it only in that case to keep the default `text` run quiet.
+ VERBOSE_ARGS=()
+ [ "$OUTPUT_FORMAT" = "stream-json" ] && VERBOSE_ARGS=(--verbose)
"$AGENT" -p \
--dangerously-skip-permissions \
--disallowedTools "Bash(git push:*)" "Bash(gh:*)" \
- --output-format=text \
+ --output-format="$OUTPUT_FORMAT" \
+ ${VERBOSE_ARGS[@]+"${VERBOSE_ARGS[@]}"} \
--model "$MODEL" < "$PROMPT_WITH_CONTEXT" &
AGENT_PID=$!
spinner "$AGENT_PID" & SPINNER_PID=$!
@@ -395,6 +459,40 @@ while true; do
echo "⚠ No work-item branch was created (still on '$CUR_BRANCH').
Check the agent output above." >&2
fi
+ # Update mode: advance the .last-sync marker to $BASE_HEAD, so the
+ # next `update` run scopes from here. Bundle the marker bump into
+ # the agent's sync commit when there is one (so it ships in the
+ # same PR); otherwise — if the agent saw nothing to sync and stayed
+ # on BASE — make a tiny marker-only branch off BASE.
+ if [ "$MODE" = "update" ]; then
+ if [ "$CUR_BRANCH" != "$BASE" ] && [ "$CUR_BRANCH" !=
"$TOOLING_REF" ]; then
+ printf '%s\n' "$BASE_HEAD" > tools/spec-loop/.last-sync
+ if ! git diff --quiet -- tools/spec-loop/.last-sync
2>/dev/null; then
+ git add tools/spec-loop/.last-sync
+ if git commit --amend --no-edit >/dev/null 2>&1; then
+ echo "[ marker ] amended .last-sync = $BASE_HEAD into
$CUR_BRANCH"
+ else
+ echo "⚠ Could not amend .last-sync into '$CUR_BRANCH'
— bump it by hand." >&2
+ fi
+ fi
+ elif [ "$CUR_BRANCH" = "$BASE" ]; then
+ cur_marker=""
+ [ -f tools/spec-loop/.last-sync ] && cur_marker="$(tr -d
'[:space:]' < tools/spec-loop/.last-sync)"
+ if [ "$cur_marker" != "$BASE_HEAD" ]; then
+ marker_branch="advance-last-sync-${BASE_HEAD:0:7}"
+ if git checkout -b "$marker_branch" >/dev/null 2>&1; then
+ printf '%s\n' "$BASE_HEAD" > tools/spec-loop/.last-sync
+ git add tools/spec-loop/.last-sync
+ if git commit -m "chore(spec-loop): advance .last-sync
to $BASE_HEAD" >/dev/null 2>&1; then
+ echo "[ marker ] advanced .last-sync on
$marker_branch"
+ echo " push it with: git push -u origin
$marker_branch"
+ CUR_BRANCH="$marker_branch"
+ fi
+ fi
+ fi
+ fi
+ fi
+
# Safety guard: a build/update iteration must never commit to the base.
if [ "$CUR_BRANCH" = "$BASE" ] && [ "$(git rev-parse HEAD)" !=
"$BASE_HEAD" ]; then
echo "✗ This iteration committed to '$BASE' instead of a work-item
branch." >&2
