This is an automated email from the ASF dual-hosted git repository.
zclllyybb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1979bb02ac0 [fix](ci) Harden Codex review workflow (#64329)
1979bb02ac0 is described below
commit 1979bb02ac011b21a5a5ed9a92127b8817f22afe
Author: zclllyybb <[email protected]>
AuthorDate: Wed Jun 10 11:00:49 2026 +0800
[fix](ci) Harden Codex review workflow (#64329)
## Proposed changes
Follow-up to the merged Litefuse payload-size fix in #64328.
This patch keeps the previous Litefuse 413/pagination/trace-shape fixes
and adds the latest CI hardening:
- Renames the automated review workflows from `opencode-*` to
`code-review-*`, and removes the remaining `opencode` wording from
workflow-local paths.
- Removes the temporary gh-only / MCP-forbid prompt restrictions that
were added while diagnosing the GitHub MCP failure path. The workflow
still verifies the real outcome by reading GitHub Reviews for the
current head SHA after Codex exits.
- Enables Codex memories in CI and syncs `$CODEX_HOME/memories` with
`oss://doris-community-ci/codex/memories.tar.gz` before and after
review. Memory sync is best-effort and does not block review completion.
- Keeps Litefuse recording best-effort so a successful code review is
not failed only because trace upload or verification fails.
The original payload-limit fixes are still included:
- removes the workflow's 4KB context override;
- makes `--max-context-json-chars <= 0` inherit `--max-json-chars`;
- paginates legacy Litefuse observation readback during verification;
- adaptively retries Litefuse 413 responses without creating extra
traces;
- preserves one trace per run and original observation ordering across
ingestion chunks.
## Commits
- `500ebc6319b` `[chore](ci) Rename code review workflows`
- `57a0dd223e3` `[fix](ci) Relax code review submission tooling`
- `746e46d0bbb` `[feat](ci) Sync Codex memories for reviews`
## Validation
Fork workflow validation:
- Pushed the same workflow head to `origin/master` first.
- Opened temporary fork PR https://github.com/zclllyybb/doris/pull/33
from `codex-review-validation-20260610-memory-rename` and triggered
`/review`.
- Code review run succeeded:
https://github.com/zclllyybb/doris/actions/runs/27249379410/job/80470334233
- GitHub review was submitted by `github-actions` on commit
`b14d07483c2d4e2c3dd00a9e8da00e52a9fff4ac`.
- Litefuse trace `0a88ea57df463c2f975fc0081a612f02` was uploaded and
verified with trace input/output present, all agent-message context
windows present, 60 events, 35 observations, no 413 retries, and one
ingestion request under the configured payload cap.
- First memory download saw the expected initial `NoSuchKey` for
`codex/memories.tar.gz` and continued without failing the review; memory
upload then succeeded with a 7,735-byte archive.
- Temporary validation PR and branch were closed/deleted after
validation.
Earlier payload-limit validation retained from the previous iteration:
- Unit-style chunking check: splitting into multiple chunks preserved
event bodies, event order, and trace id.
- Simulated Litefuse 413 on multi-event batches: uploader split/retried,
preserved event order, and uploaded all 16 events successfully.
- Simulated Litefuse 413 on one oversized observation: uploader shrank
the single observation and retried successfully.
- Synthetic dry-run with 267 Codex events and default context
truncation: split into 24 requests, max request size 395324 bytes under
a 400000-byte test cap.
- Synthetic dry-run with a single huge observation and default context
truncation: max request size 341060 bytes under a 400000-byte test cap.
- Verified the failed trace from
https://github.com/apache/doris/actions/runs/27213447314/job/80351165150
using paginated legacy readback: trace
`45202ab41ddbfd099877fa7be5cc1256` has 105 observations across pages, 21
agent messages, one turn input, and zero observations missing I/O.
- Diagnosed
https://github.com/apache/doris/actions/runs/27213705642/job/80352669355
as a live Litefuse 413 from the old master helper
`f48efb28db486837db5375e6fd0bcfc95ea67a05` despite `--max-payload-bytes
4000000`.
---
.github/scripts/emit_litefuse_otel_io.py | 98 ++++++++++++++++++----
...-review-comment.yml => code-review-comment.yml} | 2 +-
...de-review-runner.yml => code-review-runner.yml} | 61 +++++++++++---
...sync-result.yml => code-review-sync-result.yml} | 0
4 files changed, 135 insertions(+), 26 deletions(-)
diff --git a/.github/scripts/emit_litefuse_otel_io.py
b/.github/scripts/emit_litefuse_otel_io.py
index 4a93d5262d4..351b5b5cbf3 100644
--- a/.github/scripts/emit_litefuse_otel_io.py
+++ b/.github/scripts/emit_litefuse_otel_io.py
@@ -23,6 +23,7 @@ import json
import os
import secrets
import time
+import urllib.error
import urllib.parse
import urllib.request
@@ -521,18 +522,29 @@ def shrink_event_for_payload(event, max_payload_bytes):
body = shrunk.get("body") if isinstance(shrunk.get("body"), dict) else {}
event_name = body.get("name")
- for max_chars in (2_000, 1_000, 500, 200, 80):
+ def candidate_with_limits(max_chars, max_context_events=None):
candidate = json.loads(json.dumps(shrunk, ensure_ascii=False))
candidate_body = candidate.get("body") if
isinstance(candidate.get("body"), dict) else {}
input_object = candidate_body.get("input")
if isinstance(input_object, dict):
context_events =
input_object.get("events_since_previous_agent_message")
if isinstance(context_events, list):
- input_object["events_since_previous_agent_message"] = [
+ compact_events = [
compact_context_event(context_event, max_chars)
for context_event in context_events
if isinstance(context_event, dict)
]
+ if max_context_events is not None:
+ omitted_count = max(len(compact_events) -
max_context_events, 0)
+ if max_context_events <= 0:
+ compact_events = []
+ else:
+ compact_events = compact_events[-max_context_events:]
+ if omitted_count:
+ input_object[
+ "events_since_previous_agent_message_omitted_count"
+ ] = omitted_count
+ input_object["events_since_previous_agent_message"] =
compact_events
if isinstance(input_object.get("previous_agent_message"), dict):
previous_message = input_object["previous_agent_message"]
previous_message["text"] = truncate_text(
@@ -557,10 +569,20 @@ def shrink_event_for_payload(event, max_payload_bytes):
metadata = candidate_body.get("metadata")
if metadata not in (None, ""):
candidate_body["metadata"] = truncate_json(metadata, max_chars)
+ return candidate
+
+ for max_chars in (2_000, 1_000, 500, 200, 80):
+ candidate = candidate_with_limits(max_chars)
if json_payload_bytes({"batch": [candidate]}) <= max_payload_bytes:
return candidate
+ for max_context_events in (50, 20, 10, 5, 2, 1, 0):
+ for max_chars in (80, 40, 20, 10):
+ candidate = candidate_with_limits(max_chars, max_context_events)
+ if json_payload_bytes({"batch": [candidate]}) <= max_payload_bytes:
+ return candidate
+
raise RuntimeError(
"Litefuse ingestion event is too large after truncation: "
f"{json_payload_bytes({'batch': [event]})} bytes > {max_payload_bytes}
bytes; "
@@ -622,21 +644,51 @@ def post_payload_once(endpoint, public_key, secret_key,
payload):
}
+def retry_payload_chunks_after_413(payload, request_size, max_payload_bytes):
+ batch = payload.get("batch") or []
+ if not batch:
+ raise RuntimeError(
+ "Litefuse ingestion returned 413 for an empty payload chunk"
+ )
+
+ next_limit = max(1_000, min(max_payload_bytes - 1, request_size // 2))
+ if len(batch) == 1:
+ event = shrink_event_for_payload(batch[0], next_limit)
+ return [({"batch": [event]}, json_payload_bytes({"batch": [event]}))]
+
+ return chunk_payload(payload, next_limit)
+
+
def post_payload(endpoint, public_key, secret_key, payload, max_payload_bytes):
statuses = []
success_count = 0
request_sizes = []
+ retry_count = 0
chunks = chunk_payload(payload, max_payload_bytes)
- for chunk, request_size in chunks:
- status = post_payload_once(endpoint, public_key, secret_key, chunk)
+ while chunks:
+ chunk, request_size = chunks.pop(0)
+ try:
+ status = post_payload_once(endpoint, public_key, secret_key, chunk)
+ except urllib.error.HTTPError as exc:
+ if exc.code != 413:
+ raise
+ retry_count += 1
+ chunks = (
+ retry_payload_chunks_after_413(
+ chunk, request_size, max_payload_bytes
+ )
+ + chunks
+ )
+ continue
statuses.append(status["status"])
success_count += int(status.get("success_count") or 0)
request_sizes.append(request_size)
return {
"statuses": statuses,
- "request_count": len(chunks),
+ "request_count": len(statuses),
"request_sizes": request_sizes,
"max_request_size": max(request_sizes) if request_sizes else 0,
+ "payload_too_large_retries": retry_count,
"success_count": success_count,
}
@@ -673,16 +725,30 @@ def fetch_observations_v2(base_url, public_key,
secret_key, trace_id):
return json.loads(response.read().decode())
-def fetch_observations_legacy(base_url, public_key, secret_key, trace_id):
+def fetch_observations_legacy(
+ base_url, public_key, secret_key, trace_id, max_pages=10
+):
auth = base64.b64encode(f"{public_key}:{secret_key}".encode()).decode()
- params = urllib.parse.urlencode({"traceId": trace_id, "limit": "100",
"page": "1"})
- request = urllib.request.Request(
- f"{base_url.rstrip('/')}/api/public/observations?{params}",
- headers={"Authorization": f"Basic {auth}"},
- method="GET",
- )
- with urllib.request.urlopen(request, timeout=30) as response:
- return json.loads(response.read().decode())
+ limit = 100
+ rows = []
+ last_payload = {}
+ for page in range(1, max_pages + 1):
+ params = urllib.parse.urlencode(
+ {"traceId": trace_id, "limit": str(limit), "page": str(page)}
+ )
+ request = urllib.request.Request(
+ f"{base_url.rstrip('/')}/api/public/observations?{params}",
+ headers={"Authorization": f"Basic {auth}"},
+ method="GET",
+ )
+ with urllib.request.urlopen(request, timeout=30) as response:
+ payload = json.loads(response.read().decode())
+ last_payload = payload if isinstance(payload, dict) else {}
+ page_rows = observation_rows_from_v2(last_payload)
+ rows.extend(page_rows)
+ if len(page_rows) < limit:
+ break
+ return {**last_payload, "data": rows}
def observation_rows_from_v2(payload):
@@ -908,7 +974,7 @@ def parse_args():
parser.add_argument("--max-input-chars", type=int, default=200_000)
parser.add_argument("--max-output-chars", type=int, default=200_000)
parser.add_argument("--max-json-chars", type=int, default=40_000)
- parser.add_argument("--max-context-json-chars", type=int, default=4_000)
+ parser.add_argument("--max-context-json-chars", type=int, default=0)
parser.add_argument("--max-payload-bytes", type=int, default=4_000_000)
parser.add_argument("--verify", action="store_true")
parser.add_argument("--dry-run", action="store_true")
@@ -922,6 +988,8 @@ def parse_args():
def main():
args = parse_args()
endpoint = args.endpoint or
f"{args.base_url.rstrip('/')}/api/public/ingestion"
+ if args.max_context_json_chars <= 0:
+ args.max_context_json_chars = args.max_json_chars
input_text = read_text(args.input_file, args.max_input_chars)
output_text = (
diff --git a/.github/workflows/opencode-review-comment.yml
b/.github/workflows/code-review-comment.yml
similarity index 98%
rename from .github/workflows/opencode-review-comment.yml
rename to .github/workflows/code-review-comment.yml
index d042da95ac5..89eea813b40 100644
--- a/.github/workflows/opencode-review-comment.yml
+++ b/.github/workflows/code-review-comment.yml
@@ -59,7 +59,7 @@ jobs:
github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'COLLABORATOR'
)
- uses: ./.github/workflows/opencode-review-runner.yml
+ uses: ./.github/workflows/code-review-runner.yml
secrets: inherit
with:
pr_number: ${{ needs.resolve-pr.outputs.pr_number }}
diff --git a/.github/workflows/opencode-review-runner.yml
b/.github/workflows/code-review-runner.yml
similarity index 90%
rename from .github/workflows/opencode-review-runner.yml
rename to .github/workflows/code-review-runner.yml
index 44d4db3d8e2..99baccae58e 100644
--- a/.github/workflows/opencode-review-runner.yml
+++ b/.github/workflows/code-review-runner.yml
@@ -76,6 +76,13 @@ jobs:
cli_auth_credentials_store = "file"
approval_policy = "never"
+ [features]
+ memories = true
+
+ [memories]
+ use_memories = true
+ generate_memories = true
+
[shell_environment_policy]
inherit = "all"
@@ -92,9 +99,32 @@ jobs:
OSS_ENDPOINT: oss-cn-hongkong.aliyuncs.com
OSS_CODEX_AUTH_OBJECT: oss://doris-community-ci/codex/auth.json
+ - name: Sync Codex memories from OSS
+ continue-on-error: true
+ run: |
+ install -m 700 -d "$CODEX_HOME/memories"
+ archive="$RUNNER_TEMP/codex-memories.tar.gz"
+
+ if ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f
"$OSS_CODEX_MEMORIES_OBJECT" "$archive"; then
+ tar -tzf "$archive" | awk '
+ $0 != "memories" && $0 !~ /^memories\// { bad = 1 }
+ END { exit bad }
+ '
+ tar -xzf "$archive" -C "$CODEX_HOME"
+ install -m 700 -d "$CODEX_HOME/memories"
+ chmod -R go-rwx "$CODEX_HOME/memories" || true
+ else
+ echo "No Codex memories archive could be downloaded; continuing
with empty memories."
+ fi
+ env:
+ OSS_AK: ${{ secrets.OSS_AK }}
+ OSS_SK: ${{ secrets.OSS_SK }}
+ OSS_ENDPOINT: oss-cn-hongkong.aliyuncs.com
+ OSS_CODEX_MEMORIES_OBJECT:
oss://doris-community-ci/codex/memories.tar.gz
+
- name: Prepare review context directory
run: |
- review_context_dir="$(mktemp -d
"$GITHUB_WORKSPACE/.opencode-review.XXXXXX")"
+ review_context_dir="$(mktemp -d
"$GITHUB_WORKSPACE/.code-review.XXXXXX")"
review_context_rel="$(basename "$review_context_dir")"
printf 'REVIEW_CONTEXT_DIR=%s\n' "$review_context_dir" >>
"$GITHUB_ENV"
printf 'REVIEW_CONTEXT_REL=%s\n' "$review_context_rel" >>
"$GITHUB_ENV"
@@ -175,9 +205,7 @@ jobs:
You are performing an automated code review inside a GitHub Actions
runner. The gh CLI is available and authenticated via GH_TOKEN.
The current directory is the code repository for the PR to be
reviewed.
You MUST NOT attempt to access any files outside the current
directory. and you DO NOT need to. But this does not prevent you from normally
using any skill or web fetch tools.
- You MUST use the local gh CLI, including gh api when needed, for
pull request review submission.
- Do NOT use GitHub MCP connector write tools such as
github_add_review_to_pr.
- You can comment on the pull request through the GH_TOKEN-backed gh
CLI.
+ You can comment on the pull request.
Proceed with all subsequent research at the HIGHEST level of
thought, aiming to identify all issues and submit all comments in JSON format.
Context:
@@ -267,11 +295,6 @@ jobs:
fi
fi
- if [ -z "$failure_reason" ] \
- && jq -e 'select(.type == "item.completed" and (.item // {}).type
== "mcp_tool_call" and (.item // {}).tool == "github_add_review_to_pr")'
"$REVIEW_CONTEXT_DIR/codex-events.jsonl" >/dev/null; then
- failure_reason="Codex attempted to submit the review through the
GitHub MCP connector instead of the GH_TOKEN-backed gh CLI."
- fi
-
if [ -z "$failure_reason" ]; then
reviews_file="$REVIEW_CONTEXT_DIR/pr_reviews_after_codex.json"
reviews_api_ok=false
@@ -309,6 +332,7 @@ jobs:
- name: Record review I/O to Litefuse
if: ${{ always() }}
+ continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LANGFUSE_PUBLIC_KEY: ${{ secrets.LANGFUSE_PK }}
@@ -347,7 +371,6 @@ jobs:
--model "gpt-5.5" \
--reasoning-effort "$REASONING_EFFORT" \
--environment "github-actions" \
- --max-context-json-chars 4000 \
--max-payload-bytes 4000000 \
--min-observations 4 \
--min-step-observations 2 \
@@ -355,6 +378,24 @@ jobs:
--verify-sleep-seconds 5 \
--verify
+ - name: Sync Codex memories back to OSS
+ if: ${{ always() }}
+ continue-on-error: true
+ run: |
+ if [ ! -d "$CODEX_HOME/memories" ]; then
+ echo "No Codex memories directory found; skipping memories sync."
+ exit 0
+ fi
+
+ archive="$RUNNER_TEMP/codex-memories.tar.gz"
+ tar -C "$CODEX_HOME" -czf "$archive" memories
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f
"$archive" "$OSS_CODEX_MEMORIES_OBJECT"
+ env:
+ OSS_AK: ${{ secrets.OSS_AK }}
+ OSS_SK: ${{ secrets.OSS_SK }}
+ OSS_ENDPOINT: oss-cn-hongkong.aliyuncs.com
+ OSS_CODEX_MEMORIES_OBJECT:
oss://doris-community-ci/codex/memories.tar.gz
+
- name: Sync Codex auth back to OSS
if: ${{ always() }}
run: |
diff --git a/.github/workflows/opencode-review-sync-result.yml
b/.github/workflows/code-review-sync-result.yml
similarity index 100%
rename from .github/workflows/opencode-review-sync-result.yml
rename to .github/workflows/code-review-sync-result.yml
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
