JNSimba opened a new pull request, #70651:
URL: https://github.com/apache/doris/pull/70651

   ### Background
   
   Doris streaming jobs consume source changes and recover from persisted 
offsets. These regression suites cover CDC ingestion, S3 task retries, and FE 
restart recovery.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Test
   
   Problem Summary:
   
   The auto-resume suite reads the wrong failure-counter row and can swallow a 
failed check, while the FE restart suite references an undefined status 
variable and does not verify newly arriving data after recovery. Existing CDC 
checks also lack explicit rollback and successive committed changes to the same 
key. Strengthen these seven suites with bounded waits, committed markers, and 
complete expected-output comparisons; no production behavior changes.
   
   ### Details
   
   - **Change map:** 337 changed lines: 205 in seven regression suites 
(+183/-22) and 132 added lines in seven expected-output files. No production 
code, protocol, dependency, or documentation changes.
   - **Failure:** Auto-resume checks can continue after a timeout, restart 
validation can fail on its PENDING branch, and status/counter checks alone do 
not establish recovered data correctness.
   - **Cause:** Incorrect counter indexing and string comparisons, a missing 
exception rethrow, an undefined variable, and missing phase-specific data 
checks.
   - **Scenarios:** Roll back INSERT/UPDATE/DELETE on MySQL, PostgreSQL, and 
OceanBase, then commit a marker and compare full rows. While paused, commit 
UPDATE/DELETE/INSERT on the same complete key and delete another key; resume 
and check the final contents. For S3 recovery, verify existing rows before and 
after FE restart, then upload a new file and verify both the offset and all 21 
rows.
   - **What existing tests miss:** Rolled-back changes leaking downstream, 
stale values or resurrected deletes after successive key changes, and a 
recovered job that retains its state but no longer consumes new input.
   - **Fails when broken:** Missing markers, incorrect values or key sets, 
unexpected offsets, failed waits, and output mismatches all fail the suite. 
Fault injection is removed in finally; each restart run uses an isolated object 
prefix and preserves the shared fixtures.
   - **Stability evidence:** Each of the seven suites passed five complete runs 
in the verified modes, with zero failed, fatal, or skipped suites in those 35 
successful runs. PostgreSQL incremental waits now use bounded progress polling 
instead of two fixed one-minute sleeps.
   
   ### Testing
   
   - Built ASAN FE/BE and the CDC client; built the regression framework 
including its unit tests.
   - In a Linux development environment, ran the MySQL/PostgreSQL/OceanBase 
basic suites, both composite-primary-key suites, and auto-resume five times 
each. Ran FE restart in `not_cloud` mode five times. All 35 successful rounds 
used normal expected-output comparisons.
   - S3 tests used an isolated MinIO bucket containing byte-identical copies of 
the existing CSV fixtures. Restart repetitions used otherwise identical per-run 
suite copies with unique names to retain their Docker environments and logs.
   - Expected outputs were initially derived from fixture and transaction data, 
then verified by the normal regression runs. All 14 tested case/output files 
match the submitted versions.
   - Cloud restart was attempted but blocked before data verification: 
FoundationDB could not initialize inotify because the host resource limit was 
exhausted. No Cloud data checks are claimed as passing.
   - `git diff --check` passed.
   
   ### Release note
   
   None.
   
   ### Check List (For Author)
   
   - Behavior changed: No production behavior changes.
   - Does this need documentation: No.
   - Pre-submission status: Partially verified: all seven suites passed five 
runs in the modes listed above; FE restart in Cloud mode remains blocked by the 
test environment.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6), with remote Codex for validation.
   - Generated: Regression test changes, expected-output baselines, and this 
description.
   - Human review: Final reviewer approval is pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to