suxiaogang223 commented on PR #69886: URL: https://github.com/apache/doris/pull/69886#issuecomment-6974000665
# Local self-review after rebase PR: https://github.com/apache/doris/pull/69886 Head: `7ad7d7890cea93e3aff26a36afea70aac1842aa7` Base: `098d0bb62bbc82d9bd999d391efe1c20df89930b` FE selects Rust/JNI before input, BE writes and prepares files, the report acknowledgement transfers ownership, and FE publishes one Paimon snapshot. Rebase preserves this invocation chain. The previous FE UT merge conflict is resolved by advancing the combined connector surface to API 35. ## Part 1.3 checkpoints | Checkpoint | Conclusion | | --- | --- | | 1. Goal and proof | Rebase the Rust append writer onto master and rerun CI. The SPI version test and both surface tests verify the resolved collision; Paimon behavior tests exercise writer selection, descriptors and v14 decoding. | | 2. Scope and reuse | Range-diff shows only the SPI POM/version assertion changes in the first replayed commit; the other eight commits replay unchanged. Both upstream path-partition and feature write-diagnostic methods are retained. No thirdparty diff remains. | | 3. Concurrency and locks | Each pipeline local state owns a writer. The admission mutex protects the active count; SDK calls, workload refresh and file deletion run outside it. Arrow release on Rust threads restores the query context. FE transaction state remains synchronized. | | 4. Lifecycle and static initialization | Native stop precedes Arrow-pool/budget release. Prepared-file cleanup retains messages, builder and tracker until report resolution. Failure exits release admission/credits; metrics initialize lazily. These paths are unchanged by rebase. | | 5. Dynamic configuration | Both BE limits remain mutable. New writers read current values; active writer budgets stay fixed. Online adjustment and retry are covered by existing BE tests. | | 6. Compatibility | SPI major 35 separates the combined surface from upstream 34. Missing descriptors/JNI methods fail safely. Functional mixed-version Paimon reads/writes are an explicitly accepted limitation, including with Rust writes disabled; no compatibility layer is added. | | 7. Thrift/Protobuf IDs | Read all five touched definitions on HEAD, master, 4.2, 4.1, 4.0 and 3.1. Common field IDs, names, types and optionality agree; all added fields are optional; descriptor ID 7 stays reserved. Upstream BLOB fields 21-23 are retained and scan field 43 is free. No protobuf change. | | 8. Parallel paths | Rust reads and Rust/JNI writes retain the shared pinned descriptor. REST-vended credentials select JNI before input consumption and reconstruct refreshable RESTTokenFileIO. Upstream scan API additions are preserved. | | 9. Conditions and errors | SDK failures become non-OK Status and never trigger JNI fallback after input. Wide-row checks use the admitted budget and oversized batches advance by one row. No new condition or error-suppression path is introduced by rebase. | | 10. Test coverage | Existing tests cover null/scalar conversion, wide rows, cancellation, admission/credit release, prepare/open failure, report ownership, commit framing and REST refresh. Updated SQL cases await CI; no real STS expiry or process-crash test is claimed. | | 11. Test results | On the rebased head, 1,300 FE/Java tests completed with zero failures/errors and one skipped, including all 165 SPI tests. Both recorded baselines passed. Prior 59 ASAN BE tests are older evidence for unchanged writer sources; current BE and SQL execution awaits CI. Generated SQL outputs are unchanged. | | 12. Observability | Writer selection, batches/rows, Arrow/native/fixed credit and failure/abort timings remain in profiles. Active/credit/rejection metrics and bounded FE transaction labels remain intact. No per-row INFO logging is added. | | 13. Persistence and failover | FE remains the snapshot committer through the existing transaction framework; no new EditLog format. COMMITTING/OUTCOME_UNKNOWN preserve files. Durable recovery of an in-flight external transaction across FE death remains a framework limitation. | | 14. Atomicity and crash cleanup | All BE messages feed one FE snapshot commit. ACK, definite rejection and ambiguous reports have distinct ownership. Abort cleans only unpublished files; process death or deletion failure can leave orphans for Paimon cleanup. | | 15. FE/BE propagation | Sink/scan descriptors and commit version still match BE consumers; JNI uses standard Thrift and includes fe-thrift in its isolated plugin. SPI forwarding and the combined baseline agree with API 35. | | 16. Performance and memory | 8 MiB/65,536-row batch targets and bounded commit exports remain. Arrow is Doris-tracked; native/fixed/metadata credits are estimates, not complete RSS enforcement. Spill, revocation and queued admission are outside the accepted scope; concurrency admission can fail a statement and limits are adjustable online. | | 17. Other issues | Reviewed the upstream delta for overlap: new scan SPI, BLOB wire definitions, read-only Arrow optimizations and Fluss runtime cache initialization. No new conflict with this feature was found. The earlier statistics/test_replace_table Jindo missing-class failure also occurs on unrelated PRs and remains outside this rebase. | ## Verification (local) - VERIFIED: `run-fe-ut.sh --run` selecting Paimon*Test and all 37 SPI classes, on 7ad7d7890cea93e3aff26a36afea70aac1842aa7, Maven cache disabled: BUILD SUCCESS; 1,300 tests, zero failures/errors, one skipped. All 165 SPI tests passed. Log: `output/paimon-write-050/fe-tests-rebase-20261118.log`. - VERIFIED: `FE_MAVEN_THREADS=4 MVN_OPT=-Dmaven.build.cache.enabled=false ./build.sh --fe --be-java-extensions` with the same Paimon-focused extension selection as before: successfully built Doris, including Checkstyle and packaging/layout validation. Log: `output/paimon-write-050/fe-package-rebase-20261118.log`. - VERIFIED: range-diff, exact source comparison for Paimon BE/Java paths, both SPI baseline tests, and all required branch protocol comparisons passed. User runtime configuration diffs are preserved exactly. - NOT_RUN locally: BE compilation/tests and SQL regression on this rebased tree; these are assigned to the requested CI rerun. Older BE/SQL evidence remains explicitly tied to its original head. Result: **PASS**, one local self-review round after rebase; no unresolved findings. This review is not an independent review or a claim that the pending CI has passed. <!-- doris-repo-review:v1:begin --> ```yaml schema: doris-repo-review/v1 status: PASS pr: apache/doris#69886 commit: 7ad7d7890cea93e3aff26a36afea70aac1842aa7 base: 098d0bb62bbc82d9bd999d391efe1c20df89930b reviewed_at: 2026-11-18T09:06:27+00:00 reviewer: suxiaogang223 model: gpt-6.1-sol effort: xhigh findings: {blocker: 0, major: 0, minor: 0, nit: 0} rounds: 1 converged: true ``` <!-- doris-repo-review:v1:end --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
