suxiaogang223 commented on PR #69886: URL: https://github.com/apache/doris/pull/69886#issuecomment-6980975406
# Local self-review after rebase PR: https://github.com/apache/doris/pull/69886 Head: `fde0e2d56f1a0999020a226f3662672bb88db41c` Base: `29c1083f65fe92bb5a778a660fde3e8671ef537b` FE selects Rust/JNI before input, BE writes and prepares files, the report acknowledgement transfers ownership, and FE publishes one Paimon snapshot. Rebase preserves this invocation chain. All nine feature patches replay unchanged; SPI API 35 is retained. The new upstream delta was checked for changes to the APIs and headers this feature uses. ## Part 1.3 checkpoints | Checkpoint | Conclusion | | --- | --- | | 1. Goal and proof | Rebase the Rust append writer onto master and rerun CI. The SPI version test and both surface tests verify the resolved collision; Paimon behavior tests exercise writer selection, descriptors and v14 decoding. | | 2. Scope and reuse | All nine entries in range-diff are unchanged. Paimon BE/Java code and tests, the SPI and Thrift definitions are byte-identical to the previous head. No new feature code, compatibility layer or thirdparty change is added. | | 3. Concurrency and locks | Each pipeline local state owns a writer. The admission mutex protects the active count; SDK calls, workload refresh and file deletion run outside it. Arrow release on Rust threads restores the query context. FE transaction state remains synchronized. | | 4. Lifecycle and static initialization | Native stop precedes Arrow-pool/budget release. Prepared-file cleanup retains messages, builder and tracker until report resolution. Failure exits release admission/credits; metrics initialize lazily. These paths are unchanged by rebase. | | 5. Dynamic configuration | Both BE limits remain mutable. New writers read current values; active writer budgets stay fixed. Online adjustment and retry are covered by existing BE tests. | | 6. Compatibility | SPI major 35 separates the combined surface from upstream 34. Missing descriptors/JNI methods fail safely. Functional mixed-version Paimon reads/writes are an explicitly accepted limitation, including with Rust writes disabled; no compatibility layer is added. | | 7. Thrift/Protobuf IDs | Read all five touched definitions on HEAD, master, 4.2, 4.1, 4.0 and 3.1. Common field IDs, names, types and optionality agree; all added fields are optional; descriptor ID 7 stays reserved. Upstream BLOB fields 21-23 are retained and scan field 43 is free. No protobuf change. | | 8. Parallel paths | Rust reads and Rust/JNI writes retain the shared pinned descriptor. REST-vended credentials select JNI before input consumption and reconstruct refreshable RESTTokenFileIO. Upstream scan API additions are preserved. | | 9. Conditions and errors | SDK failures become non-OK Status and never trigger JNI fallback after input. Wide-row checks use the admitted budget and oversized batches advance by one row. No new condition or error-suppression path is introduced by rebase. | | 10. Test coverage | Existing tests cover null/scalar conversion, wide rows, cancellation, admission/credit release, prepare/open failure, report ownership, commit framing and REST refresh. Updated SQL cases await CI; no real STS expiry or process-crash test is claimed. | | 11. Test results | At prior head 7ad7d7890ce, 1,300 FE/Java tests completed with zero failures/errors and one skipped, including 165 SPI tests; Java packaging passed. Prior-head CI FE UT, BE UT and Cloud UT passed. No fresh build/UT/SQL run is claimed for this rebase. Current static hygiene, range-diff and protocol checks passed. | | 12. Observability | Writer selection, batches/rows, Arrow/native/fixed credit and failure/abort timings remain in profiles. Active/credit/rejection metrics and bounded FE transaction labels remain intact. No per-row INFO logging is added. | | 13. Persistence and failover | FE remains the snapshot committer through the existing transaction framework; no new EditLog format. COMMITTING/OUTCOME_UNKNOWN preserve files. Durable recovery of an in-flight external transaction across FE death remains a framework limitation. | | 14. Atomicity and crash cleanup | All BE messages feed one FE snapshot commit. ACK, definite rejection and ambiguous reports have distinct ownership. Abort cleans only unpublished files; process death or deletion failure can leave orphans for Paimon cleanup. | | 15. FE/BE propagation | Sink/scan descriptors and commit version still match BE consumers; JNI uses standard Thrift and includes fe-thrift in its isolated plugin. SPI forwarding and the combined baseline agree with API 35. | | 16. Performance and memory | 8 MiB/65,536-row batch targets and bounded commit exports remain. Arrow is Doris-tracked; native/fixed/metadata credits are estimates, not complete RSS enforcement. Spill, revocation and queued admission are outside the accepted scope; concurrency admission can fail a statement and limits are adjustable online. | | 17. Other issues | Reviewed the five new upstream commits for overlap. The object-storage refactor changes RuntimeState error-log storage and ExecEnv storage clients, not the report/budget API used here; this feature references none of the removed S3 headers. Other upstream changes concern row-id compression, aggregate-state coercion, function hashing and reviewer assignment. No additional feature defect found. | ## Verification and remaining CI work - VERIFIED (static): all nine feature patches are unchanged in range-diff; Paimon writer/reader/Java/SPI/tests and Thrift source comparisons are empty against 7ad7d7890ce. - VERIFIED (static): build-support/check-build-hygiene.sh passed, and all five touched Thrift definitions were compared on HEAD, master and branches 4.2/4.1/4.0/3.1. No field collision/reuse/new required field. - VERIFIED: the user runtime configuration changes exactly match the pre-rebase stash; no user docs/native files were changed. - Prior evidence, not new runs: 1,300 local FE/Java tests (one skipped), Java packaging, and CI FE UT/BE UT/Cloud UT passed at 7ad7d7890ce. CI binary compilation also succeeded there; build 6627 subsequently failed in the CI Jindo Java launch with E2BIG. - NOT_RUN locally on the current rebased tree: compilation, UT and SQL execution; these are assigned to the requested CI rerun. The CI helper correction reported by the author is to be verified by that run. Result: **PASS**, one local self-review round after this rebase; no unresolved findings in the feature diff. This is not an independent review or a claim that pending CI has passed. <!-- doris-repo-review:v1:begin --> ```yaml schema: doris-repo-review/v1 status: PASS pr: apache/doris#69886 commit: fde0e2d56f1a0999020a226f3662672bb88db41c base: 29c1083f65fe92bb5a778a660fde3e8671ef537b reviewed_at: 2026-11-18T14:08:03+00:00 reviewer: suxiaogang223 model: gpt-6.1-sol effort: xhigh findings: {blocker: 0, major: 0, minor: 0, nit: 0} rounds: 1 converged: true ``` <!-- doris-repo-review:v1:end --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
