suxiaogang223 commented on PR #69886:
URL: https://github.com/apache/doris/pull/69886#issuecomment-6968388158

   Local review completed with the Doris `code-review` skill for 
`3834b66be057a6acd1f0c70542e56ebc37b18116` (base 
`c212aa81dd479f6cb821701c1d6f4298ce01e4a5`). **PASS: no unresolved findings.**
   
   The REST credential-refresh regression is fixed: JNI receives the effective 
catalog context and remote identity, rebuilds RESTTokenFileIO without catalog 
schema lookups, and refreshes expired tokens for subsequent file operations. 
These tables select JNI before input consumption because the Rust adapter 
cannot refresh REST credentials. The branch is rebased on master and the 
connector SPI is 34.0.
   
   Validation:
   
   - FE build and Paimon Java plugin packaging passed, including Checkstyle and 
plugin-layout checks.
   - 1,141 related FE/Java tests: zero failures/errors, one skipped. Full 
connector SPI suite: 165 tests passed. New REST tests cover descriptor 
context/identity, writer selection, and expired-token refresh between two 
actual file creations using a local HTTP server.
   - The prior 59 ASAN BE tests (23 FFI writer tests) passed at 9b77afc3914. BE 
writer/test source is unchanged by this Java follow-up and rebase; no new BE 
rebuild is claimed. Earlier SQL smoke/distributed fault tests predate this fix. 
Updated wide-row SQL, real STS expiry and process-crash integration are not 
claimed as locally rerun.
   - Compared affected protocol definitions across master and branches 
4.2/4.1/4.0/3.1: no ID/type/name/optionality conflicts.
   
   The author's confirmed scope still applies: mixed-version Paimon execution 
may return errors safely; default scheduling can exceed admission limits and 
fail, with online adjustment/lower parallelism and retry; native memory 
accounting is estimated, with no spill/revocation or full RSS enforcement. 
These limitations are documented in the PR.
   
   <details>
   <summary>Doris code-review Part 1.3: individual checkpoint 
conclusions</summary>
   
   | Checkpoint | Conclusion |
   | --- | --- |
   | 1. Goal and proof | Supported append writes select Rust before input; 
unsupported cases select JNI. The REST follow-up restores credential refresh 
while retaining the pinned schema. BE tests cover memory release, failure 
cleanup and v14 messages; new Java tests cover refresh and routing. |
   | 2. Small and focused | Reuses the common descriptor and SDK 
RESTTokenFileIO; eight Java files change in the follow-up, with no Thrift, C++ 
or dependency change. Rebase also advances this feature's connector SPI to 34 
over upstream 33. |
   | 3. Concurrency and locks | Pipeline local state owns a writer. Rust buffer 
release may run on SDK threads and attaches the query context. A short mutex 
protects admission slots; native calls, workload refresh and abort IO run 
outside it. FE transaction state uses synchronized sections; IO is outside 
them. No lock-order cycle found. |
   | 4. Lifetimes and static initialization | Native stop precedes Arrow-pool 
and budget release. Prepared metadata keeps builder/messages/tracker until 
report resolution. Failure paths return credits and slots. Metrics initialize 
lazily. REST API/token concurrency is owned by the existing SDK wrapper. No new 
lifetime defect found. |
   | 5. Configuration | Both BE limits are mutable and checked positive. New 
writers observe changes without restart; existing writers keep their budget. 
Tests cover online adjustment and retry. No new config is added by the REST 
fix. |
   | 6. Compatibility | v14 bodies/DPCM framing remain consistent. Missing 
descriptors and mismatched JNI methods fail safely. Mixed-version read/write 
functionality is explicitly outside the author's accepted scope; even disabled 
Rust writes require matching binaries. This limitation is documented and is not 
reported again as a new defect. |
   | 7. Thrift/Protobuf | Read affected definitions on 
HEAD/master/4.2/4.1/4.0/3.1 and compared common IDs, names, types and 
optionality. No collisions or reused IDs; new fields optional; descriptor ID 7 
remains reserved. No protobuf change. Exact definitions recorded in 
`thrift-review-rest-fix.txt`. |
   | 8. Parallel paths | Rust reads and Rust/JNI writes reuse pinned metadata 
and storage normalization. JNI REST refresh is retained; Rust excludes this 
provider behavior before consuming rows. The existing Rust read 
credential-refresh capability is not expanded. |
   | 9. Conditions and error handling | REST conditions correspond to a real 
SDK FileIO/option and empty catalog environments are handled. SDK errors become 
Status; no fallback occurs after input consumption. Wide rows use the admitted 
budget and form a nonempty batch even above the normal target. No ignored new 
Status found. |
   | 10. Test coverage | Tests cover scalar/null conversion, batching, wide 
rows, native/query/fixed credit, online limits, cancellation, prepare/open 
failures, report ownership, v14 framing and JNI descriptor semantics. REST 
tests add remote identity/context, provider routing and expired-token refresh 
between files. Existing two-BE SQL fault tests cover before/after/uncertain 
commit outcomes. Updated wide-row SQL awaits CI; no real STS service or 
process-crash integration test was run. |
   | 11. Test results | The final Paimon FE/Java run completed 1,141 tests, 
zero failures/errors, one skipped. The complete SPI suite passed all 165 tests. 
Prior 59 ASAN BE tests passed at 9b77afc3914; all BE writer/test content is 
unchanged by the Java follow-up and rebase. SQL output files were generated by 
the regression runner, not handwritten. Java tests deserialize and reserialize 
the v14 fixture byte-for-byte. |
   | 12. Observability | Profiles expose backend/selection reason, 
rows/batches, fixed/native/Arrow memory, admission and failure counters/timers. 
BE exposes active writer and credit metrics; FE uses bounded 
commit/abort/retry/unknown/cleanup labels. REST fallback is visible as 
`rest_vended_credentials`. No added per-row INFO logging or credential-bearing 
SDK text. |
   | 13. Transactions/persistence/failover | FE remains the only snapshot 
committer and uses existing transaction management. No new EditLog. 
Pre-publication failure aborts; COMMITTING/OUTCOME_UNKNOWN never deletes 
potentially visible files. Durable recovery of an in-flight external 
transaction across FE death remains an existing framework limitation. |
   | 14. Atomicity and crash cleanup | All BE messages feed one FE commit. 
ACK/rejection/ambiguity have distinct file ownership. Committed state survives 
committer-close errors; late aborted reports are cleaned outside locks. BE/FE 
death or failed deletion can leave uncommitted orphans for Paimon cleanup, as 
accepted in the design. |
   | 15. FE/BE propagation | Session switch, sink/scan descriptors and commit 
version match consumers. JNI uses standard Thrift serialization and ships 
fe-thrift in its isolated plugin. Forwarding SPI methods and surface baselines 
are updated, with API 34. REST refresh carries catalog options/remote identity 
through existing optional fields. |
   | 16. Performance and memory | Batch target is 8 MiB/65,536 rows; a wider 
admitted row is sent alone. Commit export is bounded by 512 bodies/8 MiB per 
envelope and 64 MiB total. Arrow is Doris-tracked; fixed/native/metadata 
credits are estimates, not full RSS enforcement. Spill/revocation and queued 
admission are not claimed. Default scheduling can hit the eight-writer cap; 
online adjustment/lower parallelism and retry is the approved policy. Redundant 
FE REST token fetch is removed. |
   | 17. Other issues | Checked classloader/package closure, CHAR fallback, 
branch identity, schema pinning, read-path validation and rebase integration. 
No additional actionable defect confirmed. Existing REST endpoint access/LOAD 
and catalog privilege requirements are stated in the PR description. |
   
   </details>
   
   <!-- doris-repo-review:v1:begin -->
   ```yaml
   schema: doris-repo-review/v1
   status: PASS
   pr: apache/doris#69886
   commit: 3834b66be057a6acd1f0c70542e56ebc37b18116
   base: c212aa81dd479f6cb821701c1d6f4298ce01e4a5
   reviewed_at: 2026-11-18T03:43:56+00:00
   reviewer: suxiaogang223
   model: gpt-6.1-sol
   effort: xhigh
   findings: {blocker: 0, major: 0, minor: 0, nit: 0}
   rounds: 2
   converged: true
   ```
   <!-- doris-repo-review:v1:end -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to