suxiaogang223 commented on PR #69886:
URL: https://github.com/apache/doris/pull/69886#issuecomment-6968388158
Local review completed with the Doris `code-review` skill for
`3834b66be057a6acd1f0c70542e56ebc37b18116` (base
`c212aa81dd479f6cb821701c1d6f4298ce01e4a5`). **PASS: no unresolved findings.**
The REST credential-refresh regression is fixed: JNI receives the effective
catalog context and remote identity, rebuilds RESTTokenFileIO without catalog
schema lookups, and refreshes expired tokens for subsequent file operations.
These tables select JNI before input consumption because the Rust adapter
cannot refresh REST credentials. The branch is rebased on master and the
connector SPI is 34.0.
Validation:
- FE build and Paimon Java plugin packaging passed, including Checkstyle and
plugin-layout checks.
- 1,141 related FE/Java tests: zero failures/errors, one skipped. Full
connector SPI suite: 165 tests passed. New REST tests cover descriptor
context/identity, writer selection, and expired-token refresh between two
actual file creations using a local HTTP server.
- The prior 59 ASAN BE tests (23 FFI writer tests) passed at 9b77afc3914. BE
writer/test source is unchanged by this Java follow-up and rebase; no new BE
rebuild is claimed. Earlier SQL smoke/distributed fault tests predate this fix.
Updated wide-row SQL, real STS expiry and process-crash integration are not
claimed as locally rerun.
- Compared affected protocol definitions across master and branches
4.2/4.1/4.0/3.1: no ID/type/name/optionality conflicts.
The author's confirmed scope still applies: mixed-version Paimon execution
may return errors safely; default scheduling can exceed admission limits and
fail, with online adjustment/lower parallelism and retry; native memory
accounting is estimated, with no spill/revocation or full RSS enforcement.
These limitations are documented in the PR.
<details>
<summary>Doris code-review Part 1.3: individual checkpoint
conclusions</summary>
| Checkpoint | Conclusion |
| --- | --- |
| 1. Goal and proof | Supported append writes select Rust before input;
unsupported cases select JNI. The REST follow-up restores credential refresh
while retaining the pinned schema. BE tests cover memory release, failure
cleanup and v14 messages; new Java tests cover refresh and routing. |
| 2. Small and focused | Reuses the common descriptor and SDK
RESTTokenFileIO; eight Java files change in the follow-up, with no Thrift, C++
or dependency change. Rebase also advances this feature's connector SPI to 34
over upstream 33. |
| 3. Concurrency and locks | Pipeline local state owns a writer. Rust buffer
release may run on SDK threads and attaches the query context. A short mutex
protects admission slots; native calls, workload refresh and abort IO run
outside it. FE transaction state uses synchronized sections; IO is outside
them. No lock-order cycle found. |
| 4. Lifetimes and static initialization | Native stop precedes Arrow-pool
and budget release. Prepared metadata keeps builder/messages/tracker until
report resolution. Failure paths return credits and slots. Metrics initialize
lazily. REST API/token concurrency is owned by the existing SDK wrapper. No new
lifetime defect found. |
| 5. Configuration | Both BE limits are mutable and checked positive. New
writers observe changes without restart; existing writers keep their budget.
Tests cover online adjustment and retry. No new config is added by the REST
fix. |
| 6. Compatibility | v14 bodies/DPCM framing remain consistent. Missing
descriptors and mismatched JNI methods fail safely. Mixed-version read/write
functionality is explicitly outside the author's accepted scope; even disabled
Rust writes require matching binaries. This limitation is documented and is not
reported again as a new defect. |
| 7. Thrift/Protobuf | Read affected definitions on
HEAD/master/4.2/4.1/4.0/3.1 and compared common IDs, names, types and
optionality. No collisions or reused IDs; new fields optional; descriptor ID 7
remains reserved. No protobuf change. Exact definitions recorded in
`thrift-review-rest-fix.txt`. |
| 8. Parallel paths | Rust reads and Rust/JNI writes reuse pinned metadata
and storage normalization. JNI REST refresh is retained; Rust excludes this
provider behavior before consuming rows. The existing Rust read
credential-refresh capability is not expanded. |
| 9. Conditions and error handling | REST conditions correspond to a real
SDK FileIO/option and empty catalog environments are handled. SDK errors become
Status; no fallback occurs after input consumption. Wide rows use the admitted
budget and form a nonempty batch even above the normal target. No ignored new
Status found. |
| 10. Test coverage | Tests cover scalar/null conversion, batching, wide
rows, native/query/fixed credit, online limits, cancellation, prepare/open
failures, report ownership, v14 framing and JNI descriptor semantics. REST
tests add remote identity/context, provider routing and expired-token refresh
between files. Existing two-BE SQL fault tests cover before/after/uncertain
commit outcomes. Updated wide-row SQL awaits CI; no real STS service or
process-crash integration test was run. |
| 11. Test results | The final Paimon FE/Java run completed 1,141 tests,
zero failures/errors, one skipped. The complete SPI suite passed all 165 tests.
Prior 59 ASAN BE tests passed at 9b77afc3914; all BE writer/test content is
unchanged by the Java follow-up and rebase. SQL output files were generated by
the regression runner, not handwritten. Java tests deserialize and reserialize
the v14 fixture byte-for-byte. |
| 12. Observability | Profiles expose backend/selection reason,
rows/batches, fixed/native/Arrow memory, admission and failure counters/timers.
BE exposes active writer and credit metrics; FE uses bounded
commit/abort/retry/unknown/cleanup labels. REST fallback is visible as
`rest_vended_credentials`. No added per-row INFO logging or credential-bearing
SDK text. |
| 13. Transactions/persistence/failover | FE remains the only snapshot
committer and uses existing transaction management. No new EditLog.
Pre-publication failure aborts; COMMITTING/OUTCOME_UNKNOWN never deletes
potentially visible files. Durable recovery of an in-flight external
transaction across FE death remains an existing framework limitation. |
| 14. Atomicity and crash cleanup | All BE messages feed one FE commit.
ACK/rejection/ambiguity have distinct file ownership. Committed state survives
committer-close errors; late aborted reports are cleaned outside locks. BE/FE
death or failed deletion can leave uncommitted orphans for Paimon cleanup, as
accepted in the design. |
| 15. FE/BE propagation | Session switch, sink/scan descriptors and commit
version match consumers. JNI uses standard Thrift serialization and ships
fe-thrift in its isolated plugin. Forwarding SPI methods and surface baselines
are updated, with API 34. REST refresh carries catalog options/remote identity
through existing optional fields. |
| 16. Performance and memory | Batch target is 8 MiB/65,536 rows; a wider
admitted row is sent alone. Commit export is bounded by 512 bodies/8 MiB per
envelope and 64 MiB total. Arrow is Doris-tracked; fixed/native/metadata
credits are estimates, not full RSS enforcement. Spill/revocation and queued
admission are not claimed. Default scheduling can hit the eight-writer cap;
online adjustment/lower parallelism and retry is the approved policy. Redundant
FE REST token fetch is removed. |
| 17. Other issues | Checked classloader/package closure, CHAR fallback,
branch identity, schema pinning, read-path validation and rebase integration.
No additional actionable defect confirmed. Existing REST endpoint access/LOAD
and catalog privilege requirements are stated in the PR description. |
</details>
<!-- doris-repo-review:v1:begin -->
```yaml
schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69886
commit: 3834b66be057a6acd1f0c70542e56ebc37b18116
base: c212aa81dd479f6cb821701c1d6f4298ce01e4a5
reviewed_at: 2026-11-18T03:43:56+00:00
reviewer: suxiaogang223
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 2
converged: true
```
<!-- doris-repo-review:v1:end -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]