Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5289747725
Thanks for the comprehensive v5 revision. After reading the full discussion history and rechecking the branch-4.1 boundary, I agree that the three round-5 correctness issues are now addressed: UNKNOWN keeps its same-name fence, outcome and refresh are durable independent dimensions, and RESOLVE has an explicit audited/replayable protocol. For 4.2, I do not want to expand this into a complete external-object identity or distributed-query project. I am comfortable deferring the following, provided they are documented as explicit limitations: - external deletion and recreation of a different dataset at the same URI while a mutation is active or UNKNOWN; - a full provider-backed table-incarnation registry; - moving all Directory metadata reads into an isolated helper, unless the required JNI stress/failure evidence shows that the bounded FE executor is insufficient; - distributed fragment-parallel vector TopN. The current branch-4.1 path intentionally uses one whole-dataset split and one scanner/BE for global TopK, so the documentation should state that distributed vector search/merge is not provided in 4.2. Before approving the mutation part, I think only the following contained changes are required: 1. **Do not let an UNKNOWN job block credential rotation.** A credential-only catalog ALTER should not be classified as target-changing DDL. Locator/provider/version-selection changes must remain guarded, but watchers and refresh should be able to use newly rotated credentials; otherwise an expired session token makes reconciliation impossible and forces an emergency release without refresh. 2. **Add admission quotas for durable unresolved work.** Bound active+UNKNOWN jobs per table/catalog and globally, and reject before durable job creation when the bound is reached. Running concurrency and per-cycle reconciliation limits do not bound the number of permanent job records, fences, tombstones, or watcher entries. 3. **Keep possible-live worker accounting separate from the client/job deadline.** If catalog/cluster concurrency is documented as a hard bound, its slot cannot be automatically reused while an old child may still be running. Release the possible-live slot only after matching reap, BE process-epoch proof, or audited FORCE. The statement/job deadline may still expire independently. 4. **Define a small versioned schema-comparison contract.** This does not need to become a general Arrow canonicalization protocol. A bounded ordered representation of the fields relevant to admission—field ID, normalized name/type, nullability, and fixed-size-list dimension, plus any other intentionally guarded fields—is sufficient, as long as Java and Rust compare the same specified representation across replay and upgrade. For target identity in 4.2, a persisted catalog identity plus provider and normalized stable dataset locator is sufficient; same-URI external replacement can remain the documented unsupported race above. There is no need to add a large incarnation subsystem solely for that corner case. With these four changes and the deferred limitations stated clearly, I would consider the v5 design aligned with the practical 4.2 scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
