Doris-Breakwater commented on issue #67858: URL: https://github.com/apache/doris/issues/67858#issuecomment-5631247657
Initial maintainer triage (source-checked against `4.0.3-rc03` and `branch-4.0`; no controlled reproduction was run): ### Assessment There is a real restart-time availability risk in this area, and the loss of the underlying error in `BetaRowset::load_segment()` is confirmed. However, the multi-hour failure is **not yet explained by the proposed root-cause chain**, because one important claim is contradicted by the same-version FE/BE code. **Verified facts** - In non-cloud mode, `s_storage_resource_mgr` is process-local and is not persisted. Among production non-cloud paths, `push_storage_policy_callback()` is what constructs the S3/HDFS filesystem and calls `put_storage_resource()`. - A remote-rowset read can therefore miss this map after restart. `RowsetMeta::remote_storage_resource()` has no on-demand local-mode lookup; `physical_fs()` logs the detailed status and returns null; `BetaRowset::load_segment()` then replaces it with the generic `get fs failed`. Adding `resource_id` would improve the symptom, although propagating the original status would be better. - The cloud-mode guard itself is not evidence of a missing local recovery path. Its body calls `storage_engine().to_cloud().sync_storage_vault()`, while `to_cloud()` checks that the engine type is `CLOUD`. Simply removing the guard would make the local path hit that check; local mode needs a distinct FE-backed mechanism. **Contradiction that must be resolved** `report_tablet_callback()` explicitly reports the BE's in-memory storage-policy and resource ID/version lists, including empty lists after a restart. `ReportHandler.storagePolicyReport()` compares those lists with FE metadata, and `diffPolicy()` / `diffResource()` add missing entries to a `PushStoragePolicyTask`. The default tablet-report interval is 60 seconds (plus up to 5 seconds of random delay), and a changed/initial master heartbeat can also wake report workers immediately. Consequently, the code already intends to repopulate the map after a BE restart. A brief failure before the first successful report/push is plausible from source inspection, but failures persisting for hours require another failure in that pipeline (or a different condition) and cannot currently be attributed only to `is_cloud_mode()`. The production timeline also needs reconciliation: `_cooldown_data()` obtains the filesystem through `get_resource_by_storage_policy_id()`. If the two partitions successfully cooled through the same resource while older-rowset reads still failed, the global map existed at that time and subsequent reads should have found it. Exact completion timestamps and resource IDs could show whether the old rowsets referenced a different/stale resource ID or whether the successful cooldown occurred later than the policy attachment. ### Information needed 1. A controlled run with a full scan immediately before restart, immediately after restart, and again after at least two configured tablet-report intervals, with no intervening DDL/load/policy change. Please include the effective `report_tablet_interval_seconds` and `report_random_wait` values. 2. FE logs from the first post-restart heartbeat through those report intervals, especially lines containing `backend[...] reports policies`, `after diff policy`, and any agent-task submission/RPC error. 3. BE INFO/WARNING logs for the same window, including `failed to report tablet`, `successfully submit task` for `PUSH_STORAGE_POLICY`, and `successfully update s3 resource` or `update s3 resource failed`. 4. The exact `resource_id` from the read failure and the current FE resource ID/version, plus confirmation that the resource was not dropped/recreated. Please redact credentials from resource properties. 5. For the September 7 sequence, timestamps for the actual cooldown upload completion and the last failing read, not only the policy-attachment time. ### Recommended next steps - First locate the break across BE report -> FE diff -> task submission -> BE resource construction using the logs above, then fix that specific stage and add a restart regression/integration test. - Independently harden startup so remote reads do not race the initial local-mode resource synchronization; an eager post-heartbeat push/rebuild is consistent with the existing architecture. Do not reuse the cloud vault sync by only removing the mode check. - Independently preserve the `remote_storage_resource()` failure through the read stack (or at minimum include `resource_id`, tablet ID, and rowset ID in `load_segment()`'s error). Current triage: **credible bug / startup race and observability defect confirmed; claimed permanent non-cloud re-sync absence and the multi-hour root cause remain unverified pending the report/push evidence above.** The issue currently has no labels; a bug plus storage-policy/cooldown area label would be appropriate under the repository's label taxonomy. Breakwater-GitHub-Analysis-Slot: slot_680b283fba8a -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
