764276020 commented on issue #68537:
URL: https://github.com/apache/doris/issues/68537#issuecomment-5864716356
Hello @Doris-Breakwater
Detailed logs for this incident .
**Deployment / object under change**
- Build: `doris-3.1.4-rc02-7f5ba43de6` (tag `3.1.4-rc02`), cloud /
compute-storage-separation, FDB meta service
- Operation: `ALTER TABLE xxx.xx ADD INDEX ... USING INVERTED`
- `job_id = 1789639373133` (FE `SchemaChangeJobV2`), `table_id = 207246110`
- shadow `index_id = 1789639373134`; failing shadow tablet `1789639379947`
(base tablet `1757832974720`)
- storage vault bucket: `s3dwcloud-doris-c2-vol1`
---
### 1. Recycler: the in-progress (PREPARED) index is recycled
```
2026-09-24 14:32:36 sts-doris-dwcloud-recycler-5100-1 I20260924
14:17:46.145588 542 recycler.cpp:1085] begin to recycle index,
instance_id=1638888 table_id=207246110 index_id=1789639373134 state=PREPARED
2026-09-24 14:32:36 sts-doris-dwcloud-recycler-5100-1 I20260924
14:17:46.148905 542 recycler.cpp:1517] begin to recycle tablets of the index
table_id=207246110 index_id=1789639373134 partition_id=-1
```
Reading note: the left column is the log-collector row time; the
message-internal timestamp is `14:17:46.145588` (recycler process clock).
`partition_id=-1` means index-level recycling, i.e. every tablet of the shadow
index (all partitions), which is the code path that removes
`meta_tablet_idx_key`.
### 2. BE (`doris-dwcloud-be-w4sata-31`): conversion runs normally for ~22
minutes, then the commit fails
```
I20260924 12:11:59.172073 2499 task_worker_pool.cpp:2167] get alter table
task, signature: 1789639379947
I20260924 12:17:53.672082 2501 task_worker_pool.cpp:2167] get alter table
task, signature: 1789639379947
I20260924 12:22:04.149425 2500 task_worker_pool.cpp:2167] get alter table
task, signature: 1789639379947
... (task polled every few minutes) ...
I20260924 13:57:27.142143 2500 cloud_meta_mgr.cpp:1787] Skip version hole
filling for new schema change tablet 1789639379947 with alter_version -1
I20260924 13:57:27.157553 2500 cloud_schema_change_job.cpp:153] Begin to
alter tablet. base_tablet_id=1757832974720, new_tablet_id=1789639379947,
alter_version=25647, job_id=1789639373133
I20260924 13:57:27.157934 2500 cloud_schema_change_job.cpp:254] Begin to
convert historical rowsets for new_tablet from base_tablet.
base_tablet=1757832974720, new_tablet=1789639379947, job_id=1789639373133
I20260924 13:57:27.157976 2500 cloud_schema_change_job.cpp:275] schema
change type, sc_sorting: 0, sc_directly: 1, base_tablet=1757832974720,
new_tablet=1789639379947
```
Object storage writes for this tablet then proceed normally (32
`create_multi_upload_request` + 32 `complete_multipart_upload`, 16 rowsets
flushed; each rowset ≈ 241 MB `.dat` + ≈ 275 MB `.idx`):
```
I20260924 13:58:07.145941 2500 s3_file_writer.cpp:88]
create_multi_upload_request
s3://s3dwcloud-doris-c2-vol1/data/1789639379947/02000000078f8b2f5d4468a8eb35de535033262fc6ebb8ad_0.dat
I20260924 13:58:50.133966 482503 s3_file_writer.cpp:406]
complete_multipart_upload
s3://s3dwcloud-doris-c2-vol1/data/1789639379947/02000000078f8b2f5d4468a8eb35de535033262fc6ebb8ad_0.dat
size=241219147 number_parts=8 s3_write_buffer_size=33554432
I20260924 14:15:19.334806 537442 s3_file_writer.cpp:406]
complete_multipart_upload
s3://s3dwcloud-doris-c2-vol1/data/1789639379947/02000000079037db5d4468a8eb35de535033262fc6ebb8ad_3.dat
size=241232607 number_parts=8 s3_write_buffer_size=33554432
I20260924 14:19:27.115656 2095720 s3_obj_storage_client.cpp:239] UploadPart
cost=5080ms, request_id=18D82D64ABED2F0B, bucket=s3dwcloud-doris-c2-vol1,
key=data/1789639379947/02000000079037db5d4468a8eb35de535033262fc6ebb8ad_6.idx,
part_num=1, upload_id=...
I20260924 14:19:27.302443 2500 s3_file_writer.cpp:406]
complete_multipart_upload
s3://s3dwcloud-doris-c2-vol1/data/1789639379947/02000000079037db5d4468a8eb35de535033262fc6ebb8ad_6.idx
size=146927616 number_parts=5 s3_write_buffer_size=33554432
I20260924 14:19:29.280812 2500 segment_creator.cpp:309]
tablet_id:1789639379947, flushing rowset_dir: ,
rowset_id:02000000079037db5d4468a8eb35de535033262fc6ebb8ad, data
size:125997412, index size:146927616
```
Then, 103 seconds after the recycler line above:
```
W20260924 14:19:29.806116 2500 task_worker_pool.cpp:307] failed to alter
tablet|signature=1789639379947|base_tablet_id=1757832974720|new_tablet_id=1789639379947|error=[INTERNAL_ERROR]failed
to commit rowset: failed to get tablet_idx, err=KeyNotFound
tablet_id=1789639379947
key=01106d657461000110313633383838380001107461626c65745f696e646578000112000001a0aed1cbeb
```
### 3. MetaService: the tablet meta itself is already gone
```
msg: "meta_service_job.cpp:1138 failed to get new tablet meta (not found)
instance_id=1638888 tablet_id=1789639379947
key=01106d657461000110313633383838380001107461626c6574000112000000000c5a531e12000001a0aed1b14e12000001994702915912000001a0aed1cbeb
err=KeyNotFound"
```
### 4. FE: a single failed task cancels the whole job
```
2026-09-24 14:19:29,808 WARN (thrift-server-pool-315|66386)
[MasterImpl.finishTask():101] finish task reports bad. request:
TFinishTaskRequest(backend:TBackend(host:doris-dwcloud-be-w4sata-31.doris-dwcloud-be-w4sata.doris-dwcloud.svc.colo.gzgalocal,
be_port:9060, http_port:8040, brpc_port:8060, id:1743196020771),
task_type:ALTER, signature:1789639379947,
task_status:TStatus(status_code:INTERNAL_ERROR,
error_msgs:[(doris-dwcloud-be-w4sata-31.doris-dwcloud-be-w4sata.doris-dwcloud.svc.colo.gzgalocal)[INTERNAL_ERROR]failed
to commit rowset: failed to get tablet_idx, err=KeyNotFound
tablet_id=1789639379947
key=01106d657461000110313633383838380001107461626c65745f696e646578000112000001a0aed1cbeb]),
report_version:178705738452804)
2026-09-24 14:19:30,241 WARN (schema-change-pool-22|688037)
[SchemaChangeJobV2.runRunningJob():619] schema change task failed, job:
1789639373133, failedTimes: 1, maxFailedTimes: 0, err: task type: ALTER,
status_code: INTERNAL_ERROR, status_message:
[(doris-dwcloud-be-w4sata-31.doris-dwcloud-be-w4sata.doris-dwcloud.svc.colo.gzgalocal)[INTERNAL_ERROR]failed
to commit rowset: failed to get tablet_idx, err=KeyNotFound
tablet_id=1789639379947
key=01106d657461000110313633383838380001107461626c65745f696e646578000112000001a0aed1cbeb]
2026-09-24 14:19:30,291 INFO (schema-change-pool-22|688037)
[SchemaChangeJobV2.cancelImpl():838] set table's state to NORMAL when cancel,
table id: 207246110, job id: 1789639373133
2026-09-24 14:19:30,485 INFO (schema-change-pool-22|688037)
[SchemaChangeJobV2.cancelImpl():841] cancel SCHEMA_CHANGE job 1789639373133,
err: errCode = 2, detailMessage = schema change tasks failed, error reason:
task type: ALTER, status_code: INTERNAL_ERROR, status_message:
[(doris-dwcloud-be-w4sata-31.doris-dwcloud-be-w4sata.doris-dwcloud.svc.colo.gzgalocal)[INTERNAL_ERROR]failed
to commit rowset: failed to get tablet_idx, err=KeyNotFound
tablet_id=1789639379947
key=01106d657461000110313633383838380001107461626c65745f696e646578000112000001a0aed1cbeb]
```
After the cancel, other BEs of the same job keep reporting (253 rows between
14:19:36 and 14:30):
```
2026-09-24 14:19:36,474 WARN (thrift-server-pool-179602|572505)
[MasterImpl.finishTask():101] finish task reports bad. request:
TFinishTaskRequest(backend:TBackend(host:doris-dwcloud-be-w4sata-2.doris-dwcloud-be-w4sata.doris-dwcloud.svc.colo.gzgalocal,
be_port:9060, http_port:8040, brpc_port:8060, id:13750875), task_type:ALTER,
signature:1789639374431, task_status:TStatus(status_code:INTERNAL_ERROR,
error_msgs:[(doris-dwcloud-be-w4sata-2.doris-dwcloud-be-w4sata.doris-dwcloud.svc.colo.gzgalocal)[INTERNAL_ERROR]failed
to commit rowset: stale perpare rowset request, instance_id=1638888
tablet_id=1789639374431 job id=1789639373133
rowset_id=0200000002413138b04d70ea8c9d5d5227e8cd54ce5ced84]),
report_version:178938202914362)
```
and the cancelled job is replayed on the FE checkpointer:
```
2026-09-24 14:30:07,671 INFO (leaderCheckpointer|66130)
[SchemaChangeJobV2.replayCancelled():994] replay cancelled schema change job:
1789639373133
2026-09-24 14:30:07,671 INFO (leaderCheckpointer|66130)
[SchemaChangeJobV2.replayCancelled():996] set table's state to NORMAL when
replay cancelled, table id: 207246110, job id: 1789639373133
```
and 14 minutes later the recycled tablet is still unresolvable:
```
I20260924 14:33:18.244233 2407 cloud_warm_up_manager.cpp:689]
recycle_cache: tablet_id=1789639379947, num_rowsets=1
W20260924 14:33:18.244154 2407 status.h:427] meet error status:
[NOT_FOUND]failed to get tablet meta: failed to get tablet_idx, err=KeyNotFound
tablet_id=1789639379947
key=01106d657461000110313633383838380001107461626c65745f696e646578000112000001a0aed1cbeb
W20260924 14:33:18.244251 2407 cloud_tablet_mgr.cpp:369] failed to sync
tablet meta 1789639379947
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
