Shawn created HDDS-6721:
---------------------------
Summary: Ozone key deletion often dose not delete the real data
blocks in datanode
Key: HDDS-6721
URL: https://issues.apache.org/jira/browse/HDDS-6721
Project: Apache Ozone
Issue Type: Bug
Reporter: Shawn
When I delete a key of ozone (by using either ozone cli or aws cli), it seems
ozone does not clean up the real data blocks in datanodes. The key is gone at
the meta data layer (OM), but not at datanode layer.
I did the following investigation.
OM layer: it seems OM functions correctly. I do see the log saying it deletes
the block. Also The rocksDB table deletedTable is empty.
SCM layer: it does receive the deletion command from OM. And it tries to delete
the block. But it seems it only add the transaction in memory, and dose not
record the transaction in the rocksDB, and thus it does not call datanode to do
the deletion.
* No log likes "Totally added x blocks to be deleted for y datanodes" in SCM
leader node.
* The deletedBlocks rocksDB table does not have the info of the containerID
being deleted.
* And in audit log, it has the log "2022-05-09 18:18:46,553 | INFO | SCMAudit
| user=om/[email protected] | ip=100.116.76.9 |
op=DELETE_KEY_BLOCK
{KeyBlockToDelete=BlockGroup[groupID='/s3v/shawn-test2/tmp/pvc.yaml',
blockIDs=[conID: 9001 locID: 109611004723209001 bcsId: 0]]} | ret=SUCCESS |"
Datanode layer: not receive any command from SCM to delete the blocks. The
blocks of the deleting container still exists on disk. The rocksDB table
delete_txns of that container is empty.
I also tried to restart the SCM to flush the ratis logs. But it ends up the
same as above.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
