[
https://issues.apache.org/jira/browse/HDDS-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538449#comment-17538449
]
Ethan Rose commented on HDDS-6721:
----------------------------------
Thanks for the input [~szetszwo]. RATIS-1583 has been filed for time based
Ratis snapshots. I have linked it to this Jira.
> Ozone key deletion often dose not delete the real data blocks in datanode
> -------------------------------------------------------------------------
>
> Key: HDDS-6721
> URL: https://issues.apache.org/jira/browse/HDDS-6721
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Shawn
> Priority: Major
>
> When I delete a key of ozone (by using either ozone cli or aws cli), it seems
> ozone does not clean up the real data blocks in datanodes. The key is gone at
> the meta data layer (OM), but not at datanode layer.
> I did the following investigation.
> OM layer: it seems OM functions correctly. I do see the log saying it deletes
> the block. Also The rocksDB table deletedTable is empty.
> SCM layer: it does receive the deletion command from OM. And it tries to
> delete the block. But it seems it only add the transaction in memory, and
> dose not record the transaction in the rocksDB, and thus it does not call
> datanode to do the deletion.
> * No log likes "Totally added x blocks to be deleted for y datanodes" in SCM
> leader node.
> * The deletedBlocks rocksDB table does not have the info of the containerID
> being deleted.
> * And in audit log, it has the log "2022-05-09 18:18:46,553 | INFO |
> SCMAudit | user=om/[email protected] | ip=100.116.76.9 |
> op=DELETE_KEY_BLOCK
> {KeyBlockToDelete=BlockGroup[groupID='/s3v/shawn-test2/tmp/pvc.yaml',
> blockIDs=[conID: 9001 locID: 109611004723209001 bcsId: 0]]} | ret=SUCCESS |"
> Datanode layer: not receive any command from SCM to delete the blocks. The
> blocks of the deleting container still exists on disk. The rocksDB table
> delete_txns of that container is empty.
> I also tried to restart the SCM to flush the ratis logs. But it ends up the
> same as above.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
