errose28 commented on PR #7548: URL: https://github.com/apache/ozone/pull/7548#issuecomment-2657356918
> This can be a similar method to check if the key is readable without downloading the key. Yes I agree this tool would fit well under the `ozone debug replicas verify` proposal, I think we just need to work out some of the implementation details. > * The container status check cannot be bypassed. You need to execute getContainer before executing GetBlock. Only normal containers can perform GetBlock. I don't follow this. `GetBlock` only needs a block token generated by the OM or datanode. Calling SCM's `getContainer` is not required, nor is it done on the read or reconstruct paths where this API is currently used. I am also not sure what is meant by a normal container here. Does this mean one where replicas are not missing or deleted? In that case reading the block metadata would fail as expected. > The `headBlocks` mentioned in the document will check the DB and Block files, and its return value will clearly indicate whether it is missing metadata or Block files, or other exceptions This sounds like a good addition. I think the new API should be called something like `verifyBlocks` instead of `headBlock` though. Head requests usually retrieve metadata in the same way that `getBlock` already does, but the description here returns results about metadata verification, not the actual metadata. > We only need to save the mapping of container -> key of the key being checked in memory. We don't need to save the mapping of all keys. We can control the number of keys being checked through parameters. Got it. I don't think it's clear in the doc whether the steps are applied to a single key or a whole section of the namespace. For the proposed `ozone debug replicas verify` command we would like it to operate on a specified portion of the namespace. This allows the command to parallelize listing commands across buckets and run verifications for a list batch concurrently while the next listing batch is being fetched. It sounds like the block verification API is looking to batch requests for multiple blocks in one request to the DN. In this case the per-key mapping should probably be datanode to block, not container to block, since blocks for the key may be under the same datanode but in different containers. > The internal version we are currently using (using `listKeys` to obtain metadata from OM) can reach a speed of 70k/s (online users have no obvious perception), and the bottleneck here is OM's `listKeys`. If `listKeysLight` is used, it can reach a higher speed. I don't see how this would work because `listKeysLight` does not include block information. That is what makes it fast. However we need both the block locations in the cluster and the block tokens to read them to run the verification checks. [ListKeysResponse](https://github.com/apache/ozone/blob/79ca95672ac40e886acfc544910f8b551ed88060/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto#L1491) returns [KeyInfo](https://github.com/apache/ozone/blob/79ca95672ac40e886acfc544910f8b551ed88060/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto#L1144) objects which if you drill down have the [block token](https://github.com/apache/ozone/blob/79ca95672ac40e886acfc544910f8b551ed88060/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto#L1090) present. The [BasicKeyInfo](https://github.com/apache/ozone/blob/79ca95672ac40e886acfc544910f8b551ed88060/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto#L1497) in [ListKeysLightResp onse](https://github.com/apache/ozone/blob/79ca95672ac40e886acfc544910f8b551ed88060/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto#L1496) has no informtation that can be used to locate or read the blocks. Since the list keys API is paginated, the bottleneck on list keys would likely be alleviated by starting the block checks for one listing batch while concurrently fetching the next listing batch as mentioned above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
