[
https://issues.apache.org/jira/browse/HDFS-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154787#comment-16154787
]
Weiwei Yang commented on HDFS-12235:
------------------------------------
Hi [~nandakumar131]
Thanks for helping to review this patch. Please see my comments below
bq. Should we have metadataManager.start() inside KeyManager#start()?
Good suggestion, I have done so in v8 patch.
bq. If DeleteKeyBlocksResult#isSuccess returns false, we should log it in KSM.
Makes sense to me, addressed in v8 patch.
bq. With DeleteKeyBlocksResult and KsmKeyBlocks in
StorageContainerManager#deleteKeyBlocks call now SCM is aware of KSM and knows
the key to block mapping, which SCM doesn't have to know...
I agree ideally SCM should know nothing about the key, but we found there is no
other good alternative to get this work neatly other than passing the key. The
problem is how KSM handles the ACKs from SCM. Suppose the key is not included
in the message exchange, SCM will response a list of delete block results (each
block might be successful or fail). Like following
B1 : success,
B2 : success,
B3 : fail
...
When KSM gets this info, how can it know which key is successfully deleted?
Following is what we have in KSM DB
key1 : \[B11, B12 ...\],
key2 : \[B21, B22 ...\],
key3 : \[B31, B32 ...\],
...
By scanning all the keys in KSM DB, get all blocks of each key and see if they
are successfully deleted (comparing to the result)? Thats too heavy, isn't it?
In SCM and DN layer, we have the *TRANSACTION* concept, we formatted deleted
blocks info into a tx log and make the deletion API TX based. We could do
similar in KSM - SCM layer but that might introduce over-complexity. Passing
the key is similar to the idea of a TX, think the key's name is the txid. Once
we get key deletion result back, we simply locate this key in KSM DB and delete
the entry, that will be much more efficient. And talk about the generic
purpose, this API should work for generic purpose too. Just don't think you are
passing the object key, it could be anything else, a real TXID or any other
identity for a group of blocks. Please let me know if this makes sense.
Thanks
> Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
> -----------------------------------------------------------------------
>
> Key: HDFS-12235
> URL: https://issues.apache.org/jira/browse/HDFS-12235
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
> Affects Versions: HDFS-7240
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Attachments: HDFS-12235-HDFS-7240.001.patch,
> HDFS-12235-HDFS-7240.002.patch, HDFS-12235-HDFS-7240.003.patch,
> HDFS-12235-HDFS-7240.004.patch, HDFS-12235-HDFS-7240.005.patch,
> HDFS-12235-HDFS-7240.006.patch, HDFS-12235-HDFS-7240.007.patch
>
>
> KSM and SCM interaction for delete key operation, both KSM and SCM stores key
> state info in a backlog, KSM needs to scan this log and send block-deletion
> command to SCM, once SCM is fully aware of the message, KSM removes the key
> completely from namespace. See more from the design doc under HDFS-11922,
> this is task break down 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
