[
https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547651#comment-16547651
]
Shashikant Banerjee edited comment on HDDS-181 at 7/18/18 10:19 AM:
--------------------------------------------------------------------
Thanks [~msingh] for the review.
{quote}At this time, let us also raise jira to track keys which are never
committed by client because the client went away. Such keys should be garbage
collected.
{quote}
Right now, on a Datanode all the open keys in a container will be tracked by
openContainerBlockMap. Even if the client goes away, the pending keys will
still be there in the openContainerBlockMap and will be committed once the
container gets closed. So essenially, this map will have those keys for which
the client never issue a commit operation till the container close arrives.
Let's discuss on this more in order to gain more clarity on this.
{quote}The current patch maintains an in-memory map of the open keys. This
information should be persisted as well to allow the keys to be committed after
a restart.
{quote}
I think we can reconstruct this map by basically reading the individual chunk
file and updating the sizes in the map for all the blocks for which the block
entry does not exists in the container Db on datanode. This info will be lost
only when the Datanode dies, which will trigger closing of open containers on
other datanodes and the replicating the container by Replica Manger. In such
cases , if the datanode comes back up, we need this info for garbage collection
and we might not need this info for committing the open blocks once again. I
think we can do this in a separate jira(for now, It has been mentioned as a
TODO item in the code). Let's discuss this more offline.
Rest all the review comments are addressed in patch v4.
was (Author: shashikant):
Thanks [~msingh] for the review.
{quote}At this time, let us also raise jira to track keys which are never
committed by client because the client went away. Such keys should be garbage
collected.
{quote}
Right now, on a Datanode all the open keys in a container will be tracked by
openContainerBlockMap. Even if the client goes away, the pending keys will
still be there in the openContainerBlockMap and will be committed once the
container gets closed. So essenially, this map will have those keys for which
the client never issue a commit operation till the container close arrives.
Let's discuss on this more in order to gain more clarity on this.
{quote}The current patch maintains an in-memory map of the open keys. This
information should be persisted as well to allow the keys to be committed after
a restart.
{quote}
I think we can reconstruct this map by basically reading the individual chunk
file and updating the sizes in the map for all the blocks for which the block
entry does not exists in the container Db on datanode. This info will be lost
only when the Datanode dies, which will trigger closing of open containers on
other datanodes and the replicating the container by Replica Manger. In such
cases , if the datanode comes back up, we need this info for garbage collection
and we might not need this info for committing the open blocks once again. I
think we can do this in a separate jira(for now, It has been mentioned as a
TODO item in the code).
Rest all the review comments are addressed in patch v4.
> CloseContainer should commit all pending open Keys on a datanode
> ----------------------------------------------------------------
>
> Key: HDDS-181
> URL: https://issues.apache.org/jira/browse/HDDS-181
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
> Reporter: Shashikant Banerjee
> Assignee: Shashikant Banerjee
> Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch,
> HDDS-181.04.patch
>
>
> A close container command arrives in the Datanode by the SCM heartBeat
> response.It will then be queued up over the ratis pipeline. Once the command
> execution starts inside the Datanode, it will mark the container in CLOSING
> State. All the pending open keys for the container now will be committed
> followed by the transition of the container state from CLOSING to CLOSED. For
> achieving this, all the open keys for a container need to be tracked.
> This Jira aims to address this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
