[jira] [Comment Edited] (HDDS-155) Implement KeyValueContainer and adopt new disk layout for the containers

2018-06-14 Thread Bharat Viswanadham (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDDS-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513300#comment-16513300
 ] 

Bharat Viswanadham edited comment on HDDS-155 at 6/15/18 3:57 AM:
--

I will commit this shortly.

shaded-client issue is not caused by this patch.


was (Author: bharatviswa):
I will commit this shortly.

shaded-client issue is not caused by this patch.

> Implement KeyValueContainer and adopt new disk layout for the containers
> 
>
> Key: HDDS-155
> URL: https://issues.apache.org/jira/browse/HDDS-155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-155-HDDS-48.00.patch, HDDS-155-HDDS-48.01.patch, 
> HDDS-155-HDDS-48.02.patch, HDDS-155-HDDS-48.03.patch, 
> HDDS-155-HDDS-48.04.patch, HDDS-155-HDDS-48.05.patch, 
> HDDS-155-HDDS-48.06.patch, HDDS-155-HDDS-48.07.patch, 
> HDDS-155-HDDS-48.08.patch
>
>
> This Jira is to add following:
>  # Implement Container Interface
>  # Use new directory layout proposed in the design document.
>  a. Data location (chunks)
>  b. Meta location (DB and .container files)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDDS-155) Implement KeyValueContainer and adopt new disk layout for the containers

2018-06-14 Thread Bharat Viswanadham (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDDS-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513133#comment-16513133
 ] 

Bharat Viswanadham edited comment on HDDS-155 at 6/14/18 11:32 PM:
---

Hi [~hanishakoneru]

Thanks for the review.

Addressed review comments and Jenkins reported issues in patch v06.


was (Author: bharatviswa):
Hi [~hanishakoneru]

Thanks for the review.

Addressed review comments in patch v06.

> Implement KeyValueContainer and adopt new disk layout for the containers
> 
>
> Key: HDDS-155
> URL: https://issues.apache.org/jira/browse/HDDS-155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-155-HDDS-48.00.patch, HDDS-155-HDDS-48.01.patch, 
> HDDS-155-HDDS-48.02.patch, HDDS-155-HDDS-48.03.patch, 
> HDDS-155-HDDS-48.04.patch, HDDS-155-HDDS-48.05.patch, 
> HDDS-155-HDDS-48.06.patch
>
>
> This Jira is to add following:
>  # Implement Container Interface
>  # Use new directory layout proposed in the design document.
>  a. Data location (chunks)
>  b. Meta location (DB and .container files)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDDS-155) Implement KeyValueContainer and adopt new disk layout for the containers

2018-06-14 Thread Hanisha Koneru (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDDS-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513041#comment-16513041
 ] 

Hanisha Koneru edited comment on HDDS-155 at 6/14/18 9:59 PM:
--

Thanks for the update [~bharatviswa].

Some comments (Many of them are just NITs.):
 # Can we rename the following in {{DatanodeContainerProtocol.proto}}
 CONTAINER_REQUIRED_FILES_CREATE_ERROR -> CONTAINER_FILES_CREATE_ERROR
 CONTAINER_CHECKSUM_FILE_CALCULATE_ERROR -> CONTAINER_CHECKSUM_ERROR
 # {{KeyValueContainer}} Line 131, 133 : Instead of 
"String.valueOf(containerId)", we can reuse containerName
 # {{KeyValueContainer#updateRequiredFiles()}} -> when update succeeds, the 
backup files must be deleted.
 # In all scenarios where the createContainer fails, we should delete the 
containerBase dir.
 # Can we rename {{createRequiredFiles}} and {{updateRequiredFiles}} to 
createContainerFile and updateContainerFile? Required doesn’t tell us instantly 
what the files are. I think it is ok not to mention that the checksum file is 
also created as part of createContainerFile.
 # Line 225 : "Unable to delete container checksum backup file” -> Since we are 
deleting the temporary file, we can have "Unable to delete container temporary 
checksum file”.
 # {{KeyValueContainer#updateRequiredFiles()}}, when throwing 
INVALID_CONTAINER_STATE or when the restore fails, we should set the state of 
the container to INVALID.
 # When restoring from back up files, can we add a info log message saying 
"update failed, so restoring the container files."
 # Javadoc of computeChecksum is misleading. "Create checksum file of the 
.container file.” -> “Computer checksum of the .container file”.
 # Can we have a file handle to {{.db file}} in {{KeyValueContainerData}} as 
this would be used by every key operation.
 # {{KeyValueContainerUtil#verifyIsNewContainer()}} -> existence of 
containerBasePath should be sufficient condition to fail createContainer.


was (Author: hanishakoneru):
Thanks for the update [~bharatviswa].

Some comments (Many of them are just NITs.):
 # Can we rename the following in {{DatanodeContainerProtocol.proto}}
 CONTAINER_REQUIRED_FILES_CREATE_ERROR -> CONTAINER_FILES_CREATE_ERROR
 CONTAINER_CHECKSUM_FILE_CALCULATE_ERROR -> CONTAINER_CHECKSUM_ERROR
 # {{KeyValueContainer}} Line 131, 133 : Instead of 
"String.valueOf(containerId)", we can reuse containerName
 # {{KeyValueContainer#updateRequiredFiles()}} -> when update succeeds, the 
backup files must be deleted. In all scenarios where the createContainer fails, 
we should delete the containerBase dir.
 # Can we rename {{createRequiredFiles}} and {{updateRequiredFiles}} to 
createContainerFile and updateContainerFile? Required doesn’t tell us instantly 
what the files are. I think it is ok not to mention that the checksum file is 
also created as part of createContainerFile.
 # Line 225 : "Unable to delete container checksum backup file” -> Since we are 
deleting the temporary file, we can have "Unable to delete container temporary 
checksum file”.
 # {{KeyValueContainer#updateRequiredFiles()}}, when throwing 
INVALID_CONTAINER_STATE or when the restore fails, we should set the state of 
the container to INVALID.
 # When restoring from back up files, can we add a info log message saying 
"update failed, so restoring the container files."
 # Javadoc of computeChecksum is misleading. "Create checksum file of the 
.container file.” -> “Computer checksum of the .container file”.
 # Can we have a file handle to {{.db file}} in {{KeyValueContainerData}} as 
this would be used by every key operation.
 # {{KeyValueContainerUtil#verifyIsNewContainer()}} -> existence of 
containerBasePath should be sufficient condition to fail createContainer.

> Implement KeyValueContainer and adopt new disk layout for the containers
> 
>
> Key: HDDS-155
> URL: https://issues.apache.org/jira/browse/HDDS-155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-155-HDDS-48.00.patch, HDDS-155-HDDS-48.01.patch, 
> HDDS-155-HDDS-48.02.patch, HDDS-155-HDDS-48.03.patch, 
> HDDS-155-HDDS-48.04.patch, HDDS-155-HDDS-48.05.patch
>
>
> This Jira is to add following:
>  # Implement Container Interface
>  # Use new directory layout proposed in the design document.
>  a. Data location (chunks)
>  b. Meta location (DB and .container files)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDDS-155) Implement KeyValueContainer and adopt new disk layout for the containers

2018-06-14 Thread Hanisha Koneru (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDDS-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513041#comment-16513041
 ] 

Hanisha Koneru edited comment on HDDS-155 at 6/14/18 9:59 PM:
--

Thanks for the update [~bharatviswa].

Some comments (Many of them are just NITs.):
 # Can we rename the following in {{DatanodeContainerProtocol.proto}}
 CONTAINER_REQUIRED_FILES_CREATE_ERROR -> CONTAINER_FILES_CREATE_ERROR
 CONTAINER_CHECKSUM_FILE_CALCULATE_ERROR -> CONTAINER_CHECKSUM_ERROR
 # {{KeyValueContainer}} Line 131, 133 : Instead of 
"String.valueOf(containerId)", we can reuse containerName
 # {{KeyValueContainer#updateRequiredFiles()}} -> when update succeeds, the 
backup files must be deleted. In all scenarios where the createContainer fails, 
we should delete the containerBase dir.
 # Can we rename {{createRequiredFiles}} and {{updateRequiredFiles}} to 
createContainerFile and updateContainerFile? Required doesn’t tell us instantly 
what the files are. I think it is ok not to mention that the checksum file is 
also created as part of createContainerFile.
 # Line 225 : "Unable to delete container checksum backup file” -> Since we are 
deleting the temporary file, we can have "Unable to delete container temporary 
checksum file”.
 # {{KeyValueContainer#updateRequiredFiles()}}, when throwing 
INVALID_CONTAINER_STATE or when the restore fails, we should set the state of 
the container to INVALID.
 # When restoring from back up files, can we add a info log message saying 
"update failed, so restoring the container files."
 # Javadoc of computeChecksum is misleading. "Create checksum file of the 
.container file.” -> “Computer checksum of the .container file”.
 # Can we have a file handle to {{.db file}} in {{KeyValueContainerData}} as 
this would be used by every key operation.
 # {{KeyValueContainerUtil#verifyIsNewContainer()}} -> existence of 
containerBasePath should be sufficient condition to fail createContainer.


was (Author: hanishakoneru):
Thanks for the update [~bharatviswa].

Some comments (Many of them are just NITs.):
 # Can we rename the following in {{DatanodeContainerProtocol.proto}}
 CONTAINER_REQUIRED_FILES_CREATE_ERROR -> CONTAINER_FILES_CREATE_ERROR
 CONTAINER_CHECKSUM_FILE_CALCULATE_ERROR -> CONTAINER_CHECKSUM_ERROR
 # {{KeyValueContainer}} Line 131, 133 : Instead of 
"String.valueOf(containerId)", we can reuse containerName
 # {{KeyValueContainer#createRequiredFiles()}} -> there is a possibility that 
the rename of .containerFile succeeds but .checksum file fails. In this 
scenario and all other scenarios where the createContainer fails, we should 
delete the containerBase dir.
 # Can we rename {{createRequiredFiles}} and {{updateRequiredFiles}} to 
createContainerFile and updateContainerFile? Required doesn’t tell us instantly 
what the files are. I think it is ok not to mention that the checksum file is 
also created as part of createContainerFile.
 # Line 225 : "Unable to delete container checksum backup file” -> Since we are 
deleting the temporary file, we can have "Unable to delete container temporary 
checksum file”.
 # {{KeyValueContainer#updateRequiredFiles()}}, when throwing 
INVALID_CONTAINER_STATE or when the restore fails, we should set the state of 
the container to INVALID.
 # When restoring from back up files, can we add a info log message saying 
"update failed, so restoring the container files."
 # Javadoc of computeChecksum is misleading. "Create checksum file of the 
.container file.” -> “Computer checksum of the .container file”.
 # Can we have a file handle to {{.db file}} in {{KeyValueContainerData}} as 
this would be used by every key operation.
 # {{KeyValueContainerUtil#verifyIsNewContainer()}} -> existence of 
containerBasePath should be sufficient condition to fail createContainer.

> Implement KeyValueContainer and adopt new disk layout for the containers
> 
>
> Key: HDDS-155
> URL: https://issues.apache.org/jira/browse/HDDS-155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-155-HDDS-48.00.patch, HDDS-155-HDDS-48.01.patch, 
> HDDS-155-HDDS-48.02.patch, HDDS-155-HDDS-48.03.patch, 
> HDDS-155-HDDS-48.04.patch, HDDS-155-HDDS-48.05.patch
>
>
> This Jira is to add following:
>  # Implement Container Interface
>  # Use new directory layout proposed in the design document.
>  a. Data location (chunks)
>  b. Meta location (DB and .container files)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands,

[jira] [Comment Edited] (HDDS-155) Implement KeyValueContainer and adopt new disk layout for the containers

2018-06-14 Thread Bharat Viswanadham (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDDS-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512841#comment-16512841
 ] 

Bharat Viswanadham edited comment on HDDS-155 at 6/14/18 6:16 PM:
--

Rebased patch to use new code changes done in HDDS-156

Will submit the patch, once HDDS-156 gets committed.


was (Author: bharatviswa):
Rebased patch to use new code changes done in HDDS-155.

Will submit the patch, once HDDS-155 gets committed.

> Implement KeyValueContainer and adopt new disk layout for the containers
> 
>
> Key: HDDS-155
> URL: https://issues.apache.org/jira/browse/HDDS-155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-155-HDDS-48.00.patch, HDDS-155-HDDS-48.01.patch, 
> HDDS-155-HDDS-48.02.patch, HDDS-155-HDDS-48.03.patch, 
> HDDS-155-HDDS-48.04.patch, HDDS-155-HDDS-48.05.patch
>
>
> This Jira is to add following:
>  # Implement Container Interface
>  # Use new directory layout proposed in the design document.
>  a. Data location (chunks)
>  b. Meta location (DB and .container files)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org