[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[ https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455175#comment-16455175 ] Hudson commented on HDFS-12195: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-12195. Ozone: DeleteKey-1: KSM replies delete key request (omalley: rev bc413a69cc6eb86eb24a2937723d1d919ef0a4b1) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/MetadataManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeyManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/ksm/TestKeySpaceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeySpaceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/MetadataManagerImpl.java > Ozone: DeleteKey-1: KSM replies delete key request asynchronously > - > > Key: HDFS-12195 > URL: https://issues.apache.org/jira/browse/HDFS-12195 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Yuanbo Liu >Priority: Major > Attachments: HDFS-12195-HDFS-7240.001.patch, > HDFS-12195-HDFS-7240.002.patch, HDFS-12195-HDFS-7240.003.patch, > HDFS-12195-HDFS-7240.004.patch, client-ksm.png > > > We will implement delete key in ozone in multiple child tasks, this is 1 of > the child task to implement client to scm communication. We need to do it in > async manner, once key state is changed in ksm metadata, ksm is ready to > reply client with a successful message. Actual deletes on other layers will > happen some time later. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[ https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450924#comment-16450924 ] Hudson commented on HDFS-12195: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-12195. Ozone: DeleteKey-1: KSM replies delete key request (aengineer: rev 1f5353d7b64b02f597013f9ce5f002d0b5881c95) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeySpaceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/MetadataManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/ksm/TestKeySpaceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeyManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/MetadataManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java > Ozone: DeleteKey-1: KSM replies delete key request asynchronously > - > > Key: HDFS-12195 > URL: https://issues.apache.org/jira/browse/HDFS-12195 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Yuanbo Liu >Priority: Major > Attachments: HDFS-12195-HDFS-7240.001.patch, > HDFS-12195-HDFS-7240.002.patch, HDFS-12195-HDFS-7240.003.patch, > HDFS-12195-HDFS-7240.004.patch, client-ksm.png > > > We will implement delete key in ozone in multiple child tasks, this is 1 of > the child task to implement client to scm communication. We need to do it in > async manner, once key state is changed in ksm metadata, ksm is ready to > reply client with a successful message. Actual deletes on other layers will > happen some time later. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[ https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107586#comment-16107586 ] Anu Engineer commented on HDFS-12195: - +1, I will commit this shortly. Thank you for taking care of this issue. > Ozone: DeleteKey-1: KSM replies delete key request asynchronously > - > > Key: HDFS-12195 > URL: https://issues.apache.org/jira/browse/HDFS-12195 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Yuanbo Liu > Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch, > HDFS-12195-HDFS-7240.002.patch, HDFS-12195-HDFS-7240.003.patch, > HDFS-12195-HDFS-7240.004.patch > > > We will implement delete key in ozone in multiple child tasks, this is 1 of > the child task to implement client to scm communication. We need to do it in > async manner, once key state is changed in ksm metadata, ksm is ready to > reply client with a successful message. Actual deletes on other layers will > happen some time later. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107153#comment-16107153
]
Hadoop QA commented on HDFS-12195:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12195 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879592/HDFS-12195-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8ea61e4d0eab 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision |
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106924#comment-16106924
]
Weiwei Yang commented on HDFS-12195:
Hi [~yuanbo]
Looks good. Some nits
1. Can we use all small letters prefix "#deleting#"?
2. the java doc for getDeletedKeyName() is still a bit confusing, please revise
it to
{code}
// Returns the DB key name of a deleted key in KSM metadata store.
// The name for a deleted key has prefix #deleting# followed by the actual key
name.
byte[] getDeletedKeyName(byte[] keyName);
{code}
3. Following method seems only be used in test case, please add
{{@VisibleForTesting}} tag
{code}
MetadataStore getStore();
{code}
others seems good.
Thanks.
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch,
> HDFS-12195-HDFS-7240.002.patch, HDFS-12195-HDFS-7240.003.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106896#comment-16106896
]
Hadoop QA commented on HDFS-12195:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12195 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879562/HDFS-12195-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 60cae575c0f9 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 6fb800f |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20492/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106787#comment-16106787
]
Yuanbo Liu commented on HDFS-12195:
---
[~anu]/[~vagarychen]/[~nandakumar131]/[~cheersyang]
Thanks for your review.
{quote}
please rename FAILED_KEY_NOT_FOUND ==> KEY_NOT_FOUND
{quote}
It's renamed in
{{KeySpaceManagerProtocolServerSideTranslatorPB#exceptionToResponseStatus}}, so
I didn't change it in the latest patch.
I've exposed metadata store in metadata manager instead of creating
listKeys(keyPrefix, maxKeys).
Other comments are addressed, upload v3 patch.
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch,
> HDFS-12195-HDFS-7240.002.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105303#comment-16105303
]
Weiwei Yang commented on HDFS-12195:
Hi [~yuanbo]
Some more comments
*OzoneConsts.java*
Can we change line 97 to:
{code}
public static final String DELETING_KEY_PREFIX = "#deleting#"
{code}
it seems not necessary to call "#" as a prefix.
*MetadataManager#getDeletedKeyName*
line 109 to 115: I don't think the java doc is correct. This method is used to
simply return a DB key for a key which is under deletion.
And like [~anu], [~vagarychen] both commented, please remove {{listKeys}} from
KSM API.
Thank you.
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch,
> HDFS-12195-HDFS-7240.002.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103962#comment-16103962
]
Chen Liang commented on HDFS-12195:
---
Appears to me that the idea of having {{KeySpaceManager#listKeys(keyPrefix,
maxKeys)}} is that, we need a way to list all keys that start with a special
prefix i.e. #deleting# for ALL volumes and buckets. While current listKeys are
listing keys within a particular volume+bucket prefix, and requires to specify
a volume and bucket. It also appears to me that the only reason why this is in
KeySpaceManager is to help unit test. I think an alternative way to do unit
test can be directly calling the metaStore's new {{listKeys(String keyPrefix,
int maxKeys)}}, then we don't need this call from KeySpaceManager.
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch,
> HDFS-12195-HDFS-7240.002.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103783#comment-16103783
]
Nandakumar commented on HDFS-12195:
---
Thanks for working on this [~yuanbo].
Apart for the nits that Anu has mentioned, is there any reason for having
{{KeySpaceManager#listKeys(keyPrefix, maxKeys)}}? I don't see any need for
having this method in KeySpaceManager.
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch,
> HDFS-12195-HDFS-7240.002.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103426#comment-16103426
]
Anu Engineer commented on HDFS-12195:
-
[~yuanbo] Thank you for updating the patch. It looks quite good, I have 2 nits,
I think we can commit this after those are addressed.
*KeyManager.java*
{code}
List listKeys(String keyPrefix, int maxKeys)
throws IOException;
{code}
* Key Manager contains API's needed by the client. While I agree that this is
needed by you, I don't think this should be part of KeyManager.
* please rename FAILED_KEY_NOT_FOUND ==> KEY_NOT_FOUND
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch,
> HDFS-12195-HDFS-7240.002.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103185#comment-16103185
]
Hadoop QA commented on HDFS-12195:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
1 unchanged - 0 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
59s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 9
unchanged - 0 fixed = 10 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
32s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12195 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879155/HDFS-12195-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9c5fd323c576 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 5b3cac7 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103060#comment-16103060
]
Yuanbo Liu commented on HDFS-12195:
---
[~anu]/[~cheersyang] Thanks for your review.
{quote}
OzoneConsts.java
{quote}
addressed.
{quote}
MetadataManagerImpl.java
{quote}
The keyName here is not a complete key name like "/volume/bucket/key", this
value comes from the client. that's why I sent volume and bucket name to
construct a complete key name. I use {{getDeletedKeyName(bye[] keyName)}} to
replace the old one, what's your thoughts?
{quote}
KeySpaceManagerImpl.java
{quote}
As [~cheersyang] suggested, we can use {{writeBatch}} instead because this
method is an atomic operation. Test case has been added in v2 patch.
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102720#comment-16102720
]
Anu Engineer commented on HDFS-12195:
-
*OzoneConsts.java*
{code}
public static final String KSM_DELETING_KEY_PREFIX =
KSM_USER_PREFIX + "DELETING" + KSM_USER_PREFIX + KSM_VOLUME_PREFIX;
{code}
I thought [~cheersyang] wanted to use #deleting# as the prefix and not
$deleting$. My apologies, if some of my older emails or docs created this
confusion.
*MetadataManagerImpl.java*
getDeletingKey() -- Why are we sending volume name, bucket name and key name,
isn't the key name complete with just keyName. Also shouldn't this function
just call into getDBKeyForKey() and add those bytes to the Deleting_Prefix?
Also, should we rename this function to getDeletedKeyName ?
*KeySpaceManagerImpl.java*
* I see that we have removed the lookup function for a key. So what happens if
I attempt to delete a non-existent key? I don't think your code is introducing
any new issue, but just wondering how that is handled today. In fact, I think
we should have KEY_NOT_FOUND as a different error from FAILED_KEY_DELETION.
* I know that you are doing this under a writelock(), but I think this should
be under a transaction.
{code}
metadataManager.delete(objectKey);
metadataManager.put(deletingKey, objectValue);
{code}
* Here is a case where having a write lock is not good enough.
## Say the delete(objectKey), call works, but the put fails due to some
database error.
## In that case we have some ghost blocks since if the client tried to delete
the key again we will return error, But we have not cleaned up the blocks. In
other words, the delete and put must either work or not.
* Just out of paranoia, I would flip the order of deletes and put. That is I
would put and then update the actual delete. if we get a failure like the above
one, the pending delete will get deleted -- That is blocks would be gone, and
we have a damaged key -- which we can delete later.
* Can we have some test cases that confirm that key got moved to the new
location in the database ?
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101737#comment-16101737
]
Weiwei Yang commented on HDFS-12195:
Hi [~yuanbo]
I know this is still working in progress, but some early comments. We are using
ksm.db to store keys under deletion so that we can do "atomic" update, by
leveraging db batch operation. Please see more in {{MetadataStore#writeBatch}},
please use this API to do batch update of moving a key in ksm.db.
Thanks
> Ozone: DeleteKey-1: KSM replies delete key request asynchronously
> -
>
> Key: HDFS-12195
> URL: https://issues.apache.org/jira/browse/HDFS-12195
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Attachments: client-ksm.png, HDFS-12195-HDFS-7240.001.patch
>
>
> We will implement delete key in ozone in multiple child tasks, this is 1 of
> the child task to implement client to scm communication. We need to do it in
> async manner, once key state is changed in ksm metadata, ksm is ready to
> reply client with a successful message. Actual deletes on other layers will
> happen some time later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[
https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101182#comment-16101182
]
Hadoop QA commented on HDFS-12195:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 4 new +
0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.ozShell.TestOzoneShell |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12195 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878776/HDFS-12195-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9fbcd81ec06f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / c2d08e6 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
|
[jira] [Commented] (HDFS-12195) Ozone: DeleteKey-1: KSM replies delete key request asynchronously
[ https://issues.apache.org/jira/browse/HDFS-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099749#comment-16099749 ] Yuanbo Liu commented on HDFS-12195: --- working on it > Ozone: DeleteKey-1: KSM replies delete key request asynchronously > - > > Key: HDFS-12195 > URL: https://issues.apache.org/jira/browse/HDFS-12195 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Yuanbo Liu > Attachments: client-ksm.png > > > We will implement delete key in ozone in multiple child tasks, this is 1 of > the child task to implement client to scm communication. We need to do it in > async manner, once key state is changed in ksm metadata, ksm is ready to > reply client with a successful message. Actual deletes on other layers will > happen some time later. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
