[jira] [Commented] (HDDS-1834) parent directories not found in secure setup due to ACL check
[ https://issues.apache.org/jira/browse/HDDS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897527#comment-16897527 ] Anu Engineer commented on HDDS-1834: [~adoroszlai] Thanks for checking. I just pushed the patch into ozone-0.4.1. Thanks for catching the missing step and reminding us. > parent directories not found in secure setup due to ACL check > - > > Key: HDDS-1834 > URL: https://issues.apache.org/jira/browse/HDDS-1834 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Blocker > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > ozonesecure-ozonefs acceptance test is failing, because {{ozone fs -mkdir > -p}} only creates key for the specific directory, not its parents. > {noformat} > ozone fs -mkdir -p o3fs://bucket1.fstest/testdir/deep > {noformat} > Previous result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/176/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/ > testdir/deep/ > {noformat} > Current result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/177/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/deep/ > {noformat} > The failure happens on first operation that tries to use {{testdir/}} > directly: > {noformat} > $ ozone fs -touch o3fs://bucket1.fstest/testdir/TOUCHFILE.txt > ls: `o3fs://bucket1.fstest/testdir': No such file or directory > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1834) parent directories not found in secure setup due to ACL check
[ https://issues.apache.org/jira/browse/HDDS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896796#comment-16896796 ] Doroszlai, Attila commented on HDDS-1834: - Thanks [~xyao] for committing it. Can you please double-check [ozone-0.4.1|https://github.com/apache/hadoop/commits/ozone-0.4.1]? I don't see the commit there. > parent directories not found in secure setup due to ACL check > - > > Key: HDDS-1834 > URL: https://issues.apache.org/jira/browse/HDDS-1834 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Blocker > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > ozonesecure-ozonefs acceptance test is failing, because {{ozone fs -mkdir > -p}} only creates key for the specific directory, not its parents. > {noformat} > ozone fs -mkdir -p o3fs://bucket1.fstest/testdir/deep > {noformat} > Previous result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/176/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/ > testdir/deep/ > {noformat} > Current result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/177/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/deep/ > {noformat} > The failure happens on first operation that tries to use {{testdir/}} > directly: > {noformat} > $ ozone fs -touch o3fs://bucket1.fstest/testdir/TOUCHFILE.txt > ls: `o3fs://bucket1.fstest/testdir': No such file or directory > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1834) parent directories not found in secure setup due to ACL check
[ https://issues.apache.org/jira/browse/HDDS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896621#comment-16896621 ] Hudson commented on HDDS-1834: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17010 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17010/]) HDDS-1834. parent directories not found in secure setup due to ACL (xyao: rev e68d8446c42a883b9cd8a1fa47d870a47db37ad6) * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneObjInfo.java > parent directories not found in secure setup due to ACL check > - > > Key: HDDS-1834 > URL: https://issues.apache.org/jira/browse/HDDS-1834 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Blocker > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > ozonesecure-ozonefs acceptance test is failing, because {{ozone fs -mkdir > -p}} only creates key for the specific directory, not its parents. > {noformat} > ozone fs -mkdir -p o3fs://bucket1.fstest/testdir/deep > {noformat} > Previous result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/176/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/ > testdir/deep/ > {noformat} > Current result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/177/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/deep/ > {noformat} > The failure happens on first operation that tries to use {{testdir/}} > directly: > {noformat} > $ ozone fs -touch o3fs://bucket1.fstest/testdir/TOUCHFILE.txt > ls: `o3fs://bucket1.fstest/testdir': No such file or directory > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1834) parent directories not found in secure setup due to ACL check
[ https://issues.apache.org/jira/browse/HDDS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893175#comment-16893175 ] Xiaoyu Yao commented on HDDS-1834: -- Further look into the changes from HDDS-1300/HDDS-1185 on KeyManagerImpl#getFileStatus where we return a faked parent status if there are children under it even though the parent key does not exist. {code} List keys = metadataManager.listKeys(volumeName, bucketName, null, dirKey, 1); if (keys.iterator().hasNext()) { return new OzoneFileStatus(keyName); } {code} > parent directories not found in secure setup due to ACL check > - > > Key: HDDS-1834 > URL: https://issues.apache.org/jira/browse/HDDS-1834 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Blocker > > ozonesecure-ozonefs acceptance test is failing, because {{ozone fs -mkdir > -p}} only creates key for the specific directory, not its parents. > {noformat} > ozone fs -mkdir -p o3fs://bucket1.fstest/testdir/deep > {noformat} > Previous result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/176/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/ > testdir/deep/ > {noformat} > Current result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/177/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/deep/ > {noformat} > The failure happens on first operation that tries to use {{testdir/}} > directly: > {noformat} > $ ozone fs -touch o3fs://bucket1.fstest/testdir/TOUCHFILE.txt > ls: `o3fs://bucket1.fstest/testdir': No such file or directory > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1834) parent directories not found in secure setup due to ACL check
[ https://issues.apache.org/jira/browse/HDDS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892940#comment-16892940 ] Xiaoyu Yao commented on HDDS-1834: -- This is related to BasicOzoneFileSystem change in HDDS-1481 as [~ljain] has mentioned earlier. This specific test uses ozonesecure docker compose with basic ozonefs which will be affected by this change, i.e., parent dir not being created like OzoneFileSystem. The native ACL check can be enabled by setting ozone.acl.authorizer.class=org.apache.hadoop.ozone.security.acl.OzoneNativeAuthorizer. Otherwise, the default authorizer will allow all to make it easier for the dev-test environment. > parent directories not found in secure setup due to ACL check > - > > Key: HDDS-1834 > URL: https://issues.apache.org/jira/browse/HDDS-1834 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Blocker > > ozonesecure-ozonefs acceptance test is failing, because {{ozone fs -mkdir > -p}} only creates key for the specific directory, not its parents. > {noformat} > ozone fs -mkdir -p o3fs://bucket1.fstest/testdir/deep > {noformat} > Previous result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/176/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/ > testdir/deep/ > {noformat} > Current result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/177/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/deep/ > {noformat} > The failure happens on first operation that tries to use {{testdir/}} > directly: > {noformat} > $ ozone fs -touch o3fs://bucket1.fstest/testdir/TOUCHFILE.txt > ls: `o3fs://bucket1.fstest/testdir': No such file or directory > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1834) parent directories not found in secure setup due to ACL check
[ https://issues.apache.org/jira/browse/HDDS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892790#comment-16892790 ] Lokesh Jain commented on HDDS-1834: --- The problem exists in general for checkAccess. There are two bugs associated with checkAccess. # In OzoneFileSystem use cases, for access of a descendant checkAccess of any ancestor is not done. Currently while accessing a/b/c.txt we do not check the access for a/ and a/b/ and do a access check only for the path a/b/c.txt # In HDDS-1481 while doing mkdir, the ancestor directories are not created if they do not exist. checkAccess method only checks for the key provided and therefore fails with KEY_NOT_FOUND error. It should do a check for existence of a directory using getFileStatus. KeyManagerImpl#checkAccess:1645-1657 {code:java} OmKeyInfo keyInfo = metadataManager.getKeyTable().get(objectKey); if (keyInfo == null) { objectKey = OzoneFSUtils.addTrailingSlashIfNeeded(objectKey); keyInfo = metadataManager.getKeyTable().get(objectKey); if(keyInfo == null) { keyInfo = metadataManager.getOpenKeyTable().get(objectKey); if (keyInfo == null) { throw new OMException("Key not found, checkAccess failed. Key:" + objectKey, KEY_NOT_FOUND); } } } {code} Example illustrating the problem 2. {code:java} ozone sh key list o3://om/fstest/bucket1/ [ { "version" : 0, "md5hash" : null, "createdOn" : "Thu, 25 Jul 2019 11:26:02 GMT", "modifiedOn" : "Thu, 25 Jul 2019 11:26:02 GMT", "size" : 0, "keyName" : "testdir/deep/", "type" : null }, { "version" : 0, "md5hash" : null, "createdOn" : "Thu, 25 Jul 2019 11:26:09 GMT", "modifiedOn" : "Thu, 01 Jan 1970 00:12:54 GMT", "size" : 22808, "keyName" : "testdir/deep/MOVED.TXT", "type" : null }, { "version" : 0, "md5hash" : null, "createdOn" : "Thu, 25 Jul 2019 11:26:18 GMT", "modifiedOn" : "Thu, 01 Jan 1970 00:12:44 GMT", "size" : 22808, "keyName" : "testdir/deep/PUTFILE.txt", "type" : null } ] ozone sh key info o3://om/fstest/bucket1/testdir KEY_NOT_FOUND Key not found, checkAccess failed. Key:/fstest/bucket1/testdir/ {code} > parent directories not found in secure setup due to ACL check > - > > Key: HDDS-1834 > URL: https://issues.apache.org/jira/browse/HDDS-1834 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Blocker > > ozonesecure-ozonefs acceptance test is failing, because {{ozone fs -mkdir > -p}} only creates key for the specific directory, not its parents. > {noformat} > ozone fs -mkdir -p o3fs://bucket1.fstest/testdir/deep > {noformat} > Previous result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/176/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/ > testdir/deep/ > {noformat} > Current result: > {noformat:title=https://ci.anzix.net/job/ozone-nightly/177/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s16-t3-k2} > $ ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r > '.[].keyName' > testdir/deep/ > {noformat} > The failure happens on first operation that tries to use {{testdir/}} > directly: > {noformat} > $ ozone fs -touch o3fs://bucket1.fstest/testdir/TOUCHFILE.txt > ls: `o3fs://bucket1.fstest/testdir': No such file or directory > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org