[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713581#comment-16713581 ] Hudson commented on HDDS-909: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15577 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15577/]) HDDS-909 : Default implementation for Ozone acls. Contributed by Ajay (aengineer: rev 5e773efd7839c4807f98f00d286b9325c1fe81df) * (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneAclException.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneObj.java * (add) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneAccessAuthorizer.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManager.java * (add) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmAcls.java * (add) hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/security/acl/TestOzoneObjInfo.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/RequestContext.java * (add) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneObjInfo.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-909.00.patch, HDDS-909.01.patch, HDDS-909.02.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-909: -- Resolution: Fixed Fix Version/s: 0.4.0 Status: Resolved (was: Patch Available) > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-909.00.patch, HDDS-909.01.patch, HDDS-909.02.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713575#comment-16713575 ] Anu Engineer commented on HDDS-909: --- +1, I have committed this to trunk. [~ajayydv] Thanks for the contribution, [~jnp] Thanks for the reviews. > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-909.00.patch, HDDS-909.01.patch, HDDS-909.02.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13835) RBF: Unable to add files after changing the order
[
https://issues.apache.org/jira/browse/HDFS-13835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713515#comment-16713515
]
Ayush Saxena commented on HDFS-13835:
-
Thanx [~ramkumar] and [~elgoiri] for the discussion.
I guess
bq. ./hdfs dfsrouteradmin -update /apps ns1,ns2 /tmp1,/tmp2 -order RANDOM (And
if i try to add any files to the mount entry /apps its giving errors)
This statement seems to be the culprit to me.
Firstly before update the scenerio was somewhat like:
/apps pointed to ns1-->/tmp1 and ns2-->/tmp2 (for which the actual
destinations exists)
After the update
/apps pointed to ns1-->/tmp1,/tmp2 and ns2-->/tmp1,/tmp2 (these destination
doesn't exist)
The intention was to keep the dest pointing as before only but if we see in the
update command::
While parsing the arguments we split the namespace only with COMMA not the
destinations.
{code:java}
String mount = parameters[i++];
String[] nss = parameters[i++].split(",");
String dest = parameters[i++];
{code}
And we consider the destination as one.while putting it in the destMap.The
following logic in the code might give more clarity.
{code:java}
Map destMap = new LinkedHashMap<>();
for (String ns : nss) {
destMap.put(ns, dest);
}
{code}
The command is made in way such that if you give ns1,ns2 and dest as /tmp it
will mount ns1-->/tmp and ns2-->/tmp.This seems quite obvious too if we provide
more number of destinations than the namespace.how we will actually decide
which one destinations goes to one NS.
So I guess ORDER change has nothing as such related which could stop creating a
file if the destination exist.It deals just with the logic where to direct the
request.Here the error message received also gives the HINT
bq. No such file or directory:
Pls correct if I misunderstood something in the context :)
> RBF: Unable to add files after changing the order
> -
>
> Key: HDFS-13835
> URL: https://issues.apache.org/jira/browse/HDFS-13835
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: venkata ram kumar ch
>Assignee: venkata ram kumar ch
>Priority: Critical
>
> When a mount point it pointing to multiple sub cluster by default the order
> is HASH.
> But After changing the order from HASH to RANDOM i am unable to add files to
> that mount point.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-871) XceiverClientRatis#watchForCommit closes raft client at every call
[ https://issues.apache.org/jira/browse/HDDS-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713495#comment-16713495 ] Shashikant Banerjee commented on HDDS-871: -- This will be fixed as a part of HDDS-870. > XceiverClientRatis#watchForCommit closes raft client at every call > -- > > Key: HDDS-871 > URL: https://issues.apache.org/jira/browse/HDDS-871 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > > Currently XceiverClientRatis#watchForCommit closes raft client at every call. > We should cache the raft client and reuse it for future watchForCommit calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713496#comment-16713496 ] Hudson commented on HDFS-14001: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15576 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15576/]) HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the (inigoiri: rev 8fc0d04517912766a3102f3e611f7d0fabd2f815) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java * (delete) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/aliasmap/TestInMemoryAliasMap.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapAliasmap.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/blockaliasmap/impl/TestInMemoryLevelDBAliasMapClient.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/aliasmap/ITestInMemoryAliasMap.java * (edit) hadoop-tools/hadoop-fs2img/src/test/java/org/apache/hadoop/hdfs/server/namenode/ITestProvidedImplementation.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/aliasmap/InMemoryAliasMap.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/aliasmap/InMemoryLevelDBAliasMapServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch, HDFS-14001.005.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713483#comment-16713483 ] Íñigo Goiri edited comment on HDFS-14001 at 12/8/18 2:29 AM: - [^HDFS-14001.005.patch] LGTM. I tested this on Azure and the alias map is sent over succesfully. The failed unit tests is a usual suspect and the checkstyle is following consistency with the rest of the code. +1 Committing. was (Author: elgoiri): [^HDFS-14001.005.patch] LGTM. The failed unit tests is a usual suspect and the checkstyle is following consistency with the rest of the code. +1 Committing. > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch, HDFS-14001.005.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713483#comment-16713483 ] Íñigo Goiri commented on HDFS-14001: [^HDFS-14001.005.patch] LGTM. The failed unit tests is a usual suspect and the checkstyle is following consistency with the rest of the code. +1 Committing. > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch, HDFS-14001.005.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14001: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) Thanks [~virajith] for the feature! Committed to trunk. > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch, HDFS-14001.005.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713482#comment-16713482
]
Hadoop QA commented on HDDS-909:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} root: The patch generated 3 new + 0 unchanged -
0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 16s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
29s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdds.scm.pipeline.TestPipelineClose |
| | hadoop.ozone.om.TestOzoneManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-909 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951075/HDDS-909.02.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 316af4f4f9a6 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 154449f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDDS-Build/1900/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1900/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1900/testReport/ |
| Max. process+thread count | 1188 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/common hadoop-ozone/common
hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1900/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Ajay Kumar
>
[jira] [Updated] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-909: Attachment: HDDS-909.02.patch > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch, HDDS-909.01.patch, HDDS-909.02.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713466#comment-16713466
]
Ajay Kumar commented on HDDS-909:
-
[~anu] thanks for comments.
{quote}Not sure I understand why we are returning OZONE_URI_DELIMITER in the
default case.{quote}
Added default option to satisfy style check. Resource type shouldn't be
something else. New patch throws an Exception for default case. Let me know if
you think we should do something different.
{quote}In TestOzoneObjInfo
.setResType(RESOURCE_TYPE)
SetResourceType(resource) is not very easy to understand, did you intended to
write the setResType(Volume_Type){quote}
Removed the field, directly using enum now.
{quote}in TestOzoneObjInfo#testGetBucketName - line 69, 70 looks same as 72,73.
Just want to make sure it is intentional.{quote}
Thanks for catching it, removed.
Fixed the checkstyle issued for length.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDDS-909.00.patch, HDDS-909.01.patch, HDDS-909.02.patch
>
>
> Default implementation for Ozone acls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[
https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713464#comment-16713464
]
Hadoop QA commented on HDFS-14001:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 2s{color} | {color:orange} root: The patch generated 1 new + 427 unchanged
- 0 fixed = 428 total (was 427) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14001 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951051/HDFS-14001.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 849d15c36350 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 154449f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[ https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713455#comment-16713455 ] Tsz Wo Nicholas Sze commented on HDDS-870: -- Filed RATIS-453 to fix the retry behavior in Ratis. > Avoid creating block sized buffer in ChunkGroupOutputStream > --- > > Key: HDDS-870 > URL: https://issues.apache.org/jira/browse/HDDS-870 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-870.000.patch, HDDS-870.001.patch, > HDDS-870.002.patch, HDDS-870.003.patch, HDDS-870.004.patch, > HDDS-870.005.patch, HDDS-870.006.patch, HDDS-870.007.patch, > HDDS-870.008.patch, HDDS-870.009.patch > > > Currently, for a key, we create a block size byteBuffer in order for caching > data. This can be replaced with an array of buffers of size flush buffer size > configured for handling 2 node failures as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713440#comment-16713440
]
Lukas Majercak commented on HDFS-14134:
---
The unit tests are expected to fail, I can fix them once we agree on how the
retry policy should behave.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713433#comment-16713433
]
Hadoop QA commented on HDDS-909:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} root: The patch generated 4 new + 0 unchanged -
0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 6s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
54s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.om.TestOzoneManager |
| | hadoop.ozone.om.TestOmMetrics |
| | hadoop.ozone.container.TestContainerReplication |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.ozone.om.TestOmAcls |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-909 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951058/HDDS-909.01.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux a35b7dc609d4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 154449f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDDS-Build/1898/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1898/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1898/testReport/ |
| Max. process+thread count | 1255 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/common hadoop-ozone/common
hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1898/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
>
[jira] [Commented] (HDDS-115) GRPC: Support secure gRPC endpoint with mTLS
[
https://issues.apache.org/jira/browse/HDDS-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713421#comment-16713421
]
Hadoop QA commented on HDDS-115:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} HDDS-4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s{color} | {color:green} HDDS-4 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} HDDS-4 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} HDDS-4 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} HDDS-4 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 2m
4s{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 40s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
15s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 9 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-115 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951063/HDDS-115-HDDS-4.009.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle
shellcheck |
| uname | Linux c813d010d6e5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | HDDS-4 / 5227f75 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| shellcheck | v0.4.6 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDDS-Build/1899/artifact/out/patch-mvninstall-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1899/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1899/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDDS-Build/1899/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 197 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1899/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> GRPC: Support secure gRPC endpoint with mTLS
> -
>
> Key: HDDS-115
> URL: https://issues.apache.org/jira/browse/HDDS-115
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713419#comment-16713419
]
Hadoop QA commented on HDFS-14134:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 45s{color} | {color:orange} root: The patch generated 2 new + 70 unchanged -
0 fixed = 72 total (was 70) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 23s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.crypto.key.kms.TestLoadBalancingKMSClientProvider
|
| | hadoop.io.retry.TestDefaultRetryPolicy |
| | hadoop.io.retry.TestFailoverProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14134 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951049/HDFS-14134.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e77d6c1073ca 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 154449f |
|
[jira] [Commented] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[
https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713417#comment-16713417
]
Hadoop QA commented on HDFS-14001:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 5s{color} | {color:orange} root: The patch generated 3 new + 425 unchanged
- 0 fixed = 428 total (was 425) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestPread |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14001 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951038/HDFS-14001.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6f9c93273e1c 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 154449f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs |
[jira] [Comment Edited] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713393#comment-16713393
]
Ajay Kumar edited comment on HDDS-909 at 12/7/18 11:06 PM:
---
Patch v1 with following changes:
* adds test class to test negative test case for volume, bucket and key.
* removes field path form {{OzoneObj}} and adds fields for volume bucket and
key in its child class. Path is constructed from these fields.
* Added test case for getKeyName.
was (Author: ajayydv):
Patch v2 with following changes:
* adds test class to test negative test case for volume, bucket and key.
* removes field path form {{OzoneObj}} and adds fields for volume bucket and
key in its child class. Path is constructed from these fields.
* Added test case for getKeyName.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDDS-909.00.patch, HDDS-909.01.patch
>
>
> Default implementation for Ozone acls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-909: Attachment: HDDS-909.01.patch > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch, HDDS-909.01.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713401#comment-16713401
]
Anu Engineer commented on HDDS-909:
---
Thanks for updating the patch. Some minor comments inline.
# {code}
default:
return OzoneConsts.OZONE_URI_DELIMITER;
{code}
Not sure I understand why we are returning OZONE_URI_DELIMITER in the default
case.
# In TestOzoneObjInfo
{code}
.setResType(RESOURCE_TYPE)
{code}
SetResourceType(resource) is not very easy to understand, did you intended to
write the setResType(Volume_Type)
# in TestOzoneObjInfo#testGetBucketName - line 69, 70 looks same as 72,73. Just
want to make sure it is intentional.
+1, once these are addressed and there might be some checkStyle issues that you
might need to take care from the jenkins.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDDS-909.00.patch, HDDS-909.01.patch
>
>
> Default implementation for Ozone acls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-115) GRPC: Support secure gRPC endpoint with mTLS
[ https://issues.apache.org/jira/browse/HDDS-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-115: Attachment: HDDS-115-HDDS-4.009.patch > GRPC: Support secure gRPC endpoint with mTLS > - > > Key: HDDS-115 > URL: https://issues.apache.org/jira/browse/HDDS-115 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HDDS-115-HDDS-4.001.patch, HDDS-115-HDDS-4.002.patch, > HDDS-115-HDDS-4.003.patch, HDDS-115-HDDS-4.004.patch, > HDDS-115-HDDS-4.005.patch, HDDS-115-HDDS-4.006.patch, > HDDS-115-HDDS-4.008.patch, HDDS-115-HDDS-4.009.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713398#comment-16713398
]
Hadoop QA commented on HDFS-14134:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 3s{color} | {color:orange} root: The patch generated 2 new + 70 unchanged -
0 fixed = 72 total (was 70) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 57s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
42s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.crypto.key.kms.TestLoadBalancingKMSClientProvider
|
| | hadoop.io.retry.TestDefaultRetryPolicy |
| | hadoop.io.retry.TestFailoverProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14134 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951030/HDFS-14134.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 124b32531eb9 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 154449f |
|
[jira] [Comment Edited] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713393#comment-16713393
]
Ajay Kumar edited comment on HDDS-909 at 12/7/18 11:07 PM:
---
Patch v1 with following changes:
* adds test class to test negative test case for volume, bucket and key.
* removes field path form {{OzoneObj}} and adds fields for volume bucket and
key in its child class. Path is constructed from these fields.
* Added test case for getKeyName.
* Fixed test failure related to this patch.
was (Author: ajayydv):
Patch v1 with following changes:
* adds test class to test negative test case for volume, bucket and key.
* removes field path form {{OzoneObj}} and adds fields for volume bucket and
key in its child class. Path is constructed from these fields.
* Added test case for getKeyName.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDDS-909.00.patch, HDDS-909.01.patch
>
>
> Default implementation for Ozone acls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-910) Expose OMMetrics
[
https://issues.apache.org/jira/browse/HDDS-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713396#comment-16713396
]
Hadoop QA commented on HDDS-910:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 14s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
41s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.om.TestOzoneManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-910 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951052/HDDS-910.00.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 4e48ca45ad0a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 154449f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1897/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1897/testReport/ |
| Max. process+thread count | 1264 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/integration-test hadoop-ozone/ozone-manager U:
hadoop-ozone |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1897/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Expose OMMetrics
>
>
> Key: HDDS-910
> URL: https://issues.apache.org/jira/browse/HDDS-910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-910.00.patch
>
>
> Implement MetricsSource interface, so that external metrics can collect the
> OMMetrics.
>
> From *MetricsSource.java:*
> It registers with \{@link MetricsSystem}, which periodically polls it to
> collect \{@link MetricsRecord} and passes
[jira] [Updated] (HDDS-115) GRPC: Support secure gRPC endpoint with mTLS
[ https://issues.apache.org/jira/browse/HDDS-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-115: Attachment: HDDS-115-HDDS-4.008.patch > GRPC: Support secure gRPC endpoint with mTLS > - > > Key: HDDS-115 > URL: https://issues.apache.org/jira/browse/HDDS-115 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HDDS-115-HDDS-4.001.patch, HDDS-115-HDDS-4.002.patch, > HDDS-115-HDDS-4.003.patch, HDDS-115-HDDS-4.004.patch, > HDDS-115-HDDS-4.005.patch, HDDS-115-HDDS-4.006.patch, > HDDS-115-HDDS-4.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713393#comment-16713393
]
Ajay Kumar commented on HDDS-909:
-
Patch v2 with following changes:
* adds test class to test negative test case for volume, bucket and key.
* removes field path form {{OzoneObj}} and adds fields for volume bucket and
key in its child class. Path is constructed from these fields.
* Added test case for getKeyName.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDDS-909.00.patch, HDDS-909.01.patch
>
>
> Default implementation for Ozone acls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-14001: -- Status: Patch Available (was: Open) > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch, HDFS-14001.005.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14081) hdfs dfsadmin -metasave metasave_test results NPE
[
https://issues.apache.org/jira/browse/HDFS-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713363#comment-16713363
]
Shweta commented on HDFS-14081:
---
Thanks for the insightful description [~kihwal]. I will work on disabling the
metasave for the Standby NN and will post a patch soon.
> hdfs dfsadmin -metasave metasave_test results NPE
> -
>
> Key: HDFS-14081
> URL: https://issues.apache.org/jira/browse/HDFS-14081
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: Shweta
>Assignee: Shweta
>Priority: Major
> Attachments: HDFS-14081.001.patch, HDFS-14081.002.patch,
> HDFS-14081.003.patch, HDFS-14081.004.patch
>
>
> Race condition is encountered while adding Block to
> postponedMisreplicatedBlocks which in turn tried to retrieve Block from
> BlockManager in which it may not be present.
> This happens in HA, metasave in first NN succeeded but failed in second NN,
> StackTrace showing NPE is as follows:
> {code}
> 2018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 24 on 8020, call Call#1 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from
> 172.26.9.163:602342018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server:
> IPC Server handler 24 on 8020, call Call#1 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from
> 172.26.9.163:60234java.lang.NullPointerException at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseSourceDatanodes(BlockManager.java:2175)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.dumpBlockMeta(BlockManager.java:830)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.metaSave(BlockManager.java:762)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1782)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1766)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.metaSave(NameNodeRpcServer.java:1320)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.metaSave(ClientNamenodeProtocolServerSideTranslatorPB.java:928)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-910) Expose OMMetrics
[
https://issues.apache.org/jira/browse/HDDS-910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-910:
Attachment: HDDS-910.00.patch
> Expose OMMetrics
>
>
> Key: HDDS-910
> URL: https://issues.apache.org/jira/browse/HDDS-910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-910.00.patch
>
>
> Implement MetricsSource interface, so that external metrics can collect the
> OMMetrics.
>
> From *MetricsSource.java:*
> It registers with \{@link MetricsSystem}, which periodically polls it to
> collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713358#comment-16713358
]
Lukas Majercak commented on HDFS-14134:
---
For the retry policy changes, maybe it would make sense to just RETRY when the
exception is RemoteException and the operation is not idempotent/atmostonce.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-910) Expose OMMetrics
[
https://issues.apache.org/jira/browse/HDDS-910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-910:
Target Version/s: 0.4.0
Status: Patch Available (was: In Progress)
> Expose OMMetrics
>
>
> Key: HDDS-910
> URL: https://issues.apache.org/jira/browse/HDDS-910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-910.00.patch
>
>
> Implement MetricsSource interface, so that external metrics can collect the
> OMMetrics.
>
> From *MetricsSource.java:*
> It registers with \{@link MetricsSystem}, which periodically polls it to
> collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[
https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713356#comment-16713356
]
Virajith Jalaparti commented on HDFS-14001:
---
[^HDFS-14001.005.patch] fixes the {{HiddenField}} checkstyle error as well.
> [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
> --
>
> Key: HDFS-14001
> URL: https://issues.apache.org/jira/browse/HDFS-14001
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch,
> HDFS-14001.003.patch, HDFS-14001.004.patch, HDFS-14001.005.patch
>
>
> Currently, we generate the fsimage and the alias map in one machine. When we
> start the other NNs, we use bootstrapStandby to propagate the fsimage.
> However, we need to copy the Alias Map by hand. We should copy also the Alias
> Map as part of this process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-14001: -- Status: Open (was: Patch Available) > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch, HDFS-14001.005.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-14001: -- Attachment: HDFS-14001.005.patch > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch, HDFS-14001.005.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-881) Add support to transform client requests to OM into Ratis requests
[
https://issues.apache.org/jira/browse/HDDS-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713354#comment-16713354
]
Hadoop QA commented on HDDS-881:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange} hadoop-ozone: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 8 line(s) with tabs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} root generated 0 new + 13 unchanged - 1 fixed = 13
total (was 14) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 30s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
43s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.om.TestOzoneManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-881 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951045/HDDS-881.002.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux ea6e407c40fc 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 154449f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDDS-Build/1896/artifact/out/diff-checkstyle-hadoop-ozone.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDDS-Build/1896/artifact/out/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1896/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1896/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDDS-Build/1896/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1114 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager U: hadoop-ozone |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1896/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support to transform client requests to OM into Ratis requests
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-14134:
--
Attachment: HDFS-14134.001.patch
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-14134:
--
Attachment: (was: HDFS-14134.001.patch)
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[
https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713338#comment-16713338
]
Hadoop QA commented on HDFS-14131:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
43s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
32m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14131 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951041/HDFS-14131-HDFS-12943.000.patch
|
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux 77257f83cb43 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-12943 / 47d7260 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 415 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25732/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Create user guide for "Consistent reads from Observer" feature.
> ---
>
> Key: HDFS-14131
> URL: https://issues.apache.org/jira/browse/HDFS-14131
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: HDFS-12943
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-14131-HDFS-12943.000.patch
>
>
> The documentation should give an overview of the feature, explain
> configuration parameters, startup procedure, give an example of recommended
> deployment.
> It should include the description of Fast Edits Tailing HDFS-13150, as this
> is required for efficient reads from Observer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713326#comment-16713326
]
Lukas Majercak commented on HDFS-14134:
---
Reuploaded the patch, because this guy Yetus took my pdf file as patch.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-910) Expose OMMetrics
[
https://issues.apache.org/jira/browse/HDDS-910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDDS-910 started by Bharat Viswanadham.
---
> Expose OMMetrics
>
>
> Key: HDDS-910
> URL: https://issues.apache.org/jira/browse/HDDS-910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>
> Implement MetricsSource interface, so that external metrics can collect the
> OMMetrics.
>
> From *MetricsSource.java:*
> It registers with \{@link MetricsSystem}, which periodically polls it to
> collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-910) Expose OMMetrics
Bharat Viswanadham created HDDS-910:
---
Summary: Expose OMMetrics
Key: HDDS-910
URL: https://issues.apache.org/jira/browse/HDDS-910
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham
Implement MetricsSource interface, so that external metrics can collect the
OMMetrics.
>From *MetricsSource.java:*
It registers with \{@link MetricsSystem}, which periodically polls it to
collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713314#comment-16713314
]
Lukas Majercak commented on HDFS-14134:
---
Added HDFS-14134_retrypolicy_change_proposal.pdf to illustrate the proposed
changes in the FailoverOnNetworkExceptionRetry retry policy.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713319#comment-16713319
]
Hadoop QA commented on HDFS-14134:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-14134 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14134 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25734/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-14134:
--
Attachment: HDFS-14134_retrypolicy_change_proposal.pdf
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713303#comment-16713303
]
Íñigo Goiri commented on HDFS-12861:
We have this code currently in production.
I don't think [~mf_borge] is working on it anymore.
I can take over the review if needed.
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: María Fernanda Borge
>Priority: Major
> Attachments: HDFS-12861-10-april-18.patch
>
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713301#comment-16713301
]
Íñigo Goiri commented on HDFS-14134:
There are a bunch of related JIRAs:
* HADOOP-9792
* HADOOP-7896
* HDFS-5371
* HDFS-4974
* HDFS-2393
* HDFS-1973
The most relevant is HADOOP-9792.
[~sureshms] you have involved in most of those JIRAs.
Can you chime in?
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-881) Add support to transform client requests to OM into Ratis requests
[ https://issues.apache.org/jira/browse/HDDS-881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-881: Attachment: HDDS-881.002.patch > Add support to transform client requests to OM into Ratis requests > -- > > Key: HDDS-881 > URL: https://issues.apache.org/jira/browse/HDDS-881 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-881.001.patch, HDDS-881.002.patch > > > When OM receives a request, we need to transform the request into Ratis > server compatible request so that the OM's Ratis server can process that > request. > In this Jira, we just add the support to convert a client request received by > OM into a RaftClient request. This transformed request would later be passed > onto the OM's Ratis server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-881) Add support to transform client requests to OM into Ratis requests
[ https://issues.apache.org/jira/browse/HDDS-881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-881: Target Version/s: 0.5.0 Status: Patch Available (was: Open) Separated the Ratis protocol from other OzoneManager protocols so that the ratis part can be compiled with proto3. > Add support to transform client requests to OM into Ratis requests > -- > > Key: HDDS-881 > URL: https://issues.apache.org/jira/browse/HDDS-881 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-881.001.patch, HDDS-881.002.patch > > > When OM receives a request, we need to transform the request into Ratis > server compatible request so that the OM's Ratis server can process that > request. > In this Jira, we just add the support to convert a client request received by > OM into a RaftClient request. This transformed request would later be passed > onto the OM's Ratis server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14134:
---
Status: Patch Available (was: Open)
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713290#comment-16713290 ] Chao Sun commented on HDFS-14131: - And here's the [link|https://github.com/sunchao/hadoop/blob/HDFS-14131/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md] to the rendered page. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-14131: Attachment: HDFS-14131-HDFS-12943.000.patch > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-14131: Attachment: (was: HDFS-14131-HDFS-12943.000.patch) > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713289#comment-16713289 ] Chao Sun commented on HDFS-14131: - Attached patch v0. I'm sure there will be something missing :-) Comments are welcome especially from [~shv], [~xkrogen], [~vagarychen] and [~zero45]. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-14131: Status: Patch Available (was: Open) > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-14131: Attachment: HDFS-14131-HDFS-12943.000.patch > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-14001: -- Status: Open (was: Patch Available) > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[
https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713277#comment-16713277
]
Virajith Jalaparti commented on HDFS-14001:
---
[^HDFS-14001.004.patch] removes the use of {{IO_FILE_BUFFER_SIZE}} in
{{InMemoryAliasMap}} for the copy and replaces this with the appropriate
{{IOUtils.copyBytes}} method. This should fix one of the checkstyle issues. I
think we can ignore the others.
> [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
> --
>
> Key: HDFS-14001
> URL: https://issues.apache.org/jira/browse/HDFS-14001
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch,
> HDFS-14001.003.patch, HDFS-14001.004.patch
>
>
> Currently, we generate the fsimage and the alias map in one machine. When we
> start the other NNs, we use bootstrapStandby to propagate the fsimage.
> However, we need to copy the Alias Map by hand. We should copy also the Alias
> Map as part of this process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-14001: -- Status: Patch Available (was: Open) > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-14001: -- Attachment: HDFS-14001.004.patch > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch, HDFS-14001.004.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14129) RBF: Create new policy provider for router
[
https://issues.apache.org/jira/browse/HDFS-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713255#comment-16713255
]
Hadoop QA commented on HDFS-14129:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
39s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
43s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
55s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 58s{color} | {color:orange} root: The patch generated 11 new + 80 unchanged
- 0 fixed = 91 total (was 80) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
56s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
40s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
52s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
| | org.apache.hadoop.hdfs.protocolPB.RouterPolicyProvider.getServices() may
expose internal representation by returning RouterPolicyProvider.rbfServices
At RouterPolicyProvider.java:by returning RouterPolicyProvider.rbfServices At
RouterPolicyProvider.java:[line 36] |
\\
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713240#comment-16713240
]
Hadoop QA commented on HDDS-909:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} root: The patch generated 2 new + 0 unchanged -
0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 29m 33s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
29s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.om.TestOmMetrics |
| | hadoop.ozone.container.TestContainerReplication |
| | hadoop.ozone.om.TestOzoneManager |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-909 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951027/HDDS-909.00.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 946a12986b71 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 154449f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDDS-Build/1895/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1895/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1895/testReport/ |
| Max. process+thread count | 1333 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/common hadoop-ozone/common
hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1895/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-14134:
--
Attachment: HDFS-14134.001.patch
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where file does
> not have the attribute, NN throws an IOException with message "could not find
> attr". The current client retry policy determines the action for that to be
> FAILOVER_AND_RETRY. The client then fails over and retries until it reaches
> the maximum number of retries. Supposedly, the client should be able to tell
> that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-14134:
--
Attachment: HDFS-14134.001.patch
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-14134:
--
Attachment: (was: HDFS-14134.001.patch)
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-14134:
--
Description:
Currently, some operations that throw IOException on the NameNode are evaluated
by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail fast.
For example, when calling getXAttr("user.some_attr", file") where the file does
not have the attribute, NN throws an IOException with message "could not find
attr". The current client retry policy determines the action for that to be
FAILOVER_AND_RETRY. The client then fails over and retries until it reaches the
maximum number of retries. Supposedly, the client should be able to tell that
this exception is normal and fail fast.
Moreover, even if the action was FAIL, the RetryInvocationHandler looks at all
the retry actions from all requests, and FAILOVER_AND_RETRY takes precedence
over FAIL action.
was:
Currently, some operations that throw IOException on the NameNode are evaluated
by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail fast.
For example, when calling getXAttr("user.some_attr", file") where file does not
have the attribute, NN throws an IOException with message "could not find
attr". The current client retry policy determines the action for that to be
FAILOVER_AND_RETRY. The client then fails over and retries until it reaches the
maximum number of retries. Supposedly, the client should be able to tell that
this exception is normal and fail fast.
Moreover, even if the action was FAIL, the RetryInvocationHandler looks at all
the retry actions from all requests, and FAILOVER_AND_RETRY takes precedence
over FAIL action.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713233#comment-16713233
]
Lukas Majercak commented on HDFS-14134:
---
Added a patch to demonstrate the issue.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where file does
> not have the attribute, NN throws an IOException with message "could not find
> attr". The current client retry policy determines the action for that to be
> FAILOVER_AND_RETRY. The client then fails over and retries until it reaches
> the maximum number of retries. Supposedly, the client should be able to tell
> that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-14134:
--
Description:
Currently, some operations that throw IOException on the NameNode are evaluated
by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail fast.
For example, when calling getXAttr("user.some_attr", file") where file does not
have the attribute, NN throws an IOException with message "could not find
attr". The current client retry policy determines the action for that to be
FAILOVER_AND_RETRY. The client then fails over and retries until it reaches the
maximum number of retries. Supposedly, the client should be able to tell that
this exception is normal and fail fast.
Moreover, even if the action was FAIL, the RetryInvocationHandler looks at all
the retry actions from all requests, and FAILOVER_AND_RETRY takes precedence
over FAIL action.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where file does
> not have the attribute, NN throws an IOException with message "could not find
> attr". The current client retry policy determines the action for that to be
> FAILOVER_AND_RETRY. The client then fails over and retries until it reaches
> the maximum number of retries. Supposedly, the client should be able to tell
> that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713232#comment-16713232 ] Ajay Kumar commented on HDDS-909: - [~anu], [~jnp] thanks for comments. Path field complicates whole validation process. I like [~jnp] suggestion on constructing path from individual components. Will update the patch accordingly. [~anu] With this change mentioned delimiter problem will not be applicable anymore. > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
Lukas Majercak created HDFS-14134: - Summary: Idempotent operations throwing RemoteException should not be retried by the client Key: HDFS-14134 URL: https://issues.apache.org/jira/browse/HDFS-14134 Project: Hadoop HDFS Issue Type: Bug Components: hdfs, hdfs-client, ipc Reporter: Lukas Majercak Assignee: Lukas Majercak -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713223#comment-16713223 ] Jitendra Nath Pandey commented on HDDS-909: --- I wonder, why we need path at all in OzoneObj. We can just track volume/buckets and keys. These fields need to exist only in OzoneObjInfo class. > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713223#comment-16713223 ] Jitendra Nath Pandey edited comment on HDDS-909 at 12/7/18 7:41 PM: I wonder, why we need path at all in OzoneObj. We can just track volume/buckets and keys. These fields need to exist only in OzoneObjInfo class. The getPath should rather construct the path from these fields in OzoneObj. was (Author: jnp): I wonder, why we need path at all in OzoneObj. We can just track volume/buckets and keys. These fields need to exist only in OzoneObjInfo class. > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[
https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713220#comment-16713220
]
Anu Engineer commented on HDDS-909:
---
Thanks for the patch. I think there is an issue, please verify if what I am
saying is true.
we do this operation to break a string to paths
{code:java}
pathComponents = (List) StringUtils.getStringCollection(getPath(),
OzoneConsts.OZONE_URI_DELIMITER);
{code}
Later we do
{code:java}
@Override
public String getKeyName() {
if (pathComponents.size() > 2) {
return pathComponents.get(2);
} else {
return null;
}
}
{code}
I am worried that this will fail if the key itself has OZONE URI DELIMITER.
That is a legal key name.
Since I was looking at this I also realized we don't have tests for the
getKeyName call, It would be nice to add some tests for getKeyName, and
specifically for this case. There are probably some check style issues too, but
we can wait for Jenkins to flag them.
> Default implementation for Ozone acls
> -
>
> Key: HDDS-909
> URL: https://issues.apache.org/jira/browse/HDDS-909
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDDS-909.00.patch
>
>
> Default implementation for Ozone acls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-909: Status: Patch Available (was: Open) > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-909: Attachment: HDDS-909.00.patch > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-879) MultipartUpload: Add InitiateMultipartUpload in ozone
[
https://issues.apache.org/jira/browse/HDDS-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713197#comment-16713197
]
Hadoop QA commented on HDDS-879:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 3s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
15s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.ozone.container.TestContainerReplication |
| | hadoop.ozone.om.TestOzoneManager |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-879 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951024/HDDS-879.06.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 802b3c2e7ddc 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 6c852f2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1894/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1894/testReport/ |
| Max. process+thread count | 1213 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/common hadoop-ozone/client hadoop-ozone/common
hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1894/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> MultipartUpload: Add InitiateMultipartUpload in ozone
> -
>
> Key: HDDS-879
> URL: https://issues.apache.org/jira/browse/HDDS-879
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat
[jira] [Commented] (HDDS-815) Rename Ozone/HDDS config keys prefixed with 'dfs'
[
https://issues.apache.org/jira/browse/HDDS-815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713181#comment-16713181
]
Dinesh Chitlangia commented on HDDS-815:
failures unrelated to patch
> Rename Ozone/HDDS config keys prefixed with 'dfs'
> -
>
> Key: HDDS-815
> URL: https://issues.apache.org/jira/browse/HDDS-815
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Dinesh Chitlangia
>Priority: Major
> Labels: newbie
> Attachments: HDDS-815.001.patch
>
>
> The following Ozone config keys are prefixed with dfs which is the prefix
> used by HDFS. Instead we should prefix them with either HDDS or Ozone.
> {code}
> dfs.container.ipc
> dfs.container.ipc.random.port
> dfs.container.ratis.datanode.storage.dir
> dfs.container.ratis.enabled
> dfs.container.ratis.ipc
> dfs.container.ratis.ipc.random.port
> dfs.container.ratis.num.container.op.executors
> dfs.container.ratis.num.write.chunk.threads
> dfs.container.ratis.replication.level
> dfs.container.ratis.rpc.type
> dfs.container.ratis.segment.preallocated.size
> dfs.container.ratis.segment.size
> dfs.container.ratis.statemachinedata.sync.timeout
> dfs.ratis.client.request.max.retries
> dfs.ratis.client.request.retry.interval
> dfs.ratis.client.request.timeout.duration
> dfs.ratis.leader.election.minimum.timeout.duration
> dfs.ratis.server.failure.duration
> dfs.ratis.server.request.timeout.duration
> dfs.ratis.server.retry-cache.timeout.duration
> dfs.ratis.snapshot.threshold
> {code}
> Additionally, _dfs.container.ipc_ should be changed to
> _dfs.container.ipc.port_.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14084) Need for more stats in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713168#comment-16713168
]
Erik Krogen commented on HDFS-14084:
Nice, v002 became surprisingly easy :) A few comments:
* Can we create a separate log for this e.g. {code}private static final Logger
PERF_LOG = LoggerFactory.getLogger(DFSClient.getClass().getCanonicalName() +
"_Performance");{code} or something like that? Then you can more easily
enable/disable this logging without having to affect the log level of the rest
of DFSClient.
* Can you please post an example of what the log output looks like?
* You took a few things that were guarded by {{isDebugEnabled()}} and now have
them execute regardless of log level. I would prefer to see all of this code
guarded by {{isDebugEnabled()}} so that none of it is evaluated unless output
will be produced.
As a more general comment, I would still love to see this integrated with
{{StorageStatistics}}, but I think the logging approach makes sense as a
complement. It might not be a bad idea to leave this JIRA as the v002 patch and
do a follow-on to integrate with {{StorageStatistics}}.
> Need for more stats in DFSClient
>
>
> Key: HDFS-14084
> URL: https://issues.apache.org/jira/browse/HDFS-14084
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Minor
> Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch
>
>
> The usage of HDFS has changed from being used as a map-reduce filesystem, now
> it's becoming more of like a general purpose filesystem. In most of the cases
> there are issues with the Namenode so we have metrics to know the workload or
> stress on Namenode.
> However, there is a need to have more statistics collected for different
> operations/RPCs in DFSClient to know which RPC operations are taking longer
> time or to know what is the frequency of the operation.These statistics can
> be exposed to the users of DFS Client and they can periodically log or do
> some sort of flow control if the response is slow. This will also help to
> isolate HDFS issue in a mixed environment where on a node say we have Spark,
> HBase and Impala running together. We can check the throughput of different
> operation across client and isolate the problem caused because of noisy
> neighbor or network congestion or shared JVM.
> We have dealt with several problems from the field for which there is no
> conclusive evidence as to what caused the problem. If we had metrics or stats
> in DFSClient we would be better equipped to solve such complex problems.
> List of jiras for reference:
> -
> HADOOP-15538 HADOOP-15530 ( client side deadlock)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14084) Need for more stats in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713153#comment-16713153
]
Íñigo Goiri commented on HDFS-14084:
[^HDFS-14084.002.patch] looks surprisingly clean :)
This definitely needs a unit test.
Minor comments:
* You could directly do {{long startTime = Time.now();}}, no need to split.
* Should the log be trace?
> Need for more stats in DFSClient
>
>
> Key: HDFS-14084
> URL: https://issues.apache.org/jira/browse/HDFS-14084
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Minor
> Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch
>
>
> The usage of HDFS has changed from being used as a map-reduce filesystem, now
> it's becoming more of like a general purpose filesystem. In most of the cases
> there are issues with the Namenode so we have metrics to know the workload or
> stress on Namenode.
> However, there is a need to have more statistics collected for different
> operations/RPCs in DFSClient to know which RPC operations are taking longer
> time or to know what is the frequency of the operation.These statistics can
> be exposed to the users of DFS Client and they can periodically log or do
> some sort of flow control if the response is slow. This will also help to
> isolate HDFS issue in a mixed environment where on a node say we have Spark,
> HBase and Impala running together. We can check the throughput of different
> operation across client and isolate the problem caused because of noisy
> neighbor or network congestion or shared JVM.
> We have dealt with several problems from the field for which there is no
> conclusive evidence as to what caused the problem. If we had metrics or stats
> in DFSClient we would be better equipped to solve such complex problems.
> List of jiras for reference:
> -
> HADOOP-15538 HADOOP-15530 ( client side deadlock)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-879) MultipartUpload: Add InitiateMultipartUpload in ozone
[ https://issues.apache.org/jira/browse/HDDS-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-879: Attachment: HDDS-879.06.patch > MultipartUpload: Add InitiateMultipartUpload in ozone > - > > Key: HDDS-879 > URL: https://issues.apache.org/jira/browse/HDDS-879 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDDS-879.01.patch, HDDS-879.02.patch, HDDS-879.03.patch, > HDDS-879.04.patch, HDDS-879.05.patch, HDDS-879.06.patch > > > This Jira is to add initiate multipart upload. > InitiateMultipart upload does 2 things: > # Create an entry in the open table for this key > # Add multipartInfo information for this key in to multipartinfo table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-871) XceiverClientRatis#watchForCommit closes raft client at every call
[ https://issues.apache.org/jira/browse/HDDS-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713163#comment-16713163 ] Bharat Viswanadham commented on HDDS-871: - Hi [~ljain] Can we link the Jira which fixed this issue, so that it will be helpful for someone who looks into the jira's or who is watching these jira's. > XceiverClientRatis#watchForCommit closes raft client at every call > -- > > Key: HDDS-871 > URL: https://issues.apache.org/jira/browse/HDDS-871 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > > Currently XceiverClientRatis#watchForCommit closes raft client at every call. > We should cache the raft client and reuse it for future watchForCommit calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[
https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713146#comment-16713146
]
Íñigo Goiri commented on HDFS-13443:
{quote}
I feel default value for this property should be true and description should
say, "for all the routers"
{quote}
The current behavior in trunk is the equivalent to {{false}}.
I'm not sure I would consider this an incompatible change if making it {{true}}
but definitely something to keep in mind.
{quote}
The refresh API will only makes sense in the shared State Store type not local
type store. right?
{quote}
Not sure if we have any mechanism to verify that a State Store is shared or
local.
I would have a unit test though.
> RBF: Update mount table cache immediately after changing (add/update/remove)
> mount table entries.
> -
>
> Key: HDFS-13443
> URL: https://issues.apache.org/jira/browse/HDFS-13443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Labels: RBF
> Attachments: HDFS-13443-012.patch, HDFS-13443-013.patch,
> HDFS-13443-014.patch, HDFS-13443-015.patch, HDFS-13443-016.patch,
> HDFS-13443-branch-2.001.patch, HDFS-13443-branch-2.002.patch,
> HDFS-13443.001.patch, HDFS-13443.002.patch, HDFS-13443.003.patch,
> HDFS-13443.004.patch, HDFS-13443.005.patch, HDFS-13443.006.patch,
> HDFS-13443.007.patch, HDFS-13443.008.patch, HDFS-13443.009.patch,
> HDFS-13443.010.patch, HDFS-13443.011.patch
>
>
> Currently mount table cache is updated periodically, by default cache is
> updated every minute. After change in mount table, user operations may still
> use old mount table. This is bit wrong.
> To update mount table cache, maybe we can do following
> * *Add refresh API in MountTableManager which will update mount table cache.*
> * *When there is a change in mount table entries, router admin server can
> update its cache and ask other routers to update their cache*. For example if
> there are three routers R1,R2,R3 in a cluster then add mount table entry API,
> at admin server side, will perform following sequence of action
> ## user submit add mount table entry request on R1
> ## R1 adds the mount table entry in state store
> ## R1 call refresh API on R2
> ## R1 calls refresh API on R3
> ## R1 directly freshest its cache
> ## Add mount table entry response send back to user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14001) [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap
[ https://issues.apache.org/jira/browse/HDFS-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713141#comment-16713141 ] Íñigo Goiri commented on HDFS-14001: [^HDFS-14001.003.patch] looks good. I think we can fix most of those checkstyle issues. > [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap > -- > > Key: HDFS-14001 > URL: https://issues.apache.org/jira/browse/HDFS-14001 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-14001.001.patch, HDFS-14001.002.patch, > HDFS-14001.003.patch > > > Currently, we generate the fsimage and the alias map in one machine. When we > start the other NNs, we use bootstrapStandby to propagate the fsimage. > However, we need to copy the Alias Map by hand. We should copy also the Alias > Map as part of this process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-879) MultipartUpload: Add InitiateMultipartUpload in ozone
[
https://issues.apache.org/jira/browse/HDDS-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713152#comment-16713152
]
Bharat Viswanadham commented on HDDS-879:
-
Hi [~elek]
Thanks for the review.
{quote}With this patch it will be changed. The OmKeyInfo itself is used on the
rpc client side as a response. I understand why: it's so simple that we need
exactly the same object type, but it maybe useful to follow this convention
even if it's not the better.
{quote}
I feel previously this is done, as from Om*Info response we set it into
OzoneVolume and OzoneBucket, and these have an additional thing called proxy in
them, which is useful in further invoke operations on them, that might be the
reason for doing this way. But here as we need a same response, we can use I
believe. And we don't bring any additional dependencies by using a class from
another module, as this class is any way in hadoop-ozone/common and
hadoop-ozone/client is already dependent on. Let me know if you still want to
have a new class for this response.
{quote}And a nit: The finally block is not strictly required in OzoneManager
(If I understood well). we can use the AUDIT.logWriteSuccess as the last step
inside the try bock. (But again, not very important. The current implementation
also works good just with a few additional lines).
{quote}
Done.
> MultipartUpload: Add InitiateMultipartUpload in ozone
> -
>
> Key: HDDS-879
> URL: https://issues.apache.org/jira/browse/HDDS-879
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-879.01.patch, HDDS-879.02.patch, HDDS-879.03.patch,
> HDDS-879.04.patch, HDDS-879.05.patch, HDDS-879.06.patch
>
>
> This Jira is to add initiate multipart upload.
> InitiateMultipart upload does 2 things:
> # Create an entry in the open table for this key
> # Add multipartInfo information for this key in to multipartinfo table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-909) Default implementation for Ozone acls
Ajay Kumar created HDDS-909: --- Summary: Default implementation for Ozone acls Key: HDDS-909 URL: https://issues.apache.org/jira/browse/HDDS-909 Project: Hadoop Distributed Data Store Issue Type: New Feature Reporter: Ajay Kumar Assignee: Ajay Kumar Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14129) RBF: Create new policy provider for router
[
https://issues.apache.org/jira/browse/HDFS-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14129:
---
Status: Patch Available (was: Open)
> RBF: Create new policy provider for router
> --
>
> Key: HDFS-14129
> URL: https://issues.apache.org/jira/browse/HDFS-14129
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-13532
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14129-HDFS-13891.001.patch
>
>
> Router is using *{{HDFSPolicyProvider}}*. We can't add new protocol in this
> class for router, its better to create in policy provider for Router.
> {code:java}
> // Set service-level authorization security policy
> if (conf.getBoolean(HADOOP_SECURITY_AUTHORIZATION, false)) {
> this.adminServer.refreshServiceAcl(conf, new HDFSPolicyProvider());
> }
> {code}
> I got this issue when I am verified HDFS-14079 with secure cluster.
> {noformat}
> ./bin/hdfs dfsrouteradmin -ls /
> ls: Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol
> is not known.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol is
> not known.
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1520)
> at org.apache.hadoop.ipc.Client.call(Client.java:1466)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14084) Need for more stats in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713131#comment-16713131
]
Wei-Chiu Chuang commented on HDFS-14084:
v002 patch:
Suggest to use Time.monotonicNow() instead of Time.now()
Rewrite the following debug log message
{code}
LOG.debug(" RPC Client stats: " + rb.toString());
{code}
with parameterized log messages:
{code}
LOG.debug(" RPC Client stats: {}", rb);
{code}
> Need for more stats in DFSClient
>
>
> Key: HDFS-14084
> URL: https://issues.apache.org/jira/browse/HDFS-14084
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Minor
> Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch
>
>
> The usage of HDFS has changed from being used as a map-reduce filesystem, now
> it's becoming more of like a general purpose filesystem. In most of the cases
> there are issues with the Namenode so we have metrics to know the workload or
> stress on Namenode.
> However, there is a need to have more statistics collected for different
> operations/RPCs in DFSClient to know which RPC operations are taking longer
> time or to know what is the frequency of the operation.These statistics can
> be exposed to the users of DFS Client and they can periodically log or do
> some sort of flow control if the response is slow. This will also help to
> isolate HDFS issue in a mixed environment where on a node say we have Spark,
> HBase and Impala running together. We can check the throughput of different
> operation across client and isolate the problem caused because of noisy
> neighbor or network congestion or shared JVM.
> We have dealt with several problems from the field for which there is no
> conclusive evidence as to what caused the problem. If we had metrics or stats
> in DFSClient we would be better equipped to solve such complex problems.
> List of jiras for reference:
> -
> HADOOP-15538 HADOOP-15530 ( client side deadlock)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-879) MultipartUpload: Add InitiateMultipartUpload in ozone
[ https://issues.apache.org/jira/browse/HDDS-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712998#comment-16712998 ] Elek, Marton commented on HDDS-879: --- Thanks [~bharatviswa] the patch. Overall it looks very good to me. Thanks for the long comment in the KeyManagerImpl (very useful) and the additional audit events. One note: Until now we had a convention to use Om*Args as the parameter of the OzoneManagerProtocolClientSideTranslatorPB client and Om*Info as the response from this client. The RpcClient itself used simple types as the parameters and client specific objects (Such asn OzoneVolume and OzoneBucket). With this patch it will be changed. The OmKeyInfo itself is used on the rpc client side as a response. I understand why: it's so simple that we need exactly the same object type, but it maybe useful to follow this convention even if it's not the better. But not a strict requirement. This is just something which can be considered. And a nit: The finally block is not strictly required in OzoneManager (If I understood well). we can use the AUDIT.logWriteSuccess as the last step inside the try bock. (But again, not very important. The current implementation also works good just with a few additional lines). But with these comments I am +1 for the patch. > MultipartUpload: Add InitiateMultipartUpload in ozone > - > > Key: HDDS-879 > URL: https://issues.apache.org/jira/browse/HDDS-879 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDDS-879.01.patch, HDDS-879.02.patch, HDDS-879.03.patch, > HDDS-879.04.patch, HDDS-879.05.patch > > > This Jira is to add initiate multipart upload. > InitiateMultipart upload does 2 things: > # Create an entry in the open table for this key > # Add multipartInfo information for this key in to multipartinfo table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14121) Log message about the old hosts file format is misleading
[
https://issues.apache.org/jira/browse/HDFS-14121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712817#comment-16712817
]
Kitti Nanasi commented on HDFS-14121:
-
Thanks [~zvenczel] for the patch!
The patch overall looks good to me. I only have one minor comment, that the
warning message about the empty file content could be more descriptive.
> Log message about the old hosts file format is misleading
> -
>
> Key: HDFS-14121
> URL: https://issues.apache.org/jira/browse/HDFS-14121
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Zsolt Venczel
>Priority: Major
> Attachments: HDFS-14121.01.patch
>
>
> In {{CombinedHostsFileReader.readFile()}} we have the following:
> {code} LOG.warn("{} has invalid JSON format." +
> "Try the old format without top-level token defined.",
> hostsFile);{code}
> That message is trying to say that we tried parsing the hosts file as a
> well-formed JSON file and failed, so we're going to try again assuming that
> it's in the old badly-formed format. What it actually says is that the hosts
> fie is bad, and the admin should try switching to the old format. Those are
> two very different things.
> While were in there, we should refactor the logging so that instead of
> reporting that we're going to try using a different parser (who the heck
> cares?), we report that the we had to use the old parser to successfully
> parse the hosts file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712698#comment-16712698 ] Yiqun Lin edited comment on HDFS-13443 at 12/7/18 11:39 AM: Revisiting this JIRA, I am thinking one thing: The refresh API will only makes sense in the shared State Store type not local type store. right? Please attach the updated rebased on HDFS-13891 branch, current patch is for trunk. was (Author: linyiqun): Revisiting this JIRA, I am thinking one thing: The refresh API will only makes sense in the shared State Store type not local type store. Should we add a note for this? Please attach the updated rebased on HDFS-13891 branch, current patch is for trunk. > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad >Priority: Major > Labels: RBF > Attachments: HDFS-13443-012.patch, HDFS-13443-013.patch, > HDFS-13443-014.patch, HDFS-13443-015.patch, HDFS-13443-016.patch, > HDFS-13443-branch-2.001.patch, HDFS-13443-branch-2.002.patch, > HDFS-13443.001.patch, HDFS-13443.002.patch, HDFS-13443.003.patch, > HDFS-13443.004.patch, HDFS-13443.005.patch, HDFS-13443.006.patch, > HDFS-13443.007.patch, HDFS-13443.008.patch, HDFS-13443.009.patch, > HDFS-13443.010.patch, HDFS-13443.011.patch > > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712698#comment-16712698 ] Yiqun Lin commented on HDFS-13443: -- Revisiting this JIRA, I am thinking one thing: The refresh API will only makes sense in the shared State Store type not local type store. Should we add a note for this? Please attach the updated rebased on HDFS-13891 branch, current patch is for trunk. > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad >Priority: Major > Labels: RBF > Attachments: HDFS-13443-012.patch, HDFS-13443-013.patch, > HDFS-13443-014.patch, HDFS-13443-015.patch, HDFS-13443-016.patch, > HDFS-13443-branch-2.001.patch, HDFS-13443-branch-2.002.patch, > HDFS-13443.001.patch, HDFS-13443.002.patch, HDFS-13443.003.patch, > HDFS-13443.004.patch, HDFS-13443.005.patch, HDFS-13443.006.patch, > HDFS-13443.007.patch, HDFS-13443.008.patch, HDFS-13443.009.patch, > HDFS-13443.010.patch, HDFS-13443.011.patch > > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-871) XceiverClientRatis#watchForCommit closes raft client at every call
[ https://issues.apache.org/jira/browse/HDDS-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain resolved HDDS-871. -- Resolution: Duplicate > XceiverClientRatis#watchForCommit closes raft client at every call > -- > > Key: HDDS-871 > URL: https://issues.apache.org/jira/browse/HDDS-871 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > > Currently XceiverClientRatis#watchForCommit closes raft client at every call. > We should cache the raft client and reuse it for future watchForCommit calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14129) RBF: Create new policy provider for router
[
https://issues.apache.org/jira/browse/HDFS-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712647#comment-16712647
]
Ranith Sardar commented on HDFS-14129:
--
Attached the initial patch. please review it once.
> RBF: Create new policy provider for router
> --
>
> Key: HDFS-14129
> URL: https://issues.apache.org/jira/browse/HDFS-14129
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-13532
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14129-HDFS-13891.001.patch
>
>
> Router is using *{{HDFSPolicyProvider}}*. We can't add new protocol in this
> class for router, its better to create in policy provider for Router.
> {code:java}
> // Set service-level authorization security policy
> if (conf.getBoolean(HADOOP_SECURITY_AUTHORIZATION, false)) {
> this.adminServer.refreshServiceAcl(conf, new HDFSPolicyProvider());
> }
> {code}
> I got this issue when I am verified HDFS-14079 with secure cluster.
> {noformat}
> ./bin/hdfs dfsrouteradmin -ls /
> ls: Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol
> is not known.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol is
> not known.
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1520)
> at org.apache.hadoop.ipc.Client.call(Client.java:1466)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14129) RBF: Create new policy provider for router
[
https://issues.apache.org/jira/browse/HDFS-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranith Sardar updated HDFS-14129:
-
Attachment: HDFS-14129-HDFS-13891.001.patch
> RBF: Create new policy provider for router
> --
>
> Key: HDFS-14129
> URL: https://issues.apache.org/jira/browse/HDFS-14129
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-13532
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14129-HDFS-13891.001.patch
>
>
> Router is using *{{HDFSPolicyProvider}}*. We can't add new protocol in this
> class for router, its better to create in policy provider for Router.
> {code:java}
> // Set service-level authorization security policy
> if (conf.getBoolean(HADOOP_SECURITY_AUTHORIZATION, false)) {
> this.adminServer.refreshServiceAcl(conf, new HDFSPolicyProvider());
> }
> {code}
> I got this issue when I am verified HDFS-14079 with secure cluster.
> {noformat}
> ./bin/hdfs dfsrouteradmin -ls /
> ls: Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol
> is not known.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol is
> not known.
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1520)
> at org.apache.hadoop.ipc.Client.call(Client.java:1466)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[
https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712571#comment-16712571
]
Surendra Singh Lilhore edited comment on HDFS-13443 at 12/7/18 9:56 AM:
Thanks [~arshad.mohammad] for the patch.
Latest patch LGTM.
minor nits
{quote}+| dfs.federation.router.mount-table.cache.update | false | If true,
Mount table cache is updated whenever a mount table entry is added, modified or
removed. |
{quote}
I feel default value for this property should be true and description
should say, "*for all the routers*"
was (Author: surendrasingh):
Thanks [~arshad.mohammad] for the patch.
Latest path LGTM.
minor nits
{quote}+| dfs.federation.router.mount-table.cache.update | false | If true,
Mount table cache is updated whenever a mount table entry is added, modified or
removed. |
{quote}
I feel default value for this property should be true and description
should say, "*for all the routers*"
> RBF: Update mount table cache immediately after changing (add/update/remove)
> mount table entries.
> -
>
> Key: HDFS-13443
> URL: https://issues.apache.org/jira/browse/HDFS-13443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Labels: RBF
> Attachments: HDFS-13443-012.patch, HDFS-13443-013.patch,
> HDFS-13443-014.patch, HDFS-13443-015.patch, HDFS-13443-016.patch,
> HDFS-13443-branch-2.001.patch, HDFS-13443-branch-2.002.patch,
> HDFS-13443.001.patch, HDFS-13443.002.patch, HDFS-13443.003.patch,
> HDFS-13443.004.patch, HDFS-13443.005.patch, HDFS-13443.006.patch,
> HDFS-13443.007.patch, HDFS-13443.008.patch, HDFS-13443.009.patch,
> HDFS-13443.010.patch, HDFS-13443.011.patch
>
>
> Currently mount table cache is updated periodically, by default cache is
> updated every minute. After change in mount table, user operations may still
> use old mount table. This is bit wrong.
> To update mount table cache, maybe we can do following
> * *Add refresh API in MountTableManager which will update mount table cache.*
> * *When there is a change in mount table entries, router admin server can
> update its cache and ask other routers to update their cache*. For example if
> there are three routers R1,R2,R3 in a cluster then add mount table entry API,
> at admin server side, will perform following sequence of action
> ## user submit add mount table entry request on R1
> ## R1 adds the mount table entry in state store
> ## R1 call refresh API on R2
> ## R1 calls refresh API on R3
> ## R1 directly freshest its cache
> ## Add mount table entry response send back to user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[
https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712571#comment-16712571
]
Surendra Singh Lilhore commented on HDFS-13443:
---
Thanks [~arshad.mohammad] for the patch.
Latest path LGTM.
minor nits
{quote}+| dfs.federation.router.mount-table.cache.update | false | If true,
Mount table cache is updated whenever a mount table entry is added, modified or
removed. |
{quote}
I feel default value for this property should be true and description
should say, "*for all the routers*"
> RBF: Update mount table cache immediately after changing (add/update/remove)
> mount table entries.
> -
>
> Key: HDFS-13443
> URL: https://issues.apache.org/jira/browse/HDFS-13443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Labels: RBF
> Attachments: HDFS-13443-012.patch, HDFS-13443-013.patch,
> HDFS-13443-014.patch, HDFS-13443-015.patch, HDFS-13443-016.patch,
> HDFS-13443-branch-2.001.patch, HDFS-13443-branch-2.002.patch,
> HDFS-13443.001.patch, HDFS-13443.002.patch, HDFS-13443.003.patch,
> HDFS-13443.004.patch, HDFS-13443.005.patch, HDFS-13443.006.patch,
> HDFS-13443.007.patch, HDFS-13443.008.patch, HDFS-13443.009.patch,
> HDFS-13443.010.patch, HDFS-13443.011.patch
>
>
> Currently mount table cache is updated periodically, by default cache is
> updated every minute. After change in mount table, user operations may still
> use old mount table. This is bit wrong.
> To update mount table cache, maybe we can do following
> * *Add refresh API in MountTableManager which will update mount table cache.*
> * *When there is a change in mount table entries, router admin server can
> update its cache and ask other routers to update their cache*. For example if
> there are three routers R1,R2,R3 in a cluster then add mount table entry API,
> at admin server side, will perform following sequence of action
> ## user submit add mount table entry request on R1
> ## R1 adds the mount table entry in state store
> ## R1 call refresh API on R2
> ## R1 calls refresh API on R3
> ## R1 directly freshest its cache
> ## Add mount table entry response send back to user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-99) Adding SCM Audit log
[ https://issues.apache.org/jira/browse/HDDS-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712506#comment-16712506 ] Yiqun Lin commented on HDDS-99: --- LGTM, It will be better to let others have a double check, :). > Adding SCM Audit log > > > Key: HDDS-99 > URL: https://issues.apache.org/jira/browse/HDDS-99 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > Labels: alpha2 > Attachments: HDDS-99.001.patch, HDDS-99.002.patch > > > This ticket is opened to add SCM audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13916) Distcp SnapshotDiff not completely implemented for supporting WebHdfs
[
https://issues.apache.org/jira/browse/HDFS-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712476#comment-16712476
]
Hadoop QA commented on HDFS-13916:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} hadoop-tools/hadoop-distcp: The patch generated 0
new + 30 unchanged - 1 fixed = 30 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
58s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13916 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941443/HDFS-13916.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b3352ffd7685 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6c852f2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25729/testReport/ |
| Max. process+thread count | 413 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25729/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Distcp SnapshotDiff not completely implemented for supporting WebHdfs
>
