[jira] [Updated] (HDDS-1750) Add block allocation metric for pipelines in SCM
[ https://issues.apache.org/jira/browse/HDDS-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1750: Status: Patch Available (was: Open) > Add block allocation metric for pipelines in SCM > > > Key: HDDS-1750 > URL: https://issues.apache.org/jira/browse/HDDS-1750 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This Jira aims to add block allocation metrics for pipelines in SCM. This > would help in determining the distribution of block allocations among various > pipelines in SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14313) Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du
[ https://issues.apache.org/jira/browse/HDFS-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879582#comment-16879582 ] Lisheng Sun commented on HDFS-14313: Hi [~elgoiri] [~jojochuang] [~zvenczel] [~hexiaoqiao] Could you have time to review this patch? Thank you . > Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory > instead of df/du > > > Key: HDFS-14313 > URL: https://issues.apache.org/jira/browse/HDFS-14313 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, performance >Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0 >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14313.000.patch, HDFS-14313.001.patch, > HDFS-14313.002.patch, HDFS-14313.003.patch, HDFS-14313.004.patch > > > There are two ways of DU/DF getting used space that are insufficient. > # Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. > # Running DF is inaccurate when the disk sharing by multiple datanode or > other servers. > Getting hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfos in memory > is very small and accurate. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14483) Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9
[ https://issues.apache.org/jira/browse/HDFS-14483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879581#comment-16879581 ] Lisheng Sun commented on HDFS-14483: Hi [~stack] Could you have time to continue reviewing this patch? Thank you. > Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9 > -- > > Key: HDFS-14483 > URL: https://issues.apache.org/jira/browse/HDFS-14483 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Zheng Hu >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14483.branch-2.8.v1.patch, > HDFS-14483.branch-2.9.v1.patch, HDFS-14483.branch-2.9.v1.patch, > HDFS-14483.branch-2.9.v2 (2).patch, HDFS-14483.branch-2.9.v2.patch, > HDFS-14483.branch-2.9.v2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13694) Making md5 computing being in parallel with image loading
[ https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879580#comment-16879580 ] Lisheng Sun commented on HDFS-13694: Thank [~elgoiri] for merging this patch. I think we need to cherry-pick this improvement to all branches. Thanks. > Making md5 computing being in parallel with image loading > - > > Key: HDFS-13694 > URL: https://issues.apache.org/jira/browse/HDFS-13694 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-13694-001.patch, HDFS-13694-002.patch, > HDFS-13694-003.patch, HDFS-13694-004.patch, HDFS-13694-005.patch, > HDFS-13694-006.patch, HDFS-13694-007.patch > > > During namenode image loading, it firstly compute the md5 and then load the > image. Actually these two steps can be in parallel. > Test this patch against a fsimage of a 70PB 2.4 cluster (200million files > and 300million blocks), the image loading time be reduced from 1210 seconds > to 1105 seconds.So it can reduce up to about 10% of time. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-200) Create Dead Node Watcher
[ https://issues.apache.org/jira/browse/HDDS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar reassigned HDDS-200: --- Assignee: (was: Ajay Kumar) > Create Dead Node Watcher > > > Key: HDDS-200 > URL: https://issues.apache.org/jira/browse/HDDS-200 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Priority: Major > > This will be based on HDDS-195. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-200) Create Dead Node Watcher
[ https://issues.apache.org/jira/browse/HDDS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879560#comment-16879560 ] Ajay Kumar commented on HDDS-200: - This can be useful for replication manager and monitoring. > Create Dead Node Watcher > > > Key: HDDS-200 > URL: https://issues.apache.org/jira/browse/HDDS-200 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > > This will be based on HDDS-195. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access
[ https://issues.apache.org/jira/browse/HDDS-1611?focusedWorklogId=272738&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272738 ] ASF GitHub Bot logged work on HDDS-1611: Author: ASF GitHub Bot Created on: 05/Jul/19 21:03 Start Date: 05/Jul/19 21:03 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #973: HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/973#issuecomment-508861891 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 32 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 2 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 73 | Maven dependency ordering for branch | | +1 | mvninstall | 487 | trunk passed | | +1 | compile | 249 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 761 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 170 | trunk passed | | 0 | spotbugs | 318 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 516 | trunk passed | | -0 | patch | 380 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 46 | Maven dependency ordering for patch | | +1 | mvninstall | 451 | the patch passed | | +1 | compile | 281 | the patch passed | | +1 | cc | 281 | the patch passed | | +1 | javac | 281 | the patch passed | | +1 | checkstyle | 91 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | -1 | whitespace | 0 | The patch has 12 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | shadedclient | 694 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 174 | the patch passed | | +1 | findbugs | 530 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 250 | hadoop-hdds in the patch passed. | | -1 | unit | 1214 | hadoop-ozone in the patch failed. | | +1 | asflicense | 56 | The patch does not generate ASF License warnings. | | | | 6427 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.security.acl.TestOzoneNativeAuthorizer | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-973/11/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/973 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc shellcheck shelldocs | | uname | Linux 15029d967599 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9c90729 | | Default Java | 1.8.0_212 | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/11/artifact/out/whitespace-eol.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/11/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/11/testReport/ | | Max. process+thread count | 4959 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/client hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozonefs hadoop-ozone/tools U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/11/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and
[jira] [Commented] (HDFS-14630) Configuration.getTimeDurationHelper() should not log time unit warning in info log.
[
https://issues.apache.org/jira/browse/HDFS-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879510#comment-16879510
]
Hadoop QA commented on HDFS-14630:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
52s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
1s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}250m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14630 |
| JIRA Patch URL |
https://issues.a
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[ https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=272725&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272725 ] ASF GitHub Bot logged work on HDDS-1550: Author: ASF GitHub Bot Created on: 05/Jul/19 19:23 Start Date: 05/Jul/19 19:23 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1050: HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1050#issuecomment-508844432 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 71 | Maven dependency ordering for branch | | +1 | mvninstall | 526 | trunk passed | | +1 | compile | 268 | trunk passed | | +1 | checkstyle | 82 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 892 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 167 | trunk passed | | 0 | spotbugs | 314 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 507 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 34 | Maven dependency ordering for patch | | +1 | mvninstall | 437 | the patch passed | | +1 | compile | 252 | the patch passed | | +1 | javac | 252 | the patch passed | | +1 | checkstyle | 73 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 674 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 161 | the patch passed | | +1 | findbugs | 598 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 264 | hadoop-hdds in the patch passed. | | -1 | unit | 1948 | hadoop-ozone in the patch failed. | | +1 | asflicense | 56 | The patch does not generate ASF License warnings. | | | | 7221 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.scm.node.TestQueryNode | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1050/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1050 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux db574bf4cd12 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9c90729 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1050/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1050/3/testReport/ | | Max. process+thread count | 5344 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1050/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272725) Time Spent: 3h 10m (was: 3h) > MiniOzoneCluster is not shutting down
[jira] [Work logged] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access
[ https://issues.apache.org/jira/browse/HDDS-1611?focusedWorklogId=272721&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272721 ] ASF GitHub Bot logged work on HDDS-1611: Author: ASF GitHub Bot Created on: 05/Jul/19 18:51 Start Date: 05/Jul/19 18:51 Worklog Time Spent: 10m Work Description: ajayydv commented on pull request #973: HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/973#discussion_r300772241 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java ## @@ -118,6 +118,10 @@ * */ public static final String OZONE_ADMINISTRATORS = "ozone.administrators"; + /** + * Make every user an admin. Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272721) Time Spent: 5.5h (was: 5h 20m) > Evaluate ACL on volume bucket key and prefix to authorize access > - > > Key: HDDS-1611 > URL: https://issues.apache.org/jira/browse/HDDS-1611 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access
[
https://issues.apache.org/jira/browse/HDDS-1611?focusedWorklogId=272720&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272720
]
ASF GitHub Bot logged work on HDDS-1611:
Author: ASF GitHub Bot
Created on: 05/Jul/19 18:50
Start Date: 05/Jul/19 18:50
Worklog Time Spent: 10m
Work Description: ajayydv commented on pull request #973: HDDS-1611.
Evaluate ACL on volume bucket key and prefix to authorize access. Contributed
by Ajay Kumar.
URL: https://github.com/apache/hadoop/pull/973#discussion_r300771920
##
File path:
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/TestOzoneAcls.java
##
@@ -103,8 +103,8 @@ public void testAclParse() {
testMatrix.put(" world::rw", Boolean.TRUE);
testMatrix.put(" world::a", Boolean.TRUE);
-testMatrix.put(" world:bilbo:w", Boolean.FALSE);
-testMatrix.put(" world:bilbo:rw", Boolean.FALSE);
+testMatrix.put(" world:bilbo:w", Boolean.TRUE);
+testMatrix.put(" world:bilbo:rw", Boolean.TRUE);
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272720)
Time Spent: 5h 20m (was: 5h 10m)
> Evaluate ACL on volume bucket key and prefix to authorize access
> -
>
> Key: HDDS-1611
> URL: https://issues.apache.org/jira/browse/HDDS-1611
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1769) Update Chukwa dependency
Eric Yang created HDDS-1769: --- Summary: Update Chukwa dependency Key: HDDS-1769 URL: https://issues.apache.org/jira/browse/HDDS-1769 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Eric Yang Hadoop, HBase, and ZooKeeper have released new version and older version are no longer available from download site. Chukwa dependencies need to be updated to stay current. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1769) Update Chukwa dependency
[ https://issues.apache.org/jira/browse/HDDS-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HDDS-1769. - Resolution: Invalid Sorry, filed under wrong project. > Update Chukwa dependency > > > Key: HDDS-1769 > URL: https://issues.apache.org/jira/browse/HDDS-1769 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Eric Yang >Priority: Major > > Hadoop, HBase, and ZooKeeper have released new version and older version are > no longer available from download site. Chukwa dependencies need to be > updated to stay current. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access
[
https://issues.apache.org/jira/browse/HDDS-1611?focusedWorklogId=272714&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272714
]
ASF GitHub Bot logged work on HDDS-1611:
Author: ASF GitHub Bot
Created on: 05/Jul/19 18:29
Start Date: 05/Jul/19 18:29
Worklog Time Spent: 10m
Work Description: ajayydv commented on pull request #973: HDDS-1611.
Evaluate ACL on volume bucket key and prefix to authorize access. Contributed
by Ajay Kumar.
URL: https://github.com/apache/hadoop/pull/973#discussion_r300767949
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -2276,11 +2325,21 @@ public void commitKey(OmKeyArgs args, long clientID)
@Override
public OmKeyLocationInfo allocateBlock(OmKeyArgs args, long clientID,
- ExcludeList excludeList)
- throws IOException {
+ ExcludeList excludeList) throws IOException {
if(isAclEnabled) {
- checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ try {
+checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.WRITE,
Review comment:
Created HDDS-1768 to track it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 272714)
Time Spent: 5h 10m (was: 5h)
> Evaluate ACL on volume bucket key and prefix to authorize access
> -
>
> Key: HDDS-1611
> URL: https://issues.apache.org/jira/browse/HDDS-1611
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1768) Audit permission failures from authorizer
Ajay Kumar created HDDS-1768: Summary: Audit permission failures from authorizer Key: HDDS-1768 URL: https://issues.apache.org/jira/browse/HDDS-1768 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Ajay Kumar Audit permission failures from authorizer -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1544) Support default Acls for volume, bucket, keys and prefix
[ https://issues.apache.org/jira/browse/HDDS-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879468#comment-16879468 ] Ajay Kumar commented on HDDS-1544: -- [~anu], [~xyao] attaching draft patch for review. Can't be submitted before [HDDS-1611]. > Support default Acls for volume, bucket, keys and prefix > > > Key: HDDS-1544 > URL: https://issues.apache.org/jira/browse/HDDS-1544 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1544.00.patch > > > Add dAcls for volume, bucket, keys and prefix -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1544) Support default Acls for volume, bucket, keys and prefix
[ https://issues.apache.org/jira/browse/HDDS-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar reassigned HDDS-1544: Assignee: Ajay Kumar > Support default Acls for volume, bucket, keys and prefix > > > Key: HDDS-1544 > URL: https://issues.apache.org/jira/browse/HDDS-1544 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1544.00.patch > > > Add dAcls for volume, bucket, keys and prefix -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1544) Support default Acls for volume, bucket, keys and prefix
[ https://issues.apache.org/jira/browse/HDDS-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1544: - Attachment: HDDS-1544.00.patch > Support default Acls for volume, bucket, keys and prefix > > > Key: HDDS-1544 > URL: https://issues.apache.org/jira/browse/HDDS-1544 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Priority: Major > Attachments: HDDS-1544.00.patch > > > Add dAcls for volume, bucket, keys and prefix -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14257) NPE when given the Invalid path to create target dir
[
https://issues.apache.org/jira/browse/HDFS-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879461#comment-16879461
]
Íñigo Goiri commented on HDFS-14257:
This should include a unit test triggering the NPE before the fix in Mkdir and
capturing the new exception now.
> NPE when given the Invalid path to create target dir
>
>
> Key: HDFS-14257
> URL: https://issues.apache.org/jira/browse/HDFS-14257
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Harshakiran Reddy
>Assignee: hemanthboyina
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14257.patch
>
>
> bin> ./hdfs dfs -mkdir hdfs://{color:red}hacluster2 /hacluster1{color}dest2/
> {noformat}
> -mkdir: Fatal internal error
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2714)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3229)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1618)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1742)
> at
> org.apache.hadoop.fs.shell.Mkdir.processNonexistentPath(Mkdir.java:74)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:287)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
> bin>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14257) NPE when given the Invalid path to create target dir
[
https://issues.apache.org/jira/browse/HDFS-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879456#comment-16879456
]
Hadoop QA commented on HDFS-14257:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
57s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14257 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12973801/HDFS-14257.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 05a63f56f813 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c9c3429 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27155/testReport/ |
| Max. process+thread count | 1389 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27155/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> NPE when given the Invalid path
[jira] [Comment Edited] (HDFS-14593) RBF: RouterAdmin should be able to remove expired routers from Routers Information
[ https://issues.apache.org/jira/browse/HDFS-14593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879446#comment-16879446 ] Íñigo Goiri edited comment on HDFS-14593 at 7/5/19 5:40 PM: Thanks [~tasanuma], some comments: * I think we can make the method in BaseRecord return -1 by default so we don't need to tweak all the subclasses. * BaseRecord#shouldBeDeleted() with the summations looks a little weird to read, can we expand the if condition and the summation to make it more readable? * We may want to keep supporting 0 for expiration as in MountTable#getExpirationMs(). * Having to sleep 11 seconds for the expiration is a little too much; can we make this tighter? * The ZK unit tests looks related. was (Author: elgoiri): * I think we can make the method in BaseRecord return -1 by default so we don't need to tweak all the subclasses. * BaseRecord#shouldBeDeleted() with the summations looks a little weird to read, can we expand the if condition and the summation to make it more readable? * We may want to keep supporting 0 for expiration as in MountTable#getExpirationMs(). * Having to sleep 11 seconds for the expiration is a little too much; can we make this tighter? > RBF: RouterAdmin should be able to remove expired routers from Routers > Information > -- > > Key: HDFS-14593 > URL: https://issues.apache.org/jira/browse/HDFS-14593 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Attachments: HDFS-14593.001.patch, HDFS-14593.002.patch, > HDFS-14593.003.patch > > > Currently, any router seems to exist in the Router Information eternally. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14593) RBF: RouterAdmin should be able to remove expired routers from Routers Information
[ https://issues.apache.org/jira/browse/HDFS-14593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879446#comment-16879446 ] Íñigo Goiri commented on HDFS-14593: * I think we can make the method in BaseRecord return -1 by default so we don't need to tweak all the subclasses. * BaseRecord#shouldBeDeleted() with the summations looks a little weird to read, can we expand the if condition and the summation to make it more readable? * We may want to keep supporting 0 for expiration as in MountTable#getExpirationMs(). * Having to sleep 11 seconds for the expiration is a little too much; can we make this tighter? > RBF: RouterAdmin should be able to remove expired routers from Routers > Information > -- > > Key: HDFS-14593 > URL: https://issues.apache.org/jira/browse/HDFS-14593 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Attachments: HDFS-14593.001.patch, HDFS-14593.002.patch, > HDFS-14593.003.patch > > > Currently, any router seems to exist in the Router Information eternally. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14625) Make DefaultAuditLogger class in FSnamesystem to Abstract
[
https://issues.apache.org/jira/browse/HDFS-14625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879443#comment-16879443
]
Íñigo Goiri commented on HDFS-14625:
As we are moving it to a separate class, I think we can use this chance to make
some cleanup in the new DefaultAuditLogger:
* Make the fields protected and provide getters if needed. Even private if that
helps the OO pattern.
* We can remove the type in the constructor as {{public Set debugCmdSet
= new HashSet<>();}}.
* Extra space in {{static class FSNamesystemAuditLogger extends
DefaultAuditLogger {}}.
* Add short javadocs to the interesting fields:
{code}
/** List of commands to provide debug messages. */
public Set debugCmdSet = new HashSet<>();
{code}
> Make DefaultAuditLogger class in FSnamesystem to Abstract
> --
>
> Key: HDFS-14625
> URL: https://issues.apache.org/jira/browse/HDFS-14625
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14625 (1).patch, HDFS-14625.patch
>
>
> As per +HDFS-13270+ Audit logger for Router , we can make DefaultAuditLogger
> in FSnamesystem to be Abstract and common
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14624) When decommissioning a node, log remaining blocks to replicate periodically
[
https://issues.apache.org/jira/browse/HDFS-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879440#comment-16879440
]
Íñigo Goiri commented on HDFS-14624:
+1 on [^HDFS-14624.002.patch].
> When decommissioning a node, log remaining blocks to replicate periodically
> ---
>
> Key: HDFS-14624
> URL: https://issues.apache.org/jira/browse/HDFS-14624
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-14624.001.patch, HDFS-14624.002.patch
>
>
> When a node is marked for decommission, there is a monitor thread which runs
> every 30 seconds by default, and checks if the node still has pending blocks
> to be replicated before the node can complete replication.
> There are two existing debug level messages logged in the monitor thread,
> DatanodeAdminManager$Monitor.check(), which log the correct information
> already, first as the pending blocks are replicated:
> {code:java}
> LOG.debug("Node {} still has {} blocks to replicate "
> + "before it is a candidate to finish {}.",
> dn, blocks.size(), dn.getAdminState());{code}
> And then after the initial set of blocks has completed and a rescan happens:
> {code:java}
> LOG.debug("Node {} {} healthy."
> + " It needs to replicate {} more blocks."
> + " {} is still in progress.", dn,
> isHealthy ? "is": "isn't", blocks.size(), dn.getAdminState());{code}
> I would like to propose moving these messages to INFO level so it is easier
> to monitor decommission progress over time from the Namenode log.
> Based on the default settings, this would result in at most 1 log message per
> node being decommissioned every 30 seconds. The reason this is at the most,
> is because the monitor thread stops after checking after 500K blocks and
> therefore in practice it could be as little as 1 log message per 30 seconds,
> even if many DNs are being decommissioned at the same time.
> Note that the namenode webUI does display the above information, but having
> this in the NN logs would allow progress to be tracked more easily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13694) Making md5 computing being in parallel with image loading
[ https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13694: --- Attachment: HDFS-13694-007.patch > Making md5 computing being in parallel with image loading > - > > Key: HDFS-13694 > URL: https://issues.apache.org/jira/browse/HDFS-13694 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-13694-001.patch, HDFS-13694-002.patch, > HDFS-13694-003.patch, HDFS-13694-004.patch, HDFS-13694-005.patch, > HDFS-13694-006.patch, HDFS-13694-007.patch > > > During namenode image loading, it firstly compute the md5 and then load the > image. Actually these two steps can be in parallel. > Test this patch against a fsimage of a 70PB 2.4 cluster (200million files > and 300million blocks), the image loading time be reduced from 1210 seconds > to 1105 seconds.So it can reduce up to about 10% of time. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13694) Making md5 computing being in parallel with image loading
[ https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13694: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) > Making md5 computing being in parallel with image loading > - > > Key: HDFS-13694 > URL: https://issues.apache.org/jira/browse/HDFS-13694 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-13694-001.patch, HDFS-13694-002.patch, > HDFS-13694-003.patch, HDFS-13694-004.patch, HDFS-13694-005.patch, > HDFS-13694-006.patch > > > During namenode image loading, it firstly compute the md5 and then load the > image. Actually these two steps can be in parallel. > Test this patch against a fsimage of a 70PB 2.4 cluster (200million files > and 300million blocks), the image loading time be reduced from 1210 seconds > to 1105 seconds.So it can reduce up to about 10% of time. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13694) Making md5 computing being in parallel with image loading
[ https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879438#comment-16879438 ] Íñigo Goiri commented on HDFS-13694: Merged the pull request. Thanks [~leosun08] for working on this and [~hexiaoqiao] for the reviews. > Making md5 computing being in parallel with image loading > - > > Key: HDFS-13694 > URL: https://issues.apache.org/jira/browse/HDFS-13694 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13694-001.patch, HDFS-13694-002.patch, > HDFS-13694-003.patch, HDFS-13694-004.patch, HDFS-13694-005.patch, > HDFS-13694-006.patch > > > During namenode image loading, it firstly compute the md5 and then load the > image. Actually these two steps can be in parallel. > Test this patch against a fsimage of a 70PB 2.4 cluster (200million files > and 300million blocks), the image loading time be reduced from 1210 seconds > to 1105 seconds.So it can reduce up to about 10% of time. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13694) Making md5 computing being in parallel with image loading
[ https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879435#comment-16879435 ] Hudson commented on HDFS-13694: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16866 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16866/]) HDFS-13694. Making md5 computing being in parallel with image loading. (elgoiri: rev 9c907294860a48f0d1676a31bda15795b7a6771a) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java > Making md5 computing being in parallel with image loading > - > > Key: HDFS-13694 > URL: https://issues.apache.org/jira/browse/HDFS-13694 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13694-001.patch, HDFS-13694-002.patch, > HDFS-13694-003.patch, HDFS-13694-004.patch, HDFS-13694-005.patch, > HDFS-13694-006.patch > > > During namenode image loading, it firstly compute the md5 and then load the > image. Actually these two steps can be in parallel. > Test this patch against a fsimage of a 70PB 2.4 cluster (200million files > and 300million blocks), the image loading time be reduced from 1210 seconds > to 1105 seconds.So it can reduce up to about 10% of time. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879434#comment-16879434
]
Íñigo Goiri commented on HDFS-12703:
[~sodonnell], yes we should definetely cover what happens with a node in that
weird state.
[~hexiaoqiao] feel free to follow up; it would be ideal to have a unit test for
this.
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch, HDFS-12703.002.patch,
> HDFS-12703.003.patch, HDFS-12703.004.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1550) MiniOzoneCluster is not shutting down all the threads during shutdown.
[ https://issues.apache.org/jira/browse/HDDS-1550?focusedWorklogId=272671&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272671 ] ASF GitHub Bot logged work on HDDS-1550: Author: ASF GitHub Bot Created on: 05/Jul/19 16:56 Start Date: 05/Jul/19 16:56 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1050: HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1050#issuecomment-508815511 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 12 | https://github.com/apache/hadoop/pull/1050 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1050 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1050/2/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272671) Time Spent: 3h (was: 2h 50m) > MiniOzoneCluster is not shutting down all the threads during shutdown. > -- > > Key: HDDS-1550 > URL: https://issues.apache.org/jira/browse/HDDS-1550 > Project: Hadoop Distributed Data Store > Issue Type: Test > Components: test >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > MiniOzoneCluster does not shutdown all the threads during shutdown. All the > threads must be shutdown to close the cluster correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1767) ContainerStateMachine should have its own executors for executing applyTransaction calls
Lokesh Jain created HDDS-1767: - Summary: ContainerStateMachine should have its own executors for executing applyTransaction calls Key: HDDS-1767 URL: https://issues.apache.org/jira/browse/HDDS-1767 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Lokesh Jain Assignee: Lokesh Jain Currently ContainerStateMachine uses the executors provided by XceiverServerRatis for executing applyTransaction calls. This would result in two or more ContainerStateMachine to share the same set of executors. Delay or load in one ContainerStateMachine would adversely affect the performance of other state machines in such a case. It is better to have separate set of executors for each ContainerStateMachine. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1766) ContainerStateMachine is unable to increment lastAppliedIndex
Lokesh Jain created HDDS-1766: - Summary: ContainerStateMachine is unable to increment lastAppliedIndex Key: HDDS-1766 URL: https://issues.apache.org/jira/browse/HDDS-1766 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Lokesh Jain ContainerStateMachine#updateLastApplied currently updates the lastAppliedTermIndex using applyTransactionCompletionMap. There are null entries in the applyTransactionCompletionMap causing the lastAppliedIndex to not be incremented. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14257) NPE when given the Invalid path to create target dir
[
https://issues.apache.org/jira/browse/HDFS-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina updated HDFS-14257:
-
Attachment: HDFS-14257.patch
Status: Patch Available (was: Open)
> NPE when given the Invalid path to create target dir
>
>
> Key: HDFS-14257
> URL: https://issues.apache.org/jira/browse/HDFS-14257
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Harshakiran Reddy
>Assignee: hemanthboyina
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14257.patch
>
>
> bin> ./hdfs dfs -mkdir hdfs://{color:red}hacluster2 /hacluster1{color}dest2/
> {noformat}
> -mkdir: Fatal internal error
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2714)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3229)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1618)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1742)
> at
> org.apache.hadoop.fs.shell.Mkdir.processNonexistentPath(Mkdir.java:74)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:287)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
> bin>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14630) Configuration.getTimeDurationHelper() should not log time unit warning in info log.
[
https://issues.apache.org/jira/browse/HDFS-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina updated HDFS-14630:
-
Attachment: HDFS-14630.patch
Status: Patch Available (was: Open)
> Configuration.getTimeDurationHelper() should not log time unit warning in
> info log.
> ---
>
> Key: HDFS-14630
> URL: https://issues.apache.org/jira/browse/HDFS-14630
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: hemanthboyina
>Priority: Minor
> Attachments: HDFS-14630.patch
>
>
> To solve [HDFS-12920|https://issues.apache.org/jira/browse/HDFS-12920] issue
> we configured "dfs.client.datanode-restart.timeout" without time unit. No log
> file is full of
> {noformat}
> 2019-06-22 20:13:14,605 | INFO | pool-12-thread-1 | No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> org.apache.hadoop.conf.Configuration.logDeprecation(Configuration.java:1409){noformat}
> No need to log this, just give the behavior in property description.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1765) destroyPipeline scheduled from finalizeAndDestroyPipeline fails for short dead node interval
[
https://issues.apache.org/jira/browse/HDDS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879243#comment-16879243
]
Supratim Deka commented on HDDS-1765:
-
similar symptom but not the same problem. Linking for reference.
> destroyPipeline scheduled from finalizeAndDestroyPipeline fails for short
> dead node interval
>
>
> Key: HDDS-1765
> URL: https://issues.apache.org/jira/browse/HDDS-1765
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Supratim Deka
>Priority: Major
>
> This happens when
> OZONE_SCM_PIPELINE_DESTROY_TIMEOUT exceeds the value of
> OZONE_SCM_DEADNODE_INTERVAL. This is the case for start-chaos.sh
> When a Datanode is shutdown, SCM Stale node handler calls
> finalizeAndDestroyPipeline() which schedules destroyPipeline() operation with
> a delay
> of OZONE_SCM_PIPELINE_DESTROY_TIMEOUT. By the time this gets scheduled, dead
> node handler would have destroyed the pipeline.
>
> {code:java}
> 2019-07-05 14:45:16,358 INFO pipeline.SCMPipelineManager
> (SCMPipelineManager.java:finalizeAndDestroyPipeline(307)) - destroying
> pipeline:Pipeline[ Id: ef60537a-0a82-4fea-a574-109c881fa140, Nodes:
> 7947bf32-faaa-4b34-bf1e-2752a929938c{ip: 192.168.1.6, host: 192.168.1.6,
> networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE,
> State:CLOSED]
> 2019-07-05 14:45:16,363 INFO pipeline.PipelineStateManager
> (PipelineStateManager.java:removePipeline(108)) - Pipeline Pipeline[ Id:
> ef60537a-0a82-4fea-a574-109c881fa140, Nodes:
> 7947bf32-faaa-4b34-bf1e-2752a929938c{ip: 192.168.1.6, host: 192.168.1.6,
> networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE,
> State:CLOSED] removed from db
> ...
> 2019-07-05 14:46:12,400 WARN pipeline.RatisPipelineUtils
> (RatisPipelineUtils.java:destroyPipeline(66)) - Pipeline destroy failed for
> pipeline=PipelineID=ef60537a-0a82-4fea-a574-109c881fa140
> dn=7947bf32-faaa-4b34-bf1e-2752a929938c\{ip: 192.168.1.6, host: 192.168.1.6,
> networkLocation: /default-rack, certSerialId: null}
> 2019-07-05 14:46:12,401 ERROR pipeline.SCMPipelineManager
> (Scheduler.java:lambda$schedule$1(70)) - Destroy pipeline failed for
> pipeline:Pipeline[ Id: ef60537a-0a82-4fea-a574-109c881fa140, Nodes:
> 7947bf32-faaa-4b34-bf1e-2752a929938c\{ip: 192.168.1.6, host: 192.168.1.6,
> networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE,
> State:OPEN]
> org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException:
> PipelineID=ef60537a-0a82-4fea-a574-109c881fa140 not found
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:132)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.removePipeline(PipelineStateMap.java:322)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.removePipeline(PipelineStateManager.java:107)
> at
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(SCMPipelineManager.java:401)
> at
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.destroyPipeline(SCMPipelineManager.java:387)
> at
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.lambda$finalizeAndDestroyPipeline$0(SCMPipelineManager.java:321)
> at
> org.apache.hadoop.utils.Scheduler.lambda$schedule$1(Scheduler.java:68)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1765) destroyPipeline scheduled from finalizeAndDestroyPipeline fails for short dead node interval
Supratim Deka created HDDS-1765:
---
Summary: destroyPipeline scheduled from finalizeAndDestroyPipeline
fails for short dead node interval
Key: HDDS-1765
URL: https://issues.apache.org/jira/browse/HDDS-1765
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: SCM
Reporter: Supratim Deka
This happens when
OZONE_SCM_PIPELINE_DESTROY_TIMEOUT exceeds the value of
OZONE_SCM_DEADNODE_INTERVAL. This is the case for start-chaos.sh
When a Datanode is shutdown, SCM Stale node handler calls
finalizeAndDestroyPipeline() which schedules destroyPipeline() operation with a
delay
of OZONE_SCM_PIPELINE_DESTROY_TIMEOUT. By the time this gets scheduled, dead
node handler would have destroyed the pipeline.
{code:java}
2019-07-05 14:45:16,358 INFO pipeline.SCMPipelineManager
(SCMPipelineManager.java:finalizeAndDestroyPipeline(307)) - destroying
pipeline:Pipeline[ Id: ef60537a-0a82-4fea-a574-109c881fa140, Nodes:
7947bf32-faaa-4b34-bf1e-2752a929938c{ip: 192.168.1.6, host: 192.168.1.6,
networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE,
State:CLOSED]
2019-07-05 14:45:16,363 INFO pipeline.PipelineStateManager
(PipelineStateManager.java:removePipeline(108)) - Pipeline Pipeline[ Id:
ef60537a-0a82-4fea-a574-109c881fa140, Nodes:
7947bf32-faaa-4b34-bf1e-2752a929938c{ip: 192.168.1.6, host: 192.168.1.6,
networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE,
State:CLOSED] removed from db
...
2019-07-05 14:46:12,400 WARN pipeline.RatisPipelineUtils
(RatisPipelineUtils.java:destroyPipeline(66)) - Pipeline destroy failed for
pipeline=PipelineID=ef60537a-0a82-4fea-a574-109c881fa140
dn=7947bf32-faaa-4b34-bf1e-2752a929938c\{ip: 192.168.1.6, host: 192.168.1.6,
networkLocation: /default-rack, certSerialId: null}
2019-07-05 14:46:12,401 ERROR pipeline.SCMPipelineManager
(Scheduler.java:lambda$schedule$1(70)) - Destroy pipeline failed for
pipeline:Pipeline[ Id: ef60537a-0a82-4fea-a574-109c881fa140, Nodes:
7947bf32-faaa-4b34-bf1e-2752a929938c\{ip: 192.168.1.6, host: 192.168.1.6,
networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE,
State:OPEN]
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException:
PipelineID=ef60537a-0a82-4fea-a574-109c881fa140 not found
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:132)
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.removePipeline(PipelineStateMap.java:322)
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.removePipeline(PipelineStateManager.java:107)
at
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(SCMPipelineManager.java:401)
at
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.destroyPipeline(SCMPipelineManager.java:387)
at
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.lambda$finalizeAndDestroyPipeline$0(SCMPipelineManager.java:321)
at
org.apache.hadoop.utils.Scheduler.lambda$schedule$1(Scheduler.java:68)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1728) Add metrics for leader's latency in ContainerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1728?focusedWorklogId=272556&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272556 ] ASF GitHub Bot logged work on HDDS-1728: Author: ASF GitHub Bot Created on: 05/Jul/19 12:21 Start Date: 05/Jul/19 12:21 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1022: HDDS-1728. Add metrics for leader's latency in ContainerStateMachine. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1022#issuecomment-508740993 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 8 | https://github.com/apache/hadoop/pull/1022 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1022 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1022/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272556) Time Spent: 50m (was: 40m) > Add metrics for leader's latency in ContainerStateMachine > - > > Key: HDDS-1728 > URL: https://issues.apache.org/jira/browse/HDDS-1728 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > This jira proposes to add metrics around leaders round trip reply to ratis > client. This will be done via startTransaction api -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879187#comment-16879187
]
He Xiaoqiao edited comment on HDFS-12703 at 7/5/19 12:20 PM:
-
Thanks [~elgoiri], [~sodonnell] for your comment, I would like to follow up
this issue and submit patch later based on above suggestions if [~xuel1] has no
time. Thanks again.
was (Author: hexiaoqiao):
Thanks [~elgoiri], [~sodonnell] for your comment, I would like to follow up
this issue and submit patch based on above suggestions if [~xuel1] has no time.
Thanks again.
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch, HDFS-12703.002.patch,
> HDFS-12703.003.patch, HDFS-12703.004.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879187#comment-16879187
]
He Xiaoqiao commented on HDFS-12703:
Thanks [~elgoiri], [~sodonnell] for your comment, I would like to follow up
this issue and submit patch based on above suggestions if [~xuel1] has no time.
Thanks again.
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch, HDFS-12703.002.patch,
> HDFS-12703.003.patch, HDFS-12703.004.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1729) Ozone Client should timeout if the put block futures are taking a long time
[ https://issues.apache.org/jira/browse/HDDS-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1729: Resolution: Not A Problem Status: Resolved (was: Patch Available) As [~shashikant] commented on the pull request, this is not an issue. > Ozone Client should timeout if the put block futures are taking a long time > --- > > Key: HDDS-1729 > URL: https://issues.apache.org/jira/browse/HDDS-1729 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Ozone client currently enqueues a put future to the future map, However if > the pipeline is slow, the client does not timeout and wait for the future to > finish. For reasonable latency in the system, the client should timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1729) Ozone Client should timeout if the put block futures are taking a long time
[ https://issues.apache.org/jira/browse/HDDS-1729?focusedWorklogId=272549&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272549 ] ASF GitHub Bot logged work on HDDS-1729: Author: ASF GitHub Bot Created on: 05/Jul/19 12:08 Start Date: 05/Jul/19 12:08 Worklog Time Spent: 10m Work Description: mukul1987 commented on issue #1024: HDDS-1729. Ozone Client should timeout if the put block futures are taking a long time. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1024#issuecomment-508737983 Thanks for the review @bshashikant, yes the patch is not required. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272549) Time Spent: 40m (was: 0.5h) > Ozone Client should timeout if the put block futures are taking a long time > --- > > Key: HDDS-1729 > URL: https://issues.apache.org/jira/browse/HDDS-1729 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Ozone client currently enqueues a put future to the future map, However if > the pipeline is slow, the client does not timeout and wait for the future to > finish. For reasonable latency in the system, the client should timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1729) Ozone Client should timeout if the put block futures are taking a long time
[ https://issues.apache.org/jira/browse/HDDS-1729?focusedWorklogId=272550&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272550 ] ASF GitHub Bot logged work on HDDS-1729: Author: ASF GitHub Bot Created on: 05/Jul/19 12:08 Start Date: 05/Jul/19 12:08 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #1024: HDDS-1729. Ozone Client should timeout if the put block futures are taking a long time. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1024 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272550) Time Spent: 50m (was: 40m) > Ozone Client should timeout if the put block futures are taking a long time > --- > > Key: HDDS-1729 > URL: https://issues.apache.org/jira/browse/HDDS-1729 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Ozone client currently enqueues a put future to the future map, However if > the pipeline is slow, the client does not timeout and wait for the future to > finish. For reasonable latency in the system, the client should timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[
https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879179#comment-16879179
]
Hadoop QA commented on HDFS-14586:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 5 new + 44 unchanged - 0 fixed = 49 total (was 44) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 24s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestTrash |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14586 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12973762/HDFS-14586.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3a0d8e96c98a 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 928edb2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/27153/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27153/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27153/testReport/ |
| Max. process+thread count | 1472 (vs. ulimit of 1) |
| mod
[jira] [Comment Edited] (HDDS-1758) Add replication and key deletion tests to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879165#comment-16879165 ] Mukul Kumar Singh edited comment on HDDS-1758 at 7/5/19 11:40 AM: -- Thanks for the review [~shashikant]. I have committed this. was (Author: msingh): Thanks for the review [~msingh]. I have committed this. > Add replication and key deletion tests to MiniOzoneChaosCluster > --- > > Key: HDDS-1758 > URL: https://issues.apache.org/jira/browse/HDDS-1758 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This jira adds capability for deleting keys and also to test Replication > Manager code in MiniOzoneChaosCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1758) Add replication and key deletion tests to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1758: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review [~msingh]. I have committed this. > Add replication and key deletion tests to MiniOzoneChaosCluster > --- > > Key: HDDS-1758 > URL: https://issues.apache.org/jira/browse/HDDS-1758 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This jira adds capability for deleting keys and also to test Replication > Manager code in MiniOzoneChaosCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1758) Add replication and key deletion tests to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879162#comment-16879162 ] Hudson commented on HDDS-1758: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16865 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16865/]) HDDS-1758. Add replication and key deletion tests to (github: rev c9c3429a16d3b380f4a9ff90a185151360602319) * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneChaosCluster.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java > Add replication and key deletion tests to MiniOzoneChaosCluster > --- > > Key: HDDS-1758 > URL: https://issues.apache.org/jira/browse/HDDS-1758 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This jira adds capability for deleting keys and also to test Replication > Manager code in MiniOzoneChaosCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1758) Add replication and key deletion tests to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-1758?focusedWorklogId=272542&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272542 ] ASF GitHub Bot logged work on HDDS-1758: Author: ASF GitHub Bot Created on: 05/Jul/19 11:34 Start Date: 05/Jul/19 11:34 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #1049: HDDS-1758. Add replication and key deletion tests to MiniOzoneChaosCluster. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1049 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 272542) Time Spent: 40m (was: 0.5h) > Add replication and key deletion tests to MiniOzoneChaosCluster > --- > > Key: HDDS-1758 > URL: https://issues.apache.org/jira/browse/HDDS-1758 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This jira adds capability for deleting keys and also to test Replication > Manager code in MiniOzoneChaosCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879128#comment-16879128 ] hu yongfa commented on HDFS-14586: -- junit tests [^HDFS-14586.003.patch] > Trash missing delete the folder which near timeout checkpoint > - > > Key: HDFS-14586 > URL: https://issues.apache.org/jira/browse/HDFS-14586 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hu yongfa >Assignee: hu yongfa >Priority: Major > Attachments: HDFS-14586.001.patch, HDFS-14586.002.patch, > HDFS-14586.003.patch > > > when trash timeout checkpoint coming, trash will delete the old folder first, > then create a new checkpoint folder. > as the delete action may spend a long time, such as 2 minutes, so the new > checkpoint folder created late. > at the next trash timeout checkpoint, trash will skip delete the new > checkpoint folder, because the new checkpoint folder is > less than a checkpoint interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu yongfa updated HDFS-14586: - Attachment: HDFS-14586.003.patch > Trash missing delete the folder which near timeout checkpoint > - > > Key: HDFS-14586 > URL: https://issues.apache.org/jira/browse/HDFS-14586 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hu yongfa >Assignee: hu yongfa >Priority: Major > Attachments: HDFS-14586.001.patch, HDFS-14586.002.patch, > HDFS-14586.003.patch > > > when trash timeout checkpoint coming, trash will delete the old folder first, > then create a new checkpoint folder. > as the delete action may spend a long time, such as 2 minutes, so the new > checkpoint folder created late. > at the next trash timeout checkpoint, trash will skip delete the new > checkpoint folder, because the new checkpoint folder is > less than a checkpoint interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14361) SNN will always upload fsimage
[
https://issues.apache.org/jira/browse/HDFS-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879053#comment-16879053
]
hunshenshi commented on HDFS-14361:
---
[~starphin] today I review this code. I think you don't understand my point.
{code:java}
for (; i < uploads.size(); i++) {
Future upload = uploads.get(i);
try {
// TODO should there be some smarts here about retries nodes that are not
the active NN?
if (upload.get() == TransferFsImage.TransferResult.SUCCESS) {
success = true;
//avoid getting the rest of the results - we don't care since we had a
successful upload
break;
}
} catch (ExecutionException e) {
ioe = new IOException("Exception during image upload", e);
break;
} catch (InterruptedException e) {
ie = e;
break;
}
}
if (ie == null && ioe == null) {
//Update only when response from remote about success or
lastUploadTime = monotonicNow();
// we are primary if we successfully updated the ANN
this.isPrimaryCheckPointer = success;
}
{code}
If `upload.get`occure error, the error will be catch, then ioe or ie will not
null, so the code `this.isPrimaryCheckPointer = success` can't be executed,
*the isPrimaryCheckPointer is the value of last time.*
*Is it ok ?*
Let me know your point. Thanks.
> SNN will always upload fsimage
> --
>
> Key: HDFS-14361
> URL: https://issues.apache.org/jira/browse/HDFS-14361
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 3.2.0
>Reporter: hunshenshi
>Priority: Major
> Fix For: 3.2.0
>
>
> Related to -HDFS-12248.-
> {code:java}
> boolean sendRequest = isPrimaryCheckPointer
> || secsSinceLastUpload >= checkpointConf.getQuietPeriod();
> doCheckpoint(sendRequest);
> {code}
> If sendRequest is true, SNN will upload fsimage. But isPrimaryCheckPointer
> always is true,
> {code:java}
> if (ie == null && ioe == null) {
> //Update only when response from remote about success or
> lastUploadTime = monotonicNow();
> // we are primary if we successfully updated the ANN
> this.isPrimaryCheckPointer = success;
> }
> {code}
> isPrimaryCheckPointer should be outside the if condition.
> If the ANN update was not successful, then isPrimaryCheckPointer should be
> set to false.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14361) SNN will always upload fsimage
[
https://issues.apache.org/jira/browse/HDFS-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879053#comment-16879053
]
hunshenshi edited comment on HDFS-14361 at 7/5/19 8:05 AM:
---
[~starphin] today I review this code. I think you don't understand my point.
{code:java}
for (; i < uploads.size(); i++) {
Future upload = uploads.get(i);
try {
// TODO should there be some smarts here about retries nodes that are not
the active NN?
if (upload.get() == TransferFsImage.TransferResult.SUCCESS) {
success = true;
//avoid getting the rest of the results - we don't care since we had a
successful upload
break;
}
} catch (ExecutionException e) {
ioe = new IOException("Exception during image upload", e);
break;
} catch (InterruptedException e) {
ie = e;
break;
}
}
if (ie == null && ioe == null) {
//Update only when response from remote about success or
lastUploadTime = monotonicNow();
// we are primary if we successfully updated the ANN
this.isPrimaryCheckPointer = success;
}
{code}
If `upload.get`occure error, the error will be catch, then ioe or ie will not
null, so the code `this.isPrimaryCheckPointer = success` can't be executed,
*the isPrimaryCheckPointer is the value of last time.*
*Is it ok ?*
Let me know your point. Thanks.
was (Author: hunhun):
[~starphin] today I review this code. I think you don't understand my point.
{code:java}
for (; i < uploads.size(); i++) {
Future upload = uploads.get(i);
try {
// TODO should there be some smarts here about retries nodes that are not
the active NN?
if (upload.get() == TransferFsImage.TransferResult.SUCCESS) {
success = true;
//avoid getting the rest of the results - we don't care since we had a
successful upload
break;
}
} catch (ExecutionException e) {
ioe = new IOException("Exception during image upload", e);
break;
} catch (InterruptedException e) {
ie = e;
break;
}
}
if (ie == null && ioe == null) {
//Update only when response from remote about success or
lastUploadTime = monotonicNow();
// we are primary if we successfully updated the ANN
this.isPrimaryCheckPointer = success;
}
{code}
If `upload.get`occure error, the error will be catch, then ioe or ie will not
null, so the code `this.isPrimaryCheckPointer = success` can't be executed,
*the isPrimaryCheckPointer is the value of last time.*
*Is it ok ?*
Let me know your point. Thanks.
> SNN will always upload fsimage
> --
>
> Key: HDFS-14361
> URL: https://issues.apache.org/jira/browse/HDFS-14361
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 3.2.0
>Reporter: hunshenshi
>Priority: Major
> Fix For: 3.2.0
>
>
> Related to -HDFS-12248.-
> {code:java}
> boolean sendRequest = isPrimaryCheckPointer
> || secsSinceLastUpload >= checkpointConf.getQuietPeriod();
> doCheckpoint(sendRequest);
> {code}
> If sendRequest is true, SNN will upload fsimage. But isPrimaryCheckPointer
> always is true,
> {code:java}
> if (ie == null && ioe == null) {
> //Update only when response from remote about success or
> lastUploadTime = monotonicNow();
> // we are primary if we successfully updated the ANN
> this.isPrimaryCheckPointer = success;
> }
> {code}
> isPrimaryCheckPointer should be outside the if condition.
> If the ANN update was not successful, then isPrimaryCheckPointer should be
> set to false.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14257) NPE when given the Invalid path to create target dir
[
https://issues.apache.org/jira/browse/HDFS-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879013#comment-16879013
]
hemanthboyina commented on HDFS-14257:
--
added a null check for the path
now /bin # ./hdfs dfs -mkdir hdfs://hacluster1
{noformat}
-mkdir: Pathname from hdfs://hacluster1 is not a valid DFS filename.
Usage: hadoop fs [generic options]
[-appendToFile ... ]
[-cat [-ignoreCrc] ...]
[-checksum ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] [-d] [-t ] ... ]
[-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] ... ]
[-count [-q] [-h] [-v] [-t []] [-u] [-x] [-e] ...]
[-cp [-f] [-p | -p[topax]] [-d] ... ]
[-createSnapshot []]
[-deleteSnapshot ]
[-df [-h] [ ...]]
[-du [-s] [-h] [-v] [-x] ...]
[-expunge]
[-find ... ...]
[-get [-f] [-p] [-ignoreCrc] [-crc] ... ]
{noformat}
is it okay ?
> NPE when given the Invalid path to create target dir
>
>
> Key: HDFS-14257
> URL: https://issues.apache.org/jira/browse/HDFS-14257
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Harshakiran Reddy
>Assignee: hemanthboyina
>Priority: Major
> Labels: RBF
>
> bin> ./hdfs dfs -mkdir hdfs://{color:red}hacluster2 /hacluster1{color}dest2/
> {noformat}
> -mkdir: Fatal internal error
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2714)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3229)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1618)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1742)
> at
> org.apache.hadoop.fs.shell.Mkdir.processNonexistentPath(Mkdir.java:74)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:287)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
> bin>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
