[jira] [Commented] (HDFS-8533) Mismatch in displaying the "MissingBlock" count in fsck and in other metric reports
[
https://issues.apache.org/jira/browse/HDFS-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265015#comment-16265015
]
regis le bretonnic commented on HDFS-8533:
--
Hi
As fsck is ok, is there a way to identify the file or block raised as failed by
JMX for trying to fix it manually ?
> Mismatch in displaying the "MissingBlock" count in fsck and in other metric
> reports
> ---
>
> Key: HDFS-8533
> URL: https://issues.apache.org/jira/browse/HDFS-8533
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
>
> Number of DN = 2
> Step 1: Write a file with replication factor - 3 .
> Step 2: Corrupt a replica in DN1
> Step 3: DN2 is down.
> Missing Block count in report is as follows
> Fsck report: *0*
> Jmx, "dfsadmin -report" , UI, logs : *1*
> In fsck , only block whose replicas are all missed and not been corrupted are
> counted
> {code}
> if (totalReplicasPerBlock == 0 && !isCorrupt) {
> // If the block is corrupted, it means all its available replicas are
> // corrupted. We don't mark it as missing given these available
> replicas
> // might still be accessible as the block might be incorrectly marked
> as
> // corrupted by client machines.
> {code}
> While in other reports even if all the replicas are corrupted , block is been
> considered as missed.
> Please provide your thoughts : can we make missing block count consistent
> across all the reports same as implemented for fsck?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12833) In Distcp, Delete option not having the proper usage message.
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
usharani updated HDFS-12833:
Status: Patch Available (was: Open)
> In Distcp, Delete option not having the proper usage message.
> -
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Attachments: HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12833) In Distcp, Delete option not having the proper usage message.
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
usharani updated HDFS-12833:
Attachment: HDFS-12833.patch
Harshakiran Reddy thanks for reporting...
It make sense fix this issueplease review..
> In Distcp, Delete option not having the proper usage message.
> -
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Attachments: HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12840) Creating a replicated file in a EC zone does not correctly serialized in EditLogs
[
https://issues.apache.org/jira/browse/HDFS-12840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264935#comment-16264935
]
SammiChen commented on HDFS-12840:
--
Hi [~eddyxu], thanks for reporting and fix this. I'm not able to apply the
01.patch locally against latest trunk code while 00.patch is OK to apply. Can
you double check if the patch format is correct?
> Creating a replicated file in a EC zone does not correctly serialized in
> EditLogs
> -
>
> Key: HDFS-12840
> URL: https://issues.apache.org/jira/browse/HDFS-12840
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12840.00.patch, HDFS-12840.01.patch,
> HDFS-12840.reprod.patch, editsStored
>
>
> When create a replicated file in an existing EC zone, the edit logs does not
> differentiate it from an EC file. When {{FSEditLogLoader}} to replay edits,
> this file is treated as EC file, as a results, it crashes the NN because the
> blocks of this file are replicated, which does not match with {{INode}}.
> {noformat}
> ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered
> exception on operation AddBlockOp [path=/system/balancer.id,
> penultimateBlock=NULL, lastBlock=blk_1073743259_2455, RpcClientId=,
> RpcCallId=-2]
> java.lang.IllegalArgumentException: reportedBlock is not striped
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStriped.addStorage(BlockInfoStriped.java:118)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.addBlock(DatanodeStorageInfo.java:256)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlock(BlockManager.java:3141)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlockUnderConstruction(BlockManager.java:3068)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processAndHandleReportedBlock(BlockManager.java:3864)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processQueuedMessages(BlockManager.java:2916)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processQueuedMessagesForBlock(BlockManager.java:2903)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.addNewBlock(FSEditLogLoader.java:1069)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:532)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:293)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:427)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:380)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:397)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12851) Ozone: Upgrade to latest ratis build
[
https://issues.apache.org/jira/browse/HDFS-12851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264863#comment-16264863
]
Tsz Wo Nicholas Sze commented on HDFS-12851:
Just found that ContainerMetrics expects the latency in nanos (at least in the
parameter name).
{code}
//ContainerMetrics
public void incContainerOpsLatencies(ContainerProtos.Type type,
long latencyNanos) {
{code}
I think we should go deeper to see if there are more changes needed. Since it
seems quite involved, I suggest not doing this unrelated change here and file a
new JIRA.
> Ozone: Upgrade to latest ratis build
>
>
> Key: HDFS-12851
> URL: https://issues.apache.org/jira/browse/HDFS-12851
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12851-HDFS-7240.001.patch,
> HDFS-12851-HDFS-7240.002.patch, HDFS-12851-HDFS-7240.003.patch,
> HDFS-12851-HDFS-7240.004.patch
>
>
> Ratis build on Ozone needs to be upgraded to incorporate memory leak
> fixes(RATIS-135, RATIS-116), state machine optimization(RATIS-122) and Async
> client api (RATIS-113)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12851) Ozone: Upgrade to latest ratis build
[ https://issues.apache.org/jira/browse/HDFS-12851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264861#comment-16264861 ] Tsz Wo Nicholas Sze commented on HDFS-12851: > However Ratis does not has setter for some of the properties. I have raised > RATIS-154 for this. Thanks. +1 the 004 patch looks good. > Ozone: Upgrade to latest ratis build > > > Key: HDFS-12851 > URL: https://issues.apache.org/jira/browse/HDFS-12851 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12851-HDFS-7240.001.patch, > HDFS-12851-HDFS-7240.002.patch, HDFS-12851-HDFS-7240.003.patch, > HDFS-12851-HDFS-7240.004.patch > > > Ratis build on Ozone needs to be upgraded to incorporate memory leak > fixes(RATIS-135, RATIS-116), state machine optimization(RATIS-122) and Async > client api (RATIS-113) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12851) Ozone: Upgrade to latest ratis build
[ https://issues.apache.org/jira/browse/HDFS-12851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12851: - Attachment: HDFS-12851-HDFS-7240.004.patch v4 patch fixes the compilation error in the latch patch by rebasing. > Ozone: Upgrade to latest ratis build > > > Key: HDFS-12851 > URL: https://issues.apache.org/jira/browse/HDFS-12851 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12851-HDFS-7240.001.patch, > HDFS-12851-HDFS-7240.002.patch, HDFS-12851-HDFS-7240.003.patch, > HDFS-12851-HDFS-7240.004.patch > > > Ratis build on Ozone needs to be upgraded to incorporate memory leak > fixes(RATIS-135, RATIS-116), state machine optimization(RATIS-122) and Async > client api (RATIS-113) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12851) Ozone: Upgrade to latest ratis build
[
https://issues.apache.org/jira/browse/HDFS-12851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264571#comment-16264571
]
Hadoop QA commented on HDFS-12851:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
58s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
57s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 57s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
37s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
2s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
39s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Updated] (HDFS-12842) Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED
[ https://issues.apache.org/jira/browse/HDFS-12842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDFS-12842: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED > - > > Key: HDFS-12842 > URL: https://issues.apache.org/jira/browse/HDFS-12842 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar > Attachments: HDFS-12842-HDFS-7240.000.patch > > > This was initially fixed as part of HDFS-12796, but with HDFS-12756 the > change was overwritten. This jira is to bring back HDFS-12796 changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12842) Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED
[ https://issues.apache.org/jira/browse/HDFS-12842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264565#comment-16264565 ] Nanda kumar commented on HDFS-12842: Thanks [~anu] for the review, I have committed this to the feature branch. > Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED > - > > Key: HDFS-12842 > URL: https://issues.apache.org/jira/browse/HDFS-12842 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar > Attachments: HDFS-12842-HDFS-7240.000.patch > > > This was initially fixed as part of HDFS-12796, but with HDFS-12756 the > change was overwritten. This jira is to bring back HDFS-12796 changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12842) Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED
[ https://issues.apache.org/jira/browse/HDFS-12842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264534#comment-16264534 ] Nanda kumar commented on HDFS-12842: Test failures are not related, I will commit this shortly. > Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED > - > > Key: HDFS-12842 > URL: https://issues.apache.org/jira/browse/HDFS-12842 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar > Attachments: HDFS-12842-HDFS-7240.000.patch > > > This was initially fixed as part of HDFS-12796, but with HDFS-12756 the > change was overwritten. This jira is to bring back HDFS-12796 changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12851) Ozone: Upgrade to latest ratis build
[ https://issues.apache.org/jira/browse/HDFS-12851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264431#comment-16264431 ] Mukul Kumar Singh commented on HDFS-12851: -- Thanks for the review [~szetszwo], I have incorporated review comments for 1) and 2). However Ratis does not has setter for some of the properties. I have raised RATIS-154 for this. Will change settings here once RATIS-154 is fixed. > Ozone: Upgrade to latest ratis build > > > Key: HDFS-12851 > URL: https://issues.apache.org/jira/browse/HDFS-12851 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12851-HDFS-7240.001.patch, > HDFS-12851-HDFS-7240.002.patch, HDFS-12851-HDFS-7240.003.patch > > > Ratis build on Ozone needs to be upgraded to incorporate memory leak > fixes(RATIS-135, RATIS-116), state machine optimization(RATIS-122) and Async > client api (RATIS-113) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12851) Ozone: Upgrade to latest ratis build
[ https://issues.apache.org/jira/browse/HDFS-12851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12851: - Attachment: HDFS-12851-HDFS-7240.003.patch > Ozone: Upgrade to latest ratis build > > > Key: HDFS-12851 > URL: https://issues.apache.org/jira/browse/HDFS-12851 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12851-HDFS-7240.001.patch, > HDFS-12851-HDFS-7240.002.patch, HDFS-12851-HDFS-7240.003.patch > > > Ratis build on Ozone needs to be upgraded to incorporate memory leak > fixes(RATIS-135, RATIS-116), state machine optimization(RATIS-122) and Async > client api (RATIS-113) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12850) Ozone: TestContainerPersistence#testListContainer fails frequently due to timed out
[
https://issues.apache.org/jira/browse/HDFS-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264338#comment-16264338
]
Hadoop QA commented on HDFS-12850:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}133m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}199m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-12850 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899037/HDFS-12850-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4dff65b83d84 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / 3357ece |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22174/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-12842) Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED
[
https://issues.apache.org/jira/browse/HDFS-12842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264270#comment-16264270
]
Hadoop QA commented on HDFS-12842:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
23s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}144m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.qjournal.client.TestQJMWithFaults |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-12842 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898702/HDFS-12842-HDFS-7240.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5c82bb46f31f 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / 3357ece |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22173/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-8533) Mismatch in displaying the "MissingBlock" count in fsck and in other metric reports
[
https://issues.apache.org/jira/browse/HDFS-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264215#comment-16264215
]
regis le bretonnic commented on HDFS-8533:
--
I think, I'm concerned too on 2.7.3 using HDP 2.6
Ambari reports a missing block (get from JMX) that I can't find with fsck.
> Mismatch in displaying the "MissingBlock" count in fsck and in other metric
> reports
> ---
>
> Key: HDFS-8533
> URL: https://issues.apache.org/jira/browse/HDFS-8533
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
>
> Number of DN = 2
> Step 1: Write a file with replication factor - 3 .
> Step 2: Corrupt a replica in DN1
> Step 3: DN2 is down.
> Missing Block count in report is as follows
> Fsck report: *0*
> Jmx, "dfsadmin -report" , UI, logs : *1*
> In fsck , only block whose replicas are all missed and not been corrupted are
> counted
> {code}
> if (totalReplicasPerBlock == 0 && !isCorrupt) {
> // If the block is corrupted, it means all its available replicas are
> // corrupted. We don't mark it as missing given these available
> replicas
> // might still be accessible as the block might be incorrectly marked
> as
> // corrupted by client machines.
> {code}
> While in other reports even if all the replicas are corrupted , block is been
> considered as missed.
> Please provide your thoughts : can we make missing block count consistent
> across all the reports same as implemented for fsck?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12850) Ozone: TestContainerPersistence#testListContainer fails frequently due to timed out
[
https://issues.apache.org/jira/browse/HDFS-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12850:
-
Attachment: HDFS-12850-HDFS-7240.001.patch
> Ozone: TestContainerPersistence#testListContainer fails frequently due to
> timed out
> ---
>
> Key: HDFS-12850
> URL: https://issues.apache.org/jira/browse/HDFS-12850
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, test
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12850-HDFS-7240.001.patch
>
>
> Failed unit test:
> {noformat}
> java.lang.Exception: test timed out after 30 milliseconds
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:231)
> at org.apache.hadoop.utils.RocksDBStore.(RocksDBStore.java:64)
> at
> org.apache.hadoop.utils.MetadataStoreBuilder.build(MetadataStoreBuilder.java:94)
> at
> org.apache.hadoop.ozone.container.common.helpers.ContainerUtils.createMetadata(ContainerUtils.java:254)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.writeContainerInfo(ContainerManagerImpl.java:395)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.createContainer(ContainerManagerImpl.java:328)
> at
> org.apache.hadoop.ozone.container.common.impl.TestContainerPersistence.testListContainer(TestContainerPersistence.java:342)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12850) Ozone: TestContainerPersistence#testListContainer fails frequently due to timed out
[
https://issues.apache.org/jira/browse/HDFS-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12850:
-
Attachment: (was: HDFS-12850-HDFS-7240.001.patch)
> Ozone: TestContainerPersistence#testListContainer fails frequently due to
> timed out
> ---
>
> Key: HDFS-12850
> URL: https://issues.apache.org/jira/browse/HDFS-12850
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, test
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12850-HDFS-7240.001.patch
>
>
> Failed unit test:
> {noformat}
> java.lang.Exception: test timed out after 30 milliseconds
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:231)
> at org.apache.hadoop.utils.RocksDBStore.(RocksDBStore.java:64)
> at
> org.apache.hadoop.utils.MetadataStoreBuilder.build(MetadataStoreBuilder.java:94)
> at
> org.apache.hadoop.ozone.container.common.helpers.ContainerUtils.createMetadata(ContainerUtils.java:254)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.writeContainerInfo(ContainerManagerImpl.java:395)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.createContainer(ContainerManagerImpl.java:328)
> at
> org.apache.hadoop.ozone.container.common.impl.TestContainerPersistence.testListContainer(TestContainerPersistence.java:342)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12842) Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED
[ https://issues.apache.org/jira/browse/HDFS-12842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDFS-12842: --- Status: Patch Available (was: Open) > Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED > - > > Key: HDFS-12842 > URL: https://issues.apache.org/jira/browse/HDFS-12842 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar > Attachments: HDFS-12842-HDFS-7240.000.patch > > > This was initially fixed as part of HDFS-12796, but with HDFS-12756 the > change was overwritten. This jira is to bring back HDFS-12796 changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
