[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-12182: --- Resolution: Fixed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) Committed the patch to brach-2. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch, HDFS-12182.004.patch, HDFS-12182-branch-2.001.patch, > HDFS-12182-branch-2.002.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Attachment: HDFS-12182-branch-2.002.patch Fixed issue on TestMetaSave. Other tests failed on previous jira are passing locally. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch, HDFS-12182.004.patch, HDFS-12182-branch-2.001.patch, > HDFS-12182-branch-2.002.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Attachment: HDFS-12182-branch-2.001.patch Applied changes on top of branch-2 and generated patch *HDFS-12182-branch-2.001.patch*. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch, HDFS-12182.004.patch, HDFS-12182-branch-2.001.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-12182: --- Fix Version/s: 3.0.0-beta1 > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch, HDFS-12182.004.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Status: Patch Available (was: In Progress) > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch, HDFS-12182.004.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Status: In Progress (was: Patch Available) > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch, HDFS-12182.004.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Attachment: HDFS-12182.004.patch Attaching new patch version with the applied suggestions from last review. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch, HDFS-12182.004.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-12182: --- Target Version/s: 2.9.0, 3.0.0-beta1 > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-12182: --- Fix Version/s: (was: 3.0.0-alpha3) > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Attachment: HDFS-12182.003.patch Adding new patch with checkstyles reviewed. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Status: In Progress (was: Patch Available) > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Status: Patch Available (was: In Progress) > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, > HDFS-12182.003.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Status: Patch Available (was: In Progress) > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Attachment: HDFS-12182.002.patch Attaching new patch version with the suggested changes. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Attachment: HDFS-12182.001.patch Attaching patch with proposed changes to metasave. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-12182.001.patch > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Description: Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs CLI command) reports both "under replicated" and "missing" blocks under same metric *Metasave: Blocks waiting for reconstruction:* as shown on below code snippet: {noformat} synchronized (neededReconstruction) { out.println("Metasave: Blocks waiting for reconstruction: " + neededReconstruction.size()); for (Block block : neededReconstruction) { dumpBlockMeta(block, out); } } {noformat} *neededReconstruction* is an instance of *LowRedundancyBlocks*, which actually wraps 5 priority queues currently. 4 of these queues store different under replicated scenarios, but the 5th one is dedicated for corrupt blocks. Thus, metasave report may suggest some corrupt blocks are just under replicated. This can be misleading for admins and operators trying to track block missing/corruption issues, and/or other issues related to *BlockManager* metrics. I would like to propose a patch with trivial changes that would report corrupt blocks separately. was: Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs CLI command) reports both "under replicated" and "corrupt" blocks under same metric *Metasave: Blocks waiting for reconstruction:* as shown on below code snippet: {noformat} synchronized (neededReconstruction) { out.println("Metasave: Blocks waiting for reconstruction: " + neededReconstruction.size()); for (Block block : neededReconstruction) { dumpBlockMeta(block, out); } } {noformat} *neededReconstruction* is an instance of *LowRedundancyBlocks*, which actually wraps 5 priority queues currently. 4 of these queues store different under replicated scenarios, but the 5th one is dedicated for corrupt blocks. Thus, metasave report may suggest some corrupt blocks are just under replicated. This can be misleading for admins and operators trying to track block corruption issues, and/or other issues related to *BlockManager* metrics. I would like to propose a patch with trivial changes that would report corrupt blocks separately. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Description: Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs CLI command) reports both "under replicated" and "missing" blocks under same metric *Metasave: Blocks waiting for reconstruction:* as shown on below code snippet: {noformat} synchronized (neededReconstruction) { out.println("Metasave: Blocks waiting for reconstruction: " + neededReconstruction.size()); for (Block block : neededReconstruction) { dumpBlockMeta(block, out); } } {noformat} *neededReconstruction* is an instance of *LowRedundancyBlocks*, which actually wraps 5 priority queues currently. 4 of these queues store different under replicated scenarios, but the 5th one is dedicated for corrupt/missing blocks. Thus, metasave report may suggest some corrupt blocks are just under replicated. This can be misleading for admins and operators trying to track block missing/corruption issues, and/or other issues related to *BlockManager* metrics. I would like to propose a patch with trivial changes that would report corrupt blocks separately. was: Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs CLI command) reports both "under replicated" and "missing" blocks under same metric *Metasave: Blocks waiting for reconstruction:* as shown on below code snippet: {noformat} synchronized (neededReconstruction) { out.println("Metasave: Blocks waiting for reconstruction: " + neededReconstruction.size()); for (Block block : neededReconstruction) { dumpBlockMeta(block, out); } } {noformat} *neededReconstruction* is an instance of *LowRedundancyBlocks*, which actually wraps 5 priority queues currently. 4 of these queues store different under replicated scenarios, but the 5th one is dedicated for corrupt blocks. Thus, metasave report may suggest some corrupt blocks are just under replicated. This can be misleading for admins and operators trying to track block missing/corruption issues, and/or other issues related to *BlockManager* metrics. I would like to propose a patch with trivial changes that would report corrupt blocks separately. > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "missing" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt/missing > blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block missing/corruption issues, and/or other issues related to > *BlockManager* metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12182: Summary: BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks (was: BlockManager.metaSave does not distinguish between "under replicated" and "corrupt" blocks) > BlockManager.metaSave does not distinguish between "under replicated" and > "missing" blocks > -- > > Key: HDFS-12182 > URL: https://issues.apache.org/jira/browse/HDFS-12182 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha3 > > > Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs > CLI command) reports both "under replicated" and "corrupt" blocks under same > metric *Metasave: Blocks waiting for reconstruction:* as shown on below code > snippet: > {noformat} >synchronized (neededReconstruction) { > out.println("Metasave: Blocks waiting for reconstruction: " > + neededReconstruction.size()); > for (Block block : neededReconstruction) { > dumpBlockMeta(block, out); > } > } > {noformat} > *neededReconstruction* is an instance of *LowRedundancyBlocks*, which > actually wraps 5 priority queues currently. 4 of these queues store different > under replicated scenarios, but the 5th one is dedicated for corrupt blocks. > Thus, metasave report may suggest some corrupt blocks are just under > replicated. This can be misleading for admins and operators trying to track > block corruption issues, and/or other issues related to *BlockManager* > metrics. > I would like to propose a patch with trivial changes that would report > corrupt blocks separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org