[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-23 Thread Ahmed Hussein (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219709#comment-17219709
 ] 

Ahmed Hussein commented on HDFS-15622:
--

Thanks [~kihwal]!

> Deleted blocks linger in the replications queue
> ---
>
> Key: HDFS-15622
> URL: https://issues.apache.org/jira/browse/HDFS-15622
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3
>
> Attachments: HDFS-15622.001.patch, HDFS-15622.002.patch
>
>
> We had incident whereas after resolving a missing blocks incident by 
> restarting two dead nodes, there were still 8 missing, but the list was 
> empty. Metasave shows the 8 blocks are "orphaned" meaning the files were 
> already deleted. It is unclear why they were left in the replication queue.
> * The containing node was flaky and started stoped multiple time.
> * The block allocation didn't work well due to the cluster-level storage 
> space exhaustion.
> * The NN was in safe mode.
> Triggering a full block report from the node didn't have any effect. It will 
> clear up if a failover happens as the repl queue will be reinitialized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-22 Thread Ahmed Hussein (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219365#comment-17219365
 ] 

Ahmed Hussein commented on HDFS-15622:
--

Thanks [~kihwal]. I checked the failures in Junits.
I am not going to file new jiras for {{TestAddOverReplicatedStripedBlocks}} or 
{{TestStandbyCheckpoints}} until the other Junits listed in HDFS-15646 get 
fixed.

> Deleted blocks linger in the replications queue
> ---
>
> Key: HDFS-15622
> URL: https://issues.apache.org/jira/browse/HDFS-15622
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HDFS-15622.001.patch, HDFS-15622.002.patch
>
>
> We had incident whereas after resolving a missing blocks incident by 
> restarting two dead nodes, there were still 8 missing, but the list was 
> empty. Metasave shows the 8 blocks are "orphaned" meaning the files were 
> already deleted. It is unclear why they were left in the replication queue.
> * The containing node was flaky and started stoped multiple time.
> * The block allocation didn't work well due to the cluster-level storage 
> space exhaustion.
> * The NN was in safe mode.
> Triggering a full block report from the node didn't have any effect. It will 
> clear up if a failover happens as the repl queue will be reinitialized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-22 Thread Kihwal Lee (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219360#comment-17219360
 ] 

Kihwal Lee commented on HDFS-15622:
---

+1 on version 2 of the patch.

> Deleted blocks linger in the replications queue
> ---
>
> Key: HDFS-15622
> URL: https://issues.apache.org/jira/browse/HDFS-15622
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HDFS-15622.001.patch, HDFS-15622.002.patch
>
>
> We had incident whereas after resolving a missing blocks incident by 
> restarting two dead nodes, there were still 8 missing, but the list was 
> empty. Metasave shows the 8 blocks are "orphaned" meaning the files were 
> already deleted. It is unclear why they were left in the replication queue.
> * The containing node was flaky and started stoped multiple time.
> * The block allocation didn't work well due to the cluster-level storage 
> space exhaustion.
> * The NN was in safe mode.
> Triggering a full block report from the node didn't have any effect. It will 
> clear up if a failover happens as the repl queue will be reinitialized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-21 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218484#comment-17218484
 ] 

Hadoop QA commented on HDFS-15622:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
42s{color} |  | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} |  | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch does not contain any @author tags. 
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} |  | {color:green} The patch appears to include 2 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 
23s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
24s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
52s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
30s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 45s{color} |  | {color:green} branch has no errors when building and 
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  3m 
11s{color} |  | {color:blue} Used deprecated FindBugs config; considering 
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
8s{color} |  | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
14s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
12s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m 10s{color} |  | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
26s{color} |  | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 32s{color} 
|

[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-20 Thread Kihwal Lee (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218087#comment-17218087
 ] 

Kihwal Lee commented on HDFS-15622:
---

The patch looks good. the only minor nit is the name of one of the new 
variables "corruptFlag".  It sounds like it is telling something is corrupt or 
user specified it to do something with corruption. Maybe it is better to make 
it something more descriptive like "inCorruptLevel".  The logic seems fine.

> Deleted blocks linger in the replications queue
> ---
>
> Key: HDFS-15622
> URL: https://issues.apache.org/jira/browse/HDFS-15622
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HDFS-15622.001.patch
>
>
> We had incident whereas after resolving a missing blocks incident by 
> restarting two dead nodes, there were still 8 missing, but the list was 
> empty. Metasave shows the 8 blocks are "orphaned" meaning the files were 
> already deleted. It is unclear why they were left in the replication queue.
> * The containing node was flaky and started stoped multiple time.
> * The block allocation didn't work well due to the cluster-level storage 
> space exhaustion.
> * The NN was in safe mode.
> Triggering a full block report from the node didn't have any effect. It will 
> clear up if a failover happens as the repl queue will be reinitialized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-12 Thread Ahmed Hussein (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212454#comment-17212454
 ] 

Ahmed Hussein commented on HDFS-15622:
--

I had to change couple of Junit test files. I added a line to set the bcId to 
non "-1" value, so that it won't be considered deleted.

I checked the failing Junit tests:
 * {{TestFileChecksum}}: passes. The failure is unrelated. It fails 
intermittently.
 * {{TestReconstructStripedFileWithRandomECPolicy}}: Passes. The failure is 
intermittent.
 * {{TestDiskspaceQuotaUpdate}}: passes. failure is intermittent.
 * {{TestWebHDFS}}: testLargeDirectory fails for some time. HDFS-6543
 * {{TestFileChecksumCompositeCrc}}: passes
 * {{TestExternalStoragePolicySatisfier}}: HDFS-15456 is a recent fix for that 
test.

> Deleted blocks linger in the replications queue
> ---
>
> Key: HDFS-15622
> URL: https://issues.apache.org/jira/browse/HDFS-15622
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HDFS-15622.001.patch
>
>
> We had incident whereas after resolving a missing blocks incident by 
> restarting two dead nodes, there were still 8 missing, but the list was 
> empty. Metasave shows the 8 blocks are "orphaned" meaning the files were 
> already deleted. It is unclear why they were left in the replication queue.
> * The containing node was flaky and started stoped multiple time.
> * The block allocation didn't work well due to the cluster-level storage 
> space exhaustion.
> * The NN was in safe mode.
> Triggering a full block report from the node didn't have any effect. It will 
> clear up if a failover happens as the repl queue will be reinitialized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-09 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211500#comment-17211500
 ] 

Hadoop QA commented on HDFS-15622:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 32m 
35s{color} |  | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} |  | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch does not contain any @author tags. 
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} |  | {color:green} The patch appears to include 2 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
17s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
17s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m 32s{color} |  | {color:green} branch has no errors when building and 
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  3m  
7s{color} |  | {color:blue} Used deprecated FindBugs config; considering 
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
5s{color} |  | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
11s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
10s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
10s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
3s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
3s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m  6s{color} |  | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
14s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
11s{color} |  | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 51s{color} 
|