[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[ https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17802559#comment-17802559 ] Shilun Fan commented on HDFS-7408: -- Bulk update: moved all 3.4.0 non-blocker issues, please move back if it is a blocker. Retarget 3.5.0. > Add a counter in the log that shows the number of block reports processed > - > > Key: HDFS-7408 > URL: https://issues.apache.org/jira/browse/HDFS-7408 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Suresh Srinivas >Assignee: Surendra Singh Lilhore >Priority: Major > Attachments: HDFS-7408.001.patch > > > It would be great to have in the info log corresponding to block report > processing, printing information on how many block reports have been > processed. This can be useful to debug when namenode is unresponsive > especially during startup time to understand if datanodes are sending block > reports multiple times. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[
https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081501#comment-17081501
]
Hadoop QA commented on HDFS-7408:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-7408 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-7408 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12755340/HDFS-7408.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29126/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add a counter in the log that shows the number of block reports processed
> -
>
> Key: HDFS-7408
> URL: https://issues.apache.org/jira/browse/HDFS-7408
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-7408.001.patch
>
>
> It would be great to have in the info log corresponding to block report
> processing, printing information on how many block reports have been
> processed. This can be useful to debug when namenode is unresponsive
> especially during startup time to understand if datanodes are sending block
> reports multiple times.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[
https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697053#comment-16697053
]
Hadoop QA commented on HDFS-7408:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-7408 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-7408 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12755340/HDFS-7408.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25604/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add a counter in the log that shows the number of block reports processed
> -
>
> Key: HDFS-7408
> URL: https://issues.apache.org/jira/browse/HDFS-7408
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-7408.001.patch
>
>
> It would be great to have in the info log corresponding to block report
> processing, printing information on how many block reports have been
> processed. This can be useful to debug when namenode is unresponsive
> especially during startup time to understand if datanodes are sending block
> reports multiple times.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[
https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407004#comment-16407004
]
genericqa commented on HDFS-7408:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-7408 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-7408 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12755340/HDFS-7408.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23581/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add a counter in the log that shows the number of block reports processed
> -
>
> Key: HDFS-7408
> URL: https://issues.apache.org/jira/browse/HDFS-7408
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-7408.001.patch
>
>
> It would be great to have in the info log corresponding to block report
> processing, printing information on how many block reports have been
> processed. This can be useful to debug when namenode is unresponsive
> especially during startup time to understand if datanodes are sending block
> reports multiple times.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[ https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201152#comment-16201152 ] Subru Krishnan commented on HDFS-7408: -- Pushing it out from 2.9.0 due to lack of recent activity. Feel free to revert if required. > Add a counter in the log that shows the number of block reports processed > - > > Key: HDFS-7408 > URL: https://issues.apache.org/jira/browse/HDFS-7408 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Suresh Srinivas >Assignee: Surendra Singh Lilhore > Attachments: HDFS-7408.001.patch > > > It would be great to have in the info log corresponding to block report > processing, printing information on how many block reports have been > processed. This can be useful to debug when namenode is unresponsive > especially during startup time to understand if datanodes are sending block > reports multiple times. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[
https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186563#comment-16186563
]
Hadoop QA commented on HDFS-7408:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-7408 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-7408 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12755340/HDFS-7408.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21451/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add a counter in the log that shows the number of block reports processed
> -
>
> Key: HDFS-7408
> URL: https://issues.apache.org/jira/browse/HDFS-7408
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-7408.001.patch
>
>
> It would be great to have in the info log corresponding to block report
> processing, printing information on how many block reports have been
> processed. This can be useful to debug when namenode is unresponsive
> especially during startup time to understand if datanodes are sending block
> reports multiple times.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[
https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805883#comment-15805883
]
Hadoop QA commented on HDFS-7408:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HDFS-7408 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-7408 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12755340/HDFS-7408.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18058/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add a counter in the log that shows the number of block reports processed
> -
>
> Key: HDFS-7408
> URL: https://issues.apache.org/jira/browse/HDFS-7408
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-7408.001.patch
>
>
> It would be great to have in the info log corresponding to block report
> processing, printing information on how many block reports have been
> processed. This can be useful to debug when namenode is unresponsive
> especially during startup time to understand if datanodes are sending block
> reports multiple times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[
https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740303#comment-14740303
]
Surendra Singh Lilhore commented on HDFS-7408:
--
Attached initial patch based on [~atm] comment.
*Patch 001:*
1. Added new metric {{StorageBlockReportInfo}}. This metric will give the
{{lastReportTimestamp}} and {{reportCount}} for individual storage.
2. Added block report count in log.
Please review..
*JMX out for this metric :*
{code}
"StorageBlockReportCounts" :
"{\"127.0.0.1:52515\":{\"DISK\":{\"DS-2cbda63e-23f7-43e1-ac43-a1dd35fa8980\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":144186443}},\"RAM_DISK\":{\"DS-d2dcd9dc-6dce-4187-ac1b-fa75d06002fd\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":144186443}}},\"127.0.0.1:52603\":{\"ARCHIVE\":{\"DS-ac87c0ae-47f1-4911-900b-f2e2a17da94c\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":1441864334382}},\"RAM_DISK\":{\"DS-033f5a4c-5a40-472e-8c65-94c32e642aee\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":1441864334382}}},\"127.0.0.1:52469\":{\"DISK\":{\"DS-13e53daa-ffea-439e-b126-29bc391a51b0\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":144186433}},\"RAM_DISK\":{\"DS-910b64fc-c2ac-4eba-89ab-b15f38496207\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":144186443}}},\"127.0.0.1:52647\":{\"ARCHIVE\":{\"DS-3bdb084d-8f4b-4286-9c96-9d9e5ae13ea4\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":1441864335123}},\"RAM_DISK\":{\"DS-bea5f61e-4688-45e6-9365-cb20363e96e7\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":1441864335123}}},\"127.0.0.1:52559\":{\"ARCHIVE\":{\"DS-49093dbb-e560-464c-820c-eb9efcd392a3\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":1441864333793}},\"RAM_DISK\":{\"DS-03ae5f28-0877-4b9f-846e-725486d5a750\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":1441864333793}}},\"127.0.0.1:52422\":{\"DISK\":{\"DS-5e446e18-f4ca-4b40-a073-eb2c2a3561e2\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":144186443}},\"RAM_DISK\":{\"DS-ebbd1829-bd6d-4bbe-93bc-c19427f53e08\":{\"blockReportCount\":1,\"lastblockReportTimestamp\":144186443"
{code}
> Add a counter in the log that shows the number of block reports processed
> -
>
> Key: HDFS-7408
> URL: https://issues.apache.org/jira/browse/HDFS-7408
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-7408.001.patch
>
>
> It would be great to have in the info log corresponding to block report
> processing, printing information on how many block reports have been
> processed. This can be useful to debug when namenode is unresponsive
> especially during startup time to understand if datanodes are sending block
> reports multiple times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[
https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740505#comment-14740505
]
Hadoop QA commented on HDFS-7408:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 48s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 50s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 1s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 22s | The applied patch generated 3
new checkstyle issues (total was 225, now 227). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 27s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 132m 56s | Tests failed in hadoop-hdfs. |
| | | 178m 6s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755340/HDFS-7408.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f103a70 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12388/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12388/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12388/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12388/console |
This message was automatically generated.
> Add a counter in the log that shows the number of block reports processed
> -
>
> Key: HDFS-7408
> URL: https://issues.apache.org/jira/browse/HDFS-7408
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-7408.001.patch
>
>
> It would be great to have in the info log corresponding to block report
> processing, printing information on how many block reports have been
> processed. This can be useful to debug when namenode is unresponsive
> especially during startup time to understand if datanodes are sending block
> reports multiple times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[ https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216949#comment-14216949 ] Aaron T. Myers commented on HDFS-7408: -- Hey Suresh, are you envisioning this printing just a counter for the total number of block reports processed? Or total number of BRs (full and incremental?) for each DN? Or perhaps even breaking it down per individual storage, instead of per DN? Also, if we're going to put this in the logs, let's make sure it also shows up in JMX output as well. Add a counter in the log that shows the number of block reports processed - Key: HDFS-7408 URL: https://issues.apache.org/jira/browse/HDFS-7408 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas It would be great to have in the info log corresponding to block report processing, printing information on how many block reports have been processed. This can be useful to debug when namenode is unresponsive especially during startup time to understand if datanodes are sending block reports multiple times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
