[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17001162#comment-17001162
]
Íñigo Goiri commented on HDFS-15062:
My bad, logger format...
Reverted it from 3.0 just for cleanness.
I'll close it as is with just 3.3 as target.
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch,
> HDFS-15062.003.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17001157#comment-17001157
]
Eric Badger commented on HDFS-15062:
I have reverted this patch from branch-3.2 and branch-3.1. I didn't bother with
branch-3.0, since that branch is no longer active.
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch,
> HDFS-15062.003.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000268#comment-17000268
]
Hudson commented on HDFS-15062:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17779 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17779/])
HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. (inigoiri:
rev 52d7b745c6d95e799542d6409dac30d0418ce8a8)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/IncrementalBlockReportManager.java
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch,
> HDFS-15062.003.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000248#comment-17000248
]
Íñigo Goiri commented on HDFS-15062:
Thanks [~ferhui] for the patch and [~weichiu] and [~ayushtkn] for the reviews.
Pushed to trunk, branch-3.2, branch-3.1, and branch-3.0.
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch,
> HDFS-15062.003.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000238#comment-17000238
]
Íñigo Goiri commented on HDFS-15062:
The failed unit tests are unrelated, updating my vote and committing shortly.
+1 on [^HDFS-15062.003.patch]
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch,
> HDFS-15062.003.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999723#comment-16999723
]
Wei-Chiu Chuang commented on HDFS-15062:
LGTM
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch,
> HDFS-15062.003.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999720#comment-16999720
]
Hadoop QA commented on HDFS-15062:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDeadNodeDetection |
| | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.TestReconstructStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:e573ea49085 |
| JIRA Issue | HDFS-15062 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12989151/HDFS-15062.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 88a4ee862d58 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7b93575 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28542/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28542/testReport/ |
| Max. process+thread count | 3975 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999647#comment-16999647
]
Fei Hui commented on HDFS-15062:
[~weichiu] [~elgoiri]
I think adding more info is good.Upload v003 patch add nnId and duration info
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch,
> HDFS-15062.003.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999402#comment-16999402
]
Íñigo Goiri commented on HDFS-15062:
[~weichiu] was also bringing up if we needed to add more information to the log
message.
He proposed the duration but maybe there are other things too.
[~ferhui], just give it a thought to see if we should add more info.
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999281#comment-16999281
]
Fei Hui commented on HDFS-15062:
[~hexiaoqiao] Thanks for reminding me. HDFS-14997 is great work, it could
resolve the problem that sending IBRs was delayed.
This JIRA just adds key logs for quick trouble shooting.
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998834#comment-16998834
]
Xiaoqiao He commented on HDFS-15062:
It seems very similar to HDFS-14997, maybe it could solve this issue. FYI.
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998819#comment-16998819
]
Fei Hui commented on HDFS-15062:
[~weichiu] Now didn't see failing. I found that sending IBRs was delayed
because of handling invalid blocks.
I think adding log is helpful for quick trouble shooting
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998801#comment-16998801
]
Wei-Chiu Chuang commented on HDFS-15062:
What's the typical reason for failing to send IBRs? Time outs? I'm wondering if
logging the duration of the call can be useful too.
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998361#comment-16998361
]
Ayush Saxena commented on HDFS-15062:
-
v002 LGTM +1
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997857#comment-16997857
]
Hadoop QA commented on HDFS-15062:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDeadNodeDetection |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:e573ea49085 |
| JIRA Issue | HDFS-15062 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12988957/HDFS-15062.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 68f1356b8742 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 578bd10 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28535/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28535/testReport/ |
| Max. process+thread count | 2767 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997773#comment-16997773
]
Íñigo Goiri commented on HDFS-15062:
+1 on [^HDFS-15062.002.patch].
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997749#comment-16997749
]
Fei Hui commented on HDFS-15062:
[~elgoiri] Thanks for your comments!
Upload v002 patch
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997515#comment-16997515
]
Íñigo Goiri commented on HDFS-15062:
Not that it makes much of a difference with warn and this setting but it makes
sense to use the logger format:
{code}
LOG.warn("Failed to call blockReceivedAndDeleted: {}",
Arrays.toString(reports));
{code}
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997294#comment-16997294
]
Hadoop QA commented on HDFS-15062:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDeadNodeDetection |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:e573ea49085 |
| JIRA Issue | HDFS-15062 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12988913/HDFS-15062.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7c1e2b89cbc7 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / dc6cf17 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28532/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28532/testReport/ |
| Max. process+thread count | 2778 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-15062) Add LOG when sendIBRs failed
[
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997224#comment-16997224
]
Fei Hui commented on HDFS-15062:
[~weichiu] [~ayushtkn] Could you please take a look ? Thanks
> Add LOG when sendIBRs failed
>
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15062.001.patch
>
>
> {code}
> /** Send IBRs to namenode. */
> void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
> String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
> // Nothing new to report.
> return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
> LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
> namenode.blockReceivedAndDeleted(registration, bpid, reports);
> success = true;
> } finally {
> if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
> } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
> }
> }
> }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to
> pendingIBRs. Maybe we should add log for failed case. It is helpful for
> trouble shooting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
