[jira] [Commented] (HDFS-14231) DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error
[
https://issues.apache.org/jira/browse/HDFS-14231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767338#comment-16767338
]
Kitti Nanasi commented on HDFS-14231:
-
Thanks [~jojochuang] for reviewing and committing!
> DataXceiver#run() should not log exceptions caused by InvalidToken exception
> as an error
>
>
> Key: HDFS-14231
> URL: https://issues.apache.org/jira/browse/HDFS-14231
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14231.001.patch
>
>
> HDFS-10760 changed the log level from error to trace in DataXceiver#run() if
> the exception was an InvalidToken exception. I think it would be beneficial
> to log on trace level if the exception's cause was InvalidException. Like in
> the following case:
> {code:java}
> DataXceiver error processing unknown operation
> src: xxx dst: xxx
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Block
> token with block_token_identifier
> (expiryDate=1547593336220, keyId=-1735471718, userId=hbase,
> blockPoolId=BP-xxx, blockId=1245599303, access modes=[READ]) is expired.]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14231) DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error
[
https://issues.apache.org/jira/browse/HDFS-14231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761325#comment-16761325
]
Hudson commented on HDFS-14231:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15896 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15896/])
HDFS-14231. DataXceiver#run() should not log exceptions caused by (weichiu: rev
49ddd8a6ed5b40d12defb0771b4c8b53d4ffde3f)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
> DataXceiver#run() should not log exceptions caused by InvalidToken exception
> as an error
>
>
> Key: HDFS-14231
> URL: https://issues.apache.org/jira/browse/HDFS-14231
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14231.001.patch
>
>
> HDFS-10760 changed the log level from error to trace in DataXceiver#run() if
> the exception was an InvalidToken exception. I think it would be beneficial
> to log on trace level if the exception's cause was InvalidException. Like in
> the following case:
> {code:java}
> DataXceiver error processing unknown operation
> src: xxx dst: xxx
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Block
> token with block_token_identifier
> (expiryDate=1547593336220, keyId=-1735471718, userId=hbase,
> blockPoolId=BP-xxx, blockId=1245599303, access modes=[READ]) is expired.]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14231) DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error
[
https://issues.apache.org/jira/browse/HDFS-14231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760310#comment-16760310
]
Wei-Chiu Chuang commented on HDFS-14231:
+1
For posterity, this is usually seen on a secure HBase cluster, when HBase
RegionServer opens a block for a long time and then reads the block after 10
hours (block token expires after 10 hours by default). So nothing to panic
about.
Additionally, the block token expiration message is logged prior to this
message, like the following:
{noformat}
2019-01-16 03:32:23,809 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Block token verification failed: op=REQUEST_SHORT_CIRCUIT_FDS,
remoteAddress=unix:/var/run/hdfs-sockets/dn, message=Block token with
block_token_identifi
er (expiryDate=1547569931036, keyId=, userId=hbase, blockPoolId=,
blockId=, access modes=[READ]) is expired.{noformat}
So no need to log it again, especially not at ERROR level.
This is similar to HDFS-10760. The difference is that in this case, DataXceiver
thread just got created, and the exception is thrown from a different code path.
> DataXceiver#run() should not log exceptions caused by InvalidToken exception
> as an error
>
>
> Key: HDFS-14231
> URL: https://issues.apache.org/jira/browse/HDFS-14231
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Attachments: HDFS-14231.001.patch
>
>
> HDFS-10760 changed the log level from error to trace in DataXceiver#run() if
> the exception was an InvalidToken exception. I think it would be beneficial
> to log on trace level if the exception's cause was InvalidException. Like in
> the following case:
> {code:java}
> DataXceiver error processing unknown operation
> src: xxx dst: xxx
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Block
> token with block_token_identifier
> (expiryDate=1547593336220, keyId=-1735471718, userId=hbase,
> blockPoolId=BP-xxx, blockId=1245599303, access modes=[READ]) is expired.]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14231) DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error
[
https://issues.apache.org/jira/browse/HDFS-14231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752213#comment-16752213
]
Hadoop QA commented on HDFS-14231:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSClientRetries |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14231 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12956300/HDFS-14231.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b48f9b7cfb49 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 84bb980 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26053/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26053/testReport/ |
| Max. process+thread count | 4158 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs
