[jira] [Work logged] (HDFS-15937) Reduce memory used during datanode layout upgrade
[
https://issues.apache.org/jira/browse/HDFS-15937?focusedWorklogId=574598=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574598
]
ASF GitHub Bot logged work on HDFS-15937:
-
Author: ASF GitHub Bot
Created on: 31/Mar/21 05:30
Start Date: 31/Mar/21 05:30
Worklog Time Spent: 10m
Work Description: jojochuang commented on a change in pull request #2838:
URL: https://github.com/apache/hadoop/pull/2838#discussion_r604604531
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
##
@@ -1071,12 +1071,26 @@ private static void linkAllBlocks(File fromDir, File
fromBbwDir, File toDir,
}
private static class LinkArgs {
-File src;
-File dst;
+private File srcDir;
+private File dstDir;
+private String blockFile;
+
+LinkArgs(File srcDir, File dstDir, String blockFile) {
+ this.srcDir = srcDir;
+ this.dstDir = dstDir;
+ this.blockFile = blockFile;
+}
+
+public File src() {
Review comment:
it was meant to cause less code churn --> src ==> src() :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 574598)
Time Spent: 50m (was: 40m)
> Reduce memory used during datanode layout upgrade
> -
>
> Key: HDFS-15937
> URL: https://issues.apache.org/jira/browse/HDFS-15937
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.3.0, 3.1.4, 3.2.2, 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Attachments: heap-dump-after.png, heap-dump-before.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When the datanode block layout is upgrade from -56 (256x256) to -57 (32x32),
> we have found the datanode uses a lot more memory than usual.
> For each volume, the blocks are scanned and a list is created holding a
> series of LinkArgs objects. This object contains a File object for the block
> source and destination. The file object stores the path as a string, eg:
> /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta
> /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825
> This is string is repeated for every block and meta file on the DN, and much
> of the string is the same each time, leading to a large amount of memory.
> If we change the linkArgs to store:
> * Src Path without the block, eg
> /data01/dfs/dn/previous.tmp/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0
> * Dest Path without the block eg
> /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir10
> * Block / Meta file name, eg blk_12345678_1001 or blk_12345678_1001.meta
> Then ensure were reuse the same file object for repeated src and dest paths,
> we can save most of the memory without reworking the logic of the code.
> The current logic works along the source paths recursively, so you can easily
> re-use the src path object.
> For the destination path, there are only 32x32 (1024) distinct paths, so we
> can simply cache them in a hashMap and lookup the re-useable object each time.
> I tested locally by generating 100k block files and attempting the layout
> upgrade. A heap dump showed the 100k blocks using about 140MB of heap. That
> is close to 1.5GB per 1M blocks.
> After the change outlined above the same 100K blocks used about 20MB of heap,
> so 200MB per million blocks.
> A general DN sizing recommendation is 1GB of heap per 1M blocks, so the
> upgrade should be able to happen within the pre-upgrade heap.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312065#comment-17312065
]
Renukaprasad C commented on HDFS-15863:
---
Thanks [~hexiaoqiao][~surendralilhore][~elgoiri] for review & feedback.
> RBF: Validation message to be corrected in FairnessPolicyController
> ---
>
> Key: HDFS-15863
> URL: https://issues.apache.org/jira/browse/HDFS-15863
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: HDFS-15863.001.patch, HDFS-15863.002.patch,
> HDFS-15863.003.patch, HDFS-15863.004.patch, HDFS-15863.005.patch
>
>
> org.apache.hadoop.hdfs.server.federation.fairness.StaticRouterRpcFairnessPolicyController#validateCount
> When dfs.federation.router.handler.count is lessthan the total dedicated
> handlers for all NS, then error message shows 0 & -ve values in error
> message, instead can show the actual configured values.
> Current message is : "Available handlers -5 lower than min 0 for nsId nn1"
> This can be changed to: "Configured handlers
> ${DFS_ROUTER_HANDLER_COUNT_KEY}=10 lower than min 15 for nsId nn1", where 10
> is hander count & 15 is sum of dedicated handler count.
> Related to: HDFS-14090
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15932) Improve the balancer error message when process exits abnormally.
[ https://issues.apache.org/jira/browse/HDFS-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312059#comment-17312059 ] Renukaprasad C commented on HDFS-15932: --- Thanks [~hexiaoqiao] & [~brahmareddy] > Improve the balancer error message when process exits abnormally. > - > > Key: HDFS-15932 > URL: https://issues.apache.org/jira/browse/HDFS-15932 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3 > > Attachments: HDFS-15932.001.patch > > > The HDFS balancer exits abnormally. The content of the pid file is not > cleaned up, and the new balancer cannot be started. > Start the balancer (start-balancer.sh threshold 5) > Kill the balancer process (kill -9 ) > Re-execute the balancer, there will be an error message. > -- Balancer is running as process . Stop it first. > (But process already stopped, error message can be more detailed) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15863:
---
Fix Version/s: 3.4.0
> RBF: Validation message to be corrected in FairnessPolicyController
> ---
>
> Key: HDFS-15863
> URL: https://issues.apache.org/jira/browse/HDFS-15863
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: HDFS-15863.001.patch, HDFS-15863.002.patch,
> HDFS-15863.003.patch, HDFS-15863.004.patch, HDFS-15863.005.patch
>
>
> org.apache.hadoop.hdfs.server.federation.fairness.StaticRouterRpcFairnessPolicyController#validateCount
> When dfs.federation.router.handler.count is lessthan the total dedicated
> handlers for all NS, then error message shows 0 & -ve values in error
> message, instead can show the actual configured values.
> Current message is : "Available handlers -5 lower than min 0 for nsId nn1"
> This can be changed to: "Configured handlers
> ${DFS_ROUTER_HANDLER_COUNT_KEY}=10 lower than min 15 for nsId nn1", where 10
> is hander count & 15 is sum of dedicated handler count.
> Related to: HDFS-14090
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15863:
---
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks [~prasad-acit] for your works. Thanks [~elgoiri] and
[~surendralilhore] for your reviews.
> RBF: Validation message to be corrected in FairnessPolicyController
> ---
>
> Key: HDFS-15863
> URL: https://issues.apache.org/jira/browse/HDFS-15863
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Minor
> Attachments: HDFS-15863.001.patch, HDFS-15863.002.patch,
> HDFS-15863.003.patch, HDFS-15863.004.patch, HDFS-15863.005.patch
>
>
> org.apache.hadoop.hdfs.server.federation.fairness.StaticRouterRpcFairnessPolicyController#validateCount
> When dfs.federation.router.handler.count is lessthan the total dedicated
> handlers for all NS, then error message shows 0 & -ve values in error
> message, instead can show the actual configured values.
> Current message is : "Available handlers -5 lower than min 0 for nsId nn1"
> This can be changed to: "Configured handlers
> ${DFS_ROUTER_HANDLER_COUNT_KEY}=10 lower than min 15 for nsId nn1", where 10
> is hander count & 15 is sum of dedicated handler count.
> Related to: HDFS-14090
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15863:
---
Issue Type: Improvement (was: Bug)
> RBF: Validation message to be corrected in FairnessPolicyController
> ---
>
> Key: HDFS-15863
> URL: https://issues.apache.org/jira/browse/HDFS-15863
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Minor
> Attachments: HDFS-15863.001.patch, HDFS-15863.002.patch,
> HDFS-15863.003.patch, HDFS-15863.004.patch, HDFS-15863.005.patch
>
>
> org.apache.hadoop.hdfs.server.federation.fairness.StaticRouterRpcFairnessPolicyController#validateCount
> When dfs.federation.router.handler.count is lessthan the total dedicated
> handlers for all NS, then error message shows 0 & -ve values in error
> message, instead can show the actual configured values.
> Current message is : "Available handlers -5 lower than min 0 for nsId nn1"
> This can be changed to: "Configured handlers
> ${DFS_ROUTER_HANDLER_COUNT_KEY}=10 lower than min 15 for nsId nn1", where 10
> is hander count & 15 is sum of dedicated handler count.
> Related to: HDFS-14090
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15937) Reduce memory used during datanode layout upgrade
[
https://issues.apache.org/jira/browse/HDFS-15937?focusedWorklogId=574582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574582
]
ASF GitHub Bot logged work on HDFS-15937:
-
Author: ASF GitHub Bot
Created on: 31/Mar/21 04:52
Start Date: 31/Mar/21 04:52
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on a change in pull request #2838:
URL: https://github.com/apache/hadoop/pull/2838#discussion_r604588150
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
##
@@ -1071,12 +1071,26 @@ private static void linkAllBlocks(File fromDir, File
fromBbwDir, File toDir,
}
private static class LinkArgs {
-File src;
-File dst;
+private File srcDir;
+private File dstDir;
+private String blockFile;
+
+LinkArgs(File srcDir, File dstDir, String blockFile) {
+ this.srcDir = srcDir;
+ this.dstDir = dstDir;
+ this.blockFile = blockFile;
+}
+
+public File src() {
Review comment:
`getSrc` shoud be more graceful? The same as method `dst(...)` +
`blockFile()`.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
##
@@ -1161,10 +1176,12 @@ public Void call() throws IOException {
*/
@Override
public int compare(LinkArgs a, LinkArgs b) {
+File asrc = a.src();
+File bsrc = b.src();
return ComparisonChain.start().
-compare(a.src.getName(), b.src.getName()).
-compare(a.src, b.src).
-compare(a.dst, b.dst).
+compare(asrc.getName(), bsrc.getName()).
+compare(asrc, bsrc).
Review comment:
is it duplicated comparator between `compare(asrc.getName(),
bsrc.getName())` and `compare(asrc, bsrc)`?
For UnixFileSystem, the implement is as following. Not sure if it is same
for other FileSystem instance.
> public int compare(File f1, File f2) {
> return f1.getPath().compareTo(f2.getPath());
> }
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
##
@@ -1345,8 +1363,18 @@ public boolean accept(File dir, String name) {
throw new IOException("Failed to mkdirs " + blockLocation);
}
}
- idBasedLayoutSingleLinks.add(new LinkArgs(new File(from, blockName),
- new File(blockLocation, blockName)));
+ /**
+ * The destination path is 32x32, so 1024 distinct paths. Therefore
+ * we cache the destination path and reuse the same File object on
+ * potentially thousands of blocks located on this volume.
+ * This method is called recursively so the cache is passed through
+ * each recursive call. There is one cache per volume, and it is only
+ * accessed by a single thread so no locking is needed.
+ */
+ File cachedDest = pathCache
+ .computeIfAbsent(blockLocation, k -> blockLocation);
+ idBasedLayoutSingleLinks.add(new LinkArgs(from,
Review comment:
Here it reuses prefix path of blocks and no `File` instances created to
reduce memory used, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 574582)
Time Spent: 40m (was: 0.5h)
> Reduce memory used during datanode layout upgrade
> -
>
> Key: HDFS-15937
> URL: https://issues.apache.org/jira/browse/HDFS-15937
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.3.0, 3.1.4, 3.2.2, 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Attachments: heap-dump-after.png, heap-dump-before.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When the datanode block layout is upgrade from -56 (256x256) to -57 (32x32),
> we have found the datanode uses a lot more memory than usual.
> For each volume, the blocks are scanned and a list is created holding a
> series of LinkArgs objects. This object contains a File object for the block
> source and destination. The file object stores the path as a string, eg:
> /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta
>
[jira] [Updated] (HDFS-15932) Improve the balancer error message when process exits abnormally.
[ https://issues.apache.org/jira/browse/HDFS-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15932: --- Fix Version/s: 3.2.3 3.1.5 3.4.0 3.3.1 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) LGTM. +1. Committed to trunk and cherry pick to branch-3.3/branch-3.2/branch-3.1. Thanks [~prasad-acit] for your improvement and Thanks [~brahmareddy] for your reviews. > Improve the balancer error message when process exits abnormally. > - > > Key: HDFS-15932 > URL: https://issues.apache.org/jira/browse/HDFS-15932 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Fix For: 3.3.1, 3.4.0, 3.1.5, 3.2.3 > > Attachments: HDFS-15932.001.patch > > > The HDFS balancer exits abnormally. The content of the pid file is not > cleaned up, and the new balancer cannot be started. > Start the balancer (start-balancer.sh threshold 5) > Kill the balancer process (kill -9 ) > Re-execute the balancer, there will be an error message. > -- Balancer is running as process . Stop it first. > (But process already stopped, error message can be more detailed) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15932) Improve the balancer error message when process exits abnormally.
[ https://issues.apache.org/jira/browse/HDFS-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15932: --- Issue Type: Improvement (was: Bug) > Improve the balancer error message when process exits abnormally. > - > > Key: HDFS-15932 > URL: https://issues.apache.org/jira/browse/HDFS-15932 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HDFS-15932.001.patch > > > The HDFS balancer exits abnormally. The content of the pid file is not > cleaned up, and the new balancer cannot be started. > Start the balancer (start-balancer.sh threshold 5) > Kill the balancer process (kill -9 ) > Re-execute the balancer, there will be an error message. > -- Balancer is running as process . Stop it first. > (But process already stopped, error message can be more detailed) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15932) Improve the balancer error message when process exits abnormally.
[
https://issues.apache.org/jira/browse/HDFS-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312043#comment-17312043
]
Hadoop QA commented on HDFS-15932:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
24s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 30s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s{color} | {color:green}{color} | {color:green} There were no new shellcheck
issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
17s{color} | {color:green}{color} | {color:green} There were no new shelldocs
issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 8s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
35s{color} | {color:green}{color} | {color:green} hadoop-common in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} The patch does not generate
ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 46s{color} |
{color:black}{color} | {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/561/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15932 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13023134/HDFS-15932.001.patch |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 36ca556957c7 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 03e42efa30b |
| Test Results |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/561/testReport/ |
| Max. process+thread count | 511 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/561/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Improve the balancer error message when process exits abnormally.
> -
>
> Key: HDFS-15932
> URL: https://issues.apache.org/jira/browse/HDFS-15932
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Major
> Attachments:
[jira] [Work logged] (HDFS-15931) Fix non-static inner classes for better memory management
[
https://issues.apache.org/jira/browse/HDFS-15931?focusedWorklogId=574558=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574558
]
ASF GitHub Bot logged work on HDFS-15931:
-
Author: ASF GitHub Bot
Created on: 31/Mar/21 03:46
Start Date: 31/Mar/21 03:46
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2830:
URL: https://github.com/apache/hadoop/pull/2830#issuecomment-810736829
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 9s | | trunk passed |
| +1 :green_heart: | compile | 5m 12s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 4m 48s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 12s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 46s | | the patch passed |
| +1 :green_heart: | compile | 5m 8s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 5m 8s | | the patch passed |
| +1 :green_heart: | compile | 4m 46s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 4m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2830/4/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 4 new + 227 unchanged - 9 fixed =
231 total (was 236) |
| +1 :green_heart: | mvnsite | 1m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 5s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 338m 58s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2830/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 25m 13s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2830/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 482m 17s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.datanode.TestIncrementalBrVariations |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestStateAlignmentContextWithHA |
| | hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.TestDFSShell |
| |
[jira] [Updated] (HDFS-15936) Solve BlockSender#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu updated HDFS-15936:
Summary: Solve BlockSender#sendPacket() does not record SocketTimeout
exception (was: Solve SocketTimeoutException#sendPacket() does not record
SocketTimeout exception)
> Solve BlockSender#sendPacket() does not record SocketTimeout exception
> --
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In BlockSender#sendPacket(), if a SocketTimeout exception occurs, no
> information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu updated HDFS-15936:
Description:
In BlockSender#sendPacket(), if a SocketTimeout exception occurs, no
information is recorded here.
try {
..
} catch (IOException e) {
if (e instanceof SocketTimeoutException) {
/*
* writing to client timed out. This happens if the client reads
* part of a block and then decides not to read the rest (but leaves
* the socket open).
*
* Reporting of this case is done in DataXceiver#run
*/
}
}
No records are generated here, which is not conducive to troubleshooting.
We should add a line of warning type log.
was:
In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs, no
information is recorded here.
try {
..
} catch (IOException e) {
if (e instanceof SocketTimeoutException) {
/*
* writing to client timed out. This happens if the client reads
* part of a block and then decides not to read the rest (but leaves
* the socket open).
*
* Reporting of this case is done in DataXceiver#run
*/
}
}
No records are generated here, which is not conducive to troubleshooting.
We should add a line of warning type log.
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In BlockSender#sendPacket(), if a SocketTimeout exception occurs, no
> information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15932) Improve the balancer error message when process exits abnormally.
[ https://issues.apache.org/jira/browse/HDFS-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312028#comment-17312028 ] Brahma Reddy Battula commented on HDFS-15932: - [~prasad-acit] thanks for patch.. Patch lgtm. > Improve the balancer error message when process exits abnormally. > - > > Key: HDFS-15932 > URL: https://issues.apache.org/jira/browse/HDFS-15932 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HDFS-15932.001.patch > > > The HDFS balancer exits abnormally. The content of the pid file is not > cleaned up, and the new balancer cannot be started. > Start the balancer (start-balancer.sh threshold 5) > Kill the balancer process (kill -9 ) > Re-execute the balancer, there will be an error message. > -- Balancer is running as process . Stop it first. > (But process already stopped, error message can be more detailed) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15939) Solve the problem that DataXceiverServer#run() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15939:
--
Labels: pull-request-available (was: )
> Solve the problem that DataXceiverServer#run() does not record SocketTimeout
> exception
> --
>
> Key: HDFS-15939
> URL: https://issues.apache.org/jira/browse/HDFS-15939
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In DataXceiverServer#run(), if a SocketTimeout exception occurs, no
> information will be recorded here.
> try {
> ..
> } catch (SocketTimeoutException ignored){
> // wake up to see if should continue to run
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add some warning type logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15939) Solve the problem that DataXceiverServer#run() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-15939 started by JiangHua Zhu.
---
> Solve the problem that DataXceiverServer#run() does not record SocketTimeout
> exception
> --
>
> Key: HDFS-15939
> URL: https://issues.apache.org/jira/browse/HDFS-15939
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In DataXceiverServer#run(), if a SocketTimeout exception occurs, no
> information will be recorded here.
> try {
> ..
> } catch (SocketTimeoutException ignored){
> // wake up to see if should continue to run
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add some warning type logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15939) Solve the problem that DataXceiverServer#run() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15939?focusedWorklogId=574550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574550
]
ASF GitHub Bot logged work on HDFS-15939:
-
Author: ASF GitHub Bot
Created on: 31/Mar/21 03:30
Start Date: 31/Mar/21 03:30
Worklog Time Spent: 10m
Work Description: jianghuazhu opened a new pull request #2841:
URL: https://github.com/apache/hadoop/pull/2841
…cord SocketTimeout exception.
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.)
For more details, please see
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 574550)
Remaining Estimate: 0h
Time Spent: 10m
> Solve the problem that DataXceiverServer#run() does not record SocketTimeout
> exception
> --
>
> Key: HDFS-15939
> URL: https://issues.apache.org/jira/browse/HDFS-15939
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In DataXceiverServer#run(), if a SocketTimeout exception occurs, no
> information will be recorded here.
> try {
> ..
> } catch (SocketTimeoutException ignored){
> // wake up to see if should continue to run
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add some warning type logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14383) Compute datanode load based on StoragePolicy
[
https://issues.apache.org/jira/browse/HDFS-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14383:
---
Fix Version/s: 3.3.1
> Compute datanode load based on StoragePolicy
>
>
> Key: HDFS-14383
> URL: https://issues.apache.org/jira/browse/HDFS-14383
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.3, 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-14383-01.patch, HDFS-14383-02.patch
>
>
> Datanode load check logic needs to be changed because existing computation
> will not consider StoragePolicy.
> DatanodeManager#getInServiceXceiverAverage
> {code}
> public double getInServiceXceiverAverage() {
> double avgLoad = 0;
> final int nodes = getNumDatanodesInService();
> if (nodes != 0) {
> final int xceivers = heartbeatManager
> .getInServiceXceiverCount();
> avgLoad = (double)xceivers/nodes;
> }
> return avgLoad;
> }
> {code}
>
> For example: with 10 nodes (HOT), average 50 xceivers and 90 nodes (COLD)
> with average 10 xceivers the calculated threshold by the NN is 28 (((500 +
> 900)/100)*2), which means those 10 nodes (the whole HOT tier) becomes
> unavailable when the COLD tier nodes are barely in use. Turning this check
> off helps to mitigate this issue, however the
> dfs.namenode.replication.considerLoad helps to "balance" the load of the DNs,
> upon turning it off can lead to situations where specific DNs are
> "overloaded".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15253) Set default throttle value on dfs.image.transfer.bandwidthPerSec
[ https://issues.apache.org/jira/browse/HDFS-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-15253: --- Fix Version/s: 3.3.1 > Set default throttle value on dfs.image.transfer.bandwidthPerSec > > > Key: HDFS-15253 > URL: https://issues.apache.org/jira/browse/HDFS-15253 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can > use maximum available bandwidth for fsimage transfers during checkpoint. I > think we should throttle this. Many users were experienced namenode failover > when transferring large image size along with fsimage replication on > dfs.namenode.name.dir. eg. >25Gb. > Thought to set, > dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s) > dfs.namenode.checkpoint.txns=200 (Default is 1M, good to avoid frequent > checkpoint. However, the default checkpoint runs every 6 hours once) > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15921) Improve the log for the Storage Policy Operations
[ https://issues.apache.org/jira/browse/HDFS-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312011#comment-17312011 ] Xiaoqiao He commented on HDFS-15921: +1 on [^HDFS-15921.001.patch]. Thanks [~bpatel] for your works. Will commit shortly if no other comments. > Improve the log for the Storage Policy Operations > - > > Key: HDFS-15921 > URL: https://issues.apache.org/jira/browse/HDFS-15921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Minor > Attachments: HDFS-15921.001.patch > > > Improve the log for the Storage Policy Operations -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15506) [JDK 11] Fix javadoc errors in hadoop-hdfs module
[
https://issues.apache.org/jira/browse/HDFS-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-15506:
---
Fix Version/s: 3.3.1
> [JDK 11] Fix javadoc errors in hadoop-hdfs module
> -
>
> Key: HDFS-15506
> URL: https://issues.apache.org/jira/browse/HDFS-15506
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Xieming Li
>Priority: Major
> Labels: newbie
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-15506.001.patch, HDFS-15506.002.patch
>
>
> {noformat}
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:43:
> error: self-closing element not allowed
> [ERROR] *
> [ERROR]^
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:682:
> error: malformed HTML
> [ERROR]* a NameNode per second. Values <= 0 disable throttling. This
> affects
> [ERROR]^
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:1780:
> error: exception not thrown: java.io.FileNotFoundException
> [ERROR]* @throws FileNotFoundException
> [ERROR] ^
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java:176:
> error: @param name not found
> [ERROR]* @param mtime The snapshot creation time set by Time.now().
> [ERROR] ^
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java:2187:
> error: exception not thrown: java.lang.Exception
> [ERROR]* @exception Exception if the filesystem does not exist.
> [ERROR] ^
> {noformat}
> Full error log:
> https://gist.github.com/aajisaka/a0c16f0408a623e798dd7df29fbddf82
> How to reproduce the failure:
> * Remove {{true}} from pom.xml
> * Run {{mvn process-sources javadoc:javadoc-no-fork}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15508) [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module
[
https://issues.apache.org/jira/browse/HDFS-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-15508:
---
Fix Version/s: 3.3.1
> [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module
> -
>
> Key: HDFS-15508
> URL: https://issues.apache.org/jira/browse/HDFS-15508
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: newbie
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-15508.01.patch
>
>
> {noformat}
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/package-info.java:21:
> error: reference not found
> [ERROR] * Implementations should extend {@link
> AbstractDelegationTokenSecretManager}.
> [ERROR] ^
> {noformat}
> Full error log:
> https://gist.github.com/aajisaka/a7dde76a4ba2942f60bf6230ec9ed6e1
> How to reproduce the failure:
> * Remove {{true}} from pom.xml
> * Run {{mvn process-sources javadoc:javadoc-no-fork}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15331) Remove invalid exclusions that minicluster dependency on HDFS
[ https://issues.apache.org/jira/browse/HDFS-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-15331: --- Fix Version/s: 3.3.1 > Remove invalid exclusions that minicluster dependency on HDFS > - > > Key: HDFS-15331 > URL: https://issues.apache.org/jira/browse/HDFS-15331 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Fix For: 3.3.1, 3.4.0 > > > Ozone has split into independent repo, but the invalid exclusions (kubernetes > client) that minicluster dependency on HDFS is kept. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15507) [JDK 11] Fix javadoc errors in hadoop-hdfs-client module
[
https://issues.apache.org/jira/browse/HDFS-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-15507:
---
Fix Version/s: 3.3.1
> [JDK 11] Fix javadoc errors in hadoop-hdfs-client module
>
>
> Key: HDFS-15507
> URL: https://issues.apache.org/jira/browse/HDFS-15507
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Xieming Li
>Priority: Major
> Labels: newbie
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-15507.001.patch, HDFS-15507.002.patch
>
>
> {noformat}
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java:32:
> error: self-closing element not allowed
> [ERROR] *
> [ERROR]^
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1245:
> error: unexpected text
> [ERROR]* Same as {@link #create(String, FsPermission, EnumSet, boolean,
> short, long,
> [ERROR] ^
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java:161:
> error: reference not found
> [ERROR]* {@link HdfsConstants#LEASE_HARDLIMIT_PERIOD hard limit}. Until
> the
> [ERROR] ^
> {noformat}
> Full error log:
> https://gist.github.com/aajisaka/7ab1c48a9bd7a0fdb11fa82eb04874d5
> How to reproduce the failure:
> * Remove {{true}} from pom.xml
> * Run {{mvn process-sources javadoc:javadoc-no-fork}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15651) Client could not obtain block when DN CommandProcessingThread exit
[
https://issues.apache.org/jira/browse/HDFS-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-15651:
---
Fix Version/s: 3.3.1
> Client could not obtain block when DN CommandProcessingThread exit
> --
>
> Key: HDFS-15651
> URL: https://issues.apache.org/jira/browse/HDFS-15651
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Aiphago
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-15651.001.patch, HDFS-15651.002.patch,
> HDFS-15651.patch
>
>
> In our cluster, we applied the HDFS-14997 improvement.
> We find one case that CommandProcessingThread will exit due to OOM error.
> OOM error was caused by our one abnormal application that running on this DN
> node.
> {noformat}
> 2020-10-18 10:27:12,604 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: Command processor
> encountered fatal exception and exit.
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:717)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:173)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:222)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2005)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:671)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:617)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1247)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.access$1000(BPServiceActor.java:1194)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread$3.run(BPServiceActor.java:1299)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1221)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1208)
> {noformat}
> Here the main point is that CommandProcessingThread crashed will lead a very
> bad impact. All the NN response commands will not be processed by DN side.
> We enabled the block token to access the data, but here the DN command
> DNA_ACCESSKEYUPDATE is not processed on time by DN. And then we see lots of
> Sasl error due to key expiration in DN log:
> {noformat}
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=xxx, keyId=xx,
> userId=xxx, blockPoolId=, blockId=xxx, access modes=[READ]), since the
> required block key (keyID=xxx) doesn't exist.]
> {noformat}
>
> For the impact in client side, our users receive lots of 'could not obtain
> block' error with BlockMissingException.
> CommandProcessingThread is a critical thread, it should always be running.
> {code:java}
> /**
>* CommandProcessingThread that process commands asynchronously.
>*/
> class CommandProcessingThread extends Thread {
> private final BPServiceActor actor;
> private final BlockingQueue queue;
> ...
> @Override
> public void run() {
> try {
> processQueue();
> } catch (Throwable t) {
> LOG.error("{} encountered fatal exception and exit.", getName(), t);
> <=== should not exit this thread
> }
> }
> {code}
> Once a unexpected error happened, a better handing should be:
> * catch the exception, appropriately deal with the error and let
> processQueue continue to run
> or
> * exit the DN process to let admin user investigate this
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15932) Improve the balancer error message when process exits abnormally.
[ https://issues.apache.org/jira/browse/HDFS-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312004#comment-17312004 ] Xiaoqiao He commented on HDFS-15932: Trigger Jenkins manually: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/561, Let's wait what he says. > Improve the balancer error message when process exits abnormally. > - > > Key: HDFS-15932 > URL: https://issues.apache.org/jira/browse/HDFS-15932 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HDFS-15932.001.patch > > > The HDFS balancer exits abnormally. The content of the pid file is not > cleaned up, and the new balancer cannot be started. > Start the balancer (start-balancer.sh threshold 5) > Kill the balancer process (kill -9 ) > Re-execute the balancer, there will be an error message. > -- Balancer is running as process . Stop it first. > (But process already stopped, error message can be more detailed) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15850) Superuser actions should be reported to external enforcers
[ https://issues.apache.org/jira/browse/HDFS-15850?focusedWorklogId=574541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574541 ] ASF GitHub Bot logged work on HDFS-15850: - Author: ASF GitHub Bot Created on: 31/Mar/21 03:14 Start Date: 31/Mar/21 03:14 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on a change in pull request #2784: URL: https://github.com/apache/hadoop/pull/2784#discussion_r604564623 ## File path: hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml ## @@ -310,4 +310,10 @@ + Review comment: can we revert the formatting change in findbugsExcludeFile.xml as the assert should fix the spotbugs issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 574541) Time Spent: 3h 20m (was: 3h 10m) > Superuser actions should be reported to external enforcers > -- > > Key: HDFS-15850 > URL: https://issues.apache.org/jira/browse/HDFS-15850 > Project: Hadoop HDFS > Issue Type: Task > Components: security >Affects Versions: 3.3.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15850.v1.patch, HDFS-15850.v2.patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently, HDFS superuser checks or actions are not reported to external > enforcers like Ranger and the audit report provided by such external enforces > are not complete and are missing the superuser actions. To fix this, add a > new method to "AccessControlEnforcer" for all superuser checks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15850) Superuser actions should be reported to external enforcers
[
https://issues.apache.org/jira/browse/HDFS-15850?focusedWorklogId=574538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574538
]
ASF GitHub Bot logged work on HDFS-15850:
-
Author: ASF GitHub Bot
Created on: 31/Mar/21 03:11
Start Date: 31/Mar/21 03:11
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2784:
URL: https://github.com/apache/hadoop/pull/2784#issuecomment-810725402
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 2s | | trunk passed |
| +1 :green_heart: | compile | 5m 11s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 4m 56s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 15s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 46s | | the patch passed |
| +1 :green_heart: | compile | 5m 21s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 5m 21s | | the patch passed |
| +1 :green_heart: | compile | 5m 30s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 5m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 33s | | hadoop-hdfs-project: The
patch generated 0 new + 498 unchanged - 6 fixed = 498 total (was 504) |
| +1 :green_heart: | mvnsite | 2m 9s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 347m 8s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2784/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 23m 46s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2784/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 493m 55s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestDFSShell |
| |
[jira] [Commented] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311999#comment-17311999
]
Xiaoqiao He commented on HDFS-15863:
+1 on [^HDFS-15863.005.patch]. Thanks [~prasad-acit].
> RBF: Validation message to be corrected in FairnessPolicyController
> ---
>
> Key: HDFS-15863
> URL: https://issues.apache.org/jira/browse/HDFS-15863
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Minor
> Attachments: HDFS-15863.001.patch, HDFS-15863.002.patch,
> HDFS-15863.003.patch, HDFS-15863.004.patch, HDFS-15863.005.patch
>
>
> org.apache.hadoop.hdfs.server.federation.fairness.StaticRouterRpcFairnessPolicyController#validateCount
> When dfs.federation.router.handler.count is lessthan the total dedicated
> handlers for all NS, then error message shows 0 & -ve values in error
> message, instead can show the actual configured values.
> Current message is : "Available handlers -5 lower than min 0 for nsId nn1"
> This can be changed to: "Configured handlers
> ${DFS_ROUTER_HANDLER_COUNT_KEY}=10 lower than min 15 for nsId nn1", where 10
> is hander count & 15 is sum of dedicated handler count.
> Related to: HDFS-14090
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15939) Solve the problem that DataXceiverServer#run() does not record SocketTimeout exception
JiangHua Zhu created HDFS-15939:
---
Summary: Solve the problem that DataXceiverServer#run() does not
record SocketTimeout exception
Key: HDFS-15939
URL: https://issues.apache.org/jira/browse/HDFS-15939
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: JiangHua Zhu
In DataXceiverServer#run(), if a SocketTimeout exception occurs, no information
will be recorded here.
try {
..
} catch (SocketTimeoutException ignored){
// wake up to see if should continue to run
}
No records are generated here, which is not conducive to troubleshooting.
We should add some warning type logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15939) Solve the problem that DataXceiverServer#run() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu reassigned HDFS-15939:
---
Assignee: JiangHua Zhu
> Solve the problem that DataXceiverServer#run() does not record SocketTimeout
> exception
> --
>
> Key: HDFS-15939
> URL: https://issues.apache.org/jira/browse/HDFS-15939
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>
> In DataXceiverServer#run(), if a SocketTimeout exception occurs, no
> information will be recorded here.
> try {
> ..
> } catch (SocketTimeoutException ignored){
> // wake up to see if should continue to run
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add some warning type logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15921) Improve the log for the Storage Policy Operations
[ https://issues.apache.org/jira/browse/HDFS-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311976#comment-17311976 ] Bhavik Patel commented on HDFS-15921: - [~hemanthboyina] & [~tasanuma] please review > Improve the log for the Storage Policy Operations > - > > Key: HDFS-15921 > URL: https://issues.apache.org/jira/browse/HDFS-15921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Minor > Attachments: HDFS-15921.001.patch > > > Improve the log for the Storage Policy Operations -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?focusedWorklogId=574516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574516
]
ASF GitHub Bot logged work on HDFS-15936:
-
Author: ASF GitHub Bot
Created on: 31/Mar/21 02:08
Start Date: 31/Mar/21 02:08
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on pull request #2836:
URL: https://github.com/apache/hadoop/pull/2836#issuecomment-810702621
@virajjasani, @cxorm , thank you for your attention.
I have submitted some new code, can you review it for me?
Looking forward to your feedback again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 574516)
Time Spent: 1h 10m (was: 1h)
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15937) Reduce memory used during datanode layout upgrade
[
https://issues.apache.org/jira/browse/HDFS-15937?focusedWorklogId=574460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574460
]
ASF GitHub Bot logged work on HDFS-15937:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 23:40
Start Date: 30/Mar/21 23:40
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2838:
URL: https://github.com/apache/hadoop/pull/2838#issuecomment-810647490
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 30 unchanged - 3
fixed = 30 total (was 33) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 227m 53s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2838/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 312m 54s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2838/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0359a8b54474 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6c30ed2108877a3e628a4f52b4e4578f769da7f7 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK
[jira] [Work logged] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[
https://issues.apache.org/jira/browse/HDFS-15887?focusedWorklogId=574426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574426
]
ASF GitHub Bot logged work on HDFS-15887:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 21:24
Start Date: 30/Mar/21 21:24
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2839:
URL: https://github.com/apache/hadoop/pull/2839#issuecomment-810587475
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2839/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 26 unchanged -
0 fixed = 27 total (was 26) |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 358m 30s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2839/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 463m 20s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| |
[jira] [Work logged] (HDFS-15909) Make fnmatch cross platform
[
https://issues.apache.org/jira/browse/HDFS-15909?focusedWorklogId=574414=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574414
]
ASF GitHub Bot logged work on HDFS-15909:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 21:00
Start Date: 30/Mar/21 21:00
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2792:
URL: https://github.com/apache/hadoop/pull/2792#issuecomment-810574297
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 30s | | trunk passed |
| +1 :green_heart: | compile | 2m 43s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 40s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 51m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 2m 29s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | cc | 2m 29s |
[/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2792/7/artifact/out/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 4 new + 42 unchanged
- 4 fixed = 46 total (was 46) |
| +1 :green_heart: | golang | 2m 29s | | the patch passed |
| +1 :green_heart: | javac | 2m 29s | | the patch passed |
| +1 :green_heart: | compile | 2m 31s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | cc | 2m 31s | | the patch passed |
| +1 :green_heart: | golang | 2m 31s | | the patch passed |
| +1 :green_heart: | javac | 2m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 108m 22s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 182m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2792/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2792 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux e1d60575eb80 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c4d3ed7ae36d142518451c9e7ac4e1136907dbc8 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2792/7/testReport/ |
| Max. process+thread count | 604 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2792/7/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by |
[jira] [Updated] (HDFS-15927) Catch polymorphic type by reference
[
https://issues.apache.org/jira/browse/HDFS-15927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15927:
---
Affects Version/s: (was: 3.4.0)
> Catch polymorphic type by reference
> ---
>
> Key: HDFS-15927
> URL: https://issues.apache.org/jira/browse/HDFS-15927
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Need to catch polymorphic exception types by reference in order to realize
> the polymorphic usage, if any. Otherwise, the functionality of the caught
> object is restricted to only that of the base class.
> Also, the following warning gets reported since it's currently caught by
> value -
> {code:java}
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:138:22:
> warning: catching polymorphic type 'const class hdfs::uri_parse_error' by
> value [-Wcatch-value=]
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:151:27:
> warning: catching polymorphic type 'struct hdfs::ha_parse_error' by value
> [-Wcatch-value=]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15927) Catch polymorphic type by reference
[
https://issues.apache.org/jira/browse/HDFS-15927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri resolved HDFS-15927.
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Catch polymorphic type by reference
> ---
>
> Key: HDFS-15927
> URL: https://issues.apache.org/jira/browse/HDFS-15927
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Need to catch polymorphic exception types by reference in order to realize
> the polymorphic usage, if any. Otherwise, the functionality of the caught
> object is restricted to only that of the base class.
> Also, the following warning gets reported since it's currently caught by
> value -
> {code:java}
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:138:22:
> warning: catching polymorphic type 'const class hdfs::uri_parse_error' by
> value [-Wcatch-value=]
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:151:27:
> warning: catching polymorphic type 'struct hdfs::ha_parse_error' by value
> [-Wcatch-value=]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15927) Catch polymorphic type by reference
[
https://issues.apache.org/jira/browse/HDFS-15927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311801#comment-17311801
]
Íñigo Goiri commented on HDFS-15927:
Thanks [~gautham] for the fix.
Merged PR 2824.
> Catch polymorphic type by reference
> ---
>
> Key: HDFS-15927
> URL: https://issues.apache.org/jira/browse/HDFS-15927
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Need to catch polymorphic exception types by reference in order to realize
> the polymorphic usage, if any. Otherwise, the functionality of the caught
> object is restricted to only that of the base class.
> Also, the following warning gets reported since it's currently caught by
> value -
> {code:java}
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:138:22:
> warning: catching polymorphic type 'const class hdfs::uri_parse_error' by
> value [-Wcatch-value=]
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:151:27:
> warning: catching polymorphic type 'struct hdfs::ha_parse_error' by value
> [-Wcatch-value=]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15929) Replace RAND_pseudo_bytes in util.cc
[
https://issues.apache.org/jira/browse/HDFS-15929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15929:
---
Affects Version/s: (was: 3.4.0)
> Replace RAND_pseudo_bytes in util.cc
>
>
> Key: HDFS-15929
> URL: https://issues.apache.org/jira/browse/HDFS-15929
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following
> warning during compilation that it's deprecated -
> {code}
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:
> In function ‘std::string hdfs::GetRandomClientName()’:
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
> warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated
> [-Wdeprecated-declarations]
>78 | RAND_pseudo_bytes([0], 8);
> | ^
> In file included from /usr/include/openssl/e_os2.h:13,
> from /usr/include/openssl/ossl_typ.h:19,
> from /usr/include/openssl/rand.h:14,
> from
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h:29,
> from
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:19:
> /usr/include/openssl/rand.h:44:1: note: declared here
>44 | DEPRECATEDIN_1_1_0(int RAND_pseudo_bytes(unsigned char *buf, int num))
> | ^~
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
> warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated
> [-Wdeprecated-declarations]
>78 | RAND_pseudo_bytes([0], 8);
> | ^
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15929) Replace RAND_pseudo_bytes in util.cc
[
https://issues.apache.org/jira/browse/HDFS-15929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311798#comment-17311798
]
Íñigo Goiri commented on HDFS-15929:
Thanks [~gautham] for the fix.
Merged PR 2826.
> Replace RAND_pseudo_bytes in util.cc
>
>
> Key: HDFS-15929
> URL: https://issues.apache.org/jira/browse/HDFS-15929
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following
> warning during compilation that it's deprecated -
> {code}
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:
> In function ‘std::string hdfs::GetRandomClientName()’:
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
> warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated
> [-Wdeprecated-declarations]
>78 | RAND_pseudo_bytes([0], 8);
> | ^
> In file included from /usr/include/openssl/e_os2.h:13,
> from /usr/include/openssl/ossl_typ.h:19,
> from /usr/include/openssl/rand.h:14,
> from
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h:29,
> from
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:19:
> /usr/include/openssl/rand.h:44:1: note: declared here
>44 | DEPRECATEDIN_1_1_0(int RAND_pseudo_bytes(unsigned char *buf, int num))
> | ^~
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
> warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated
> [-Wdeprecated-declarations]
>78 | RAND_pseudo_bytes([0], 8);
> | ^
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15929) Replace RAND_pseudo_bytes in util.cc
[
https://issues.apache.org/jira/browse/HDFS-15929?focusedWorklogId=574404=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574404
]
ASF GitHub Bot logged work on HDFS-15929:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 20:36
Start Date: 30/Mar/21 20:36
Worklog Time Spent: 10m
Work Description: goiri merged pull request #2826:
URL: https://github.com/apache/hadoop/pull/2826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 574404)
Time Spent: 1h 40m (was: 1.5h)
> Replace RAND_pseudo_bytes in util.cc
>
>
> Key: HDFS-15929
> URL: https://issues.apache.org/jira/browse/HDFS-15929
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following
> warning during compilation that it's deprecated -
> {code}
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:
> In function ‘std::string hdfs::GetRandomClientName()’:
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
> warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated
> [-Wdeprecated-declarations]
>78 | RAND_pseudo_bytes([0], 8);
> | ^
> In file included from /usr/include/openssl/e_os2.h:13,
> from /usr/include/openssl/ossl_typ.h:19,
> from /usr/include/openssl/rand.h:14,
> from
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h:29,
> from
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:19:
> /usr/include/openssl/rand.h:44:1: note: declared here
>44 | DEPRECATEDIN_1_1_0(int RAND_pseudo_bytes(unsigned char *buf, int num))
> | ^~
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
> warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated
> [-Wdeprecated-declarations]
>78 | RAND_pseudo_bytes([0], 8);
> | ^
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15929) Replace RAND_pseudo_bytes in util.cc
[
https://issues.apache.org/jira/browse/HDFS-15929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri resolved HDFS-15929.
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Replace RAND_pseudo_bytes in util.cc
>
>
> Key: HDFS-15929
> URL: https://issues.apache.org/jira/browse/HDFS-15929
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following
> warning during compilation that it's deprecated -
> {code}
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:
> In function ‘std::string hdfs::GetRandomClientName()’:
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
> warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated
> [-Wdeprecated-declarations]
>78 | RAND_pseudo_bytes([0], 8);
> | ^
> In file included from /usr/include/openssl/e_os2.h:13,
> from /usr/include/openssl/ossl_typ.h:19,
> from /usr/include/openssl/rand.h:14,
> from
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h:29,
> from
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:19:
> /usr/include/openssl/rand.h:44:1: note: declared here
>44 | DEPRECATEDIN_1_1_0(int RAND_pseudo_bytes(unsigned char *buf, int num))
> | ^~
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
> warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated
> [-Wdeprecated-declarations]
>78 | RAND_pseudo_bytes([0], 8);
> | ^
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15927) Catch polymorphic type by reference
[
https://issues.apache.org/jira/browse/HDFS-15927?focusedWorklogId=574403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574403
]
ASF GitHub Bot logged work on HDFS-15927:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 20:35
Start Date: 30/Mar/21 20:35
Worklog Time Spent: 10m
Work Description: goiri merged pull request #2824:
URL: https://github.com/apache/hadoop/pull/2824
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 574403)
Time Spent: 1h (was: 50m)
> Catch polymorphic type by reference
> ---
>
> Key: HDFS-15927
> URL: https://issues.apache.org/jira/browse/HDFS-15927
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Need to catch polymorphic exception types by reference in order to realize
> the polymorphic usage, if any. Otherwise, the functionality of the caught
> object is restricted to only that of the base class.
> Also, the following warning gets reported since it's currently caught by
> value -
> {code:java}
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:138:22:
> warning: catching polymorphic type 'const class hdfs::uri_parse_error' by
> value [-Wcatch-value=]
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:151:27:
> warning: catching polymorphic type 'struct hdfs::ha_parse_error' by value
> [-Wcatch-value=]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15931) Fix non-static inner classes for better memory management
[
https://issues.apache.org/jira/browse/HDFS-15931?focusedWorklogId=574372=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574372
]
ASF GitHub Bot logged work on HDFS-15931:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 19:56
Start Date: 30/Mar/21 19:56
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2830:
URL: https://github.com/apache/hadoop/pull/2830#issuecomment-810537015
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 40s | | trunk passed |
| +1 :green_heart: | compile | 5m 10s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 5m 30s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 36s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 45s | | the patch passed |
| +1 :green_heart: | compile | 4m 57s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 4m 57s | | the patch passed |
| +1 :green_heart: | compile | 4m 39s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 4m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2830/3/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 4 new + 225 unchanged - 9 fixed =
229 total (was 234) |
| +1 :green_heart: | mvnsite | 1m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 6s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 358m 42s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2830/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 25m 1s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2830/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 504m 42s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.datanode.TestIncrementalBrVariations |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
[jira] [Work logged] (HDFS-15929) Replace RAND_pseudo_bytes in util.cc
[
https://issues.apache.org/jira/browse/HDFS-15929?focusedWorklogId=574367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574367
]
ASF GitHub Bot logged work on HDFS-15929:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 19:48
Start Date: 30/Mar/21 19:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2826:
URL: https://github.com/apache/hadoop/pull/2826#issuecomment-810532265
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 20s | | trunk passed |
| +1 :green_heart: | compile | 2m 38s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 44s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 51m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 2m 29s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | cc | 2m 29s | |
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 40 unchanged
- 6 fixed = 40 total (was 46) |
| +1 :green_heart: | golang | 2m 29s | | the patch passed |
| +1 :green_heart: | javac | 2m 29s | | the patch passed |
| +1 :green_heart: | compile | 2m 31s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | cc | 2m 31s | |
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new
+ 40 unchanged - 6 fixed = 40 total (was 46) |
| +1 :green_heart: | golang | 2m 31s | | the patch passed |
| +1 :green_heart: | javac | 2m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 59s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 105m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2826/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2826 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux f884c103d0df 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6befd42c75252f0897888e732dcff65cc55e1570 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2826/5/testReport/ |
| Max. process+thread count | 626 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2826/5/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was
[jira] [Work logged] (HDFS-15927) Catch polymorphic type by reference
[
https://issues.apache.org/jira/browse/HDFS-15927?focusedWorklogId=574366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574366
]
ASF GitHub Bot logged work on HDFS-15927:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 19:45
Start Date: 30/Mar/21 19:45
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2824:
URL: https://github.com/apache/hadoop/pull/2824#issuecomment-810530775
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 34s | | trunk passed |
| +1 :green_heart: | compile | 2m 39s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 38s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 51m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 2m 29s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | cc | 2m 29s | |
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 44 unchanged
- 2 fixed = 44 total (was 46) |
| +1 :green_heart: | golang | 2m 29s | | the patch passed |
| +1 :green_heart: | javac | 2m 29s | | the patch passed |
| +1 :green_heart: | compile | 2m 30s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | cc | 2m 30s | |
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new
+ 44 unchanged - 2 fixed = 44 total (was 46) |
| +1 :green_heart: | golang | 2m 30s | | the patch passed |
| +1 :green_heart: | javac | 2m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 35m 1s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 108m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2824/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2824 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 012cc96289c9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0b9284d1c3b1e85a54d403493dcd933b222beb8f |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2824/3/testReport/ |
| Max. process+thread count | 541 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2824/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-15932) Improve the balancer error message when process exits abnormally.
[ https://issues.apache.org/jira/browse/HDFS-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311744#comment-17311744 ] Renukaprasad C commented on HDFS-15932: --- Message change in case of abnormal balancer process exit & try to start the balancer for next time. No UT change required. Build failure is not related to the changes done. [~brahma] can you plz look into the patch when you find time? > Improve the balancer error message when process exits abnormally. > - > > Key: HDFS-15932 > URL: https://issues.apache.org/jira/browse/HDFS-15932 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HDFS-15932.001.patch > > > The HDFS balancer exits abnormally. The content of the pid file is not > cleaned up, and the new balancer cannot be started. > Start the balancer (start-balancer.sh threshold 5) > Kill the balancer process (kill -9 ) > Re-execute the balancer, there will be an error message. > -- Balancer is running as process . Stop it first. > (But process already stopped, error message can be more detailed) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15938) Fix java doc in FSEditLog
[
https://issues.apache.org/jira/browse/HDFS-15938?focusedWorklogId=574310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574310
]
ASF GitHub Bot logged work on HDFS-15938:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 18:28
Start Date: 30/Mar/21 18:28
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2837:
URL: https://github.com/apache/hadoop/pull/2837#issuecomment-810481589
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 12s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2837/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 127 unchanged
- 0 fixed = 128 total (was 127) |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 330m 22s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2837/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 426m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[jira] [Updated] (HDFS-15928) Replace RAND_pseudo_bytes in rpc_engine.cc
[
https://issues.apache.org/jira/browse/HDFS-15928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15928:
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Affects Version/s: (was: 3.4.0)
Status: Patch Available (was: Open)
> Replace RAND_pseudo_bytes in rpc_engine.cc
> --
>
> Key: HDFS-15928
> URL: https://issues.apache.org/jira/browse/HDFS-15928
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following
> warning during compilation that it's deprecated -
> {code}
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc:124:40:
> warning: 'int RAND_pseudo_bytes(unsigned char*, int)' is deprecated
> [-Wdeprecated-declarations]
> [WARNING] from
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc:18:
> [WARNING] /usr/include/openssl/rand.h:44:1: note: declared here
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15928) Replace RAND_pseudo_bytes in rpc_engine.cc
[
https://issues.apache.org/jira/browse/HDFS-15928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15928:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Replace RAND_pseudo_bytes in rpc_engine.cc
> --
>
> Key: HDFS-15928
> URL: https://issues.apache.org/jira/browse/HDFS-15928
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following
> warning during compilation that it's deprecated -
> {code}
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc:124:40:
> warning: 'int RAND_pseudo_bytes(unsigned char*, int)' is deprecated
> [-Wdeprecated-declarations]
> [WARNING] from
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc:18:
> [WARNING] /usr/include/openssl/rand.h:44:1: note: declared here
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15928) Replace RAND_pseudo_bytes in rpc_engine.cc
[
https://issues.apache.org/jira/browse/HDFS-15928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311683#comment-17311683
]
Íñigo Goiri commented on HDFS-15928:
Thanks [~gautham] for the patch.
Merged PR 2825.
> Replace RAND_pseudo_bytes in rpc_engine.cc
> --
>
> Key: HDFS-15928
> URL: https://issues.apache.org/jira/browse/HDFS-15928
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following
> warning during compilation that it's deprecated -
> {code}
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc:124:40:
> warning: 'int RAND_pseudo_bytes(unsigned char*, int)' is deprecated
> [-Wdeprecated-declarations]
> [WARNING] from
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc:18:
> [WARNING] /usr/include/openssl/rand.h:44:1: note: declared here
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15928) Replace RAND_pseudo_bytes in rpc_engine.cc
[
https://issues.apache.org/jira/browse/HDFS-15928?focusedWorklogId=574280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574280
]
ASF GitHub Bot logged work on HDFS-15928:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 17:37
Start Date: 30/Mar/21 17:37
Worklog Time Spent: 10m
Work Description: goiri merged pull request #2825:
URL: https://github.com/apache/hadoop/pull/2825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 574280)
Time Spent: 50m (was: 40m)
> Replace RAND_pseudo_bytes in rpc_engine.cc
> --
>
> Key: HDFS-15928
> URL: https://issues.apache.org/jira/browse/HDFS-15928
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following
> warning during compilation that it's deprecated -
> {code}
> [WARNING]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc:124:40:
> warning: 'int RAND_pseudo_bytes(unsigned char*, int)' is deprecated
> [-Wdeprecated-declarations]
> [WARNING] from
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc:18:
> [WARNING] /usr/include/openssl/rand.h:44:1: note: declared here
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15937) Reduce memory used during datanode layout upgrade
[
https://issues.apache.org/jira/browse/HDFS-15937?focusedWorklogId=574251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574251
]
ASF GitHub Bot logged work on HDFS-15937:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 16:41
Start Date: 30/Mar/21 16:41
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2838:
URL: https://github.com/apache/hadoop/pull/2838#issuecomment-810412030
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 12s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 55s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2838/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 31 unchanged -
2 fixed = 32 total (was 33) |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 232m 30s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2838/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 318m 50s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2838/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 70d639a759ad 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git
[jira] [Work started] (HDFS-15931) Fix non-static inner classes for better memory management
[ https://issues.apache.org/jira/browse/HDFS-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15931 started by Viraj Jasani. --- > Fix non-static inner classes for better memory management > - > > Key: HDFS-15931 > URL: https://issues.apache.org/jira/browse/HDFS-15931 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > If an inner class does not need to reference its enclosing instance, it can > be static. This prevents a common cause of memory leaks and uses less memory > per instance of the enclosing class. > Came across DataNodeProperties as a non static inner class defined in > MiniDFSCluster without holding any implicit reference to MiniDFSCluster. > Taking this opportunity to find other non-static inner classes that are not > holding implicit reference to their respective enclosing instances. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15931) Fix non-static inner classes for better memory management
[ https://issues.apache.org/jira/browse/HDFS-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HDFS-15931: Status: Patch Available (was: In Progress) > Fix non-static inner classes for better memory management > - > > Key: HDFS-15931 > URL: https://issues.apache.org/jira/browse/HDFS-15931 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > If an inner class does not need to reference its enclosing instance, it can > be static. This prevents a common cause of memory leaks and uses less memory > per instance of the enclosing class. > Came across DataNodeProperties as a non static inner class defined in > MiniDFSCluster without holding any implicit reference to MiniDFSCluster. > Taking this opportunity to find other non-static inner classes that are not > holding implicit reference to their respective enclosing instances. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15923) RBF: Authentication failed when rename accross sub clusters
[
https://issues.apache.org/jira/browse/HDFS-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311555#comment-17311555
]
Jinglun commented on HDFS-15923:
Hi [~zhengzhuobinzzb], thanks your question !
In the current design the journal and the distcp procedure are all done with
the Router's kerberos credential (a super user). Both the journal path and the
yarn queue are configured by the administrator. The super user's credential is
also used for preserving all the permissions in distcp. So we shouldn't use the
user's ugi. The user's ugi won't have write access of the journal path. The ugi
doesn't have access of the super user's yarn queue too.
But there is an issue about the user's ugi: "The Router doesn't do any
permission check before doing the Router Federation Rename". We should check
both the source and the dst with the user's ugi before submitting the Balance
Job.
Let me know your thoughts. If you also agree with the permission issue, are you
interested in fixing it ?
> RBF: Authentication failed when rename accross sub clusters
>
>
> Key: HDFS-15923
> URL: https://issues.apache.org/jira/browse/HDFS-15923
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: zhuobin zheng
>Priority: Major
> Labels: RBF, pull-request-available, rename
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Rename accross subcluster with RBF and Kerberos environment. Will encounter
> the following two errors:
> # Save Object to journal.
> # Precheck try to get src file status
> So, we need use Proxy UGI doAs create DistcpProcedure and TrashProcedure and
> submit Job.
> In patch i use proxy ugi doAs above method. It worked.
> But there are another strange thing and this patch not solve:
> Router use ugi itself to submit the Distcp job. But not user ugi or proxy
> ugi. This may cause excessive distcp permissions.
> First: Save Object to journal.
> {code:java}
> // code placeholder
> 2021-03-23 14:01:16,233 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:408)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
> at
> org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
> at org.apache.hadoop.ipc.Client.call(Client.java:1452)
> at org.apache.hadoop.ipc.Client.call(Client.java:1405)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy11.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:376)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy12.create(Unknown Source)
> at
>
[jira] [Work logged] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?focusedWorklogId=574162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574162 ] ASF GitHub Bot logged work on HDFS-15887: - Author: ASF GitHub Bot Created on: 30/Mar/21 13:39 Start Date: 30/Mar/21 13:39 Worklog Time Spent: 10m Work Description: jianghuazhu opened a new pull request #2839: URL: https://github.com/apache/hadoop/pull/2839 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 574162) Remaining Estimate: 0h Time Spent: 10m > Make LogRoll and TailEdits execute in parallel > -- > > Key: HDFS-15887 > URL: https://issues.apache.org/jira/browse/HDFS-15887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: edit_files.jpg > > Time Spent: 10m > Remaining Estimate: 0h > > In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, > and when a checkpoint occurs, it will compete with TailEdits for lock > (FSNamesystem#cpLock). > Usually, it takes a long time to execute checkpoint, which will cause the > size of the generated edit log file to be relatively large. > For example, here is an actual effect: > The StandbyCheckpointer log is triggered as follows : edit_files.jpg > 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State > Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering > checkpoint because there have been 5142154 txns since the last checkpoint, > which exceeds the configured threshold 100 > When loading an edit log with a large amount of data, the processing time > will be longer. We should make the edit log size as even as possible, which > is good for the operation of the system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15887: -- Labels: pull-request-available (was: ) > Make LogRoll and TailEdits execute in parallel > -- > > Key: HDFS-15887 > URL: https://issues.apache.org/jira/browse/HDFS-15887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Attachments: edit_files.jpg > > Time Spent: 10m > Remaining Estimate: 0h > > In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, > and when a checkpoint occurs, it will compete with TailEdits for lock > (FSNamesystem#cpLock). > Usually, it takes a long time to execute checkpoint, which will cause the > size of the generated edit log file to be relatively large. > For example, here is an actual effect: > The StandbyCheckpointer log is triggered as follows : edit_files.jpg > 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State > Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering > checkpoint because there have been 5142154 txns since the last checkpoint, > which exceeds the configured threshold 100 > When loading an edit log with a large amount of data, the processing time > will be longer. We should make the edit log size as even as possible, which > is good for the operation of the system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15887 started by JiangHua Zhu. --- > Make LogRoll and TailEdits execute in parallel > -- > > Key: HDFS-15887 > URL: https://issues.apache.org/jira/browse/HDFS-15887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Attachments: edit_files.jpg > > Time Spent: 10m > Remaining Estimate: 0h > > In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, > and when a checkpoint occurs, it will compete with TailEdits for lock > (FSNamesystem#cpLock). > Usually, it takes a long time to execute checkpoint, which will cause the > size of the generated edit log file to be relatively large. > For example, here is an actual effect: > The StandbyCheckpointer log is triggered as follows : edit_files.jpg > 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State > Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering > checkpoint because there have been 5142154 txns since the last checkpoint, > which exceeds the configured threshold 100 > When loading an edit log with a large amount of data, the processing time > will be longer. We should make the edit log size as even as possible, which > is good for the operation of the system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15253) Set default throttle value on dfs.image.transfer.bandwidthPerSec
[ https://issues.apache.org/jira/browse/HDFS-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-15253: --- Release Note: The configuration dfs.image.transfer.bandwidthPerSec which defines the maximum bandwidth available for fsimage transfer is changed from 0 (meaning no throttle at all) to 50MB/s. > Set default throttle value on dfs.image.transfer.bandwidthPerSec > > > Key: HDFS-15253 > URL: https://issues.apache.org/jira/browse/HDFS-15253 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can > use maximum available bandwidth for fsimage transfers during checkpoint. I > think we should throttle this. Many users were experienced namenode failover > when transferring large image size along with fsimage replication on > dfs.namenode.name.dir. eg. >25Gb. > Thought to set, > dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s) > dfs.namenode.checkpoint.txns=200 (Default is 1M, good to avoid frequent > checkpoint. However, the default checkpoint runs every 6 hours once) > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15646) Track failing tests in HDFS
[
https://issues.apache.org/jira/browse/HDFS-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311520#comment-17311520
]
Ahmed Hussein commented on HDFS-15646:
--
Thanks [~ayushtkn],
Last time I checked the qbt-reports , there were only the dynamo test failing.
Do you know if the daily build runs all the unit tests? or only the modules
with the delta commits?
If the latter, then the frequency of getting OOM depends on the commits (which
modules were triggered everyday).
When there are a large number of unit tests to execute, then Some guesses:
* Too many threads are running. There could be some unit tests that make this
problem worse.
* The entropy is not enough. this leads to threads stalling. See HADOOP-16810.
I tried to set the JVM parameters for tests related to intermediate data
encryption, but I don't think that this is enough.
> Track failing tests in HDFS
> ---
>
> Key: HDFS-15646
> URL: https://issues.apache.org/jira/browse/HDFS-15646
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Priority: Blocker
>
> There are several Units that are consistently failing on Yetus for a log
> period of time.
> The list keeps growing and it is driving the repository into unstable
> status. Qbt reports more than *40 failing unit tests* on average.
> Personally, over the last week, with every submitted patch, I have to spend a
> considerable time looking at the same stack trace to double check whether or
> not the patch contributes to those failures.
> I found out that the majority of those tests were failing for quite sometime
> but +no Jiras were filed+.
> The main problem of those consistent failures is that they have side effect
> on the runtime of the other Junits by sucking up resources such as memory and
> ports.
> {{StripedFile}} and {{EC}} tests in particular are 100% show-ups in the list
> of bad tests.
> I looked at those tests and they certainly need some improvements (i.e.,
> HDFS-15459). Is any one interested in those test cases? Can we just turn them
> off?
> I like to give some heads-up that we need some more collaboration to enforce
> the stability of the code set.
> * For all developers, please, {color:#ff}file a Jira once you see a
> failing test whether it is unrelated to your patch or not{color}. This gives
> heads-up to other developers about the potential failures. Please do not stop
> at commenting on your patch "_+this is unrelated to my work+_".
> * Volunteer to dedicate more time on fixing flaky tests.
> * Periodically, make sure that the list of failing tests does not exceed a
> certain number of tests. We have Qbt reports to monitor that, but there is no
> follow up on its status.
> * We should consider aggressive strategies such as blocking any merges until
> the code is brought back to stability.
> * We need a clear and well-defined process to address Yetus issues:
> configuration, investigating running out of memory, slowness..etc.
> * Turn-off the Junits within the modules that are not being actively used in
> the community (i.e., EC, stripedFiles, or..etc.).
>
> CC: [~aajisaka], [~elgoiri], [~kihwal], [~daryn], [~weichiu]
> Do you guys have any thoughts on the current status of the HDFS ?
>
> +The following list is a quick list of failing Junits from Qbt reports:+
>
> !https://ci-hadoop.apache.org/static/0ead8630/images/16x16/document_add.png!
> [org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSProviderCaching|https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/299/testReport/org.apache.hadoop.crypto.key.kms.server/TestKMS/testKMSProviderCaching/]1.5
>
> sec[1|https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/299/]
> !https://ci-hadoop.apache.org/static/0ead8630/images/16x16/document_add.png!
> [org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata|https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/299/testReport/org.apache.hadoop.fs.azure/TestBlobMetadata/testFolderMetadata/]42
>
> ms[3|https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/297/]
> !https://ci-hadoop.apache.org/static/0ead8630/images/16x16/document_add.png!
> [org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata|https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/299/testReport/org.apache.hadoop.fs.azure/TestBlobMetadata/testFirstContainerVersionMetadata/]46
>
> ms[3|https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/297/]
> !https://ci-hadoop.apache.org/static/0ead8630/images/16x16/document_add.png!
>
[jira] [Work logged] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?focusedWorklogId=574092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574092
]
ASF GitHub Bot logged work on HDFS-15936:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 11:57
Start Date: 30/Mar/21 11:57
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2836:
URL: https://github.com/apache/hadoop/pull/2836#issuecomment-810161274
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 36 unchanged - 1
fixed = 36 total (was 37) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 230m 1s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2836/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 315m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2836/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2836 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 99e9a30d71b6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[jira] [Updated] (HDFS-15937) Reduce memory used during datanode layout upgrade
[ https://issues.apache.org/jira/browse/HDFS-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDFS-15937: - Attachment: heap-dump-before.png heap-dump-after.png > Reduce memory used during datanode layout upgrade > - > > Key: HDFS-15937 > URL: https://issues.apache.org/jira/browse/HDFS-15937 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.0, 3.1.4, 3.2.2, 3.4.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Attachments: heap-dump-after.png, heap-dump-before.png > > Time Spent: 10m > Remaining Estimate: 0h > > When the datanode block layout is upgrade from -56 (256x256) to -57 (32x32), > we have found the datanode uses a lot more memory than usual. > For each volume, the blocks are scanned and a list is created holding a > series of LinkArgs objects. This object contains a File object for the block > source and destination. The file object stores the path as a string, eg: > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825 > This is string is repeated for every block and meta file on the DN, and much > of the string is the same each time, leading to a large amount of memory. > If we change the linkArgs to store: > * Src Path without the block, eg > /data01/dfs/dn/previous.tmp/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0 > * Dest Path without the block eg > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir10 > * Block / Meta file name, eg blk_12345678_1001 or blk_12345678_1001.meta > Then ensure were reuse the same file object for repeated src and dest paths, > we can save most of the memory without reworking the logic of the code. > The current logic works along the source paths recursively, so you can easily > re-use the src path object. > For the destination path, there are only 32x32 (1024) distinct paths, so we > can simply cache them in a hashMap and lookup the re-useable object each time. > I tested locally by generating 100k block files and attempting the layout > upgrade. A heap dump showed the 100k blocks using about 140MB of heap. That > is close to 1.5GB per 1M blocks. > After the change outlined above the same 100K blocks used about 20MB of heap, > so 200MB per million blocks. > A general DN sizing recommendation is 1GB of heap per 1M blocks, so the > upgrade should be able to happen within the pre-upgrade heap. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15937) Reduce memory used during datanode layout upgrade
[ https://issues.apache.org/jira/browse/HDFS-15937?focusedWorklogId=574059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574059 ] ASF GitHub Bot logged work on HDFS-15937: - Author: ASF GitHub Bot Created on: 30/Mar/21 11:21 Start Date: 30/Mar/21 11:21 Worklog Time Spent: 10m Work Description: sodonnel opened a new pull request #2838: URL: https://github.com/apache/hadoop/pull/2838 When the datanode block layout is upgrade from -56 (256x256) to -57 (32x32), we have found the datanode uses a lot more memory than usual. For each volume, the blocks are scanned and a list is created holding a series of LinkArgs objects. This object contains a File object for the block source and destination. The file object stores the path as a string, eg: /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825 This is string is repeated for every block and meta file on the DN, and much of the string is the same each time, leading to a large amount of memory. If we change the linkArgs to store: * Src Path without the block, eg /data01/dfs/dn/previous.tmp/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0 * Dest Path without the block eg /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir10 * Block / Meta file name, eg blk_12345678_1001 or blk_12345678_1001.meta Then ensure were reuse the same file object for repeated src and dest paths, we can save most of the memory without reworking the logic of the code. The current logic works along the source paths recursively, so you can easily re-use the src path object. For the destination path, there are only 32x32 (1024) distinct paths, so we can simply cache them in a hashMap and lookup the re-useable object each time. I tested locally by generating 100k block files and attempting the layout upgrade. A heap dump showed the 100k blocks using about 140MB of heap. That is close to 1.5GB per 1M blocks. After the change outlined above the same 100K blocks used about 20MB of heap, so 200MB per million blocks. A general DN sizing recommendation is 1GB of heap per 1M blocks, so the upgrade should be able to happen within the pre-upgrade heap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 574059) Remaining Estimate: 0h Time Spent: 10m > Reduce memory used during datanode layout upgrade > - > > Key: HDFS-15937 > URL: https://issues.apache.org/jira/browse/HDFS-15937 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.0, 3.1.4, 3.2.2, 3.4.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > When the datanode block layout is upgrade from -56 (256x256) to -57 (32x32), > we have found the datanode uses a lot more memory than usual. > For each volume, the blocks are scanned and a list is created holding a > series of LinkArgs objects. This object contains a File object for the block > source and destination. The file object stores the path as a string, eg: > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825 > This is string is repeated for every block and meta file on the DN, and much > of the string is the same each time, leading to a large amount of memory. > If we change the linkArgs to store: > * Src Path without the block, eg > /data01/dfs/dn/previous.tmp/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0 > * Dest Path without the block eg > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir10 > * Block / Meta file name, eg blk_12345678_1001 or blk_12345678_1001.meta > Then ensure were reuse the same file object for repeated src and dest paths, > we can save most of the memory without reworking the logic of the code. > The current logic works along the source paths recursively, so you can easily > re-use the src path object. > For the destination path, there are only 32x32 (1024) distinct paths, so we
[jira] [Updated] (HDFS-15937) Reduce memory used during datanode layout upgrade
[ https://issues.apache.org/jira/browse/HDFS-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15937: -- Labels: pull-request-available (was: ) > Reduce memory used during datanode layout upgrade > - > > Key: HDFS-15937 > URL: https://issues.apache.org/jira/browse/HDFS-15937 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.0, 3.1.4, 3.2.2, 3.4.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When the datanode block layout is upgrade from -56 (256x256) to -57 (32x32), > we have found the datanode uses a lot more memory than usual. > For each volume, the blocks are scanned and a list is created holding a > series of LinkArgs objects. This object contains a File object for the block > source and destination. The file object stores the path as a string, eg: > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825 > This is string is repeated for every block and meta file on the DN, and much > of the string is the same each time, leading to a large amount of memory. > If we change the linkArgs to store: > * Src Path without the block, eg > /data01/dfs/dn/previous.tmp/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0 > * Dest Path without the block eg > /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir10 > * Block / Meta file name, eg blk_12345678_1001 or blk_12345678_1001.meta > Then ensure were reuse the same file object for repeated src and dest paths, > we can save most of the memory without reworking the logic of the code. > The current logic works along the source paths recursively, so you can easily > re-use the src path object. > For the destination path, there are only 32x32 (1024) distinct paths, so we > can simply cache them in a hashMap and lookup the re-useable object each time. > I tested locally by generating 100k block files and attempting the layout > upgrade. A heap dump showed the 100k blocks using about 140MB of heap. That > is close to 1.5GB per 1M blocks. > After the change outlined above the same 100K blocks used about 20MB of heap, > so 200MB per million blocks. > A general DN sizing recommendation is 1GB of heap per 1M blocks, so the > upgrade should be able to happen within the pre-upgrade heap. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15938) Fix java doc in FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15938: -- Labels: pull-request-available (was: ) > Fix java doc in FSEditLog > - > > Key: HDFS-15938 > URL: https://issues.apache.org/jira/browse/HDFS-15938 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Fix java doc in > org.apache.hadoop.hdfs.server.namenode.FSEditLog#logAddCacheDirectiveInfo. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15938) Fix java doc in FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-15938?focusedWorklogId=574057=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574057 ] ASF GitHub Bot logged work on HDFS-15938: - Author: ASF GitHub Bot Created on: 30/Mar/21 11:20 Start Date: 30/Mar/21 11:20 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request #2837: URL: https://github.com/apache/hadoop/pull/2837 JIRA: [HDFS-15938](https://issues.apache.org/jira/browse/HDFS-15938) Fix java doc in org.apache.hadoop.hdfs.server.namenode.FSEditLog#logAddCacheDirectiveInfo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 574057) Remaining Estimate: 0h Time Spent: 10m > Fix java doc in FSEditLog > - > > Key: HDFS-15938 > URL: https://issues.apache.org/jira/browse/HDFS-15938 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Fix java doc in > org.apache.hadoop.hdfs.server.namenode.FSEditLog#logAddCacheDirectiveInfo. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15938) Fix java doc in FSEditLog
tomscut created HDFS-15938: -- Summary: Fix java doc in FSEditLog Key: HDFS-15938 URL: https://issues.apache.org/jira/browse/HDFS-15938 Project: Hadoop HDFS Issue Type: Wish Reporter: tomscut Assignee: tomscut Fix java doc in org.apache.hadoop.hdfs.server.namenode.FSEditLog#logAddCacheDirectiveInfo. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15937) Reduce memory used during datanode layout upgrade
Stephen O'Donnell created HDFS-15937: Summary: Reduce memory used during datanode layout upgrade Key: HDFS-15937 URL: https://issues.apache.org/jira/browse/HDFS-15937 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.2, 3.1.4, 3.3.0, 3.4.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell When the datanode block layout is upgrade from -56 (256x256) to -57 (32x32), we have found the datanode uses a lot more memory than usual. For each volume, the blocks are scanned and a list is created holding a series of LinkArgs objects. This object contains a File object for the block source and destination. The file object stores the path as a string, eg: /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825 This is string is repeated for every block and meta file on the DN, and much of the string is the same each time, leading to a large amount of memory. If we change the linkArgs to store: * Src Path without the block, eg /data01/dfs/dn/previous.tmp/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0 * Dest Path without the block eg /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir10 * Block / Meta file name, eg blk_12345678_1001 or blk_12345678_1001.meta Then ensure were reuse the same file object for repeated src and dest paths, we can save most of the memory without reworking the logic of the code. The current logic works along the source paths recursively, so you can easily re-use the src path object. For the destination path, there are only 32x32 (1024) distinct paths, so we can simply cache them in a hashMap and lookup the re-useable object each time. I tested locally by generating 100k block files and attempting the layout upgrade. A heap dump showed the 100k blocks using about 140MB of heap. That is close to 1.5GB per 1M blocks. After the change outlined above the same 100K blocks used about 20MB of heap, so 200MB per million blocks. A general DN sizing recommendation is 1GB of heap per 1M blocks, so the upgrade should be able to happen within the pre-upgrade heap. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15924) Log4j will cause Server handler blocked when audit log boom.
[
https://issues.apache.org/jira/browse/HDFS-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311354#comment-17311354
]
Qi Zhu commented on HDFS-15924:
---
Thanks a lot [~weichiu] for reply, i think HDFS-15720 may give me a little
relieve.
And looking forward to HADOOP-16206.
Which will update to log4j 2.
Thanks.
> Log4j will cause Server handler blocked when audit log boom.
>
>
> Key: HDFS-15924
> URL: https://issues.apache.org/jira/browse/HDFS-15924
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Qi Zhu
>Priority: Major
> Attachments: image-2021-03-26-16-18-03-341.png,
> image-2021-03-26-16-19-42-165.png
>
>
> !image-2021-03-26-16-18-03-341.png|width=707,height=234!
> !image-2021-03-26-16-19-42-165.png|width=824,height=198!
> The thread blocked when audit log boom show in above.
> Such as [https://dzone.com/articles/log4j-thread-deadlock-case] , it seems
> the same case when heavy load, should we update to Log4j2 or other things we
> can do to improve it in heavy audit log.
>
> {code:java}
> /**
> Call the appenders in the hierrachy starting at
> this. If no appenders could be found, emit a
> warning.
> This method calls all the appenders inherited from the
> hierarchy circumventing any evaluation of whether to log or not
> to log the particular log request.
> @param event the event to log. */
> public void callAppenders(LoggingEvent event) {
> int writes = 0;
> for(Category c = this; c != null; c=c.parent) {
> // Protected against simultaneous call to addAppender,
> removeAppender,...
> synchronized(c) {
> if(c.aai != null) {
> writes += c.aai.appendLoopOnAppenders(event);
> }
> if(!c.additive) {
> break;
> }
> }
> }
> if(writes == 0) {
> repository.emitNoAppenderWarning(this);
> }
> }{code}
> The log4j code, use the global synchronized, it will cause this happened.
> cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] [~ferhui]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15924) Log4j will cause Server handler blocked when audit log boom.
[
https://issues.apache.org/jira/browse/HDFS-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311345#comment-17311345
]
Wei-Chiu Chuang commented on HDFS-15924:
Check this out: HDFS-15720. The additional properties supported may give you a
little relieve.
> Log4j will cause Server handler blocked when audit log boom.
>
>
> Key: HDFS-15924
> URL: https://issues.apache.org/jira/browse/HDFS-15924
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Qi Zhu
>Priority: Major
> Attachments: image-2021-03-26-16-18-03-341.png,
> image-2021-03-26-16-19-42-165.png
>
>
> !image-2021-03-26-16-18-03-341.png|width=707,height=234!
> !image-2021-03-26-16-19-42-165.png|width=824,height=198!
> The thread blocked when audit log boom show in above.
> Such as [https://dzone.com/articles/log4j-thread-deadlock-case] , it seems
> the same case when heavy load, should we update to Log4j2 or other things we
> can do to improve it in heavy audit log.
>
> {code:java}
> /**
> Call the appenders in the hierrachy starting at
> this. If no appenders could be found, emit a
> warning.
> This method calls all the appenders inherited from the
> hierarchy circumventing any evaluation of whether to log or not
> to log the particular log request.
> @param event the event to log. */
> public void callAppenders(LoggingEvent event) {
> int writes = 0;
> for(Category c = this; c != null; c=c.parent) {
> // Protected against simultaneous call to addAppender,
> removeAppender,...
> synchronized(c) {
> if(c.aai != null) {
> writes += c.aai.appendLoopOnAppenders(event);
> }
> if(!c.additive) {
> break;
> }
> }
> }
> if(writes == 0) {
> repository.emitNoAppenderWarning(this);
> }
> }{code}
> The log4j code, use the global synchronized, it will cause this happened.
> cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] [~ferhui]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?focusedWorklogId=573993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573993
]
ASF GitHub Bot logged work on HDFS-15936:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 09:50
Start Date: 30/Mar/21 09:50
Worklog Time Spent: 10m
Work Description: cxorm commented on a change in pull request #2836:
URL: https://github.com/apache/hadoop/pull/2836#discussion_r603942892
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
##
@@ -631,6 +631,7 @@ private int sendPacket(ByteBuffer pkt, int maxChunks,
OutputStream out,
*
* Reporting of this case is done in DataXceiver#run
*/
+LOG.warn("There was a timeout when sending packets: ", e);
Review comment:
I am fine with the update though the exception would be threw and
processed in `DataXceiver#run.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 573993)
Time Spent: 50m (was: 40m)
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?focusedWorklogId=573983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573983
]
ASF GitHub Bot logged work on HDFS-15936:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 09:40
Start Date: 30/Mar/21 09:40
Worklog Time Spent: 10m
Work Description: cxorm commented on a change in pull request #2836:
URL: https://github.com/apache/hadoop/pull/2836#discussion_r603942103
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
##
@@ -631,6 +631,7 @@ private int sendPacket(ByteBuffer pkt, int maxChunks,
OutputStream out,
*
* Reporting of this case is done in DataXceiver#run
Review comment:
I am fine with the update though the exception would be threw and
processed in `DataXceiver#run`
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
##
@@ -631,6 +631,7 @@ private int sendPacket(ByteBuffer pkt, int maxChunks,
OutputStream out,
*
* Reporting of this case is done in DataXceiver#run
Review comment:
I am fine with the update though the exception would be threw and
processed in `DataXceiver#run`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 573983)
Time Spent: 0.5h (was: 20m)
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?focusedWorklogId=573984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573984
]
ASF GitHub Bot logged work on HDFS-15936:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 09:40
Start Date: 30/Mar/21 09:40
Worklog Time Spent: 10m
Work Description: cxorm commented on a change in pull request #2836:
URL: https://github.com/apache/hadoop/pull/2836#discussion_r603942892
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
##
@@ -631,6 +631,7 @@ private int sendPacket(ByteBuffer pkt, int maxChunks,
OutputStream out,
*
* Reporting of this case is done in DataXceiver#run
*/
+LOG.warn("There was a timeout when sending packets: ", e);
Review comment:
I am fine with the update though the exception would be threw and
processed in DataXceiver#run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 573984)
Time Spent: 40m (was: 0.5h)
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-15887: Description: In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and when a checkpoint occurs, it will compete with TailEdits for lock (FSNamesystem#cpLock). Usually, it takes a long time to execute checkpoint, which will cause the size of the generated edit log file to be relatively large. For example, here is an actual effect: The StandbyCheckpointer log is triggered as follows : edit_files.jpg 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint because there have been 5142154 txns since the last checkpoint, which exceeds the configured threshold 100 When loading an edit log with a large amount of data, the processing time will be longer. We should make the edit log size as even as possible, which is good for the operation of the system. was: In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and when a checkpoint occurs, it will compete with TailEdits for lock (FSNamesystem#cpLock). Usually, it takes a long time to execute checkpoint, which will cause the size of the generated edit log file to be relatively large. For example, here is an actual effect: The StandbyCheckpointer log is triggered as follows : edit_files.jpg 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint because there have been 5142154 txns since the last checkpoint, which exceeds the configured threshold 100 When loading an edit log with a large amount of data, the processing time will be longer. We should make the edit log size as even as possible, which is good for the operation of the system. > Make LogRoll and TailEdits execute in parallel > -- > > Key: HDFS-15887 > URL: https://issues.apache.org/jira/browse/HDFS-15887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: edit_files.jpg > > > In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, > and when a checkpoint occurs, it will compete with TailEdits for lock > (FSNamesystem#cpLock). > Usually, it takes a long time to execute checkpoint, which will cause the > size of the generated edit log file to be relatively large. > For example, here is an actual effect: > The StandbyCheckpointer log is triggered as follows : edit_files.jpg > 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State > Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering > checkpoint because there have been 5142154 txns since the last checkpoint, > which exceeds the configured threshold 100 > When loading an edit log with a large amount of data, the processing time > will be longer. We should make the edit log size as even as possible, which > is good for the operation of the system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-15887: Description: In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and when a checkpoint occurs, it will compete with TailEdits for lock (FSNamesystem#cpLock). Usually, it takes a long time to execute checkpoint, which will cause the size of the generated edit log file to be relatively large. For example, here is an actual effect: The StandbyCheckpointer log is triggered as follows : edit_files.jpg 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint because there have been 5142154 txns since the last checkpoint, which exceeds the configured threshold 100 When loading an edit log with a large amount of data, the processing time will be longer. We should make the edit log size as even as possible, which is good for the operation of the system. was: In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and when a checkpoint occurs, it will compete with TailEdits for lock (FSNamesystem#cpLock). Usually, it takes a long time to execute checkpoint, which will cause the size of the generated edit log file to be relatively large. For example, here is an actual effect: The StandbyCheckpointer log is triggered as follows : edit_files.jpg 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint because there have been 5142154 txns since the last checkpoint, which exceeds the configured threshold 100 > Make LogRoll and TailEdits execute in parallel > -- > > Key: HDFS-15887 > URL: https://issues.apache.org/jira/browse/HDFS-15887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: edit_files.jpg > > > In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, > and when a checkpoint occurs, it will compete with TailEdits for lock > (FSNamesystem#cpLock). > Usually, it takes a long time to execute checkpoint, which will cause the > size of the generated edit log file to be relatively large. > For example, here is an actual effect: > The StandbyCheckpointer log is triggered as follows : edit_files.jpg > 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State > Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering > checkpoint because there have been 5142154 txns since the last checkpoint, > which exceeds the configured threshold 100 > When loading an edit log with a large amount of data, the processing time > will be longer. We should make the edit log size as even as possible, which > is good for the operation of the system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-15887: Description: In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and when a checkpoint occurs, it will compete with TailEdits for lock (FSNamesystem#cpLock). Usually, it takes a long time to execute checkpoint, which will cause the size of the generated edit log file to be relatively large. For example, here is an actual effect: The StandbyCheckpointer log is triggered as follows : edit_files.jpg 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint because there have been 5142154 txns since the last checkpoint, which exceeds the configured threshold 100 was: When checkpoint action and roll edit log action occur at the same time, the generated edit file will be larger, which is unfavorable for the tail edit file. For example, here is an actual effect: The StandbyCheckpointer log is triggered as follows : edit_files.jpg 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint because there have been 5142154 txns since the last checkpoint, which exceeds the configured threshold 100 > Make LogRoll and TailEdits execute in parallel > -- > > Key: HDFS-15887 > URL: https://issues.apache.org/jira/browse/HDFS-15887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: edit_files.jpg > > > In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, > and when a checkpoint occurs, it will compete with TailEdits for lock > (FSNamesystem#cpLock). > Usually, it takes a long time to execute checkpoint, which will cause the > size of the generated edit log file to be relatively large. > For example, here is an actual effect: > The StandbyCheckpointer log is triggered as follows : edit_files.jpg > 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State > Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering > checkpoint because there have been 5142154 txns since the last checkpoint, > which exceeds the configured threshold 100 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-15887: Summary: Make LogRoll and TailEdits execute in parallel (was: Solve the uneven rolling edit log file) > Make LogRoll and TailEdits execute in parallel > -- > > Key: HDFS-15887 > URL: https://issues.apache.org/jira/browse/HDFS-15887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: edit_files.jpg > > > When checkpoint action and roll edit log action occur at the same time, the > generated edit file will be larger, which is unfavorable for the tail edit > file. > For example, here is an actual effect: > The StandbyCheckpointer log is triggered as follows : edit_files.jpg > 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State > Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering > checkpoint because there have been 5142154 txns since the last checkpoint, > which exceeds the configured threshold 100 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?focusedWorklogId=573934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573934
]
ASF GitHub Bot logged work on HDFS-15936:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 07:54
Start Date: 30/Mar/21 07:54
Worklog Time Spent: 10m
Work Description: virajjasani commented on a change in pull request #2836:
URL: https://github.com/apache/hadoop/pull/2836#discussion_r603865296
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
##
@@ -631,6 +631,7 @@ private int sendPacket(ByteBuffer pkt, int maxChunks,
OutputStream out,
*
* Reporting of this case is done in DataXceiver#run
*/
+LOG.warn("There was a timeout when sending packets: ", e);
Review comment:
nit: `LOG.warn("Sending packets timed out.", e);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 573934)
Time Spent: 20m (was: 10m)
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15850) Superuser actions should be reported to external enforcers
[
https://issues.apache.org/jira/browse/HDFS-15850?focusedWorklogId=573926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573926
]
ASF GitHub Bot logged work on HDFS-15850:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 07:45
Start Date: 30/Mar/21 07:45
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2784:
URL: https://github.com/apache/hadoop/pull/2784#issuecomment-809995195
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 57s | | trunk passed |
| +1 :green_heart: | compile | 6m 6s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 5m 41s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 31s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 54s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 10s | | the patch passed |
| +1 :green_heart: | compile | 6m 4s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 6m 4s | | the patch passed |
| +1 :green_heart: | compile | 5m 35s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 5m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 19s | | hadoop-hdfs-project: The
patch generated 0 new + 498 unchanged - 6 fixed = 498 total (was 504) |
| +1 :green_heart: | mvnsite | 2m 7s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 16s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 376m 31s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2784/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 23m 3s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2784/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 533m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestStateAlignmentContextWithHA |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| |
[jira] [Updated] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15936:
--
Labels: pull-request-available (was: )
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?focusedWorklogId=573894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573894
]
ASF GitHub Bot logged work on HDFS-15936:
-
Author: ASF GitHub Bot
Created on: 30/Mar/21 06:41
Start Date: 30/Mar/21 06:41
Worklog Time Spent: 10m
Work Description: jianghuazhu opened a new pull request #2836:
URL: https://github.com/apache/hadoop/pull/2836
…SocketTimeout exception.
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.)
For more details, please see
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 573894)
Remaining Estimate: 0h
Time Spent: 10m
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-15936 started by JiangHua Zhu.
---
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
[
https://issues.apache.org/jira/browse/HDFS-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu reassigned HDFS-15936:
---
Assignee: JiangHua Zhu
> Solve SocketTimeoutException#sendPacket() does not record SocketTimeout
> exception
> -
>
> Key: HDFS-15936
> URL: https://issues.apache.org/jira/browse/HDFS-15936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>
> In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs,
> no information is recorded here.
> try {
>..
> } catch (IOException e) {
>if (e instanceof SocketTimeoutException) {
> /*
> * writing to client timed out. This happens if the client reads
> * part of a block and then decides not to read the rest (but leaves
> * the socket open).
> *
> * Reporting of this case is done in DataXceiver#run
> */
>}
> }
> No records are generated here, which is not conducive to troubleshooting.
> We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15936) Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception
JiangHua Zhu created HDFS-15936:
---
Summary: Solve SocketTimeoutException#sendPacket() does not record
SocketTimeout exception
Key: HDFS-15936
URL: https://issues.apache.org/jira/browse/HDFS-15936
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: JiangHua Zhu
In SocketTimeoutException#sendPacket(), if a SocketTimeout exception occurs, no
information is recorded here.
try {
..
} catch (IOException e) {
if (e instanceof SocketTimeoutException) {
/*
* writing to client timed out. This happens if the client reads
* part of a block and then decides not to read the rest (but leaves
* the socket open).
*
* Reporting of this case is done in DataXceiver#run
*/
}
}
No records are generated here, which is not conducive to troubleshooting.
We should add a line of warning type log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
