[jira] [Work logged] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?focusedWorklogId=652548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652548
]
ASF GitHub Bot logged work on HDFS-16229:
-
Author: ASF GitHub Bot
Created on: 18/Sep/21 05:37
Start Date: 18/Sep/21 05:37
Worklog Time Spent: 10m
Work Description: ayushtkn commented on pull request #3451:
URL: https://github.com/apache/hadoop/pull/3451#issuecomment-922195008
Thanx @jianghuazhu for the contribution!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652548)
Time Spent: 1h 10m (was: 1h)
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17417024#comment-17417024
]
Ayush Saxena commented on HDFS-16229:
-
Committed to trunk.
Thanx [~jianghuazhu] for the contribution!!!
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HDFS-16229.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?focusedWorklogId=652547=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652547
]
ASF GitHub Bot logged work on HDFS-16229:
-
Author: ASF GitHub Bot
Created on: 18/Sep/21 05:37
Start Date: 18/Sep/21 05:37
Worklog Time Spent: 10m
Work Description: ayushtkn merged pull request #3451:
URL: https://github.com/apache/hadoop/pull/3451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652547)
Time Spent: 1h (was: 50m)
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-16229:
Priority: Trivial (was: Major)
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16107) Split RPC configuration to isolate RPC
[
https://issues.apache.org/jira/browse/HDFS-16107?focusedWorklogId=652544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652544
]
ASF GitHub Bot logged work on HDFS-16107:
-
Author: ASF GitHub Bot
Created on: 18/Sep/21 05:26
Start Date: 18/Sep/21 05:26
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3170:
URL: https://github.com/apache/hadoop/pull/3170#discussion_r711494952
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##
@@ -3192,23 +3196,41 @@ protected Server(String bindAddress, int port,
if (queueSizePerHandler != -1) {
this.maxQueueSize = handlerCount * queueSizePerHandler;
} else {
- this.maxQueueSize = handlerCount * conf.getInt(
- CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_KEY,
- CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_DEFAULT);
+ this.maxQueueSize = conf.getInt(getQueueClassPrefix() + "." +
+ CommonConfigurationKeys.SERVER_HANDLER_QUEUE_SIZE_KEY, 0);
+ if (this.maxQueueSize < 1) {
+this.maxQueueSize = handlerCount * conf.getInt(
+CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_KEY,
+CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_DEFAULT);
+ }
+}
+int tmpMaxRespSize = conf.getInt(getQueueClassPrefix() + "." +
+CommonConfigurationKeys.SERVER_RPC_MAX_RESPONSE_SIZE_KEY, 0);
+if (tmpMaxRespSize < 1) {
+ this.maxRespSize = conf.getInt(
+ CommonConfigurationKeys.IPC_SERVER_RPC_MAX_RESPONSE_SIZE_KEY,
+ CommonConfigurationKeys.IPC_SERVER_RPC_MAX_RESPONSE_SIZE_DEFAULT);
+} else {
+ this.maxRespSize = tmpMaxRespSize;
}
-this.maxRespSize = conf.getInt(
-CommonConfigurationKeys.IPC_SERVER_RPC_MAX_RESPONSE_SIZE_KEY,
-CommonConfigurationKeys.IPC_SERVER_RPC_MAX_RESPONSE_SIZE_DEFAULT);
if (numReaders != -1) {
this.readThreads = numReaders;
} else {
- this.readThreads = conf.getInt(
- CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_KEY,
- CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_DEFAULT);
+ this.readThreads = conf.getInt(getQueueClassPrefix() + "." +
+ CommonConfigurationKeys.SERVER_RPC_READ_THREADS_KEY, 0);
Review comment:
Hi @jianghuazhu , if we share default value with
```IPC_SERVER_RPC_READ_THREADS_KEY```, maybe we could change
```
this.readThreads = conf.getInt(getQueueClassPrefix() + "." +
CommonConfigurationKeys.SERVER_RPC_READ_THREADS_KEY, 0);
if (this.readThreads < 1) {
this.readThreads = conf.getInt(
CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_KEY,
CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_DEFAULT);
}
```
to
```this.readThreads = conf.getInt(prefix + "." +
CommonConfigurationKeys.SERVER_RPC_READ_THREADS_KEY,
CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_DEFAULT);```.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652544)
Time Spent: 3h 20m (was: 3h 10m)
> Split RPC configuration to isolate RPC
> --
>
> Key: HDFS-16107
> URL: https://issues.apache.org/jira/browse/HDFS-16107
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> For RPC of different ports, there are some common configurations, such as:
> ipc.server.read.threadpool.size
> ipc.server.read.connection-queue.size
> ipc.server.handler.queue.size
> Once we configure these values, it will affect all requests (including client
> and requests within the cluster).
> It is necessary for us to split these configurations to adapt to different
> ports, such as:
> ipc.8020.server.read.threadpool.size
> ipc.8021.server.read.threadpool.size
> ipc.8020.server.read.connection-queue.size
> ipc.8021.server.read.connection-queue.size
> The advantage of this is to isolate the RPC to deal with the pressure of
> requests from all sides.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Updated] (HDFS-16230) Minor bug in TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16230: -- Labels: pull-request-available (was: ) > Minor bug in TestStorageRestore > --- > > Key: HDFS-16230 > URL: https://issues.apache.org/jira/browse/HDFS-16230 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Thomas Leplus >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Strings being immutable, you need to use the trim() method return value. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16230) Minor bug in TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-16230?focusedWorklogId=652530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652530 ] ASF GitHub Bot logged work on HDFS-16230: - Author: ASF GitHub Bot Created on: 18/Sep/21 03:53 Start Date: 18/Sep/21 03:53 Worklog Time Spent: 10m Work Description: thomasleplus opened a new pull request #3455: URL: https://github.com/apache/hadoop/pull/3455 Strings being immutable, you need to use the trim() method return value. ### Description of PR Minor bug in TestStorageRestore. ### How was this patch tested? Issue is trivial. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 652530) Remaining Estimate: 0h Time Spent: 10m > Minor bug in TestStorageRestore > --- > > Key: HDFS-16230 > URL: https://issues.apache.org/jira/browse/HDFS-16230 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Thomas Leplus >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > > Strings being immutable, you need to use the trim() method return value. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16230) Minor bug in TestStorageRestore
Thomas Leplus created HDFS-16230: Summary: Minor bug in TestStorageRestore Key: HDFS-16230 URL: https://issues.apache.org/jira/browse/HDFS-16230 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Thomas Leplus Strings being immutable, you need to use the trim() method return value. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu updated HDFS-16229:
Component/s: documentation
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?focusedWorklogId=652522=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652522
]
ASF GitHub Bot logged work on HDFS-16229:
-
Author: ASF GitHub Bot
Created on: 18/Sep/21 02:30
Start Date: 18/Sep/21 02:30
Worklog Time Spent: 10m
Work Description: jianghuazhu removed a comment on pull request #3451:
URL: https://github.com/apache/hadoop/pull/3451#issuecomment-922164124
@ayushtkn @virajjasani, can you review it?
There are some exceptions in jenkins, such as:
hadoop.hdfs.TestDistributedFileSystem
hadoop.hdfs.TestReconstructStripedFileWithValidator
hadoop.hdfs.TestDecommissionWithStriped
hadoop.hdfs.TestDFSInotifyEventInputStream
hadoop.hdfs.TestViewDistributedFileSystemContract
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.TestRollingUpgradeRollback
It seems that these exceptions are caused by other reasons, not much related
to the code I submitted, because I only submitted some documents and comments.
thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652522)
Time Spent: 50m (was: 40m)
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?focusedWorklogId=652521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652521
]
ASF GitHub Bot logged work on HDFS-16229:
-
Author: ASF GitHub Bot
Created on: 18/Sep/21 02:29
Start Date: 18/Sep/21 02:29
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on pull request #3451:
URL: https://github.com/apache/hadoop/pull/3451#issuecomment-922164124
@ayushtkn @virajjasani, can you review it?
There are some exceptions in jenkins, such as:
hadoop.hdfs.TestDistributedFileSystem
hadoop.hdfs.TestReconstructStripedFileWithValidator
hadoop.hdfs.TestDecommissionWithStriped
hadoop.hdfs.TestDFSInotifyEventInputStream
hadoop.hdfs.TestViewDistributedFileSystemContract
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.TestRollingUpgradeRollback
It seems that these exceptions are caused by other reasons, not much related
to the code I submitted, because I only submitted some documents and comments.
thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652521)
Time Spent: 40m (was: 0.5h)
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?focusedWorklogId=652520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652520
]
ASF GitHub Bot logged work on HDFS-16229:
-
Author: ASF GitHub Bot
Created on: 18/Sep/21 02:28
Start Date: 18/Sep/21 02:28
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on pull request #3451:
URL: https://github.com/apache/hadoop/pull/3451#issuecomment-922163995
@ayushtkn @virajjasani, can you review it?
There are some exceptions in jenkins, such as:
hadoop.hdfs.TestDistributedFileSystem
hadoop.hdfs.TestReconstructStripedFileWithValidator
hadoop.hdfs.TestDecommissionWithStriped
hadoop.hdfs.TestDFSInotifyEventInputStream
hadoop.hdfs.TestViewDistributedFileSystemContract
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.TestRollingUpgradeRollback
It seems that these exceptions are caused by other reasons, not much related
to the code I submitted, because I only submitted some documents and comments.
thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652520)
Time Spent: 0.5h (was: 20m)
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16227) testMoverWithStripedFile fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-16227?focusedWorklogId=652503=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652503
]
ASF GitHub Bot logged work on HDFS-16227:
-
Author: ASF GitHub Bot
Created on: 18/Sep/21 00:03
Start Date: 18/Sep/21 00:03
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3429:
URL: https://github.com/apache/hadoop/pull/3429#issuecomment-922140659
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 24s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 49s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 33 unchanged - 1
fixed = 33 total (was 34) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 228m 47s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 339m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3429/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3429 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 1655d0a4cc97 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 887156444d80a7b762d4caa2f289b36e259b3da9 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3429/6/testReport/ |
| Max. process+thread count | 3018 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3429/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
|
[jira] [Updated] (HDFS-16213) Flaky test TestFsDatasetImpl#testDnRestartWithHardLink
[
https://issues.apache.org/jira/browse/HDFS-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated HDFS-16213:
Status: Patch Available (was: In Progress)
> Flaky test TestFsDatasetImpl#testDnRestartWithHardLink
> --
>
> Key: HDFS-16213
> URL: https://issues.apache.org/jira/browse/HDFS-16213
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Failure case:
> [here|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3359/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR]
> testDnRestartWithHardLink(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 7.768 s <<< FAILURE![ERROR]
> testDnRestartWithHardLink(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 7.768 s <<< FAILURE!java.lang.AssertionError at
> org.junit.Assert.fail(Assert.java:87) at
> org.junit.Assert.assertTrue(Assert.java:42) at
> org.junit.Assert.assertTrue(Assert.java:53) at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testDnRestartWithHardLink(TestFsDatasetImpl.java:1344)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16213) Flaky test TestFsDatasetImpl#testDnRestartWithHardLink
[
https://issues.apache.org/jira/browse/HDFS-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-16213 started by Viraj Jasani.
---
> Flaky test TestFsDatasetImpl#testDnRestartWithHardLink
> --
>
> Key: HDFS-16213
> URL: https://issues.apache.org/jira/browse/HDFS-16213
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Failure case:
> [here|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3359/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR]
> testDnRestartWithHardLink(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 7.768 s <<< FAILURE![ERROR]
> testDnRestartWithHardLink(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 7.768 s <<< FAILURE!java.lang.AssertionError at
> org.junit.Assert.fail(Assert.java:87) at
> org.junit.Assert.assertTrue(Assert.java:42) at
> org.junit.Assert.assertTrue(Assert.java:53) at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testDnRestartWithHardLink(TestFsDatasetImpl.java:1344)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16227) testMoverWithStripedFile fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-16227?focusedWorklogId=652420=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652420
]
ASF GitHub Bot logged work on HDFS-16227:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 18:23
Start Date: 17/Sep/21 18:23
Worklog Time Spent: 10m
Work Description: virajjasani commented on a change in pull request #3429:
URL: https://github.com/apache/hadoop/pull/3429#discussion_r711264932
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
##
@@ -872,7 +872,6 @@ void initConfWithStripe(Configuration conf) {
public void testMoverWithStripedFile() throws Exception {
final Configuration conf = new HdfsConfiguration();
initConfWithStripe(conf);
-
Review comment:
> Not strongly opposed but not ideal.
I agree with your review, addressed in the latest commit. No change in
logic, just the refactor as you have suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652420)
Time Spent: 2h 40m (was: 2.5h)
> testMoverWithStripedFile fails intermittently
> -
>
> Key: HDFS-16227
> URL: https://issues.apache.org/jira/browse/HDFS-16227
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> TestMover#testMoverWithStripedFile fails intermittently with stacktrace:
> {code:java}
> [ERROR]
> testMoverWithStripedFile(org.apache.hadoop.hdfs.server.mover.TestMover) Time
> elapsed: 48.439 s <<< FAILURE![ERROR]
> testMoverWithStripedFile(org.apache.hadoop.hdfs.server.mover.TestMover) Time
> elapsed: 48.439 s <<< FAILURE!java.lang.AssertionError: expected:
> but was: at org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:120) at
> org.junit.Assert.assertEquals(Assert.java:146) at
> org.apache.hadoop.hdfs.server.mover.TestMover.testMoverWithStripedFile(TestMover.java:965)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
> e.g
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3386/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16227) testMoverWithStripedFile fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-16227?focusedWorklogId=652384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652384
]
ASF GitHub Bot logged work on HDFS-16227:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 16:40
Start Date: 17/Sep/21 16:40
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #3429:
URL: https://github.com/apache/hadoop/pull/3429#discussion_r711203232
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
##
@@ -872,7 +872,6 @@ void initConfWithStripe(Configuration conf) {
public void testMoverWithStripedFile() throws Exception {
final Configuration conf = new HdfsConfiguration();
initConfWithStripe(conf);
-
Review comment:
If the goal is to avoid the 150 lines issue, I would prefer trying to
make refactor the method a little instead of just removing lines that hurt
readability. Not strongly opposed but not ideal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652384)
Time Spent: 2.5h (was: 2h 20m)
> testMoverWithStripedFile fails intermittently
> -
>
> Key: HDFS-16227
> URL: https://issues.apache.org/jira/browse/HDFS-16227
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> TestMover#testMoverWithStripedFile fails intermittently with stacktrace:
> {code:java}
> [ERROR]
> testMoverWithStripedFile(org.apache.hadoop.hdfs.server.mover.TestMover) Time
> elapsed: 48.439 s <<< FAILURE![ERROR]
> testMoverWithStripedFile(org.apache.hadoop.hdfs.server.mover.TestMover) Time
> elapsed: 48.439 s <<< FAILURE!java.lang.AssertionError: expected:
> but was: at org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:120) at
> org.junit.Assert.assertEquals(Assert.java:146) at
> org.apache.hadoop.hdfs.server.mover.TestMover.testMoverWithStripedFile(TestMover.java:965)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
> e.g
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3386/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?focusedWorklogId=652357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652357
]
ASF GitHub Bot logged work on HDFS-16229:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 15:51
Start Date: 17/Sep/21 15:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3451:
URL: https://github.com/apache/hadoop/pull/3451#issuecomment-921902708
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 24s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 359m 7s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3451/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +0 :ok: | asflicense | 0m 43s | | ASF License check generated no
output? |
| | | 464m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestReconstructStripedFileWithValidator |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.TestDFSInotifyEventInputStream |
| | hadoop.hdfs.TestViewDistributedFileSystemContract |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestRollingUpgradeRollback |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3451/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3451 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 4742ec491730 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 30382a1b071a5ba4f187e14a12b359b6cca1d895 |
| Default Java | Private
[jira] [Updated] (HDFS-15113) Missing IBR when NameNode restart if open processCommand async feature
[
https://issues.apache.org/jira/browse/HDFS-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15113:
---
Fix Version/s: 3.2.4
> Missing IBR when NameNode restart if open processCommand async feature
> --
>
> Key: HDFS-15113
> URL: https://issues.apache.org/jira/browse/HDFS-15113
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Blocker
> Fix For: 3.3.0, 3.2.4
>
> Attachments: HDFS-15113.001.patch, HDFS-15113.002.patch,
> HDFS-15113.003.patch, HDFS-15113.004.patch, HDFS-15113.005.patch,
> HDFS-15113.addendum.patch
>
>
> Recently, I meet one case that NameNode missing block after restart which is
> related with HDFS-14997.
> a. during NameNode restart, it will return command `DNA_REGISTER` to DataNode
> when receive some RPC request from DataNode.
> b. when DataNode receive `DNA_REGISTER` command, it will run #reRegister
> async.
> {code:java}
> void reRegister() throws IOException {
> if (shouldRun()) {
> // re-retrieve namespace info to make sure that, if the NN
> // was restarted, we still match its version (HDFS-2120)
> NamespaceInfo nsInfo = retrieveNamespaceInfo();
> // and re-register
> register(nsInfo);
> scheduler.scheduleHeartbeat();
> // HDFS-9917,Standby NN IBR can be very huge if standby namenode is down
> // for sometime.
> if (state == HAServiceState.STANDBY || state ==
> HAServiceState.OBSERVER) {
> ibrManager.clearIBRs();
> }
> }
> }
> {code}
> c. As we know, #register will trigger BR immediately.
> d. because #reRegister run async, so we could not make sure which one run
> first between send FBR and clear IBR. If clean IBR run first, it will be OK.
> But if send FBR first then clear IBR, it will missing some blocks received
> between these two time point until next FBR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15113) Missing IBR when NameNode restart if open processCommand async feature
[
https://issues.apache.org/jira/browse/HDFS-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416727#comment-17416727
]
Xiaoqiao He commented on HDFS-15113:
Cherry-pick to branch-3.2.
> Missing IBR when NameNode restart if open processCommand async feature
> --
>
> Key: HDFS-15113
> URL: https://issues.apache.org/jira/browse/HDFS-15113
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Blocker
> Fix For: 3.3.0, 3.2.4
>
> Attachments: HDFS-15113.001.patch, HDFS-15113.002.patch,
> HDFS-15113.003.patch, HDFS-15113.004.patch, HDFS-15113.005.patch,
> HDFS-15113.addendum.patch
>
>
> Recently, I meet one case that NameNode missing block after restart which is
> related with HDFS-14997.
> a. during NameNode restart, it will return command `DNA_REGISTER` to DataNode
> when receive some RPC request from DataNode.
> b. when DataNode receive `DNA_REGISTER` command, it will run #reRegister
> async.
> {code:java}
> void reRegister() throws IOException {
> if (shouldRun()) {
> // re-retrieve namespace info to make sure that, if the NN
> // was restarted, we still match its version (HDFS-2120)
> NamespaceInfo nsInfo = retrieveNamespaceInfo();
> // and re-register
> register(nsInfo);
> scheduler.scheduleHeartbeat();
> // HDFS-9917,Standby NN IBR can be very huge if standby namenode is down
> // for sometime.
> if (state == HAServiceState.STANDBY || state ==
> HAServiceState.OBSERVER) {
> ibrManager.clearIBRs();
> }
> }
> }
> {code}
> c. As we know, #register will trigger BR immediately.
> d. because #reRegister run async, so we could not make sure which one run
> first between send FBR and clear IBR. If clean IBR run first, it will be OK.
> But if send FBR first then clear IBR, it will missing some blocks received
> between these two time point until next FBR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15075) Remove process command timing from BPServiceActor
[ https://issues.apache.org/jira/browse/HDFS-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416724#comment-17416724 ] Xiaoqiao He commented on HDFS-15075: Cherry-pick to branch-3.2 > Remove process command timing from BPServiceActor > - > > Key: HDFS-15075 > URL: https://issues.apache.org/jira/browse/HDFS-15075 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.3.0, 3.2.4 > > Attachments: HDFS-15075.001.patch, HDFS-15075.002.patch, > HDFS-15075.003.patch, HDFS-15075.004.patch, HDFS-15075.005.patch, > HDFS-15075.006.patch, HDFS-15075.007.patch, HDFS-15075.008.patch, > HDFS-15075.009.patch > > > HDFS-14997 moved the command processing into async. > Right now, we are checking the time to add to a queue. > We should remove this one and maybe move the timing within the thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15651) Client could not obtain block when DN CommandProcessingThread exit
[
https://issues.apache.org/jira/browse/HDFS-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416725#comment-17416725
]
Xiaoqiao He commented on HDFS-15651:
Cherry-pick to branch-3.2.
> Client could not obtain block when DN CommandProcessingThread exit
> --
>
> Key: HDFS-15651
> URL: https://issues.apache.org/jira/browse/HDFS-15651
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Aiphago
>Priority: Major
> Fix For: 3.3.1, 3.4.0, 3.2.4
>
> Attachments: HDFS-15651.001.patch, HDFS-15651.002.patch,
> HDFS-15651.patch
>
>
> In our cluster, we applied the HDFS-14997 improvement.
> We find one case that CommandProcessingThread will exit due to OOM error.
> OOM error was caused by our one abnormal application that running on this DN
> node.
> {noformat}
> 2020-10-18 10:27:12,604 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: Command processor
> encountered fatal exception and exit.
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:717)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:173)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:222)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2005)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:671)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:617)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1247)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.access$1000(BPServiceActor.java:1194)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread$3.run(BPServiceActor.java:1299)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1221)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1208)
> {noformat}
> Here the main point is that CommandProcessingThread crashed will lead a very
> bad impact. All the NN response commands will not be processed by DN side.
> We enabled the block token to access the data, but here the DN command
> DNA_ACCESSKEYUPDATE is not processed on time by DN. And then we see lots of
> Sasl error due to key expiration in DN log:
> {noformat}
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=xxx, keyId=xx,
> userId=xxx, blockPoolId=, blockId=xxx, access modes=[READ]), since the
> required block key (keyID=xxx) doesn't exist.]
> {noformat}
>
> For the impact in client side, our users receive lots of 'could not obtain
> block' error with BlockMissingException.
> CommandProcessingThread is a critical thread, it should always be running.
> {code:java}
> /**
>* CommandProcessingThread that process commands asynchronously.
>*/
> class CommandProcessingThread extends Thread {
> private final BPServiceActor actor;
> private final BlockingQueue queue;
> ...
> @Override
> public void run() {
> try {
> processQueue();
> } catch (Throwable t) {
> LOG.error("{} encountered fatal exception and exit.", getName(), t);
> <=== should not exit this thread
> }
> }
> {code}
> Once a unexpected error happened, a better handing should be:
> * catch the exception, appropriately deal with the error and let
> processQueue continue to run
> or
> * exit the DN process to let admin user investigate this
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15651) Client could not obtain block when DN CommandProcessingThread exit
[
https://issues.apache.org/jira/browse/HDFS-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15651:
---
Fix Version/s: 3.2.4
> Client could not obtain block when DN CommandProcessingThread exit
> --
>
> Key: HDFS-15651
> URL: https://issues.apache.org/jira/browse/HDFS-15651
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Aiphago
>Priority: Major
> Fix For: 3.3.1, 3.4.0, 3.2.4
>
> Attachments: HDFS-15651.001.patch, HDFS-15651.002.patch,
> HDFS-15651.patch
>
>
> In our cluster, we applied the HDFS-14997 improvement.
> We find one case that CommandProcessingThread will exit due to OOM error.
> OOM error was caused by our one abnormal application that running on this DN
> node.
> {noformat}
> 2020-10-18 10:27:12,604 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: Command processor
> encountered fatal exception and exit.
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:717)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:173)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:222)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2005)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:671)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:617)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1247)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.access$1000(BPServiceActor.java:1194)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread$3.run(BPServiceActor.java:1299)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1221)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1208)
> {noformat}
> Here the main point is that CommandProcessingThread crashed will lead a very
> bad impact. All the NN response commands will not be processed by DN side.
> We enabled the block token to access the data, but here the DN command
> DNA_ACCESSKEYUPDATE is not processed on time by DN. And then we see lots of
> Sasl error due to key expiration in DN log:
> {noformat}
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=xxx, keyId=xx,
> userId=xxx, blockPoolId=, blockId=xxx, access modes=[READ]), since the
> required block key (keyID=xxx) doesn't exist.]
> {noformat}
>
> For the impact in client side, our users receive lots of 'could not obtain
> block' error with BlockMissingException.
> CommandProcessingThread is a critical thread, it should always be running.
> {code:java}
> /**
>* CommandProcessingThread that process commands asynchronously.
>*/
> class CommandProcessingThread extends Thread {
> private final BPServiceActor actor;
> private final BlockingQueue queue;
> ...
> @Override
> public void run() {
> try {
> processQueue();
> } catch (Throwable t) {
> LOG.error("{} encountered fatal exception and exit.", getName(), t);
> <=== should not exit this thread
> }
> }
> {code}
> Once a unexpected error happened, a better handing should be:
> * catch the exception, appropriately deal with the error and let
> processQueue continue to run
> or
> * exit the DN process to let admin user investigate this
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15075) Remove process command timing from BPServiceActor
[ https://issues.apache.org/jira/browse/HDFS-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15075: --- Fix Version/s: 3.2.4 > Remove process command timing from BPServiceActor > - > > Key: HDFS-15075 > URL: https://issues.apache.org/jira/browse/HDFS-15075 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.3.0, 3.2.4 > > Attachments: HDFS-15075.001.patch, HDFS-15075.002.patch, > HDFS-15075.003.patch, HDFS-15075.004.patch, HDFS-15075.005.patch, > HDFS-15075.006.patch, HDFS-15075.007.patch, HDFS-15075.008.patch, > HDFS-15075.009.patch > > > HDFS-14997 moved the command processing into async. > Right now, we are checking the time to add to a queue. > We should remove this one and maybe move the timing within the thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14997) BPServiceActor processes commands from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416722#comment-17416722 ] Xiaoqiao He commented on HDFS-14997: [~brahmareddy] This patch is ready now, less conflict and verify at local, I just check in for branch-3.2. If no other objection, I will check in this related patch to branch-3.2.3 also. What do you think? Thanks. > BPServiceActor processes commands from NameNode asynchronously > -- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.3.0, 3.2.4 > > Attachments: HDFS-14997-branch-3.2.001.patch, HDFS-14997.001.patch, > HDFS-14997.002.patch, HDFS-14997.003.patch, HDFS-14997.004.patch, > HDFS-14997.005.patch, HDFS-14997.addendum.patch, > image-2019-12-26-16-15-44-814.png > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14997) BPServiceActor processes commands from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14997: --- Fix Version/s: 3.2.4 Resolution: Fixed Status: Resolved (was: Patch Available) > BPServiceActor processes commands from NameNode asynchronously > -- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.2.4, 3.3.0 > > Attachments: HDFS-14997-branch-3.2.001.patch, HDFS-14997.001.patch, > HDFS-14997.002.patch, HDFS-14997.003.patch, HDFS-14997.004.patch, > HDFS-14997.005.patch, HDFS-14997.addendum.patch, > image-2019-12-26-16-15-44-814.png > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14997) BPServiceActor processes commands from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416630#comment-17416630 ] Xiaoqiao He commented on HDFS-14997: Failed unit tests seems not related to this changes. Will commit to branch-3.2 for a short while. > BPServiceActor processes commands from NameNode asynchronously > -- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14997-branch-3.2.001.patch, HDFS-14997.001.patch, > HDFS-14997.002.patch, HDFS-14997.003.patch, HDFS-14997.004.patch, > HDFS-14997.005.patch, HDFS-14997.addendum.patch, > image-2019-12-26-16-15-44-814.png > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[
https://issues.apache.org/jira/browse/HDFS-16219?focusedWorklogId=652270=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652270
]
ASF GitHub Bot logged work on HDFS-16219:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 11:14
Start Date: 17/Sep/21 11:14
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3452:
URL: https://github.com/apache/hadoop/pull/3452#issuecomment-921713830
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 62m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 22m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 40m 30s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3452/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 144m 47s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3452/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3452 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml |
| uname | Linux 51304fa9f775 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fdec0eda22fe78b114856befd8cf30019f678497 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3452/2/testReport/ |
| Max. process+thread count | 2191 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console
[jira] [Work logged] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[
https://issues.apache.org/jira/browse/HDFS-16219?focusedWorklogId=652267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652267
]
ASF GitHub Bot logged work on HDFS-16219:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 11:04
Start Date: 17/Sep/21 11:04
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3452:
URL: https://github.com/apache/hadoop/pull/3452#issuecomment-921707811
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 55m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | shadedclient | 20m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 35m 23s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3452/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 134m 44s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestDisableNameservices |
| | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3452/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3452 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml |
| uname | Linux ff5eb637ddd8 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fdec0eda22fe78b114856befd8cf30019f678497 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3452/3/testReport/ |
| Max. process+thread count | 2075 (vs. ulimit of 5500) |
| modules | C:
[jira] [Work logged] (HDFS-16107) Split RPC configuration to isolate RPC
[ https://issues.apache.org/jira/browse/HDFS-16107?focusedWorklogId=652213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652213 ] ASF GitHub Bot logged work on HDFS-16107: - Author: ASF GitHub Bot Created on: 17/Sep/21 09:14 Start Date: 17/Sep/21 09:14 Worklog Time Spent: 10m Work Description: jianghuazhu commented on pull request #3170: URL: https://github.com/apache/hadoop/pull/3170#issuecomment-921640568 @tomscut @virajjasani, I have submitted some changes, can you review it? thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 652213) Time Spent: 3h 10m (was: 3h) > Split RPC configuration to isolate RPC > -- > > Key: HDFS-16107 > URL: https://issues.apache.org/jira/browse/HDFS-16107 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > For RPC of different ports, there are some common configurations, such as: > ipc.server.read.threadpool.size > ipc.server.read.connection-queue.size > ipc.server.handler.queue.size > Once we configure these values, it will affect all requests (including client > and requests within the cluster). > It is necessary for us to split these configurations to adapt to different > ports, such as: > ipc.8020.server.read.threadpool.size > ipc.8021.server.read.threadpool.size > ipc.8020.server.read.connection-queue.size > ipc.8021.server.read.connection-queue.size > The advantage of this is to isolate the RPC to deal with the pressure of > requests from all sides. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16107) Split RPC configuration to isolate RPC
[
https://issues.apache.org/jira/browse/HDFS-16107?focusedWorklogId=652207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652207
]
ASF GitHub Bot logged work on HDFS-16107:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 08:58
Start Date: 17/Sep/21 08:58
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3170:
URL: https://github.com/apache/hadoop/pull/3170#issuecomment-921628998
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 50s | | trunk passed |
| +1 :green_heart: | compile | 22m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 22m 7s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 7s | | the patch passed |
| +1 :green_heart: | compile | 19m 50s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 45s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 219m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3170/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3170 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux af57c15e2ac0 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c7f6f3a3ad69b1cc52e3e679458e5ac3324af382 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3170/8/testReport/ |
| Max. process+thread count | 2236 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3170/8/console |
| versions | git=2.25.1 maven=3.6.3
[jira] [Work logged] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?focusedWorklogId=652201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652201 ] ASF GitHub Bot logged work on HDFS-16219: - Author: ASF GitHub Bot Created on: 17/Sep/21 08:46 Start Date: 17/Sep/21 08:46 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3452: URL: https://github.com/apache/hadoop/pull/3452#issuecomment-921621460 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | docker | 11m 1s | | Docker failed to build yetus/hadoop:1c0b2edde93. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/3452 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3452/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 652201) Time Spent: 20m (was: 10m) > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16219: -- Labels: pull-request-available (was: ) > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?focusedWorklogId=652195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652195 ] ASF GitHub Bot logged work on HDFS-16219: - Author: ASF GitHub Bot Created on: 17/Sep/21 08:32 Start Date: 17/Sep/21 08:32 Worklog Time Spent: 10m Work Description: virajjasani opened a new pull request #3452: URL: https://github.com/apache/hadoop/pull/3452 ### Description of PR When dfs.federation.router.federation.rename.option is set to DISTCP, if dfs.federation.router.federation.rename.map and dfs.federation.router.federation.rename.bandwidth are not provided with default values, DFSRouter fails to launch. ### How was this patch tested? Local dev testing. ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 652195) Remaining Estimate: 0h Time Spent: 10m > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416541#comment-17416541 ] Viraj Jasani commented on HDFS-16219: - I see, yes even RouterFedBalance builder also use default values 10 and 10 [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/rbfbalance/RouterFedBalance.java#L85-L88]. Makes sense to update the same in hdfs-rbf-default. > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-16229:
--
Labels: pull-request-available (was: )
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416534#comment-17416534 ] Akira Ajisaka commented on HDFS-16219: -- I suppose the default value seems 10 and 10 in federation balance. https://github.com/apache/hadoop/blob/77299ae992b16066dd61e4fec9ff63b863ae2e21/hadoop-tools/hadoop-federation-balance/src/site/markdown/HDFSFederationBalance.md > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?focusedWorklogId=652166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652166
]
ASF GitHub Bot logged work on HDFS-16229:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 08:05
Start Date: 17/Sep/21 08:05
Worklog Time Spent: 10m
Work Description: jianghuazhu opened a new pull request #3451:
URL: https://github.com/apache/hadoop/pull/3451
### Description of PR
Remove the use of obsolete BLOCK_DELETION_INCREMENT.
Details: HDFS-16229
### How was this patch tested?
This jira is mainly to solve and comment-related, so there is no need for
excessive testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652166)
Remaining Estimate: 0h
Time Spent: 10m
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HDFS-16219: -- Thank you [~vjasani] for your comment. Reopened this. > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu updated HDFS-16229:
Component/s: namenode
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
JiangHua Zhu created HDFS-16229:
---
Summary: Remove the use of obsolete BLOCK_DELETION_INCREMENT
Key: HDFS-16229
URL: https://issues.apache.org/jira/browse/HDFS-16229
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: JiangHua Zhu
As early as HDFS-13831, it was already
Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
BLOCK_DELETION_INCREMENT.
E.g:
/**
* From the given list, incrementally remove the blocks from blockManager
* Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
* ensure that other waiters on the lock can get in. See HDFS-2938
*
* @param blocks
* An instance of {@link BlocksMapUpdateInfo} which contains a list
* of blocks that need to be removed from blocksMap
*/
void removeBlocks(BlocksMapUpdateInfo blocks) {
..
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416523#comment-17416523 ] Viraj Jasani commented on HDFS-16219: - [~aajisaka] I just realized that even after HDFS-16217 is in, if we provide router rename option as DISTCP, we still face this issue. I think we might still want to fix this by providing default values of rename.map and rename.bandwidth in hdfs-rbf-default, as 20 and 100 respectively. Thought? > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16229) Remove the use of obsolete BLOCK_DELETION_INCREMENT
[
https://issues.apache.org/jira/browse/HDFS-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu reassigned HDFS-16229:
---
Assignee: JiangHua Zhu
> Remove the use of obsolete BLOCK_DELETION_INCREMENT
> ---
>
> Key: HDFS-16229
> URL: https://issues.apache.org/jira/browse/HDFS-16229
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
>
> As early as HDFS-13831, it was already
> Removed the obsolete BLOCK_DELETION_INCREMENT, but ignored another use of
> BLOCK_DELETION_INCREMENT.
> E.g:
>/**
> * From the given list, incrementally remove the blocks from blockManager
> * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
> * ensure that other waiters on the lock can get in. See HDFS-2938
> *
> * @param blocks
> * An instance of {@link BlocksMapUpdateInfo} which contains a list
> * of blocks that need to be removed from blocksMap
> */
>void removeBlocks(BlocksMapUpdateInfo blocks) {
>..
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16217) RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources
[
https://issues.apache.org/jira/browse/HDFS-16217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-16217:
-
Summary: RBF: Set default value of
hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config
resources (was: RBF: Set default value of
hdfs.fedbalance.procedure.scheduler.journal.uri)
> RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by
> adding appropriate config resources
>
>
> Key: HDFS-16217
> URL: https://issues.apache.org/jira/browse/HDFS-16217
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Environment: Hadoop 3.3.0 with patches
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When dfs.federation.router.federation.rename.option is set to DISTCP and
> hdfs.fedbalance.procedure.scheduler.journal.uri is not set, DFSRouter fails
> to launch.
> {quote}
> 2021-09-08 15:39:11,818 ERROR
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
> router
> java.lang.NullPointerException
> at java.base/java.net.URI$Parser.parse(URI.java:3104)
> at java.base/java.net.URI.(URI.java:600)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.initRouterFedRename(RouterRpcServer.java:444)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.(RouterRpcServer.java:419)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> {quote}
> hdfs.fedbalance.procedure.scheduler.journal.uri is
> hdfs://localhost:8020/tmp/procedure by default, however, the default value is
> not used in DFSRouter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16217) RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
[
https://issues.apache.org/jira/browse/HDFS-16217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved HDFS-16217.
--
Fix Version/s: 3.4.0
Resolution: Fixed
Merged the PR.
> RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
> -
>
> Key: HDFS-16217
> URL: https://issues.apache.org/jira/browse/HDFS-16217
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Environment: Hadoop 3.3.0 with patches
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When dfs.federation.router.federation.rename.option is set to DISTCP and
> hdfs.fedbalance.procedure.scheduler.journal.uri is not set, DFSRouter fails
> to launch.
> {quote}
> 2021-09-08 15:39:11,818 ERROR
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
> router
> java.lang.NullPointerException
> at java.base/java.net.URI$Parser.parse(URI.java:3104)
> at java.base/java.net.URI.(URI.java:600)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.initRouterFedRename(RouterRpcServer.java:444)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.(RouterRpcServer.java:419)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> {quote}
> hdfs.fedbalance.procedure.scheduler.journal.uri is
> hdfs://localhost:8020/tmp/procedure by default, however, the default value is
> not used in DFSRouter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HDFS-16219. -- Resolution: Duplicate Fixed as part of HDFS-16217. > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-16219: - Comment: was deleted (was: Merged the PR.) > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HDFS-16219: -- > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.4.0 > > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HDFS-16219. -- Fix Version/s: 3.4.0 Resolution: Fixed Merged the PR. > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.4.0 > > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
[ https://issues.apache.org/jira/browse/HDFS-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-16219: - Fix Version/s: (was: 3.4.0) > RBF: Set default map tasks and bandwidth in RouterFederationRename > -- > > Key: HDFS-16219 > URL: https://issues.apache.org/jira/browse/HDFS-16219 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: Hadoop 3.3.0 with patches >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > > If dfs.federation.router.federation.rename.map or > dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails > to launch. > This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16217) RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
[
https://issues.apache.org/jira/browse/HDFS-16217?focusedWorklogId=652145=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652145
]
ASF GitHub Bot logged work on HDFS-16217:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 07:39
Start Date: 17/Sep/21 07:39
Worklog Time Spent: 10m
Work Description: aajisaka merged pull request #3434:
URL: https://github.com/apache/hadoop/pull/3434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652145)
Time Spent: 1h 50m (was: 1h 40m)
> RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
> -
>
> Key: HDFS-16217
> URL: https://issues.apache.org/jira/browse/HDFS-16217
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Environment: Hadoop 3.3.0 with patches
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When dfs.federation.router.federation.rename.option is set to DISTCP and
> hdfs.fedbalance.procedure.scheduler.journal.uri is not set, DFSRouter fails
> to launch.
> {quote}
> 2021-09-08 15:39:11,818 ERROR
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
> router
> java.lang.NullPointerException
> at java.base/java.net.URI$Parser.parse(URI.java:3104)
> at java.base/java.net.URI.(URI.java:600)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.initRouterFedRename(RouterRpcServer.java:444)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.(RouterRpcServer.java:419)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> {quote}
> hdfs.fedbalance.procedure.scheduler.journal.uri is
> hdfs://localhost:8020/tmp/procedure by default, however, the default value is
> not used in DFSRouter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16217) RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
[
https://issues.apache.org/jira/browse/HDFS-16217?focusedWorklogId=652146=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652146
]
ASF GitHub Bot logged work on HDFS-16217:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 07:39
Start Date: 17/Sep/21 07:39
Worklog Time Spent: 10m
Work Description: aajisaka commented on pull request #3434:
URL: https://github.com/apache/hadoop/pull/3434#issuecomment-921578739
Merged. Thank you @virajjasani and @goiri
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652146)
Time Spent: 2h (was: 1h 50m)
> RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
> -
>
> Key: HDFS-16217
> URL: https://issues.apache.org/jira/browse/HDFS-16217
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Environment: Hadoop 3.3.0 with patches
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When dfs.federation.router.federation.rename.option is set to DISTCP and
> hdfs.fedbalance.procedure.scheduler.journal.uri is not set, DFSRouter fails
> to launch.
> {quote}
> 2021-09-08 15:39:11,818 ERROR
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
> router
> java.lang.NullPointerException
> at java.base/java.net.URI$Parser.parse(URI.java:3104)
> at java.base/java.net.URI.(URI.java:600)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.initRouterFedRename(RouterRpcServer.java:444)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.(RouterRpcServer.java:419)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> {quote}
> hdfs.fedbalance.procedure.scheduler.journal.uri is
> hdfs://localhost:8020/tmp/procedure by default, however, the default value is
> not used in DFSRouter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16217) RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
[
https://issues.apache.org/jira/browse/HDFS-16217?focusedWorklogId=652142=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652142
]
ASF GitHub Bot logged work on HDFS-16217:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 07:30
Start Date: 17/Sep/21 07:30
Worklog Time Spent: 10m
Work Description: virajjasani commented on pull request #3434:
URL: https://github.com/apache/hadoop/pull/3434#issuecomment-921567270
Thanks @goiri @aajisaka for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652142)
Time Spent: 1h 40m (was: 1.5h)
> RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
> -
>
> Key: HDFS-16217
> URL: https://issues.apache.org/jira/browse/HDFS-16217
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Environment: Hadoop 3.3.0 with patches
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When dfs.federation.router.federation.rename.option is set to DISTCP and
> hdfs.fedbalance.procedure.scheduler.journal.uri is not set, DFSRouter fails
> to launch.
> {quote}
> 2021-09-08 15:39:11,818 ERROR
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
> router
> java.lang.NullPointerException
> at java.base/java.net.URI$Parser.parse(URI.java:3104)
> at java.base/java.net.URI.(URI.java:600)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.initRouterFedRename(RouterRpcServer.java:444)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.(RouterRpcServer.java:419)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> {quote}
> hdfs.fedbalance.procedure.scheduler.journal.uri is
> hdfs://localhost:8020/tmp/procedure by default, however, the default value is
> not used in DFSRouter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16213) Flaky test TestFsDatasetImpl#testDnRestartWithHardLink
[
https://issues.apache.org/jira/browse/HDFS-16213?focusedWorklogId=652141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652141
]
ASF GitHub Bot logged work on HDFS-16213:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 07:29
Start Date: 17/Sep/21 07:29
Worklog Time Spent: 10m
Work Description: virajjasani commented on pull request #3386:
URL: https://github.com/apache/hadoop/pull/3386#issuecomment-921565797
Thanks for the reviews @aajisaka @LeonGao91
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652141)
Time Spent: 7h 40m (was: 7.5h)
> Flaky test TestFsDatasetImpl#testDnRestartWithHardLink
> --
>
> Key: HDFS-16213
> URL: https://issues.apache.org/jira/browse/HDFS-16213
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Failure case:
> [here|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3359/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR]
> testDnRestartWithHardLink(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 7.768 s <<< FAILURE![ERROR]
> testDnRestartWithHardLink(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 7.768 s <<< FAILURE!java.lang.AssertionError at
> org.junit.Assert.fail(Assert.java:87) at
> org.junit.Assert.assertTrue(Assert.java:42) at
> org.junit.Assert.assertTrue(Assert.java:53) at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testDnRestartWithHardLink(TestFsDatasetImpl.java:1344)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16227) testMoverWithStripedFile fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-16227?focusedWorklogId=652140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652140
]
ASF GitHub Bot logged work on HDFS-16227:
-
Author: ASF GitHub Bot
Created on: 17/Sep/21 07:28
Start Date: 17/Sep/21 07:28
Worklog Time Spent: 10m
Work Description: virajjasani commented on a change in pull request #3429:
URL: https://github.com/apache/hadoop/pull/3429#discussion_r710818845
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
##
@@ -872,7 +872,6 @@ void initConfWithStripe(Configuration conf) {
public void testMoverWithStripedFile() throws Exception {
final Configuration conf = new HdfsConfiguration();
initConfWithStripe(conf);
-
Review comment:
Are you fine with this @goiri? After the recent commit, the method has
exactly 150 lines, one more line will create checkstyle warning.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 652140)
Time Spent: 2h 20m (was: 2h 10m)
> testMoverWithStripedFile fails intermittently
> -
>
> Key: HDFS-16227
> URL: https://issues.apache.org/jira/browse/HDFS-16227
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> TestMover#testMoverWithStripedFile fails intermittently with stacktrace:
> {code:java}
> [ERROR]
> testMoverWithStripedFile(org.apache.hadoop.hdfs.server.mover.TestMover) Time
> elapsed: 48.439 s <<< FAILURE![ERROR]
> testMoverWithStripedFile(org.apache.hadoop.hdfs.server.mover.TestMover) Time
> elapsed: 48.439 s <<< FAILURE!java.lang.AssertionError: expected:
> but was: at org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:120) at
> org.junit.Assert.assertEquals(Assert.java:146) at
> org.apache.hadoop.hdfs.server.mover.TestMover.testMoverWithStripedFile(TestMover.java:965)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
> e.g
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3386/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16224) testBalancerWithObserverWithFailedNode times out
[ https://issues.apache.org/jira/browse/HDFS-16224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-16224. -- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed I've committed the fix. Thanks [~LeonG]! > testBalancerWithObserverWithFailedNode times out > > > Key: HDFS-16224 > URL: https://issues.apache.org/jira/browse/HDFS-16224 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > testBalancerWithObserverWithFailedNode fails intermittently. > > Seems it is because of datanode cannot shutdown because we need to wait for > datanodes to finish retries to failed observer. > > Jenkins report: > > [ERROR] > testBalancerWithObserverWithFailedNode(org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes) > Time elapsed: 180.144 s <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds at java.lang.Object.wait(Native Method) at > java.lang.Thread.join(Thread.java:1252) at > java.lang.Thread.join(Thread.java:1326) at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.join(BPServiceActor.java:632) > at > org.apache.hadoop.hdfs.server.datanode.BPOfferService.join(BPOfferService.java:360) > at > org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.shutDownAll(BlockPoolManager.java:119) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2169) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:2166) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:2156) > at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2135) > at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2109) > at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2102) > at > org.apache.hadoop.hdfs.qjournal.MiniQJMHACluster.shutdown(MiniQJMHACluster.java:189) > at > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes.testBalancerWithObserver(TestBalancerWithHANameNodes.java:240) > at > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes.testBalancerWithObserverWithFailedNode(TestBalancerWithHANameNodes.java:197) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at > java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16224) testBalancerWithObserverWithFailedNode times out
[ https://issues.apache.org/jira/browse/HDFS-16224?focusedWorklogId=652135=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652135 ] ASF GitHub Bot logged work on HDFS-16224: - Author: ASF GitHub Bot Created on: 17/Sep/21 06:59 Start Date: 17/Sep/21 06:59 Worklog Time Spent: 10m Work Description: Jing9 merged pull request #3425: URL: https://github.com/apache/hadoop/pull/3425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 652135) Time Spent: 50m (was: 40m) > testBalancerWithObserverWithFailedNode times out > > > Key: HDFS-16224 > URL: https://issues.apache.org/jira/browse/HDFS-16224 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Trivial > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > testBalancerWithObserverWithFailedNode fails intermittently. > > Seems it is because of datanode cannot shutdown because we need to wait for > datanodes to finish retries to failed observer. > > Jenkins report: > > [ERROR] > testBalancerWithObserverWithFailedNode(org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes) > Time elapsed: 180.144 s <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds at java.lang.Object.wait(Native Method) at > java.lang.Thread.join(Thread.java:1252) at > java.lang.Thread.join(Thread.java:1326) at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.join(BPServiceActor.java:632) > at > org.apache.hadoop.hdfs.server.datanode.BPOfferService.join(BPOfferService.java:360) > at > org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.shutDownAll(BlockPoolManager.java:119) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2169) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:2166) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:2156) > at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2135) > at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2109) > at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2102) > at > org.apache.hadoop.hdfs.qjournal.MiniQJMHACluster.shutdown(MiniQJMHACluster.java:189) > at > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes.testBalancerWithObserver(TestBalancerWithHANameNodes.java:240) > at > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes.testBalancerWithObserverWithFailedNode(TestBalancerWithHANameNodes.java:197) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at > java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
