[jira] [Updated] (HDFS-15273) CacheReplicationMonitor hold lock for long time and lead to NN out of service
[
https://issues.apache.org/jira/browse/HDFS-15273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-15273:
---
Status: Open (was: Patch Available)
> CacheReplicationMonitor hold lock for long time and lead to NN out of service
> -
>
> Key: HDFS-15273
> URL: https://issues.apache.org/jira/browse/HDFS-15273
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: caching, namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15273.001.patch
>
>
> CacheReplicationMonitor scan Cache Directives and Cached BlockMap
> periodically. If we add more and more cache directives,
> CacheReplicationMonitor will cost very long time to rescan all of cache
> directives and cache blocks. Meanwhile, scan operation hold global write
> lock, during scan period, NameNode could not process other request.
> So I think we should warn this risk to end user who turn on CacheManager
> feature before improve this implement.
> {code:java}
> private void rescan() throws InterruptedException {
> scannedDirectives = 0;
> scannedBlocks = 0;
> try {
> namesystem.writeLock();
> try {
> lock.lock();
> if (shutdown) {
> throw new InterruptedException("CacheReplicationMonitor was " +
> "shut down.");
> }
> curScanCount = completedScanCount + 1;
> } finally {
> lock.unlock();
> }
> resetStatistics();
> rescanCacheDirectives();
> rescanCachedBlockMap();
> blockManager.getDatanodeManager().resetLastCachingDirectiveSentTime();
> } finally {
> namesystem.writeUnlock();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15273) CacheReplicationMonitor hold lock for long time and lead to NN out of service
[
https://issues.apache.org/jira/browse/HDFS-15273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-15273:
---
Status: Patch Available (was: Open)
The patch still applies. Submit the patch to go through the precommit tests.
> CacheReplicationMonitor hold lock for long time and lead to NN out of service
> -
>
> Key: HDFS-15273
> URL: https://issues.apache.org/jira/browse/HDFS-15273
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: caching, namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15273.001.patch
>
>
> CacheReplicationMonitor scan Cache Directives and Cached BlockMap
> periodically. If we add more and more cache directives,
> CacheReplicationMonitor will cost very long time to rescan all of cache
> directives and cache blocks. Meanwhile, scan operation hold global write
> lock, during scan period, NameNode could not process other request.
> So I think we should warn this risk to end user who turn on CacheManager
> feature before improve this implement.
> {code:java}
> private void rescan() throws InterruptedException {
> scannedDirectives = 0;
> scannedBlocks = 0;
> try {
> namesystem.writeLock();
> try {
> lock.lock();
> if (shutdown) {
> throw new InterruptedException("CacheReplicationMonitor was " +
> "shut down.");
> }
> curScanCount = completedScanCount + 1;
> } finally {
> lock.unlock();
> }
> resetStatistics();
> rescanCacheDirectives();
> rescanCachedBlockMap();
> blockManager.getDatanodeManager().resetLastCachingDirectiveSentTime();
> } finally {
> namesystem.writeUnlock();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16422) Fix thread safety of EC decoding during concurrent preads
[ https://issues.apache.org/jira/browse/HDFS-16422?focusedWorklogId=726126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726126 ] ASF GitHub Bot logged work on HDFS-16422: - Author: ASF GitHub Bot Created on: 14/Feb/22 06:02 Start Date: 14/Feb/22 06:02 Worklog Time Spent: 10m Work Description: jojochuang commented on pull request #3881: URL: https://github.com/apache/hadoop/pull/3881#issuecomment-1038678297 Not sure there's a reentrant lock inside AbstractNativeRawEncoder but it's the read lock that's held, not write lock, to improve concurrency ([HADOOP-15499](https://issues.apache.org/jira/browse/HADOOP-15499) I suppose data race problem is considered? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 726126) Time Spent: 3.5h (was: 3h 20m) > Fix thread safety of EC decoding during concurrent preads > - > > Key: HDFS-16422 > URL: https://issues.apache.org/jira/browse/HDFS-16422 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsclient, ec, erasure-coding >Affects Versions: 3.3.0, 3.3.1 >Reporter: daimin >Assignee: daimin >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.3 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Reading data on an erasure-coded file with missing replicas(internal block of > block group) will cause online reconstruction: read dataUnits part of data > and decode them into the target missing data. Each DFSStripedInputStream > object has a RawErasureDecoder object, and when we doing pread concurrently, > RawErasureDecoder.decode will be invoked concurrently too. > RawErasureDecoder.decode is not thread safe, as a result of that we get wrong > data from pread occasionally. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16455) RBF: Router should explicitly specify the value of `jute.maxbuffer` in hadoop configuration files like core-site.xml
[
https://issues.apache.org/jira/browse/HDFS-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDFS-16455:
--
Assignee: Max Xie
> RBF: Router should explicitly specify the value of `jute.maxbuffer` in hadoop
> configuration files like core-site.xml
>
>
> Key: HDFS-16455
> URL: https://issues.apache.org/jira/browse/HDFS-16455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Based on the current design for delegation token in secure Router, the total
> number of tokens store and update in zookeeper using
> ZKDelegationTokenManager.
> But the default value of system property `jute.maxbuffer` is just 4MB, if
> Router store too many tokens in zk, it will throw IOException `{{{}Packet
> lenxx is out of range{}}}` and all Router will crash.
>
> In our cluster, Routers crashed because of it. The crash logs are below
> {code:java}
> 2022-02-09 02:15:51,607 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Token renewal for identifier: (token for xxx: HDFS_DELEGATION_TOKEN
> owner=xxx/scheduler, renewer=hadoop, realUser=, issueDate=1644344146305,
> maxDate=1644948946305, sequenceNumbe
> r=27136070, masterKeyId=1107); total currentTokens 279548 2022-02-09
> 02:16:07,632 WARN org.apache.zookeeper.ClientCnxn: Session 0x1000172775a0012
> for server zkurl:2181, unexpected e
> rror, closing socket connection and attempting reconnect
> java.io.IOException: Packet len4194553 is out of range!
> at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
> 2022-02-09 02:16:07,733 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 1254 on default port 9001, call Call#144 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getDelegationToken from
> ip:46534
> java.lang.RuntimeException: Could not increment shared counter !!
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:582)
> {code}
> When we restart a Router, it crashed again
> {code:java}
> 2022-02-09 03:14:17,308 INFO
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager:
> Starting to load key cache.
> 2022-02-09 03:14:17,310 INFO
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager:
> Loaded key cache.
> 2022-02-09 03:14:32,930 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x205584be35b0001 for server zkurl:2181, unexpected
> error, closing socket connection and attempting reconnect
> java.io.IOException: Packet len4194478 is out of range!
> at
> org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
> 2022-02-09 03:14:33,030 ERROR
> org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl:
> Error starting threads for z
> kDelegationTokens
> java.io.IOException: Could not start PathChildrenCache for tokens {code}
> Finnally, we config `-Djute.maxbuffer=1000` in hadoop-env,sh to fix this
> issue.
> After dig it, we found the number of the znode
> `/ZKDTSMRoot/ZKDTSMTokensRoot`'s children node was more than 25, which's
> data size was over 4MB.
>
> Maybe we should explicitly specify the value of `jute.maxbuffer` in hadoop
> configuration files like core-site.xml, hdfs-rbf-site.xml to configure a
> larger value
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16396) Reconfig slow peer parameters for datanode
[
https://issues.apache.org/jira/browse/HDFS-16396?focusedWorklogId=726083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726083
]
ASF GitHub Bot logged work on HDFS-16396:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 02:55
Start Date: 14/Feb/22 02:55
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3827:
URL: https://github.com/apache/hadoop/pull/3827#discussion_r805472758
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java
##
@@ -57,26 +60,26 @@
* for outlier detection. If the number of samples is below this then
* outlier detection is skipped.
*/
- private final long minOutlierDetectionSamples;
+ private volatile long minOutlierDetectionSamples;
/**
* Threshold in milliseconds below which a DataNode is definitely not slow.
*/
- private final long lowThresholdMs;
+ private volatile long lowThresholdMs;
/**
* Minimum number of nodes to run outlier detection.
*/
- private final long minOutlierDetectionNodes;
+ private volatile long minOutlierDetectionNodes;
public DataNodePeerMetrics(final String name, Configuration conf) {
this.name = name;
minOutlierDetectionSamples = conf.getLong(
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_KEY,
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_DEFAULT);
lowThresholdMs =
-conf.getLong(DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
+conf.getLong(DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_DEFAULT);
minOutlierDetectionNodes =
-
conf.getLong(DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
+conf.getLong(DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_DEFAULT);
this.slowNodeDetector =
new OutlierDetector(minOutlierDetectionNodes, lowThresholdMs);
Review comment:
Thanks @tasanuma for your comment, and it makes sense to me. I updated
the code, please have a look when you are free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 726083)
Time Spent: 4h (was: 3h 50m)
> Reconfig slow peer parameters for datanode
> --
>
> Key: HDFS-16396
> URL: https://issues.apache.org/jira/browse/HDFS-16396
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> In large clusters, rolling restart datanodes takes a long time. We can make
> slow peers parameters and slow disks parameters in datanode reconfigurable to
> facilitate cluster operation and maintenance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16431) Truncate CallerContext in client side
[ https://issues.apache.org/jira/browse/HDFS-16431?focusedWorklogId=726024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726024 ] ASF GitHub Bot logged work on HDFS-16431: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:36 Start Date: 14/Feb/22 02:36 Worklog Time Spent: 10m Work Description: smarthanwang commented on pull request #3909: URL: https://github.com/apache/hadoop/pull/3909#issuecomment-1037808131 trigger jenkins -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 726024) Time Spent: 1h 40m (was: 1.5h) > Truncate CallerContext in client side > - > > Key: HDFS-16431 > URL: https://issues.apache.org/jira/browse/HDFS-16431 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nn >Reporter: Chengwei Wang >Assignee: Chengwei Wang >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > The context of CallerContext would be truncated when it exceeds the maximum > allowed length in server side. I think it's better to do check and truncate > in client side to reduce the unnecessary overhead of network and memory for > NN. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?focusedWorklogId=726006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726006 ] ASF GitHub Bot logged work on HDFS-16449: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:34 Start Date: 14/Feb/22 02:34 Worklog Time Spent: 10m Work Description: aajisaka merged pull request #3967: URL: https://github.com/apache/hadoop/pull/3967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 726006) Time Spent: 1h (was: 50m) > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.3 > > Time Spent: 1h > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16434) Add opname to read/write lock for remaining operations
[ https://issues.apache.org/jira/browse/HDFS-16434?focusedWorklogId=725998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725998 ] ASF GitHub Bot logged work on HDFS-16434: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:34 Start Date: 14/Feb/22 02:34 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3915: URL: https://github.com/apache/hadoop/pull/3915#issuecomment-1038532614 Hi @tasanuma @ayushtkn @Hexiaoqiao , could you please take a look at this. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725998) Time Spent: 1h (was: 50m) > Add opname to read/write lock for remaining operations > -- > > Key: HDFS-16434 > URL: https://issues.apache.org/jira/browse/HDFS-16434 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > In this issue at > [HDFS-10872|https://issues.apache.org/jira/browse/HDFS-10872], we add opname > to read and write locks. However, there are still many operations that have > not been completed. When analyzing some operations that hold locks for a long > time, we can only find specific methods through stack. I suggest that these > remaining operations be completed to facilitate later performance > optimization. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16422) Fix thread safety of EC decoding during concurrent preads
[ https://issues.apache.org/jira/browse/HDFS-16422?focusedWorklogId=725990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725990 ] ASF GitHub Bot logged work on HDFS-16422: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:33 Start Date: 14/Feb/22 02:33 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3881: URL: https://github.com/apache/hadoop/pull/3881#issuecomment-1038554948 Thanks for the fix. Should we also make the encoder thread safe? https://github.com/apache/hadoop/blob/7f72a4cdd5753645a11054db09657938f0ed19ce/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureEncoder.java#L40-L41 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725990) Time Spent: 3h 20m (was: 3h 10m) > Fix thread safety of EC decoding during concurrent preads > - > > Key: HDFS-16422 > URL: https://issues.apache.org/jira/browse/HDFS-16422 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsclient, ec, erasure-coding >Affects Versions: 3.3.0, 3.3.1 >Reporter: daimin >Assignee: daimin >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.3 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Reading data on an erasure-coded file with missing replicas(internal block of > block group) will cause online reconstruction: read dataUnits part of data > and decode them into the target missing data. Each DFSStripedInputStream > object has a RawErasureDecoder object, and when we doing pread concurrently, > RawErasureDecoder.decode will be invoked concurrently too. > RawErasureDecoder.decode is not thread safe, as a result of that we get wrong > data from pread occasionally. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15382) Split FsDatasetImpl from blockpool lock to blockpool volume lock
[
https://issues.apache.org/jira/browse/HDFS-15382?focusedWorklogId=725972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725972
]
ASF GitHub Bot logged work on HDFS-15382:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 02:31
Start Date: 14/Feb/22 02:31
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3941:
URL: https://github.com/apache/hadoop/pull/3941#issuecomment-1037381679
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 12 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 0s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 1m 24s |
[/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3941/2/artifact/out/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 464 unchanged
- 1 fixed = 465 total (was 465) |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 352 unchanged - 5
fixed = 352 total (was 357) |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 311m 9s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 435m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3941/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3941 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 2c67dae8e98b 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1cfe96b481092e7d3f16cd5328752de7ebc550b5 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Work logged] (HDFS-16396) Reconfig slow peer parameters for datanode
[
https://issues.apache.org/jira/browse/HDFS-16396?focusedWorklogId=725929=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725929
]
ASF GitHub Bot logged work on HDFS-16396:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 02:28
Start Date: 14/Feb/22 02:28
Worklog Time Spent: 10m
Work Description: tasanuma commented on a change in pull request #3827:
URL: https://github.com/apache/hadoop/pull/3827#discussion_r805442838
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java
##
@@ -57,26 +60,26 @@
* for outlier detection. If the number of samples is below this then
* outlier detection is skipped.
*/
- private final long minOutlierDetectionSamples;
+ private volatile long minOutlierDetectionSamples;
/**
* Threshold in milliseconds below which a DataNode is definitely not slow.
*/
- private final long lowThresholdMs;
+ private volatile long lowThresholdMs;
/**
* Minimum number of nodes to run outlier detection.
*/
- private final long minOutlierDetectionNodes;
+ private volatile long minOutlierDetectionNodes;
public DataNodePeerMetrics(final String name, Configuration conf) {
this.name = name;
minOutlierDetectionSamples = conf.getLong(
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_KEY,
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_DEFAULT);
lowThresholdMs =
-conf.getLong(DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
+conf.getLong(DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_DEFAULT);
minOutlierDetectionNodes =
-
conf.getLong(DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
+conf.getLong(DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_DEFAULT);
Review comment:
Why don't you import the static values of `..._DEFAULT` as well?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java
##
@@ -57,26 +60,26 @@
* for outlier detection. If the number of samples is below this then
* outlier detection is skipped.
*/
- private final long minOutlierDetectionSamples;
+ private volatile long minOutlierDetectionSamples;
/**
* Threshold in milliseconds below which a DataNode is definitely not slow.
*/
- private final long lowThresholdMs;
+ private volatile long lowThresholdMs;
/**
* Minimum number of nodes to run outlier detection.
*/
- private final long minOutlierDetectionNodes;
+ private volatile long minOutlierDetectionNodes;
public DataNodePeerMetrics(final String name, Configuration conf) {
this.name = name;
minOutlierDetectionSamples = conf.getLong(
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_KEY,
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_DEFAULT);
lowThresholdMs =
-conf.getLong(DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
+conf.getLong(DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_DEFAULT);
minOutlierDetectionNodes =
-
conf.getLong(DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
+conf.getLong(DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_DEFAULT);
this.slowNodeDetector =
new OutlierDetector(minOutlierDetectionNodes, lowThresholdMs);
Review comment:
`this.slowNodeDetector` has to update `minOutlierDetectionNodes` and
`lowThresholdMs` after reconfiguring them, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 725929)
Time Spent: 3h 50m (was: 3h 40m)
> Reconfig slow peer parameters for datanode
> --
>
> Key: HDFS-16396
> URL: https://issues.apache.org/jira/browse/HDFS-16396
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> In large clusters, rolling restart datanodes takes a long time. We can make
> slow peers parameters and slow disks parameters in datanode
[jira] [Work logged] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?focusedWorklogId=725891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725891 ] ASF GitHub Bot logged work on HDFS-16449: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:26 Start Date: 14/Feb/22 02:26 Worklog Time Spent: 10m Work Description: aajisaka commented on pull request #3967: URL: https://github.com/apache/hadoop/pull/3967#issuecomment-1038412832 Merged. Thank you @GuoPhilipse and @ayushtkn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725891) Time Spent: 50m (was: 40m) > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.3 > > Time Spent: 50m > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16440) RBF: Support router get HAServiceStatus with Lifeline RPC address
[ https://issues.apache.org/jira/browse/HDFS-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YulongZ updated HDFS-16440: --- Attachment: HDFS-16440.004.patch > RBF: Support router get HAServiceStatus with Lifeline RPC address > - > > Key: HDFS-16440 > URL: https://issues.apache.org/jira/browse/HDFS-16440 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: YulongZ >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-16440.001.patch, HDFS-16440.003.patch, > HDFS-16440.004.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > NamenodeHeartbeatService gets HAServiceStatus using > NNHAServiceTarget.getProxy. When we set a special > dfs.namenode.lifeline.rpc-address , NamenodeHeartbeatService may get > HAServiceStatus using NNHAServiceTarget.getHealthMonitorProxy. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16422) Fix thread safety of EC decoding during concurrent preads
[ https://issues.apache.org/jira/browse/HDFS-16422?focusedWorklogId=725872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725872 ] ASF GitHub Bot logged work on HDFS-16422: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:15 Start Date: 14/Feb/22 02:15 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3881: URL: https://github.com/apache/hadoop/pull/3881#issuecomment-1038554948 Thanks for the fix. Should we also make the encoder thread safe? https://github.com/apache/hadoop/blob/7f72a4cdd5753645a11054db09657938f0ed19ce/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureEncoder.java#L40-L41 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725872) Time Spent: 3h 10m (was: 3h) > Fix thread safety of EC decoding during concurrent preads > - > > Key: HDFS-16422 > URL: https://issues.apache.org/jira/browse/HDFS-16422 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsclient, ec, erasure-coding >Affects Versions: 3.3.0, 3.3.1 >Reporter: daimin >Assignee: daimin >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.3 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Reading data on an erasure-coded file with missing replicas(internal block of > block group) will cause online reconstruction: read dataUnits part of data > and decode them into the target missing data. Each DFSStripedInputStream > object has a RawErasureDecoder object, and when we doing pread concurrently, > RawErasureDecoder.decode will be invoked concurrently too. > RawErasureDecoder.decode is not thread safe, as a result of that we get wrong > data from pread occasionally. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16396) Reconfig slow peer parameters for datanode
[
https://issues.apache.org/jira/browse/HDFS-16396?focusedWorklogId=725870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725870
]
ASF GitHub Bot logged work on HDFS-16396:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 02:03
Start Date: 14/Feb/22 02:03
Worklog Time Spent: 10m
Work Description: tasanuma commented on a change in pull request #3827:
URL: https://github.com/apache/hadoop/pull/3827#discussion_r805442838
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java
##
@@ -57,26 +60,26 @@
* for outlier detection. If the number of samples is below this then
* outlier detection is skipped.
*/
- private final long minOutlierDetectionSamples;
+ private volatile long minOutlierDetectionSamples;
/**
* Threshold in milliseconds below which a DataNode is definitely not slow.
*/
- private final long lowThresholdMs;
+ private volatile long lowThresholdMs;
/**
* Minimum number of nodes to run outlier detection.
*/
- private final long minOutlierDetectionNodes;
+ private volatile long minOutlierDetectionNodes;
public DataNodePeerMetrics(final String name, Configuration conf) {
this.name = name;
minOutlierDetectionSamples = conf.getLong(
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_KEY,
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_DEFAULT);
lowThresholdMs =
-conf.getLong(DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
+conf.getLong(DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_DEFAULT);
minOutlierDetectionNodes =
-
conf.getLong(DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
+conf.getLong(DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_DEFAULT);
Review comment:
Why don't you import the static values of `..._DEFAULT` as well?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java
##
@@ -57,26 +60,26 @@
* for outlier detection. If the number of samples is below this then
* outlier detection is skipped.
*/
- private final long minOutlierDetectionSamples;
+ private volatile long minOutlierDetectionSamples;
/**
* Threshold in milliseconds below which a DataNode is definitely not slow.
*/
- private final long lowThresholdMs;
+ private volatile long lowThresholdMs;
/**
* Minimum number of nodes to run outlier detection.
*/
- private final long minOutlierDetectionNodes;
+ private volatile long minOutlierDetectionNodes;
public DataNodePeerMetrics(final String name, Configuration conf) {
this.name = name;
minOutlierDetectionSamples = conf.getLong(
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_KEY,
DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_DEFAULT);
lowThresholdMs =
-conf.getLong(DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
+conf.getLong(DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY,
DFSConfigKeys.DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_DEFAULT);
minOutlierDetectionNodes =
-
conf.getLong(DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
+conf.getLong(DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY,
DFSConfigKeys.DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_DEFAULT);
this.slowNodeDetector =
new OutlierDetector(minOutlierDetectionNodes, lowThresholdMs);
Review comment:
`this.slowNodeDetector` has to update `minOutlierDetectionNodes` and
`lowThresholdMs` after reconfiguring them, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 725870)
Time Spent: 3h 40m (was: 3.5h)
> Reconfig slow peer parameters for datanode
> --
>
> Key: HDFS-16396
> URL: https://issues.apache.org/jira/browse/HDFS-16396
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> In large clusters, rolling restart datanodes takes a long time. We can make
> slow peers parameters and slow disks parameters in datanode
[jira] [Comment Edited] (HDFS-16450) Give priority to releasing DNs with less free space
[
https://issues.apache.org/jira/browse/HDFS-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489575#comment-17489575
]
yanbin.zhang edited comment on HDFS-16450 at 2/14/22, 1:52 AM:
---
Could you please give some comments or suggestions. [~weichiu] [~hexiaoqiao]
[~ferhui]
was (Author: it_singer):
Could you please give some comments or suggestions?[~weichiu]
> Give priority to releasing DNs with less free space
> ---
>
> Key: HDFS-16450
> URL: https://issues.apache.org/jira/browse/HDFS-16450
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: yanbin.zhang
>Assignee: yanbin.zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16450.001.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When deleting redundant replicas, the one with the least free space should be
> prioritized.
> {code:java}
> //BlockPlacementPolicyDefault#chooseReplicaToDelete
> final DatanodeStorageInfo storage;
> if (oldestHeartbeatStorage != null) {
> storage = oldestHeartbeatStorage;
> } else if (minSpaceStorage != null) {
> storage = minSpaceStorage;
> } else {
> return null;
> }
> excessTypes.remove(storage.getStorageType());
> return storage; {code}
> Change the above logic to the following:
> {code:java}
> //BlockPlacementPolicyDefault#chooseReplicaToDelete
> final DatanodeStorageInfo storage;
> if (minSpaceStorage != null) {
> storage = minSpaceStorage;
> } else if (oldestHeartbeatStorage != null) {
> storage = oldestHeartbeatStorage;
> } else {
> return null;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16434) Add opname to read/write lock for remaining operations
[ https://issues.apache.org/jira/browse/HDFS-16434?focusedWorklogId=725864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725864 ] ASF GitHub Bot logged work on HDFS-16434: - Author: ASF GitHub Bot Created on: 14/Feb/22 01:31 Start Date: 14/Feb/22 01:31 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3915: URL: https://github.com/apache/hadoop/pull/3915#issuecomment-1038532614 Hi @tasanuma @ayushtkn @Hexiaoqiao , could you please take a look at this. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725864) Time Spent: 50m (was: 40m) > Add opname to read/write lock for remaining operations > -- > > Key: HDFS-16434 > URL: https://issues.apache.org/jira/browse/HDFS-16434 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > In this issue at > [HDFS-10872|https://issues.apache.org/jira/browse/HDFS-10872], we add opname > to read and write locks. However, there are still many operations that have > not been completed. When analyzing some operations that hold locks for a long > time, we can only find specific methods through stack. I suggest that these > remaining operations be completed to facilitate later performance > optimization. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-16449: - Fix Version/s: 3.2.4 Backported to branch-3.2. > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.3 > > Time Spent: 40m > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-16449: - Fix Version/s: 3.3.3 Backported to branch-3.3. > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Time Spent: 40m > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned HDFS-16449: Assignee: guophilipse > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-16449: - Issue Type: Bug (was: Improvement) > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HDFS-16449. -- Fix Version/s: 3.4.0 Resolution: Fixed Merged the PR into trunk. > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?focusedWorklogId=725835=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725835 ] ASF GitHub Bot logged work on HDFS-16449: - Author: ASF GitHub Bot Created on: 13/Feb/22 20:38 Start Date: 13/Feb/22 20:38 Worklog Time Spent: 10m Work Description: aajisaka commented on pull request #3967: URL: https://github.com/apache/hadoop/pull/3967#issuecomment-1038412832 Merged. Thank you @GuoPhilipse and @ayushtkn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725835) Time Spent: 40m (was: 0.5h) > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16449) Fix hadoop web site release notes and changelog not available
[ https://issues.apache.org/jira/browse/HDFS-16449?focusedWorklogId=725834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725834 ] ASF GitHub Bot logged work on HDFS-16449: - Author: ASF GitHub Bot Created on: 13/Feb/22 20:38 Start Date: 13/Feb/22 20:38 Worklog Time Spent: 10m Work Description: aajisaka merged pull request #3967: URL: https://github.com/apache/hadoop/pull/3967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725834) Time Spent: 0.5h (was: 20m) > Fix hadoop web site release notes and changelog not available > - > > Key: HDFS-16449 > URL: https://issues.apache.org/jira/browse/HDFS-16449 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Fix hadoop web site release notes and changelog not available -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491596#comment-17491596
]
Hadoop QA commented on HDFS-16456:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
11s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
57s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 26s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 27m
9s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/754/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new +
4 unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 43s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green}{color} |
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Status: Open (was: Patch Available)
> EC: Decommission a rack with only on dn will fail when the rack number is
> equal with replication
>
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Priority: Critical
> Attachments: HDFS-16456.001.patch
>
>
> In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
> # Enable EC policy, such as RS-6-3-1024k.
> # The rack number in this cluster is equal with the replication number(9)
> # A rack only has one DN, and decommission this DN.
> The root cause is in
> BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
> give a limit parameter maxNodesPerRack for choose targets. In this scenario,
> the maxNodesPerRack is 1, which means each rack can only be chosen one
> datanode.
> {code:java}
> protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
>...
> // If more replicas than racks, evenly spread the replicas.
> // This calculation rounds up.
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> return new int[] {numOfReplicas, maxNodesPerRack};
> } {code}
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> here will be called, where totalNumOfReplicas=9 and numOfRacks=9
> When we decommission one dn which is only one node in its rack, the
> chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
> will throw NotEnoughReplicasException, but the exception will not be caught
> and fail to fallback to chooseEvenlyFromRemainingRacks() function.
> When decommission, after choose targets, verifyBlockPlacement() function will
> return the total rack number contains the invalid rack, and
> BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false
> and it will also cause decommission fail.
> {code:java}
> public BlockPlacementStatus verifyBlockPlacement(DatanodeInfo[] locs,
> int numberOfReplicas) {
> if (locs == null)
> locs = DatanodeDescriptor.EMPTY_ARRAY;
> if (!clusterMap.hasClusterEverBeenMultiRack()) {
> // only one rack
> return new BlockPlacementStatusDefault(1, 1, 1);
> }
> // Count locations on different racks.
> Set racks = new HashSet<>();
> for (DatanodeInfo dn : locs) {
> racks.add(dn.getNetworkLocation());
> }
> return new BlockPlacementStatusDefault(racks.size(), numberOfReplicas,
> clusterMap.getNumOfRacks());
> } {code}
> {code:java}
> public boolean isPlacementPolicySatisfied() {
> return requiredRacks <= currentRacks || currentRacks >= totalRacks;
> }{code}
> According to the above description, we should make the below modify to fix it:
> # In startDecommission() or stopDecommission(), we should also change the
> numOfRacks in class NetworkTopology. Or choose targets may fail for the
> maxNodesPerRack is too small. And even choose targets success,
> isPlacementPolicySatisfied will also return false cause decommission fail.
> # In BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder(), the first
> chooseOnce() function should also be put in try..catch..., or it will not
> fallback to call chooseEvenlyFromRemainingRacks() when throw exception.
> # In chooseEvenlyFromRemainingRacks(), this numResultsOflastChoose =
> results.size(); code should be move to after chooseOnce(), or it will throw
> lastException and make choose targets failed.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Affects Version/s: (was: 3.1.1)
> EC: Decommission a rack with only on dn will fail when the rack number is
> equal with replication
>
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Priority: Critical
> Attachments: HDFS-16456.001.patch
>
>
> In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
> # Enable EC policy, such as RS-6-3-1024k.
> # The rack number in this cluster is equal with the replication number(9)
> # A rack only has one DN, and decommission this DN.
> The root cause is in
> BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
> give a limit parameter maxNodesPerRack for choose targets. In this scenario,
> the maxNodesPerRack is 1, which means each rack can only be chosen one
> datanode.
> {code:java}
> protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
>...
> // If more replicas than racks, evenly spread the replicas.
> // This calculation rounds up.
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> return new int[] {numOfReplicas, maxNodesPerRack};
> } {code}
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> here will be called, where totalNumOfReplicas=9 and numOfRacks=9
> When we decommission one dn which is only one node in its rack, the
> chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
> will throw NotEnoughReplicasException, but the exception will not be caught
> and fail to fallback to chooseEvenlyFromRemainingRacks() function.
> When decommission, after choose targets, verifyBlockPlacement() function will
> return the total rack number contains the invalid rack, and
> BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false
> and it will also cause decommission fail.
> {code:java}
> public BlockPlacementStatus verifyBlockPlacement(DatanodeInfo[] locs,
> int numberOfReplicas) {
> if (locs == null)
> locs = DatanodeDescriptor.EMPTY_ARRAY;
> if (!clusterMap.hasClusterEverBeenMultiRack()) {
> // only one rack
> return new BlockPlacementStatusDefault(1, 1, 1);
> }
> // Count locations on different racks.
> Set racks = new HashSet<>();
> for (DatanodeInfo dn : locs) {
> racks.add(dn.getNetworkLocation());
> }
> return new BlockPlacementStatusDefault(racks.size(), numberOfReplicas,
> clusterMap.getNumOfRacks());
> } {code}
> {code:java}
> public boolean isPlacementPolicySatisfied() {
> return requiredRacks <= currentRacks || currentRacks >= totalRacks;
> }{code}
> According to the above description, we should make the below modify to fix it:
> # In startDecommission() or stopDecommission(), we should also change the
> numOfRacks in class NetworkTopology. Or choose targets may fail for the
> maxNodesPerRack is too small. And even choose targets success,
> isPlacementPolicySatisfied will also return false cause decommission fail.
> # In BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder(), the first
> chooseOnce() function should also be put in try..catch..., or it will not
> fallback to call chooseEvenlyFromRemainingRacks() when throw exception.
> # In chooseEvenlyFromRemainingRacks(), this numResultsOflastChoose =
> results.size(); code should be move to after chooseOnce(), or it will throw
> lastException and make choose targets failed.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Status: Patch Available (was: Open)
> EC: Decommission a rack with only on dn will fail when the rack number is
> equal with replication
>
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Priority: Critical
> Attachments: HDFS-16456.001.patch
>
>
> In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
> # Enable EC policy, such as RS-6-3-1024k.
> # The rack number in this cluster is equal with the replication number(9)
> # A rack only has one DN, and decommission this DN.
> The root cause is in
> BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
> give a limit parameter maxNodesPerRack for choose targets. In this scenario,
> the maxNodesPerRack is 1, which means each rack can only be chosen one
> datanode.
> {code:java}
> protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
>...
> // If more replicas than racks, evenly spread the replicas.
> // This calculation rounds up.
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> return new int[] {numOfReplicas, maxNodesPerRack};
> } {code}
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> here will be called, where totalNumOfReplicas=9 and numOfRacks=9
> When we decommission one dn which is only one node in its rack, the
> chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
> will throw NotEnoughReplicasException, but the exception will not be caught
> and fail to fallback to chooseEvenlyFromRemainingRacks() function.
> When decommission, after choose targets, verifyBlockPlacement() function will
> return the total rack number contains the invalid rack, and
> BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false
> and it will also cause decommission fail.
> {code:java}
> public BlockPlacementStatus verifyBlockPlacement(DatanodeInfo[] locs,
> int numberOfReplicas) {
> if (locs == null)
> locs = DatanodeDescriptor.EMPTY_ARRAY;
> if (!clusterMap.hasClusterEverBeenMultiRack()) {
> // only one rack
> return new BlockPlacementStatusDefault(1, 1, 1);
> }
> // Count locations on different racks.
> Set racks = new HashSet<>();
> for (DatanodeInfo dn : locs) {
> racks.add(dn.getNetworkLocation());
> }
> return new BlockPlacementStatusDefault(racks.size(), numberOfReplicas,
> clusterMap.getNumOfRacks());
> } {code}
> {code:java}
> public boolean isPlacementPolicySatisfied() {
> return requiredRacks <= currentRacks || currentRacks >= totalRacks;
> }{code}
> According to the above description, we should make the below modify to fix it:
> # In startDecommission() or stopDecommission(), we should also change the
> numOfRacks in class NetworkTopology. Or choose targets may fail for the
> maxNodesPerRack is too small. And even choose targets success,
> isPlacementPolicySatisfied will also return false cause decommission fail.
> # In BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder(), the first
> chooseOnce() function should also be put in try..catch..., or it will not
> fallback to call chooseEvenlyFromRemainingRacks() when throw exception.
> # In chooseEvenlyFromRemainingRacks(), this numResultsOflastChoose =
> results.size(); code should be move to after chooseOnce(), or it will throw
> lastException and make choose targets failed.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Description:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
...
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where totalNumOfReplicas=9 and numOfRacks=9
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
When decommission, after choose targets, verifyBlockPlacement() function will
return the total rack number contains the invalid rack, and
BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false and
it will also cause decommission fail.
{code:java}
public BlockPlacementStatus verifyBlockPlacement(DatanodeInfo[] locs,
int numberOfReplicas) {
if (locs == null)
locs = DatanodeDescriptor.EMPTY_ARRAY;
if (!clusterMap.hasClusterEverBeenMultiRack()) {
// only one rack
return new BlockPlacementStatusDefault(1, 1, 1);
}
// Count locations on different racks.
Set racks = new HashSet<>();
for (DatanodeInfo dn : locs) {
racks.add(dn.getNetworkLocation());
}
return new BlockPlacementStatusDefault(racks.size(), numberOfReplicas,
clusterMap.getNumOfRacks());
} {code}
{code:java}
public boolean isPlacementPolicySatisfied() {
return requiredRacks <= currentRacks || currentRacks >= totalRacks;
}{code}
According to the above description, we should make the below modify to fix it:
# In startDecommission() or stopDecommission(), we should also change the
numOfRacks in class NetworkTopology. Or choose targets may fail for the
maxNodesPerRack is too small. And even choose targets success,
isPlacementPolicySatisfied will also return false cause decommission fail.
# In BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder(), the first
chooseOnce() function should also be put in try..catch..., or it will not
fallback to call chooseEvenlyFromRemainingRacks() when throw exception.
# In chooseEvenlyFromRemainingRacks(), this numResultsOflastChoose =
results.size(); code should be move to after chooseOnce(), or it will throw
lastException and make choose targets failed.
was:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where
[jira] [Comment Edited] (HDFS-14530) libhdfspp is missing functions like hdfsWrite
[
https://issues.apache.org/jira/browse/HDFS-14530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491436#comment-17491436
]
Ahmed Riza edited comment on HDFS-14530 at 2/13/22, 2:19 PM:
-
I can confirm what [~write2kish...@gmail.com] has found. I've checked the
`libhdfspp` distributed with *hadoop-3.3.1* as well as one I've built from
latest master. Both versions do not contain an implementation of the
`hdfsWrite` function. Example:
{code:java}
strings hadoop-3.3.1/lib/native/libhdfspp.so.0.1.0 | grep hdfsWrite{code}
returns no results, In contrast, we can see that the `hdfsRead` function is
available, for example:
{code:java}
strings hadoop-3.3.1/lib/native/libhdfspp.so.0.1.0 | grep hdfsRead
hdfsReadStatisticsGetRemoteBytesRead
hdfsRead
hdfsReadStatistics
hdfsReadStatisticsGetRemoteBytesRead
hdfsRead
hdfsRead
hdfsReadStatisticsGetRemoteBytesRead
{code}
Reading the through the code base, as far as I can tell, there's currently no
write implementation:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.h
The implementation of the library is new to me, but willing to learn and
contribute if I can to a write api.
was (Author: ahmedriza):
I can confirm what [~write2kish...@gmail.com] has found. I've checked the
`libhdfspp` distributed with *hadoop-3.3.1* as well as one I've built from
latest master. Both versions do not contain an implementation of the
`hdfsWrite` function. Example:
{code:java}
strings hadoop-3.3.1/lib/native/libhdfspp.so.0.1.0 | grep hdfsWrite{code}
returns no results, In contrast, we can see that the `hdfsRead` function is
available, for example:
{code:java}
strings hadoop-3.3.1/lib/native/libhdfspp.so.0.1.0 | grep hdfsRead
hdfsReadStatisticsGetRemoteBytesRead
hdfsRead
hdfsReadStatistics
hdfsReadStatisticsGetRemoteBytesRead
hdfsRead
hdfsRead
hdfsReadStatisticsGetRemoteBytesRead
{code}
It would be great, if someone could confirm whether there's a plan to add
`hdfsWrite` support to `libhdfspp`. The implementation of the library is new
to me, but willing to learn and contribute if I can.
> libhdfspp is missing functions like hdfsWrite
> -
>
> Key: HDFS-14530
> URL: https://issues.apache.org/jira/browse/HDFS-14530
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs++
>Affects Versions: 3.1.0, 2.8.5
>Reporter: Krishna Kishore
>Priority: Major
> Fix For: 3.1.0, 2.8.5
>
>
> I have downloaded code from [https://github.com/apache/hadoop] and compiled
> libhdfspp. I couldn't find how to use this library, libhdfspp.so . I see that
> some functions like hdfsWrite(), hdfsHSync() are missing from this library.
> This library is unusable without the write functionality. Please let me know
> if there is a way to use the write functionality with current hdfs++ code or
> if there are plans for adding this functionality.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Description:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where totalNumOfReplicas=9 and numOfRacks=9
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
When decommission, after choose targets, verifyBlockPlacement() function will
return the total rack number contains the invalid rack, and
BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false and
it will also cause decommission fail.
{code:java}
public BlockPlacementStatus verifyBlockPlacement(DatanodeInfo[] locs,
int numberOfReplicas) {
if (locs == null)
locs = DatanodeDescriptor.EMPTY_ARRAY;
if (!clusterMap.hasClusterEverBeenMultiRack()) {
// only one rack
return new BlockPlacementStatusDefault(1, 1, 1);
}
// Count locations on different racks.
Set racks = new HashSet<>();
for (DatanodeInfo dn : locs) {
racks.add(dn.getNetworkLocation());
}
return new BlockPlacementStatusDefault(racks.size(), numberOfReplicas,
clusterMap.getNumOfRacks());
} {code}
{code:java}
public boolean isPlacementPolicySatisfied() {
return requiredRacks <= currentRacks || currentRacks >= totalRacks;
}
{code}
was:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Description:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where totalNumOfReplicas=9 and numOfRacks=9
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
When decommission, after choose targets, verifyBlockPlacement() function will
return the total rack number contains the invalid rack, and
BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false and
it will also cause decommission fail.
{code:java}
public BlockPlacementStatus verifyBlockPlacement(DatanodeInfo[] locs,
int numberOfReplicas) {
if (locs == null)
locs = DatanodeDescriptor.EMPTY_ARRAY;
if (!clusterMap.hasClusterEverBeenMultiRack()) {
// only one rack
return new BlockPlacementStatusDefault(1, 1, 1);
}
// Count locations on different racks.
Set racks = new HashSet<>();
for (DatanodeInfo dn : locs) {
racks.add(dn.getNetworkLocation());
}
return new BlockPlacementStatusDefault(racks.size(), numberOfReplicas,
clusterMap.getNumOfRacks());
} {code}
{code:java}
public boolean isPlacementPolicySatisfied() {
return requiredRacks <= currentRacks || currentRacks >= totalRacks;
}
{code}
was:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Description:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where totalNumOfReplicas=9 and numOfRacks=9
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
When decommission, after choose targets, verifyBlockPlacement() function will
return the total rack number contains the invalid rack, and
BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false and
it will also cause decommission fail.
{code:java}
public boolean isPlacementPolicySatisfied() {
return requiredRacks <= currentRacks || currentRacks >= totalRacks;
}
{code}
was:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where totalNumOfReplicas=9 and numOfRacks=9
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
When decommission, after choose targets, verifyBlockPlacement() function will
return the total rack number contains the invalid rack, and
BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false and
it will also cause decommission fail.
> EC:
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Priority: Critical (was: Major)
> EC: Decommission a rack with only on dn will fail when the rack number is
> equal with replication
>
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1, 3.4.0
>Reporter: caozhiqiang
>Priority: Critical
> Attachments: HDFS-16456.001.patch
>
>
> In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
> # Enable EC policy, such as RS-6-3-1024k.
> # The rack number in this cluster is equal with the replication number(9)
> # A rack only has one DN, and decommission this DN.
> The root cause is in
> BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
> give a limit parameter maxNodesPerRack for choose targets. In this scenario,
> the maxNodesPerRack is 1, which means each rack can only be chosen one
> datanode.
>
> {code:java}
> protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
> int clusterSize = clusterMap.getNumOfLeaves();
> int totalNumOfReplicas = numOfChosen + numOfReplicas;
> if (totalNumOfReplicas > clusterSize) {
> numOfReplicas -= (totalNumOfReplicas-clusterSize);
> totalNumOfReplicas = clusterSize;
> }
> // No calculation needed when there is only one rack or picking one node.
> int numOfRacks = clusterMap.getNumOfRacks();
> // HDFS-14527 return default when numOfRacks = 0 to avoid
> // ArithmeticException when calc maxNodesPerRack at following logic.
> if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
> return new int[] {numOfReplicas, totalNumOfReplicas};
> }
> // If more racks than replicas, put one replica per rack.
> if (totalNumOfReplicas < numOfRacks) {
> return new int[] {numOfReplicas, 1};
> }
> // If more replicas than racks, evenly spread the replicas.
> // This calculation rounds up.
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> return new int[] {numOfReplicas, maxNodesPerRack};
> } {code}
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> here will be called, where totalNumOfReplicas=9 and numOfRacks=9
>
> When we decommission one dn which is only one node in its rack, the
> chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
> will throw NotEnoughReplicasException, but the exception will not be caught
> and fail to fallback to chooseEvenlyFromRemainingRacks() function.
> When decommission, after choose targets, verifyBlockPlacement() function will
> return the total rack number contains the invalid rack, and
> BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false
> and it will also cause decommission fail.
> {code:java}
> public boolean isPlacementPolicySatisfied() {
> return requiredRacks <= currentRacks || currentRacks >= totalRacks;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Description:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where totalNumOfReplicas=9 and numOfRacks=9
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
When decommission, after choose targets, verifyBlockPlacement() function will
return the total rack number contains the invalid rack, and
BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false and
it will also cause decommission fail.
was:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where totalNumOfReplicas=9 and numOfRacks=9
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
> EC: Decommission a rack with only on dn will fail when the rack number is
> equal with replication
>
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Affects Version/s: 3.4.0
Status: Patch Available (was: Open)
> EC: Decommission a rack with only on dn will fail when the rack number is
> equal with replication
>
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1, 3.4.0
>Reporter: caozhiqiang
>Priority: Major
> Attachments: HDFS-16456.001.patch
>
>
> In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
> # Enable EC policy, such as RS-6-3-1024k.
> # The rack number in this cluster is equal with the replication number(9)
> # A rack only has one DN, and decommission this DN.
> The root cause is in
> BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
> give a limit parameter maxNodesPerRack for choose targets. In this scenario,
> the maxNodesPerRack is 1, which means each rack can only be chosen one
> datanode.
>
> {code:java}
> protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
> int clusterSize = clusterMap.getNumOfLeaves();
> int totalNumOfReplicas = numOfChosen + numOfReplicas;
> if (totalNumOfReplicas > clusterSize) {
> numOfReplicas -= (totalNumOfReplicas-clusterSize);
> totalNumOfReplicas = clusterSize;
> }
> // No calculation needed when there is only one rack or picking one node.
> int numOfRacks = clusterMap.getNumOfRacks();
> // HDFS-14527 return default when numOfRacks = 0 to avoid
> // ArithmeticException when calc maxNodesPerRack at following logic.
> if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
> return new int[] {numOfReplicas, totalNumOfReplicas};
> }
> // If more racks than replicas, put one replica per rack.
> if (totalNumOfReplicas < numOfRacks) {
> return new int[] {numOfReplicas, 1};
> }
> // If more replicas than racks, evenly spread the replicas.
> // This calculation rounds up.
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> return new int[] {numOfReplicas, maxNodesPerRack};
> } {code}
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> here will be called, where totalNumOfReplicas=9 and numOfRacks=9
>
> When we decommission one dn which is only one node in its rack, the
> chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
> will throw NotEnoughReplicasException, but the exception will not be caught
> and fail to fallback to chooseEvenlyFromRemainingRacks() function.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated HDFS-16456:
---
Description:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
{code:java}
protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
int clusterSize = clusterMap.getNumOfLeaves();
int totalNumOfReplicas = numOfChosen + numOfReplicas;
if (totalNumOfReplicas > clusterSize) {
numOfReplicas -= (totalNumOfReplicas-clusterSize);
totalNumOfReplicas = clusterSize;
}
// No calculation needed when there is only one rack or picking one node.
int numOfRacks = clusterMap.getNumOfRacks();
// HDFS-14527 return default when numOfRacks = 0 to avoid
// ArithmeticException when calc maxNodesPerRack at following logic.
if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
return new int[] {numOfReplicas, totalNumOfReplicas};
}
// If more racks than replicas, put one replica per rack.
if (totalNumOfReplicas < numOfRacks) {
return new int[] {numOfReplicas, 1};
}
// If more replicas than racks, evenly spread the replicas.
// This calculation rounds up.
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
return new int[] {numOfReplicas, maxNodesPerRack};
} {code}
int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
here will be called, where totalNumOfReplicas=9 and numOfRacks=9
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
was:
In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
# Enable EC policy, such as RS-6-3-1024k.
# The rack number in this cluster is equal with the replication number(9)
# A rack only has one DN, and decommission this DN.
The root cause is in
BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
give a limit parameter maxNodesPerRack for choose targets. In this scenario,
the maxNodesPerRack is 1, which means each rack can only be chosen one datanode.
When we decommission one dn which is only one node in its rack, the
chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
will throw NotEnoughReplicasException, but the exception will not be caught and
fail to fallback to chooseEvenlyFromRemainingRacks() function.
> EC: Decommission a rack with only on dn will fail when the rack number is
> equal with replication
>
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1
>Reporter: caozhiqiang
>Priority: Major
> Attachments: HDFS-16456.001.patch
>
>
> In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
> # Enable EC policy, such as RS-6-3-1024k.
> # The rack number in this cluster is equal with the replication number(9)
> # A rack only has one DN, and decommission this DN.
> The root cause is in
> BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
> give a limit parameter maxNodesPerRack for choose targets. In this scenario,
> the maxNodesPerRack is 1, which means each rack can only be chosen one
> datanode.
>
> {code:java}
> protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
> int clusterSize = clusterMap.getNumOfLeaves();
> int totalNumOfReplicas = numOfChosen + numOfReplicas;
> if (totalNumOfReplicas > clusterSize) {
> numOfReplicas -= (totalNumOfReplicas-clusterSize);
> totalNumOfReplicas = clusterSize;
> }
> // No calculation needed when there is only one rack or picking one node.
> int numOfRacks = clusterMap.getNumOfRacks();
> // HDFS-14527 return default when numOfRacks = 0 to avoid
> // ArithmeticException when calc maxNodesPerRack at following logic.
> if (numOfRacks <= 1 || totalNumOfReplicas <= 1) {
> return new int[] {numOfReplicas, totalNumOfReplicas};
> }
> // If more racks than replicas, put one replica per rack.
> if (totalNumOfReplicas
[jira] [Updated] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[ https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caozhiqiang updated HDFS-16456: --- Attachment: HDFS-16456.001.patch > EC: Decommission a rack with only on dn will fail when the rack number is > equal with replication > > > Key: HDFS-16456 > URL: https://issues.apache.org/jira/browse/HDFS-16456 > Project: Hadoop HDFS > Issue Type: Bug > Components: ec, namenode >Affects Versions: 3.1.1 >Reporter: caozhiqiang >Priority: Major > Attachments: HDFS-16456.001.patch > > > In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason: > # Enable EC policy, such as RS-6-3-1024k. > # The rack number in this cluster is equal with the replication number(9) > # A rack only has one DN, and decommission this DN. > The root cause is in > BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will > give a limit parameter maxNodesPerRack for choose targets. In this scenario, > the maxNodesPerRack is 1, which means each rack can only be chosen one > datanode. > When we decommission one dn which is only one node in its rack, the > chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder() > will throw NotEnoughReplicasException, but the exception will not be caught > and fail to fallback to chooseEvenlyFromRemainingRacks() function. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
caozhiqiang created HDFS-16456: -- Summary: EC: Decommission a rack with only on dn will fail when the rack number is equal with replication Key: HDFS-16456 URL: https://issues.apache.org/jira/browse/HDFS-16456 Project: Hadoop HDFS Issue Type: Bug Components: ec, namenode Affects Versions: 3.1.1 Reporter: caozhiqiang In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason: # Enable EC policy, such as RS-6-3-1024k. # The rack number in this cluster is equal with the replication number(9) # A rack only has one DN, and decommission this DN. The root cause is in BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will give a limit parameter maxNodesPerRack for choose targets. In this scenario, the maxNodesPerRack is 1, which means each rack can only be chosen one datanode. When we decommission one dn which is only one node in its rack, the chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder() will throw NotEnoughReplicasException, but the exception will not be caught and fail to fallback to chooseEvenlyFromRemainingRacks() function. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
