Szymon Orzechowski created HDFS-17548:
-----------------------------------------
Summary: excessive NO_REQUIRED_STORAGE_TYPE messages
Key: HDFS-17548
URL: https://issues.apache.org/jira/browse/HDFS-17548
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.3.4
Reporter: Szymon Orzechowski
Notification of unavailable storageType has been implemented in HDFS-15815.
Yesterday we noted a failure on our production cluster. As a side result of
analyzing the reasons for the failure, we found additional error messages:
nn-3.wphadoop.dc-2.jumbo.xxxx_hadoop-hdfs-namenode.log.out:2024-06-07
00:35:23,381 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not enough
replicas was chosen. Reason: \{NO_REQUIRED_STORAGE_TYPE=1}
These tell us very little and seem to make absolutely no sense in the case of
our cluster (12 racks, no storage policies enabled nor storage types defined).
However, in 100% of cases they occur directly (or almost directly) after
messages like:
nn-3.wphadoop.dc-2.jumbo.xxxx_hadoop-hdfs-namenode.log.out-2024-06-07
00:35:23,380 INFO org.apache.hadoop.ipc.Server: IPC Server handler 25 on
default port 8020, call#9866 Retry#0
org.apache.hadoop.hdfs.protocol.ClientProtocol. create from 10.32.20.25:35130:
org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException:
The directory item limit of /user/gobblin/loghost/failures/dot_ma/undefined is
exceeded: limit=1048576 items=1048576
Which leads me to the conclusion that in this case the NO_REQUIRED_STORAGE_TYPE
errors are raised due to reaching the limit specified in property
dfs.namenode.fs-limits.max-directory-items. Perhaps they should be restricted
as they provide no information and actually report a non-existent problem.
Additionally, immediately after clearing the
/user/gobblin/loghost/failures/dot_ma/undefined directory, the
NO_REQUIRED_STORAGE_TYPE messages stopped appearing.
---
I would also like to take this opportunity to ask where to find any list,
specifying meaning of values used in the NO_REQUIRED_STORAGE_TYPE=1 messages
(in this case, 1)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
