[
https://issues.apache.org/jira/browse/HADOOP-19554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17948249#comment-17948249
]
ASF GitHub Bot commented on HADOOP-19554:
-----------------------------------------
cnauroth commented on code in PR #7651:
URL: https://github.com/apache/hadoop/pull/7651#discussion_r2067072005
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java:
##########
@@ -393,6 +396,8 @@ int getCurrentDirectoryIndex() {
*/
public Path getLocalPathForWrite(String pathStr, long size,
Configuration conf, boolean checkWrite) throws IOException {
+ LOG.debug("searchng for directory for file at {}, size = {};
checkWrite={}",
Review Comment:
Minor typo: "searching".
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java:
##########
@@ -406,27 +411,40 @@ public Path getLocalPathForWrite(String pathStr, long
size,
pathStr = pathStr.substring(1);
}
Path returnPath = null;
-
- if(size == SIZE_UNKNOWN) { //do roulette selection: pick dir with
probability
- //proportional to available size
- long[] availableOnDisk = new long[ctx.dirDF.length];
- long totalAvailable = 0;
-
- //build the "roulette wheel"
- for(int i =0; i < ctx.dirDF.length; ++i) {
- final DF target = ctx.dirDF[i];
- // attempt to recreate the dir so that getAvailable() is valid
- // if it fails, getAvailable() will return 0, so the dir will
- // be declared unavailable.
- // return value is logged at debug to keep spotbugs quiet.
- final boolean b = new File(target.getDirPath()).mkdirs();
- LOG.debug("mkdirs of {}={}", target, b);
- availableOnDisk[i] = target.getAvailable();
- totalAvailable += availableOnDisk[i];
+
+ final int dirCount = ctx.dirDF.length;
+ long[] availableOnDisk = new long[dirCount];
+ long totalAvailable = 0;
+
+ StringBuilder pathNames = new StringBuilder();
+
+ //build the "roulette wheel"
+ for (int i =0; i < dirCount; ++i) {
+ final DF target = ctx.dirDF[i];
+ // attempt to recreate the dir so that getAvailable() is valid
+ // if it fails, getAvailable() will return 0, so the dir will
+ // be declared unavailable.
+ // return value is logged at debug to keep spotbugs quiet.
+ final String name = target.getDirPath();
+ pathNames.append(" ").append(name);
+ final File dirPath = new File(name);
+ if (!dirPath.exists()) {
+ LOG.debug("creating buffer dir {}}", name);
+ dirPath.mkdirs();
Review Comment:
Do you want to debug-log the boolean return value? It should almost always
be `true` now that you added the `exists()` check, but it can still be `false`
for edge cases like permissions violations.
> LocalDirAllocator still doesn't always recover from directory tree deletion
> ---------------------------------------------------------------------------
>
> Key: HADOOP-19554
> URL: https://issues.apache.org/jira/browse/HADOOP-19554
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.4.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> In HADOOP-18636, LocalDirAllocator was modified to recreate missing dirs. But
> it appears that there are still codepaths which don't do that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
