[
https://issues.apache.org/jira/browse/HDFS-11119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651987#comment-15651987
]
Anu Engineer edited comment on HDFS-11119 at 11/9/16 8:45 PM:
--------------------------------------------------------------
[~arpitagarwal] Thanks for the patch. Some minor comments
The way time is specified is little confusing. Just wanted to make sure that my
understanding is correct.
{noformat}
public static final String DFS_DATANODE_DISK_CHECK_TIMEOUT_DEFAULT ="10m";
public static final String DFS_DATANODE_DISK_CHECK_MIN_GAP_DEFAULT ="15m";
{noformat}
This is milliseconds right ? but if that is hardcoded in code, then why take
an "m" if all of these are time units in milliseconds.
When I first read code I read 15m as 15 minutes and later realized that it is
15 milliseconds. I think we should pick a larger min gap and also if possible
specify that the time unit is not flexible.
In hdfs-default.xml also can we specify that the time unit is in milliseconds.
It is obvious from code, but I am worried some user might try to specify
something like H.
{{StorageLocationsChecker.java#check}}
{noformat}
if (goodLocations.size() == 0) {
throw new IOException("All directories in "
+ DFS_DATANODE_DATA_DIR_KEY + " are invalid: "
+ failedLocations);
}
{noformat}
Should we Log.error this case too ?
{{StorageLocationsChecker.java}}
Executors.newCachedThreadPool()
Can we take a ThreadFactory so we can set the name of threads in this pool ?
Plus, Are these threads daemons ?
nit : I am presuming the change in Datanode startup is coming in a later patch
?
was (Author: anu):
[~arpitagarwal] Thanks for the patch. Some minor comments
The way time is specified is little confusing. Just wanted to make sure that my
understanding is correct.
{noformat}
public static final String DFS_DATANODE_DISK_CHECK_TIMEOUT_DEFAULT ="10m";
public static final String DFS_DATANODE_DISK_CHECK_MIN_GAP_DEFAULT ="15m";
{noformat}
This is milliseconds right ? but if that is hardcoded in code, then why take
an "m" if all of these are time units in milliseconds.
When I first read code I read 15m as 15 minutes and later realized that it is
15 milliseconds. I think we should pick a larger min gap and also if possible
specify that the time unit is not flexible.
In hdfs-default.xml also can we specify that the time unit is in milliseconds.
It is obvious from code, but I am worried some user might try to specify
something like H.
{{StorageLocationsChecker.java#check}}
{noformat}
if (goodLocations.size() == 0) {
throw new IOException("All directories in "
+ DFS_DATANODE_DATA_DIR_KEY + " are invalid: "
+ failedLocations);
}
{noformat}
Should we Log.error this case too ?
{{StorageLocationsChecker.java}}
Executors.newCachedThreadPool()
Can we a ThreadFactory so we can set the name of threads in this pool ? Plus,
Are these threads daemons ?
nit : I am presuming the change in Datanode startup is coming in a later patch
?
> Support for parallel checking of StorageLocations on DataNode startup
> ---------------------------------------------------------------------
>
> Key: HDFS-11119
> URL: https://issues.apache.org/jira/browse/HDFS-11119
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
>
> The {{AsyncChecker}} support introduced by HDFS-11114 can be used to
> parallelize checking {{StorageLocation}} s on Datanode startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
