[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-11296:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.9.0
Status: Resolved (was: Patch Available)
+1. The failed tests aren't related. Committed to trunk and branch-2. Thanks
[~manojg] for the contribution. Thanks [~eddyxu] for the review.
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HDFS-11296.01.patch, HDFS-11296.02.patch,
> HDFS-11296.03.patch, HDFS-11296-branch-2.01.patch,
> HDFS-11296-branch-2.02.patch, HDFS-11296-branch-2.03.patch
>
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11296:
--
Attachment: HDFS-11296.03.patch
HDFS-11296-branch-2.03.patch
Attached v03 patch with the TestNameNodeMXBean fix.
[~eddyxu], [~mingma], kindly take a look at the latest patch.
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11296.01.patch, HDFS-11296.02.patch,
> HDFS-11296.03.patch, HDFS-11296-branch-2.01.patch,
> HDFS-11296-branch-2.02.patch, HDFS-11296-branch-2.03.patch
>
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11296:
--
Attachment: HDFS-11296.02.patch
HDFS-11296-branch-2.02.patch
Thanks for the review [~mingma] and [~eddyxu].
Attached 02 patch with System.currentTimeMillis() replaced with the util
Time.now(). Kindly take a look.
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11296.01.patch, HDFS-11296.02.patch,
> HDFS-11296-branch-2.01.patch, HDFS-11296-branch-2.02.patch
>
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11296:
--
Attachment: HDFS-11296-branch-2.01.patch
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11296-branch-2.01.patch, HDFS-11296.01.patch
>
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11296:
--
Status: Patch Available (was: Open)
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11296.01.patch
>
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11296:
--
Description:
Currently it is possible to configure an expiry time in milliseconds for a
DataNode in maintenance state. As per the design, the expiry attribute is an
absolute time, beyond which NameNode starts to stop the ongoing maintenance
operation for that DataNode. Internally in the code, this expiry time is read
and checked against {{Time.monotonicNow()}} making the expiry based on more of
JVM's runtime, which is very difficult to configure for any external user. The
goal is to make the expiry time an absolute epoch time, so that its easy to
configure for external users.
{noformat}
{
"hostName": ,
"port": ,
"adminState": "IN_MAINTENANCE",
"maintenanceExpireTimeInMS":
}
{noformat}
DatanodeInfo.java
{noformat}
public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS) {
return Time.monotonicNow() < maintenanceExpireTimeInMS;
}
{noformat}
was:
Currently it is possible to configure an expiry time in milliseconds for a
DataNode in maintenance state. As per the design, the expiry attribute is an
absolute time, beyond which NameNode starts to stop the ongoing maintenance
operation for that DataNode. Internally in the code, this expiry time is read
and checked against {{Time.monotonicNow()}} making the expiry based on more of
JVM's runtime, which is very difficult to configure for any external user. The
goal is to make the expiry time an absolute epoch time, so that its easy to
configure for external users.
{noformat}
{
"hostName": ,
"port": ,
"adminState": "IN_MAINTENANCE",
"maintenanceExpireTimeInMS":
}
{noformat}
DatanodeInfo.java
{noformat}
public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS) {
return Time.monotonicNow() < maintenanceExpireTimeInMS;
}
{noformat}
Component/s: (was: datanode)
(was: namenode)
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11296.01.patch
>
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11296:
--
Attachment: HDFS-11296.01.patch
Attached v01 patch to address the following
1. Made {{DatanideInfo#maintenanceNotExpired}} to check expiry time against
the current epoch time.
2. Modified TestMaintenanceState to configure maintenance state expiration with
epoch time
Manual verification:
-- Configured {{maintenanceExpireTimeInMS}} with current epoch time (taken
using shell command date +%s ) + 5 mins
-- Ran the command: hdfs dfsadmin -refreshNodes
-- Verified DN going to maintenance state
-- Verified time expiry exactly at 5th min and DNs put back to Normal state
[~eddyxu], [~mingma], can you please take a look at the patch ?
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11296.01.patch
>
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11296:
--
Summary: Maintenance state expiry should be an epoch time and not jvm
monotonic (was: Maintenance state expiry should be a epoch time and not jvm
monotonic)
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
