[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r494037751
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r494037751
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486549682
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
My final query then, why can't have one NVDIMM like one SSD as this also
movable and peristent..?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486548053 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: So, NVDIMM is peristent storage and RAM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486545111 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: Ok. Got it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486535639
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
ok, By design if you dn't want to move.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483125623
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Balancer and mover will not move the blocks based on the `isTransient` (
they call getMovableTypes(..))..The blocks which are in NVDIMM shouldn't moved
I feel(as this also exists in RAM and no need to move),but as per this change
it will move.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483125623
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Balancer and mover will not move the blocks based on the `isTransient` (
they call getMovableTypes(..))..the blocks which NVDIMM shouldn't moved I
feel,but as per this change it will move.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r482249623
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
As this will be co-exists. I was suggesting to change the method name to
something like "isNvdimm" which was revlant here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r482248330
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
Review comment:
As we are adding only "NVDIMM",So I expect one line change here but I
missed that you added one more param to enum.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r476644846
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
will RAM_DISK and NVDIMM co-exist..? if co-exist's, why can't we name
NVDIM itself..?
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
Review comment:
Let's add only this line
##
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
##
@@ -88,13 +92,14 @@ The effective storage policy can be retrieved by the
"[`storagepolicies -getStor
### Configuration
* **dfs.storage.policy.enabled** - for enabling/disabling the storage policy
feature. The default value is `true`.
-* **dfs.storage.default.policy** - Set the default storage policy with the
policy name. The default value is `HOT`. All possible policies are defined in
enum StoragePolicy, including `LAZY_PERSIST` `ALL_SSD` `ONE_SSD` `HOT` `WARM`
`COLD` and `PROVIDED`.
+* **dfs.storage.default.policy** - Set the default storage policy with the
policy name. The default value is `HOT`. All possible policies are defined in
enum StoragePolicy, including `LAZY_PERSIST` `ALL_SSD` `ONE_SSD` `HOT` `WARM`
`COLD` `ALL_NVDIMM` and `PROVIDED`.
Review comment:
can we maintain same order here also..?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
is this will not valid anymore.?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
