[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=491118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491118 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 25/Sep/20 13:43 Start Date: 25/Sep/20 13:43 Worklog Time Spent: 10m Work Description: brahmareddybattula merged pull request #2189: URL: https://github.com/apache/hadoop/pull/2189 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 491118) Time Spent: 12h 10m (was: 12h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 12h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=491061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491061
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 25/Sep/20 13:39
Start Date: 25/Sep/20 13:39
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r494037751
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 491061)
Time Spent: 12h (was: 11h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 12h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=491042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491042 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 25/Sep/20 13:37 Start Date: 25/Sep/20 13:37 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698212333 @huangtianhua and @YaYun-Wang thanks for review. @liuml07 thanks for review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 491042) Time Spent: 11h 50m (was: 11h 40m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 11h 50m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=490995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-490995 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 25/Sep/20 13:33 Start Date: 25/Sep/20 13:33 Worklog Time Spent: 10m Work Description: brahmareddybattula edited a comment on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698212333 @huangtianhua and @YaYun-Wang thanks for contribution. @liuml07 thanks for review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 490995) Time Spent: 11h 40m (was: 11.5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 11h 40m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=490068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-490068 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 24/Sep/20 08:57 Start Date: 24/Sep/20 08:57 Worklog Time Spent: 10m Work Description: brahmareddybattula merged pull request #2189: URL: https://github.com/apache/hadoop/pull/2189 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 490068) Time Spent: 11.5h (was: 11h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 11.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=490066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-490066 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 24/Sep/20 08:55 Start Date: 24/Sep/20 08:55 Worklog Time Spent: 10m Work Description: brahmareddybattula edited a comment on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698212333 @huangtianhua and @YaYun-Wang thanks for contribution. @liuml07 thanks for review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 490066) Time Spent: 11h 20m (was: 11h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 11h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=490065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-490065 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 24/Sep/20 08:55 Start Date: 24/Sep/20 08:55 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698212333 @huangtianhua and @YaYun-Wang thanks for review. @liuml07 thanks for review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 490065) Time Spent: 11h 10m (was: 11h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 11h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489976
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 24/Sep/20 04:53
Start Date: 24/Sep/20 04:53
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r494037751
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 489976)
Time Spent: 11h (was: 10h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 11h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489954 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 24/Sep/20 02:41 Start Date: 24/Sep/20 02:41 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698076876 Yeah; for this checkstyle in test we can keep it as-is for the sake of code consistency. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489954) Time Spent: 10h 50m (was: 10h 40m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 10h 50m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489946 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 24/Sep/20 02:17 Start Date: 24/Sep/20 02:17 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698070550 @liuml07 , thanks, and we have updated the code, seems there is a checkstyle about the storage enum name of 'nvdimm' should keep as it is to consistent with other storage types, what do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489946) Time Spent: 10h 40m (was: 10.5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489792 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 23/Sep/20 19:40 Start Date: 23/Sep/20 19:40 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-668645247 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489792) Time Spent: 10.5h (was: 10h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 10.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489791 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 23/Sep/20 19:39 Start Date: 23/Sep/20 19:39 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-668517822 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489791) Time Spent: 10h 20m (was: 10h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 10h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489417=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489417
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 23/Sep/20 06:32
Start Date: 23/Sep/20 06:32
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697162885
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 16 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 19s | | trunk passed |
| +1 :green_heart: | compile | 26m 40s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 22m 27s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 48s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 29s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 13s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 16s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 3s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 37s | | the patch passed |
| +1 :green_heart: | compile | 20m 54s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 20m 54s |
[/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt)
| root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 32 new + 131 unchanged -
32 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 20m 54s | | the patch passed |
| +1 :green_heart: | compile | 16m 48s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 16m 48s |
[/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 37 new + 126
unchanged - 37 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 16m 48s | | the patch passed |
| -0 :warning: | checkstyle | 2m 57s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 16 new + 723 unchanged - 6 fixed = 739 total (was
729) |
| +1 :green_heart: | mvnsite | 4m 3s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 14m 15s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 6s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 21s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 22s | | hadoop-common in the patch
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489354=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489354 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 23/Sep/20 04:56 Start Date: 23/Sep/20 04:56 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697078022 No more comments on the latest commit. Let's wait for the QA and run failing tests locally. I'll commit shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489354) Time Spent: 10h (was: 9h 50m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 10h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489320 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 23/Sep/20 04:53 Start Date: 23/Sep/20 04:53 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697068485 @liuml07 Sorry, we didn't catch you, what other changes we have to do? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489320) Time Spent: 9h 50m (was: 9h 40m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 9h 50m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489210=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489210
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 23/Sep/20 04:44
Start Date: 23/Sep/20 04:44
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r492866266
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes()
throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+.setStorageDirectory(
+new StorageDirectory(
+StorageLocation.parse("[NVDIMM]"+volDir.getPath(
+.setStorageID("storage-id")
+.setConf(conf)
+.build();
+assertEquals("", 3L, volume5.getReserved());
Review comment:
Usually we can, but following original code style here is bad. When it
fails, the original code gives up empty string. My code shows your expected
value and actual value so you can debug. Please change it.
We can also file a JIRA to update all such cases where `assertEquals` can be
improved.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 489210)
Time Spent: 9h 40m (was: 9.5h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=489075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489075
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 23/Sep/20 04:32
Start Date: 23/Sep/20 04:32
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r492598805
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes()
throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+.setStorageDirectory(
+new StorageDirectory(
+StorageLocation.parse("[NVDIMM]"+volDir.getPath(
+.setStorageID("storage-id")
+.setConf(conf)
+.build();
+assertEquals("", 3L, volume5.getReserved());
Review comment:
In order to be consistent with the original code, the `assertEquals()`
here has three parameters, such as, lines 196 and 204 of the original code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 489075)
Time Spent: 9.5h (was: 9h 20m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488934 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 23/Sep/20 04:21 Start Date: 23/Sep/20 04:21 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696742400 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 488934) Time Spent: 9h 20m (was: 9h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 9h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488728 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 23/Sep/20 02:06 Start Date: 23/Sep/20 02:06 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697078022 No more comments on the latest commit. Let's wait for the QA and run failing tests locally. I'll commit shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 488728) Time Spent: 9h 10m (was: 9h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 9h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488704 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 23/Sep/20 01:29 Start Date: 23/Sep/20 01:29 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697068485 @liuml07 Sorry, we didn't catch you, what other changes we have to do? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 488704) Time Spent: 9h (was: 8h 50m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 9h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488388
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 22/Sep/20 16:17
Start Date: 22/Sep/20 16:17
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r492866266
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes()
throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+.setStorageDirectory(
+new StorageDirectory(
+StorageLocation.parse("[NVDIMM]"+volDir.getPath(
+.setStorageID("storage-id")
+.setConf(conf)
+.build();
+assertEquals("", 3L, volume5.getReserved());
Review comment:
Usually we can, but following original code style here is bad. When it
fails, the original code gives up empty string. My code shows your expected
value and actual value so you can debug. Please change it.
We can also file a JIRA to update all such cases where `assertEquals` can be
improved.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 488388)
Time Spent: 8h 50m (was: 8h 40m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488331 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 22/Sep/20 15:15 Start Date: 22/Sep/20 15:15 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696788468 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | buf | 0m 1s | buf was not available. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 31m 35s | trunk passed | | +1 :green_heart: | compile | 24m 47s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 21m 59s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 25s | trunk passed | | +1 :green_heart: | mvnsite | 4m 35s | trunk passed | | +1 :green_heart: | shadedclient | 24m 36s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 44s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 9m 38s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 12s | the patch passed | | +1 :green_heart: | compile | 27m 37s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | cc | 27m 37s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 22 new + 141 unchanged - 22 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 27m 37s | the patch passed | | +1 :green_heart: | compile | 24m 0s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | cc | 24m 0s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 25 new + 138 unchanged - 25 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 24m 0s | the patch passed | | -0 :warning: | checkstyle | 3m 34s | root: The patch generated 16 new + 723 unchanged - 6 fixed = 739 total (was 729) | | +1 :green_heart: | mvnsite | 4m 59s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 19m 21s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 56s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 47s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 11m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 11m 45s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 39s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 107m 50s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. | | | | 356m 14s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHAAppend | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.TestDFSClientRetries | | | hadoop.hdfs.TestStripedFileAppend | | | hadoop.hdfs.TestReadStripedFileWithMissingBlocks | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | |
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488312 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 22/Sep/20 14:51 Start Date: 22/Sep/20 14:51 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696772998 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | buf | 0m 0s | buf was not available. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 29m 35s | trunk passed | | +1 :green_heart: | compile | 24m 47s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 19m 58s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 14s | trunk passed | | +1 :green_heart: | mvnsite | 3m 51s | trunk passed | | +1 :green_heart: | shadedclient | 23m 24s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 16s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 46s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 56s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 9m 1s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 24m 13s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | cc | 24m 13s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 27 new + 136 unchanged - 27 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 24m 13s | the patch passed | | +1 :green_heart: | compile | 20m 3s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | cc | 20m 3s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 30 new + 133 unchanged - 30 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 20m 3s | the patch passed | | -0 :warning: | checkstyle | 3m 12s | root: The patch generated 4 new + 725 unchanged - 4 fixed = 729 total (was 729) | | +1 :green_heart: | mvnsite | 3m 52s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 49s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 16s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 44s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 9m 53s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 10m 33s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 29s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 132m 31s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. | | | | 356m 13s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.security.TestRaceWhenRelogin | | | hadoop.hdfs.TestQuota | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.TestDecommissionWithBackoffMonitor | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.TestFileChecksum | | Subsystem | Report/Notes |
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488308 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 22/Sep/20 14:45 Start Date: 22/Sep/20 14:45 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696769048 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | buf | 0m 1s | buf was not available. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 30m 3s | trunk passed | | +1 :green_heart: | compile | 24m 5s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 20m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 12s | trunk passed | | +1 :green_heart: | mvnsite | 4m 14s | trunk passed | | +1 :green_heart: | shadedclient | 23m 31s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 9s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 54s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 8m 43s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 15s | the patch passed | | +1 :green_heart: | compile | 24m 1s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | cc | 24m 1s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 10 new + 153 unchanged - 10 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 24m 1s | the patch passed | | +1 :green_heart: | compile | 20m 48s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | cc | 20m 48s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 6 new + 157 unchanged - 6 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 20m 48s | the patch passed | | -0 :warning: | checkstyle | 3m 7s | root: The patch generated 4 new + 725 unchanged - 4 fixed = 729 total (was 729) | | +1 :green_heart: | mvnsite | 4m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 16m 31s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 9s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 39s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 9m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 10m 19s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 18s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 134m 19s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. | | | | 357m 54s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.TestFileChecksum | | Subsystem | Report/Notes |
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488300 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 22/Sep/20 14:28 Start Date: 22/Sep/20 14:28 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696758750 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | buf | 0m 1s | buf was not available. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 28m 33s | trunk passed | | +1 :green_heart: | compile | 20m 57s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 3s | trunk passed | | +1 :green_heart: | mvnsite | 3m 48s | trunk passed | | +1 :green_heart: | shadedclient | 23m 2s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 10s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 37s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 8m 7s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 20m 8s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | cc | 20m 8s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 30 new + 133 unchanged - 30 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 20m 8s | the patch passed | | +1 :green_heart: | compile | 17m 39s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | cc | 17m 39s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 5 new + 158 unchanged - 5 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 17m 39s | the patch passed | | -0 :warning: | checkstyle | 3m 2s | root: The patch generated 16 new + 723 unchanged - 6 fixed = 739 total (was 729) | | +1 :green_heart: | mvnsite | 3m 50s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 16m 22s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 11s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 35s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 10m 10s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 15s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 113m 52s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 57s | The patch does not generate ASF License warnings. | | | | 319m 19s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.qjournal.server.TestJournalNodeSync | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | |
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488297 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 22/Sep/20 14:26 Start Date: 22/Sep/20 14:26 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696757028 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | buf | 0m 1s | buf was not available. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 58s | trunk passed | | +1 :green_heart: | compile | 21m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 18m 16s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 2s | trunk passed | | +1 :green_heart: | mvnsite | 4m 5s | trunk passed | | +1 :green_heart: | shadedclient | 21m 51s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 22s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 48s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 41s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 8m 14s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 6s | the patch passed | | +1 :green_heart: | compile | 20m 55s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | cc | 20m 55s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 9 new + 154 unchanged - 9 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 20m 55s | the patch passed | | +1 :green_heart: | compile | 18m 15s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | cc | 18m 15s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 14 new + 149 unchanged - 14 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 18m 15s | the patch passed | | -0 :warning: | checkstyle | 2m 59s | root: The patch generated 16 new + 725 unchanged - 6 fixed = 741 total (was 731) | | +1 :green_heart: | mvnsite | 4m 9s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 20s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 29s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 49s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 10m 10s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 30s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 113m 41s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 11s | The patch does not generate ASF License warnings. | | | | 319m 59s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestByteBufferPread | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.TestFileAppend4 | | |
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488291 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 22/Sep/20 14:21 Start Date: 22/Sep/20 14:21 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696754216 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | buf | 0m 0s | buf was not available. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 27m 55s | trunk passed | | +1 :green_heart: | compile | 21m 10s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 18m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 0s | trunk passed | | +1 :green_heart: | mvnsite | 3m 55s | trunk passed | | +1 :green_heart: | shadedclient | 21m 39s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 22s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 50s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 38s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 8m 16s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 53s | the patch passed | | +1 :green_heart: | compile | 20m 54s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | cc | 20m 54s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 35 new + 128 unchanged - 35 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 20m 54s | the patch passed | | +1 :green_heart: | compile | 18m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | cc | 18m 50s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 34 new + 129 unchanged - 34 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 18m 50s | the patch passed | | -0 :warning: | checkstyle | 2m 58s | root: The patch generated 4 new + 727 unchanged - 4 fixed = 731 total (was 731) | | +1 :green_heart: | mvnsite | 4m 2s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 6s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 21s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 1s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 47s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 24s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 115m 39s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | | | | 321m 38s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.mover.TestStorageMover | | | hadoop.hdfs.TestDatanodeLayoutUpgrade | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestMultiThreadedHflush | | | hadoop.hdfs.TestSnapshotCommands | | |
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488282 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 22/Sep/20 14:02 Start Date: 22/Sep/20 14:02 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696742400 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | buf | 0m 1s | buf was not available. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 29m 17s | trunk passed | | +1 :green_heart: | compile | 24m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 20m 46s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 27s | trunk passed | | +1 :green_heart: | mvnsite | 4m 25s | trunk passed | | +1 :green_heart: | shadedclient | 25m 57s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 17s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 9m 9s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 7s | the patch passed | | +1 :green_heart: | compile | 22m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | cc | 22m 40s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 32 new + 131 unchanged - 32 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 22m 40s | the patch passed | | +1 :green_heart: | compile | 18m 59s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | cc | 18m 59s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 7 new + 156 unchanged - 7 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 18m 59s | the patch passed | | -0 :warning: | checkstyle | 2m 57s | root: The patch generated 4 new + 725 unchanged - 4 fixed = 729 total (was 729) | | +1 :green_heart: | mvnsite | 4m 4s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 14s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 49s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 30s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 18s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 97m 36s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | | | | 318m 7s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestSnapshotCommands | | | hadoop.hdfs.TestGetFileChecksum | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.server.datanode.TestBPOfferService | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksum |
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=488112=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488112
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 22/Sep/20 09:32
Start Date: 22/Sep/20 09:32
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r492598805
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes()
throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+.setStorageDirectory(
+new StorageDirectory(
+StorageLocation.parse("[NVDIMM]"+volDir.getPath(
+.setStorageID("storage-id")
+.setConf(conf)
+.build();
+assertEquals("", 3L, volume5.getReserved());
Review comment:
In order to be consistent with the original code, the `assertEquals()`
here has three parameters, such as, lines 196 and 204 of the original code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 488112)
Time Spent: 7.5h (was: 7h 20m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=487562=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487562
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 22/Sep/20 03:16
Start Date: 22/Sep/20 03:16
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491857797
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map storageTypeStats =
(Map)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
-assertEquals(3l, storageTypeStats.get("nodesInService"));
+assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
I have not used Java 7 for a while, but I remember vaguely this is
actually supported?
https://docs.oracle.com/javase/specs/jls/se7/html/jls-14.html
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map storageTypeStats =
(Map)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
-assertEquals(3l, storageTypeStats.get("nodesInService"));
+assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
Hadoop releases before 2.10 are all end of life (EoL). Hadoop 2.10 is
the only version using Java 7. We do not need any support, compile or runtime,
for Java versions before Java 7.
Hadoop 3.x are all using Java 8+. We do not need any Java 7 support in
Hadoop 3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487562)
Time Spent: 7h 20m (was: 7h 10m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=487465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487465
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 22/Sep/20 03:04
Start Date: 22/Sep/20 03:04
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491852471
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map storageTypeStats =
(Map)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
-assertEquals(3l, storageTypeStats.get("nodesInService"));
+assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
`storageType` is a parameter of "java.lang.String" , and `switch()`
does not support "java.lang.String" before java 1.7. So, will `if-else ` be
more appropriate here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 487465)
Time Spent: 7h 10m (was: 7h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=486810=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486810
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 08:11
Start Date: 21/Sep/20 08:11
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491859739
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map storageTypeStats =
(Map)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
-assertEquals(3l, storageTypeStats.get("nodesInService"));
+assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
Hadoop releases before 2.10 are all end of life (EoL). Hadoop 2.10 is
the only version using Java 7. We do not need any support, compile or runtime,
for Java versions before Java 7.
Hadoop 3.x are all using Java 8+. We do not need any Java 7 support in
Hadoop 3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 486810)
Time Spent: 7h (was: 6h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=486809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486809
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 08:07
Start Date: 21/Sep/20 08:07
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491857797
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map storageTypeStats =
(Map)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
-assertEquals(3l, storageTypeStats.get("nodesInService"));
+assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
I have not used Java 7 for a while, but I remember vaguely this is
actually supported?
https://docs.oracle.com/javase/specs/jls/se7/html/jls-14.html
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 486809)
Time Spent: 6h 50m (was: 6h 40m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=486806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486806
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 21/Sep/20 07:56
Start Date: 21/Sep/20 07:56
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491852471
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map storageTypeStats =
(Map)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
-assertEquals(3l, storageTypeStats.get("nodesInService"));
+assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
`storageType` is a parameter of "java.lang.String" , and `switch()`
does not support "java.lang.String" before java 1.7. So, will `if-else ` be
more appropriate here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 486806)
Time Spent: 6h 40m (was: 6.5h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=486622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486622
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 20/Sep/20 07:40
Start Date: 20/Sep/20 07:40
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r487387683
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##
@@ -2298,9 +2298,9 @@ private void cacheBlock(String bpid, long blockId) {
": volume was not an instance of FsVolumeImpl.");
return;
}
-if (volume.isTransientStorage()) {
+if (volume.isRAMStorage()) {
LOG.warn("Caching not supported on block with id " + blockId +
- " since the volume is backed by RAM.");
+ " since the volume is backed by RAM_DISK or NVDIMM.");
Review comment:
nit: we can make it clear what the volume is.
```
LOG.warn("Caching not supported on block with id {} since the volume "
+ "is backed by {} which is RAM.", blockId, volume);
```
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/net/TestDFSNetworkTopology.java
##
@@ -182,15 +212,16 @@ public void testGetStorageTypeInfo() throws Exception {
*/
@Test
public void testAddAndRemoveTopology() throws Exception {
-String[] newRack = {"/l1/d1/r1", "/l1/d1/r3", "/l1/d3/r3", "/l1/d3/r3"};
-String[] newHost = {"nhost1", "nhost2", "nhost3", "nhost4"};
+String[] newRack = {"/l1/d1/r1", "/l1/d1/r3", "/l1/d3/r3", "/l1/d3/r3",
+"/l1/d3/r4"};
+String[] newHost = {"nhost1", "nhost2", "nhost3", "nhost4", "nhost5"};
String[] newips = {"30.30.30.30", "31.31.31.31", "32.32.32.32",
-"33.33.33.33"};
+"33.33.33.33", "33.33.33.34"};
Review comment:
nit: why not use `34.34.34.34`?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map storageTypeStats =
(Map)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
-assertEquals(3l, storageTypeStats.get("nodesInService"));
+assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
As this `if-else if-else if-else` getting longer, let's use switch case?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
##
@@ -43,14 +43,15 @@ public void testDataDirParsing() throws Throwable {
File dir5 = new File("/dir5");
File dir6 = new File("/dir6");
+File dir7 = new File("/dir7");
// Verify that a valid string is correctly parsed, and that storage
// type is not case-sensitive and we are able to handle white-space between
// storage type and URI.
String locations1 = "[disk]/dir0,[DISK]/dir1,[sSd]/dir2,[disK]/dir3," +
-"[ram_disk]/dir4,[disk]/dir5, [disk] /dir6, [disk] ";
+"[ram_disk]/dir4,[disk]/dir5, [disk] /dir6, [disk], [nvdimm]/dir7";
Review comment:
The invalid 8th URI has ending space deliberately for testing. Let's
keep it, aka
```
"[ram_disk]/dir4,[disk]/dir5, [disk] /dir6, [disk] , [nvdimm]/dir7";
```
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/net/TestDFSNetworkTopology.java
##
@@ -120,10 +125,11 @@ public void testGetStorageTypeInfo() throws Exception {
HashMap> d2info =
d2.getChildrenStorageInfo();
assertEquals(1, d2info.keySet().size());
-assertTrue(d2info.get("r3").size() == 3);
+assertTrue(d2info.get("r3").size() == 4);
Review comment:
nit: replace with `assertEquals(4, d2info.get("r3").size())`;
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes()
throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+.setStorageDirectory(
+new StorageDirectory(
+StorageLocation.parse("[NVDIMM]"+volDir.getPath(
+.setStorageID("storage-id")
+.setConf(conf)
+.build();
+assertEquals("", 3L, volume5.getReserved());
Review comment:
nit: Let's use better assertion statement
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=485491=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-485491 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 17/Sep/20 01:41 Start Date: 17/Sep/20 01:41 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-693756560 Sorry, I have been busy in other stuff. The recent review I have no other comments except a few trivial ones. I'll post that and commit this week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 485491) Time Spent: 6h 20m (was: 6h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 6h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=485486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-485486 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 17/Sep/20 01:31 Start Date: 17/Sep/20 01:31 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-693753888 @liuml07 and @brahmareddybattula Sorry to disturb you again, I wonder if there is any update about this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 485486) Time Spent: 6h 10m (was: 6h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 6h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=483723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483723 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 14/Sep/20 01:25 Start Date: 14/Sep/20 01:25 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-691761278 @brahmareddybattula Hi brahma, would you review it again, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 483723) Time Spent: 6h (was: 5h 50m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 6h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=483721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483721 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 14/Sep/20 01:24 Start Date: 14/Sep/20 01:24 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-691760924 @liuml07 That sounds good, thank you very much:) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 483721) Time Spent: 5h 50m (was: 5h 40m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=483268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483268 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 12/Sep/20 21:39 Start Date: 12/Sep/20 21:39 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690912424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 483268) Time Spent: 5h 40m (was: 5.5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=483181=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483181 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 12/Sep/20 21:29 Start Date: 12/Sep/20 21:29 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690839476 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 483181) Time Spent: 5.5h (was: 5h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 5.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=482864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482864 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 12/Sep/20 20:28 Start Date: 12/Sep/20 20:28 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690829884 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 482864) Time Spent: 5h 20m (was: 5h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 5h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=482773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482773 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 12/Sep/20 20:20 Start Date: 12/Sep/20 20:20 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690839476 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 482773) Time Spent: 5h 10m (was: 5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 5h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=482574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482574
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 20:03
Start Date: 12/Sep/20 20:03
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716471
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
> My final query then, why can't have one NVDIMM like one SSD as this
also movable and peristent..?
Considering NVDIMM is faster, so NVDIMM does not use `FsDatasetCache()`
which SSD needs in the design.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
> So, NVDIMM is peristent storage and RAM.
yes, that’s right.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 482574)
Time Spent: 5h (was: 4h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481939 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 07:42 Start Date: 11/Sep/20 07:42 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690932868 No this is not a blocker @huangtianhua I will review (again) and commit hopefully this week if there is no other comments. Thanks, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481939) Time Spent: 4h 50m (was: 4h 40m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 50m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481925 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 06:54 Start Date: 11/Sep/20 06:54 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690912424 @liuml07 We have checked for several times, the failure tests are not related with this patch, and I found almost all of the HDFS-* patches are not pass the hadoop-hdfs unit tests, like https://github.com/apache/hadoop/pull/2275 https://github.com/apache/hadoop/pull/2205 So maybe the failure tests are not blocker? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481925) Time Spent: 4h 40m (was: 4.5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481857 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 02:43 Start Date: 11/Sep/20 02:43 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690839476 Will check again later this week. Ideally we can get a clean QA. Could you check the test failures and make sure they are not related? Thanks, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481857) Time Spent: 4.5h (was: 4h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481846 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 02:08 Start Date: 11/Sep/20 02:08 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690829884 @liuml07 and @brahmareddybattula So if all of the codes are ok, would you please to approve? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481846) Time Spent: 4h 20m (was: 4h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481837 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 01:14 Start Date: 11/Sep/20 01:14 Worklog Time Spent: 10m Work Description: YaYun-Wang commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716810 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: > So, NVDIMM is peristent storage and RAM. yes, that’s right. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481837) Time Spent: 4h 10m (was: 4h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481836
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 11/Sep/20 01:12
Start Date: 11/Sep/20 01:12
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716471
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
> My final query then, why can't have one NVDIMM like one SSD as this
also movable and peristent..?
Considering NVDIMM is faster, so NVDIMM does not use `FsDatasetCache()`
which SSD needs in the design.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481836)
Time Spent: 4h (was: 3h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481641
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 18:27
Start Date: 10/Sep/20 18:27
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486549682
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
My final query then, why can't have one NVDIMM like one SSD as this also
movable and peristent..?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481641)
Time Spent: 3h 50m (was: 3h 40m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481640 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 10/Sep/20 18:25 Start Date: 10/Sep/20 18:25 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486548053 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: So, NVDIMM is peristent storage and RAM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481640) Time Spent: 3h 40m (was: 3.5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481637 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 10/Sep/20 18:21 Start Date: 10/Sep/20 18:21 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486545111 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: Ok. Got it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481637) Time Spent: 3.5h (was: 3h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 3.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481626
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 18:07
Start Date: 10/Sep/20 18:07
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486535639
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
ok, By design if you dn't want to move.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481626)
Time Spent: 3h 20m (was: 3h 10m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=480696=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480696
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 09/Sep/20 09:14
Start Date: 09/Sep/20 09:14
Worklog Time Spent: 10m
Work Description: huangtianhua commented on a change in pull request
#2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r485463441
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
@brahmareddybattula would you please have a look for this? Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 480696)
Time Spent: 3h 10m (was: 3h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=479211=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479211
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 18:16
Start Date: 04/Sep/20 18:16
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483780758
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Thanks for clarification, @YaYun-Wang I think now we both are on the
same page. @brahmareddybattula Does this make sense to you? Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479211)
Time Spent: 3h (was: 2h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=479050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479050 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 04/Sep/20 10:01 Start Date: 04/Sep/20 10:01 Worklog Time Spent: 10m Work Description: YaYun-Wang commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r477304478 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: The "isTransientStorage" method is still available. In the original code,` isTransient()` and `isTransientStorage` methods are used to determine whether to support FsDatasetCache, Persistent, Quota, and Movable. FsDatasetCache will be used When the storage type is persistent. NVDIMM is RAM to some extent, which is fast. However, NVDIMM is a persistent storage type. Then, `isTransient()` and `isTransientStorage() ` used to determine whether to support FsDatasetCache can't meet the requirements. Therefore, we add `isRAM()` and `isRAMStorage() ` methods to decide whether cache is supported or not. And the other functions, such as, Persistent, Quota, and Movable judged by`isTransient()` and `isTransientStorage() ` methods. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 479050) Time Spent: 2h 50m (was: 2h 40m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 2h 50m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=479047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479047
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 09:55
Start Date: 04/Sep/20 09:55
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483515521
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Sorry, I made a big mistake in the above reply. Our design idea is:
NVDIMM supports mover and balancer, and `isTransient() ` applied in the case.`
isRAM()` is only used to “FsDatasetCache” judgment. So the current code is
reasonable, i think it‘s not necessary to modify the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479047)
Time Spent: 2h 40m (was: 2.5h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=479046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479046
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 09:54
Start Date: 04/Sep/20 09:54
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483515521
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Sorry, I made a big mistake in the above reply. Our design idea is:
NVDIMM supports mover and balancer, and `isTransient() ` applied in the case.
isRAM() is only used to “FsDatasetCache” judgment. So the current code is
reasonable, i think it‘s not necessary to modify the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479046)
Time Spent: 2.5h (was: 2h 20m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478972
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 07:18
Start Date: 04/Sep/20 07:18
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
NVDIMM is special RAM, the data above can be stored persistently. It can
be regarded as a general hardware device. We don't have to consider what
storage type it is, balancer and mover can be applied on NVDIMM, therefore, I
think it is better to use isRAM to determine whether to use mover . In
addition, neither RAM nor nvdimm need FsDatasetCache, and isTransient() used to
determine whether FsDatasetCache is needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478972)
Time Spent: 2h 20m (was: 2h 10m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478956
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 06:31
Start Date: 04/Sep/20 06:31
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483414755
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
> neither RAM nor nvdimm need FsDatasetCache, and isTransient() used to
determine whether FsDatasetCache is needed
I think we have agreed on this.
> I think it is better to use isRAM to determine whether to use mover
So just to be clear, if we use `isRAM()` to determine, both RAM_DISK and
NVDIMM will return true and thus will simply disable Balancer and Mover. I was
proposing that only when we do not want to move those NVDIMM data replicas
around (disk/node) ever. Is that the design here? I'm fine with that but I
think this contradicts with the statement that _"It can be regarded as a
general hardware device. "_
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478956)
Time Spent: 2h 10m (was: 2h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478955
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 06:24
Start Date: 04/Sep/20 06:24
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483194125
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Oh, I was thinking that allowing Balancer to move the NVDIMM data is by
design since they are not volatile. But if that is *not* case, then we can
update Balancer code by replacing `isTransient()` call with `isRAM()` call. Not
sure if this makes more sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478955)
Time Spent: 2h (was: 1h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478954
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 06:20
Start Date: 04/Sep/20 06:20
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
NVDIMM is special RAM, the data above can be stored persistently. It can
be regarded as a general hardware device. We don't have to consider what
storage type it is. Therefore, I think it is better to use isRAM to determine
whether to use mover . In addition, neither RAM nor nvdimm need FsDatasetCache,
and isTransient() used to determine whether FsDatasetCache is needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478954)
Time Spent: 1h 50m (was: 1h 40m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478953
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 06:19
Start Date: 04/Sep/20 06:19
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
NVDIMM is RAM, however, the data above can be stored persistently. It
can be regarded as a general hardware device. We don't have to consider what
storage type it is. Therefore, I think it is better to use isRAM to determine
whether to use mover . In addition, neither RAM nor nvdimm need FsDatasetCache,
and isTransient() used to determine whether FsDatasetCache is needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478953)
Time Spent: 1h 40m (was: 1.5h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478768
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 03/Sep/20 19:03
Start Date: 03/Sep/20 19:03
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483194125
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Oh, I was thinking that allowing Balancer to move the NVDIMM data is by
design since they are not volatile. But if that is case, then we can update
Balancer code by replacing `isTransient()` call with `isRAM()` call. Not sure
if this makes more sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478768)
Time Spent: 1.5h (was: 1h 20m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478697=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478697
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 03/Sep/20 17:00
Start Date: 03/Sep/20 17:00
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483125623
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Balancer and mover will not move the blocks based on the `isTransient` (
they call getMovableTypes(..))..The blocks which are in NVDIMM shouldn't moved
I feel(as this also exists in RAM and no need to move),but as per this change
it will move.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478697)
Time Spent: 1h 20m (was: 1h 10m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478692
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 03/Sep/20 16:59
Start Date: 03/Sep/20 16:59
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483125623
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Balancer and mover will not move the blocks based on the `isTransient` (
they call getMovableTypes(..))..the blocks which NVDIMM shouldn't moved I
feel,but as per this change it will move.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478692)
Time Spent: 1h 10m (was: 1h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478079
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 02/Sep/20 19:30
Start Date: 02/Sep/20 19:30
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r482341228
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
No. Both RAM_DISK and NVDIMM will return true here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478079)
Time Spent: 1h (was: 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=477996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-477996
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 02/Sep/20 17:38
Start Date: 02/Sep/20 17:38
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r482249623
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
As this will be co-exists. I was suggesting to change the method name to
something like "isNvdimm" which was revlant here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 477996)
Time Spent: 50m (was: 40m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=477995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-477995
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 02/Sep/20 17:36
Start Date: 02/Sep/20 17:36
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r482248330
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
Review comment:
As we are adding only "NVDIMM",So I expect one line change here but I
missed that you added one more param to enum.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 477995)
Time Spent: 40m (was: 0.5h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=477681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-477681 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 02/Sep/20 07:21 Start Date: 02/Sep/20 07:21 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-685405577 @liuml07 and @brahmareddybattula please help to review the latest code again? thanks very much. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 477681) Time Spent: 0.5h (was: 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=475152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-475152 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 27/Aug/20 06:19 Start Date: 27/Aug/20 06:19 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-681615978 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | buf | 0m 0s | buf was not available. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 25m 44s | trunk passed | | +1 :green_heart: | compile | 19m 9s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 16m 45s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 0s | trunk passed | | +1 :green_heart: | mvnsite | 4m 3s | trunk passed | | +1 :green_heart: | shadedclient | 21m 36s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 32s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 4s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 35s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 58s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 18m 39s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | cc | 18m 38s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 12 new + 151 unchanged - 12 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 18m 38s | the patch passed | | +1 :green_heart: | compile | 17m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | cc | 17m 50s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 10 new + 153 unchanged - 10 fixed = 163 total (was 163) | | +1 :green_heart: | javac | 17m 50s | the patch passed | | -0 :warning: | checkstyle | 2m 57s | root: The patch generated 3 new + 725 unchanged - 4 fixed = 728 total (was 729) | | +1 :green_heart: | mvnsite | 4m 4s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 14s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 15s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 54s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 56s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 18s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 101m 28s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 57s | The patch does not generate ASF License warnings. | | | | 297m 7s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.balancer.TestBalancer | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=475135=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-475135 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 27/Aug/20 04:11 Start Date: 27/Aug/20 04:11 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-681369503 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | buf | 0m 0s | buf was not available. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 1m 8s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 0m 22s | root in trunk failed. | | -1 :x: | compile | 0m 22s | root in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 22s | root in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in root | | -1 :x: | mvnsite | 1m 48s | hadoop-common in trunk failed. | | -1 :x: | mvnsite | 0m 22s | hadoop-hdfs in trunk failed. | | -1 :x: | mvnsite | 0m 22s | hadoop-hdfs-client in trunk failed. | | +1 :green_heart: | shadedclient | 3m 18s | branch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 22s | hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 23s | hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 22s | hadoop-hdfs-client in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 23s | hadoop-common in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javadoc | 0m 21s | hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javadoc | 0m 21s | hadoop-hdfs-client in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | +0 :ok: | spotbugs | 6m 37s | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 :x: | findbugs | 0m 21s | hadoop-common in trunk failed. | | -1 :x: | findbugs | 0m 23s | hadoop-hdfs in trunk failed. | | -1 :x: | findbugs | 0m 22s | hadoop-hdfs-client in trunk failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 0m 22s | hadoop-common in the patch failed. | | -1 :x: | mvninstall | 0m 22s | hadoop-hdfs in the patch failed. | | -1 :x: | mvninstall | 0m 22s | hadoop-hdfs-client in the patch failed. | | -1 :x: | compile | 0m 21s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | cc | 0m 21s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 0m 21s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 22s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | cc | 0m 22s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 22s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in root | | -1 :x: | mvnsite | 0m 22s | hadoop-common in the patch failed. | | -1 :x: | mvnsite | 0m 22s | hadoop-hdfs in the patch failed. | | -1 :x: | mvnsite | 0m 22s | hadoop-hdfs-client in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 0m 22s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 33s | hadoop-common in the patch failed with JDK
