[jira] [Commented] (HDFS-15438) Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy
[
https://issues.apache.org/jira/browse/HDFS-15438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190978#comment-17190978
]
Ayush Saxena commented on HDFS-15438:
-
+1
> Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy
> --
>
> Key: HDFS-15438
> URL: https://issues.apache.org/jira/browse/HDFS-15438
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Reporter: AMC-team
>Priority: Major
> Attachments: HDFS-15438.000.patch, HDFS-15438.001.patch, Screen Shot
> 2020-09-03 at 4.33.53 PM.png
>
>
> In HDFS disk balancer, the config parameter
> "dfs.disk.balancer.max.disk.errors" is to control the value of maximum number
> of errors we can ignore for a specific move between two disks before it is
> abandoned.
> The parameter can accept value that >= 0. And setting the value to 0 should
> mean no error tolerance. However, setting the value to 0 will simply don't do
> the block copy even there is no disk error occur because the while loop
> condition *item.getErrorCount() < getMaxError(item)* will not satisfied.
> {code:java}
> // Gets the next block that we can copy
> private ExtendedBlock getBlockToCopy(FsVolumeSpi.BlockIterator iter,
> DiskBalancerWorkItem item) {
> while (!iter.atEnd() && item.getErrorCount() < getMaxError(item)) {
> try {
> ... //get the block
> } catch (IOException e) {
> item.incErrorCount();
> }
>if (item.getErrorCount() >= getMaxError(item)) {
> item.setErrMsg("Error count exceeded.");
> LOG.info("Maximum error count exceeded. Error count: {} Max error:{}
> ",
> item.getErrorCount(), item.getMaxDiskErrors());
> }
> {code}
> *How to fix*
> Change the while loop condition to support value 0.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190948#comment-17190948 ] Ayush Saxena edited comment on HDFS-15554 at 9/5/20, 4:45 AM: -- Thanx [~fengnanli] for sharing the use case. I just saw there are bunch of test failures as well. so, that even gives me a feel, and TBH I don't think the current workflow is anyway 'wrong', It is just that it doesn't satisfy your use case. In general, I am not in favor of going ahead with such an incompatible way. The best I can suggest is, back this with a configuration, turn that off by default, and you guys can enable it, as per need, This may even save you from fixing the tests as well. was (Author: ayushtkn): Thanx [~fengnanli] for sharing the use case. I just saw there are bunch of test failures as well. so, that even gives me a feel, and TBH I don't think the current workflow is anyway 'wrong', It is just that it doesn't satisfy your use case. I am in general no favour of going ahead with such an incompatible way. The best I can suggest is, back this with a configuration, turn that off by default, and you guys can enable it, as per need, This may even prevent you from fixing the tests as well. > RBF: force router check file existence in destinations before adding/updating > mount points > -- > > Key: HDFS-15554 > URL: https://issues.apache.org/jira/browse/HDFS-15554 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Adding/Updating mount points right now is only a router action without > validation in the downstream namenodes for the destination files/directories. > In practice we have set up the dangling mount points and when clients call > listStatus they would get the file returned, but then if they try to access > the file FileNotFoundException would be thrown out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[
https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190977#comment-17190977
]
Ayush Saxena commented on HDFS-14694:
-
Thanx [~leosun08] for the update, Couple of comments :
{code:java}
+ if (recoverOnCloseException) {
+try {
+ dfsClient.endFileLease(fileId);
+ dfsClient.recoverLease(src);
+ leaseRecovered = true;
+} catch (Exception e) {
+ // Ignore exception rendered by recoverLease. Throw original
+ // exception
+}
+ }
{code}
* This can be refactored into a single method and can be used at both places,
You can extend a javadoc for the new method, explaining what it does.
{code:java}
- private void completeFile() throws IOException {
+ protected void completeFile() throws IOException {
{code}
* Add {{@VisibleForTesting}} and I think default access, should be enough?
{code:java}
+ public boolean isLeaseRecovered() {
{code}
* Similarly here.
{code:java}
+String RECOVER_ON_CLOSE_EXCEPTION_KEY =
+PREFIX + "recover.on.close.exception";
{code}
* Can we change to {{recover.lease.on.close.exception}}?
{code:java}
GenericTestUtils.waitFor(() -> {
boolean closed;
try {
closed = cluster.getFileSystem().isFileClosed(
new Path("/testExceptionInCloseWithoutRecoverLease"));
} catch (IOException e) {
return false;
}
return closed;
}, 1000, 5000);
}
{code}
* The wait can be refactored to a method, and you can use in both the tests.
Nit:
{code:java}
+ Assert.assertTrue(cluster.getFileSystem()
+Assert.assertFalse(cluster.getFileSystem().isFileClosed(
{code}
* No need for {{Assert.}} {{assertTrue}} and {{assertFalse}} are already
imported.
> Call recoverLease on DFSOutputStream close exception
>
>
> Key: HDFS-14694
> URL: https://issues.apache.org/jira/browse/HDFS-14694
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Chen Zhang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch,
> HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch,
> HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch,
> HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch
>
>
> HDFS uses file-lease to manage opened files, when a file is not closed
> normally, NN will recover lease automatically after hard limit exceeded. But
> for a long running service(e.g. HBase), the hdfs-client will never die and NN
> don't have any chances to recover the file.
> Usually client program needs to handle exceptions by themself to avoid this
> condition(e.g. HBase automatically call recover lease for files that not
> closed normally), but in our experience, most services (in our company) don't
> process this condition properly, which will cause lots of files in abnormal
> status or even data loss.
> This Jira propose to add a feature that call recoverLease operation
> automatically when DFSOutputSteam close encounters exception. It should be
> disabled by default, but when somebody builds a long-running service based on
> HDFS, they can enable this option.
> We've add this feature to our internal Hadoop distribution for more than 3
> years, it's quite useful according our experience.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.
[
https://issues.apache.org/jira/browse/HDFS-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190958#comment-17190958
]
Hadoop QA commented on HDFS-15422:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.10 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
59s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} branch-2.10 passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} branch-2.10 passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} branch-2.10 passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} branch-2.10 passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
39s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} branch-2.10 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| |
[jira] [Commented] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190948#comment-17190948 ] Ayush Saxena commented on HDFS-15554: - Thanx [~fengnanli] for sharing the use case. I just saw there are bunch of test failures as well. so, that even gives me a feel, and TBH I don't think the current workflow is anyway 'wrong', It is just that it doesn't satisfy your use case. I am in general no favour of going ahead with such an incompatible way. The best I can suggest is, back this with a configuration, turn that off by default, and you guys can enable it, as per need, This may even prevent you from fixing the tests as well. > RBF: force router check file existence in destinations before adding/updating > mount points > -- > > Key: HDFS-15554 > URL: https://issues.apache.org/jira/browse/HDFS-15554 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Adding/Updating mount points right now is only a router action without > validation in the downstream namenodes for the destination files/directories. > In practice we have set up the dangling mount points and when clients call > listStatus they would get the file returned, but then if they try to access > the file FileNotFoundException would be thrown out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.
[ https://issues.apache.org/jira/browse/HDFS-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-15422: Status: Patch Available (was: Open) > Reported IBR is partially replaced with stored info when queuing. > - > > Key: HDFS-15422 > URL: https://issues.apache.org/jira/browse/HDFS-15422 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Kihwal Lee >Priority: Critical > Attachments: HDFS-15422-branch-2.10.001.patch > > > When queueing an IBR (incremental block report) on a standby namenode, some > of the reported information is being replaced with the existing stored > information. This can lead to false block corruption. > We had a namenode, after transitioning to active, started reporting missing > blocks with "SIZE_MISMATCH" as corrupt reason. These were blocks that were > appended and the sizes were actually correct on the datanodes. Upon further > investigation, it was determined that the namenode was queueing IBRs with > altered information. > Although it sounds bad, I am not making it blocker -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.
[ https://issues.apache.org/jira/browse/HDFS-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190943#comment-17190943 ] Masatake Iwasaki commented on HDFS-15422: - I'm looking into the code while attaching the diff to kick unit tests. > Reported IBR is partially replaced with stored info when queuing. > - > > Key: HDFS-15422 > URL: https://issues.apache.org/jira/browse/HDFS-15422 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Kihwal Lee >Priority: Critical > Attachments: HDFS-15422-branch-2.10.001.patch > > > When queueing an IBR (incremental block report) on a standby namenode, some > of the reported information is being replaced with the existing stored > information. This can lead to false block corruption. > We had a namenode, after transitioning to active, started reporting missing > blocks with "SIZE_MISMATCH" as corrupt reason. These were blocks that were > appended and the sizes were actually correct on the datanodes. Upon further > investigation, it was determined that the namenode was queueing IBRs with > altered information. > Although it sounds bad, I am not making it blocker -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.
[ https://issues.apache.org/jira/browse/HDFS-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-15422: Attachment: HDFS-15422-branch-2.10.001.patch > Reported IBR is partially replaced with stored info when queuing. > - > > Key: HDFS-15422 > URL: https://issues.apache.org/jira/browse/HDFS-15422 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Kihwal Lee >Priority: Critical > Attachments: HDFS-15422-branch-2.10.001.patch > > > When queueing an IBR (incremental block report) on a standby namenode, some > of the reported information is being replaced with the existing stored > information. This can lead to false block corruption. > We had a namenode, after transitioning to active, started reporting missing > blocks with "SIZE_MISMATCH" as corrupt reason. These were blocks that were > appended and the sizes were actually correct on the datanodes. Upon further > investigation, it was determined that the namenode was queueing IBRs with > altered information. > Although it sounds bad, I am not making it blocker -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190917#comment-17190917 ] Fengnan Li commented on HDFS-15554: --- Thanks for the pointer [~ayushtkn]! I think both tickets are useful. Besides the shared consistency goal, HDFS-13506 is saving one command. However there are many cases where the downstream dirs should not be created. In our use case we are dealing with daily ingestion data where for some days there is no data and creating an empty dir will cause confusion for others. On the other hand, if we just create a dir per mount point we would end up with a lot of unused dirs and inodes, which will have a negative impact on NN performance since we are talking about a huge number of tables over a couple years of daily data. This change will have admins either create NN dir first if needed or not create the dangling(wrong) mount points. This may be incompatible with the current workflow but I don't think the current workflow is fully correct since we already went through some outages due to this. I am working on the tests as discussed with [~elgoiri]. It is non trivial I have to say. > RBF: force router check file existence in destinations before adding/updating > mount points > -- > > Key: HDFS-15554 > URL: https://issues.apache.org/jira/browse/HDFS-15554 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Adding/Updating mount points right now is only a router action without > validation in the downstream namenodes for the destination files/directories. > In practice we have set up the dangling mount points and when clients call > listStatus they would get the file returned, but then if they try to access > the file FileNotFoundException would be thrown out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190883#comment-17190883 ] Ayush Saxena commented on HDFS-15554: - Thanx [~fengnanli] for the report, I just had a quick look on the description(couldn't check the code), and seems like you want to avoid adding a mount entry if the destination directory doesn't exist, so as to avoid FNF's later, right? Will HDFS-13506 help? It aims to create the directory if it doesn't exist in the actual NS. Blocking addition of mount entries if destination directory isn't there would be an incompatible change as well. An admin might add a mount entry first and then create the destination directories, that is something. I am pretty sure, if we throw an exception a lot of test scripts are gonna break. > RBF: force router check file existence in destinations before adding/updating > mount points > -- > > Key: HDFS-15554 > URL: https://issues.apache.org/jira/browse/HDFS-15554 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Adding/Updating mount points right now is only a router action without > validation in the downstream namenodes for the destination files/directories. > In practice we have set up the dangling mount points and when clients call > listStatus they would get the file returned, but then if they try to access > the file FileNotFoundException would be thrown out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=479211=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479211
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 18:16
Start Date: 04/Sep/20 18:16
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483780758
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Thanks for clarification, @YaYun-Wang I think now we both are on the
same page. @brahmareddybattula Does this make sense to you? Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479211)
Time Spent: 3h (was: 2h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[
https://issues.apache.org/jira/browse/HDFS-15554?focusedWorklogId=479201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479201
]
ASF GitHub Bot logged work on HDFS-15554:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 17:45
Start Date: 04/Sep/20 17:45
Worklog Time Spent: 10m
Work Description: fengnanli commented on a change in pull request #2266:
URL: https://github.com/apache/hadoop/pull/2266#discussion_r483767772
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java
##
@@ -562,11 +595,35 @@ public GetDestinationResponse getDestination(
LOG.error("Cannot get location for {}: {}",
src, ioe.getMessage());
}
-if (nsIds.isEmpty() && !locations.isEmpty()) {
- String nsId = locations.get(0).getNameserviceId();
- nsIds.add(nsId);
+return nsIds;
+ }
+
+ /**
+ * Verify the file exists in destination nameservices to avoid dangling
+ * mount points.
+ *
+ * @param entry the new mount points added, could be from add or update.
+ * @return destination nameservices where the file doesn't exist.
+ * @throws IOException
+ */
+ private List verifyFileInDestinations(MountTable entry)
Review comment:
If that's the case, I will try to fix all in one batch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479201)
Time Spent: 2h (was: 1h 50m)
> RBF: force router check file existence in destinations before adding/updating
> mount points
> --
>
> Key: HDFS-15554
> URL: https://issues.apache.org/jira/browse/HDFS-15554
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Adding/Updating mount points right now is only a router action without
> validation in the downstream namenodes for the destination files/directories.
> In practice we have set up the dangling mount points and when clients call
> listStatus they would get the file returned, but then if they try to access
> the file FileNotFoundException would be thrown out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[
https://issues.apache.org/jira/browse/HDFS-15554?focusedWorklogId=479198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479198
]
ASF GitHub Bot logged work on HDFS-15554:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 17:29
Start Date: 04/Sep/20 17:29
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2266:
URL: https://github.com/apache/hadoop/pull/2266#discussion_r483760982
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java
##
@@ -562,11 +595,35 @@ public GetDestinationResponse getDestination(
LOG.error("Cannot get location for {}: {}",
src, ioe.getMessage());
}
-if (nsIds.isEmpty() && !locations.isEmpty()) {
- String nsId = locations.get(0).getNameserviceId();
- nsIds.add(nsId);
+return nsIds;
+ }
+
+ /**
+ * Verify the file exists in destination nameservices to avoid dangling
+ * mount points.
+ *
+ * @param entry the new mount points added, could be from add or update.
+ * @return destination nameservices where the file doesn't exist.
+ * @throws IOException
+ */
+ private List verifyFileInDestinations(MountTable entry)
Review comment:
Eventually we may want to make all the tests be correct, but for now I'm
fine setting up the config for just the new tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479198)
Time Spent: 1h 50m (was: 1h 40m)
> RBF: force router check file existence in destinations before adding/updating
> mount points
> --
>
> Key: HDFS-15554
> URL: https://issues.apache.org/jira/browse/HDFS-15554
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Adding/Updating mount points right now is only a router action without
> validation in the downstream namenodes for the destination files/directories.
> In practice we have set up the dangling mount points and when clients call
> listStatus they would get the file returned, but then if they try to access
> the file FileNotFoundException would be thrown out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
[ https://issues.apache.org/jira/browse/HDFS-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15558: --- Status: Patch Available (was: Open) > ViewDistributedFileSystem#recoverLease should call super.recoverLease when > there are no mounts configured > - > > Key: HDFS-15558 > URL: https://issues.apache.org/jira/browse/HDFS-15558 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13522) Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190767#comment-17190767
]
CR Hota commented on HDFS-13522:
[~elgoiri] Thanks for following-up.
[~hemanthboyina] Thanks for uploading the patch and feel free to take this
jira. I can also help with the code review. Meanwhile can you help me
understand if the consistency guarantees are same with and without router or
router relaxes the consistency guarantees ? This was a discussion point when we
were last working on this. Please refer to the notes in the thread. The last
design doc which was uploaded was intended to allow routers in the middle to
still honor the same consistency guarantees that client to
Nameode/ObserverNamenode honor without routers.
> Support observer node from Router-Based Federation
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-13522.001.patch, HDFS-13522_WIP.patch, RBF_
> Observer support.pdf, Router+Observer RPC clogging.png,
> ShortTerm-Routers+Observer.png
>
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15551) Tiny Improve for DeadNode detector
[
https://issues.apache.org/jira/browse/HDFS-15551?focusedWorklogId=479104=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479104
]
ASF GitHub Bot logged work on HDFS-15551:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 13:35
Start Date: 04/Sep/20 13:35
Worklog Time Spent: 10m
Work Description: imbajin commented on a change in pull request #2265:
URL: https://github.com/apache/hadoop/pull/2265#discussion_r483619060
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java
##
@@ -475,6 +475,7 @@ public synchronized void addNodeToDetect(DFSInputStream
dfsInputStream,
datanodeInfos.add(datanodeInfo);
}
+LOG.warn("Add datanode {} to suspectAndDeadNodes", datanodeInfo);
Review comment:
use DEBUG as default now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479104)
Time Spent: 1h 20m (was: 1h 10m)
> Tiny Improve for DeadNode detector
> --
>
> Key: HDFS-15551
> URL: https://issues.apache.org/jira/browse/HDFS-15551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Affects Versions: 3.3.0
>Reporter: dark_num
>Assignee: imbajin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> # add or improve some logs for adding local & global deadnodes
> # logic improve
> # fix typo
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15551) Tiny Improve for DeadNode detector
[ https://issues.apache.org/jira/browse/HDFS-15551?focusedWorklogId=479079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479079 ] ASF GitHub Bot logged work on HDFS-15551: - Author: ASF GitHub Bot Created on: 04/Sep/20 12:12 Start Date: 04/Sep/20 12:12 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2265: URL: https://github.com/apache/hadoop/pull/2265#issuecomment-687104767 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 28m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 27m 55s | trunk passed | | +1 :green_heart: | compile | 0m 58s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 51s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 58s | trunk passed | | +1 :green_heart: | shadedclient | 15m 30s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 26s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 24s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 49s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | compile | 0m 43s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 43s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 46s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 3s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 33s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 2m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 4s | hadoop-hdfs-client in the patch passed. | | +1 :green_heart: | asflicense | 0m 34s | The patch does not generate ASF License warnings. | | | | 104m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2265/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2265 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 53e09050e647 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 696e4fe50e4 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2265/2/testReport/ | | Max. process+thread count | 415 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2265/2/console | | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 | | Powered by | Apache
[jira] [Commented] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
[
https://issues.apache.org/jira/browse/HDFS-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190678#comment-17190678
]
Hadoop QA commented on HDFS-15558:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
50s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
55s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
30s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
59s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
|
[jira] [Work logged] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
[ https://issues.apache.org/jira/browse/HDFS-15558?focusedWorklogId=479063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479063 ] ASF GitHub Bot logged work on HDFS-15558: - Author: ASF GitHub Bot Created on: 04/Sep/20 11:03 Start Date: 04/Sep/20 11:03 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2275: URL: https://github.com/apache/hadoop/pull/2275#issuecomment-687077465 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 30m 55s | trunk passed | | +1 :green_heart: | compile | 4m 50s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 3m 55s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 1m 5s | trunk passed | | +1 :green_heart: | mvnsite | 2m 19s | trunk passed | | +1 :green_heart: | shadedclient | 19m 22s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 35s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 0s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 30s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 36s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 58s | the patch passed | | +1 :green_heart: | compile | 3m 59s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 3m 59s | the patch passed | | +1 :green_heart: | compile | 3m 37s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 3m 37s | the patch passed | | +1 :green_heart: | checkstyle | 0m 50s | the patch passed | | +1 :green_heart: | mvnsite | 1m 59s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 10s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 23s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 57s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 5m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 6s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 98m 15s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. | | | | 215m 20s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2275/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2275 | | JIRA Issue | HDFS-15558 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 54631ccca339 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 139a43e98e2 | | Default Java
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=479050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479050 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 04/Sep/20 10:01 Start Date: 04/Sep/20 10:01 Worklog Time Spent: 10m Work Description: YaYun-Wang commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r477304478 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: The "isTransientStorage" method is still available. In the original code,` isTransient()` and `isTransientStorage` methods are used to determine whether to support FsDatasetCache, Persistent, Quota, and Movable. FsDatasetCache will be used When the storage type is persistent. NVDIMM is RAM to some extent, which is fast. However, NVDIMM is a persistent storage type. Then, `isTransient()` and `isTransientStorage() ` used to determine whether to support FsDatasetCache can't meet the requirements. Therefore, we add `isRAM()` and `isRAMStorage() ` methods to decide whether cache is supported or not. And the other functions, such as, Persistent, Quota, and Movable judged by`isTransient()` and `isTransientStorage() ` methods. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 479050) Time Spent: 2h 50m (was: 2h 40m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 2h 50m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=479047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479047
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 09:55
Start Date: 04/Sep/20 09:55
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483515521
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Sorry, I made a big mistake in the above reply. Our design idea is:
NVDIMM supports mover and balancer, and `isTransient() ` applied in the case.`
isRAM()` is only used to “FsDatasetCache” judgment. So the current code is
reasonable, i think it‘s not necessary to modify the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479047)
Time Spent: 2h 40m (was: 2.5h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=479046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479046
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 09:54
Start Date: 04/Sep/20 09:54
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483515521
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Sorry, I made a big mistake in the above reply. Our design idea is:
NVDIMM supports mover and balancer, and `isTransient() ` applied in the case.
isRAM() is only used to “FsDatasetCache” judgment. So the current code is
reasonable, i think it‘s not necessary to modify the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479046)
Time Spent: 2.5h (was: 2h 20m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15556) Fix NPE in DatanodeDescriptor#updateStorageStats when handle DN Lifeline
[
https://issues.apache.org/jira/browse/HDFS-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190601#comment-17190601
]
huhaiyang edited comment on HDFS-15556 at 9/4/20, 7:28 AM:
---
The current issue is the same as [HDFS-14042|
https://issues.apache.org/jira/browse/HDFS-14042].
was (Author: haiyang hu):
The current issue is the same as[HDFS-14042|
https://issues.apache.org/jira/browse/HDFS-14042].
> Fix NPE in DatanodeDescriptor#updateStorageStats when handle DN Lifeline
>
>
> Key: HDFS-15556
> URL: https://issues.apache.org/jira/browse/HDFS-15556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0
>Reporter: huhaiyang
>Priority: Critical
> Attachments: HDFS-15556.001.patch, NN-CPU.png, NN_DN.LOG
>
>
> In our cluster, the NameNode appears NPE when processing lifeline messages
> sent by the DataNode, which will cause an maxLoad exception calculated by NN.
> because DataNode is identified as busy and unable to allocate available nodes
> in choose DataNode, program loop execution results in high CPU and reduces
> the processing performance of the cluster.
> *NameNode the exception stack*:
> {code:java}
> 2020-08-25 00:59:02,977 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 5 on 8022, call Call#20535 Retry#0
> org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol.sendLifeline
> from x:34766
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.updateStorageStats(DatanodeDescriptor.java:460)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.updateHeartbeatState(DatanodeDescriptor.java:390)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.updateLifeline(HeartbeatManager.java:254)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleLifeline(DatanodeManager.java:1805)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleLifeline(FSNamesystem.java:4039)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendLifeline(NameNodeRpcServer.java:1761)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeLifelineProtocolServerSideTranslatorPB.sendLifeline(DatanodeLifelineProtocolServerSideTranslatorPB.java:62)
> at
> org.apache.hadoop.hdfs.protocol.proto.DatanodeLifelineProtocolProtos$DatanodeLifelineProtocolService$2.callBlockingMethod(DatanodeLifelineProtocolProtos.java:409)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:886)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:828)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1903)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2717)
> {code}
> {code:java}
> // DatanodeDescriptor#updateStorageStats
> ...
> for (StorageReport report : reports) {
> DatanodeStorageInfo storage = null;
> synchronized (storageMap) {
> storage =
> storageMap.get(report.getStorage().getStorageID());
> }
> if (checkFailedStorages) {
> failedStorageInfos.remove(storage);
> }
> storage.receivedHeartbeat(report); // NPE exception occurred here
> // skip accounting for capacity of PROVIDED storages!
> if (StorageType.PROVIDED.equals(storage.getStorageType())) {
> continue;
> }
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15556) Fix NPE in DatanodeDescriptor#updateStorageStats when handle DN Lifeline
[
https://issues.apache.org/jira/browse/HDFS-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190601#comment-17190601
]
huhaiyang commented on HDFS-15556:
--
The current issue is the same as[HDFS-14042|
https://issues.apache.org/jira/browse/HDFS-14042].
> Fix NPE in DatanodeDescriptor#updateStorageStats when handle DN Lifeline
>
>
> Key: HDFS-15556
> URL: https://issues.apache.org/jira/browse/HDFS-15556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0
>Reporter: huhaiyang
>Priority: Critical
> Attachments: HDFS-15556.001.patch, NN-CPU.png, NN_DN.LOG
>
>
> In our cluster, the NameNode appears NPE when processing lifeline messages
> sent by the DataNode, which will cause an maxLoad exception calculated by NN.
> because DataNode is identified as busy and unable to allocate available nodes
> in choose DataNode, program loop execution results in high CPU and reduces
> the processing performance of the cluster.
> *NameNode the exception stack*:
> {code:java}
> 2020-08-25 00:59:02,977 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 5 on 8022, call Call#20535 Retry#0
> org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol.sendLifeline
> from x:34766
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.updateStorageStats(DatanodeDescriptor.java:460)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.updateHeartbeatState(DatanodeDescriptor.java:390)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.updateLifeline(HeartbeatManager.java:254)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleLifeline(DatanodeManager.java:1805)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleLifeline(FSNamesystem.java:4039)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendLifeline(NameNodeRpcServer.java:1761)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeLifelineProtocolServerSideTranslatorPB.sendLifeline(DatanodeLifelineProtocolServerSideTranslatorPB.java:62)
> at
> org.apache.hadoop.hdfs.protocol.proto.DatanodeLifelineProtocolProtos$DatanodeLifelineProtocolService$2.callBlockingMethod(DatanodeLifelineProtocolProtos.java:409)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:886)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:828)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1903)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2717)
> {code}
> {code:java}
> // DatanodeDescriptor#updateStorageStats
> ...
> for (StorageReport report : reports) {
> DatanodeStorageInfo storage = null;
> synchronized (storageMap) {
> storage =
> storageMap.get(report.getStorage().getStorageID());
> }
> if (checkFailedStorages) {
> failedStorageInfos.remove(storage);
> }
> storage.receivedHeartbeat(report); // NPE exception occurred here
> // skip accounting for capacity of PROVIDED storages!
> if (StorageType.PROVIDED.equals(storage.getStorageType())) {
> continue;
> }
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
[ https://issues.apache.org/jira/browse/HDFS-15558?focusedWorklogId=478976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478976 ] ASF GitHub Bot logged work on HDFS-15558: - Author: ASF GitHub Bot Created on: 04/Sep/20 07:26 Start Date: 04/Sep/20 07:26 Worklog Time Spent: 10m Work Description: umamaheswararao opened a new pull request #2275: URL: https://github.com/apache/hadoop/pull/2275 https://issues.apache.org/jira/browse/HDFS-15558 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 478976) Remaining Estimate: 0h Time Spent: 10m > ViewDistributedFileSystem#recoverLease should call super.recoverLease when > there are no mounts configured > - > > Key: HDFS-15558 > URL: https://issues.apache.org/jira/browse/HDFS-15558 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
[ https://issues.apache.org/jira/browse/HDFS-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15558: -- Labels: pull-request-available (was: ) > ViewDistributedFileSystem#recoverLease should call super.recoverLease when > there are no mounts configured > - > > Key: HDFS-15558 > URL: https://issues.apache.org/jira/browse/HDFS-15558 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478972
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 07:18
Start Date: 04/Sep/20 07:18
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
NVDIMM is special RAM, the data above can be stored persistently. It can
be regarded as a general hardware device. We don't have to consider what
storage type it is, balancer and mover can be applied on NVDIMM, therefore, I
think it is better to use isRAM to determine whether to use mover . In
addition, neither RAM nor nvdimm need FsDatasetCache, and isTransient() used to
determine whether FsDatasetCache is needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478972)
Time Spent: 2h 20m (was: 2h 10m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?focusedWorklogId=478969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478969 ] ASF GitHub Bot logged work on HDFS-15554: - Author: ASF GitHub Bot Created on: 04/Sep/20 07:07 Start Date: 04/Sep/20 07:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2266: URL: https://github.com/apache/hadoop/pull/2266#issuecomment-686959347 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 30m 49s | trunk passed | | +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 31s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 22s | trunk passed | | +1 :green_heart: | mvnsite | 0m 36s | trunk passed | | +1 :green_heart: | shadedclient | 16m 36s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 52s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 10s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 31s | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 0 new + 30 unchanged - 2 fixed = 30 total (was 32) | | +1 :green_heart: | compile | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 26s | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 0 new + 30 unchanged - 2 fixed = 30 total (was 32) | | -0 :warning: | checkstyle | 0m 16s | hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 16 new + 0 unchanged - 0 fixed = 16 total (was 0) | | +1 :green_heart: | mvnsite | 0m 29s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 38s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 14s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 10m 35s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 87m 37s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterAdminCLI | | | hadoop.hdfs.server.federation.router.TestRouterMountTableCacheRefresh | | | hadoop.hdfs.server.federation.router.TestRouterFaultTolerant | | | hadoop.hdfs.server.federation.router.TestRouterAllResolver | | | hadoop.hdfs.server.federation.router.TestRouterMountTableCacheRefreshSecure | | | hadoop.hdfs.server.federation.router.TestDisableNameservices | | | hadoop.hdfs.server.federation.router.TestRouterFsck | | | hadoop.hdfs.server.federation.router.TestRouterQuota | | | hadoop.hdfs.server.federation.router.TestRouterMissingFolderMulti | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2266/2/artifact/out/Dockerfile | |
[jira] [Work logged] (HDFS-15551) Tiny Improve for DeadNode detector
[
https://issues.apache.org/jira/browse/HDFS-15551?focusedWorklogId=478967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478967
]
ASF GitHub Bot logged work on HDFS-15551:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 07:04
Start Date: 04/Sep/20 07:04
Worklog Time Spent: 10m
Work Description: leosunli commented on a change in pull request #2265:
URL: https://github.com/apache/hadoop/pull/2265#discussion_r483384047
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java
##
@@ -396,13 +395,13 @@ private void probeCallBack(Probe probe, boolean success) {
probe.getDatanodeInfo());
removeDeadNode(probe.getDatanodeInfo());
} else if (probe.getType() == ProbeType.CHECK_SUSPECT) {
-LOG.debug("Remove the node out from suspect node list: {}.",
+LOG.info("Remove the node out from suspect node list: {}.",
Review comment:
when a lot of stale relicas, it should have many supsect nodes but
not dead nodes.
These nodes all will print this log.
What is the purpose of printing this log?
The client can access normally the suspect node.
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java
##
@@ -475,6 +475,7 @@ public synchronized void addNodeToDetect(DFSInputStream
dfsInputStream,
datanodeInfos.add(datanodeInfo);
}
+LOG.warn("Add datanode {} to suspectAndDeadNodes", datanodeInfo);
Review comment:
One case: when a lot of stale relicas, will the log flood?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478967)
Time Spent: 1h (was: 50m)
> Tiny Improve for DeadNode detector
> --
>
> Key: HDFS-15551
> URL: https://issues.apache.org/jira/browse/HDFS-15551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Affects Versions: 3.3.0
>Reporter: dark_num
>Assignee: imbajin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> # add or improve some logs for adding local & global deadnodes
> # logic improve
> # fix typo
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478956
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 06:31
Start Date: 04/Sep/20 06:31
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483414755
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
> neither RAM nor nvdimm need FsDatasetCache, and isTransient() used to
determine whether FsDatasetCache is needed
I think we have agreed on this.
> I think it is better to use isRAM to determine whether to use mover
So just to be clear, if we use `isRAM()` to determine, both RAM_DISK and
NVDIMM will return true and thus will simply disable Balancer and Mover. I was
proposing that only when we do not want to move those NVDIMM data replicas
around (disk/node) ever. Is that the design here? I'm fine with that but I
think this contradicts with the statement that _"It can be regarded as a
general hardware device. "_
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478956)
Time Spent: 2h 10m (was: 2h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478955
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 06:24
Start Date: 04/Sep/20 06:24
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483194125
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
Oh, I was thinking that allowing Balancer to move the NVDIMM data is by
design since they are not volatile. But if that is *not* case, then we can
update Balancer code by replacing `isTransient()` call with `isRAM()` call. Not
sure if this makes more sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478955)
Time Spent: 2h (was: 1h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478954
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 06:20
Start Date: 04/Sep/20 06:20
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
NVDIMM is special RAM, the data above can be stored persistently. It can
be regarded as a general hardware device. We don't have to consider what
storage type it is. Therefore, I think it is better to use isRAM to determine
whether to use mover . In addition, neither RAM nor nvdimm need FsDatasetCache,
and isTransient() used to determine whether FsDatasetCache is needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478954)
Time Spent: 1h 50m (was: 1h 40m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=478953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478953
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 04/Sep/20 06:19
Start Date: 04/Sep/20 06:19
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
NVDIMM is RAM, however, the data above can be stored persistently. It
can be regarded as a general hardware device. We don't have to consider what
storage type it is. Therefore, I think it is better to use isRAM to determine
whether to use mover . In addition, neither RAM nor nvdimm need FsDatasetCache,
and isTransient() used to determine whether FsDatasetCache is needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 478953)
Time Spent: 1h 40m (was: 1.5h)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
