[jira] [Commented] (HDFS-14815) RBF: Update the quota in MountTable when calling setQuota on a MountTable src.
[ https://issues.apache.org/jira/browse/HDFS-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193997#comment-17193997 ] Hemanth Boyina commented on HDFS-14815: --- thanks for the explanation [~LiJinglun] after this change , through file system object we couldn't able to set quota on a mount point , before the change we could able to set quota so it looks like an incompatible change > RBF: Update the quota in MountTable when calling setQuota on a MountTable src. > -- > > Key: HDFS-14815 > URL: https://issues.apache.org/jira/browse/HDFS-14815 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14815.001.patch, HDFS-14815.002.patch, > HDFS-14815.003.patch, HDFS-14815.004.patch, HDFS-14815.005.patch > > > The method setQuota() can make the remote quota(the quota on real clusters) > inconsistent with the MountTable. I think we have 3 ways to fix it: > # Reject all the setQuota() rpcs if it trys to change the quota of a mount > table. > # Let setQuota() to change the mount table quota. Update the quota on zk > first and then update remote quotas. > # Do nothing. The RouterQuotaUpdateService will finally make all the remote > quota right. We can tolerate short-term inconsistencies. > I'd like option 1 because I want the RouterAdmin to be the only entrance to > update the MountTable. > Option 3 we don't need change anything, but the quota will be inconsistent > for a short-term. The remote quota will be effective immediately and > auto-changed back after a while. User might be confused about the behavior. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15555) RBF: Refresh cacheNS when SocketException occurs
[ https://issues.apache.org/jira/browse/HDFS-1?focusedWorklogId=481895=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481895 ] ASF GitHub Bot logged work on HDFS-1: - Author: ASF GitHub Bot Created on: 11/Sep/20 05:21 Start Date: 11/Sep/20 05:21 Worklog Time Spent: 10m Work Description: aajisaka commented on pull request #2267: URL: https://github.com/apache/hadoop/pull/2267#issuecomment-690880934 HI @hemanthboyina I don't have any updates here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481895) Time Spent: 1h 10m (was: 1h) > RBF: Refresh cacheNS when SocketException occurs > > > Key: HDFS-1 > URL: https://issues.apache.org/jira/browse/HDFS-1 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: HDFS 3.3.0, Java 11 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Problem: > When active NameNode is restarted and loading fsimage, DFSRouters > significantly slow down. > Investigation: > When active NameNode is restarted and loading fsimage, RouterRpcClient > receives SocketException. Since > RouterRpcClient#isUnavailableException(IOException) returns false when the > argument is SocketException, the MembershipNameNodeResolver#cacheNS is not > refreshed. That's why the order of the NameNodes returned by > MemberShipNameNodeResolver#getNamenodesForNameserviceId(String) is unchanged > and the active NameNode is still returned first. Therefore RouterRpcClient > still tries to connect to the NameNode that is loading fsimage. > After loading the fsimage, the NameNode throws StandbyException. The > exception is one of the 'Unavailable Exception' and the cacheNS is refreshed. > Workaround: > Stop NameNode and wait 1 minute before starting NameNode instead of > restarting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15555) RBF: Refresh cacheNS when SocketException occurs
[ https://issues.apache.org/jira/browse/HDFS-1?focusedWorklogId=481893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481893 ] ASF GitHub Bot logged work on HDFS-1: - Author: ASF GitHub Bot Created on: 11/Sep/20 05:13 Start Date: 11/Sep/20 05:13 Worklog Time Spent: 10m Work Description: hemanthboyina commented on pull request #2267: URL: https://github.com/apache/hadoop/pull/2267#issuecomment-690878265 any update here @aajisaka , as HDFS-15543 was modifying same part of code This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481893) Time Spent: 1h (was: 50m) > RBF: Refresh cacheNS when SocketException occurs > > > Key: HDFS-1 > URL: https://issues.apache.org/jira/browse/HDFS-1 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: HDFS 3.3.0, Java 11 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Problem: > When active NameNode is restarted and loading fsimage, DFSRouters > significantly slow down. > Investigation: > When active NameNode is restarted and loading fsimage, RouterRpcClient > receives SocketException. Since > RouterRpcClient#isUnavailableException(IOException) returns false when the > argument is SocketException, the MembershipNameNodeResolver#cacheNS is not > refreshed. That's why the order of the NameNodes returned by > MemberShipNameNodeResolver#getNamenodesForNameserviceId(String) is unchanged > and the active NameNode is still returned first. Therefore RouterRpcClient > still tries to connect to the NameNode that is loading fsimage. > After loading the fsimage, the NameNode throws StandbyException. The > exception is one of the 'Unavailable Exception' and the cacheNS is refreshed. > Workaround: > Stop NameNode and wait 1 minute before starting NameNode instead of > restarting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15456) TestExternalStoragePolicySatisfier fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15456:
--
Labels: pull-request-available test (was: test)
> TestExternalStoragePolicySatisfier fails intermittently
> ---
>
> Key: HDFS-15456
> URL: https://issues.apache.org/jira/browse/HDFS-15456
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Leon Gao
>Priority: Major
> Labels: pull-request-available, test
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestExternalStoragePolicySatisfier}} frequently times-out on hadoop trunk
> {code:bash}
> [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 421.443 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
> [ERROR]
> testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier)
> Time elapsed: 43.983 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition. Thread diagnostics:
> Timestamp: 2020-07-07 07:51:10,267
> "IPC Server handler 4 on default port 44933" daemon prio=5 tid=1138
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "ForkJoinPool-2-worker-19" daemon prio=5 tid=235 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1/current/BP-912129709-172.17.0.2-1594151429636"
> daemon prio=5 tid=1217 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
> at java.lang.Thread.run(Thread.java:748)
> "Socket Reader #1 for port 0" daemon prio=5 tid=1192 runnable
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252)
> "pool-90-thread-1" prio=5 tid=1069 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 2 on default port 37995" daemon prio=5 tid=1169
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
>
[jira] [Work logged] (HDFS-15456) TestExternalStoragePolicySatisfier fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15456?focusedWorklogId=481891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481891
]
ASF GitHub Bot logged work on HDFS-15456:
-
Author: ASF GitHub Bot
Created on: 11/Sep/20 05:07
Start Date: 11/Sep/20 05:07
Worklog Time Spent: 10m
Work Description: LeonGao91 opened a new pull request #2299:
URL: https://github.com/apache/hadoop/pull/2299
One-liner fix of this UT.
Ignore datanode load so block placement can successfully pick fallback
storage type.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481891)
Remaining Estimate: 0h
Time Spent: 10m
> TestExternalStoragePolicySatisfier fails intermittently
> ---
>
> Key: HDFS-15456
> URL: https://issues.apache.org/jira/browse/HDFS-15456
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Leon Gao
>Priority: Major
> Labels: test
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestExternalStoragePolicySatisfier}} frequently times-out on hadoop trunk
> {code:bash}
> [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 421.443 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
> [ERROR]
> testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier)
> Time elapsed: 43.983 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition. Thread diagnostics:
> Timestamp: 2020-07-07 07:51:10,267
> "IPC Server handler 4 on default port 44933" daemon prio=5 tid=1138
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "ForkJoinPool-2-worker-19" daemon prio=5 tid=235 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1/current/BP-912129709-172.17.0.2-1594151429636"
> daemon prio=5 tid=1217 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
> at java.lang.Thread.run(Thread.java:748)
> "Socket Reader #1 for port 0" daemon prio=5 tid=1192 runnable
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252)
> "pool-90-thread-1" prio=5 tid=1069 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
>
[jira] [Assigned] (HDFS-15456) TestExternalStoragePolicySatisfier fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Gao reassigned HDFS-15456:
---
Assignee: Leon Gao
> TestExternalStoragePolicySatisfier fails intermittently
> ---
>
> Key: HDFS-15456
> URL: https://issues.apache.org/jira/browse/HDFS-15456
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Leon Gao
>Priority: Major
> Labels: test
>
> {{TestExternalStoragePolicySatisfier}} frequently times-out on hadoop trunk
> {code:bash}
> [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 421.443 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
> [ERROR]
> testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier)
> Time elapsed: 43.983 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition. Thread diagnostics:
> Timestamp: 2020-07-07 07:51:10,267
> "IPC Server handler 4 on default port 44933" daemon prio=5 tid=1138
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "ForkJoinPool-2-worker-19" daemon prio=5 tid=235 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1/current/BP-912129709-172.17.0.2-1594151429636"
> daemon prio=5 tid=1217 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
> at java.lang.Thread.run(Thread.java:748)
> "Socket Reader #1 for port 0" daemon prio=5 tid=1192 runnable
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252)
> "pool-90-thread-1" prio=5 tid=1069 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 2 on default port 37995" daemon prio=5 tid=1169
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at
>
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15532:
---
Affects Version/s: 3.4.0
Status: Patch Available (was: Open)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[
https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=481871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481871
]
ASF GitHub Bot logged work on HDFS-15539:
-
Author: ASF GitHub Bot
Created on: 11/Sep/20 03:12
Start Date: 11/Sep/20 03:12
Worklog Time Spent: 10m
Work Description: smengcl commented on pull request #2258:
URL: https://github.com/apache/hadoop/pull/2258#issuecomment-690847462
> Thanks @smengcl for working on this,. The test failures like
TestDistributedFileSystem#testGetTrashRoots look related. Can you plz verify?
I'm checking. Probably need to add a line or two to clean up the old tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481871)
Time Spent: 1h 10m (was: 1h)
> When disallowing snapshot on a dir, throw exception if its trash root is not
> empty
> --
>
> Key: HDFS-15539
> URL: https://issues.apache.org/jira/browse/HDFS-15539
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
> trash root in that dir anymore (if any). The risk is the trash root will be
> left there forever.
> We need to throw an exception there and prompt the user to clean up or rename
> the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[ https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=481869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481869 ] ASF GitHub Bot logged work on HDFS-15539: - Author: ASF GitHub Bot Created on: 11/Sep/20 03:07 Start Date: 11/Sep/20 03:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2258: URL: https://github.com/apache/hadoop/pull/2258#issuecomment-690845948 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 28m 49s | trunk passed | | +1 :green_heart: | compile | 4m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 4m 15s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 1m 2s | trunk passed | | +1 :green_heart: | mvnsite | 2m 14s | trunk passed | | +1 :green_heart: | shadedclient | 19m 0s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 25s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 56s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 28s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 38s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 56s | the patch passed | | +1 :green_heart: | compile | 4m 10s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 4m 10s | the patch passed | | +1 :green_heart: | compile | 3m 46s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 3m 46s | the patch passed | | +1 :green_heart: | checkstyle | 0m 53s | the patch passed | | +1 :green_heart: | mvnsite | 1m 59s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 50s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 20s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 47s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 7m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 18s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 77m 54s | hadoop-hdfs in the patch passed. | | +0 :ok: | asflicense | 0m 38s | ASF License check generated no output? | | | | 194m 25s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.TestReencryption | | | hadoop.hdfs.server.namenode.TestFSEditLogLoader | | | hadoop.hdfs.server.namenode.TestNameNodeReconfigure | | | hadoop.hdfs.server.namenode.TestNamenodeStorageDirectives | | | hadoop.hdfs.server.namenode.TestNameEditsConfigs | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.server.namenode.TestFileLimit | | | hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile | | | hadoop.hdfs.TestAppendDifferentChecksum | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.namenode.TestCheckpoint | | | hadoop.hdfs.server.namenode.TestMetadataVersionOutput | | | hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness | | | hadoop.hdfs.server.namenode.TestFSImageWithXAttr | | | hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocksWithRandomECPolicy | | | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier | | | hadoop.hdfs.TestDFSStripedInputStream | | | hadoop.hdfs.server.namenode.TestStoragePolicySatisfierWithHA | | |
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15532:
--
Labels: pull-request-available (was: )
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=481864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481864
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 11/Sep/20 03:00
Start Date: 11/Sep/20 03:00
Worklog Time Spent: 10m
Work Description: umamaheswararao opened a new pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298
https://issues.apache.org/jira/browse/HDFS-15532
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481864)
Remaining Estimate: 0h
Time Spent: 10m
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15532:
---
Summary: listFiles on root/InternalDir will fail if fallback root has file
(was: listFiles on root will fail if fallback root has file)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481857 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 02:43 Start Date: 11/Sep/20 02:43 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690839476 Will check again later this week. Ideally we can get a clean QA. Could you check the test failures and make sure they are not related? Thanks, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481857) Time Spent: 4.5h (was: 4h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481846 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 02:08 Start Date: 11/Sep/20 02:08 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690829884 @liuml07 and @brahmareddybattula So if all of the codes are ok, would you please to approve? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481846) Time Spent: 4h 20m (was: 4h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481837 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 01:14 Start Date: 11/Sep/20 01:14 Worklog Time Spent: 10m Work Description: YaYun-Wang commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716810 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: > So, NVDIMM is peristent storage and RAM. yes, that’s right. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481837) Time Spent: 4h 10m (was: 4h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481836
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 11/Sep/20 01:12
Start Date: 11/Sep/20 01:12
Worklog Time Spent: 10m
Work Description: YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716471
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
> My final query then, why can't have one NVDIMM like one SSD as this
also movable and peristent..?
Considering NVDIMM is faster, so NVDIMM does not use `FsDatasetCache()`
which SSD needs in the design.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481836)
Time Spent: 4h (was: 3h 50m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?focusedWorklogId=481828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481828 ] ASF GitHub Bot logged work on HDFS-15554: - Author: ASF GitHub Bot Created on: 11/Sep/20 00:59 Start Date: 11/Sep/20 00:59 Worklog Time Spent: 10m Work Description: fengnanli commented on pull request #2266: URL: https://github.com/apache/hadoop/pull/2266#issuecomment-690809794 The TestRouterRpcMultiDestination test passed locally. @goiri Can you help commit it? Thanks a lot! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481828) Time Spent: 3h 20m (was: 3h 10m) > RBF: force router check file existence in destinations before adding/updating > mount points > -- > > Key: HDFS-15554 > URL: https://issues.apache.org/jira/browse/HDFS-15554 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > Adding/Updating mount points right now is only a router action without > validation in the downstream namenodes for the destination files/directories. > In practice we have set up the dangling mount points and when clients call > listStatus they would get the file returned, but then if they try to access > the file FileNotFoundException would be thrown out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[
https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=481795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481795
]
ASF GitHub Bot logged work on HDFS-15539:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 23:51
Start Date: 10/Sep/20 23:51
Worklog Time Spent: 10m
Work Description: smengcl commented on a change in pull request #2258:
URL: https://github.com/apache/hadoop/pull/2258#discussion_r486694935
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
##
@@ -2442,4 +2442,38 @@ public void testGetTrashRootOnEZInSnapshottableDir()
}
}
}
+
+ @Test
+ public void testDisallowSnapshotShouldThrowWhenTrashRootExists()
+ throws IOException {
+Configuration conf = getTestConfiguration();
+MiniDFSCluster cluster =
+new MiniDFSCluster.Builder(conf).numDataNodes(1).build();
+try {
+ DistributedFileSystem dfs = cluster.getFileSystem();
+ Path testDir = new Path("/disallowss/test1/");
+ Path file0path = new Path(testDir, "file-0");
+ dfs.create(file0path);
+ dfs.allowSnapshot(testDir);
+ // Create trash root manually
+ Path testDirTrashRoot = new Path(testDir, FileSystem.TRASH_PREFIX);
+ dfs.mkdirs(testDirTrashRoot);
+ // Try disallowing snapshot, should throw
+ try {
+dfs.disallowSnapshot(testDir);
+fail("Should have thrown IOException when trash root exists inside "
Review comment:
Thanks! I have updated accordingly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481795)
Time Spent: 50m (was: 40m)
> When disallowing snapshot on a dir, throw exception if its trash root is not
> empty
> --
>
> Key: HDFS-15539
> URL: https://issues.apache.org/jira/browse/HDFS-15539
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
> trash root in that dir anymore (if any). The risk is the trash root will be
> left there forever.
> We need to throw an exception there and prompt the user to clean up or rename
> the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?focusedWorklogId=481758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481758 ] ASF GitHub Bot logged work on HDFS-15554: - Author: ASF GitHub Bot Created on: 10/Sep/20 22:32 Start Date: 10/Sep/20 22:32 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2266: URL: https://github.com/apache/hadoop/pull/2266#issuecomment-690766821 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 33m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 21s | trunk passed | | +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 40s | trunk passed | | +1 :green_heart: | shadedclient | 15m 26s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 54s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 10s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 32s | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 0 new + 30 unchanged - 2 fixed = 30 total (was 32) | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 28s | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 0 new + 30 unchanged - 2 fixed = 30 total (was 32) | | +1 :green_heart: | checkstyle | 0m 17s | the patch passed | | +1 :green_heart: | mvnsite | 0m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 13m 56s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 14s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 8m 39s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 114m 33s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2266/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2266 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint | | uname | Linux 490356438ed4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 9960c01a25c | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions |
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[
https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=481668=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481668
]
ASF GitHub Bot logged work on HDFS-15568:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 19:49
Start Date: 10/Sep/20 19:49
Worklog Time Spent: 10m
Work Description: szetszwo commented on a change in pull request #2296:
URL: https://github.com/apache/hadoop/pull/2296#discussion_r486594478
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##
@@ -448,22 +455,31 @@ public String createSnapshot(final LeaseManager
leaseManager,
"snapshot IDs and ID rollover is not supported.");
}
int n = numSnapshots.get();
-if (n >= maxSnapshotFSLimit) {
- // We have reached the maximum snapshot limit
- throw new SnapshotException(
- "Failed to create snapshot: there are already " + (n + 1)
- + " snapshot(s) and the max snapshot limit is "
- + maxSnapshotFSLimit);
-}
-
-srcRoot.addSnapshot(snapshotCounter, snapshotName, leaseManager,
-this.captureOpenFiles, maxSnapshotLimit, mtime);
+checkSnapshotLimit(maxSnapshotFSLimit, n);
+srcRoot.addSnapshot(this, snapshotName, leaseManager, mtime);
//create success, update id
snapshotCounter++;
numSnapshots.getAndIncrement();
return Snapshot.getSnapshotPath(snapshotRoot, snapshotName);
}
+
+ void checkSnapshotLimit(int limit, int numSnapshots)
Review comment:
I suggest to add limit type to the error message as below.
```
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
index 266c0a71241..7a47ab4000d 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
@@ -190,8 +190,7 @@ public Snapshot addSnapshot(INodeDirectory snapshotRoot,
+ n + " snapshot(s) and the snapshot quota is "
+ snapshotQuota);
}
-snapshotManager.checkSnapshotLimit(snapshotManager.
-getMaxSnapshotLimit(), n);
+snapshotManager.checkPerDirectorySnapshotLimit(n);
final Snapshot s = new Snapshot(id, name, snapshotRoot);
final byte[] nameBytes = s.getRoot().getLocalNameBytes();
final int i = searchSnapshot(nameBytes);
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
index 0a2e18c3dc3..7c482074486 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
@@ -455,7 +455,7 @@ public String createSnapshot(final LeaseManager
leaseManager,
"snapshot IDs and ID rollover is not supported.");
}
int n = numSnapshots.get();
-checkSnapshotLimit(maxSnapshotFSLimit, n);
+checkFileSystemSnapshotLimit(n);
srcRoot.addSnapshot(this, snapshotName, leaseManager, mtime);
//create success, update id
@@ -464,12 +464,19 @@ public String createSnapshot(final LeaseManager
leaseManager,
return Snapshot.getSnapshotPath(snapshotRoot, snapshotName);
}
- void checkSnapshotLimit(int limit, int numSnapshots)
- throws SnapshotException {
+ void checkFileSystemSnapshotLimit(int n) throws SnapshotException {
+checkSnapshotLimit(maxSnapshotFSLimit, n, "file system");
+ }
+
+ void checkPerDirectorySnapshotLimit(int n) throws SnapshotException {
+checkSnapshotLimit(maxSnapshotLimit, n, "per directory");
+ }
+
+ private void checkSnapshotLimit(int limit, int numSnapshots,
+ String type) throws SnapshotException {
if (numSnapshots >= limit) {
- String msg = "there are already " + (numSnapshots + 1)
- + " snapshot(s) and the max snapshot limit is "
- + limit;
+ String msg = "There are already " + (numSnapshots + 1)
+ + " snapshot(s) and the " + type + " snapshot limit is " + limit;
if (fsdir.isImageLoaded()) {
// We have reached the maximum snapshot limit
throw new SnapshotException(
```
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193808#comment-17193808 ] Íñigo Goiri commented on HDFS-13293: Let's try, but keep in mind that toucking something like CallerContext in common is usually controversial. > RBF: The RouterRPCServer should transfer CallerContext and client ip to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Fei Hui >Priority: Major > Attachments: HDFS-13293.001.patch > > > Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[
https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=481660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481660
]
ASF GitHub Bot logged work on HDFS-15568:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 19:21
Start Date: 10/Sep/20 19:21
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2296:
URL: https://github.com/apache/hadoop/pull/2296#discussion_r486579348
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
##
@@ -508,6 +508,10 @@ FSNamesystem getFSNamesystem() {
return namesystem;
}
+ public boolean isImageLoaded() {
Review comment:
Add javadoc
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##
@@ -368,6 +368,13 @@ void assertFirstSnapshot(INodeDirectory dir,
}
}
+ boolean captureOpenFiles() {
Review comment:
javadoc
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##
@@ -368,6 +368,13 @@ void assertFirstSnapshot(INodeDirectory dir,
}
}
+ boolean captureOpenFiles() {
+return captureOpenFiles;
+ }
+
+ int getMaxSnapshotLimit() {
Review comment:
VisibleForTesting
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java
##
@@ -133,4 +137,68 @@ public void testValidateSnapshotIDWidth() throws Exception
{
getMaxSnapshotID() < Snapshot.CURRENT_STATE_ID);
}
+ @Test
+ public void SnapshotLimitOnRestart() throws Exception {
+final Configuration conf = new Configuration();
+final Path snapshottableDir
+= new Path("/" + getClass().getSimpleName());
+int numSnapshots = 5;
+conf.setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, numSnapshots);
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_FILESYSTEM_LIMIT,
+numSnapshots * 2);
+MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(0).build();
+cluster.waitActive();
+DistributedFileSystem hdfs = cluster.getFileSystem();
+hdfs.mkdirs(snapshottableDir);
+hdfs.allowSnapshot(snapshottableDir);
+int i = 0;
+for (; i < numSnapshots; i++) {
+ hdfs.createSnapshot(snapshottableDir, "s" + i);
+}
+try {
+ hdfs.createSnapshot(snapshottableDir, "s" + i);
+ Assert.fail("Expected SnapshotException not thrown");
+} catch (SnapshotException se) {
+ Assert.assertTrue(
+ StringUtils.toLowerCase(se.getMessage()).contains(
+ "max snapshot limit"));
+}
+
+// now change max snapshot directory limit to 2 and restart namenode
+cluster.getNameNode().getConf().setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, 2);
+cluster.restartNameNodes();
+
+// make sure edits of all previous 5 create snapshots are replayed
+Assert.assertEquals(numSnapshots, cluster.getNamesystem().
+getSnapshotManager().getNumSnapshots());
+
+// make sure namenode has the new snapshot limit configured as 2
+Assert.assertEquals(2,
+cluster.getNamesystem().getSnapshotManager().getMaxSnapshotLimit());
+
+// Any new snapshot creation should still fail
+try {
+ hdfs.createSnapshot(snapshottableDir, "s" + i);
+ Assert.fail("Expected SnapshotException not thrown");
+} catch (SnapshotException se) {
+ Assert.assertTrue(
Review comment:
LambdaTestUtils
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java
##
@@ -133,4 +137,68 @@ public void testValidateSnapshotIDWidth() throws Exception
{
getMaxSnapshotID() < Snapshot.CURRENT_STATE_ID);
}
+ @Test
+ public void SnapshotLimitOnRestart() throws Exception {
+final Configuration conf = new Configuration();
+final Path snapshottableDir
+= new Path("/" + getClass().getSimpleName());
+int numSnapshots = 5;
+conf.setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, numSnapshots);
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_FILESYSTEM_LIMIT,
+numSnapshots * 2);
+MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(0).build();
+cluster.waitActive();
+DistributedFileSystem hdfs = cluster.getFileSystem();
+hdfs.mkdirs(snapshottableDir);
+hdfs.allowSnapshot(snapshottableDir);
+int i = 0;
+for (; i < numSnapshots; i++) {
+ hdfs.createSnapshot(snapshottableDir, "s" + i);
+}
+try {
+ hdfs.createSnapshot(snapshottableDir, "s" + i);
+ Assert.fail("Expected
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481641
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 18:27
Start Date: 10/Sep/20 18:27
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486549682
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
My final query then, why can't have one NVDIMM like one SSD as this also
movable and peristent..?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481641)
Time Spent: 3h 50m (was: 3h 40m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481640 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 10/Sep/20 18:25 Start Date: 10/Sep/20 18:25 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486548053 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: So, NVDIMM is peristent storage and RAM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481640) Time Spent: 3h 40m (was: 3.5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481637 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 10/Sep/20 18:21 Start Date: 10/Sep/20 18:21 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486545111 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: Ok. Got it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481637) Time Spent: 3.5h (was: 3h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 3.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15565) Remove the invalid code in the Balancer#doBalance() method.
[
https://issues.apache.org/jira/browse/HDFS-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193774#comment-17193774
]
Mingliang Liu commented on HDFS-15565:
--
Closing this issue and removing the fix versions.
> Remove the invalid code in the Balancer#doBalance() method.
> ---
>
> Key: HDFS-15565
> URL: https://issues.apache.org/jira/browse/HDFS-15565
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover
>Reporter: jianghua zhu
>Assignee: jianghua zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15565.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In the Balancer#doBalance() method, an invalid line of code is added, as
> follows:
> static private int doBalance(Collection namenodes,
> Collection nsIds, final BalancerParameters p, Configuration conf)
> throws IOException, InterruptedException
> { ... System.out.println("Time Stamp Iteration# Bytes Already Moved Bytes
> Left To Move Bytes Being Moved"); ... }
>
> I think it was originally used for testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Closed] (HDFS-15565) Remove the invalid code in the Balancer#doBalance() method.
[
https://issues.apache.org/jira/browse/HDFS-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu closed HDFS-15565.
> Remove the invalid code in the Balancer#doBalance() method.
> ---
>
> Key: HDFS-15565
> URL: https://issues.apache.org/jira/browse/HDFS-15565
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover
>Reporter: jianghua zhu
>Assignee: jianghua zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15565.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In the Balancer#doBalance() method, an invalid line of code is added, as
> follows:
> static private int doBalance(Collection namenodes,
> Collection nsIds, final BalancerParameters p, Configuration conf)
> throws IOException, InterruptedException
> { ... System.out.println("Time Stamp Iteration# Bytes Already Moved Bytes
> Left To Move Bytes Being Moved"); ... }
>
> I think it was originally used for testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15565) Remove the invalid code in the Balancer#doBalance() method.
[
https://issues.apache.org/jira/browse/HDFS-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-15565:
-
Fix Version/s: (was: 3.3.0)
> Remove the invalid code in the Balancer#doBalance() method.
> ---
>
> Key: HDFS-15565
> URL: https://issues.apache.org/jira/browse/HDFS-15565
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover
>Reporter: jianghua zhu
>Assignee: jianghua zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15565.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In the Balancer#doBalance() method, an invalid line of code is added, as
> follows:
> static private int doBalance(Collection namenodes,
> Collection nsIds, final BalancerParameters p, Configuration conf)
> throws IOException, InterruptedException
> { ... System.out.println("Time Stamp Iteration# Bytes Already Moved Bytes
> Left To Move Bytes Being Moved"); ... }
>
> I think it was originally used for testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481626
]
ASF GitHub Bot logged work on HDFS-15025:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 18:07
Start Date: 10/Sep/20 18:07
Worklog Time Spent: 10m
Work Description: brahmareddybattula commented on a change in pull
request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486535639
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+return isRAM;
+ }
Review comment:
ok, By design if you dn't want to move.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481626)
Time Spent: 3h 20m (was: 3h 10m)
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15569) Speed up the Storage#doRecover during datanode rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Boyina updated HDFS-15569:
--
Description:
When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not having
enough memory upgrade failed , Adjusted memory configurations and re upgraded
datanode ,
Now datanode upgrade has taken more time , on analyzing found that
Storage#deleteDir has taken more time in RECOVER_UPGRADE state
{code:java}
"Thread-28" #270 daemon prio=5 os_prio=0 tid=0x7fed5a9b8000 nid=0x2b5c
runnable [0x7fdcdad2a000]"Thread-28" #270 daemon prio=5 os_prio=0
tid=0x7fed5a9b8000 nid=0x2b5c runnable [0x7fdcdad2a000]
java.lang.Thread.State: RUNNABLE at java.io.UnixFileSystem.delete0(Native
Method) at java.io.UnixFileSystem.delete(UnixFileSystem.java:265) at
java.io.File.delete(File.java:1041) at
org.apache.hadoop.fs.FileUtil.deleteImpl(FileUtil.java:229) at
org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:270) at
org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:153) at
org.apache.hadoop.hdfs.server.common.Storage.deleteDir(Storage.java:1348) at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.doRecover(Storage.java:782)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:174)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:224)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:253)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:455)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:389)
- locked <0x7fdf08ec7548> (a
org.apache.hadoop.hdfs.server.datanode.DataStorage) at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:557)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1761)
- locked <0x7fdf08ec7598> (a
org.apache.hadoop.hdfs.server.datanode.DataNode) at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1697)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:392)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:282)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:822)
at java.lang.Thread.run(Thread.java:748) {code}
was:
When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not having
enough memory upgrade failed , Adjusted memory configurations and re upgraded
datanode ,
Now datanode upgrade has taken more time , on analyzing found that
Storage#deleteDir has taken more time in Recover_Upgrade state
> Speed up the Storage#doRecover during datanode rolling upgrade
> ---
>
> Key: HDFS-15569
> URL: https://issues.apache.org/jira/browse/HDFS-15569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hemanth Boyina
>Assignee: Hemanth Boyina
>Priority: Major
>
> When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not
> having enough memory upgrade failed , Adjusted memory configurations and re
> upgraded datanode ,
> Now datanode upgrade has taken more time , on analyzing found that
> Storage#deleteDir has taken more time in RECOVER_UPGRADE state
> {code:java}
> "Thread-28" #270 daemon prio=5 os_prio=0 tid=0x7fed5a9b8000 nid=0x2b5c
> runnable [0x7fdcdad2a000]"Thread-28" #270 daemon prio=5 os_prio=0
> tid=0x7fed5a9b8000 nid=0x2b5c runnable [0x7fdcdad2a000]
> java.lang.Thread.State: RUNNABLE at java.io.UnixFileSystem.delete0(Native
> Method) at java.io.UnixFileSystem.delete(UnixFileSystem.java:265) at
> java.io.File.delete(File.java:1041) at
> org.apache.hadoop.fs.FileUtil.deleteImpl(FileUtil.java:229) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:270) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
>
[jira] [Commented] (HDFS-15569) Speed up the Storage#doRecover during datanode rolling upgrade
[ https://issues.apache.org/jira/browse/HDFS-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193701#comment-17193701 ] Hemanth Boyina commented on HDFS-15569: --- One solution can be rename the current directory to current.tmp directory and delete the current.tmp in parallel , as the current directory doesn't exists now datanode upgrade can be proceeded > Speed up the Storage#doRecover during datanode rolling upgrade > --- > > Key: HDFS-15569 > URL: https://issues.apache.org/jira/browse/HDFS-15569 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hemanth Boyina >Assignee: Hemanth Boyina >Priority: Major > > When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not > having enough memory upgrade failed , Adjusted memory configurations and re > upgraded datanode , > Now datanode upgrade has taken more time , on analyzing found that > Storage#deleteDir has taken more time in Recover_Upgrade state -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15569) Speed up the Storage#doRecover during datanode rolling upgrade
Hemanth Boyina created HDFS-15569: - Summary: Speed up the Storage#doRecover during datanode rolling upgrade Key: HDFS-15569 URL: https://issues.apache.org/jira/browse/HDFS-15569 Project: Hadoop HDFS Issue Type: Improvement Reporter: Hemanth Boyina Assignee: Hemanth Boyina When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not having enough memory upgrade failed , Adjusted memory configurations and re upgraded datanode , Now datanode upgrade has taken more time , on analyzing found that Storage#deleteDir has taken more time in Recover_Upgrade state -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15516) Add info for create flags in NameNode audit logs
[ https://issues.apache.org/jira/browse/HDFS-15516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193586#comment-17193586 ] jianghua zhu edited comment on HDFS-15516 at 9/10/20, 12:35 PM: [~linyiqun], [~chaosun] , do you have other ideas to share? was (Author: jianghuazhu): [~linyiqun], [~sunchao] , do you have other ideas to share? > Add info for create flags in NameNode audit logs > > > Key: HDFS-15516 > URL: https://issues.apache.org/jira/browse/HDFS-15516 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Shashikant Banerjee >Assignee: jianghua zhu >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15516.001.patch, HDFS-15516.002.patch, > HDFS-15516.003.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, if file create happens with flags like overwrite , the audit logs > doesn't seem to contain the info regarding the flags in the audit logs. It > would be useful to add info regarding the create options in the audit logs > similar to Rename ops. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15516) Add info for create flags in NameNode audit logs
[ https://issues.apache.org/jira/browse/HDFS-15516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193586#comment-17193586 ] jianghua zhu commented on HDFS-15516: - [~linyiqun], [~sunchao] , do you have other ideas to share? > Add info for create flags in NameNode audit logs > > > Key: HDFS-15516 > URL: https://issues.apache.org/jira/browse/HDFS-15516 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Shashikant Banerjee >Assignee: jianghua zhu >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15516.001.patch, HDFS-15516.002.patch, > HDFS-15516.003.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, if file create happens with flags like overwrite , the audit logs > doesn't seem to contain the info regarding the flags in the audit logs. It > would be useful to add info regarding the create options in the audit logs > similar to Rename ops. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[ https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=481397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481397 ] ASF GitHub Bot logged work on HDFS-15568: - Author: ASF GitHub Bot Created on: 10/Sep/20 11:49 Start Date: 10/Sep/20 11:49 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2296: URL: https://github.com/apache/hadoop/pull/2296#issuecomment-690215954 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 40m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 32m 7s | trunk passed | | +1 :green_heart: | compile | 1m 16s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 1m 8s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 47s | trunk passed | | +1 :green_heart: | mvnsite | 1m 15s | trunk passed | | +1 :green_heart: | shadedclient | 17m 51s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 50s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 19s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 11s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 8s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 1m 1s | the patch passed | | -0 :warning: | checkstyle | 0m 40s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 89 unchanged - 0 fixed = 92 total (was 89) | | +1 :green_heart: | mvnsite | 1m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 27s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 46s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 16s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 80m 10s | hadoop-hdfs in the patch passed. | | -1 :x: | asflicense | 0m 35s | The patch generated 4 ASF License warnings. | | | | 209m 18s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS | | | hadoop.fs.viewfs.TestViewFileSystemHdfs | | | hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.fs.viewfs.TestViewFsAtHdfsRoot | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2296/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2296 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 94a92cfd8c29 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / e5fe3262702 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle |
[jira] [Work logged] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15563?focusedWorklogId=481372=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481372
]
ASF GitHub Bot logged work on HDFS-15563:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 11:08
Start Date: 10/Sep/20 11:08
Worklog Time Spent: 10m
Work Description: bshashikant commented on pull request #2295:
URL: https://github.com/apache/hadoop/pull/2295#issuecomment-690161198
Thanks @smengcl for the contribution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481372)
Time Spent: 40m (was: 0.5h)
> Incorrect getTrashRoot return value when a non-snapshottable dir prefix
> matches the path of a snapshottable dir
> ---
>
> Key: HDFS-15563
> URL: https://issues.apache.org/jira/browse/HDFS-15563
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 3.4.0
>Reporter: Nilotpal Nandi
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
> enabled.
> Root cause analysis:
> {{SnapshottableDirectoryStatus}} paths retrived inside
> {{DFSClient#getSnapshotRoot}} aren't appended with '/', causing some
> directories with the same path prefix to be mistakenly classified as
> snapshottable directory.
> Thanks [~shashikant] for the test case addition.
> ---
> Repro:
> {code:java}
> 1. snapshottable directory present in the cluster
> hdfs lsSnapshottableDir
> drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
> drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536
> /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable
> directory
> hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it
> failed
> hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.
> {code}
> "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
> "*/user/hrt_4/newdir/subdir/.Trash*"
> as clear from the msg here:
> {noformat}
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15563?focusedWorklogId=481356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481356
]
ASF GitHub Bot logged work on HDFS-15563:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 10:25
Start Date: 10/Sep/20 10:25
Worklog Time Spent: 10m
Work Description: bshashikant merged pull request #2295:
URL: https://github.com/apache/hadoop/pull/2295
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481356)
Time Spent: 0.5h (was: 20m)
> Incorrect getTrashRoot return value when a non-snapshottable dir prefix
> matches the path of a snapshottable dir
> ---
>
> Key: HDFS-15563
> URL: https://issues.apache.org/jira/browse/HDFS-15563
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 3.4.0
>Reporter: Nilotpal Nandi
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
> enabled.
> Root cause analysis:
> {{SnapshottableDirectoryStatus}} paths retrived inside
> {{DFSClient#getSnapshotRoot}} aren't appended with '/', causing some
> directories with the same path prefix to be mistakenly classified as
> snapshottable directory.
> Thanks [~shashikant] for the test case addition.
> ---
> Repro:
> {code:java}
> 1. snapshottable directory present in the cluster
> hdfs lsSnapshottableDir
> drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
> drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536
> /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable
> directory
> hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it
> failed
> hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.
> {code}
> "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
> "*/user/hrt_4/newdir/subdir/.Trash*"
> as clear from the msg here:
> {noformat}
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15563:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Incorrect getTrashRoot return value when a non-snapshottable dir prefix
> matches the path of a snapshottable dir
> ---
>
> Key: HDFS-15563
> URL: https://issues.apache.org/jira/browse/HDFS-15563
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 3.4.0
>Reporter: Nilotpal Nandi
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
> enabled.
> Root cause analysis:
> {{SnapshottableDirectoryStatus}} paths retrived inside
> {{DFSClient#getSnapshotRoot}} aren't appended with '/', causing some
> directories with the same path prefix to be mistakenly classified as
> snapshottable directory.
> Thanks [~shashikant] for the test case addition.
> ---
> Repro:
> {code:java}
> 1. snapshottable directory present in the cluster
> hdfs lsSnapshottableDir
> drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
> drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536
> /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable
> directory
> hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it
> failed
> hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.
> {code}
> "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
> "*/user/hrt_4/newdir/subdir/.Trash*"
> as clear from the msg here:
> {noformat}
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14997) BPServiceActor processes commands from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193490#comment-17193490 ] Stephen O'Donnell commented on HDFS-14997: -- [~Captainhzy] Do you have this patch included on your cluster and you see the heartbeat thread blocked for a long time? What type of command is holding the lock? Do you have any jstack traces showing the problem? > BPServiceActor processes commands from NameNode asynchronously > -- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch, > HDFS-14997.003.patch, HDFS-14997.004.patch, HDFS-14997.005.patch, > HDFS-14997.addendum.patch, image-2019-12-26-16-15-44-814.png > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15548) Allow configuring DISK/ARCHIVE storage types on same device mount
[ https://issues.apache.org/jira/browse/HDFS-15548?focusedWorklogId=481308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481308 ] ASF GitHub Bot logged work on HDFS-15548: - Author: ASF GitHub Bot Created on: 10/Sep/20 08:41 Start Date: 10/Sep/20 08:41 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2288: URL: https://github.com/apache/hadoop/pull/2288#issuecomment-690086118 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 22s | trunk passed | | +1 :green_heart: | compile | 1m 18s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 1m 9s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 56s | trunk passed | | +1 :green_heart: | mvnsite | 1m 18s | trunk passed | | +1 :green_heart: | shadedclient | 18m 7s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 52s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 20s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 8s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 11s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javac | 1m 9s | hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 2 new + 602 unchanged - 0 fixed = 604 total (was 602) | | +1 :green_heart: | compile | 1m 2s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | javac | 1m 2s | hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 2 new + 586 unchanged - 0 fixed = 588 total (was 586) | | -0 :warning: | checkstyle | 0m 50s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 569 unchanged - 0 fixed = 576 total (was 569) | | +1 :green_heart: | mvnsite | 1m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 48s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 46s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 15s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 13s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 113m 15s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 202m 26s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2288/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2288 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 64599a30402d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[
https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=481302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481302
]
ASF GitHub Bot logged work on HDFS-15568:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 08:19
Start Date: 10/Sep/20 08:19
Worklog Time Spent: 10m
Work Description: bshashikant opened a new pull request #2296:
URL: https://github.com/apache/hadoop/pull/2296
please see https://issues.apache.org/jira/browse/HDFS-15568
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481302)
Remaining Estimate: 0h
Time Spent: 10m
> namenode start failed to start when dfs.namenode.snapshot.max.limit set
> ---
>
> Key: HDFS-15568
> URL: https://issues.apache.org/jira/browse/HDFS-15568
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> 11:35:05.872 AM ERROR NameNode
> Failed to start namenode.
> org.apache.hadoop.hdfs.protocol.SnapshotException: Failed to add snapshot:
> there are already 20 snapshot(s) and the max snapshot limit is 20
> at
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.addSnapshot(DirectorySnapshottableFeature.java:181)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSnapshot(INodeDirectory.java:285)
> at
> org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:447)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:802)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:287)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:760)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:337)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:755)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:646)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:717)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:933)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1670)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1737)
> {code}
> Steps to reproduce:
> --
> directory level snapshot limit set - 100
> Created 100 snapshots
> deleted all 100 snapshots (in-oder)
> No snapshot exist
> Then, directory level snapshot limit set - 20
> HDFS restart
> Namenode start failed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[
https://issues.apache.org/jira/browse/HDFS-15568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15568:
--
Labels: pull-request-available (was: )
> namenode start failed to start when dfs.namenode.snapshot.max.limit set
> ---
>
> Key: HDFS-15568
> URL: https://issues.apache.org/jira/browse/HDFS-15568
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> 11:35:05.872 AM ERROR NameNode
> Failed to start namenode.
> org.apache.hadoop.hdfs.protocol.SnapshotException: Failed to add snapshot:
> there are already 20 snapshot(s) and the max snapshot limit is 20
> at
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.addSnapshot(DirectorySnapshottableFeature.java:181)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSnapshot(INodeDirectory.java:285)
> at
> org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:447)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:802)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:287)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:760)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:337)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:755)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:646)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:717)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:933)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1670)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1737)
> {code}
> Steps to reproduce:
> --
> directory level snapshot limit set - 100
> Created 100 snapshots
> deleted all 100 snapshots (in-oder)
> No snapshot exist
> Then, directory level snapshot limit set - 20
> HDFS restart
> Namenode start failed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
Shashikant Banerjee created HDFS-15568:
--
Summary: namenode start failed to start when
dfs.namenode.snapshot.max.limit set
Key: HDFS-15568
URL: https://issues.apache.org/jira/browse/HDFS-15568
Project: Hadoop HDFS
Issue Type: Sub-task
Components: snapshots
Reporter: Nilotpal Nandi
Assignee: Shashikant Banerjee
{code:java}
11:35:05.872 AM ERROR NameNode
Failed to start namenode.
org.apache.hadoop.hdfs.protocol.SnapshotException: Failed to add snapshot:
there are already 20 snapshot(s) and the max snapshot limit is 20
at
org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.addSnapshot(DirectorySnapshottableFeature.java:181)
at
org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSnapshot(INodeDirectory.java:285)
at
org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:447)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:802)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:287)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:760)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:337)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1164)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:755)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:646)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:717)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:960)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:933)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1670)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1737)
{code}
Steps to reproduce:
--
directory level snapshot limit set - 100
Created 100 snapshots
deleted all 100 snapshots (in-oder)
No snapshot exist
Then, directory level snapshot limit set - 20
HDFS restart
Namenode start failed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15565) Remove the invalid code in the Balancer#doBalance() method.
[
https://issues.apache.org/jira/browse/HDFS-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193406#comment-17193406
]
jianghua zhu commented on HDFS-15565:
-
[~sunchao] , [~hexiaoqiao] , thank you very much for your suggestions.
> Remove the invalid code in the Balancer#doBalance() method.
> ---
>
> Key: HDFS-15565
> URL: https://issues.apache.org/jira/browse/HDFS-15565
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover
>Reporter: jianghua zhu
>Assignee: jianghua zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.0
>
> Attachments: HDFS-15565.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In the Balancer#doBalance() method, an invalid line of code is added, as
> follows:
> static private int doBalance(Collection namenodes,
> Collection nsIds, final BalancerParameters p, Configuration conf)
> throws IOException, InterruptedException
> { ... System.out.println("Time Stamp Iteration# Bytes Already Moved Bytes
> Left To Move Bytes Being Moved"); ... }
>
> I think it was originally used for testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15565) Remove the invalid code in the Balancer#doBalance() method.
[
https://issues.apache.org/jira/browse/HDFS-15565?focusedWorklogId=481275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481275
]
ASF GitHub Bot logged work on HDFS-15565:
-
Author: ASF GitHub Bot
Created on: 10/Sep/20 06:45
Start Date: 10/Sep/20 06:45
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on a change in pull request #2292:
URL: https://github.com/apache/hadoop/pull/2292#discussion_r486101882
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
##
@@ -705,7 +705,6 @@ static private int doBalance(Collection namenodes,
LOG.info("excluded nodes = " + p.getExcludedNodes());
LOG.info("source nodes = " + p.getSourceNodes());
checkKeytabAndInit(conf);
-System.out.println("Time Stamp Iteration# Bytes Already
Moved Bytes Left To Move Bytes Being Moved");
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481275)
Time Spent: 50m (was: 40m)
> Remove the invalid code in the Balancer#doBalance() method.
> ---
>
> Key: HDFS-15565
> URL: https://issues.apache.org/jira/browse/HDFS-15565
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover
>Reporter: jianghua zhu
>Assignee: jianghua zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.0
>
> Attachments: HDFS-15565.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In the Balancer#doBalance() method, an invalid line of code is added, as
> follows:
> static private int doBalance(Collection namenodes,
> Collection nsIds, final BalancerParameters p, Configuration conf)
> throws IOException, InterruptedException
> { ... System.out.println("Time Stamp Iteration# Bytes Already Moved Bytes
> Left To Move Bytes Being Moved"); ... }
>
> I think it was originally used for testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193397#comment-17193397 ] Fei Hui commented on HDFS-13293: [~aajisaka][~elgoiri] I think maybe we need to do three things. * File a new jira and extend CallerContext in hadoop common, it can contains many key value pairs. * add real client ip to the caller context in this jira. *hadoop.caller.context.enabled* has been used by audit log ,should we add new parameter? * File a new jira and Fix the way Yarn use CallerContext (Add key value to the context) What do you think ? > RBF: The RouterRPCServer should transfer CallerContext and client ip to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Fei Hui >Priority: Major > Attachments: HDFS-13293.001.patch > > > Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
