[hadoop] branch trunk updated: HDFS-16014: Fix an issue in checking native pmdk lib by 'hadoop checknative' command (#3762)

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 8e08f43  HDFS-16014: Fix an issue in checking native pmdk lib by 
'hadoop checknative' command (#3762)
8e08f43 is described below

commit 8e08f43e0386e355c6e32282e5f5eadc5904ec2d
Author: PHILO-HE 
AuthorDate: Tue Dec 14 17:15:12 2021 +0800

HDFS-16014: Fix an issue in checking native pmdk lib by 'hadoop 
checknative' command (#3762)
---
 .../main/java/org/apache/hadoop/io/nativeio/NativeIO.java |  2 +-
 .../native/src/org/apache/hadoop/io/nativeio/pmdk_load.c  | 15 ---
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
index 79b489b..ebe7f21 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
@@ -141,7 +141,7 @@ public class NativeIO {
   }
 }
 
-// Denotes the state of supporting PMDK. The value is set by JNI.
+// Denotes the state of supporting PMDK. The actual value is set via JNI.
 private static SupportState pmdkSupportState =
 SupportState.UNSUPPORTED;
 
diff --git 
a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c
 
b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c
index 502508c..f1a1df5 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c
+++ 
b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c
@@ -35,13 +35,14 @@
 #endif
 
 PmdkLibLoader * pmdkLoader;
+// 1 represents loaded. Otherwise, not loaded.
+int pmdkLoaded;
 
 /**
  *  pmdk_load.c
  *  Utility of loading the libpmem library and the required functions.
  *  Building of this codes won't rely on any libpmem source codes, but running
  *  into this will rely on successfully loading of the dynamic library.
- *
  */
 
 static const char* load_functions() {
@@ -56,6 +57,10 @@ static const char* load_functions() {
   return NULL;
 }
 
+/**
+ * It should be idempotent to call this function for checking
+ * whether PMDK lib is successfully loaded.
+ */
 void load_pmdk_lib(char* err, size_t err_len) {
   const char* errMsg;
   const char* library = NULL;
@@ -67,10 +72,13 @@ void load_pmdk_lib(char* err, size_t err_len) {
 
   err[0] = '\0';
 
-  if (pmdkLoader != NULL) {
+  if (pmdkLoaded == 1) {
 return;
   }
-  pmdkLoader = calloc(1, sizeof(PmdkLibLoader));
+
+  if (pmdkLoader == NULL) {
+pmdkLoader = calloc(1, sizeof(PmdkLibLoader));
+  }
 
   // Load PMDK library
   #ifdef UNIX
@@ -103,4 +111,5 @@ void load_pmdk_lib(char* err, size_t err_len) {
   }
 
   pmdkLoader->libname = strdup(library);
+  pmdkLoaded = 1;
 }

-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch trunk updated (00d483d -> 1364847)

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git.


from 00d483d  HDFS-16324. Fix error log in BlockManagerSafeMode (#3661)
 add 1364847  HDFS-15788. Correct the statement for pmem cache to reflect 
cache persistence support (#3761)

No new revisions were added by this update.

Summary of changes:
 .../hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md  | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch trunk updated (82522d6 -> df4006e)

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git.


from 82522d6  HADOOP-17281 Implement FileSystem.listStatusIterator() in 
S3AFileSystem (#2354)
 add df4006e  HDFS-15253 Default checkpoint transfer speed, 50mb per second 
(#2366)

No new revisions were added by this update.

Summary of changes:
 .../src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java   | 2 +-
 hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.1 updated: HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new b1eb478  HDFS-15080. Fix the issue in reading persistent memory cached 
data with an offset. Contributed by Feilong He.
b1eb478 is described below

commit b1eb47865c7ec9796f966c7b134758bf456d0d84
Author: Rakesh Radhakrishnan 
AuthorDate: Wed Jan 8 14:25:17 2020 +0530

HDFS-15080. Fix the issue in reading persistent memory cached data with an 
offset. Contributed by Feilong He.

(cherry picked from commit 7030722e5d9f376245a9ab0a6a883538b6c55f82)
---
 .../hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
index faf0935..1ce2910 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
@@ -821,7 +821,7 @@ class FsDatasetImpl implements FsDatasetSpi {
   if (addr != -1) {
 LOG.debug("Get InputStream by cache address.");
 return FsDatasetUtil.getDirectInputStream(
-addr, info.getBlockDataLength());
+addr + seekOffset, info.getBlockDataLength() - seekOffset);
   }
   LOG.debug("Get InputStream by cache file path.");
   return FsDatasetUtil.getInputStreamAndSeek(


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.2 updated: HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new d7f9a90  HDFS-15080. Fix the issue in reading persistent memory cached 
data with an offset. Contributed by Feilong He.
d7f9a90 is described below

commit d7f9a90e2c7e367c57b7e19ce0497bad05b95e1a
Author: Rakesh Radhakrishnan 
AuthorDate: Wed Jan 8 14:25:17 2020 +0530

HDFS-15080. Fix the issue in reading persistent memory cached data with an 
offset. Contributed by Feilong He.

(cherry picked from commit 7030722e5d9f376245a9ab0a6a883538b6c55f82)
---
 .../hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
index faf0935..1ce2910 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
@@ -821,7 +821,7 @@ class FsDatasetImpl implements FsDatasetSpi {
   if (addr != -1) {
 LOG.debug("Get InputStream by cache address.");
 return FsDatasetUtil.getDirectInputStream(
-addr, info.getBlockDataLength());
+addr + seekOffset, info.getBlockDataLength() - seekOffset);
   }
   LOG.debug("Get InputStream by cache file path.");
   return FsDatasetUtil.getInputStreamAndSeek(


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch trunk updated: HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 7030722  HDFS-15080. Fix the issue in reading persistent memory cached 
data with an offset. Contributed by Feilong He.
7030722 is described below

commit 7030722e5d9f376245a9ab0a6a883538b6c55f82
Author: Rakesh Radhakrishnan 
AuthorDate: Wed Jan 8 14:25:17 2020 +0530

HDFS-15080. Fix the issue in reading persistent memory cached data with an 
offset. Contributed by Feilong He.
---
 .../hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
index f356949..0feb3b7 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
@@ -822,7 +822,7 @@ class FsDatasetImpl implements FsDatasetSpi {
   if (addr != -1) {
 LOG.debug("Get InputStream by cache address.");
 return FsDatasetUtil.getDirectInputStream(
-addr, info.getBlockDataLength());
+addr + seekOffset, info.getBlockDataLength() - seekOffset);
   }
   LOG.debug("Get InputStream by cache file path.");
   return FsDatasetUtil.getInputStreamAndSeek(


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.1 updated: HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 8b748c0  HDFS-14740. Recover data blocks from persistent memory read 
cache during datanode restarts. Contributed by Feilong He.
8b748c0 is described below

commit 8b748c0308ab504f66f2a984eaec1a3d87116f2f
Author: Rakesh Radhakrishnan 
AuthorDate: Thu Jan 2 11:44:00 2020 +0530

HDFS-14740. Recover data blocks from persistent memory read cache during 
datanode restarts. Contributed by Feilong He.

(cherry picked from commit d79cce20abbbf321f6dcce03f4087544124a7cd2)
---
 .../org/apache/hadoop/io/nativeio/NativeIO.java|  21 ++-
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   |  25 +--
 .../apache/hadoop/io/nativeio/TestNativeIO.java|  13 +-
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |  10 +-
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |  17 +-
 .../datanode/fsdataset/impl/FsDatasetCache.java|  28 +++-
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  15 +-
 .../datanode/fsdataset/impl/MappableBlock.java |   7 +
 .../fsdataset/impl/MappableBlockLoader.java|   7 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |   7 +
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |   6 +
 .../impl/NativePmemMappableBlockLoader.java|  31 +++-
 .../fsdataset/impl/NativePmemMappedBlock.java  |   9 +-
 .../fsdataset/impl/PmemMappableBlockLoader.java|  37 -
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  10 +-
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 120 +++---
 .../src/main/resources/hdfs-default.xml|  15 +-
 .../site/markdown/CentralizedCacheManagement.md|   6 +-
 .../impl/TestCacheByPmemMappableBlockLoader.java   |  20 ++-
 ...BlockLoader.java => TestPmemCacheRecovery.java} | 183 +++--
 20 files changed, 425 insertions(+), 162 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
index f2b3456..feea6d8 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
@@ -223,28 +223,31 @@ public class NativeIO {
  * JNI wrapper of persist memory operations.
  */
 public static class Pmem {
-  // check whether the address is a Pmem address or DIMM address
+  // Check whether the address is a Pmem address or DIMM address
   public static boolean isPmem(long address, long length) {
 return NativeIO.POSIX.isPmemCheck(address, length);
   }
 
-  // create a pmem file and memory map it
-  public static PmemMappedRegion mapBlock(String path, long length) {
-return NativeIO.POSIX.pmemCreateMapFile(path, length);
+  // Map a file in persistent memory, if the given file exists,
+  // directly map it. If not, create the named file on persistent memory
+  // and then map it.
+  public static PmemMappedRegion mapBlock(
+  String path, long length, boolean isFileExist) {
+return NativeIO.POSIX.pmemMapFile(path, length, isFileExist);
   }
 
-  // unmap a pmem file
+  // Unmap a pmem file
   public static boolean unmapBlock(long address, long length) {
 return NativeIO.POSIX.pmemUnMap(address, length);
   }
 
-  // copy data from disk file(src) to pmem file(dest), without flush
+  // Copy data from disk file(src) to pmem file(dest), without flush
   public static void memCopy(byte[] src, long dest, boolean isPmem,
   long length) {
 NativeIO.POSIX.pmemCopy(src, dest, isPmem, length);
   }
 
-  // flush the memory content to persistent storage
+  // Flush the memory content to persistent storage
   public static void memSync(PmemMappedRegion region) {
 if (region.isPmem()) {
   NativeIO.POSIX.pmemDrain();
@@ -260,8 +263,8 @@ public class NativeIO {
 
 private static native String getPmdkLibPath();
 private static native boolean isPmemCheck(long address, long length);
-private static native PmemMappedRegion pmemCreateMapFile(String path,
-long length);
+private static native PmemMappedRegion pmemMapFile(String path,
+long length, boolean isFileExist);
 private static native boolean pmemUnMap(long address, long length);
 private static native void pmemCopy(byte[] src, long dest, boolean isPmem,
 long length);
diff --git 
a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
 
b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
in

[hadoop] branch branch-3.2 updated: HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new a27f9bd  HDFS-14740. Recover data blocks from persistent memory read 
cache during datanode restarts. Contributed by Feilong He.
a27f9bd is described below

commit a27f9bd28e7ca6b94b7ca3210400c296d0defdb2
Author: Rakesh Radhakrishnan 
AuthorDate: Thu Jan 2 11:44:00 2020 +0530

HDFS-14740. Recover data blocks from persistent memory read cache during 
datanode restarts. Contributed by Feilong He.

(cherry picked from commit d79cce20abbbf321f6dcce03f4087544124a7cd2)
---
 .../org/apache/hadoop/io/nativeio/NativeIO.java|  21 ++-
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   |  25 +--
 .../apache/hadoop/io/nativeio/TestNativeIO.java|  13 +-
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |  10 +-
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |  17 +-
 .../datanode/fsdataset/impl/FsDatasetCache.java|  28 +++-
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  15 +-
 .../datanode/fsdataset/impl/MappableBlock.java |   7 +
 .../fsdataset/impl/MappableBlockLoader.java|   7 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |   7 +
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |   6 +
 .../impl/NativePmemMappableBlockLoader.java|  31 +++-
 .../fsdataset/impl/NativePmemMappedBlock.java  |   9 +-
 .../fsdataset/impl/PmemMappableBlockLoader.java|  37 -
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  10 +-
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 120 +++---
 .../src/main/resources/hdfs-default.xml|  15 +-
 .../site/markdown/CentralizedCacheManagement.md|   5 +-
 .../impl/TestCacheByPmemMappableBlockLoader.java   |  20 ++-
 ...BlockLoader.java => TestPmemCacheRecovery.java} | 183 +++--
 20 files changed, 424 insertions(+), 162 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
index 973afa3..160b8e0 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
@@ -224,28 +224,31 @@ public class NativeIO {
  * JNI wrapper of persist memory operations.
  */
 public static class Pmem {
-  // check whether the address is a Pmem address or DIMM address
+  // Check whether the address is a Pmem address or DIMM address
   public static boolean isPmem(long address, long length) {
 return NativeIO.POSIX.isPmemCheck(address, length);
   }
 
-  // create a pmem file and memory map it
-  public static PmemMappedRegion mapBlock(String path, long length) {
-return NativeIO.POSIX.pmemCreateMapFile(path, length);
+  // Map a file in persistent memory, if the given file exists,
+  // directly map it. If not, create the named file on persistent memory
+  // and then map it.
+  public static PmemMappedRegion mapBlock(
+  String path, long length, boolean isFileExist) {
+return NativeIO.POSIX.pmemMapFile(path, length, isFileExist);
   }
 
-  // unmap a pmem file
+  // Unmap a pmem file
   public static boolean unmapBlock(long address, long length) {
 return NativeIO.POSIX.pmemUnMap(address, length);
   }
 
-  // copy data from disk file(src) to pmem file(dest), without flush
+  // Copy data from disk file(src) to pmem file(dest), without flush
   public static void memCopy(byte[] src, long dest, boolean isPmem,
   long length) {
 NativeIO.POSIX.pmemCopy(src, dest, isPmem, length);
   }
 
-  // flush the memory content to persistent storage
+  // Flush the memory content to persistent storage
   public static void memSync(PmemMappedRegion region) {
 if (region.isPmem()) {
   NativeIO.POSIX.pmemDrain();
@@ -261,8 +264,8 @@ public class NativeIO {
 
 private static native String getPmdkLibPath();
 private static native boolean isPmemCheck(long address, long length);
-private static native PmemMappedRegion pmemCreateMapFile(String path,
-long length);
+private static native PmemMappedRegion pmemMapFile(String path,
+long length, boolean isFileExist);
 private static native boolean pmemUnMap(long address, long length);
 private static native void pmemCopy(byte[] src, long dest, boolean isPmem,
 long length);
diff --git 
a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
 
b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
in

[hadoop] branch trunk updated: HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new d79cce2  HDFS-14740. Recover data blocks from persistent memory read 
cache during datanode restarts. Contributed by Feilong He.
d79cce2 is described below

commit d79cce20abbbf321f6dcce03f4087544124a7cd2
Author: Rakesh Radhakrishnan 
AuthorDate: Thu Jan 2 11:44:00 2020 +0530

HDFS-14740. Recover data blocks from persistent memory read cache during 
datanode restarts. Contributed by Feilong He.
---
 .../org/apache/hadoop/io/nativeio/NativeIO.java|  21 ++-
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   |  25 +--
 .../apache/hadoop/io/nativeio/TestNativeIO.java|  13 +-
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |  10 +-
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |  17 +-
 .../datanode/fsdataset/impl/FsDatasetCache.java|  28 +++-
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  15 +-
 .../datanode/fsdataset/impl/MappableBlock.java |   7 +
 .../fsdataset/impl/MappableBlockLoader.java|   7 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |   7 +
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |   6 +
 .../impl/NativePmemMappableBlockLoader.java|  31 +++-
 .../fsdataset/impl/NativePmemMappedBlock.java  |   9 +-
 .../fsdataset/impl/PmemMappableBlockLoader.java|  37 -
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  10 +-
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 120 +++---
 .../src/main/resources/hdfs-default.xml|  15 +-
 .../site/markdown/CentralizedCacheManagement.md|   5 +-
 .../impl/TestCacheByPmemMappableBlockLoader.java   |  20 ++-
 ...BlockLoader.java => TestPmemCacheRecovery.java} | 183 +++--
 20 files changed, 424 insertions(+), 162 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
index 973afa3..160b8e0 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
@@ -224,28 +224,31 @@ public class NativeIO {
  * JNI wrapper of persist memory operations.
  */
 public static class Pmem {
-  // check whether the address is a Pmem address or DIMM address
+  // Check whether the address is a Pmem address or DIMM address
   public static boolean isPmem(long address, long length) {
 return NativeIO.POSIX.isPmemCheck(address, length);
   }
 
-  // create a pmem file and memory map it
-  public static PmemMappedRegion mapBlock(String path, long length) {
-return NativeIO.POSIX.pmemCreateMapFile(path, length);
+  // Map a file in persistent memory, if the given file exists,
+  // directly map it. If not, create the named file on persistent memory
+  // and then map it.
+  public static PmemMappedRegion mapBlock(
+  String path, long length, boolean isFileExist) {
+return NativeIO.POSIX.pmemMapFile(path, length, isFileExist);
   }
 
-  // unmap a pmem file
+  // Unmap a pmem file
   public static boolean unmapBlock(long address, long length) {
 return NativeIO.POSIX.pmemUnMap(address, length);
   }
 
-  // copy data from disk file(src) to pmem file(dest), without flush
+  // Copy data from disk file(src) to pmem file(dest), without flush
   public static void memCopy(byte[] src, long dest, boolean isPmem,
   long length) {
 NativeIO.POSIX.pmemCopy(src, dest, isPmem, length);
   }
 
-  // flush the memory content to persistent storage
+  // Flush the memory content to persistent storage
   public static void memSync(PmemMappedRegion region) {
 if (region.isPmem()) {
   NativeIO.POSIX.pmemDrain();
@@ -261,8 +264,8 @@ public class NativeIO {
 
 private static native String getPmdkLibPath();
 private static native boolean isPmemCheck(long address, long length);
-private static native PmemMappedRegion pmemCreateMapFile(String path,
-long length);
+private static native PmemMappedRegion pmemMapFile(String path,
+long length, boolean isFileExist);
 private static native boolean pmemUnMap(long address, long length);
 private static native void pmemCopy(byte[] src, long dest, boolean isPmem,
 long length);
diff --git 
a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
 
b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
index b0b5151..1d7c508 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/native/

[hadoop] 02/10: HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 0c6824120af4c95d912949f99bc5df57983ec9b3
Author: Rakesh Radhakrishnan 
AuthorDate: Fri Mar 29 00:18:15 2019 +0530

HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. 
Contributed by Rakesh R

(cherry picked from commit f3f51284d57ef2e0c7e968b6eea56eab578f7e93)
---
 .../datanode/fsdataset/impl/FsDatasetCache.java| 132 ++---
 .../fsdataset/impl/MappableBlockLoader.java|  20 ++
 .../datanode/fsdataset/impl/MemoryCacheStats.java  | 212 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  24 ++-
 .../datanode/TestFsDatasetCacheRevocation.java |  31 +--
 .../{ => fsdataset/impl}/TestFsDatasetCache.java   |  10 +-
 6 files changed, 298 insertions(+), 131 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
index 9efd11a..f90a4b1 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
@@ -52,7 +52,6 @@ import org.apache.hadoop.hdfs.DFSConfigKeys;
 import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
 import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
 import org.apache.hadoop.hdfs.server.datanode.DatanodeUtil;
-import org.apache.hadoop.io.nativeio.NativeIO;
 import org.apache.hadoop.util.Time;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -131,104 +130,10 @@ public class FsDatasetCache {
 
   private final long revocationPollingMs;
 
-  /**
-   * The approximate amount of cache space in use.
-   *
-   * This number is an overestimate, counting bytes that will be used only
-   * if pending caching operations succeed.  It does not take into account
-   * pending uncaching operations.
-   *
-   * This overestimate is more useful to the NameNode than an underestimate,
-   * since we don't want the NameNode to assign us more replicas than
-   * we can cache, because of the current batch of operations.
-   */
-  private final UsedBytesCount usedBytesCount;
-
-  public static class PageRounder {
-private final long osPageSize =
-NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize();
-
-/**
- * Round up a number to the operating system page size.
- */
-public long roundUp(long count) {
-  return (count + osPageSize - 1) & (~(osPageSize - 1));
-}
-
-/**
- * Round down a number to the operating system page size.
- */
-public long roundDown(long count) {
-  return count & (~(osPageSize - 1));
-}
-  }
-
-  private class UsedBytesCount {
-private final AtomicLong usedBytes = new AtomicLong(0);
-
-private final PageRounder rounder = new PageRounder();
-
-/**
- * Try to reserve more bytes.
- *
- * @param countThe number of bytes to add.  We will round this
- * up to the page size.
- *
- * @return The new number of usedBytes if we succeeded;
- * -1 if we failed.
- */
-long reserve(long count) {
-  count = rounder.roundUp(count);
-  while (true) {
-long cur = usedBytes.get();
-long next = cur + count;
-if (next > maxBytes) {
-  return -1;
-}
-if (usedBytes.compareAndSet(cur, next)) {
-  return next;
-}
-  }
-}
-
-/**
- * Release some bytes that we're using.
- *
- * @param countThe number of bytes to release.  We will round this
- * up to the page size.
- *
- * @return The new number of usedBytes.
- */
-long release(long count) {
-  count = rounder.roundUp(count);
-  return usedBytes.addAndGet(-count);
-}
-
-/**
- * Release some bytes that we're using rounded down to the page size.
- *
- * @param countThe number of bytes to release.  We will round this
- * down to the page size.
- *
- * @return The new number of usedBytes.
- */
-long releaseRoundDown(long count) {
-  count = rounder.roundDown(count);
-  return usedBytes.addAndGet(-count);
-}
-
-long get() {
-  return usedBytes.get();
-}
-  }
-
-  /**
-   * The total cache capacity in bytes.
-   */
-  private final long maxBytes;
-
   private final MappableBlockLoader mappableBlockLoader;
 
+  private final MemoryCacheStats memCacheStats;
+
   /**
* Number of cache commands that could not be completed successfully
*/
@@ -240,12 +145,10 @@ public class 

[hadoop] 10/10: HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit a6cdcf676e781f229654474927aa16583d34884e
Author: Rakesh Radhakrishnan 
AuthorDate: Sun Sep 22 22:02:54 2019 +0530

HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. 
Contributed by Feilong He.

(cherry picked from commit 659c88801d008bb352d10a1cb3bd0e401486cc9b)
---
 .../hadoop-common/src/CMakeLists.txt   |  2 +-
 .../org/apache/hadoop/io/nativeio/NativeIO.java| 28 +-
 .../apache/hadoop/util/NativeLibraryChecker.java   | 10 
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   | 14 ++-
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.c  | 28 +++---
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.h  |  5 
 .../datanode/fsdataset/impl/FsDatasetCache.java| 15 +---
 7 files changed, 72 insertions(+), 30 deletions(-)

diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt 
b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
index 771c685..10591f6 100644
--- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
+++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
@@ -170,7 +170,7 @@ if(REQUIRE_PMDK)
 set(CMAKE_FIND_LIBRARY_SUFFIXES ${STORED_CMAKE_FIND_LIBRARY_SUFFIXES})
 
 if(PMDK_LIBRARY)
-GET_FILENAME_COMPONENT(HADOOP_PMDK_LIBRARY ${PMDK_LIBRARY} NAME)
+GET_FILENAME_COMPONENT(HADOOP_PMDK_LIBRARY ${PMDK_LIBRARY} REALPATH)
 set(PMDK_SOURCE_FILES ${SRC}/io/nativeio/pmdk_load.c)
 else(PMDK_LIBRARY)
 MESSAGE(FATAL_ERROR "The required PMDK library is NOT found. 
PMDK_LIBRARY=${PMDK_LIBRARY}")
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
index 1d0eab7..973afa3 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
@@ -120,16 +120,19 @@ public class NativeIO {
   public String getMessage() {
 String msg;
 switch (stateCode) {
+// -1 represents UNSUPPORTED.
 case -1:
-  msg = "The native code is built without PMDK support.";
+  msg = "The native code was built without PMDK support.";
   break;
+// 1 represents PMDK_LIB_NOT_FOUND.
 case 1:
-  msg = "The native code is built with PMDK support, but PMDK libs " +
-  "are NOT found in execution environment or failed to be loaded.";
+  msg = "The native code was built with PMDK support, but PMDK libs " +
+  "were NOT found in execution environment or failed to be 
loaded.";
   break;
+// 0 represents SUPPORTED.
 case 0:
-  msg = "The native code is built with PMDK support, and PMDK libs " +
-  "are loaded successfully.";
+  msg = "The native code was built with PMDK support, and PMDK libs " +
+  "were loaded successfully.";
   break;
 default:
   msg = "The state code: " + stateCode + " is unrecognized!";
@@ -140,7 +143,7 @@ public class NativeIO {
 
 // Denotes the state of supporting PMDK. The value is set by JNI.
 private static SupportState pmdkSupportState =
-SupportState.PMDK_LIB_NOT_FOUND;
+SupportState.UNSUPPORTED;
 
 private static final Logger LOG = LoggerFactory.getLogger(NativeIO.class);
 
@@ -177,6 +180,14 @@ public class NativeIO {
   LOG.error("The state code: " + stateCode + " is unrecognized!");
 }
 
+public static String getPmdkSupportStateMessage() {
+  if (getPmdkLibPath() != null) {
+return pmdkSupportState.getMessage() +
+" The pmdk lib path: " + getPmdkLibPath();
+  }
+  return pmdkSupportState.getMessage();
+}
+
 public static boolean isPmdkAvailable() {
   LOG.info(pmdkSupportState.getMessage());
   return pmdkSupportState == SupportState.SUPPORTED;
@@ -242,8 +253,13 @@ public class NativeIO {
   NativeIO.POSIX.pmemSync(region.getAddress(), region.getLength());
 }
   }
+
+  public static String getPmdkLibPath() {
+return POSIX.getPmdkLibPath();
+  }
 }
 
+private static native String getPmdkLibPath();
 private static native boolean isPmemCheck(long address, long length);
 private static native PmemMappedRegion pmemCreateMapFile(String path,
 long length);
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java
 
b/hado

[hadoop] 06/10: HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 6eb5fb596ffa44e4c4fe67e960f396cb4b37c80f
Author: Sammi Chen 
AuthorDate: Wed Jun 5 21:33:00 2019 +0800

HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed 
by Feilong He.

(cherry picked from commit d1aad444907e1fc5314e8e64529e57c51ed7561c)
---
 BUILDING.txt   |  28 +++
 dev-support/bin/dist-copynativelibs|   8 +
 hadoop-common-project/hadoop-common/pom.xml|   2 +
 .../hadoop-common/src/CMakeLists.txt   |  21 ++
 .../hadoop-common/src/config.h.cmake   |   1 +
 .../org/apache/hadoop/io/nativeio/NativeIO.java| 135 ++-
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   | 252 +
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.c  | 106 +
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.h  |  95 
 .../apache/hadoop/io/nativeio/TestNativeIO.java| 153 +
 .../datanode/fsdataset/impl/FsDatasetCache.java|  22 ++
 .../datanode/fsdataset/impl/FsDatasetImpl.java |   8 +
 .../datanode/fsdataset/impl/FsDatasetUtil.java |  22 ++
 .../datanode/fsdataset/impl/MappableBlock.java |   6 +
 .../fsdataset/impl/MappableBlockLoader.java|  11 +-
 .../fsdataset/impl/MappableBlockLoaderFactory.java |   4 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |   8 +-
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |   5 +
 ...der.java => NativePmemMappableBlockLoader.java} | 166 +++---
 ...MappedBlock.java => NativePmemMappedBlock.java} |  49 ++--
 .../fsdataset/impl/PmemMappableBlockLoader.java|  10 +-
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |   5 +
 22 files changed, 1009 insertions(+), 108 deletions(-)

diff --git a/BUILDING.txt b/BUILDING.txt
index 1b900c3..0698469 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -86,6 +86,8 @@ Optional packages:
   $ sudo apt-get install fuse libfuse-dev
 * ZStandard compression
 $ sudo apt-get install zstd
+* PMDK library for storage class memory(SCM) as HDFS cache backend
+  Please refer to http://pmem.io/ and https://github.com/pmem/pmdk
 
 
--
 Maven main modules:
@@ -262,6 +264,32 @@ Maven build goals:
invoke, run 'mvn dependency-check:aggregate'. Note that this plugin
requires maven 3.1.1 or greater.
 
+ PMDK library build options:
+
+   The Persistent Memory Development Kit (PMDK), formerly known as NVML, is a 
growing
+   collection of libraries which have been developed for various use cases, 
tuned,
+   validated to production quality, and thoroughly documented. These libraries 
are built
+   on the Direct Access (DAX) feature available in both Linux and Windows, 
which allows
+   applications directly load/store access to persistent memory by 
memory-mapping files
+   on a persistent memory aware file system.
+
+   It is currently an optional component, meaning that Hadoop can be built 
without
+   this dependency. Please Note the library is used via dynamic module. For 
getting
+   more details please refer to the official sites:
+   http://pmem.io/ and https://github.com/pmem/pmdk.
+
+  * -Drequire.pmdk is used to build the project with PMDK libraries forcibly. 
With this
+option provided, the build will fail if libpmem library is not found. If 
this option
+is not given, the build will generate a version of Hadoop with 
libhadoop.so.
+And storage class memory(SCM) backed HDFS cache is still supported without 
PMDK involved.
+Because PMDK can bring better caching write/read performance, it is 
recommended to build
+the project with this option if user plans to use SCM backed HDFS cache.
+  * -Dpmdk.lib is used to specify a nonstandard location for PMDK libraries if 
they are not
+under /usr/lib or /usr/lib64.
+  * -Dbundle.pmdk is used to copy the specified libpmem libraries into the 
distribution tar
+package. This option requires that -Dpmdk.lib is specified. With 
-Dbundle.pmdk provided,
+the build will fail if -Dpmdk.lib is not specified.
+
 
--
 Building components separately
 
diff --git a/dev-support/bin/dist-copynativelibs 
b/dev-support/bin/dist-copynativelibs
index 67d2edf..4a783f0 100755
--- a/dev-support/bin/dist-copynativelibs
+++ b/dev-support/bin/dist-copynativelibs
@@ -96,6 +96,12 @@ for i in "$@"; do
 --isalbundle=*)
   ISALBUNDLE=${i#*=}
 ;;
+--pmdklib=*)
+  PMDKLIB=${i#*=}
+;;
+--pmdkbundle=*)
+  PMDKBUNDLE=${i#*=}
+;;
 --opensslbinbundle=*)
   OPENSSLBINBUNDLE=${i#*=}
 ;;
@@ -153,6 +159,8 @@ if [[ -d "${LIB_DIR}" ]]; then
   bundle_native_lib "${OPENSSLLIBBUNDLE}" &

[hadoop] 04/10: HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit f3571c1f096e771060dae4d31d2d8508d42d04b9
Author: Rakesh Radhakrishnan 
AuthorDate: Wed May 8 17:20:21 2019 +0530

HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by 
Feilong He.
---
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |  14 --
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |  22 ---
 .../datanode/fsdataset/impl/FsDatasetCache.java|  31 ++--
 .../datanode/fsdataset/impl/FsDatasetImpl.java |   2 +
 .../fsdataset/impl/MappableBlockLoader.java|  19 +-
 .../fsdataset/impl/MappableBlockLoaderFactory.java |  47 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  21 +--
 .../fsdataset/impl/PmemMappableBlockLoader.java|  40 ++---
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  10 +-
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 197 +++--
 .../src/main/resources/hdfs-default.xml|  24 ---
 .../impl/TestCacheByPmemMappableBlockLoader.java   |  26 ++-
 .../fsdataset/impl/TestFsDatasetCache.java |   5 +-
 13 files changed, 256 insertions(+), 202 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index bad8352..f2df396 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -26,8 +26,6 @@ import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant;
-import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader;
-import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator;
 import org.apache.hadoop.hdfs.web.URLConnectionFactory;
@@ -392,22 +390,10 @@ public class DFSConfigKeys extends 
CommonConfigurationKeys {
   public static final String DFS_DATANODE_CACHE_REVOCATION_POLLING_MS = 
"dfs.datanode.cache.revocation.polling.ms";
   public static final long DFS_DATANODE_CACHE_REVOCATION_POLLING_MS_DEFAULT = 
500L;
 
-  // Currently, the available cache loaders are MemoryMappableBlockLoader,
-  // PmemMappableBlockLoader. MemoryMappableBlockLoader is the default cache
-  // loader to cache block replica to memory.
-  public static final String DFS_DATANODE_CACHE_LOADER_CLASS =
-  "dfs.datanode.cache.loader.class";
-  public static final Class
-  DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT =
-  MemoryMappableBlockLoader.class;
   // Multiple dirs separated by "," are acceptable.
   public static final String DFS_DATANODE_CACHE_PMEM_DIRS_KEY =
   "dfs.datanode.cache.pmem.dirs";
   public static final String DFS_DATANODE_CACHE_PMEM_DIRS_DEFAULT = "";
-  // The cache capacity of persistent memory
-  public static final String DFS_DATANODE_CACHE_PMEM_CAPACITY_KEY =
-  "dfs.datanode.cache.pmem.capacity";
-  public static final long DFS_DATANODE_CACHE_PMEM_CAPACITY_DEFAULT = 0L;
 
   public static final String 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_KEY = 
"dfs.namenode.datanode.registration.ip-hostname-check";
   public static final boolean 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_DEFAULT = true;
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
index 6ee8e92..139ad77 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
@@ -27,10 +27,6 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHO
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHOLD_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_KEY;
-import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_LOADER_CLASS;
-import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT;
-import static 
org.apache.hadoop.hdfs.

[hadoop] 07/10: HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 75c66bf044d21c5742cdfb2e8eaebb1ac95e736f
Author: Rakesh Radhakrishnan 
AuthorDate: Mon Jul 15 13:02:37 2019 +0530

HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He.

(cherry picked from commit e98adb00b7da8fa913b86ecf2049444b1d8617d4)
---
 .../{MemoryCacheStats.java => CacheStats.java} |  6 +--
 .../datanode/fsdataset/impl/FsDatasetCache.java| 54 ++
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  4 +-
 .../fsdataset/impl/MappableBlockLoader.java|  3 +-
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  8 ++--
 .../impl/NativePmemMappableBlockLoader.java|  5 +-
 .../fsdataset/impl/PmemMappableBlockLoader.java|  9 +++-
 .../impl/TestCacheByPmemMappableBlockLoader.java   | 14 +++---
 .../fsdataset/impl/TestFsDatasetCache.java |  2 +-
 9 files changed, 45 insertions(+), 60 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
similarity index 97%
rename from 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
rename to 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
index d276c27..f79b7c7 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
@@ -27,7 +27,7 @@ import com.google.common.annotations.VisibleForTesting;
 /**
  * Keeps statistics for the memory cache.
  */
-class MemoryCacheStats {
+class CacheStats {
 
   /**
* The approximate amount of cache space in use.
@@ -47,7 +47,7 @@ class MemoryCacheStats {
*/
   private final long maxBytes;
 
-  MemoryCacheStats(long maxBytes) {
+  CacheStats(long maxBytes) {
 this.usedBytesCount = new UsedBytesCount();
 this.maxBytes = maxBytes;
   }
@@ -81,7 +81,7 @@ class MemoryCacheStats {
   private class UsedBytesCount {
 private final AtomicLong usedBytes = new AtomicLong(0);
 
-private MemoryCacheStats.PageRounder rounder = new PageRounder();
+private CacheStats.PageRounder rounder = new PageRounder();
 
 /**
  * Try to reserve more bytes.
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
index 37e548e..1514927 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
@@ -23,7 +23,6 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION_POLLING_MS;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION_POLLING_MS_DEFAULT;
 
-import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 import com.google.common.util.concurrent.ThreadFactoryBuilder;
 
@@ -137,7 +136,7 @@ public class FsDatasetCache {
*/
   private final MappableBlockLoader cacheLoader;
 
-  private final MemoryCacheStats memCacheStats;
+  private final CacheStats memCacheStats;
 
   /**
* Number of cache commands that could not be completed successfully
@@ -178,30 +177,17 @@ public class FsDatasetCache {
   ".  Reconfigure this to " + minRevocationPollingMs);
 }
 this.revocationPollingMs = confRevocationPollingMs;
-// Both lazy writer and read cache are sharing this statistics.
-this.memCacheStats = new MemoryCacheStats(
-dataset.datanode.getDnConf().getMaxLockedMemory());
 
 this.cacheLoader = MappableBlockLoaderFactory.createCacheLoader(
 this.getDnConf());
-cacheLoader.initialize(this);
-  }
-
-  /**
-   * Check if pmem cache is enabled.
-   */
-  private boolean isPmemCacheEnabled() {
-return !cacheLoader.isTransientCache();
+// Both lazy writer and read cache are sharing this statistics.
+this.memCacheStats = cacheLoader.initialize(this.getDnConf());
   }
 
   DNConf getDnConf() {
 return this.dataset.datanode.getDnConf();
   }
 
-  MemoryCacheStats getMemCacheStats() {
-return memCacheStats;
-  }
-
   /**
* Get the cache path if the replica is cached in

[hadoop] 01/10: HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 3f6f095c066f5f72e1883a3927dbad8b6b5b3cbf
Author: Uma Maheswara Rao G 
AuthorDate: Thu Mar 14 22:21:08 2019 -0700

HDFS-14354: Refactor MappableBlock to align with the implementation of SCM 
cache. Contributed by Feilong He.

(cherry picked from commit ba50a36a3ead628c3d44d384f7ed4d2b3a55dd07)
---
 .../datanode/fsdataset/impl/FsDatasetCache.java|  15 +-
 .../datanode/fsdataset/impl/MappableBlock.java | 155 +
 .../fsdataset/impl/MappableBlockLoader.java|  80 +++
 ...leBlock.java => MemoryMappableBlockLoader.java} | 111 +--
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |  54 +++
 5 files changed, 184 insertions(+), 231 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
index 767b150..9efd11a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
@@ -227,6 +227,8 @@ public class FsDatasetCache {
*/
   private final long maxBytes;
 
+  private final MappableBlockLoader mappableBlockLoader;
+
   /**
* Number of cache commands that could not be completed successfully
*/
@@ -236,7 +238,7 @@ public class FsDatasetCache {
*/
   final AtomicLong numBlocksFailedToUncache = new AtomicLong(0);
 
-  public FsDatasetCache(FsDatasetImpl dataset) {
+  public FsDatasetCache(FsDatasetImpl dataset) throws IOException {
 this.dataset = dataset;
 this.maxBytes = dataset.datanode.getDnConf().getMaxLockedMemory();
 ThreadFactory workerFactory = new ThreadFactoryBuilder()
@@ -268,6 +270,7 @@ public class FsDatasetCache {
   ".  Reconfigure this to " + minRevocationPollingMs);
 }
 this.revocationPollingMs = confRevocationPollingMs;
+this.mappableBlockLoader = new MemoryMappableBlockLoader();
   }
 
   /**
@@ -461,14 +464,14 @@ public class FsDatasetCache {
   return;
 }
 try {
-  mappableBlock = MappableBlock.
-  load(length, blockIn, metaIn, blockFileName);
+  mappableBlock = mappableBlockLoader.load(length, blockIn, metaIn,
+  blockFileName, key);
 } catch (ChecksumException e) {
   // Exception message is bogus since this wasn't caused by a file read
   LOG.warn("Failed to cache " + key + ": checksum verification 
failed.");
   return;
 } catch (IOException e) {
-  LOG.warn("Failed to cache " + key, e);
+  LOG.warn("Failed to cache the block [key=" + key + "]!", e);
   return;
 }
 synchronized (FsDatasetCache.this) {
@@ -498,9 +501,7 @@ public class FsDatasetCache {
   }
   LOG.debug("Caching of {} was aborted.  We are now caching only {} "
   + "bytes in total.", key, usedBytesCount.get());
-  if (mappableBlock != null) {
-mappableBlock.close();
-  }
+  IOUtils.closeQuietly(mappableBlock);
   numBlocksFailedToCache.incrementAndGet();
 
   synchronized (FsDatasetCache.this) {
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
index 45aa364..0fff327 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
@@ -18,164 +18,21 @@
 
 package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl;
 
-import java.io.BufferedInputStream;
-import java.io.Closeable;
-import java.io.DataInputStream;
-import java.io.FileInputStream;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.MappedByteBuffer;
-import java.nio.channels.FileChannel;
-import java.nio.channels.FileChannel.MapMode;
-
-import org.apache.commons.io.IOUtils;
 import org.apache.hadoop.classification.InterfaceAudience;
 import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.fs.ChecksumException;
-import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader;
-import org.apache.hadoop.io.nativeio.NativeIO;
-import org.apache.hadoop.util.DataChecksum;
 
-import com.google.common.a

[hadoop] 08/10: HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit c6d59e6a4796b626fcf0046f9bc435f4f2217259
Author: Rakesh Radhakrishnan 
AuthorDate: Mon Jul 15 13:18:23 2019 +0530

HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed 
by Feilong He.

(cherry picked from commit 30a8f840f1572129fe7d02f8a784c47ab57ce89a)
---
 .../src/site/markdown/CentralizedCacheManagement.md| 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
index 7568949..8880ea5 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
@@ -32,6 +32,8 @@ Centralized cache management in HDFS has many significant 
advantages.
 
 4.  Centralized caching can improve overall cluster memory utilization. When 
relying on the OS buffer cache at each DataNode, repeated reads of a block will 
result in all *n* replicas of the block being pulled into buffer cache. With 
centralized cache management, a user can explicitly pin only *m* of the *n* 
replicas, saving *n-m* memory.
 
+5.  HDFS supports non-volatile storage class memory (SCM, also known as 
persistent memory) cache in Linux platform. User can enable either memory cache 
or SCM cache for a DataNode. Memory cache and SCM cache can coexist among 
DataNodes. In the current implementation, the cache data in SCM will be cleaned 
up when DataNode restarts. Persistent HDFS cache support on SCM will be 
considered in the future.
+
 Use Cases
 -
 
@@ -200,11 +202,21 @@ Configuration
 
 In order to lock block files into memory, the DataNode relies on native JNI 
code found in `libhadoop.so` or `hadoop.dll` on Windows. Be sure to [enable 
JNI](../hadoop-common/NativeLibraries.html) if you are using HDFS centralized 
cache management.
 
+Currently, there are two implementations for persistent memory cache. The 
default one is pure Java based implementation and the other is native 
implementation which leverages PMDK library to improve the performance of cache 
write and cache read.
+
+To enable PMDK based implementation, please follow the below steps.
+
+1. Install PMDK library. Please refer to the official site http://pmem.io/ for 
detailed information.
+
+2. Build Hadoop with PMDK support. Please refer to "PMDK library build 
options" section in `BUILDING.txt` in the source code.
+
+To verify that PMDK is correctly detected by Hadoop, run the `hadoop 
checknative` command.
+
 ### Configuration Properties
 
  Required
 
-Be sure to configure the following:
+Be sure to configure one of the following properties for DRAM cache or 
persistent memory cache. Please note that DRAM cache and persistent cache 
cannot coexist on a DataNode.
 
 *   dfs.datanode.max.locked.memory
 
@@ -212,6 +224,10 @@ Be sure to configure the following:
 
 This setting is shared with the [Lazy Persist Writes 
feature](./MemoryStorage.html). The Data Node will ensure that the combined 
memory used by Lazy Persist Writes and Centralized Cache Management does not 
exceed the amount configured in `dfs.datanode.max.locked.memory`.
 
+*   dfs.datanode.cache.pmem.dirs
+
+This property specifies the cache volume of persistent memory. For 
multiple volumes, they should be separated by “,”, e.g. “/mnt/pmem0, 
/mnt/pmem1”. The default value is empty. If this property is configured, the 
volume capacity will be detected. And there is no need to configure 
`dfs.datanode.max.locked.memory`.
+
  Optional
 
 The following properties are not required, but may be specified for tuning:


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.2 updated (54d3e21 -> a6cdcf6)

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a change to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git.


from 54d3e21  HDFS-14923. Remove dead code from HealthMonitor. Contributed 
by Fei Hui.
 new 3f6f095  HDFS-14354: Refactor MappableBlock to align with the 
implementation of SCM cache. Contributed by Feilong He.
 new 0c68241  HDFS-14393. Refactor FsDatasetCache for SCM cache 
implementation. Contributed by Rakesh R
 new 6bc73a9  HDFS-14355 : Implement HDFS cache on SCM by using pure java 
mapped byte buffer. Contributed by Feilong He.
 new f3571c1  HDFS-14401. Refine the implementation for HDFS cache on SCM. 
Contributed by Feilong He.
 new 0fe720d  HDFS-14402. Use FileChannel.transferTo() method for 
transferring block to SCM cache. Contributed by Feilong He.
 new 6eb5fb5  HDFS-14356. Implement HDFS cache on SCM with native PMDK 
libs. Contributed by Feilong He.
 new 75c66bf  HDFS-14458. Report pmem stats to namenode. Contributed by 
Feilong He.
 new c6d59e6  HDFS-14357. Update documentation for HDFS cache on SCM 
support. Contributed by Feilong He.
 new dc2fad4  HDFS-14700. Clean up pmem cache before setting pmem cache 
capacity. Contributed by Feilong He.
 new a6cdcf6  HDFS-14818. Check native pmdk lib by 'hadoop checknative' 
command. Contributed by Feilong He.

The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 BUILDING.txt   |  28 ++
 dev-support/bin/dist-copynativelibs|   8 +
 hadoop-common-project/hadoop-common/pom.xml|   2 +
 .../hadoop-common/src/CMakeLists.txt   |  21 ++
 .../hadoop-common/src/config.h.cmake   |   1 +
 .../org/apache/hadoop/io/nativeio/NativeIO.java| 151 +++-
 .../apache/hadoop/util/NativeLibraryChecker.java   |  10 +
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   | 264 ++
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.c  | 106 ++
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.h  |  90 +
 .../apache/hadoop/io/nativeio/TestNativeIO.java| 153 
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |   5 +
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |   9 +
 .../server/datanode/fsdataset/impl/CacheStats.java | 212 +++
 .../datanode/fsdataset/impl/FsDatasetCache.java| 231 ++--
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  34 +-
 .../datanode/fsdataset/impl/FsDatasetUtil.java |  49 +++
 .../datanode/fsdataset/impl/MappableBlock.java | 155 +---
 ...MappableBlock.java => MappableBlockLoader.java} | 162 +
 .../fsdataset/impl/MappableBlockLoaderFactory.java |  51 +++
 .../fsdataset/impl/MemoryMappableBlockLoader.java  | 125 +++
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |  59 +++
 .../impl/NativePmemMappableBlockLoader.java| 192 ++
 .../fsdataset/impl/NativePmemMappedBlock.java  |  85 +
 .../fsdataset/impl/PmemMappableBlockLoader.java| 149 
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  69 
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 398 +
 .../src/main/resources/hdfs-default.xml|  10 +
 .../site/markdown/CentralizedCacheManagement.md|  18 +-
 .../datanode/TestFsDatasetCacheRevocation.java |  31 +-
 .../impl/TestCacheByPmemMappableBlockLoader.java   | 325 +
 .../{ => fsdataset/impl}/TestFsDatasetCache.java   |  15 +-
 32 files changed, 2845 insertions(+), 373 deletions(-)
 create mode 100644 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c
 create mode 100644 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.h
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
 copy 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/{MappableBlock.java
 => MappableBlockLoader.java} (56%)
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoaderFactory.java
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappedBlock.java
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java
 create mode 100644 
hadoop-hdfs-project/

[hadoop] 05/10: HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 0fe720dc485908b58b1b4d1fc7edf2f65ab446af
Author: Rakesh Radhakrishnan 
AuthorDate: Sun May 26 14:30:11 2019 +0530

HDFS-14402. Use FileChannel.transferTo() method for transferring block to 
SCM cache. Contributed by Feilong He.

(cherry picked from commit 37900c5639f8ba8d41b9fedc3d41ee0fbda7d5db)
---
 .../fsdataset/impl/MappableBlockLoader.java|  59 +++
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  59 ---
 .../fsdataset/impl/PmemMappableBlockLoader.java| 110 +++--
 3 files changed, 75 insertions(+), 153 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
index 044e5c5..3ec8416 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
@@ -18,10 +18,16 @@
 
 package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl;
 
+import com.google.common.base.Preconditions;
+import org.apache.commons.io.IOUtils;
 import org.apache.hadoop.classification.InterfaceAudience;
 import org.apache.hadoop.classification.InterfaceStability;
 import org.apache.hadoop.hdfs.ExtendedBlockId;
+import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader;
+import org.apache.hadoop.util.DataChecksum;
 
+import java.io.BufferedInputStream;
+import java.io.DataInputStream;
 import java.io.FileInputStream;
 import java.io.IOException;
 import java.nio.ByteBuffer;
@@ -108,6 +114,59 @@ public abstract class MappableBlockLoader {
   }
 
   /**
+   * Verifies the block's checksum. This is an I/O intensive operation.
+   */
+  protected void verifyChecksum(long length, FileInputStream metaIn,
+FileChannel blockChannel, String blockFileName)
+  throws IOException {
+// Verify the checksum from the block's meta file
+// Get the DataChecksum from the meta file header
+BlockMetadataHeader header =
+BlockMetadataHeader.readHeader(new DataInputStream(
+new BufferedInputStream(metaIn, BlockMetadataHeader
+.getHeaderSize(;
+FileChannel metaChannel = null;
+try {
+  metaChannel = metaIn.getChannel();
+  if (metaChannel == null) {
+throw new IOException(
+"Block InputStream meta file has no FileChannel.");
+  }
+  DataChecksum checksum = header.getChecksum();
+  final int bytesPerChecksum = checksum.getBytesPerChecksum();
+  final int checksumSize = checksum.getChecksumSize();
+  final int numChunks = (8 * 1024 * 1024) / bytesPerChecksum;
+  ByteBuffer blockBuf = ByteBuffer.allocate(numChunks * bytesPerChecksum);
+  ByteBuffer checksumBuf = ByteBuffer.allocate(numChunks * checksumSize);
+  // Verify the checksum
+  int bytesVerified = 0;
+  while (bytesVerified < length) {
+Preconditions.checkState(bytesVerified % bytesPerChecksum == 0,
+"Unexpected partial chunk before EOF");
+assert bytesVerified % bytesPerChecksum == 0;
+int bytesRead = fillBuffer(blockChannel, blockBuf);
+if (bytesRead == -1) {
+  throw new IOException("checksum verification failed: premature EOF");
+}
+blockBuf.flip();
+// Number of read chunks, including partial chunk at end
+int chunks = (bytesRead + bytesPerChecksum - 1) / bytesPerChecksum;
+checksumBuf.limit(chunks * checksumSize);
+fillBuffer(metaChannel, checksumBuf);
+checksumBuf.flip();
+checksum.verifyChunkedSums(blockBuf, checksumBuf, blockFileName,
+bytesVerified);
+// Success
+bytesVerified += bytesRead;
+blockBuf.clear();
+checksumBuf.clear();
+  }
+} finally {
+  IOUtils.closeQuietly(metaChannel);
+}
+  }
+
+  /**
* Reads bytes into a buffer until EOF or the buffer's limit is reached.
*/
   protected int fillBuffer(FileChannel channel, ByteBuffer buf)
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
index 919835a..52d8d93 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/a

[hadoop] 03/10: HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 6bc73a983d272235dee9fbb69e27bf6440343c0d
Author: Uma Maheswara Rao G 
AuthorDate: Sat Mar 30 23:33:25 2019 -0700

HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte 
buffer. Contributed by Feilong He.
---
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |  19 ++
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |  31 ++
 .../datanode/fsdataset/impl/FsDatasetCache.java|  90 +-
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  20 +-
 .../datanode/fsdataset/impl/FsDatasetUtil.java |  27 ++
 .../fsdataset/impl/MappableBlockLoader.java|  41 ++-
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  45 ++-
 ...ockLoader.java => PmemMappableBlockLoader.java} | 132 ++---
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  70 +
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 306 +++
 .../src/main/resources/hdfs-default.xml|  34 +++
 .../impl/TestCacheByPmemMappableBlockLoader.java   | 329 +
 12 files changed, 1077 insertions(+), 67 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 8877101..bad8352 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -26,6 +26,8 @@ import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant;
+import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader;
+import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator;
 import org.apache.hadoop.hdfs.web.URLConnectionFactory;
@@ -390,6 +392,23 @@ public class DFSConfigKeys extends CommonConfigurationKeys 
{
   public static final String DFS_DATANODE_CACHE_REVOCATION_POLLING_MS = 
"dfs.datanode.cache.revocation.polling.ms";
   public static final long DFS_DATANODE_CACHE_REVOCATION_POLLING_MS_DEFAULT = 
500L;
 
+  // Currently, the available cache loaders are MemoryMappableBlockLoader,
+  // PmemMappableBlockLoader. MemoryMappableBlockLoader is the default cache
+  // loader to cache block replica to memory.
+  public static final String DFS_DATANODE_CACHE_LOADER_CLASS =
+  "dfs.datanode.cache.loader.class";
+  public static final Class
+  DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT =
+  MemoryMappableBlockLoader.class;
+  // Multiple dirs separated by "," are acceptable.
+  public static final String DFS_DATANODE_CACHE_PMEM_DIRS_KEY =
+  "dfs.datanode.cache.pmem.dirs";
+  public static final String DFS_DATANODE_CACHE_PMEM_DIRS_DEFAULT = "";
+  // The cache capacity of persistent memory
+  public static final String DFS_DATANODE_CACHE_PMEM_CAPACITY_KEY =
+  "dfs.datanode.cache.pmem.capacity";
+  public static final long DFS_DATANODE_CACHE_PMEM_CAPACITY_DEFAULT = 0L;
+
   public static final String 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_KEY = 
"dfs.namenode.datanode.registration.ip-hostname-check";
   public static final boolean 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_DEFAULT = true;
 
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
index 155b800..6ee8e92 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
@@ -27,6 +27,11 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHO
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHOLD_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_LOADER_CLASS;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT;
+import static 
org.apache.hadoop.hd

[hadoop] 09/10: HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit dc2fad4bd36944fc58d90c7618f125bdaac14d85
Author: Rakesh Radhakrishnan 
AuthorDate: Fri Aug 9 14:07:54 2019 +0530

HDFS-14700. Clean up pmem cache before setting pmem cache capacity. 
Contributed by Feilong He.

(cherry picked from commit f6fa865d6fcb0ef0a25a00615f16f383e5032373)
---
 .../datanode/fsdataset/impl/PmemVolumeManager.java| 19 ---
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
index 2d77f7a..969d18b 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
@@ -218,6 +218,8 @@ public final class PmemVolumeManager {
   try {
 File pmemDir = new File(volumes[n]);
 File realPmemDir = verifyIfValidPmemVolume(pmemDir);
+// Clean up the cache left before, if any.
+cleanup(realPmemDir);
 this.pmemVolumes.add(realPmemDir.getPath());
 long maxBytes;
 if (maxBytesPerPmem == -1) {
@@ -242,17 +244,20 @@ public final class PmemVolumeManager {
   throw new IOException(
   "At least one valid persistent memory volume is required!");
 }
-cleanup();
+  }
+
+  void cleanup(File realPmemDir) {
+try {
+  FileUtils.cleanDirectory(realPmemDir);
+} catch (IOException e) {
+  LOG.error("Failed to clean up " + realPmemDir.getPath(), e);
+}
   }
 
   void cleanup() {
 // Remove all files under the volume.
-for (String pmemDir: pmemVolumes) {
-  try {
-FileUtils.cleanDirectory(new File(pmemDir));
-  } catch (IOException e) {
-LOG.error("Failed to clean up " + pmemDir, e);
-  }
+for (String pmemVolume : pmemVolumes) {
+  cleanup(new File(pmemVolume));
 }
   }
 


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] 05/10: HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit f982f9662322dbe32e1d5a703f0fabc8720815e2
Author: Rakesh Radhakrishnan 
AuthorDate: Sun May 26 14:30:11 2019 +0530

HDFS-14402. Use FileChannel.transferTo() method for transferring block to 
SCM cache. Contributed by Feilong He.

(cherry picked from commit 37900c5639f8ba8d41b9fedc3d41ee0fbda7d5db)
---
 .../fsdataset/impl/MappableBlockLoader.java|  59 +++
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  59 ---
 .../fsdataset/impl/PmemMappableBlockLoader.java| 110 +++--
 3 files changed, 75 insertions(+), 153 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
index 044e5c5..3ec8416 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
@@ -18,10 +18,16 @@
 
 package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl;
 
+import com.google.common.base.Preconditions;
+import org.apache.commons.io.IOUtils;
 import org.apache.hadoop.classification.InterfaceAudience;
 import org.apache.hadoop.classification.InterfaceStability;
 import org.apache.hadoop.hdfs.ExtendedBlockId;
+import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader;
+import org.apache.hadoop.util.DataChecksum;
 
+import java.io.BufferedInputStream;
+import java.io.DataInputStream;
 import java.io.FileInputStream;
 import java.io.IOException;
 import java.nio.ByteBuffer;
@@ -108,6 +114,59 @@ public abstract class MappableBlockLoader {
   }
 
   /**
+   * Verifies the block's checksum. This is an I/O intensive operation.
+   */
+  protected void verifyChecksum(long length, FileInputStream metaIn,
+FileChannel blockChannel, String blockFileName)
+  throws IOException {
+// Verify the checksum from the block's meta file
+// Get the DataChecksum from the meta file header
+BlockMetadataHeader header =
+BlockMetadataHeader.readHeader(new DataInputStream(
+new BufferedInputStream(metaIn, BlockMetadataHeader
+.getHeaderSize(;
+FileChannel metaChannel = null;
+try {
+  metaChannel = metaIn.getChannel();
+  if (metaChannel == null) {
+throw new IOException(
+"Block InputStream meta file has no FileChannel.");
+  }
+  DataChecksum checksum = header.getChecksum();
+  final int bytesPerChecksum = checksum.getBytesPerChecksum();
+  final int checksumSize = checksum.getChecksumSize();
+  final int numChunks = (8 * 1024 * 1024) / bytesPerChecksum;
+  ByteBuffer blockBuf = ByteBuffer.allocate(numChunks * bytesPerChecksum);
+  ByteBuffer checksumBuf = ByteBuffer.allocate(numChunks * checksumSize);
+  // Verify the checksum
+  int bytesVerified = 0;
+  while (bytesVerified < length) {
+Preconditions.checkState(bytesVerified % bytesPerChecksum == 0,
+"Unexpected partial chunk before EOF");
+assert bytesVerified % bytesPerChecksum == 0;
+int bytesRead = fillBuffer(blockChannel, blockBuf);
+if (bytesRead == -1) {
+  throw new IOException("checksum verification failed: premature EOF");
+}
+blockBuf.flip();
+// Number of read chunks, including partial chunk at end
+int chunks = (bytesRead + bytesPerChecksum - 1) / bytesPerChecksum;
+checksumBuf.limit(chunks * checksumSize);
+fillBuffer(metaChannel, checksumBuf);
+checksumBuf.flip();
+checksum.verifyChunkedSums(blockBuf, checksumBuf, blockFileName,
+bytesVerified);
+// Success
+bytesVerified += bytesRead;
+blockBuf.clear();
+checksumBuf.clear();
+  }
+} finally {
+  IOUtils.closeQuietly(metaChannel);
+}
+  }
+
+  /**
* Reads bytes into a buffer until EOF or the buffer's limit is reached.
*/
   protected int fillBuffer(FileChannel channel, ByteBuffer buf)
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
index 919835a..52d8d93 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/a

[hadoop] 04/10: HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 3b2a7aabf7af538fa337b8c155295a922f014b3e
Author: Rakesh Radhakrishnan 
AuthorDate: Wed May 8 17:20:21 2019 +0530

HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by 
Feilong He.
---
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |  14 --
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |  22 ---
 .../datanode/fsdataset/impl/FsDatasetCache.java|  31 ++--
 .../datanode/fsdataset/impl/FsDatasetImpl.java |   2 +
 .../fsdataset/impl/MappableBlockLoader.java|  19 +-
 .../fsdataset/impl/MappableBlockLoaderFactory.java |  47 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  21 +--
 .../fsdataset/impl/PmemMappableBlockLoader.java|  40 ++---
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  10 +-
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 197 +++--
 .../src/main/resources/hdfs-default.xml|  24 ---
 .../impl/TestCacheByPmemMappableBlockLoader.java   |  26 ++-
 .../fsdataset/impl/TestFsDatasetCache.java |   5 +-
 13 files changed, 256 insertions(+), 202 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 8ad1652..ce0c6d3 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -25,8 +25,6 @@ import org.apache.hadoop.hdfs.net.DFSNetworkTopology;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant;
-import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader;
-import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator;
 import org.apache.hadoop.hdfs.web.URLConnectionFactory;
@@ -382,22 +380,10 @@ public class DFSConfigKeys extends 
CommonConfigurationKeys {
   public static final String DFS_DATANODE_CACHE_REVOCATION_POLLING_MS = 
"dfs.datanode.cache.revocation.polling.ms";
   public static final long DFS_DATANODE_CACHE_REVOCATION_POLLING_MS_DEFAULT = 
500L;
 
-  // Currently, the available cache loaders are MemoryMappableBlockLoader,
-  // PmemMappableBlockLoader. MemoryMappableBlockLoader is the default cache
-  // loader to cache block replica to memory.
-  public static final String DFS_DATANODE_CACHE_LOADER_CLASS =
-  "dfs.datanode.cache.loader.class";
-  public static final Class
-  DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT =
-  MemoryMappableBlockLoader.class;
   // Multiple dirs separated by "," are acceptable.
   public static final String DFS_DATANODE_CACHE_PMEM_DIRS_KEY =
   "dfs.datanode.cache.pmem.dirs";
   public static final String DFS_DATANODE_CACHE_PMEM_DIRS_DEFAULT = "";
-  // The cache capacity of persistent memory
-  public static final String DFS_DATANODE_CACHE_PMEM_CAPACITY_KEY =
-  "dfs.datanode.cache.pmem.capacity";
-  public static final long DFS_DATANODE_CACHE_PMEM_CAPACITY_DEFAULT = 0L;
 
   public static final String 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_KEY = 
"dfs.namenode.datanode.registration.ip-hostname-check";
   public static final boolean 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_DEFAULT = true;
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
index 6ee8e92..139ad77 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
@@ -27,10 +27,6 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHO
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHOLD_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_KEY;
-import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_LOADER_CLASS;
-import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT;
-import static 
org.apache.hadoop.hdfs.

[hadoop] 06/10: HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 9281b725503369fb5eb73350db4773a41e7dac0c
Author: Sammi Chen 
AuthorDate: Wed Jun 5 21:33:00 2019 +0800

HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed 
by Feilong He.

(cherry picked from commit d1aad444907e1fc5314e8e64529e57c51ed7561c)
---
 BUILDING.txt   |  28 +++
 dev-support/bin/dist-copynativelibs|   8 +
 hadoop-common-project/hadoop-common/pom.xml|   2 +
 .../hadoop-common/src/CMakeLists.txt   |  21 ++
 .../hadoop-common/src/config.h.cmake   |   1 +
 .../org/apache/hadoop/io/nativeio/NativeIO.java| 135 ++-
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   | 252 +
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.c  | 106 +
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.h  |  95 
 .../apache/hadoop/io/nativeio/TestNativeIO.java| 153 +
 .../datanode/fsdataset/impl/FsDatasetCache.java|  22 ++
 .../datanode/fsdataset/impl/FsDatasetImpl.java |   8 +
 .../datanode/fsdataset/impl/FsDatasetUtil.java |  22 ++
 .../datanode/fsdataset/impl/MappableBlock.java |   6 +
 .../fsdataset/impl/MappableBlockLoader.java|  11 +-
 .../fsdataset/impl/MappableBlockLoaderFactory.java |   4 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |   8 +-
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |   5 +
 ...der.java => NativePmemMappableBlockLoader.java} | 166 +++---
 ...MappedBlock.java => NativePmemMappedBlock.java} |  49 ++--
 .../fsdataset/impl/PmemMappableBlockLoader.java|  10 +-
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |   5 +
 22 files changed, 1009 insertions(+), 108 deletions(-)

diff --git a/BUILDING.txt b/BUILDING.txt
index 247d2fa..a18834b 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -82,6 +82,8 @@ Optional packages:
   $ sudo apt-get install fuse libfuse-dev
 * ZStandard compression
 $ sudo apt-get install zstd
+* PMDK library for storage class memory(SCM) as HDFS cache backend
+  Please refer to http://pmem.io/ and https://github.com/pmem/pmdk
 
 
--
 Maven main modules:
@@ -246,6 +248,32 @@ Maven build goals:
invoke, run 'mvn dependency-check:aggregate'. Note that this plugin
requires maven 3.1.1 or greater.
 
+ PMDK library build options:
+
+   The Persistent Memory Development Kit (PMDK), formerly known as NVML, is a 
growing
+   collection of libraries which have been developed for various use cases, 
tuned,
+   validated to production quality, and thoroughly documented. These libraries 
are built
+   on the Direct Access (DAX) feature available in both Linux and Windows, 
which allows
+   applications directly load/store access to persistent memory by 
memory-mapping files
+   on a persistent memory aware file system.
+
+   It is currently an optional component, meaning that Hadoop can be built 
without
+   this dependency. Please Note the library is used via dynamic module. For 
getting
+   more details please refer to the official sites:
+   http://pmem.io/ and https://github.com/pmem/pmdk.
+
+  * -Drequire.pmdk is used to build the project with PMDK libraries forcibly. 
With this
+option provided, the build will fail if libpmem library is not found. If 
this option
+is not given, the build will generate a version of Hadoop with 
libhadoop.so.
+And storage class memory(SCM) backed HDFS cache is still supported without 
PMDK involved.
+Because PMDK can bring better caching write/read performance, it is 
recommended to build
+the project with this option if user plans to use SCM backed HDFS cache.
+  * -Dpmdk.lib is used to specify a nonstandard location for PMDK libraries if 
they are not
+under /usr/lib or /usr/lib64.
+  * -Dbundle.pmdk is used to copy the specified libpmem libraries into the 
distribution tar
+package. This option requires that -Dpmdk.lib is specified. With 
-Dbundle.pmdk provided,
+the build will fail if -Dpmdk.lib is not specified.
+
 
--
 Building components separately
 
diff --git a/dev-support/bin/dist-copynativelibs 
b/dev-support/bin/dist-copynativelibs
index 67d2edf..4a783f0 100755
--- a/dev-support/bin/dist-copynativelibs
+++ b/dev-support/bin/dist-copynativelibs
@@ -96,6 +96,12 @@ for i in "$@"; do
 --isalbundle=*)
   ISALBUNDLE=${i#*=}
 ;;
+--pmdklib=*)
+  PMDKLIB=${i#*=}
+;;
+--pmdkbundle=*)
+  PMDKBUNDLE=${i#*=}
+;;
 --opensslbinbundle=*)
   OPENSSLBINBUNDLE=${i#*=}
 ;;
@@ -153,6 +159,8 @@ if [[ -d "${LIB_DIR}" ]]; then
   bundle_native_lib "${OPENSSLLIBBUNDLE}" &

[hadoop] 03/10: HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit b00f9d89e42c5581b8a540ffe14ae56c08e3fa12
Author: Uma Maheswara Rao G 
AuthorDate: Sat Mar 30 23:33:25 2019 -0700

HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte 
buffer. Contributed by Feilong He.
---
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |  19 ++
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |  31 ++
 .../datanode/fsdataset/impl/FsDatasetCache.java|  90 +-
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  20 +-
 .../datanode/fsdataset/impl/FsDatasetUtil.java |  27 ++
 .../fsdataset/impl/MappableBlockLoader.java|  41 ++-
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  45 ++-
 ...ockLoader.java => PmemMappableBlockLoader.java} | 132 ++---
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  70 +
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 306 +++
 .../src/main/resources/hdfs-default.xml|  34 +++
 .../impl/TestCacheByPmemMappableBlockLoader.java   | 329 +
 12 files changed, 1077 insertions(+), 67 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 02ce2f4..8ad1652 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -25,6 +25,8 @@ import org.apache.hadoop.hdfs.net.DFSNetworkTopology;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant;
+import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader;
+import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator;
 import org.apache.hadoop.hdfs.web.URLConnectionFactory;
@@ -380,6 +382,23 @@ public class DFSConfigKeys extends CommonConfigurationKeys 
{
   public static final String DFS_DATANODE_CACHE_REVOCATION_POLLING_MS = 
"dfs.datanode.cache.revocation.polling.ms";
   public static final long DFS_DATANODE_CACHE_REVOCATION_POLLING_MS_DEFAULT = 
500L;
 
+  // Currently, the available cache loaders are MemoryMappableBlockLoader,
+  // PmemMappableBlockLoader. MemoryMappableBlockLoader is the default cache
+  // loader to cache block replica to memory.
+  public static final String DFS_DATANODE_CACHE_LOADER_CLASS =
+  "dfs.datanode.cache.loader.class";
+  public static final Class
+  DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT =
+  MemoryMappableBlockLoader.class;
+  // Multiple dirs separated by "," are acceptable.
+  public static final String DFS_DATANODE_CACHE_PMEM_DIRS_KEY =
+  "dfs.datanode.cache.pmem.dirs";
+  public static final String DFS_DATANODE_CACHE_PMEM_DIRS_DEFAULT = "";
+  // The cache capacity of persistent memory
+  public static final String DFS_DATANODE_CACHE_PMEM_CAPACITY_KEY =
+  "dfs.datanode.cache.pmem.capacity";
+  public static final long DFS_DATANODE_CACHE_PMEM_CAPACITY_DEFAULT = 0L;
+
   public static final String 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_KEY = 
"dfs.namenode.datanode.registration.ip-hostname-check";
   public static final boolean 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_DEFAULT = true;
 
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
index 155b800..6ee8e92 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
@@ -27,6 +27,11 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHO
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHOLD_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_LOADER_CLASS;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT;
+import static 
org.apache.hadoop.hd

[hadoop] 09/10: HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit cd1277398a1373a8bd6fa5f700684cdc6bd6a947
Author: Rakesh Radhakrishnan 
AuthorDate: Fri Aug 9 14:07:54 2019 +0530

HDFS-14700. Clean up pmem cache before setting pmem cache capacity. 
Contributed by Feilong He.

(cherry picked from commit f6fa865d6fcb0ef0a25a00615f16f383e5032373)
---
 .../datanode/fsdataset/impl/PmemVolumeManager.java| 19 ---
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
index 2d77f7a..969d18b 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
@@ -218,6 +218,8 @@ public final class PmemVolumeManager {
   try {
 File pmemDir = new File(volumes[n]);
 File realPmemDir = verifyIfValidPmemVolume(pmemDir);
+// Clean up the cache left before, if any.
+cleanup(realPmemDir);
 this.pmemVolumes.add(realPmemDir.getPath());
 long maxBytes;
 if (maxBytesPerPmem == -1) {
@@ -242,17 +244,20 @@ public final class PmemVolumeManager {
   throw new IOException(
   "At least one valid persistent memory volume is required!");
 }
-cleanup();
+  }
+
+  void cleanup(File realPmemDir) {
+try {
+  FileUtils.cleanDirectory(realPmemDir);
+} catch (IOException e) {
+  LOG.error("Failed to clean up " + realPmemDir.getPath(), e);
+}
   }
 
   void cleanup() {
 // Remove all files under the volume.
-for (String pmemDir: pmemVolumes) {
-  try {
-FileUtils.cleanDirectory(new File(pmemDir));
-  } catch (IOException e) {
-LOG.error("Failed to clean up " + pmemDir, e);
-  }
+for (String pmemVolume : pmemVolumes) {
+  cleanup(new File(pmemVolume));
 }
   }
 


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] 01/10: HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit aab9fefddcf5f48d0e7849d0d295f0ff0ffe20f8
Author: Uma Maheswara Rao G 
AuthorDate: Thu Mar 14 22:21:08 2019 -0700

HDFS-14354: Refactor MappableBlock to align with the implementation of SCM 
cache. Contributed by Feilong He.

(cherry picked from commit ba50a36a3ead628c3d44d384f7ed4d2b3a55dd07)
---
 .../datanode/fsdataset/impl/FsDatasetCache.java|  15 +-
 .../datanode/fsdataset/impl/MappableBlock.java | 155 +
 .../fsdataset/impl/MappableBlockLoader.java|  80 +++
 ...leBlock.java => MemoryMappableBlockLoader.java} | 111 +--
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |  54 +++
 5 files changed, 184 insertions(+), 231 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
index f70d4af..b7795f6 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
@@ -227,6 +227,8 @@ public class FsDatasetCache {
*/
   private final long maxBytes;
 
+  private final MappableBlockLoader mappableBlockLoader;
+
   /**
* Number of cache commands that could not be completed successfully
*/
@@ -236,7 +238,7 @@ public class FsDatasetCache {
*/
   final AtomicLong numBlocksFailedToUncache = new AtomicLong(0);
 
-  public FsDatasetCache(FsDatasetImpl dataset) {
+  public FsDatasetCache(FsDatasetImpl dataset) throws IOException {
 this.dataset = dataset;
 this.maxBytes = dataset.datanode.getDnConf().getMaxLockedMemory();
 ThreadFactory workerFactory = new ThreadFactoryBuilder()
@@ -268,6 +270,7 @@ public class FsDatasetCache {
   ".  Reconfigure this to " + minRevocationPollingMs);
 }
 this.revocationPollingMs = confRevocationPollingMs;
+this.mappableBlockLoader = new MemoryMappableBlockLoader();
   }
 
   /**
@@ -461,14 +464,14 @@ public class FsDatasetCache {
   return;
 }
 try {
-  mappableBlock = MappableBlock.
-  load(length, blockIn, metaIn, blockFileName);
+  mappableBlock = mappableBlockLoader.load(length, blockIn, metaIn,
+  blockFileName, key);
 } catch (ChecksumException e) {
   // Exception message is bogus since this wasn't caused by a file read
   LOG.warn("Failed to cache " + key + ": checksum verification 
failed.");
   return;
 } catch (IOException e) {
-  LOG.warn("Failed to cache " + key, e);
+  LOG.warn("Failed to cache the block [key=" + key + "]!", e);
   return;
 }
 synchronized (FsDatasetCache.this) {
@@ -498,9 +501,7 @@ public class FsDatasetCache {
   }
   LOG.debug("Caching of {} was aborted.  We are now caching only {} "
   + "bytes in total.", key, usedBytesCount.get());
-  if (mappableBlock != null) {
-mappableBlock.close();
-  }
+  IOUtils.closeQuietly(mappableBlock);
   numBlocksFailedToCache.incrementAndGet();
 
   synchronized (FsDatasetCache.this) {
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
index 45aa364..0fff327 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
@@ -18,164 +18,21 @@
 
 package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl;
 
-import java.io.BufferedInputStream;
-import java.io.Closeable;
-import java.io.DataInputStream;
-import java.io.FileInputStream;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.MappedByteBuffer;
-import java.nio.channels.FileChannel;
-import java.nio.channels.FileChannel.MapMode;
-
-import org.apache.commons.io.IOUtils;
 import org.apache.hadoop.classification.InterfaceAudience;
 import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.fs.ChecksumException;
-import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader;
-import org.apache.hadoop.io.nativeio.NativeIO;
-import org.apache.hadoop.util.DataChecksum;
 
-import com.google.common.a

[hadoop] 07/10: HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit df0dcc7493792a6297707008e8a6086596571da7
Author: Rakesh Radhakrishnan 
AuthorDate: Mon Jul 15 13:02:37 2019 +0530

HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He.

(cherry picked from commit e98adb00b7da8fa913b86ecf2049444b1d8617d4)
---
 .../{MemoryCacheStats.java => CacheStats.java} |  6 +--
 .../datanode/fsdataset/impl/FsDatasetCache.java| 54 ++
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  4 +-
 .../fsdataset/impl/MappableBlockLoader.java|  3 +-
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  8 ++--
 .../impl/NativePmemMappableBlockLoader.java|  5 +-
 .../fsdataset/impl/PmemMappableBlockLoader.java|  9 +++-
 .../impl/TestCacheByPmemMappableBlockLoader.java   | 14 +++---
 .../fsdataset/impl/TestFsDatasetCache.java |  2 +-
 9 files changed, 45 insertions(+), 60 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
similarity index 97%
rename from 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
rename to 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
index d276c27..f79b7c7 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
@@ -27,7 +27,7 @@ import com.google.common.annotations.VisibleForTesting;
 /**
  * Keeps statistics for the memory cache.
  */
-class MemoryCacheStats {
+class CacheStats {
 
   /**
* The approximate amount of cache space in use.
@@ -47,7 +47,7 @@ class MemoryCacheStats {
*/
   private final long maxBytes;
 
-  MemoryCacheStats(long maxBytes) {
+  CacheStats(long maxBytes) {
 this.usedBytesCount = new UsedBytesCount();
 this.maxBytes = maxBytes;
   }
@@ -81,7 +81,7 @@ class MemoryCacheStats {
   private class UsedBytesCount {
 private final AtomicLong usedBytes = new AtomicLong(0);
 
-private MemoryCacheStats.PageRounder rounder = new PageRounder();
+private CacheStats.PageRounder rounder = new PageRounder();
 
 /**
  * Try to reserve more bytes.
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
index f72b33d..2e85eaa 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
@@ -23,7 +23,6 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION_POLLING_MS;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION_POLLING_MS_DEFAULT;
 
-import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 import com.google.common.util.concurrent.ThreadFactoryBuilder;
 
@@ -137,7 +136,7 @@ public class FsDatasetCache {
*/
   private final MappableBlockLoader cacheLoader;
 
-  private final MemoryCacheStats memCacheStats;
+  private final CacheStats memCacheStats;
 
   /**
* Number of cache commands that could not be completed successfully
@@ -178,30 +177,17 @@ public class FsDatasetCache {
   ".  Reconfigure this to " + minRevocationPollingMs);
 }
 this.revocationPollingMs = confRevocationPollingMs;
-// Both lazy writer and read cache are sharing this statistics.
-this.memCacheStats = new MemoryCacheStats(
-dataset.datanode.getDnConf().getMaxLockedMemory());
 
 this.cacheLoader = MappableBlockLoaderFactory.createCacheLoader(
 this.getDnConf());
-cacheLoader.initialize(this);
-  }
-
-  /**
-   * Check if pmem cache is enabled.
-   */
-  private boolean isPmemCacheEnabled() {
-return !cacheLoader.isTransientCache();
+// Both lazy writer and read cache are sharing this statistics.
+this.memCacheStats = cacheLoader.initialize(this.getDnConf());
   }
 
   DNConf getDnConf() {
 return this.dataset.datanode.getDnConf();
   }
 
-  MemoryCacheStats getMemCacheStats() {
-return memCacheStats;
-  }
-
   /**
* Get the cache path if the replica is cached in

[hadoop] 10/10: HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 2db0b8f6ca590452159472250dd6d082c87a0911
Author: Rakesh Radhakrishnan 
AuthorDate: Sun Sep 22 22:02:54 2019 +0530

HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. 
Contributed by Feilong He.

(cherry picked from commit 659c88801d008bb352d10a1cb3bd0e401486cc9b)
---
 .../hadoop-common/src/CMakeLists.txt   |  2 +-
 .../org/apache/hadoop/io/nativeio/NativeIO.java| 28 +-
 .../apache/hadoop/util/NativeLibraryChecker.java   | 10 
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   | 14 ++-
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.c  | 28 +++---
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.h  |  5 
 .../datanode/fsdataset/impl/FsDatasetCache.java| 15 +---
 7 files changed, 72 insertions(+), 30 deletions(-)

diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt 
b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
index 771c685..10591f6 100644
--- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
+++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
@@ -170,7 +170,7 @@ if(REQUIRE_PMDK)
 set(CMAKE_FIND_LIBRARY_SUFFIXES ${STORED_CMAKE_FIND_LIBRARY_SUFFIXES})
 
 if(PMDK_LIBRARY)
-GET_FILENAME_COMPONENT(HADOOP_PMDK_LIBRARY ${PMDK_LIBRARY} NAME)
+GET_FILENAME_COMPONENT(HADOOP_PMDK_LIBRARY ${PMDK_LIBRARY} REALPATH)
 set(PMDK_SOURCE_FILES ${SRC}/io/nativeio/pmdk_load.c)
 else(PMDK_LIBRARY)
 MESSAGE(FATAL_ERROR "The required PMDK library is NOT found. 
PMDK_LIBRARY=${PMDK_LIBRARY}")
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
index e18d70e..f2b3456 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
@@ -119,16 +119,19 @@ public class NativeIO {
   public String getMessage() {
 String msg;
 switch (stateCode) {
+// -1 represents UNSUPPORTED.
 case -1:
-  msg = "The native code is built without PMDK support.";
+  msg = "The native code was built without PMDK support.";
   break;
+// 1 represents PMDK_LIB_NOT_FOUND.
 case 1:
-  msg = "The native code is built with PMDK support, but PMDK libs " +
-  "are NOT found in execution environment or failed to be loaded.";
+  msg = "The native code was built with PMDK support, but PMDK libs " +
+  "were NOT found in execution environment or failed to be 
loaded.";
   break;
+// 0 represents SUPPORTED.
 case 0:
-  msg = "The native code is built with PMDK support, and PMDK libs " +
-  "are loaded successfully.";
+  msg = "The native code was built with PMDK support, and PMDK libs " +
+  "were loaded successfully.";
   break;
 default:
   msg = "The state code: " + stateCode + " is unrecognized!";
@@ -139,7 +142,7 @@ public class NativeIO {
 
 // Denotes the state of supporting PMDK. The value is set by JNI.
 private static SupportState pmdkSupportState =
-SupportState.PMDK_LIB_NOT_FOUND;
+SupportState.UNSUPPORTED;
 
 private static final Logger LOG = LoggerFactory.getLogger(NativeIO.class);
 
@@ -176,6 +179,14 @@ public class NativeIO {
   LOG.error("The state code: " + stateCode + " is unrecognized!");
 }
 
+public static String getPmdkSupportStateMessage() {
+  if (getPmdkLibPath() != null) {
+return pmdkSupportState.getMessage() +
+" The pmdk lib path: " + getPmdkLibPath();
+  }
+  return pmdkSupportState.getMessage();
+}
+
 public static boolean isPmdkAvailable() {
   LOG.info(pmdkSupportState.getMessage());
   return pmdkSupportState == SupportState.SUPPORTED;
@@ -241,8 +252,13 @@ public class NativeIO {
   NativeIO.POSIX.pmemSync(region.getAddress(), region.getLength());
 }
   }
+
+  public static String getPmdkLibPath() {
+return POSIX.getPmdkLibPath();
+  }
 }
 
+private static native String getPmdkLibPath();
 private static native boolean isPmemCheck(long address, long length);
 private static native PmemMappedRegion pmemCreateMapFile(String path,
 long length);
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java
 
b/hado

[hadoop] 08/10: HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit f2563dcca100098a7e121d6d0a75907eaf87fa75
Author: Rakesh Radhakrishnan 
AuthorDate: Mon Jul 15 13:18:23 2019 +0530

HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed 
by Feilong He.

(cherry picked from commit 30a8f840f1572129fe7d02f8a784c47ab57ce89a)
---
 .../src/site/markdown/CentralizedCacheManagement.md| 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
index f2de043..85cc242 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
@@ -32,6 +32,8 @@ Centralized cache management in HDFS has many significant 
advantages.
 
 4.  Centralized caching can improve overall cluster memory utilization. When 
relying on the OS buffer cache at each DataNode, repeated reads of a block will 
result in all *n* replicas of the block being pulled into buffer cache. With 
centralized cache management, a user can explicitly pin only *m* of the *n* 
replicas, saving *n-m* memory.
 
+5.  HDFS supports non-volatile storage class memory (SCM, also known as 
persistent memory) cache in Linux platform. User can enable either memory cache 
or SCM cache for a DataNode. Memory cache and SCM cache can coexist among 
DataNodes. In the current implementation, the cache data in SCM will be cleaned 
up when DataNode restarts. Persistent HDFS cache support on SCM will be 
considered in the future.
+
 Use Cases
 -
 
@@ -200,11 +202,21 @@ Configuration
 
 In order to lock block files into memory, the DataNode relies on native JNI 
code found in `libhadoop.so` or `hadoop.dll` on Windows. Be sure to [enable 
JNI](../hadoop-common/NativeLibraries.html) if you are using HDFS centralized 
cache management.
 
+Currently, there are two implementations for persistent memory cache. The 
default one is pure Java based implementation and the other is native 
implementation which leverages PMDK library to improve the performance of cache 
write and cache read.
+
+To enable PMDK based implementation, please follow the below steps.
+
+1. Install PMDK library. Please refer to the official site http://pmem.io/ for 
detailed information.
+
+2. Build Hadoop with PMDK support. Please refer to "PMDK library build 
options" section in `BUILDING.txt` in the source code.
+
+To verify that PMDK is correctly detected by Hadoop, run the `hadoop 
checknative` command.
+
 ### Configuration Properties
 
  Required
 
-Be sure to configure the following:
+Be sure to configure one of the following properties for DRAM cache or 
persistent memory cache. Please note that DRAM cache and persistent cache 
cannot coexist on a DataNode.
 
 *   dfs.datanode.max.locked.memory
 
@@ -212,6 +224,10 @@ Be sure to configure the following:
 
 This setting is shared with the [Lazy Persist Writes 
feature](./MemoryStorage.html). The Data Node will ensure that the combined 
memory used by Lazy Persist Writes and Centralized Cache Management does not 
exceed the amount configured in `dfs.datanode.max.locked.memory`.
 
+*   dfs.datanode.cache.pmem.dirs
+
+This property specifies the cache volume of persistent memory. For 
multiple volumes, they should be separated by “,”, e.g. “/mnt/pmem0, 
/mnt/pmem1”. The default value is empty. If this property is configured, the 
volume capacity will be detected. And there is no need to configure 
`dfs.datanode.max.locked.memory`.
+
  Optional
 
 The following properties are not required, but may be specified for tuning:


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] 02/10: HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git

commit 712749c1a00eb3e7ed5a3511320c2297abc10e11
Author: Rakesh Radhakrishnan 
AuthorDate: Fri Mar 29 00:18:15 2019 +0530

HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. 
Contributed by Rakesh R

(cherry picked from commit f3f51284d57ef2e0c7e968b6eea56eab578f7e93)
---
 .../datanode/fsdataset/impl/FsDatasetCache.java| 132 ++---
 .../fsdataset/impl/MappableBlockLoader.java|  20 ++
 .../datanode/fsdataset/impl/MemoryCacheStats.java  | 212 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  24 ++-
 .../datanode/TestFsDatasetCacheRevocation.java |  31 +--
 .../{ => fsdataset/impl}/TestFsDatasetCache.java   |  10 +-
 6 files changed, 298 insertions(+), 131 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
index b7795f6..f2abada 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
@@ -52,7 +52,6 @@ import org.apache.hadoop.hdfs.DFSConfigKeys;
 import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
 import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
 import org.apache.hadoop.hdfs.server.datanode.DatanodeUtil;
-import org.apache.hadoop.io.nativeio.NativeIO;
 import org.apache.hadoop.util.Time;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -131,104 +130,10 @@ public class FsDatasetCache {
 
   private final long revocationPollingMs;
 
-  /**
-   * The approximate amount of cache space in use.
-   *
-   * This number is an overestimate, counting bytes that will be used only
-   * if pending caching operations succeed.  It does not take into account
-   * pending uncaching operations.
-   *
-   * This overestimate is more useful to the NameNode than an underestimate,
-   * since we don't want the NameNode to assign us more replicas than
-   * we can cache, because of the current batch of operations.
-   */
-  private final UsedBytesCount usedBytesCount;
-
-  public static class PageRounder {
-private final long osPageSize =
-NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize();
-
-/**
- * Round up a number to the operating system page size.
- */
-public long roundUp(long count) {
-  return (count + osPageSize - 1) & (~(osPageSize - 1));
-}
-
-/**
- * Round down a number to the operating system page size.
- */
-public long roundDown(long count) {
-  return count & (~(osPageSize - 1));
-}
-  }
-
-  private class UsedBytesCount {
-private final AtomicLong usedBytes = new AtomicLong(0);
-
-private final PageRounder rounder = new PageRounder();
-
-/**
- * Try to reserve more bytes.
- *
- * @param countThe number of bytes to add.  We will round this
- * up to the page size.
- *
- * @return The new number of usedBytes if we succeeded;
- * -1 if we failed.
- */
-long reserve(long count) {
-  count = rounder.roundUp(count);
-  while (true) {
-long cur = usedBytes.get();
-long next = cur + count;
-if (next > maxBytes) {
-  return -1;
-}
-if (usedBytes.compareAndSet(cur, next)) {
-  return next;
-}
-  }
-}
-
-/**
- * Release some bytes that we're using.
- *
- * @param countThe number of bytes to release.  We will round this
- * up to the page size.
- *
- * @return The new number of usedBytes.
- */
-long release(long count) {
-  count = rounder.roundUp(count);
-  return usedBytes.addAndGet(-count);
-}
-
-/**
- * Release some bytes that we're using rounded down to the page size.
- *
- * @param countThe number of bytes to release.  We will round this
- * down to the page size.
- *
- * @return The new number of usedBytes.
- */
-long releaseRoundDown(long count) {
-  count = rounder.roundDown(count);
-  return usedBytes.addAndGet(-count);
-}
-
-long get() {
-  return usedBytes.get();
-}
-  }
-
-  /**
-   * The total cache capacity in bytes.
-   */
-  private final long maxBytes;
-
   private final MappableBlockLoader mappableBlockLoader;
 
+  private final MemoryCacheStats memCacheStats;
+
   /**
* Number of cache commands that could not be completed successfully
*/
@@ -240,12 +145,10 @@ public class 

[hadoop] branch branch-3.1 updated (287fab6 -> 2db0b8f)

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a change to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git.


from 287fab6  HDFS-14923. Remove dead code from HealthMonitor. Contributed 
by Fei Hui.
 new aab9fef  HDFS-14354: Refactor MappableBlock to align with the 
implementation of SCM cache. Contributed by Feilong He.
 new 712749c  HDFS-14393. Refactor FsDatasetCache for SCM cache 
implementation. Contributed by Rakesh R
 new b00f9d8  HDFS-14355 : Implement HDFS cache on SCM by using pure java 
mapped byte buffer. Contributed by Feilong He.
 new 3b2a7aa  HDFS-14401. Refine the implementation for HDFS cache on SCM. 
Contributed by Feilong He.
 new f982f96  HDFS-14402. Use FileChannel.transferTo() method for 
transferring block to SCM cache. Contributed by Feilong He.
 new 9281b72  HDFS-14356. Implement HDFS cache on SCM with native PMDK 
libs. Contributed by Feilong He.
 new df0dcc7  HDFS-14458. Report pmem stats to namenode. Contributed by 
Feilong He.
 new f2563dc  HDFS-14357. Update documentation for HDFS cache on SCM 
support. Contributed by Feilong He.
 new cd12773  HDFS-14700. Clean up pmem cache before setting pmem cache 
capacity. Contributed by Feilong He.
 new 2db0b8f  HDFS-14818. Check native pmdk lib by 'hadoop checknative' 
command. Contributed by Feilong He.

The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 BUILDING.txt   |  28 ++
 dev-support/bin/dist-copynativelibs|   8 +
 hadoop-common-project/hadoop-common/pom.xml|   2 +
 .../hadoop-common/src/CMakeLists.txt   |  21 ++
 .../hadoop-common/src/config.h.cmake   |   1 +
 .../org/apache/hadoop/io/nativeio/NativeIO.java| 151 +++-
 .../apache/hadoop/util/NativeLibraryChecker.java   |  10 +
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   | 264 ++
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.c  | 106 ++
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.h  |  90 +
 .../apache/hadoop/io/nativeio/TestNativeIO.java| 153 
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |   5 +
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |   9 +
 .../server/datanode/fsdataset/impl/CacheStats.java | 212 +++
 .../datanode/fsdataset/impl/FsDatasetCache.java| 231 ++--
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  34 +-
 .../datanode/fsdataset/impl/FsDatasetUtil.java |  49 +++
 .../datanode/fsdataset/impl/MappableBlock.java | 155 +---
 ...MappableBlock.java => MappableBlockLoader.java} | 162 +
 .../fsdataset/impl/MappableBlockLoaderFactory.java |  51 +++
 .../fsdataset/impl/MemoryMappableBlockLoader.java  | 125 +++
 .../datanode/fsdataset/impl/MemoryMappedBlock.java |  59 +++
 .../impl/NativePmemMappableBlockLoader.java| 192 ++
 .../fsdataset/impl/NativePmemMappedBlock.java  |  85 +
 .../fsdataset/impl/PmemMappableBlockLoader.java| 149 
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  69 
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 398 +
 .../src/main/resources/hdfs-default.xml|  10 +
 .../site/markdown/CentralizedCacheManagement.md|  18 +-
 .../datanode/TestFsDatasetCacheRevocation.java |  31 +-
 .../impl/TestCacheByPmemMappableBlockLoader.java   | 325 +
 .../{ => fsdataset/impl}/TestFsDatasetCache.java   |  15 +-
 32 files changed, 2845 insertions(+), 373 deletions(-)
 create mode 100644 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c
 create mode 100644 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.h
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
 copy 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/{MappableBlock.java
 => MappableBlockLoader.java} (56%)
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoaderFactory.java
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappedBlock.java
 create mode 100644 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java
 create mode 100644 
hadoop-hdfs-project/

[hadoop] branch trunk updated: HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 659c888  HDFS-14818. Check native pmdk lib by 'hadoop checknative' 
command. Contributed by Feilong He.
659c888 is described below

commit 659c88801d008bb352d10a1cb3bd0e401486cc9b
Author: Rakesh Radhakrishnan 
AuthorDate: Sun Sep 22 22:02:54 2019 +0530

HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. 
Contributed by Feilong He.
---
 .../hadoop-common/src/CMakeLists.txt   |  2 +-
 .../org/apache/hadoop/io/nativeio/NativeIO.java| 28 +-
 .../apache/hadoop/util/NativeLibraryChecker.java   | 10 
 .../src/org/apache/hadoop/io/nativeio/NativeIO.c   | 14 ++-
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.c  | 28 +++---
 .../src/org/apache/hadoop/io/nativeio/pmdk_load.h  |  5 
 .../datanode/fsdataset/impl/FsDatasetCache.java| 15 +---
 7 files changed, 72 insertions(+), 30 deletions(-)

diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt 
b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
index 771c685..10591f6 100644
--- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
+++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
@@ -170,7 +170,7 @@ if(REQUIRE_PMDK)
 set(CMAKE_FIND_LIBRARY_SUFFIXES ${STORED_CMAKE_FIND_LIBRARY_SUFFIXES})
 
 if(PMDK_LIBRARY)
-GET_FILENAME_COMPONENT(HADOOP_PMDK_LIBRARY ${PMDK_LIBRARY} NAME)
+GET_FILENAME_COMPONENT(HADOOP_PMDK_LIBRARY ${PMDK_LIBRARY} REALPATH)
 set(PMDK_SOURCE_FILES ${SRC}/io/nativeio/pmdk_load.c)
 else(PMDK_LIBRARY)
 MESSAGE(FATAL_ERROR "The required PMDK library is NOT found. 
PMDK_LIBRARY=${PMDK_LIBRARY}")
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
index 1d0eab7..973afa3 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
@@ -120,16 +120,19 @@ public class NativeIO {
   public String getMessage() {
 String msg;
 switch (stateCode) {
+// -1 represents UNSUPPORTED.
 case -1:
-  msg = "The native code is built without PMDK support.";
+  msg = "The native code was built without PMDK support.";
   break;
+// 1 represents PMDK_LIB_NOT_FOUND.
 case 1:
-  msg = "The native code is built with PMDK support, but PMDK libs " +
-  "are NOT found in execution environment or failed to be loaded.";
+  msg = "The native code was built with PMDK support, but PMDK libs " +
+  "were NOT found in execution environment or failed to be 
loaded.";
   break;
+// 0 represents SUPPORTED.
 case 0:
-  msg = "The native code is built with PMDK support, and PMDK libs " +
-  "are loaded successfully.";
+  msg = "The native code was built with PMDK support, and PMDK libs " +
+  "were loaded successfully.";
   break;
 default:
   msg = "The state code: " + stateCode + " is unrecognized!";
@@ -140,7 +143,7 @@ public class NativeIO {
 
 // Denotes the state of supporting PMDK. The value is set by JNI.
 private static SupportState pmdkSupportState =
-SupportState.PMDK_LIB_NOT_FOUND;
+SupportState.UNSUPPORTED;
 
 private static final Logger LOG = LoggerFactory.getLogger(NativeIO.class);
 
@@ -177,6 +180,14 @@ public class NativeIO {
   LOG.error("The state code: " + stateCode + " is unrecognized!");
 }
 
+public static String getPmdkSupportStateMessage() {
+  if (getPmdkLibPath() != null) {
+return pmdkSupportState.getMessage() +
+" The pmdk lib path: " + getPmdkLibPath();
+  }
+  return pmdkSupportState.getMessage();
+}
+
 public static boolean isPmdkAvailable() {
   LOG.info(pmdkSupportState.getMessage());
   return pmdkSupportState == SupportState.SUPPORTED;
@@ -242,8 +253,13 @@ public class NativeIO {
   NativeIO.POSIX.pmemSync(region.getAddress(), region.getLength());
 }
   }
+
+  public static String getPmdkLibPath() {
+return POSIX.getPmdkLibPath();
+  }
 }
 
+private static native String getPmdkLibPath();
 private static native boolean isPmemCheck(long address, long length);
 private static native PmemMappedRegion pmemCreateMapFile(String path,
 long l

[hadoop] branch trunk updated: HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new f6fa865  HDFS-14700. Clean up pmem cache before setting pmem cache 
capacity. Contributed by Feilong He.
f6fa865 is described below

commit f6fa865d6fcb0ef0a25a00615f16f383e5032373
Author: Rakesh Radhakrishnan 
AuthorDate: Fri Aug 9 14:07:54 2019 +0530

HDFS-14700. Clean up pmem cache before setting pmem cache capacity. 
Contributed by Feilong He.
---
 .../datanode/fsdataset/impl/PmemVolumeManager.java| 19 ---
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
index 2d77f7a..969d18b 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
@@ -218,6 +218,8 @@ public final class PmemVolumeManager {
   try {
 File pmemDir = new File(volumes[n]);
 File realPmemDir = verifyIfValidPmemVolume(pmemDir);
+// Clean up the cache left before, if any.
+cleanup(realPmemDir);
 this.pmemVolumes.add(realPmemDir.getPath());
 long maxBytes;
 if (maxBytesPerPmem == -1) {
@@ -242,17 +244,20 @@ public final class PmemVolumeManager {
   throw new IOException(
   "At least one valid persistent memory volume is required!");
 }
-cleanup();
+  }
+
+  void cleanup(File realPmemDir) {
+try {
+  FileUtils.cleanDirectory(realPmemDir);
+} catch (IOException e) {
+  LOG.error("Failed to clean up " + realPmemDir.getPath(), e);
+}
   }
 
   void cleanup() {
 // Remove all files under the volume.
-for (String pmemDir: pmemVolumes) {
-  try {
-FileUtils.cleanDirectory(new File(pmemDir));
-  } catch (IOException e) {
-LOG.error("Failed to clean up " + pmemDir, e);
-  }
+for (String pmemVolume : pmemVolumes) {
+  cleanup(new File(pmemVolume));
 }
   }
 


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch trunk updated: HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 30a8f84  HDFS-14357. Update documentation for HDFS cache on SCM 
support. Contributed by Feilong He.
30a8f84 is described below

commit 30a8f840f1572129fe7d02f8a784c47ab57ce89a
Author: Rakesh Radhakrishnan 
AuthorDate: Mon Jul 15 13:18:23 2019 +0530

HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed 
by Feilong He.
---
 .../src/site/markdown/CentralizedCacheManagement.md| 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
index 7568949..8880ea5 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/CentralizedCacheManagement.md
@@ -32,6 +32,8 @@ Centralized cache management in HDFS has many significant 
advantages.
 
 4.  Centralized caching can improve overall cluster memory utilization. When 
relying on the OS buffer cache at each DataNode, repeated reads of a block will 
result in all *n* replicas of the block being pulled into buffer cache. With 
centralized cache management, a user can explicitly pin only *m* of the *n* 
replicas, saving *n-m* memory.
 
+5.  HDFS supports non-volatile storage class memory (SCM, also known as 
persistent memory) cache in Linux platform. User can enable either memory cache 
or SCM cache for a DataNode. Memory cache and SCM cache can coexist among 
DataNodes. In the current implementation, the cache data in SCM will be cleaned 
up when DataNode restarts. Persistent HDFS cache support on SCM will be 
considered in the future.
+
 Use Cases
 -
 
@@ -200,11 +202,21 @@ Configuration
 
 In order to lock block files into memory, the DataNode relies on native JNI 
code found in `libhadoop.so` or `hadoop.dll` on Windows. Be sure to [enable 
JNI](../hadoop-common/NativeLibraries.html) if you are using HDFS centralized 
cache management.
 
+Currently, there are two implementations for persistent memory cache. The 
default one is pure Java based implementation and the other is native 
implementation which leverages PMDK library to improve the performance of cache 
write and cache read.
+
+To enable PMDK based implementation, please follow the below steps.
+
+1. Install PMDK library. Please refer to the official site http://pmem.io/ for 
detailed information.
+
+2. Build Hadoop with PMDK support. Please refer to "PMDK library build 
options" section in `BUILDING.txt` in the source code.
+
+To verify that PMDK is correctly detected by Hadoop, run the `hadoop 
checknative` command.
+
 ### Configuration Properties
 
  Required
 
-Be sure to configure the following:
+Be sure to configure one of the following properties for DRAM cache or 
persistent memory cache. Please note that DRAM cache and persistent cache 
cannot coexist on a DataNode.
 
 *   dfs.datanode.max.locked.memory
 
@@ -212,6 +224,10 @@ Be sure to configure the following:
 
 This setting is shared with the [Lazy Persist Writes 
feature](./MemoryStorage.html). The Data Node will ensure that the combined 
memory used by Lazy Persist Writes and Centralized Cache Management does not 
exceed the amount configured in `dfs.datanode.max.locked.memory`.
 
+*   dfs.datanode.cache.pmem.dirs
+
+This property specifies the cache volume of persistent memory. For 
multiple volumes, they should be separated by “,”, e.g. “/mnt/pmem0, 
/mnt/pmem1”. The default value is empty. If this property is configured, the 
volume capacity will be detected. And there is no need to configure 
`dfs.datanode.max.locked.memory`.
+
  Optional
 
 The following properties are not required, but may be specified for tuning:


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch trunk updated: HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new e98adb0  HDFS-14458. Report pmem stats to namenode. Contributed by 
Feilong He.
e98adb0 is described below

commit e98adb00b7da8fa913b86ecf2049444b1d8617d4
Author: Rakesh Radhakrishnan 
AuthorDate: Mon Jul 15 13:02:37 2019 +0530

HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He.
---
 .../{MemoryCacheStats.java => CacheStats.java} |  6 +--
 .../datanode/fsdataset/impl/FsDatasetCache.java| 54 ++
 .../datanode/fsdataset/impl/FsDatasetImpl.java |  4 +-
 .../fsdataset/impl/MappableBlockLoader.java|  3 +-
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  8 ++--
 .../impl/NativePmemMappableBlockLoader.java|  5 +-
 .../fsdataset/impl/PmemMappableBlockLoader.java|  9 +++-
 .../impl/TestCacheByPmemMappableBlockLoader.java   | 14 +++---
 .../fsdataset/impl/TestFsDatasetCache.java |  2 +-
 9 files changed, 45 insertions(+), 60 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
similarity index 97%
rename from 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
rename to 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
index d276c27..f79b7c7 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryCacheStats.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
@@ -27,7 +27,7 @@ import com.google.common.annotations.VisibleForTesting;
 /**
  * Keeps statistics for the memory cache.
  */
-class MemoryCacheStats {
+class CacheStats {
 
   /**
* The approximate amount of cache space in use.
@@ -47,7 +47,7 @@ class MemoryCacheStats {
*/
   private final long maxBytes;
 
-  MemoryCacheStats(long maxBytes) {
+  CacheStats(long maxBytes) {
 this.usedBytesCount = new UsedBytesCount();
 this.maxBytes = maxBytes;
   }
@@ -81,7 +81,7 @@ class MemoryCacheStats {
   private class UsedBytesCount {
 private final AtomicLong usedBytes = new AtomicLong(0);
 
-private MemoryCacheStats.PageRounder rounder = new PageRounder();
+private CacheStats.PageRounder rounder = new PageRounder();
 
 /**
  * Try to reserve more bytes.
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
index 37e548e..1514927 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
@@ -23,7 +23,6 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION_POLLING_MS;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_CACHE_REVOCATION_POLLING_MS_DEFAULT;
 
-import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 import com.google.common.util.concurrent.ThreadFactoryBuilder;
 
@@ -137,7 +136,7 @@ public class FsDatasetCache {
*/
   private final MappableBlockLoader cacheLoader;
 
-  private final MemoryCacheStats memCacheStats;
+  private final CacheStats memCacheStats;
 
   /**
* Number of cache commands that could not be completed successfully
@@ -178,30 +177,17 @@ public class FsDatasetCache {
   ".  Reconfigure this to " + minRevocationPollingMs);
 }
 this.revocationPollingMs = confRevocationPollingMs;
-// Both lazy writer and read cache are sharing this statistics.
-this.memCacheStats = new MemoryCacheStats(
-dataset.datanode.getDnConf().getMaxLockedMemory());
 
 this.cacheLoader = MappableBlockLoaderFactory.createCacheLoader(
 this.getDnConf());
-cacheLoader.initialize(this);
-  }
-
-  /**
-   * Check if pmem cache is enabled.
-   */
-  private boolean isPmemCacheEnabled() {
-return !cacheLoader.isTransientCache();
+// Both lazy writer and read cache are sharing this statistics.
+this.memCacheStats = cacheLoader.initialize(this.getDnConf());
   }
 
   DNConf getDnConf() {
 return this.dataset.datanode.getDnConf();
   }
 
-  MemoryCacheStats get

[hadoop] branch trunk updated: HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 37900c5  HDFS-14402. Use FileChannel.transferTo() method for 
transferring block to SCM cache. Contributed by Feilong He.
37900c5 is described below

commit 37900c5639f8ba8d41b9fedc3d41ee0fbda7d5db
Author: Rakesh Radhakrishnan 
AuthorDate: Sun May 26 14:30:11 2019 +0530

HDFS-14402. Use FileChannel.transferTo() method for transferring block to 
SCM cache. Contributed by Feilong He.
---
 .../fsdataset/impl/MappableBlockLoader.java|  59 +++
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  59 ---
 .../fsdataset/impl/PmemMappableBlockLoader.java| 110 +++--
 3 files changed, 75 insertions(+), 153 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
index 044e5c5..3ec8416 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java
@@ -18,10 +18,16 @@
 
 package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl;
 
+import com.google.common.base.Preconditions;
+import org.apache.commons.io.IOUtils;
 import org.apache.hadoop.classification.InterfaceAudience;
 import org.apache.hadoop.classification.InterfaceStability;
 import org.apache.hadoop.hdfs.ExtendedBlockId;
+import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader;
+import org.apache.hadoop.util.DataChecksum;
 
+import java.io.BufferedInputStream;
+import java.io.DataInputStream;
 import java.io.FileInputStream;
 import java.io.IOException;
 import java.nio.ByteBuffer;
@@ -108,6 +114,59 @@ public abstract class MappableBlockLoader {
   }
 
   /**
+   * Verifies the block's checksum. This is an I/O intensive operation.
+   */
+  protected void verifyChecksum(long length, FileInputStream metaIn,
+FileChannel blockChannel, String blockFileName)
+  throws IOException {
+// Verify the checksum from the block's meta file
+// Get the DataChecksum from the meta file header
+BlockMetadataHeader header =
+BlockMetadataHeader.readHeader(new DataInputStream(
+new BufferedInputStream(metaIn, BlockMetadataHeader
+.getHeaderSize(;
+FileChannel metaChannel = null;
+try {
+  metaChannel = metaIn.getChannel();
+  if (metaChannel == null) {
+throw new IOException(
+"Block InputStream meta file has no FileChannel.");
+  }
+  DataChecksum checksum = header.getChecksum();
+  final int bytesPerChecksum = checksum.getBytesPerChecksum();
+  final int checksumSize = checksum.getChecksumSize();
+  final int numChunks = (8 * 1024 * 1024) / bytesPerChecksum;
+  ByteBuffer blockBuf = ByteBuffer.allocate(numChunks * bytesPerChecksum);
+  ByteBuffer checksumBuf = ByteBuffer.allocate(numChunks * checksumSize);
+  // Verify the checksum
+  int bytesVerified = 0;
+  while (bytesVerified < length) {
+Preconditions.checkState(bytesVerified % bytesPerChecksum == 0,
+"Unexpected partial chunk before EOF");
+assert bytesVerified % bytesPerChecksum == 0;
+int bytesRead = fillBuffer(blockChannel, blockBuf);
+if (bytesRead == -1) {
+  throw new IOException("checksum verification failed: premature EOF");
+}
+blockBuf.flip();
+// Number of read chunks, including partial chunk at end
+int chunks = (bytesRead + bytesPerChecksum - 1) / bytesPerChecksum;
+checksumBuf.limit(chunks * checksumSize);
+fillBuffer(metaChannel, checksumBuf);
+checksumBuf.flip();
+checksum.verifyChunkedSums(blockBuf, checksumBuf, blockFileName,
+bytesVerified);
+// Success
+bytesVerified += bytesRead;
+blockBuf.clear();
+checksumBuf.clear();
+  }
+} finally {
+  IOUtils.closeQuietly(metaChannel);
+}
+  }
+
+  /**
* Reads bytes into a buffer until EOF or the buffer's limit is reached.
*/
   protected int fillBuffer(FileChannel channel, ByteBuffer buf)
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
index 919835a..52d8d93 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/a

[hadoop] branch trunk updated: HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He.

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 9b0aace  HDFS-14401. Refine the implementation for HDFS cache on SCM. 
Contributed by Feilong He.
9b0aace is described below

commit 9b0aace1e6c54f201784912c0b623707aa82b761
Author: Rakesh Radhakrishnan 
AuthorDate: Wed May 8 17:20:21 2019 +0530

HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by 
Feilong He.
---
 .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java |  14 --
 .../apache/hadoop/hdfs/server/datanode/DNConf.java |  22 ---
 .../datanode/fsdataset/impl/FsDatasetCache.java|  31 ++--
 .../datanode/fsdataset/impl/FsDatasetImpl.java |   2 +
 .../fsdataset/impl/MappableBlockLoader.java|  19 +-
 .../fsdataset/impl/MappableBlockLoaderFactory.java |  47 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  21 +--
 .../fsdataset/impl/PmemMappableBlockLoader.java|  40 ++---
 .../datanode/fsdataset/impl/PmemMappedBlock.java   |  10 +-
 .../datanode/fsdataset/impl/PmemVolumeManager.java | 197 +++--
 .../src/main/resources/hdfs-default.xml|  24 ---
 .../impl/TestCacheByPmemMappableBlockLoader.java   |  26 ++-
 .../fsdataset/impl/TestFsDatasetCache.java |   5 +-
 13 files changed, 256 insertions(+), 202 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index f9a4ab2..e9d0eec 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -26,8 +26,6 @@ import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault;
 import 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant;
-import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader;
-import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker;
 import 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator;
 import org.apache.hadoop.hdfs.web.URLConnectionFactory;
@@ -392,22 +390,10 @@ public class DFSConfigKeys extends 
CommonConfigurationKeys {
   public static final String DFS_DATANODE_CACHE_REVOCATION_POLLING_MS = 
"dfs.datanode.cache.revocation.polling.ms";
   public static final long DFS_DATANODE_CACHE_REVOCATION_POLLING_MS_DEFAULT = 
500L;
 
-  // Currently, the available cache loaders are MemoryMappableBlockLoader,
-  // PmemMappableBlockLoader. MemoryMappableBlockLoader is the default cache
-  // loader to cache block replica to memory.
-  public static final String DFS_DATANODE_CACHE_LOADER_CLASS =
-  "dfs.datanode.cache.loader.class";
-  public static final Class
-  DFS_DATANODE_CACHE_LOADER_CLASS_DEFAULT =
-  MemoryMappableBlockLoader.class;
   // Multiple dirs separated by "," are acceptable.
   public static final String DFS_DATANODE_CACHE_PMEM_DIRS_KEY =
   "dfs.datanode.cache.pmem.dirs";
   public static final String DFS_DATANODE_CACHE_PMEM_DIRS_DEFAULT = "";
-  // The cache capacity of persistent memory
-  public static final String DFS_DATANODE_CACHE_PMEM_CAPACITY_KEY =
-  "dfs.datanode.cache.pmem.capacity";
-  public static final long DFS_DATANODE_CACHE_PMEM_CAPACITY_DEFAULT = 0L;
 
   public static final String 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_KEY = 
"dfs.namenode.datanode.registration.ip-hostname-check";
   public static final boolean 
DFS_NAMENODE_DATANODE_REGISTRATION_IP_HOSTNAME_CHECK_DEFAULT = true;
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
index b4bf868..00ac64c 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
@@ -27,10 +27,6 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHO
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHOLD_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_DEFAULT;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CACHEREPORT_INTERVAL_MSEC_KEY;
-import static 
org.apache.had

[hadoop] branch trunk updated: HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R

[rakeshr]

This is an automated email from the ASF dual-hosted git repository.

rakeshr pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new f3f5128  HDFS-14393. Refactor FsDatasetCache for SCM cache 
implementation. Contributed by Rakesh R
f3f5128 is described below

commit f3f51284d57ef2e0c7e968b6eea56eab578f7e93
Author: Rakesh Radhakrishnan 
AuthorDate: Fri Mar 29 00:18:15 2019 +0530

HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. 
Contributed by Rakesh R
---
 .../datanode/fsdataset/impl/FsDatasetCache.java| 132 ++---
 .../fsdataset/impl/MappableBlockLoader.java|  20 ++
 .../datanode/fsdataset/impl/MemoryCacheStats.java  | 212 +
 .../fsdataset/impl/MemoryMappableBlockLoader.java  |  24 ++-
 .../datanode/TestFsDatasetCacheRevocation.java |  31 +--
 .../{ => fsdataset/impl}/TestFsDatasetCache.java   |  10 +-
 6 files changed, 298 insertions(+), 131 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
index 9efd11a..f90a4b1 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
@@ -52,7 +52,6 @@ import org.apache.hadoop.hdfs.DFSConfigKeys;
 import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
 import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
 import org.apache.hadoop.hdfs.server.datanode.DatanodeUtil;
-import org.apache.hadoop.io.nativeio.NativeIO;
 import org.apache.hadoop.util.Time;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -131,104 +130,10 @@ public class FsDatasetCache {
 
   private final long revocationPollingMs;
 
-  /**
-   * The approximate amount of cache space in use.
-   *
-   * This number is an overestimate, counting bytes that will be used only
-   * if pending caching operations succeed.  It does not take into account
-   * pending uncaching operations.
-   *
-   * This overestimate is more useful to the NameNode than an underestimate,
-   * since we don't want the NameNode to assign us more replicas than
-   * we can cache, because of the current batch of operations.
-   */
-  private final UsedBytesCount usedBytesCount;
-
-  public static class PageRounder {
-private final long osPageSize =
-NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize();
-
-/**
- * Round up a number to the operating system page size.
- */
-public long roundUp(long count) {
-  return (count + osPageSize - 1) & (~(osPageSize - 1));
-}
-
-/**
- * Round down a number to the operating system page size.
- */
-public long roundDown(long count) {
-  return count & (~(osPageSize - 1));
-}
-  }
-
-  private class UsedBytesCount {
-private final AtomicLong usedBytes = new AtomicLong(0);
-
-private final PageRounder rounder = new PageRounder();
-
-/**
- * Try to reserve more bytes.
- *
- * @param countThe number of bytes to add.  We will round this
- * up to the page size.
- *
- * @return The new number of usedBytes if we succeeded;
- * -1 if we failed.
- */
-long reserve(long count) {
-  count = rounder.roundUp(count);
-  while (true) {
-long cur = usedBytes.get();
-long next = cur + count;
-if (next > maxBytes) {
-  return -1;
-}
-if (usedBytes.compareAndSet(cur, next)) {
-  return next;
-}
-  }
-}
-
-/**
- * Release some bytes that we're using.
- *
- * @param countThe number of bytes to release.  We will round this
- * up to the page size.
- *
- * @return The new number of usedBytes.
- */
-long release(long count) {
-  count = rounder.roundUp(count);
-  return usedBytes.addAndGet(-count);
-}
-
-/**
- * Release some bytes that we're using rounded down to the page size.
- *
- * @param countThe number of bytes to release.  We will round this
- * down to the page size.
- *
- * @return The new number of usedBytes.
- */
-long releaseRoundDown(long count) {
-  count = rounder.roundDown(count);
-  return usedBytes.addAndGet(-count);
-}
-
-long get() {
-  return usedBytes.get();
-}
-  }
-
-  /**
-   * The total cache capacity in bytes.
-   */
-  private final long maxBytes;
-
   private final MappableBlockLoader mappableBlockLoader;
 
+  private final MemoryCacheStats memCacheStats;
+
   /*

[48/50] [abbrv] hadoop git commit: HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5bd5714/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestBlockStorageMovementAttemptedItems.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestBlockStorageMovementAttemptedItems.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestBlockStorageMovementAttemptedItems.java
index f85769f..f48521b 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestBlockStorageMovementAttemptedItems.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestBlockStorageMovementAttemptedItems.java
@@ -34,6 +34,7 @@ import org.apache.hadoop.hdfs.HdfsConfiguration;
 import org.apache.hadoop.hdfs.protocol.Block;
 import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
 import 
org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier.StorageTypeNodePair;
+import org.apache.hadoop.hdfs.server.sps.ExternalSPSContext;
 import org.junit.After;
 import org.junit.Before;
 import org.junit.Test;
@@ -52,7 +53,7 @@ public class TestBlockStorageMovementAttemptedItems {
   @Before
   public void setup() throws Exception {
 Configuration config = new HdfsConfiguration();
-Context ctxt = Mockito.mock(IntraSPSNameNodeContext.class);
+Context ctxt = Mockito.mock(ExternalSPSContext.class);
 SPSService sps = new StoragePolicySatisfier(config);
 Mockito.when(ctxt.isRunning()).thenReturn(true);
 Mockito.when(ctxt.isInSafeMode()).thenReturn(false);

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5bd5714/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
deleted file mode 100644
index ec5307b..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
+++ /dev/null
@@ -1,1825 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.hdfs.server.namenode.sps;
-
-import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_STORAGE_POLICY_ENABLED_KEY;
-import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_QUEUE_LIMIT_KEY;
-import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.slf4j.LoggerFactory.getLogger;
-
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.TimeoutException;
-
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.conf.ReconfigurationException;
-import org.apache.hadoop.fs.FSDataOutputStream;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.StorageType;
-import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.DFSTestUtil;
-import org.apache.hadoop.hdfs.DistributedFileSystem;
-import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
-import org.apache.hadoop.hdfs.NameNodeProxies;
-import org.apache.hadoop.hdfs.StripedFileTestUtil;
-import org.apache.hadoop.hdfs.client.HdfsAdmin;
-import org.apache.hadoop.hdfs.protocol.Block;
-import 

[50/50] [abbrv] hadoop git commit: HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.

[rakeshr]

HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e5bd5714
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e5bd5714
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e5bd5714

Branch: refs/heads/HDFS-10285
Commit: e5bd57145a05eb0b3b6333d974cadfadae3a8afd
Parents: 4c2f806
Author: Uma Maheswara Rao G 
Authored: Mon Jul 23 16:05:35 2018 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:11:13 2018 +0530

--
 .../java/org/apache/hadoop/hdfs/DFSClient.java  |   26 -
 .../hadoop/hdfs/protocol/ClientProtocol.java|   29 -
 .../hadoop/hdfs/protocol/HdfsConstants.java |   40 -
 .../ClientNamenodeProtocolTranslatorPB.java |   36 -
 .../hadoop/hdfs/protocolPB/PBHelperClient.java  |   37 -
 .../src/main/proto/ClientNamenodeProtocol.proto |   26 -
 .../federation/router/RouterRpcServer.java  |   14 -
 .../org/apache/hadoop/hdfs/DFSConfigKeys.java   |4 -
 ...tNamenodeProtocolServerSideTranslatorPB.java |   39 -
 .../apache/hadoop/hdfs/protocolPB/PBHelper.java |   99 -
 .../hadoop/hdfs/server/balancer/ExitStatus.java |3 +-
 .../server/blockmanagement/BlockManager.java|   21 +-
 .../blockmanagement/DatanodeDescriptor.java |   68 -
 .../server/blockmanagement/DatanodeManager.java |   94 +-
 .../hdfs/server/datanode/BPOfferService.java|   12 -
 .../hadoop/hdfs/server/datanode/DataNode.java   |   12 -
 .../datanode/StoragePolicySatisfyWorker.java|  217 ---
 .../apache/hadoop/hdfs/server/mover/Mover.java  |   21 -
 .../hdfs/server/namenode/FSDirXAttrOp.java  |   12 -
 .../hdfs/server/namenode/FSNamesystem.java  |8 +-
 .../hdfs/server/namenode/NameNodeRpcServer.java |   36 -
 .../sps/BlockStorageMovementNeeded.java |  121 +-
 .../hdfs/server/namenode/sps/Context.java   |5 -
 .../IntraSPSNameNodeBlockMoveTaskHandler.java   |   63 -
 .../namenode/sps/IntraSPSNameNodeContext.java   |  189 --
 .../sps/IntraSPSNameNodeFileIdCollector.java|  185 --
 .../hdfs/server/namenode/sps/SPSService.java|5 -
 .../namenode/sps/StoragePolicySatisfier.java|   44 -
 .../sps/StoragePolicySatisfyManager.java|  156 +-
 .../hdfs/server/sps/ExternalSPSContext.java |5 -
 .../sps/ExternalStoragePolicySatisfier.java |9 -
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   |   87 +-
 .../src/main/proto/DatanodeProtocol.proto   |   30 -
 .../src/main/resources/hdfs-default.xml |   14 +-
 .../src/site/markdown/ArchivalStorage.md|   22 +-
 .../org/apache/hadoop/hdfs/DFSTestUtil.java |   40 +
 .../server/datanode/TestBPOfferService.java |4 -
 .../TestStoragePolicySatisfyWorker.java |  241 ---
 .../hadoop/hdfs/server/mover/TestMover.java |7 +-
 .../namenode/TestNameNodeReconfigure.java   |   32 +-
 .../TestPersistentStoragePolicySatisfier.java   |  124 +-
 .../TestStoragePolicySatisfierWithHA.java   |  152 +-
 .../TestBlockStorageMovementAttemptedItems.java |3 +-
 .../sps/TestStoragePolicySatisfier.java | 1825 --
 ...stStoragePolicySatisfierWithStripedFile.java |   87 +-
 .../sps/TestExternalStoragePolicySatisfier.java | 1433 +-
 .../hdfs/tools/TestStoragePolicyCommands.java   |2 +-
 .../TestStoragePolicySatisfyAdminCommands.java  |   56 +-
 48 files changed, 1517 insertions(+), 4278 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5bd5714/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
index b6f9bdd..adbb133 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@@ -123,7 +123,6 @@ import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.ReencryptAction;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.RollingUpgradeAction;
-import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction;
 import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
 import org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus;
@@ -3110,10 +3109,6 @@ public class DFSClient implements java.io.Closeable, 
RemotePeerFactory,
 }
   }
 
-  public boolean 

[45/50] [abbrv] hadoop git commit: HDFS-13076: [SPS]: Resolve conflicts after rebasing HDFS-10285 branch to trunk. Contributed by Rakesh R.

[rakeshr]

HDFS-13076: [SPS]: Resolve conflicts after rebasing HDFS-10285 branch to trunk. 
Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0bdc60f5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0bdc60f5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0bdc60f5

Branch: refs/heads/HDFS-10285
Commit: 0bdc60f5a7a0600a0f0de9e0b95de9011ba8dc72
Parents: 8eabfc0
Author: Rakesh Radhakrishnan 
Authored: Thu Jul 5 10:10:13 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:11:05 2018 +0530

--
 .../hdfs/server/federation/router/RouterNamenodeProtocol.java | 6 ++
 .../hadoop/hdfs/server/federation/router/RouterRpcServer.java | 7 +++
 .../hadoop/hdfs/server/blockmanagement/BlockManager.java  | 2 +-
 .../server/namenode/sps/IntraSPSNameNodeFileIdCollector.java  | 4 ++--
 4 files changed, 16 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/0bdc60f5/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNamenodeProtocol.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNamenodeProtocol.java
 
b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNamenodeProtocol.java
index 0433650..edfb391 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNamenodeProtocol.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNamenodeProtocol.java
@@ -184,4 +184,10 @@ public class RouterNamenodeProtocol implements 
NamenodeProtocol {
 rpcServer.checkOperation(OperationCategory.READ, false);
 return false;
   }
+
+  @Override
+  public Long getNextSPSPath() throws IOException {
+// not supported
+return null;
+  }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/0bdc60f5/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
 
b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
index d93f99d..36645c9 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
@@ -2509,4 +2509,11 @@ public class RouterRpcServer extends AbstractService
 checkOperation(OperationCategory.READ, false);
 return StoragePolicySatisfyPathStatus.NOT_AVAILABLE;
   }
+
+  @Override
+  public Long getNextSPSPath() throws IOException {
+checkOperation(OperationCategory.READ, false);
+// not supported
+return null;
+  }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/0bdc60f5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index cb0de67..94ada2e 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -5070,7 +5070,7 @@ public class BlockManager implements BlockStatsMXBean {
 DFSConfigKeys.DFS_STORAGE_POLICY_ENABLED_KEY,
 DFSConfigKeys.DFS_STORAGE_POLICY_ENABLED_DEFAULT);
 String modeVal = spsMode;
-if (org.apache.commons.lang.StringUtils.isBlank(modeVal)) {
+if (org.apache.commons.lang3.StringUtils.isBlank(modeVal)) {
   modeVal = conf.get(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_MODE_KEY,
   DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_MODE_DEFAULT);
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/0bdc60f5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeFileIdCollector.java
--
diff --git 

[42/50] [abbrv] hadoop git commit: HDFS-13165: [SPS]: Collects successfully moved block details via IBR. Contributed by Rakesh R.

[rakeshr]

HDFS-13165: [SPS]: Collects successfully moved block details via IBR. 
Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/93a04547
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/93a04547
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/93a04547

Branch: refs/heads/HDFS-10285
Commit: 93a04547d5d86537bb6bcaf76165169f5b5689bb
Parents: 616d9ed
Author: Rakesh Radhakrishnan 
Authored: Sun Apr 29 11:06:59 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:57 2018 +0530

--
 .../DatanodeProtocolClientSideTranslatorPB.java |  11 +-
 .../DatanodeProtocolServerSideTranslatorPB.java |   4 +-
 .../apache/hadoop/hdfs/protocolPB/PBHelper.java |  25 ---
 .../server/blockmanagement/BlockManager.java|  86 +-
 .../sps/BlockMovementAttemptFinished.java   |  24 ++-
 .../common/sps/BlockStorageMovementTracker.java | 109 +---
 .../sps/BlocksMovementsStatusHandler.java   |  70 +---
 .../hdfs/server/datanode/BPServiceActor.java|  14 +-
 .../hadoop/hdfs/server/datanode/DataNode.java   |   7 +-
 .../datanode/StoragePolicySatisfyWorker.java|  48 ++
 .../namenode/FSDirSatisfyStoragePolicyOp.java   |  13 +-
 .../hdfs/server/namenode/FSDirXAttrOp.java  |   8 +-
 .../hdfs/server/namenode/FSDirectory.java   |   5 +-
 .../hdfs/server/namenode/FSNamesystem.java  |  30 ++--
 .../hadoop/hdfs/server/namenode/NameNode.java   |  19 ++-
 .../hdfs/server/namenode/NameNodeRpcServer.java |  46 +++--
 .../sps/BlockStorageMovementAttemptedItems.java | 167 +--
 .../hdfs/server/namenode/sps/SPSService.java|  19 ++-
 .../namenode/sps/StoragePolicySatisfier.java| 154 +++--
 .../hdfs/server/protocol/DatanodeProtocol.java  |   4 +-
 .../sps/ExternalSPSBlockMoveTaskHandler.java|  32 ++--
 .../sps/ExternalStoragePolicySatisfier.java |   3 +-
 .../src/main/proto/DatanodeProtocol.proto   |   9 -
 .../src/main/resources/hdfs-default.xml |  41 +
 .../TestNameNodePrunesMissingStorages.java  |   4 +-
 .../datanode/InternalDataNodeTestUtils.java |   4 +-
 .../SimpleBlocksMovementsStatusHandler.java |  88 ++
 .../server/datanode/TestBPOfferService.java |  12 +-
 .../hdfs/server/datanode/TestBlockRecovery.java |   4 +-
 .../server/datanode/TestDataNodeLifeline.java   |   7 +-
 .../TestDatanodeProtocolRetryPolicy.java|   4 +-
 .../server/datanode/TestFsDatasetCache.java |   4 +-
 .../TestStoragePolicySatisfyWorker.java |  76 +
 .../hdfs/server/datanode/TestStorageReport.java |   4 +-
 .../server/namenode/NNThroughputBenchmark.java  |   9 +-
 .../hdfs/server/namenode/NameNodeAdapter.java   |   4 +-
 .../hdfs/server/namenode/TestDeadDatanode.java  |   5 +-
 .../namenode/TestNameNodeReconfigure.java   |  17 +-
 .../TestBlockStorageMovementAttemptedItems.java |  88 ++
 .../sps/TestStoragePolicySatisfier.java |  73 ++--
 ...stStoragePolicySatisfierWithStripedFile.java |  40 +++--
 .../sps/TestExternalStoragePolicySatisfier.java |  44 ++---
 42 files changed, 776 insertions(+), 659 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/93a04547/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
index dcc0705..e4125dc 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
@@ -48,7 +48,6 @@ import 
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.StorageBlock
 import 
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.StorageReceivedDeletedBlocksProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.HdfsServerProtos.VersionRequestProto;
 import org.apache.hadoop.hdfs.server.protocol.BlockReportContext;
-import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMoveAttemptFinished;
 import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand;
 import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol;
 import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration;
@@ -139,8 +138,7 @@ public class DatanodeProtocolClientSideTranslatorPB 
implements
   VolumeFailureSummary volumeFailureSummary,
   boolean requestFullBlockReportLease,
   @Nonnull 

[14/50] [abbrv] hadoop git commit: HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/48baf306/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
index d3c5cb1..2f621e6 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
@@ -156,7 +156,7 @@ import 
org.apache.hadoop.hdfs.server.common.IncorrectVersionException;
 import org.apache.hadoop.hdfs.server.namenode.NameNode.OperationCategory;
 import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
 import org.apache.hadoop.hdfs.server.protocol.BlockReportContext;
-import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMovementResult;
+import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMoveAttemptFinished;
 import org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations;
 import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand;
 import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol;
@@ -1517,14 +1517,15 @@ public class NameNodeRpcServer implements 
NamenodeProtocols {
   boolean requestFullBlockReportLease,
   @Nonnull SlowPeerReports slowPeers,
   @Nonnull SlowDiskReports slowDisks,
-  BlocksStorageMovementResult[] blkMovementStatus) throws IOException {
+  BlocksStorageMoveAttemptFinished storageMovementFinishedBlks)
+  throws IOException {
 checkNNStartup();
 verifyRequest(nodeReg);
 return namesystem.handleHeartbeat(nodeReg, report,
 dnCacheCapacity, dnCacheUsed, xceiverCount, xmitsInProgress,
 failedVolumes, volumeFailureSummary, requestFullBlockReportLease,
 slowPeers, slowDisks,
-blkMovementStatus);
+storageMovementFinishedBlks);
   }
 
   @Override // DatanodeProtocol

http://git-wip-us.apache.org/repos/asf/hadoop/blob/48baf306/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
index a4372d5..a28a806 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
@@ -17,6 +17,8 @@
  */
 package org.apache.hadoop.hdfs.server.namenode;
 
+import static org.apache.hadoop.util.Time.monotonicNow;
+
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
@@ -44,7 +46,7 @@ import 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
 import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo;
 import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
 import 
org.apache.hadoop.hdfs.server.protocol.BlockStorageMovementCommand.BlockMovingInfo;
-import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMovementResult;
+import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMoveAttemptFinished;
 import org.apache.hadoop.hdfs.server.protocol.StorageReport;
 import org.apache.hadoop.hdfs.util.StripedBlockUtil;
 import org.apache.hadoop.util.Daemon;
@@ -82,25 +84,38 @@ public class StoragePolicySatisfier implements Runnable {
   /**
* Represents the collective analysis status for all blocks.
*/
-  private enum BlocksMovingAnalysisStatus {
-// Represents that, the analysis skipped due to some conditions. A such
-// condition is if block collection is in incomplete state.
-ANALYSIS_SKIPPED_FOR_RETRY,
-// Represents that, all block storage movement needed blocks found its
-// targets.
-ALL_BLOCKS_TARGETS_PAIRED,
-// Represents that, only fewer or none of the block storage movement needed
-// block found its eligible targets.
-FEW_BLOCKS_TARGETS_PAIRED,
-// Represents that, none of the blocks found for block storage movements.
-BLOCKS_ALREADY_SATISFIED,
-// Represents that, the analysis skipped due to some conditions.
-// Example conditions are if no blocks really exists in block collection or
-// if analysis is not required on ec files with unsuitable storage policies
-BLOCKS_TARGET_PAIRING_SKIPPED,
-// Represents that, All the reported blocks are satisfied the policy but
-// some of the blocks are low redundant.
-FEW_LOW_REDUNDANCY_BLOCKS
+  private 

[34/50] [abbrv] hadoop git commit: HDFS-13050: [SPS]: Create start/stop script to start external SPS process. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-13050: [SPS]: Create start/stop script to start external SPS process. 
Contributed by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9c6e8ae8
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9c6e8ae8
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9c6e8ae8

Branch: refs/heads/HDFS-10285
Commit: 9c6e8ae88c04b7b33d05f32bae0c58a8de2d5921
Parents: 1682095
Author: Rakesh Radhakrishnan 
Authored: Mon Jan 29 03:10:48 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:35 2018 +0530

--
 .../hadoop-hdfs/src/main/bin/hdfs   |   5 +
 .../server/blockmanagement/BlockManager.java|   9 ++
 .../apache/hadoop/hdfs/server/mover/Mover.java  |   2 +-
 .../hdfs/server/namenode/sps/Context.java   |   5 -
 .../namenode/sps/IntraSPSNameNodeContext.java   |   4 -
 .../sps/IntraSPSNameNodeFileIdCollector.java|  12 +-
 .../hdfs/server/namenode/sps/SPSPathIds.java|   1 +
 .../namenode/sps/StoragePolicySatisfier.java|  83 +++-
 .../sps/ExternalSPSBlockMoveTaskHandler.java|   2 +-
 .../hdfs/server/sps/ExternalSPSContext.java |  57 +---
 .../server/sps/ExternalSPSFileIDCollector.java  |  12 +-
 .../sps/ExternalStoragePolicySatisfier.java | 130 +++
 .../src/site/markdown/ArchivalStorage.md|  10 +-
 .../sps/TestStoragePolicySatisfier.java |  22 ++--
 .../sps/TestExternalStoragePolicySatisfier.java |  33 +++--
 15 files changed, 259 insertions(+), 128 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/9c6e8ae8/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
--
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
index bc6e7a4..94426a5 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
@@ -63,6 +63,7 @@ function hadoop_usage
   hadoop_add_subcommand "secondarynamenode" daemon "run the DFS secondary 
namenode"
   hadoop_add_subcommand "snapshotDiff" client "diff two snapshots of a 
directory or diff the current directory contents with a snapshot"
   hadoop_add_subcommand "storagepolicies" admin 
"list/get/set/satisfyStoragePolicy block storage policies"
+  hadoop_add_subcommand "sps" daemon "run external storagepolicysatisfier"
   hadoop_add_subcommand "version" client "print the version"
   hadoop_add_subcommand "zkfc" daemon "run the ZK Failover Controller daemon"
   hadoop_generate_usage "${HADOOP_SHELL_EXECNAME}" false
@@ -201,6 +202,10 @@ function hdfscmd_case
 storagepolicies)
   HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.StoragePolicyAdmin
 ;;
+sps)
+  HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
+  
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.server.sps.ExternalStoragePolicySatisfier
+;;
 version)
   HADOOP_CLASSNAME=org.apache.hadoop.util.VersionInfo
 ;;

http://git-wip-us.apache.org/repos/asf/hadoop/blob/9c6e8ae8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index daaa7a3..f348a33 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -94,6 +94,9 @@ import org.apache.hadoop.hdfs.server.namenode.NameNode;
 import org.apache.hadoop.hdfs.server.namenode.Namesystem;
 import org.apache.hadoop.hdfs.server.namenode.ha.HAContext;
 import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
+import 
org.apache.hadoop.hdfs.server.namenode.sps.IntraSPSNameNodeBlockMoveTaskHandler;
+import org.apache.hadoop.hdfs.server.namenode.sps.IntraSPSNameNodeContext;
+import 
org.apache.hadoop.hdfs.server.namenode.sps.IntraSPSNameNodeFileIdCollector;
 import org.apache.hadoop.hdfs.server.namenode.sps.SPSPathIds;
 import org.apache.hadoop.hdfs.server.namenode.sps.SPSService;
 import org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier;
@@ -5098,9 +5101,15 @@ public class BlockManager implements BlockStatsMXBean {
   return;
 }
 updateSPSMode(StoragePolicySatisfierMode.INTERNAL);
+sps.init(new IntraSPSNameNodeContext(this.namesystem, this, sps),
+new IntraSPSNameNodeFileIdCollector(this.namesystem.getFSDirectory(),
+sps),
+new 

[16/50] [abbrv] hadoop git commit: HDFS-12556: [SPS] : Block movement analysis should be done in read lock.

[rakeshr]

HDFS-12556: [SPS] : Block movement analysis should be done in read lock.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/691e544e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/691e544e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/691e544e

Branch: refs/heads/HDFS-10285
Commit: 691e544e577d8131def91cddd1dec56ac654554b
Parents: 48baf30
Author: Surendra Singh Lilhore 
Authored: Sat Oct 14 15:11:26 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:49 2018 +0530

--
 .../server/namenode/StoragePolicySatisfier.java | 27 +---
 .../TestPersistentStoragePolicySatisfier.java   |  2 +-
 2 files changed, 19 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/691e544e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
index a28a806..cbfba44 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
@@ -242,12 +242,25 @@ public class StoragePolicySatisfier implements Runnable {
   ItemInfo itemInfo = storageMovementNeeded.get();
   if (itemInfo != null) {
 long trackId = itemInfo.getTrackId();
-BlockCollection blockCollection =
-namesystem.getBlockCollection(trackId);
-// Check blockCollectionId existence.
+BlockCollection blockCollection;
+BlocksMovingAnalysis status = null;
+try {
+  namesystem.readLock();
+  blockCollection = namesystem.getBlockCollection(trackId);
+  // Check blockCollectionId existence.
+  if (blockCollection == null) {
+// File doesn't exists (maybe got deleted), remove trackId from
+// the queue
+storageMovementNeeded.removeItemTrackInfo(itemInfo);
+  } else {
+status =
+analyseBlocksStorageMovementsAndAssignToDN(
+blockCollection);
+  }
+} finally {
+  namesystem.readUnlock();
+}
 if (blockCollection != null) {
-  BlocksMovingAnalysis status =
-  analyseBlocksStorageMovementsAndAssignToDN(blockCollection);
   switch (status.status) {
   // Just add to monitor, so it will be retried after timeout
   case ANALYSIS_SKIPPED_FOR_RETRY:
@@ -283,10 +296,6 @@ public class StoragePolicySatisfier implements Runnable {
 storageMovementNeeded.removeItemTrackInfo(itemInfo);
 break;
   }
-} else {
-  // File doesn't exists (maybe got deleted), remove trackId from
-  // the queue
-  storageMovementNeeded.removeItemTrackInfo(itemInfo);
 }
   }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/691e544e/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPersistentStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPersistentStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPersistentStoragePolicySatisfier.java
index 5bce296..7165d06 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPersistentStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPersistentStoragePolicySatisfier.java
@@ -72,7 +72,7 @@ public class TestPersistentStoragePolicySatisfier {
   {StorageType.DISK, StorageType.ARCHIVE, StorageType.SSD}
   };
 
-  private final int timeout = 30;
+  private final int timeout = 9;
 
   /**
* Setup environment for every test case.


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[38/50] [abbrv] hadoop git commit: HDFS-13110: [SPS]: Reduce the number of APIs in NamenodeProtocol used by external satisfier. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/79925028/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSFileIDCollector.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSFileIDCollector.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSFileIDCollector.java
deleted file mode 100644
index ff277ba..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSFileIDCollector.java
+++ /dev/null
@@ -1,174 +0,0 @@
-package org.apache.hadoop.hdfs.server.sps;
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import java.io.IOException;
-import java.util.ArrayList;
-
-import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.DistributedFileSystem;
-import org.apache.hadoop.hdfs.protocol.DirectoryListing;
-import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
-import org.apache.hadoop.hdfs.server.namenode.sps.Context;
-import org.apache.hadoop.hdfs.server.namenode.sps.FileIdCollector;
-import org.apache.hadoop.hdfs.server.namenode.sps.ItemInfo;
-import org.apache.hadoop.hdfs.server.namenode.sps.SPSService;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * This class is to scan the paths recursively. If file is directory, then it
- * will scan for files recursively. If the file is non directory, then it will
- * just submit the same file to process.
- */
-@InterfaceAudience.Private
-public class ExternalSPSFileIDCollector implements FileIdCollector {
-  public static final Logger LOG =
-  LoggerFactory.getLogger(ExternalSPSFileIDCollector.class);
-  private Context cxt;
-  private DistributedFileSystem dfs;
-  private SPSService service;
-  private int maxQueueLimitToScan;
-
-  public ExternalSPSFileIDCollector(Context cxt, SPSService service) {
-this.cxt = cxt;
-this.service = service;
-this.maxQueueLimitToScan = service.getConf().getInt(
-DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_QUEUE_LIMIT_KEY,
-DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_QUEUE_LIMIT_DEFAULT);
-try {
-  // TODO: probably we could get this dfs from external context? but this 
is
-  // too specific to external.
-  dfs = getFS(service.getConf());
-} catch (IOException e) {
-  LOG.error("Unable to get the filesystem. Make sure Namenode running and "
-  + "configured namenode address is correct.", e);
-}
-  }
-
-  private DistributedFileSystem getFS(Configuration conf) throws IOException {
-return (DistributedFileSystem) FileSystem
-.get(FileSystem.getDefaultUri(conf), conf);
-  }
-
-  /**
-   * Recursively scan the given path and add the file info to SPS service for
-   * processing.
-   */
-  private long processPath(long startID, String fullPath) {
-long pendingWorkCount = 0; // to be satisfied file counter
-for (byte[] lastReturnedName = HdfsFileStatus.EMPTY_NAME;;) {
-  final DirectoryListing children;
-  try {
-children = dfs.getClient().listPaths(fullPath, lastReturnedName, 
false);
-  } catch (IOException e) {
-LOG.warn("Failed to list directory " + fullPath
-+ ". Ignore the directory and continue.", e);
-return pendingWorkCount;
-  }
-  if (children == null) {
-if (LOG.isDebugEnabled()) {
-  LOG.debug("The scanning start dir/sub dir " + fullPath
-  + " does not have childrens.");
-}
-return pendingWorkCount;
-  }
-
-  for (HdfsFileStatus child : children.getPartialListing()) {
-if (child.isFile()) {
-  service.addFileIdToProcess(new ItemInfo(startID, child.getFileId()),
-  false);
-  checkProcessingQueuesFree();
-  pendingWorkCount++; // increment to be satisfied file count
-} else {
-  String fullPathStr = child.getFullName(fullPath);
- 

[21/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e36384a4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
new file mode 100644
index 000..8dc52dc
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
@@ -0,0 +1,1779 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode.sps;
+
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_STORAGE_POLICY_ENABLED_KEY;
+import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.slf4j.LoggerFactory.getLogger;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.ReconfigurationException;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
+import org.apache.hadoop.hdfs.NameNodeProxies;
+import org.apache.hadoop.hdfs.StripedFileTestUtil;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
+import org.apache.hadoop.hdfs.protocol.LocatedBlock;
+import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
+import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
+import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
+import org.apache.hadoop.hdfs.server.datanode.DataNode;
+import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
+import org.apache.hadoop.hdfs.server.datanode.InternalDataNodeTestUtils;
+import org.apache.hadoop.hdfs.server.namenode.FSDirectory;
+import org.apache.hadoop.hdfs.server.namenode.FSEditLog;
+import org.apache.hadoop.hdfs.server.namenode.FSTreeTraverser;
+import org.apache.hadoop.hdfs.server.namenode.INode;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import com.google.common.base.Supplier;
+
+/**
+ * Tests that StoragePolicySatisfier daemon is able to check the blocks to be
+ * moved and finding its suggested target locations to move.
+ */
+public class TestStoragePolicySatisfier {
+
+  {
+GenericTestUtils.setLogLevel(
+getLogger(FSTreeTraverser.class), Level.DEBUG);
+  }
+
+  private static final String ONE_SSD = "ONE_SSD";
+  private static final String COLD = "COLD";
+  private static final Logger LOG =
+  LoggerFactory.getLogger(TestStoragePolicySatisfier.class);
+  private final Configuration config = new HdfsConfiguration();
+  private StorageType[][] allDiskTypes =
+ 

[41/50] [abbrv] hadoop git commit: HDFS-13165: [SPS]: Collects successfully moved block details via IBR. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/93a04547/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSBlockMoveTaskHandler.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSBlockMoveTaskHandler.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSBlockMoveTaskHandler.java
index 7580ba9..f5225d2 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSBlockMoveTaskHandler.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/sps/ExternalSPSBlockMoveTaskHandler.java
@@ -20,13 +20,10 @@ package org.apache.hadoop.hdfs.server.sps;
 
 import java.io.IOException;
 import java.net.Socket;
-import java.util.ArrayList;
-import java.util.List;
 import java.util.concurrent.Callable;
 import java.util.concurrent.CompletionService;
 import java.util.concurrent.ExecutorCompletionService;
 import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Future;
 import java.util.concurrent.SynchronousQueue;
 import java.util.concurrent.ThreadPoolExecutor;
 import java.util.concurrent.TimeUnit;
@@ -39,7 +36,6 @@ import org.apache.hadoop.fs.StorageType;
 import org.apache.hadoop.hdfs.DFSConfigKeys;
 import org.apache.hadoop.hdfs.DFSUtilClient;
 import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys;
-import org.apache.hadoop.hdfs.protocol.Block;
 import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import org.apache.hadoop.hdfs.protocol.datatransfer.TrustedChannelResolver;
@@ -48,15 +44,14 @@ import 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient;
 import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier;
 import org.apache.hadoop.hdfs.server.balancer.KeyManager;
 import org.apache.hadoop.hdfs.server.balancer.NameNodeConnector;
+import org.apache.hadoop.hdfs.server.common.sps.BlockDispatcher;
 import org.apache.hadoop.hdfs.server.common.sps.BlockMovementAttemptFinished;
 import org.apache.hadoop.hdfs.server.common.sps.BlockMovementStatus;
 import org.apache.hadoop.hdfs.server.common.sps.BlockStorageMovementTracker;
 import org.apache.hadoop.hdfs.server.common.sps.BlocksMovementsStatusHandler;
-import org.apache.hadoop.hdfs.server.common.sps.BlockDispatcher;
 import org.apache.hadoop.hdfs.server.namenode.sps.BlockMoveTaskHandler;
 import org.apache.hadoop.hdfs.server.namenode.sps.SPSService;
 import 
org.apache.hadoop.hdfs.server.protocol.BlockStorageMovementCommand.BlockMovingInfo;
-import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMoveAttemptFinished;
 import org.apache.hadoop.security.token.Token;
 import org.apache.hadoop.util.Daemon;
 import org.slf4j.Logger;
@@ -105,12 +100,14 @@ public class ExternalSPSBlockMoveTaskHandler implements 
BlockMoveTaskHandler {
 int ioFileBufferSize = DFSUtilClient.getIoFileBufferSize(conf);
 blkDispatcher = new BlockDispatcher(HdfsConstants.READ_TIMEOUT,
 ioFileBufferSize, connectToDnViaHostname);
+
+startMovementTracker();
   }
 
   /**
* Initializes block movement tracker daemon and starts the thread.
*/
-  public void init() {
+  private void startMovementTracker() {
 movementTrackerThread = new Daemon(this.blkMovementTracker);
 movementTrackerThread.setName("BlockStorageMovementTracker");
 movementTrackerThread.start();
@@ -156,24 +153,16 @@ public class ExternalSPSBlockMoveTaskHandler implements 
BlockMoveTaskHandler {
 // dn.incrementBlocksScheduled(blkMovingInfo.getTargetStorageType());
 LOG.debug("Received BlockMovingTask {}", blkMovingInfo);
 BlockMovingTask blockMovingTask = new BlockMovingTask(blkMovingInfo);
-Future moveCallable = mCompletionServ
-.submit(blockMovingTask);
-blkMovementTracker.addBlock(blkMovingInfo.getBlock(), moveCallable);
+mCompletionServ.submit(blockMovingTask);
   }
 
   private class ExternalBlocksMovementsStatusHandler
-  extends BlocksMovementsStatusHandler {
+  implements BlocksMovementsStatusHandler {
 @Override
-public void handle(
-List moveAttemptFinishedBlks) {
-  List blocks = new ArrayList<>();
-  for (BlockMovementAttemptFinished item : moveAttemptFinishedBlks) {
-blocks.add(item.getBlock());
-  }
-  BlocksStorageMoveAttemptFinished blkAttempted =
-  new BlocksStorageMoveAttemptFinished(
-  blocks.toArray(new Block[blocks.size()]));
-  service.notifyStorageMovementAttemptFinishedBlks(blkAttempted);
+public void handle(BlockMovementAttemptFinished attemptedMove) {
+  service.notifyStorageMovementAttemptFinishedBlk(
+  attemptedMove.getTargetDatanode(), attemptedMove.getTargetType(),
+  attemptedMove.getBlock());
 }
   }
 
@@ -194,6 +183,7 @@ public class 

[43/50] [abbrv] hadoop git commit: HDFS-13381 : [SPS]: Use DFSUtilClient#makePathFromFileId() to prepare satisfier file path. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8eabfc06/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
index b05717a..ec5307b 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
@@ -108,8 +108,6 @@ public class TestStoragePolicySatisfier {
   public static final long CAPACITY = 2 * 256 * 1024 * 1024;
   public static final String FILE = "/testMoveToSatisfyStoragePolicy";
   public static final int DEFAULT_BLOCK_SIZE = 1024;
-  private ExternalBlockMovementListener blkMoveListener =
-  new ExternalBlockMovementListener();
 
   /**
* Sets hdfs cluster.
@@ -1282,8 +1280,8 @@ public class TestStoragePolicySatisfier {
 
 //Queue limit can control the traverse logic to wait for some free
 //entry in queue. After 10 files, traverse control will be on U.
-StoragePolicySatisfier sps = new 
StoragePolicySatisfier(config);
-Context ctxt = new IntraSPSNameNodeContext(
+StoragePolicySatisfier sps = new StoragePolicySatisfier(config);
+Context ctxt = new IntraSPSNameNodeContext(
 hdfsCluster.getNamesystem(),
 hdfsCluster.getNamesystem().getBlockManager(), sps) {
   @Override
@@ -1297,8 +1295,7 @@ public class TestStoragePolicySatisfier {
   }
 };
 
-FileCollector fileIDCollector = createFileIdCollector(sps, ctxt);
-sps.init(ctxt, fileIDCollector, null, null);
+sps.init(ctxt);
 sps.getStorageMovementQueue().activate();
 
 INode rootINode = fsDir.getINode("/root");
@@ -1314,13 +1311,6 @@ public class TestStoragePolicySatisfier {
 dfs.delete(new Path("/root"), true);
   }
 
-  public FileCollector createFileIdCollector(
-  StoragePolicySatisfier sps, Context ctxt) {
-FileCollector fileIDCollector = new IntraSPSNameNodeFileIdCollector(
-hdfsCluster.getNamesystem().getFSDirectory(), sps);
-return fileIDCollector;
-  }
-
   /**
*  Test traverse when root parent got deleted.
*  1. Delete L when traversing Q
@@ -1351,8 +1341,8 @@ public class TestStoragePolicySatisfier {
 
 // Queue limit can control the traverse logic to wait for some free
 // entry in queue. After 10 files, traverse control will be on U.
-StoragePolicySatisfier sps = new 
StoragePolicySatisfier(config);
-Context ctxt = new IntraSPSNameNodeContext(
+StoragePolicySatisfier sps = new StoragePolicySatisfier(config);
+Context ctxt = new IntraSPSNameNodeContext(
 hdfsCluster.getNamesystem(),
 hdfsCluster.getNamesystem().getBlockManager(), sps) {
   @Override
@@ -1365,8 +1355,7 @@ public class TestStoragePolicySatisfier {
 return true;
   }
 };
-FileCollector fileIDCollector = createFileIdCollector(sps, ctxt);
-sps.init(ctxt, fileIDCollector, null, null);
+sps.init(ctxt);
 sps.getStorageMovementQueue().activate();
 
 INode rootINode = fsDir.getINode("/root");
@@ -1383,12 +1372,12 @@ public class TestStoragePolicySatisfier {
   }
 
   private void assertTraversal(List expectedTraverseOrder,
-  FSDirectory fsDir, StoragePolicySatisfier sps)
+  FSDirectory fsDir, StoragePolicySatisfier sps)
   throws InterruptedException {
 // Remove 10 element and make queue free, So other traversing will start.
 for (int i = 0; i < 10; i++) {
   String path = expectedTraverseOrder.remove(0);
-  ItemInfo itemInfo = sps.getStorageMovementQueue().get();
+  ItemInfo itemInfo = sps.getStorageMovementQueue().get();
   if (itemInfo == null) {
 continue;
   }
@@ -1403,7 +1392,7 @@ public class TestStoragePolicySatisfier {
 // Check other element traversed in order and E, M, U, R, S should not be
 // added in queue which we already removed from expected list
 for (String path : expectedTraverseOrder) {
-  ItemInfo itemInfo = sps.getStorageMovementQueue().get();
+  ItemInfo itemInfo = sps.getStorageMovementQueue().get();
   if (itemInfo == null) {
 continue;
   }
@@ -1717,17 +1706,17 @@ public class TestStoragePolicySatisfier {
   public void waitForAttemptedItems(long expectedBlkMovAttemptedCount,
   int timeout) throws TimeoutException, InterruptedException {
 BlockManager blockManager = hdfsCluster.getNamesystem().getBlockManager();
-final StoragePolicySatisfier sps =
-(StoragePolicySatisfier) blockManager.getSPSManager()
+final StoragePolicySatisfier sps =
+

[17/50] [abbrv] hadoop git commit: HDFS-12310: [SPS]: Provide an option to track the status of in progress requests. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-12310: [SPS]: Provide an option to track the status of in progress 
requests. Contributed by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8e0a5fb4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8e0a5fb4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8e0a5fb4

Branch: refs/heads/HDFS-10285
Commit: 8e0a5fb4453098946d9706f8f41d28ef48cb9882
Parents: 691e544
Author: Rakesh Radhakrishnan 
Authored: Fri Nov 3 08:18:14 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:54 2018 +0530

--
 .../java/org/apache/hadoop/hdfs/DFSClient.java  |  22 
 .../hadoop/hdfs/protocol/ClientProtocol.java|  21 
 .../hadoop/hdfs/protocol/HdfsConstants.java |  27 +
 .../ClientNamenodeProtocolTranslatorPB.java |  20 
 .../hadoop/hdfs/protocolPB/PBHelperClient.java  |  33 ++
 .../src/main/proto/ClientNamenodeProtocol.proto |  17 ++-
 ...tNamenodeProtocolServerSideTranslatorPB.java |  23 +++-
 .../server/blockmanagement/BlockManager.java|  12 ++
 .../namenode/BlockStorageMovementNeeded.java| 109 +++
 .../hdfs/server/namenode/NameNodeRpcServer.java |  13 ++-
 .../server/namenode/StoragePolicySatisfier.java |   8 ++
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   |  35 +-
 .../src/site/markdown/ArchivalStorage.md|   3 +-
 .../TestPersistentStoragePolicySatisfier.java   |   2 +-
 .../namenode/TestStoragePolicySatisfier.java|  67 
 .../hdfs/tools/TestStoragePolicyCommands.java   |  18 +++
 16 files changed, 424 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/8e0a5fb4/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
index 7337aa2..471ab2c 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@@ -123,6 +123,7 @@ import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.ReencryptAction;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.RollingUpgradeAction;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction;
 import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
 import org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus;
@@ -3169,4 +3170,25 @@ public class DFSClient implements java.io.Closeable, 
RemotePeerFactory,
 checkOpen();
 return new OpenFilesIterator(namenode, tracer, openFilesTypes, path);
   }
+
+  /**
+   * Check the storage policy satisfy status of the path for which
+   * {@link DFSClient#satisfyStoragePolicy(String)} is called.
+   *
+   * @return Storage policy satisfy status.
+   * 
+   * PENDING if path is in queue and not processed for satisfying
+   * the policy.
+   * IN_PROGRESS if satisfying the storage policy for path.
+   * SUCCESS if storage policy satisfied for the path.
+   * NOT_AVAILABLE if
+   * {@link DFSClient#satisfyStoragePolicy(String)} not called for
+   * path or SPS work is already finished.
+   * 
+   * @throws IOException
+   */
+  public StoragePolicySatisfyPathStatus checkStoragePolicySatisfyPathStatus(
+  String path) throws IOException {
+return namenode.checkStoragePolicySatisfyPathStatus(path);
+  }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8e0a5fb4/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
index 81d7c91..360fd63 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
@@ -45,6 +45,7 @@ import org.apache.hadoop.hdfs.inotify.EventBatchList;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.ReencryptAction;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.RollingUpgradeAction;
 

[19/50] [abbrv] hadoop git commit: HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed 
by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/80ab2ede
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/80ab2ede
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/80ab2ede

Branch: refs/heads/HDFS-10285
Commit: 80ab2edeafc6fbd4e40a3923111c6e965fbe35ca
Parents: ec9a37a
Author: Surendra Singh Lilhore 
Authored: Wed Nov 15 20:22:27 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:02 2018 +0530

--
 .../hadoop/hdfs/protocol/HdfsConstants.java |  6 +++
 .../hadoop/hdfs/protocolPB/PBHelperClient.java  |  4 ++
 .../src/main/proto/ClientNamenodeProtocol.proto |  3 +-
 .../org/apache/hadoop/hdfs/DFSConfigKeys.java   | 10 ++--
 .../server/blockmanagement/DatanodeManager.java | 12 ++---
 .../datanode/StoragePolicySatisfyWorker.java|  3 +-
 .../BlockStorageMovementAttemptedItems.java |  8 +--
 .../namenode/BlockStorageMovementNeeded.java| 46 
 .../hdfs/server/namenode/FSNamesystem.java  |  3 ++
 .../server/namenode/StoragePolicySatisfier.java | 42 ---
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   | 27 +++---
 .../src/main/resources/hdfs-default.xml | 17 --
 .../src/site/markdown/ArchivalStorage.md|  2 +-
 .../TestBlockStorageMovementAttemptedItems.java | 10 ++--
 .../namenode/TestStoragePolicySatisfier.java| 57 ++--
 15 files changed, 199 insertions(+), 51 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/80ab2ede/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
index 190a1c6..aabcdd9 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
@@ -150,6 +150,12 @@ public final class HdfsConstants {
 SUCCESS,
 
 /**
+ * Few blocks failed to move and the path is still not
+ * fully satisfied the storage policy.
+ */
+FAILURE,
+
+/**
  * Status not available.
  */
 NOT_AVAILABLE

http://git-wip-us.apache.org/repos/asf/hadoop/blob/80ab2ede/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
index 582693f..84812ab 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
@@ -3388,6 +3388,8 @@ public class PBHelperClient {
   return StoragePolicySatisfyPathStatus.IN_PROGRESS;
 case SUCCESS:
   return StoragePolicySatisfyPathStatus.SUCCESS;
+case FAILURE:
+  return StoragePolicySatisfyPathStatus.FAILURE;
 case NOT_AVAILABLE:
   return StoragePolicySatisfyPathStatus.NOT_AVAILABLE;
 default:
@@ -3404,6 +3406,8 @@ public class PBHelperClient {
   return HdfsConstants.StoragePolicySatisfyPathStatus.IN_PROGRESS;
 case SUCCESS:
   return HdfsConstants.StoragePolicySatisfyPathStatus.SUCCESS;
+case FAILURE:
+  return HdfsConstants.StoragePolicySatisfyPathStatus.FAILURE;
 case NOT_AVAILABLE:
   return HdfsConstants.StoragePolicySatisfyPathStatus.NOT_AVAILABLE;
 default:

http://git-wip-us.apache.org/repos/asf/hadoop/blob/80ab2ede/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
index 0484e75..154891d 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
@@ -851,7 +851,8 @@ message CheckStoragePolicySatisfyPathStatusResponseProto {
 PENDING = 0;
 IN_PROGRESS = 1;
 SUCCESS = 2;
-NOT_AVAILABLE = 3;
+

[11/50] [abbrv] hadoop git commit: HDFS-12225: [SPS]: Optimize extended attributes for tracking SPS movements. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-12225: [SPS]: Optimize extended attributes for tracking SPS movements. 
Contributed by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e934b3ed
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e934b3ed
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e934b3ed

Branch: refs/heads/HDFS-10285
Commit: e934b3edda0409329d5627e4a782d551f6e626e6
Parents: ba09ab3
Author: Uma Maheswara Rao G 
Authored: Wed Aug 23 15:37:03 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:37 2018 +0530

--
 .../server/blockmanagement/BlockManager.java|  21 +-
 .../server/blockmanagement/DatanodeManager.java |  14 +-
 .../hdfs/server/datanode/BPOfferService.java|   1 +
 .../BlockStorageMovementAttemptedItems.java |  95 +---
 .../namenode/BlockStorageMovementNeeded.java| 233 ++-
 .../namenode/FSDirSatisfyStoragePolicyOp.java   |  91 +++-
 .../hdfs/server/namenode/FSDirXAttrOp.java  |  11 +-
 .../hdfs/server/namenode/FSDirectory.java   |   2 +-
 .../hdfs/server/namenode/FSNamesystem.java  |   2 +-
 .../server/namenode/StoragePolicySatisfier.java | 108 ++---
 .../TestStoragePolicySatisfyWorker.java |   5 +-
 .../TestBlockStorageMovementAttemptedItems.java |  34 +--
 .../TestPersistentStoragePolicySatisfier.java   | 104 +
 .../namenode/TestStoragePolicySatisfier.java| 127 +-
 14 files changed, 589 insertions(+), 259 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/e934b3ed/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index a442a92..0ee558a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -89,7 +89,6 @@ import 
org.apache.hadoop.hdfs.server.namenode.INode.BlocksMapUpdateInfo;
 import org.apache.hadoop.hdfs.server.namenode.INodesInPath;
 import org.apache.hadoop.hdfs.server.namenode.NameNode;
 import org.apache.hadoop.hdfs.server.namenode.Namesystem;
-import org.apache.hadoop.hdfs.server.namenode.BlockStorageMovementNeeded;
 import org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier;
 import org.apache.hadoop.hdfs.server.namenode.ha.HAContext;
 import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
@@ -431,9 +430,6 @@ public class BlockManager implements BlockStatsMXBean {
   private final StoragePolicySatisfier sps;
   private final boolean storagePolicyEnabled;
   private boolean spsEnabled;
-  private final BlockStorageMovementNeeded storageMovementNeeded =
-  new BlockStorageMovementNeeded();
-
   /** Minimum live replicas needed for the datanode to be transitioned
* from ENTERING_MAINTENANCE to IN_MAINTENANCE.
*/
@@ -480,8 +476,7 @@ public class BlockManager implements BlockStatsMXBean {
 conf.getBoolean(
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_DEFAULT);
-sps = new StoragePolicySatisfier(namesystem, storageMovementNeeded, this,
-conf);
+sps = new StoragePolicySatisfier(namesystem, this, conf);
 blockTokenSecretManager = createBlockTokenSecretManager(conf);
 
 providedStorageMap = new ProvidedStorageMap(namesystem, this, conf);
@@ -5009,20 +5004,6 @@ public class BlockManager implements BlockStatsMXBean {
   }
 
   /**
-   * Set file block collection for which storage movement needed for its 
blocks.
-   *
-   * @param id
-   *  - file block collection id.
-   */
-  public void satisfyStoragePolicy(long id) {
-storageMovementNeeded.add(id);
-if (LOG.isDebugEnabled()) {
-  LOG.debug("Added block collection id {} to block "
-  + "storageMovementNeeded queue", id);
-}
-  }
-
-  /**
* Gets the storage policy satisfier instance.
*
* @return sps

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e934b3ed/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
 

[49/50] [abbrv] hadoop git commit: HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5bd5714/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeBlockMoveTaskHandler.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeBlockMoveTaskHandler.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeBlockMoveTaskHandler.java
deleted file mode 100644
index d6e92d2..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeBlockMoveTaskHandler.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.hdfs.server.namenode.sps;
-
-import java.io.IOException;
-
-import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
-import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
-import org.apache.hadoop.hdfs.server.namenode.Namesystem;
-import 
org.apache.hadoop.hdfs.server.protocol.BlockStorageMovementCommand.BlockMovingInfo;
-
-/**
- * This class handles the internal SPS block movements. This will assign block
- * movement tasks to target datanode descriptors.
- */
-@InterfaceAudience.Private
-public class IntraSPSNameNodeBlockMoveTaskHandler
-implements BlockMoveTaskHandler {
-
-  private BlockManager blockManager;
-  private Namesystem namesystem;
-
-  public IntraSPSNameNodeBlockMoveTaskHandler(BlockManager blockManager,
-  Namesystem namesytem) {
-this.blockManager = blockManager;
-this.namesystem = namesytem;
-  }
-
-  @Override
-  public void submitMoveTask(BlockMovingInfo blkMovingInfo) throws IOException 
{
-namesystem.readLock();
-try {
-  DatanodeDescriptor dn = blockManager.getDatanodeManager()
-  .getDatanode(blkMovingInfo.getTarget().getDatanodeUuid());
-  if (dn == null) {
-throw new IOException("Failed to schedule block movement task:"
-+ blkMovingInfo + " as target datanode: "
-+ blkMovingInfo.getTarget() + " doesn't exists");
-  }
-  dn.incrementBlocksScheduled(blkMovingInfo.getTargetStorageType());
-  dn.addBlocksToMoveStorage(blkMovingInfo);
-} finally {
-  namesystem.readUnlock();
-}
-  }
-}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5bd5714/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeContext.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeContext.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeContext.java
deleted file mode 100644
index 2bf4810..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/IntraSPSNameNodeContext.java
+++ /dev/null
@@ -1,189 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.hdfs.server.namenode.sps;
-
-import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
-
-import java.io.IOException;
-import 

[30/50] [abbrv] hadoop git commit: HDFS-13033: [SPS]: Implement a mechanism to do file block movements for external SPS. Contributed by Rakesh R.

[rakeshr]

HDFS-13033: [SPS]: Implement a mechanism to do file block movements for 
external SPS. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4ed7b37b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4ed7b37b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4ed7b37b

Branch: refs/heads/HDFS-10285
Commit: 4ed7b37b112791bd86792706c022a163efb6e84e
Parents: febf510
Author: Uma Maheswara Rao G 
Authored: Tue Jan 23 16:19:46 2018 -0800
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:23 2018 +0530

--
 .../hdfs/server/balancer/NameNodeConnector.java |   8 +
 .../hdfs/server/common/sps/BlockDispatcher.java | 186 +
 .../sps/BlockMovementAttemptFinished.java   |  80 ++
 .../server/common/sps/BlockMovementStatus.java  |  53 
 .../common/sps/BlockStorageMovementTracker.java | 184 +
 .../sps/BlocksMovementsStatusHandler.java   |  95 +++
 .../hdfs/server/common/sps/package-info.java|  27 ++
 .../datanode/BlockStorageMovementTracker.java   | 186 -
 .../datanode/StoragePolicySatisfyWorker.java| 271 ++-
 .../hdfs/server/namenode/FSNamesystem.java  |   4 +-
 .../namenode/sps/BlockMoveTaskHandler.java  |   3 +-
 .../sps/BlockStorageMovementAttemptedItems.java |  12 +-
 .../IntraSPSNameNodeBlockMoveTaskHandler.java   |   3 +-
 .../hdfs/server/namenode/sps/SPSService.java|  14 +-
 .../namenode/sps/StoragePolicySatisfier.java|  30 +-
 .../sps/ExternalSPSBlockMoveTaskHandler.java| 233 
 .../TestBlockStorageMovementAttemptedItems.java |   2 +-
 .../sps/TestStoragePolicySatisfier.java |   6 +-
 .../sps/TestExternalStoragePolicySatisfier.java |  69 -
 19 files changed, 997 insertions(+), 469 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/4ed7b37b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
index b0dd779..6bfbbb3 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
@@ -269,6 +269,14 @@ public class NameNodeConnector implements Closeable {
 }
   }
 
+  /**
+   * Returns fallbackToSimpleAuth. This will be true or false during calls to
+   * indicate if a secure client falls back to simple auth.
+   */
+  public AtomicBoolean getFallbackToSimpleAuth() {
+return fallbackToSimpleAuth;
+  }
+
   @Override
   public void close() {
 keyManager.close();

http://git-wip-us.apache.org/repos/asf/hadoop/blob/4ed7b37b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/sps/BlockDispatcher.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/sps/BlockDispatcher.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/sps/BlockDispatcher.java
new file mode 100644
index 000..f87fcae
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/sps/BlockDispatcher.java
@@ -0,0 +1,186 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.common.sps;
+
+import static org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import 

[25/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by 
Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e36384a4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e36384a4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e36384a4

Branch: refs/heads/HDFS-10285
Commit: e36384a42777ccc658dbc1be37e5a964bf5dec32
Parents: 80ab2ed
Author: Uma Maheswara Rao G 
Authored: Fri Dec 22 09:10:12 2017 -0800
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:06 2018 +0530

--
 .../server/blockmanagement/BlockManager.java|6 +-
 .../BlockStorageMovementAttemptedItems.java |  241 ---
 .../namenode/BlockStorageMovementNeeded.java|  574 --
 .../hdfs/server/namenode/FSNamesystem.java  |1 +
 .../hdfs/server/namenode/IntraNNSPSContext.java |   41 +
 .../server/namenode/StoragePolicySatisfier.java |  973 --
 .../sps/BlockStorageMovementAttemptedItems.java |  241 +++
 .../sps/BlockStorageMovementNeeded.java |  572 ++
 .../namenode/sps/StoragePolicySatisfier.java|  988 ++
 .../hdfs/server/namenode/sps/package-info.java  |   28 +
 .../TestBlockStorageMovementAttemptedItems.java |  196 --
 .../namenode/TestStoragePolicySatisfier.java| 1775 -
 ...stStoragePolicySatisfierWithStripedFile.java |  580 --
 .../TestBlockStorageMovementAttemptedItems.java |  196 ++
 .../sps/TestStoragePolicySatisfier.java | 1779 ++
 ...stStoragePolicySatisfierWithStripedFile.java |  580 ++
 16 files changed, 4430 insertions(+), 4341 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/e36384a4/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index c81ed6c..1cf687e 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -89,11 +89,12 @@ import 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.ReplicaState;
 import org.apache.hadoop.hdfs.server.namenode.CachedBlock;
 import org.apache.hadoop.hdfs.server.namenode.INode.BlocksMapUpdateInfo;
 import org.apache.hadoop.hdfs.server.namenode.INodesInPath;
+import org.apache.hadoop.hdfs.server.namenode.IntraNNSPSContext;
 import org.apache.hadoop.hdfs.server.namenode.NameNode;
 import org.apache.hadoop.hdfs.server.namenode.Namesystem;
-import org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier;
 import org.apache.hadoop.hdfs.server.namenode.ha.HAContext;
 import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
+import org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier;
 import org.apache.hadoop.hdfs.server.protocol.BlockCommand;
 import org.apache.hadoop.hdfs.server.protocol.BlockReportContext;
 import org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations;
@@ -478,7 +479,8 @@ public class BlockManager implements BlockStatsMXBean {
 conf.getBoolean(
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_DEFAULT);
-sps = new StoragePolicySatisfier(namesystem, this, conf);
+StoragePolicySatisfier.Context spsctxt = new IntraNNSPSContext(namesystem);
+sps = new StoragePolicySatisfier(namesystem, this, conf, spsctxt);
 blockTokenSecretManager = createBlockTokenSecretManager(conf);
 
 providedStorageMap = new ProvidedStorageMap(namesystem, this, conf);

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e36384a4/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
deleted file mode 100644
index 643255f..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
+++ /dev/null
@@ -1,241 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work 

[20/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e36384a4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
new file mode 100644
index 000..c1a2b8b
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
@@ -0,0 +1,580 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode.sps;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
+import org.apache.hadoop.hdfs.NameNodeProxies;
+import org.apache.hadoop.hdfs.StripedFileTestUtil;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
+import org.apache.hadoop.hdfs.protocol.LocatedBlock;
+import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Supplier;
+
+/**
+ * Tests that StoragePolicySatisfier daemon is able to check the striped blocks
+ * to be moved and finding its expected target locations in order to satisfy 
the
+ * storage policy.
+ */
+public class TestStoragePolicySatisfierWithStripedFile {
+
+  private static final Logger LOG = LoggerFactory
+  .getLogger(TestStoragePolicySatisfierWithStripedFile.class);
+
+  private final int stripesPerBlock = 2;
+
+  private ErasureCodingPolicy ecPolicy;
+  private int dataBlocks;
+  private int parityBlocks;
+  private int cellSize;
+  private int defaultStripeBlockSize;
+
+  private ErasureCodingPolicy getEcPolicy() {
+return StripedFileTestUtil.getDefaultECPolicy();
+  }
+
+  /**
+   * Initialize erasure coding policy.
+   */
+  @Before
+  public void init(){
+ecPolicy = getEcPolicy();
+dataBlocks = ecPolicy.getNumDataUnits();
+parityBlocks = ecPolicy.getNumParityUnits();
+cellSize = ecPolicy.getCellSize();
+defaultStripeBlockSize = cellSize * stripesPerBlock;
+  }
+
+  /**
+   * Tests to verify that all the striped blocks(data + parity blocks) are
+   * moving to satisfy the storage policy.
+   */
+  @Test(timeout = 30)
+  public void testMoverWithFullStripe() throws Exception {
+// start 10 datanodes
+int numOfDatanodes = 10;
+int storagesPerDatanode = 2;
+long capacity = 20 * defaultStripeBlockSize;
+long[][] capacities = new long[numOfDatanodes][storagesPerDatanode];
+for (int i = 0; i < numOfDatanodes; i++) {
+  for (int j = 0; j < storagesPerDatanode; j++) {
+capacities[i][j] = capacity;
+  }
+}
+
+final Configuration conf = new HdfsConfiguration();
+conf.setBoolean(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
+true);
+initConfWithStripe(conf, defaultStripeBlockSize);
+final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+.numDataNodes(numOfDatanodes)
+

[28/50] [abbrv] hadoop git commit: HDFS-12911. [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations. Contributed by Uma Maheswara Rao G

[rakeshr]

HDFS-12911. [SPS]: Modularize the SPS code and expose necessary interfaces for 
external/internal implementations. Contributed by Uma Maheswara Rao G


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a0c8e48c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a0c8e48c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a0c8e48c

Branch: refs/heads/HDFS-10285
Commit: a0c8e48c40e31d741b192a649d1a7773e2384437
Parents: 113185e
Author: Rakesh Radhakrishnan 
Authored: Fri Jan 19 08:51:49 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:15 2018 +0530

--
 .../server/blockmanagement/BlockManager.java|  61 +-
 .../namenode/FSDirSatisfyStoragePolicyOp.java   |  16 +-
 .../hdfs/server/namenode/FSDirectory.java   |   6 +-
 .../hdfs/server/namenode/FSNamesystem.java  |  10 +-
 .../namenode/sps/BlockMoveTaskHandler.java  |  44 
 .../namenode/sps/BlockMovementListener.java |  40 
 .../sps/BlockStorageMovementAttemptedItems.java |  28 +--
 .../sps/BlockStorageMovementNeeded.java | 207 ---
 .../hdfs/server/namenode/sps/Context.java   |  43 ++--
 .../server/namenode/sps/FileIdCollector.java|  43 
 .../IntraSPSNameNodeBlockMoveTaskHandler.java   |  62 ++
 .../namenode/sps/IntraSPSNameNodeContext.java   |  62 ++
 .../sps/IntraSPSNameNodeFileIdCollector.java| 178 
 .../hdfs/server/namenode/sps/ItemInfo.java  |  81 
 .../hdfs/server/namenode/sps/SPSPathIds.java|  63 ++
 .../hdfs/server/namenode/sps/SPSService.java| 107 ++
 .../namenode/sps/StoragePolicySatisfier.java| 175 +++-
 .../TestBlockStorageMovementAttemptedItems.java |  19 +-
 .../sps/TestStoragePolicySatisfier.java | 111 ++
 ...stStoragePolicySatisfierWithStripedFile.java |  19 +-
 20 files changed, 938 insertions(+), 437 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/a0c8e48c/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index c2d5162..63117ce 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -93,8 +93,8 @@ import org.apache.hadoop.hdfs.server.namenode.NameNode;
 import org.apache.hadoop.hdfs.server.namenode.Namesystem;
 import org.apache.hadoop.hdfs.server.namenode.ha.HAContext;
 import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
-import org.apache.hadoop.hdfs.server.namenode.sps.Context;
-import org.apache.hadoop.hdfs.server.namenode.sps.IntraSPSNameNodeContext;
+import org.apache.hadoop.hdfs.server.namenode.sps.SPSPathIds;
+import org.apache.hadoop.hdfs.server.namenode.sps.SPSService;
 import org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier;
 import org.apache.hadoop.hdfs.server.protocol.BlockCommand;
 import org.apache.hadoop.hdfs.server.protocol.BlockReportContext;
@@ -434,7 +434,8 @@ public class BlockManager implements BlockStatsMXBean {
   private final StoragePolicySatisfier sps;
   private final boolean storagePolicyEnabled;
   private boolean spsEnabled;
-  private Context spsctxt = null;
+  private final SPSPathIds spsPaths;
+
   /** Minimum live replicas needed for the datanode to be transitioned
* from ENTERING_MAINTENANCE to IN_MAINTENANCE.
*/
@@ -481,8 +482,8 @@ public class BlockManager implements BlockStatsMXBean {
 conf.getBoolean(
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_DEFAULT);
-spsctxt = new IntraSPSNameNodeContext(namesystem, this, conf);
-sps = new StoragePolicySatisfier(spsctxt);
+sps = new StoragePolicySatisfier(conf);
+spsPaths = new SPSPathIds();
 blockTokenSecretManager = createBlockTokenSecretManager(conf);
 
 providedStorageMap = new ProvidedStorageMap(namesystem, this, conf);
@@ -5033,8 +5034,7 @@ public class BlockManager implements BlockStatsMXBean {
   LOG.info("Storage policy satisfier is already running.");
   return;
 }
-// TODO: FSDirectory will get removed via HDFS-12911 modularization work
-sps.start(false, namesystem.getFSDirectory());
+sps.start(false);
   }
 
   /**
@@ -5070,8 +5070,7 @@ public class BlockManager implements BlockStatsMXBean {
   

[44/50] [abbrv] hadoop git commit: HDFS-13381 : [SPS]: Use DFSUtilClient#makePathFromFileId() to prepare satisfier file path. Contributed by Rakesh R.

[rakeshr]

HDFS-13381 : [SPS]: Use DFSUtilClient#makePathFromFileId() to prepare satisfier 
file path. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8eabfc06
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8eabfc06
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8eabfc06

Branch: refs/heads/HDFS-10285
Commit: 8eabfc0698304b691c51512aaa511be21a74d56f
Parents: 93a0454
Author: Uma Maheswara Rao G 
Authored: Mon Jul 2 17:22:00 2018 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:11:01 2018 +0530

--
 .../NamenodeProtocolServerSideTranslatorPB.java |  2 +-
 .../NamenodeProtocolTranslatorPB.java   |  2 +-
 .../server/blockmanagement/BlockManager.java|  2 +-
 .../hdfs/server/namenode/FSNamesystem.java  | 11 ---
 .../hdfs/server/namenode/NameNodeRpcServer.java |  8 +-
 .../hadoop/hdfs/server/namenode/Namesystem.java |  9 ---
 .../sps/BlockStorageMovementAttemptedItems.java | 72 +++--
 .../sps/BlockStorageMovementNeeded.java | 61 ++
 .../hdfs/server/namenode/sps/Context.java   | 45 ---
 .../namenode/sps/DatanodeCacheManager.java  |  4 +-
 .../hdfs/server/namenode/sps/FileCollector.java | 13 +--
 .../namenode/sps/IntraSPSNameNodeContext.java   | 54 +
 .../sps/IntraSPSNameNodeFileIdCollector.java| 14 ++--
 .../hdfs/server/namenode/sps/ItemInfo.java  | 34 
 .../hdfs/server/namenode/sps/SPSService.java| 31 +++
 .../namenode/sps/StoragePolicySatisfier.java| 61 +-
 .../sps/StoragePolicySatisfyManager.java| 20 ++---
 .../hdfs/server/protocol/NamenodeProtocol.java  |  2 +-
 .../sps/ExternalSPSBlockMoveTaskHandler.java|  4 +-
 .../hdfs/server/sps/ExternalSPSContext.java | 85 
 .../sps/ExternalSPSFilePathCollector.java   | 36 +
 .../sps/ExternalStoragePolicySatisfier.java | 30 +--
 .../src/main/proto/NamenodeProtocol.proto   |  2 +-
 .../TestBlockStorageMovementAttemptedItems.java | 16 ++--
 .../sps/TestStoragePolicySatisfier.java | 66 +--
 ...stStoragePolicySatisfierWithStripedFile.java | 41 --
 .../sps/TestExternalStoragePolicySatisfier.java | 35 +++-
 27 files changed, 346 insertions(+), 414 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/8eabfc06/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
index e4283c6..d9367fb 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
@@ -265,7 +265,7 @@ public class NamenodeProtocolServerSideTranslatorPB 
implements
   RpcController controller, GetNextSPSPathRequestProto request)
   throws ServiceException {
 try {
-  String nextSPSPath = impl.getNextSPSPath();
+  Long nextSPSPath = impl.getNextSPSPath();
   if (nextSPSPath == null) {
 return GetNextSPSPathResponseProto.newBuilder().build();
   }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8eabfc06/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
index 97dee9b..3bd5986 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
@@ -267,7 +267,7 @@ public class NamenodeProtocolTranslatorPB implements 
NamenodeProtocol,
   }
 
   @Override
-  public String getNextSPSPath() throws IOException {
+  public Long getNextSPSPath() throws IOException {
 GetNextSPSPathRequestProto req =
 GetNextSPSPathRequestProto.newBuilder().build();
 try {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8eabfc06/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

[47/50] [abbrv] hadoop git commit: HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5bd5714/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
index 18acb50..d9a93fd 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
@@ -32,34 +32,57 @@ import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_SPS_ADDRESS_KEY;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_SPS_KERBEROS_PRINCIPAL_KEY;
 import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_SPS_KEYTAB_FILE_KEY;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_SPS_MAX_OUTSTANDING_PATHS_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_STORAGE_POLICY_ENABLED_KEY;
 import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY;
 import static 
org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_DATA_TRANSFER_PROTECTION_KEY;
+import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.fail;
 
 import java.io.File;
+import java.io.FileNotFoundException;
 import java.io.IOException;
-import java.net.URI;
+import java.net.InetSocketAddress;
 import java.security.PrivilegedExceptionAction;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.List;
 import java.util.Properties;
 import java.util.concurrent.TimeoutException;
 
+import org.apache.commons.logging.LogFactory;
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.FileUtil;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.fs.permission.FsPermission;
 import org.apache.hadoop.hdfs.DFSConfigKeys;
 import org.apache.hadoop.hdfs.DFSTestUtil;
-import org.apache.hadoop.hdfs.DFSUtil;
 import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
 import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.NameNodeProxies;
+import org.apache.hadoop.hdfs.StripedFileTestUtil;
+import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
+import org.apache.hadoop.hdfs.protocol.LocatedBlock;
+import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
 import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode;
 import org.apache.hadoop.hdfs.server.balancer.NameNodeConnector;
 import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
+import org.apache.hadoop.hdfs.server.datanode.DataNode;
+import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
+import org.apache.hadoop.hdfs.server.datanode.InternalDataNodeTestUtils;
+import org.apache.hadoop.hdfs.server.namenode.FSEditLog;
+import org.apache.hadoop.hdfs.server.namenode.INode;
+import org.apache.hadoop.hdfs.server.namenode.sps.BlockMovementListener;
 import 
org.apache.hadoop.hdfs.server.namenode.sps.BlockStorageMovementAttemptedItems;
 import org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier;
-import org.apache.hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfier;
 import org.apache.hadoop.http.HttpConfig;
 import org.apache.hadoop.minikdc.MiniKdc;
 import org.apache.hadoop.security.SecurityUtil;
@@ -67,29 +90,57 @@ import org.apache.hadoop.security.UserGroupInformation;
 import org.apache.hadoop.security.authentication.util.KerberosName;
 import org.apache.hadoop.security.ssl.KeyStoreTestUtil;
 import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
 import org.junit.After;
 import org.junit.Assert;
+import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import com.google.common.base.Supplier;
 
 /**
  * Tests the external sps service plugins.
  */
-public class TestExternalStoragePolicySatisfier
-extends TestStoragePolicySatisfier {
+public class TestExternalStoragePolicySatisfier {
+  private static final String ONE_SSD = "ONE_SSD";
+  private static final String COLD = "COLD";
   private StorageType[][] allDiskTypes =
  

[37/50] [abbrv] hadoop git commit: HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by Surendra Singh.

[rakeshr]

HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by 
Surendra Singh.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/87e125a6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/87e125a6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/87e125a6

Branch: refs/heads/HDFS-10285
Commit: 87e125a6127b7e8ec599a1778b7bb195d6b186a9
Parents: 12e3125
Author: Uma Maheswara Rao G 
Authored: Wed Feb 7 02:28:23 2018 -0800
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:43 2018 +0530

--
 .../java/org/apache/hadoop/hdfs/DFSClient.java  |   4 +-
 .../hadoop/hdfs/protocol/ClientProtocol.java|   6 +-
 .../ClientNamenodeProtocolTranslatorPB.java |  14 +-
 .../src/main/proto/ClientNamenodeProtocol.proto |   8 +-
 .../federation/router/RouterRpcServer.java  |   2 +-
 .../java/org/apache/hadoop/hdfs/DFSUtil.java|  61 ---
 ...tNamenodeProtocolServerSideTranslatorPB.java |  16 +-
 .../server/blockmanagement/BlockManager.java| 255 +---
 .../blockmanagement/DatanodeDescriptor.java |  33 +-
 .../hdfs/server/common/HdfsServerConstants.java |   2 +-
 .../datanode/StoragePolicySatisfyWorker.java|  15 +-
 .../apache/hadoop/hdfs/server/mover/Mover.java  |   2 +-
 .../namenode/FSDirSatisfyStoragePolicyOp.java   |  26 +-
 .../server/namenode/FSDirStatAndListingOp.java  |   1 -
 .../hdfs/server/namenode/FSDirXAttrOp.java  |   2 +-
 .../hdfs/server/namenode/FSDirectory.java   |   2 +-
 .../hdfs/server/namenode/FSNamesystem.java  |  46 +--
 .../hadoop/hdfs/server/namenode/NameNode.java   |  30 +-
 .../hdfs/server/namenode/NameNodeRpcServer.java |  21 +-
 .../sps/BlockStorageMovementNeeded.java |   4 +-
 .../namenode/sps/IntraSPSNameNodeContext.java   |   6 +-
 .../hdfs/server/namenode/sps/SPSPathIds.java|  70 
 .../hdfs/server/namenode/sps/SPSService.java|  10 +-
 .../namenode/sps/StoragePolicySatisfier.java| 137 ---
 .../sps/StoragePolicySatisfyManager.java| 399 +++
 .../sps/ExternalStoragePolicySatisfier.java |   2 +-
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   |   2 +-
 .../namenode/TestNameNodeReconfigure.java   |  19 +-
 .../TestPersistentStoragePolicySatisfier.java   |   3 +-
 .../TestStoragePolicySatisfierWithHA.java   |   6 +-
 .../sps/TestStoragePolicySatisfier.java |  35 +-
 ...stStoragePolicySatisfierWithStripedFile.java |   6 +-
 .../sps/TestExternalStoragePolicySatisfier.java |  24 +-
 33 files changed, 665 insertions(+), 604 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/87e125a6/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
index 471ab2c..b6f9bdd 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@@ -3110,8 +3110,8 @@ public class DFSClient implements java.io.Closeable, 
RemotePeerFactory,
 }
   }
 
-  public boolean isStoragePolicySatisfierRunning() throws IOException {
-return namenode.isStoragePolicySatisfierRunning();
+  public boolean isInternalSatisfierRunning() throws IOException {
+return namenode.isInternalSatisfierRunning();
   }
 
   Tracer getTracer() {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/87e125a6/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
index 360fd63..5c51c22 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
@@ -1759,12 +1759,12 @@ public interface ClientProtocol {
   void satisfyStoragePolicy(String path) throws IOException;
 
   /**
-   * Check if StoragePolicySatisfier is running.
-   * @return true if StoragePolicySatisfier is running
+   * Check if internal StoragePolicySatisfier is running.
+   * @return true if internal StoragePolicySatisfier is running
* @throws IOException
*/
   @Idempotent
-  boolean isStoragePolicySatisfierRunning() throws IOException;
+  boolean 

[39/50] [abbrv] hadoop git commit: HDFS-13110: [SPS]: Reduce the number of APIs in NamenodeProtocol used by external satisfier. Contributed by Rakesh R.

[rakeshr]

HDFS-13110: [SPS]: Reduce the number of APIs in NamenodeProtocol used by 
external satisfier. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/79925028
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/79925028
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/79925028

Branch: refs/heads/HDFS-10285
Commit: 7992502873e3730e15d9c6487c2922a5d22666e0
Parents: 87e125a
Author: Rakesh Radhakrishnan 
Authored: Fri Feb 16 17:01:38 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:48 2018 +0530

--
 .../NamenodeProtocolServerSideTranslatorPB.java |  46 +
 .../NamenodeProtocolTranslatorPB.java   |  42 +
 .../hdfs/server/namenode/FSTreeTraverser.java   |   2 +-
 .../hdfs/server/namenode/NameNodeRpcServer.java |  32 +---
 .../server/namenode/ReencryptionHandler.java|   2 +-
 .../sps/BlockStorageMovementAttemptedItems.java |  42 +++--
 .../sps/BlockStorageMovementNeeded.java | 119 +++--
 .../hdfs/server/namenode/sps/Context.java   |  55 +++---
 .../hdfs/server/namenode/sps/FileCollector.java |  48 +
 .../server/namenode/sps/FileIdCollector.java|  43 -
 .../namenode/sps/IntraSPSNameNodeContext.java   |  39 ++---
 .../sps/IntraSPSNameNodeFileIdCollector.java|  23 +--
 .../hdfs/server/namenode/sps/ItemInfo.java  |  39 +++--
 .../hdfs/server/namenode/sps/SPSService.java|  32 ++--
 .../namenode/sps/StoragePolicySatisfier.java| 129 +-
 .../sps/StoragePolicySatisfyManager.java|   6 +-
 .../hdfs/server/protocol/NamenodeProtocol.java  |  24 +--
 .../sps/ExternalSPSBlockMoveTaskHandler.java|   4 +-
 .../hdfs/server/sps/ExternalSPSContext.java |  60 +++
 .../server/sps/ExternalSPSFileIDCollector.java  | 174 ---
 .../sps/ExternalSPSFilePathCollector.java   | 172 ++
 .../sps/ExternalStoragePolicySatisfier.java |   7 +-
 .../src/main/proto/NamenodeProtocol.proto   |  27 +--
 .../TestBlockStorageMovementAttemptedItems.java |  27 ++-
 .../sps/TestStoragePolicySatisfier.java |  52 +++---
 ...stStoragePolicySatisfierWithStripedFile.java |  15 +-
 .../sps/TestExternalStoragePolicySatisfier.java | 148 +++-
 27 files changed, 701 insertions(+), 708 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/79925028/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
index 25eafdf..ed176cc 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
@@ -35,16 +35,12 @@ import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetBlocksReq
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetBlocksResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetEditLogManifestRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetEditLogManifestResponseProto;
-import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetFilePathRequestProto;
-import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetFilePathResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetMostRecentCheckpointTxIdRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetMostRecentCheckpointTxIdResponseProto;
-import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetNextSPSPathIdRequestProto;
-import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetNextSPSPathIdResponseProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetNextSPSPathRequestProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetNextSPSPathResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetTransactionIdRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetTransactionIdResponseProto;
-import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.HasLowRedundancyBlocksRequestProto;
-import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.HasLowRedundancyBlocksResponseProto;
 import 

[22/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e36384a4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
deleted file mode 100644
index 6991ad2..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
+++ /dev/null
@@ -1,580 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.hdfs.server.namenode;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.TimeoutException;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.StorageType;
-import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.DFSTestUtil;
-import org.apache.hadoop.hdfs.DistributedFileSystem;
-import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
-import org.apache.hadoop.hdfs.NameNodeProxies;
-import org.apache.hadoop.hdfs.StripedFileTestUtil;
-import org.apache.hadoop.hdfs.client.HdfsAdmin;
-import org.apache.hadoop.hdfs.protocol.ClientProtocol;
-import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
-import org.apache.hadoop.hdfs.protocol.HdfsConstants;
-import org.apache.hadoop.hdfs.protocol.LocatedBlock;
-import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
-import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
-import org.apache.hadoop.test.GenericTestUtils;
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.Test;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.base.Supplier;
-
-/**
- * Tests that StoragePolicySatisfier daemon is able to check the striped blocks
- * to be moved and finding its expected target locations in order to satisfy 
the
- * storage policy.
- */
-public class TestStoragePolicySatisfierWithStripedFile {
-
-  private static final Logger LOG = LoggerFactory
-  .getLogger(TestStoragePolicySatisfierWithStripedFile.class);
-
-  private final int stripesPerBlock = 2;
-
-  private ErasureCodingPolicy ecPolicy;
-  private int dataBlocks;
-  private int parityBlocks;
-  private int cellSize;
-  private int defaultStripeBlockSize;
-
-  private ErasureCodingPolicy getEcPolicy() {
-return StripedFileTestUtil.getDefaultECPolicy();
-  }
-
-  /**
-   * Initialize erasure coding policy.
-   */
-  @Before
-  public void init(){
-ecPolicy = getEcPolicy();
-dataBlocks = ecPolicy.getNumDataUnits();
-parityBlocks = ecPolicy.getNumParityUnits();
-cellSize = ecPolicy.getCellSize();
-defaultStripeBlockSize = cellSize * stripesPerBlock;
-  }
-
-  /**
-   * Tests to verify that all the striped blocks(data + parity blocks) are
-   * moving to satisfy the storage policy.
-   */
-  @Test(timeout = 30)
-  public void testMoverWithFullStripe() throws Exception {
-// start 10 datanodes
-int numOfDatanodes = 10;
-int storagesPerDatanode = 2;
-long capacity = 20 * defaultStripeBlockSize;
-long[][] capacities = new long[numOfDatanodes][storagesPerDatanode];
-for (int i = 0; i < numOfDatanodes; i++) {
-  for (int j = 0; j < storagesPerDatanode; j++) {
-capacities[i][j] = capacity;
-  }
-}
-
-final Configuration conf = new HdfsConfiguration();
-conf.setBoolean(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
-true);
-initConfWithStripe(conf, defaultStripeBlockSize);
-final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
-.numDataNodes(numOfDatanodes)
-.storagesPerDatanode(storagesPerDatanode)
-   

[01/50] [abbrv] hadoop git commit: HDFS-11726. [SPS]: StoragePolicySatisfier should not select same storage type as source and destination in same datanode. Surendra Singh Lilhore. [Forced Update!]

[rakeshr]

Repository: hadoop
Updated Branches:
  refs/heads/HDFS-10285 cee21db17 -> e5bd57145 (forced update)


HDFS-11726. [SPS]: StoragePolicySatisfier should not select same storage type 
as source and destination in same datanode. Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/64575960
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/64575960
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/64575960

Branch: refs/heads/HDFS-10285
Commit: 64575960597c700fdc0d08ac2c74bb941f8601b5
Parents: 3de4131
Author: Rakesh Radhakrishnan 
Authored: Fri Jun 9 14:03:13 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:08:57 2018 +0530

--
 .../server/namenode/StoragePolicySatisfier.java | 23 ++
 .../namenode/TestStoragePolicySatisfier.java| 44 
 2 files changed, 58 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/64575960/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
index 9e2a4a0..1b2afa3 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
@@ -501,15 +501,20 @@ public class StoragePolicySatisfier implements Runnable {
 // avoid choosing a target which already has this block.
 for (int i = 0; i < sourceWithStorageList.size(); i++) {
   StorageTypeNodePair existingTypeNodePair = sourceWithStorageList.get(i);
-  StorageTypeNodePair chosenTarget = chooseTargetTypeInSameNode(blockInfo,
-  existingTypeNodePair.dn, expected);
-  if (chosenTarget != null) {
-sourceNodes.add(existingTypeNodePair.dn);
-sourceStorageTypes.add(existingTypeNodePair.storageType);
-targetNodes.add(chosenTarget.dn);
-targetStorageTypes.add(chosenTarget.storageType);
-expected.remove(chosenTarget.storageType);
-// TODO: We can increment scheduled block count for this node?
+
+  // Check whether the block replica is already placed in the expected
+  // storage type in this source datanode.
+  if (!expected.contains(existingTypeNodePair.storageType)) {
+StorageTypeNodePair chosenTarget = chooseTargetTypeInSameNode(
+blockInfo, existingTypeNodePair.dn, expected);
+if (chosenTarget != null) {
+  sourceNodes.add(existingTypeNodePair.dn);
+  sourceStorageTypes.add(existingTypeNodePair.storageType);
+  targetNodes.add(chosenTarget.dn);
+  targetStorageTypes.add(chosenTarget.storageType);
+  expected.remove(chosenTarget.storageType);
+  // TODO: We can increment scheduled block count for this node?
+}
   }
   // To avoid choosing this excludeNodes as targets later
   excludeNodes.add(existingTypeNodePair.dn);

http://git-wip-us.apache.org/repos/asf/hadoop/blob/64575960/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
index 8e08a1e..f1a4169 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
@@ -764,6 +764,50 @@ public class TestStoragePolicySatisfier {
   }
 
   /**
+   * If replica with expected storage type already exist in source DN then that
+   * DN should be skipped.
+   */
+  @Test(timeout = 30)
+  public void testSPSWhenReplicaWithExpectedStorageAlreadyAvailableInSource()
+  throws Exception {
+StorageType[][] diskTypes = new StorageType[][] {
+{StorageType.DISK, StorageType.ARCHIVE},
+{StorageType.DISK, StorageType.ARCHIVE},
+{StorageType.DISK, StorageType.ARCHIVE}};
+
+try {
+  hdfsCluster = startCluster(config, diskTypes, diskTypes.length,
+  storagesPerDatanode, capacity);
+  dfs = hdfsCluster.getFileSystem();
+  // 1. Write two 

[06/50] [abbrv] hadoop git commit: HDFS-11874. [SPS]: Document the SPS feature. Contributed by Uma Maheswara Rao G

[rakeshr]

HDFS-11874. [SPS]: Document the SPS feature. Contributed by Uma Maheswara Rao G


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b1238b21
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b1238b21
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b1238b21

Branch: refs/heads/HDFS-10285
Commit: b1238b217ce2e4b2314827bc8b6411fa25d883f0
Parents: bc4b9e6
Author: Rakesh Radhakrishnan 
Authored: Fri Jul 14 22:36:09 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:17 2018 +0530

--
 .../src/site/markdown/ArchivalStorage.md| 51 ++--
 1 file changed, 48 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/b1238b21/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
index a56cf8b..9098616 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
@@ -97,8 +97,44 @@ The effective storage policy can be retrieved by the 
"[`storagepolicies -getStor
 
 The default storage type of a datanode storage location will be DISK if it 
does not have a storage type tagged explicitly.
 
-Mover - A New Data Migration Tool
--
+Storage Policy Based Data Movement
+--
+
+Setting a new storage policy on already existing file/dir will change the 
policy in Namespace, but it will not move the blocks physically across storage 
medias.
+Following 2 options will allow users to move the blocks based on new policy 
set. So, once user change/set to a new policy on file/directory, user should 
also perform one of the following options to achieve the desired data movement. 
Note that both options cannot be allowed to run simultaneously.
+
+### Storage Policy Satisfier (SPS)
+
+When user changes the storage policy on a file/directory, user can call 
`HdfsAdmin` API `satisfyStoragePolicy()` to move the blocks as per the new 
policy set.
+The SPS daemon thread runs along with namenode and periodically scans for the 
storage mismatches between new policy set and the physical blocks placed. This 
will only track the files/directories for which user invoked 
satisfyStoragePolicy. If SPS identifies some blocks to be moved for a file, 
then it will schedule block movement tasks to datanodes. A Coordinator 
DataNode(C-DN) will track all block movements associated to a file and notify 
to namenode about movement success/failure. If there are any failures in 
movement, the SPS will re-attempt by sending new block movement task.
+
+SPS can be activated and deactivated dynamically without restarting the 
Namenode.
+
+Detailed design documentation can be found at [Storage Policy Satisfier(SPS) 
(HDFS-10285)](https://issues.apache.org/jira/browse/HDFS-10285)
+
+* **Note**: When user invokes `satisfyStoragePolicy()` API on a directory, SPS 
will consider the files which are immediate to that directory. Sub-directories 
won't be considered for satisfying the policy. Its user responsibility to call 
this API on directories recursively, to track all files under the sub tree.
+
+* HdfsAdmin API :
+`public void satisfyStoragePolicy(final Path path) throws IOException`
+
+* Arguments :
+
+| | |
+|: |: |
+| `path` | A path which requires blocks storage movement. |
+
+Configurations:
+
+*   **dfs.storage.policy.satisfier.activate** - Used to activate or deactivate 
SPS. Configuring true represents SPS is
+   activated and vice versa.
+
+*   **dfs.storage.policy.satisfier.recheck.timeout.millis** - A timeout to 
re-check the processed block storage movement
+   command results from Co-ordinator Datanode.
+
+*   **dfs.storage.policy.satisfier.self.retry.timeout.millis** - A timeout to 
retry if no block movement results reported from
+   Co-ordinator Datanode in this configured timeout.
+
+### Mover - A New Data Migration Tool
 
 A new data migration tool is added for archiving data. The tool is similar to 
Balancer. It periodically scans the files in HDFS to check if the block 
placement satisfies the storage policy. For the blocks violating the storage 
policy, it moves the replicas to a different storage type in order to fulfill 
the storage policy requirement. Note that it always tries to move block 
replicas within the same node whenever possible. If that is not possible (e.g. 
when a node doesn’t have the target storage type) then it will copy the block 
replicas to another node over the network.
 
@@ -115,6 +151,10 @@ A new data migration tool 

[23/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e36384a4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
deleted file mode 100644
index 9f733ff..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
+++ /dev/null
@@ -1,1775 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.hdfs.server.namenode;
-
-import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_STORAGE_POLICY_ENABLED_KEY;
-import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.slf4j.LoggerFactory.getLogger;
-
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.TimeoutException;
-
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.conf.ReconfigurationException;
-import org.apache.hadoop.fs.FSDataOutputStream;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.StorageType;
-import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.DFSTestUtil;
-import org.apache.hadoop.hdfs.DistributedFileSystem;
-import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
-import org.apache.hadoop.hdfs.NameNodeProxies;
-import org.apache.hadoop.hdfs.StripedFileTestUtil;
-import org.apache.hadoop.hdfs.client.HdfsAdmin;
-import org.apache.hadoop.hdfs.protocol.ClientProtocol;
-import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
-import org.apache.hadoop.hdfs.protocol.HdfsConstants;
-import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
-import org.apache.hadoop.hdfs.protocol.LocatedBlock;
-import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
-import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
-import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
-import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
-import org.apache.hadoop.hdfs.server.datanode.DataNode;
-import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
-import org.apache.hadoop.hdfs.server.datanode.InternalDataNodeTestUtils;
-import org.apache.hadoop.test.GenericTestUtils;
-import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
-import org.junit.Assert;
-import org.junit.Test;
-import org.mockito.Mockito;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.slf4j.event.Level;
-
-import com.google.common.base.Supplier;
-
-/**
- * Tests that StoragePolicySatisfier daemon is able to check the blocks to be
- * moved and finding its suggested target locations to move.
- */
-public class TestStoragePolicySatisfier {
-
-  {
-GenericTestUtils.setLogLevel(
-getLogger(FSTreeTraverser.class), Level.DEBUG);
-  }
-
-  private static final String ONE_SSD = "ONE_SSD";
-  private static final String COLD = "COLD";
-  private static final Logger LOG =
-  LoggerFactory.getLogger(TestStoragePolicySatisfier.class);
-  private final Configuration config = new HdfsConfiguration();
-  private StorageType[][] allDiskTypes =
-  new StorageType[][]{{StorageType.DISK, StorageType.DISK},
-  {StorageType.DISK, StorageType.DISK},
-  {StorageType.DISK, StorageType.DISK}};
-  private MiniDFSCluster hdfsCluster = null;
-  final private int numOfDatanodes = 3;
-  

[24/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e36384a4/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
new file mode 100644
index 000..5635621
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
@@ -0,0 +1,572 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode.sps;
+
+import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
+import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
+import org.apache.hadoop.hdfs.server.namenode.FSDirectory;
+import org.apache.hadoop.hdfs.server.namenode.FSTreeTraverser;
+import org.apache.hadoop.hdfs.server.namenode.INode;
+import org.apache.hadoop.hdfs.server.namenode.Namesystem;
+import 
org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier.ItemInfo;
+import org.apache.hadoop.hdfs.server.namenode.FSTreeTraverser.TraverseInfo;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+
+/**
+ * A Class to track the block collection IDs (Inode's ID) for which physical
+ * storage movement needed as per the Namespace and StorageReports from DN.
+ * It scan the pending directories for which storage movement is required and
+ * schedule the block collection IDs for movement. It track the info of
+ * scheduled items and remove the SPS xAttr from the file/Directory once
+ * movement is success.
+ */
+@InterfaceAudience.Private
+public class BlockStorageMovementNeeded {
+
+  public static final Logger LOG =
+  LoggerFactory.getLogger(BlockStorageMovementNeeded.class);
+
+  private final Queue storageMovementNeeded =
+  new LinkedList();
+
+  /**
+   * Map of startId and number of child's. Number of child's indicate the
+   * number of files pending to satisfy the policy.
+   */
+  private final Map pendingWorkForDirectory =
+  new HashMap();
+
+  private final Map spsStatus =
+  new ConcurrentHashMap<>();
+
+  private final Namesystem namesystem;
+
+  // List of pending dir to satisfy the policy
+  private final Queue spsDirsToBeTraveresed = new LinkedList();
+
+  private final StoragePolicySatisfier sps;
+
+  private Daemon inodeIdCollector;
+
+  private final int maxQueuedItem;
+
+  // Amount of time to cache the SUCCESS status of path before turning it to
+  // NOT_AVAILABLE.
+  private static long statusClearanceElapsedTimeMs = 30;
+
+  public BlockStorageMovementNeeded(Namesystem namesystem,
+  StoragePolicySatisfier sps, int queueLimit) {
+this.namesystem = namesystem;
+this.sps = sps;
+this.maxQueuedItem = queueLimit;
+  }
+
+  /**
+   * Add the candidate to tracking list for which storage movement
+   * expected if necessary.
+   *
+   * @param trackInfo
+   *  - track info for satisfy the policy
+   */
+  public synchronized void add(ItemInfo trackInfo) {
+spsStatus.put(trackInfo.getStartId(),
+new StoragePolicySatisfyPathStatusInfo(
+StoragePolicySatisfyPathStatus.IN_PROGRESS));
+storageMovementNeeded.add(trackInfo);
+  }
+
+  /**
+   * Add the itemInfo to tracking list for which storage movement
+   * expected if necessary.
+   * @param startId
+   *- start id
+  

[18/50] [abbrv] hadoop git commit: HDFS-12790: [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and HDFS-11968 commits. Contributed by Rakesh R.

[rakeshr]

HDFS-12790: [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and 
HDFS-11968 commits. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ec9a37a0
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ec9a37a0
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ec9a37a0

Branch: refs/heads/HDFS-10285
Commit: ec9a37a0b3c013e29cad8800f754a0a790f5991a
Parents: 8e0a5fb
Author: Rakesh Radhakrishnan 
Authored: Fri Nov 10 10:06:43 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:58 2018 +0530

--
 .../federation/router/RouterRpcServer.java  |  19 +++
 .../namenode/TestStoragePolicySatisfier.java|   9 +-
 ...stStoragePolicySatisfierWithStripedFile.java |  21 +--
 .../hdfs/tools/TestStoragePolicyCommands.java   |  57 -
 .../TestStoragePolicySatisfyAdminCommands.java  | 127 +++
 5 files changed, 162 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ec9a37a0/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
 
b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
index 027db8a..c5458f0 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
@@ -87,6 +87,7 @@ import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.ReencryptAction;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.RollingUpgradeAction;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
 import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
 import org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus;
 import org.apache.hadoop.hdfs.protocol.LastBlockWithStatus;
@@ -2490,4 +2491,22 @@ public class RouterRpcServer extends AbstractService
   public FederationRPCMetrics getRPCMetrics() {
 return this.rpcMonitor.getRPCMetrics();
   }
+
+  @Override
+  public void satisfyStoragePolicy(String path) throws IOException {
+checkOperation(OperationCategory.WRITE, false);
+  }
+
+  @Override
+  public boolean isStoragePolicySatisfierRunning() throws IOException {
+checkOperation(OperationCategory.READ, false);
+return false;
+  }
+
+  @Override
+  public StoragePolicySatisfyPathStatus checkStoragePolicySatisfyPathStatus(
+  String path) throws IOException {
+checkOperation(OperationCategory.READ, false);
+return StoragePolicySatisfyPathStatus.NOT_AVAILABLE;
+  }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/ec9a37a0/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
index f42d911..edd1aca 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
@@ -61,6 +61,7 @@ import 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
 import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
 import org.apache.hadoop.hdfs.server.datanode.DataNode;
 import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
+import org.apache.hadoop.hdfs.server.datanode.InternalDataNodeTestUtils;
 import org.apache.hadoop.test.GenericTestUtils;
 import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
 import org.junit.Assert;
@@ -912,8 +913,6 @@ public class TestStoragePolicySatisfier {
 
 int defaultStripedBlockSize =
 StripedFileTestUtil.getDefaultECPolicy().getCellSize() * 4;
-config.set(DFSConfigKeys.DFS_NAMENODE_EC_POLICIES_ENABLED_KEY,
-StripedFileTestUtil.getDefaultECPolicy().getName());
 config.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, defaultStripedBlockSize);
 config.setLong(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1L);
 

[26/50] [abbrv] hadoop git commit: HDFS-12982 : [SPS]: Reduce the locking and cleanup the Namesystem access. Contributed by Rakesh R.

[rakeshr]

HDFS-12982 : [SPS]: Reduce the locking and cleanup the Namesystem access. 
Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/113185ee
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/113185ee
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/113185ee

Branch: refs/heads/HDFS-10285
Commit: 113185eec03d6d107ac19d267741c760aaef8f9c
Parents: e36384a
Author: Surendra Singh Lilhore 
Authored: Mon Jan 8 15:13:11 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:10 2018 +0530

--
 .../server/blockmanagement/BlockManager.java|  16 +-
 .../blockmanagement/DatanodeDescriptor.java |   2 +-
 .../server/blockmanagement/DatanodeManager.java |  22 ++
 .../server/namenode/FSDirStatAndListingOp.java  |   1 +
 .../hdfs/server/namenode/FSNamesystem.java  |  44 ++-
 .../hdfs/server/namenode/IntraNNSPSContext.java |  41 --
 .../hadoop/hdfs/server/namenode/Namesystem.java |  24 ++
 .../sps/BlockStorageMovementAttemptedItems.java |  17 +-
 .../sps/BlockStorageMovementNeeded.java |  48 ++-
 .../hdfs/server/namenode/sps/Context.java   | 181 +
 .../namenode/sps/IntraSPSNameNodeContext.java   | 220 +++
 .../namenode/sps/StoragePolicySatisfier.java| 374 +--
 .../TestBlockStorageMovementAttemptedItems.java |  17 +-
 .../sps/TestStoragePolicySatisfier.java |  25 +-
 14 files changed, 742 insertions(+), 290 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/113185ee/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index 1cf687e..c2d5162 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -89,11 +89,12 @@ import 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.ReplicaState;
 import org.apache.hadoop.hdfs.server.namenode.CachedBlock;
 import org.apache.hadoop.hdfs.server.namenode.INode.BlocksMapUpdateInfo;
 import org.apache.hadoop.hdfs.server.namenode.INodesInPath;
-import org.apache.hadoop.hdfs.server.namenode.IntraNNSPSContext;
 import org.apache.hadoop.hdfs.server.namenode.NameNode;
 import org.apache.hadoop.hdfs.server.namenode.Namesystem;
 import org.apache.hadoop.hdfs.server.namenode.ha.HAContext;
 import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
+import org.apache.hadoop.hdfs.server.namenode.sps.Context;
+import org.apache.hadoop.hdfs.server.namenode.sps.IntraSPSNameNodeContext;
 import org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier;
 import org.apache.hadoop.hdfs.server.protocol.BlockCommand;
 import org.apache.hadoop.hdfs.server.protocol.BlockReportContext;
@@ -433,6 +434,7 @@ public class BlockManager implements BlockStatsMXBean {
   private final StoragePolicySatisfier sps;
   private final boolean storagePolicyEnabled;
   private boolean spsEnabled;
+  private Context spsctxt = null;
   /** Minimum live replicas needed for the datanode to be transitioned
* from ENTERING_MAINTENANCE to IN_MAINTENANCE.
*/
@@ -479,8 +481,8 @@ public class BlockManager implements BlockStatsMXBean {
 conf.getBoolean(
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_DEFAULT);
-StoragePolicySatisfier.Context spsctxt = new IntraNNSPSContext(namesystem);
-sps = new StoragePolicySatisfier(namesystem, this, conf, spsctxt);
+spsctxt = new IntraSPSNameNodeContext(namesystem, this, conf);
+sps = new StoragePolicySatisfier(spsctxt);
 blockTokenSecretManager = createBlockTokenSecretManager(conf);
 
 providedStorageMap = new ProvidedStorageMap(namesystem, this, conf);
@@ -5031,8 +5033,8 @@ public class BlockManager implements BlockStatsMXBean {
   LOG.info("Storage policy satisfier is already running.");
   return;
 }
-
-sps.start(false);
+// TODO: FSDirectory will get removed via HDFS-12911 modularization work
+sps.start(false, namesystem.getFSDirectory());
   }
 
   /**
@@ -5068,8 +5070,8 @@ public class BlockManager implements BlockStatsMXBean {
   LOG.info("Storage policy satisfier is already running.");
   return;
 }
-
-sps.start(true);
+// TODO: FSDirectory will get removed via HDFS-12911 modularization work
+sps.start(true, 

[29/50] [abbrv] hadoop git commit: HDFS-13025. [SPS]: Implement a mechanism to scan the files for external SPS. Contributed by Uma Maheswara Rao G.

[rakeshr]

HDFS-13025. [SPS]: Implement a mechanism to scan the files for external SPS. 
Contributed by Uma Maheswara Rao G.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/febf5100
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/febf5100
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/febf5100

Branch: refs/heads/HDFS-10285
Commit: febf5100fb7b40d6d4395a123f7d476b42e169a2
Parents: a0c8e48
Author: Rakesh Radhakrishnan 
Authored: Tue Jan 23 20:09:26 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:19 2018 +0530

--
 .../sps/BlockStorageMovementNeeded.java |  70 +++-
 .../hdfs/server/namenode/sps/Context.java   |   8 +
 .../IntraSPSNameNodeBlockMoveTaskHandler.java   |   2 +
 .../namenode/sps/IntraSPSNameNodeContext.java   |   7 +
 .../sps/IntraSPSNameNodeFileIdCollector.java|   6 +-
 .../hdfs/server/namenode/sps/SPSService.java|  10 +-
 .../namenode/sps/StoragePolicySatisfier.java|   8 +-
 .../server/sps/ExternalSPSFileIDCollector.java  | 156 +
 .../hadoop/hdfs/server/sps/package-info.java|  28 ++
 .../sps/TestStoragePolicySatisfier.java | 323 ++-
 .../sps/TestExternalStoragePolicySatisfier.java | 108 +++
 11 files changed, 556 insertions(+), 170 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/febf5100/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
index 39a0051..b141502 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
@@ -97,23 +97,53 @@ public class BlockStorageMovementNeeded {
   }
 
   /**
-   * Add the itemInfo to tracking list for which storage movement
-   * expected if necessary.
+   * Add the itemInfo list to tracking list for which storage movement expected
+   * if necessary.
+   *
* @param startId
-   *- start id
+   *  - start id
* @param itemInfoList
-   *- List of child in the directory
+   *  - List of child in the directory
+   * @param scanCompleted
+   *  -Indicates whether the start id directory has no more elements to
+   *  scan.
*/
   @VisibleForTesting
-  public synchronized void addAll(long startId,
-  List itemInfoList, boolean scanCompleted) {
+  public synchronized void addAll(long startId, List itemInfoList,
+  boolean scanCompleted) {
 storageMovementNeeded.addAll(itemInfoList);
+updatePendingDirScanStats(startId, itemInfoList.size(), scanCompleted);
+  }
+
+  /**
+   * Add the itemInfo to tracking list for which storage movement expected if
+   * necessary.
+   *
+   * @param itemInfoList
+   *  - List of child in the directory
+   * @param scanCompleted
+   *  -Indicates whether the ItemInfo start id directory has no more
+   *  elements to scan.
+   */
+  @VisibleForTesting
+  public synchronized void add(ItemInfo itemInfo, boolean scanCompleted) {
+storageMovementNeeded.add(itemInfo);
+// This represents sps start id is file, so no need to update pending dir
+// stats.
+if (itemInfo.getStartId() == itemInfo.getFileId()) {
+  return;
+}
+updatePendingDirScanStats(itemInfo.getStartId(), 1, scanCompleted);
+  }
+
+  private void updatePendingDirScanStats(long startId, int numScannedFiles,
+  boolean scanCompleted) {
 DirPendingWorkInfo pendingWork = pendingWorkForDirectory.get(startId);
 if (pendingWork == null) {
   pendingWork = new DirPendingWorkInfo();
   pendingWorkForDirectory.put(startId, pendingWork);
 }
-pendingWork.addPendingWorkCount(itemInfoList.size());
+pendingWork.addPendingWorkCount(numScannedFiles);
 if (scanCompleted) {
   pendingWork.markScanCompleted();
 }
@@ -250,13 +280,15 @@ public class BlockStorageMovementNeeded {
 
 @Override
 public void run() {
-  LOG.info("Starting FileInodeIdCollector!.");
+  LOG.info("Starting SPSPathIdProcessor!.");
   long lastStatusCleanTime = 0;
+  Long startINodeId = null;
   while (ctxt.isRunning()) {
-LOG.info("Running FileInodeIdCollector!.");
 try {
   if (!ctxt.isInSafeMode()) {
-Long startINodeId = ctxt.getNextSPSPathId();
+if (startINodeId 

[03/50] [abbrv] hadoop git commit: HDFS-11670: [SPS]: Add CLI command for satisfy storage policy operations. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-11670: [SPS]: Add CLI command for satisfy storage policy operations. 
Contributed by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4162d3c7
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4162d3c7
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4162d3c7

Branch: refs/heads/HDFS-10285
Commit: 4162d3c77bd71939a81bab9c2883833b49da30c0
Parents: 65094cb
Author: Uma Maheswara Rao G 
Authored: Mon Jun 19 17:16:49 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:05 2018 +0530

--
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   | 93 +++-
 .../src/site/markdown/ArchivalStorage.md| 21 +
 .../src/site/markdown/HDFSCommands.md   |  2 +
 .../hdfs/tools/TestStoragePolicyCommands.java   | 43 -
 4 files changed, 157 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/4162d3c7/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
index aeb10d9..662957c 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
@@ -23,6 +23,7 @@ import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.BlockStoragePolicySpi;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
 import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
@@ -32,6 +33,8 @@ import org.apache.hadoop.util.Tool;
 import org.apache.hadoop.util.ToolRunner;
 
 import java.io.FileNotFoundException;
+import com.google.common.base.Joiner;
+
 import java.io.IOException;
 import java.util.Arrays;
 import java.util.Collection;
@@ -245,6 +248,92 @@ public class StoragePolicyAdmin extends Configured 
implements Tool {
 }
   }
 
+  /** Command to schedule blocks to move based on specified policy. */
+  private static class SatisfyStoragePolicyCommand implements
+  AdminHelper.Command {
+@Override
+public String getName() {
+  return "-satisfyStoragePolicy";
+}
+
+@Override
+public String getShortUsage() {
+  return "[" + getName() + " -path ]\n";
+}
+
+@Override
+public String getLongUsage() {
+  TableListing listing = AdminHelper.getOptionDescriptionListing();
+  listing.addRow("", "The path of the file/directory to satisfy"
+  + " storage policy");
+  return getShortUsage() + "\n" +
+  "Schedule blocks to move based on file/directory policy.\n\n" +
+  listing.toString();
+}
+
+@Override
+public int run(Configuration conf, List args) throws IOException {
+  final String path = StringUtils.popOptionWithArgument("-path", args);
+  if (path == null) {
+System.err.println("Please specify the path for setting the storage " +
+"policy.\nUsage: " + getLongUsage());
+return 1;
+  }
+
+  final DistributedFileSystem dfs = AdminHelper.getDFS(conf);
+  try {
+dfs.satisfyStoragePolicy(new Path(path));
+System.out.println("Scheduled blocks to move based on the current"
++ " storage policy on " + path);
+  } catch (Exception e) {
+System.err.println(AdminHelper.prettifyException(e));
+return 2;
+  }
+  return 0;
+}
+  }
+
+  /** Command to check storage policy satisfier status. */
+  private static class IsSPSRunningCommand implements AdminHelper.Command {
+@Override
+public String getName() {
+  return "-isSPSRunning";
+}
+
+@Override
+public String getShortUsage() {
+  return "[" + getName() + "]\n";
+}
+
+@Override
+public String getLongUsage() {
+  return getShortUsage() + "\n" +
+  "Check the status of Storage Policy Statisfier.\n\n";
+}
+
+@Override
+public int run(Configuration conf, List args) throws IOException {
+  if (!args.isEmpty()) {
+System.err.print("Can't understand arguments: "
++ Joiner.on(" ").join(args) + "\n");
+System.err.println("Usage is " + getLongUsage());
+return 1;
+  }
+  final DistributedFileSystem dfs = AdminHelper.getDFS(conf);
+  try {
+if(dfs.getClient().isStoragePolicySatisfierRunning()){
+  

[32/50] [abbrv] hadoop git commit: HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R.

[rakeshr]

HDFS-13057: [SPS]: Revisit configurations to make SPS service modes 
internal/external/none. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4e89833e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4e89833e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4e89833e

Branch: refs/heads/HDFS-10285
Commit: 4e89833e86fe79908737ab9c1ace9fdb5711e997
Parents: 4ed7b37
Author: Uma Maheswara Rao G 
Authored: Fri Jan 26 08:57:29 2018 -0800
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:27 2018 +0530

--
 .../hadoop/hdfs/protocol/HdfsConstants.java |  39 
 .../org/apache/hadoop/hdfs/DFSConfigKeys.java   |   9 +-
 .../server/blockmanagement/BlockManager.java| 105 +++---
 .../hdfs/server/namenode/FSNamesystem.java  |   6 +-
 .../hdfs/server/namenode/FSTreeTraverser.java   |   2 +-
 .../hadoop/hdfs/server/namenode/NameNode.java   |  34 ++--
 .../sps/BlockStorageMovementNeeded.java |   2 +-
 .../namenode/sps/IntraSPSNameNodeContext.java   |   3 +
 .../hdfs/server/namenode/sps/SPSService.java|   4 +-
 .../namenode/sps/StoragePolicySatisfier.java|  17 +-
 .../server/sps/ExternalSPSFileIDCollector.java  |  32 ++-
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   |  16 +-
 .../src/main/resources/hdfs-default.xml |  11 +-
 .../src/site/markdown/ArchivalStorage.md|  17 +-
 .../TestStoragePolicySatisfyWorker.java |   5 +-
 .../hadoop/hdfs/server/mover/TestMover.java |  45 +++--
 .../hdfs/server/mover/TestStorageMover.java |   4 +-
 .../namenode/TestNameNodeReconfigure.java   | 105 +-
 .../TestPersistentStoragePolicySatisfier.java   |   9 +-
 .../TestStoragePolicySatisfierWithHA.java   |  12 +-
 .../sps/TestStoragePolicySatisfier.java | 202 +++
 ...stStoragePolicySatisfierWithStripedFile.java |  17 +-
 .../sps/TestExternalStoragePolicySatisfier.java | 112 +++---
 .../hdfs/tools/TestStoragePolicyCommands.java   |   5 +-
 .../TestStoragePolicySatisfyAdminCommands.java  |  14 +-
 25 files changed, 500 insertions(+), 327 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/4e89833e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
index aabcdd9..ab48dcd 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
@@ -129,6 +129,45 @@ public final class HdfsConstants {
   }
 
   /**
+   * Storage policy satisfier service modes.
+   */
+  public enum StoragePolicySatisfierMode {
+
+/**
+ * This mode represents that SPS service is running inside Namenode and can
+ * accept any SPS call request.
+ */
+INTERNAL,
+
+/**
+ * This mode represents that SPS service is running outside Namenode as an
+ * external service and can accept any SPS call request.
+ */
+EXTERNAL,
+
+/**
+ * This mode represents that SPS service is disabled and cannot accept any
+ * SPS call request.
+ */
+NONE;
+
+private static final Map MAP =
+new HashMap<>();
+
+static {
+  for (StoragePolicySatisfierMode a : values()) {
+MAP.put(a.name(), a);
+  }
+}
+
+/** Convert the given String to a StoragePolicySatisfierMode. */
+public static StoragePolicySatisfierMode fromString(String s) {
+  return MAP.get(StringUtils.toUpperCase(s));
+}
+  }
+
+
+  /**
* Storage policy satisfy path status.
*/
   public enum StoragePolicySatisfyPathStatus {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/4e89833e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index de1fed4..60982e6 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -23,6 +23,7 @@ import org.apache.hadoop.fs.CommonConfigurationKeys;
 import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys;
 import org.apache.hadoop.hdfs.net.DFSNetworkTopology;
 import 

[13/50] [abbrv] hadoop git commit: HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/48baf306/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
index 57e9f94..70219f6 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
@@ -203,11 +203,11 @@ public class TestStoragePolicySatisfier {
   }
 
   /**
-   * Tests to verify that the block storage movement results will be propagated
+   * Tests to verify that the block storage movement report will be propagated
* to Namenode via datanode heartbeat.
*/
   @Test(timeout = 30)
-  public void testPerTrackIdBlocksStorageMovementResults() throws Exception {
+  public void testBlksStorageMovementAttemptFinishedReport() throws Exception {
 try {
   createCluster();
   // Change policy to ONE_SSD
@@ -229,7 +229,7 @@ public class TestStoragePolicySatisfier {
   DFSTestUtil.waitExpectedStorageType(
   file, StorageType.DISK, 2, 3, dfs);
 
-  waitForBlocksMovementResult(1, 3);
+  waitForBlocksMovementAttemptReport(1, 3);
 } finally {
   shutdownCluster();
 }
@@ -276,7 +276,7 @@ public class TestStoragePolicySatisfier {
 fileName, StorageType.DISK, 2, 3, dfs);
   }
 
-  waitForBlocksMovementResult(files.size(), 3);
+  waitForBlocksMovementAttemptReport(files.size(), 3);
 } finally {
   shutdownCluster();
 }
@@ -457,7 +457,7 @@ public class TestStoragePolicySatisfier {
   DFSTestUtil.waitExpectedStorageType(
   file, StorageType.DISK, 2, 3, dfs);
 
-  waitForBlocksMovementResult(1, 3);
+  waitForBlocksMovementAttemptReport(1, 3);
 } finally {
   shutdownCluster();
 }
@@ -630,7 +630,7 @@ public class TestStoragePolicySatisfier {
   // No block movement will be scheduled as there is no target node
   // available with the required storage type.
   waitForAttemptedItems(1, 3);
-  waitForBlocksMovementResult(1, 3);
+  waitForBlocksMovementAttemptReport(1, 3);
   DFSTestUtil.waitExpectedStorageType(
   file1, StorageType.ARCHIVE, 1, 3, dfs);
   DFSTestUtil.waitExpectedStorageType(
@@ -691,7 +691,7 @@ public class TestStoragePolicySatisfier {
   DFSTestUtil.waitExpectedStorageType(
   file, StorageType.DISK, 3, 3, dfs);
 
-  waitForBlocksMovementResult(1, 3);
+  waitForBlocksMovementAttemptReport(1, 3);
 } finally {
   shutdownCluster();
 }
@@ -871,7 +871,7 @@ public class TestStoragePolicySatisfier {
   Set dns = hdfsCluster.getNamesystem()
   .getBlockManager().getDatanodeManager().getDatanodes();
   for (DatanodeDescriptor dd : dns) {
-assertNull(dd.getBlocksToMoveStorages());
+assertNull(dd.getBlocksToMoveStorages(1));
   }
 
   // Enable heart beats now
@@ -1224,7 +1224,7 @@ public class TestStoragePolicySatisfier {
   /**
* Test SPS for batch processing.
*/
-  @Test(timeout = 30)
+  @Test(timeout = 300)
   public void testBatchProcessingForSPSDirectory() throws Exception {
 try {
   StorageType[][] diskTypes = new StorageType[][] {
@@ -1252,7 +1252,7 @@ public class TestStoragePolicySatisfier {
 DFSTestUtil.waitExpectedStorageType(fileName, StorageType.ARCHIVE, 2,
 3, dfs);
   }
-  waitForBlocksMovementResult(files.size(), 3);
+  waitForBlocksMovementAttemptReport(files.size(), 3);
   String expectedLogMessage = "StorageMovementNeeded queue remaining"
   + " capacity is zero";
   assertTrue("Log output does not contain expected log message: "
@@ -1268,7 +1268,7 @@ public class TestStoragePolicySatisfier {
*  1. Delete /root when traversing Q
*  2. U, R, S should not be in queued.
*/
-  @Test
+  @Test(timeout = 30)
   public void testTraverseWhenParentDeleted() throws Exception {
 StorageType[][] diskTypes = new StorageType[][] {
 {StorageType.DISK, StorageType.ARCHIVE},
@@ -1330,7 +1330,7 @@ public class TestStoragePolicySatisfier {
*  1. Delete L when traversing Q
*  2. E, M, U, R, S should not be in queued.
*/
-  @Test
+  @Test(timeout = 30)
   public void testTraverseWhenRootParentDeleted() throws Exception {
 StorageType[][] diskTypes = new StorageType[][] {
 {StorageType.DISK, StorageType.ARCHIVE},
@@ -1387,6 +1387,82 @@ public class TestStoragePolicySatisfier {
 dfs.delete(new Path("/root"), 

[09/50] [abbrv] hadoop git commit: HDFS-12152: [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to improve thread cleanup time. Contributed by Rakesh R.

[rakeshr]

HDFS-12152: [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to 
improve thread cleanup time. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/79482a3d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/79482a3d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/79482a3d

Branch: refs/heads/HDFS-10285
Commit: 79482a3dd6fef14df6ecd2c2b02208782849efb6
Parents: 3d77345
Author: Uma Maheswara Rao G 
Authored: Wed Jul 19 00:55:26 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:29 2018 +0530

--
 .../datanode/BlockStorageMovementTracker.java   | 16 
 .../server/datanode/StoragePolicySatisfyWorker.java |  5 +++--
 2 files changed, 15 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/79482a3d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockStorageMovementTracker.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockStorageMovementTracker.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockStorageMovementTracker.java
index c7e952b..f3d2bb6 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockStorageMovementTracker.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockStorageMovementTracker.java
@@ -77,7 +77,8 @@ public class BlockStorageMovementTracker implements Runnable {
 moverTaskFutures.wait(2000);
   }
 } catch (InterruptedException ignore) {
-  // ignore
+  // Sets interrupt flag of this thread.
+  Thread.currentThread().interrupt();
 }
   }
   try {
@@ -102,12 +103,19 @@ public class BlockStorageMovementTracker implements 
Runnable {
 synchronized (moverTaskFutures) {
   moverTaskFutures.remove(trackId);
 }
-// handle completed or inprogress blocks movements per trackId.
-blksMovementsStatusHandler.handle(resultPerTrackIdList);
+if (running) {
+  // handle completed or inprogress blocks movements per trackId.
+  blksMovementsStatusHandler.handle(resultPerTrackIdList);
+}
 movementResults.remove(trackId);
   }
 }
-  } catch (ExecutionException | InterruptedException e) {
+  } catch (InterruptedException e) {
+if (running) {
+  LOG.error("Exception while moving block replica to target storage"
+  + " type", e);
+}
+  } catch (ExecutionException e) {
 // TODO: Do we need failure retries and implement the same if required.
 LOG.error("Exception while moving block replica to target storage 
type",
 e);

http://git-wip-us.apache.org/repos/asf/hadoop/blob/79482a3d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
index 196cd58..4e57805 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
@@ -137,8 +137,8 @@ public class StoragePolicySatisfyWorker {
* thread.
*/
   void stop() {
-movementTrackerThread.interrupt();
 movementTracker.stopTracking();
+movementTrackerThread.interrupt();
   }
 
   /**
@@ -147,7 +147,8 @@ public class StoragePolicySatisfyWorker {
   void waitToFinishWorkerThread() {
 try {
   movementTrackerThread.join(3000);
-} catch (InterruptedException ie) {
+} catch (InterruptedException ignore) {
+  // ignore
 }
   }
 


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[46/50] [abbrv] hadoop git commit: HDFS-13076: [SPS]: Addendum. Resolve conflicts after rebasing branch to trunk. Contributed by Rakesh R.

[rakeshr]

HDFS-13076: [SPS]: Addendum. Resolve conflicts after rebasing branch to trunk. 
Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4c2f806e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4c2f806e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4c2f806e

Branch: refs/heads/HDFS-10285
Commit: 4c2f806e056072acca9a26ee259f2bc4b90ddeda
Parents: 0bdc60f
Author: Rakesh Radhakrishnan 
Authored: Fri Jul 20 10:59:16 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:11:09 2018 +0530

--
 .../java/org/apache/hadoop/hdfs/server/datanode/DataNode.java| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/4c2f806e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
index a714602..21af33f 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
@@ -3624,8 +3624,8 @@ public class DataNode extends ReconfigurableBase
 }
 return this.diskBalancer;
   }
-}
 
   StoragePolicySatisfyWorker getStoragePolicySatisfyWorker() {
 return storagePolicySatisfyWorker;
-  }}
+  }
+}


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[33/50] [abbrv] hadoop git commit: HDFS-13075. [SPS]: Provide External Context implementation. Contributed by Uma Maheswara Rao G.

[rakeshr]

HDFS-13075. [SPS]: Provide External Context implementation. Contributed by Uma 
Maheswara Rao G.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/16820957
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/16820957
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/16820957

Branch: refs/heads/HDFS-10285
Commit: 16820957b5b0e0d1771b01aa93f9628681dc
Parents: 4e89833
Author: Surendra Singh Lilhore 
Authored: Sun Jan 28 20:46:56 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:31 2018 +0530

--
 .../NamenodeProtocolServerSideTranslatorPB.java |  67 +
 .../NamenodeProtocolTranslatorPB.java   |  58 
 .../hdfs/server/balancer/NameNodeConnector.java |  28 +-
 .../server/blockmanagement/BlockManager.java|  19 ++
 .../server/blockmanagement/DatanodeManager.java |  18 ++
 .../hdfs/server/common/HdfsServerConstants.java |   3 +-
 .../hdfs/server/namenode/NameNodeRpcServer.java |  53 +++-
 .../sps/BlockStorageMovementNeeded.java |   8 +-
 .../hdfs/server/namenode/sps/Context.java   |   9 +-
 .../namenode/sps/IntraSPSNameNodeContext.java   |  23 +-
 .../namenode/sps/StoragePolicySatisfier.java|  15 +-
 .../hdfs/server/protocol/NamenodeProtocol.java  |  46 +++-
 .../hdfs/server/sps/ExternalSPSContext.java | 271 +++
 .../src/main/proto/NamenodeProtocol.proto   |  57 
 .../sps/TestExternalStoragePolicySatisfier.java |  31 +--
 15 files changed, 652 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/16820957/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
index 90c2c49..25eafdf 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
@@ -23,6 +23,8 @@ import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
 import org.apache.hadoop.hdfs.protocol.DatanodeInfo.DatanodeInfoBuilder;
 import 
org.apache.hadoop.hdfs.protocol.proto.HdfsServerProtos.VersionRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.HdfsServerProtos.VersionResponseProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.CheckDNSpaceRequestProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.CheckDNSpaceResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.EndCheckpointRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.EndCheckpointResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.ErrorReportRequestProto;
@@ -33,10 +35,16 @@ import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetBlocksReq
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetBlocksResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetEditLogManifestRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetEditLogManifestResponseProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetFilePathRequestProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetFilePathResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetMostRecentCheckpointTxIdRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetMostRecentCheckpointTxIdResponseProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetNextSPSPathIdRequestProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetNextSPSPathIdResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetTransactionIdRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.GetTransactionIdResponseProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.HasLowRedundancyBlocksRequestProto;
+import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.HasLowRedundancyBlocksResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.IsRollingUpgradeRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.IsRollingUpgradeResponseProto;
 import 

[12/50] [abbrv] hadoop git commit: HDFS-12291: [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-12291: [SPS]: Provide a mechanism to recursively iterate and satisfy 
storage policy of all the files under the given dir. Contributed by Surendra 
Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8b2e8768
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8b2e8768
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8b2e8768

Branch: refs/heads/HDFS-10285
Commit: 8b2e8768b30055464b34668f93249e0d899b
Parents: e934b3e
Author: Uma Maheswara Rao G 
Authored: Sat Sep 30 06:31:52 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:41 2018 +0530

--
 .../org/apache/hadoop/hdfs/DFSConfigKeys.java   |   8 +
 .../java/org/apache/hadoop/hdfs/DFSUtil.java|  22 +-
 .../BlockStorageMovementAttemptedItems.java |   8 +-
 .../namenode/BlockStorageMovementNeeded.java| 277 +++---
 .../server/namenode/ReencryptionHandler.java|   1 +
 .../server/namenode/StoragePolicySatisfier.java |  43 ++-
 .../src/main/resources/hdfs-default.xml |  23 ++
 .../src/site/markdown/ArchivalStorage.md|   3 +-
 .../TestBlockStorageMovementAttemptedItems.java |   2 +-
 .../TestPersistentStoragePolicySatisfier.java   |   8 +-
 .../namenode/TestStoragePolicySatisfier.java| 377 ++-
 11 files changed, 689 insertions(+), 83 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b2e8768/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 22d525a..ec5a678 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -615,6 +615,14 @@ public class DFSConfigKeys extends CommonConfigurationKeys 
{
   "dfs.storage.policy.satisfier.enabled";
   public static final boolean DFS_STORAGE_POLICY_SATISFIER_ENABLED_DEFAULT =
   false;
+  public static final String  DFS_STORAGE_POLICY_SATISFIER_QUEUE_LIMIT_KEY =
+  "dfs.storage.policy.satisfier.queue.limit";
+  public static final int  DFS_STORAGE_POLICY_SATISFIER_QUEUE_LIMIT_DEFAULT =
+  1000;
+  public static final String DFS_SPS_WORK_MULTIPLIER_PER_ITERATION =
+  "dfs.storage.policy.satisfier.work.multiplier.per.iteration";
+  public static final int DFS_SPS_WORK_MULTIPLIER_PER_ITERATION_DEFAULT =
+  1;
   public static final String 
DFS_STORAGE_POLICY_SATISFIER_RECHECK_TIMEOUT_MILLIS_KEY =
   "dfs.storage.policy.satisfier.recheck.timeout.millis";
   public static final int 
DFS_STORAGE_POLICY_SATISFIER_RECHECK_TIMEOUT_MILLIS_DEFAULT =

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b2e8768/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
index f5ceeaf..c26599c 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
@@ -1457,7 +1457,27 @@ public class DFSUtil {
 "It should be a positive, non-zero integer value.");
 return blocksReplWorkMultiplier;
   }
-  
+
+  /**
+   * Get DFS_SPS_WORK_MULTIPLIER_PER_ITERATION from
+   * configuration.
+   *
+   * @param conf Configuration
+   * @return Value of DFS_SPS_WORK_MULTIPLIER_PER_ITERATION
+   */
+  public static int getSPSWorkMultiplier(Configuration conf) {
+int spsWorkMultiplier = conf
+.getInt(
+DFSConfigKeys.DFS_SPS_WORK_MULTIPLIER_PER_ITERATION,
+DFSConfigKeys.DFS_SPS_WORK_MULTIPLIER_PER_ITERATION_DEFAULT);
+Preconditions.checkArgument(
+(spsWorkMultiplier > 0),
+DFSConfigKeys.DFS_SPS_WORK_MULTIPLIER_PER_ITERATION +
+" = '" + spsWorkMultiplier + "' is invalid. " +
+"It should be a positive, non-zero integer value.");
+return spsWorkMultiplier;
+  }
+
   /**
* Get SPNEGO keytab Key from configuration
* 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b2e8768/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
--
diff --git 

[31/50] [abbrv] hadoop git commit: HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/4e89833e/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
index 0e3a5a3..2257608 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfierWithStripedFile.java
@@ -41,6 +41,7 @@ import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import org.apache.hadoop.hdfs.protocol.LocatedBlock;
 import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode;
 import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
 import org.apache.hadoop.test.GenericTestUtils;
 import org.junit.Assert;
@@ -103,8 +104,8 @@ public class TestStoragePolicySatisfierWithStripedFile {
 }
 
 final Configuration conf = new HdfsConfiguration();
-conf.setBoolean(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
-true);
+conf.set(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_MODE_KEY,
+StoragePolicySatisfierMode.INTERNAL.toString());
 initConfWithStripe(conf, defaultStripeBlockSize);
 final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
 .numDataNodes(numOfDatanodes)
@@ -216,8 +217,8 @@ public class TestStoragePolicySatisfierWithStripedFile {
 }
 
 final Configuration conf = new HdfsConfiguration();
-conf.setBoolean(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
-true);
+conf.set(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_MODE_KEY,
+StoragePolicySatisfierMode.INTERNAL.toString());
 initConfWithStripe(conf, defaultStripeBlockSize);
 final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
 .numDataNodes(numOfDatanodes)
@@ -328,8 +329,8 @@ public class TestStoragePolicySatisfierWithStripedFile {
 conf.set(DFSConfigKeys
 .DFS_STORAGE_POLICY_SATISFIER_RECHECK_TIMEOUT_MILLIS_KEY,
 "3000");
-conf.setBoolean(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
-true);
+conf.set(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_MODE_KEY,
+StoragePolicySatisfierMode.INTERNAL.toString());
 initConfWithStripe(conf, defaultStripeBlockSize);
 final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
 .numDataNodes(numOfDatanodes)
@@ -420,8 +421,8 @@ public class TestStoragePolicySatisfierWithStripedFile {
 }
 
 final Configuration conf = new HdfsConfiguration();
-conf.setBoolean(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
-true);
+conf.set(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_MODE_KEY,
+StoragePolicySatisfierMode.INTERNAL.toString());
 initConfWithStripe(conf, defaultStripeBlockSize);
 final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
 .numDataNodes(numOfDatanodes)

http://git-wip-us.apache.org/repos/asf/hadoop/blob/4e89833e/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
index 9a401bd..42b04da 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
@@ -31,6 +31,7 @@ import org.apache.hadoop.hdfs.DFSConfigKeys;
 import org.apache.hadoop.hdfs.DFSUtil;
 import org.apache.hadoop.hdfs.MiniDFSCluster;
 import org.apache.hadoop.hdfs.protocol.Block;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode;
 import org.apache.hadoop.hdfs.server.balancer.NameNodeConnector;
 import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
 import org.apache.hadoop.hdfs.server.namenode.sps.BlockMovementListener;
@@ -54,12 +55,19 @@ public class TestExternalStoragePolicySatisfier
   new StorageType[][]{{StorageType.DISK, StorageType.DISK},
   {StorageType.DISK, StorageType.DISK},
   {StorageType.DISK, StorageType.DISK}};

[35/50] [abbrv] hadoop git commit: HDFS-13077. [SPS]: Fix review comments of external storage policy satisfier. Contributed by Rakesh R.

[rakeshr]

HDFS-13077. [SPS]: Fix review comments of external storage policy satisfier. 
Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/12e3125c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/12e3125c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/12e3125c

Branch: refs/heads/HDFS-10285
Commit: 12e3125c8c6937383e100c5da14bb0e4cae7f2dc
Parents: 9c6e8ae
Author: Surendra Singh Lilhore 
Authored: Mon Jan 29 23:59:55 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:39 2018 +0530

--
 .../org/apache/hadoop/hdfs/DFSConfigKeys.java   |  14 +-
 .../server/blockmanagement/BlockManager.java|  33 +++-
 .../namenode/FSDirSatisfyStoragePolicyOp.java   |  15 ++
 .../hdfs/server/namenode/FSNamesystem.java  |  41 ++--
 .../hdfs/server/namenode/NameNodeRpcServer.java |  11 ++
 .../hdfs/server/namenode/sps/SPSPathIds.java|   8 +-
 .../namenode/sps/StoragePolicySatisfier.java|   6 +-
 .../hdfs/server/sps/ExternalSPSContext.java |   4 +
 .../sps/ExternalStoragePolicySatisfier.java |  30 ++-
 .../sps/TestStoragePolicySatisfier.java |   7 +-
 .../sps/TestExternalStoragePolicySatisfier.java | 195 ++-
 11 files changed, 323 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/12e3125c/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 60982e6..562325a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -611,7 +611,7 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
   public static final String  DFS_MOVER_MAX_NO_MOVE_INTERVAL_KEY = 
"dfs.mover.max-no-move-interval";
   public static final intDFS_MOVER_MAX_NO_MOVE_INTERVAL_DEFAULT = 60*1000; 
// One minute
 
-  // SPS related configurations
+  // StoragePolicySatisfier (SPS) related configurations
   public static final String  DFS_STORAGE_POLICY_SATISFIER_MODE_KEY =
   "dfs.storage.policy.satisfier.mode";
   public static final String DFS_STORAGE_POLICY_SATISFIER_MODE_DEFAULT =
@@ -640,6 +640,18 @@ public class DFSConfigKeys extends CommonConfigurationKeys 
{
   "dfs.storage.policy.satisfier.low.max-streams.preference";
   public static final boolean 
DFS_STORAGE_POLICY_SATISFIER_LOW_MAX_STREAMS_PREFERENCE_DEFAULT =
   true;
+  public static final String DFS_SPS_MAX_OUTSTANDING_PATHS_KEY =
+  "dfs.storage.policy.satisfier.max.outstanding.paths";
+  public static final int DFS_SPS_MAX_OUTSTANDING_PATHS_DEFAULT = 1;
+
+  // SPS keytab configurations, by default it is disabled.
+  public static final String  DFS_SPS_ADDRESS_KEY =
+  "dfs.storage.policy.satisfier.address";
+  public static final String  DFS_SPS_ADDRESS_DEFAULT= "0.0.0.0:0";
+  public static final String  DFS_SPS_KEYTAB_FILE_KEY =
+  "dfs.storage.policy.satisfier.keytab.file";
+  public static final String  DFS_SPS_KERBEROS_PRINCIPAL_KEY =
+  "dfs.storage.policy.satisfier.kerberos.principal";
 
   public static final String  DFS_DATANODE_ADDRESS_KEY = 
"dfs.datanode.address";
   public static final int DFS_DATANODE_DEFAULT_PORT = 9866;

http://git-wip-us.apache.org/repos/asf/hadoop/blob/12e3125c/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index f348a33..00a91a9 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -439,6 +439,7 @@ public class BlockManager implements BlockStatsMXBean {
   private final boolean storagePolicyEnabled;
   private StoragePolicySatisfierMode spsMode;
   private SPSPathIds spsPaths;
+  private final int spsOutstandingPathsLimit;
 
   /** Minimum live replicas needed for the datanode to be transitioned
* from ENTERING_MAINTENANCE to IN_MAINTENANCE.
@@ -478,14 +479,16 @@ public class BlockManager implements BlockStatsMXBean {
 

[15/50] [abbrv] hadoop git commit: HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements. Contributed by Rakesh R.

[rakeshr]

HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the block 
storage movements. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/48baf306
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/48baf306
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/48baf306

Branch: refs/heads/HDFS-10285
Commit: 48baf30640932e0ce38a2ceab6907b80a2b2aa94
Parents: 8b2e876
Author: Uma Maheswara Rao G 
Authored: Thu Oct 12 17:17:51 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:46 2018 +0530

--
 .../org/apache/hadoop/hdfs/DFSConfigKeys.java   |   8 +-
 .../DatanodeProtocolClientSideTranslatorPB.java |  12 +-
 .../DatanodeProtocolServerSideTranslatorPB.java |   4 +-
 .../apache/hadoop/hdfs/protocolPB/PBHelper.java | 150 +++-
 .../blockmanagement/DatanodeDescriptor.java |  50 ++-
 .../server/blockmanagement/DatanodeManager.java | 104 --
 .../hdfs/server/datanode/BPOfferService.java|   3 +-
 .../hdfs/server/datanode/BPServiceActor.java|  33 +-
 .../datanode/BlockStorageMovementTracker.java   |  80 ++---
 .../datanode/StoragePolicySatisfyWorker.java| 214 
 .../BlockStorageMovementAttemptedItems.java | 299 
 .../BlockStorageMovementInfosBatch.java |  61 
 .../hdfs/server/namenode/FSNamesystem.java  |  11 +-
 .../hdfs/server/namenode/NameNodeRpcServer.java |   7 +-
 .../server/namenode/StoragePolicySatisfier.java | 343 ++-
 .../protocol/BlockStorageMovementCommand.java   |  99 ++
 .../BlocksStorageMoveAttemptFinished.java   |  48 +++
 .../protocol/BlocksStorageMovementResult.java   |  74 
 .../hdfs/server/protocol/DatanodeProtocol.java  |   5 +-
 .../src/main/proto/DatanodeProtocol.proto   |  30 +-
 .../src/main/resources/hdfs-default.xml |  21 +-
 .../src/site/markdown/ArchivalStorage.md|   6 +-
 .../TestNameNodePrunesMissingStorages.java  |   5 +-
 .../datanode/InternalDataNodeTestUtils.java |   4 +-
 .../server/datanode/TestBPOfferService.java |   4 +-
 .../hdfs/server/datanode/TestBlockRecovery.java |   4 +-
 .../server/datanode/TestDataNodeLifeline.java   |   6 +-
 .../TestDatanodeProtocolRetryPolicy.java|   4 +-
 .../server/datanode/TestFsDatasetCache.java |   4 +-
 .../TestStoragePolicySatisfyWorker.java |  52 ++-
 .../hdfs/server/datanode/TestStorageReport.java |   4 +-
 .../server/namenode/NNThroughputBenchmark.java  |   6 +-
 .../hdfs/server/namenode/NameNodeAdapter.java   |   4 +-
 .../TestBlockStorageMovementAttemptedItems.java | 145 
 .../hdfs/server/namenode/TestDeadDatanode.java  |   4 +-
 .../namenode/TestStoragePolicySatisfier.java| 115 ++-
 ...stStoragePolicySatisfierWithStripedFile.java |  20 +-
 37 files changed, 908 insertions(+), 1135 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/48baf306/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index ec5a678..0dcc3f6 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -626,11 +626,15 @@ public class DFSConfigKeys extends 
CommonConfigurationKeys {
   public static final String 
DFS_STORAGE_POLICY_SATISFIER_RECHECK_TIMEOUT_MILLIS_KEY =
   "dfs.storage.policy.satisfier.recheck.timeout.millis";
   public static final int 
DFS_STORAGE_POLICY_SATISFIER_RECHECK_TIMEOUT_MILLIS_DEFAULT =
-  5 * 60 * 1000;
+  1 * 60 * 1000;
   public static final String 
DFS_STORAGE_POLICY_SATISFIER_SELF_RETRY_TIMEOUT_MILLIS_KEY =
   "dfs.storage.policy.satisfier.self.retry.timeout.millis";
   public static final int 
DFS_STORAGE_POLICY_SATISFIER_SELF_RETRY_TIMEOUT_MILLIS_DEFAULT =
-  20 * 60 * 1000;
+  5 * 60 * 1000;
+  public static final String 
DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REPLICA_MAX_STREAMS_KEY =
+  "dfs.storage.policy.satisfier.low.max-streams.preference";
+  public static final boolean 
DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REPLICA_MAX_STREAMS_DEFAULT =
+  false;
 
   public static final String  DFS_DATANODE_ADDRESS_KEY = 
"dfs.datanode.address";
   public static final int DFS_DATANODE_DEFAULT_PORT = 9866;

http://git-wip-us.apache.org/repos/asf/hadoop/blob/48baf306/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java

[40/50] [abbrv] hadoop git commit: HDFS-13166: [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls. Contributed by Rakesh R.

[rakeshr]

HDFS-13166: [SPS]: Implement caching mechanism to keep LIVE datanodes to 
minimize costly getLiveDatanodeStorageReport() calls. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/616d9ed7
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/616d9ed7
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/616d9ed7

Branch: refs/heads/HDFS-10285
Commit: 616d9ed7ed352419218412842e342b8673fdb0e6
Parents: 7992502
Author: Surendra Singh Lilhore 
Authored: Thu Mar 1 00:08:37 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:10:52 2018 +0530

--
 .../org/apache/hadoop/hdfs/DFSConfigKeys.java   |   5 +
 .../NamenodeProtocolServerSideTranslatorPB.java |  19 --
 .../NamenodeProtocolTranslatorPB.java   |  17 -
 .../hdfs/server/namenode/NameNodeRpcServer.java |  13 -
 .../hdfs/server/namenode/sps/Context.java   |  24 +-
 .../namenode/sps/DatanodeCacheManager.java  | 121 +++
 .../namenode/sps/IntraSPSNameNodeContext.java   |  23 +-
 .../namenode/sps/StoragePolicySatisfier.java| 340 ++-
 .../hdfs/server/protocol/NamenodeProtocol.java  |  16 -
 .../hdfs/server/sps/ExternalSPSContext.java |  32 +-
 .../src/main/proto/NamenodeProtocol.proto   |  25 --
 .../src/main/resources/hdfs-default.xml |  11 +
 .../src/site/markdown/ArchivalStorage.md|   2 +-
 .../TestStoragePolicySatisfyWorker.java |   3 +
 .../TestPersistentStoragePolicySatisfier.java   |   6 +
 .../TestStoragePolicySatisfierWithHA.java   |   3 +
 .../sps/TestStoragePolicySatisfier.java |   4 +
 ...stStoragePolicySatisfierWithStripedFile.java |  24 +-
 .../TestStoragePolicySatisfyAdminCommands.java  |   3 +
 19 files changed, 431 insertions(+), 260 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/616d9ed7/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 562325a..d519fbe 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -643,6 +643,11 @@ public class DFSConfigKeys extends CommonConfigurationKeys 
{
   public static final String DFS_SPS_MAX_OUTSTANDING_PATHS_KEY =
   "dfs.storage.policy.satisfier.max.outstanding.paths";
   public static final int DFS_SPS_MAX_OUTSTANDING_PATHS_DEFAULT = 1;
+  // SPS datanode cache config, defaulting to 5mins.
+  public static final String DFS_SPS_DATANODE_CACHE_REFRESH_INTERVAL_MS =
+  "dfs.storage.policy.satisfier.datanode.cache.refresh.interval.ms";
+  public static final long DFS_SPS_DATANODE_CACHE_REFRESH_INTERVAL_MS_DEFAULT =
+  30L;
 
   // SPS keytab configurations, by default it is disabled.
   public static final String  DFS_SPS_ADDRESS_KEY =

http://git-wip-us.apache.org/repos/asf/hadoop/blob/616d9ed7/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
index ed176cc..e4283c6 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
@@ -23,8 +23,6 @@ import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
 import org.apache.hadoop.hdfs.protocol.DatanodeInfo.DatanodeInfoBuilder;
 import 
org.apache.hadoop.hdfs.protocol.proto.HdfsServerProtos.VersionRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.HdfsServerProtos.VersionResponseProto;
-import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.CheckDNSpaceRequestProto;
-import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.CheckDNSpaceResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.EndCheckpointRequestProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.EndCheckpointResponseProto;
 import 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos.ErrorReportRequestProto;
@@ -277,21 +275,4 @@ public class NamenodeProtocolServerSideTranslatorPB 

[10/50] [abbrv] hadoop git commit: HDFS-12214: [SPS]: Fix review comments of StoragePolicySatisfier feature. Contributed by Rakesh R.

[rakeshr]

HDFS-12214: [SPS]: Fix review comments of StoragePolicySatisfier feature. 
Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ba09ab3b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ba09ab3b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ba09ab3b

Branch: refs/heads/HDFS-10285
Commit: ba09ab3b19d4aacf6af0e196db32b7e12ddfc237
Parents: 79482a3
Author: Uma Maheswara Rao G 
Authored: Thu Aug 17 13:21:07 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:33 2018 +0530

--
 .../hadoop-hdfs/src/main/bin/hdfs   |   2 +-
 .../org/apache/hadoop/hdfs/DFSConfigKeys.java   |   8 +-
 .../server/blockmanagement/BlockManager.java| 104 +++
 .../BlockStorageMovementAttemptedItems.java |   4 +-
 .../hdfs/server/namenode/FSNamesystem.java  |  20 ++--
 .../hadoop/hdfs/server/namenode/NameNode.java   |  22 ++--
 .../server/namenode/StoragePolicySatisfier.java |  20 ++--
 .../protocol/BlocksStorageMovementResult.java   |   2 +-
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   |  11 +-
 .../src/main/resources/hdfs-default.xml |  10 +-
 .../src/site/markdown/ArchivalStorage.md|  14 +--
 .../src/site/markdown/HDFSCommands.md   |   2 +-
 .../TestStoragePolicySatisfyWorker.java |   2 +-
 .../hadoop/hdfs/server/mover/TestMover.java |  22 ++--
 .../hdfs/server/mover/TestStorageMover.java |   2 +-
 .../TestBlockStorageMovementAttemptedItems.java |   2 +-
 .../namenode/TestNameNodeReconfigure.java   |  99 --
 .../TestPersistentStoragePolicySatisfier.java   |   6 +-
 .../namenode/TestStoragePolicySatisfier.java|  35 +--
 .../TestStoragePolicySatisfierWithHA.java   |  10 +-
 ...stStoragePolicySatisfierWithStripedFile.java |   8 ++
 .../hdfs/tools/TestStoragePolicyCommands.java   |  21 ++--
 22 files changed, 265 insertions(+), 161 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ba09ab3b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
--
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
index 38be348b..bc6e7a4 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
@@ -62,7 +62,7 @@ function hadoop_usage
   hadoop_add_subcommand "portmap" daemon "run a portmap service"
   hadoop_add_subcommand "secondarynamenode" daemon "run the DFS secondary 
namenode"
   hadoop_add_subcommand "snapshotDiff" client "diff two snapshots of a 
directory or diff the current directory contents with a snapshot"
-  hadoop_add_subcommand "storagepolicies" admin "list/get/set block storage 
policies"
+  hadoop_add_subcommand "storagepolicies" admin 
"list/get/set/satisfyStoragePolicy block storage policies"
   hadoop_add_subcommand "version" client "print the version"
   hadoop_add_subcommand "zkfc" daemon "run the ZK Failover Controller daemon"
   hadoop_generate_usage "${HADOOP_SHELL_EXECNAME}" false

http://git-wip-us.apache.org/repos/asf/hadoop/blob/ba09ab3b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 08b6fb3..22d525a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -611,10 +611,10 @@ public class DFSConfigKeys extends 
CommonConfigurationKeys {
   public static final intDFS_MOVER_MAX_NO_MOVE_INTERVAL_DEFAULT = 60*1000; 
// One minute
 
   // SPS related configurations
-  public static final String  DFS_STORAGE_POLICY_SATISFIER_ACTIVATE_KEY =
-  "dfs.storage.policy.satisfier.activate";
-  public static final boolean DFS_STORAGE_POLICY_SATISFIER_ACTIVATE_DEFAULT =
-  true;
+  public static final String  DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY =
+  "dfs.storage.policy.satisfier.enabled";
+  public static final boolean DFS_STORAGE_POLICY_SATISFIER_ENABLED_DEFAULT =
+  false;
   public static final String 
DFS_STORAGE_POLICY_SATISFIER_RECHECK_TIMEOUT_MILLIS_KEY =
   "dfs.storage.policy.satisfier.recheck.timeout.millis";
   public static final int 
DFS_STORAGE_POLICY_SATISFIER_RECHECK_TIMEOUT_MILLIS_DEFAULT =

http://git-wip-us.apache.org/repos/asf/hadoop/blob/ba09ab3b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

[27/50] [abbrv] hadoop git commit: HDFS-12911. [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations. Contributed by Uma Maheswara Rao G

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/a0c8e48c/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
index 2a7bde5..9354044 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
@@ -72,7 +72,6 @@ import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
 import org.junit.After;
 import org.junit.Assert;
 import org.junit.Test;
-import org.mockito.Mockito;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.slf4j.event.Level;
@@ -147,12 +146,11 @@ public class TestStoragePolicySatisfier {
 startAdditionalDNs(config, 3, numOfDatanodes, newtypes,
 storagesPerDatanode, capacity, hdfsCluster);
 
-dfs.satisfyStoragePolicy(new Path(file));
-
 hdfsCluster.triggerHeartbeats();
+dfs.satisfyStoragePolicy(new Path(file));
 // Wait till namenode notified about the block location details
-DFSTestUtil.waitExpectedStorageType(
-file, StorageType.ARCHIVE, 3, 3, dfs);
+DFSTestUtil.waitExpectedStorageType(file, StorageType.ARCHIVE, 3, 35000,
+dfs);
   }
 
   @Test(timeout = 30)
@@ -1284,6 +1282,7 @@ public class TestStoragePolicySatisfier {
 {StorageType.ARCHIVE, StorageType.SSD},
 {StorageType.DISK, StorageType.DISK}};
 config.setLong("dfs.block.size", DEFAULT_BLOCK_SIZE);
+config.setInt(DFS_STORAGE_POLICY_SATISFIER_QUEUE_LIMIT_KEY, 10);
 hdfsCluster = startCluster(config, diskTypes, diskTypes.length,
 storagesPerDatanode, capacity);
 dfs = hdfsCluster.getFileSystem();
@@ -1299,19 +1298,28 @@ public class TestStoragePolicySatisfier {
 
 //Queue limit can control the traverse logic to wait for some free
 //entry in queue. After 10 files, traverse control will be on U.
-StoragePolicySatisfier sps = Mockito.mock(StoragePolicySatisfier.class);
-Mockito.when(sps.isRunning()).thenReturn(true);
-Context ctxt = Mockito.mock(Context.class);
-config.setInt(DFS_STORAGE_POLICY_SATISFIER_QUEUE_LIMIT_KEY, 10);
-Mockito.when(ctxt.getConf()).thenReturn(config);
-Mockito.when(ctxt.isRunning()).thenReturn(true);
-Mockito.when(ctxt.isInSafeMode()).thenReturn(false);
-Mockito.when(ctxt.isFileExist(Mockito.anyLong())).thenReturn(true);
-BlockStorageMovementNeeded movmentNeededQueue =
-new BlockStorageMovementNeeded(ctxt);
+StoragePolicySatisfier sps = new StoragePolicySatisfier(config);
+Context ctxt = new IntraSPSNameNodeContext(hdfsCluster.getNamesystem(),
+hdfsCluster.getNamesystem().getBlockManager(), sps) {
+  @Override
+  public boolean isInSafeMode() {
+return false;
+  }
+
+  @Override
+  public boolean isRunning() {
+return true;
+  }
+};
+
+FileIdCollector fileIDCollector =
+new IntraSPSNameNodeFileIdCollector(fsDir, sps);
+sps.init(ctxt, fileIDCollector, null);
+sps.getStorageMovementQueue().activate();
+
 INode rootINode = fsDir.getINode("/root");
-movmentNeededQueue.addToPendingDirQueue(rootINode.getId());
-movmentNeededQueue.init(fsDir);
+hdfsCluster.getNamesystem().getBlockManager()
+.addSPSPathId(rootINode.getId());
 
 //Wait for thread to reach U.
 Thread.sleep(1000);
@@ -1321,7 +1329,7 @@ public class TestStoragePolicySatisfier {
 // Remove 10 element and make queue free, So other traversing will start.
 for (int i = 0; i < 10; i++) {
   String path = expectedTraverseOrder.remove(0);
-  long trackId = movmentNeededQueue.get().getTrackId();
+  long trackId = sps.getStorageMovementQueue().get().getFileId();
   INode inode = fsDir.getInode(trackId);
   assertTrue("Failed to traverse tree, expected " + path + " but got "
   + inode.getFullPathName(), path.equals(inode.getFullPathName()));
@@ -1332,7 +1340,7 @@ public class TestStoragePolicySatisfier {
 // Check other element traversed in order and R,S should not be added in
 // queue which we already removed from expected list
 for (String path : expectedTraverseOrder) {
-  long trackId = movmentNeededQueue.get().getTrackId();
+  long trackId = sps.getStorageMovementQueue().get().getFileId();
   INode inode = fsDir.getInode(trackId);
   assertTrue("Failed to traverse tree, expected " + path + " but got "
   + inode.getFullPathName(), path.equals(inode.getFullPathName()));
@@ -1352,6 +1360,7 @@ public class 

[07/50] [abbrv] hadoop git commit: HDFS-12146. [SPS]: Fix TestStoragePolicySatisfierWithStripedFile#testSPSWhenFileHasLowRedundancyBlocks. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-12146. [SPS]: Fix 
TestStoragePolicySatisfierWithStripedFile#testSPSWhenFileHasLowRedundancyBlocks.
 Contributed by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/48d532c6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/48d532c6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/48d532c6

Branch: refs/heads/HDFS-10285
Commit: 48d532c606e21f3f80c97503dde1552bfd66dbc8
Parents: b1238b2
Author: Rakesh Radhakrishnan 
Authored: Mon Jul 17 22:40:03 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:21 2018 +0530

--
 .../server/namenode/TestStoragePolicySatisfier.java |  9 +
 .../TestStoragePolicySatisfierWithStripedFile.java  | 16 
 2 files changed, 13 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/48d532c6/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
index be7236b..10ceae7 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
@@ -1025,12 +1025,13 @@ public class TestStoragePolicySatisfier {
   list.add(cluster.stopDataNode(0));
   list.add(cluster.stopDataNode(0));
   cluster.restartNameNodes();
-  cluster.restartDataNode(list.get(0), true);
-  cluster.restartDataNode(list.get(1), true);
+  cluster.restartDataNode(list.get(0), false);
+  cluster.restartDataNode(list.get(1), false);
   cluster.waitActive();
   fs.satisfyStoragePolicy(filePath);
-  Thread.sleep(3000 * 6);
-  cluster.restartDataNode(list.get(2), true);
+  DFSTestUtil.waitExpectedStorageType(filePath.toString(),
+  StorageType.ARCHIVE, 2, 3, cluster.getFileSystem());
+  cluster.restartDataNode(list.get(2), false);
   DFSTestUtil.waitExpectedStorageType(filePath.toString(),
   StorageType.ARCHIVE, 3, 3, cluster.getFileSystem());
 } finally {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/48d532c6/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
index f905ead..c070113 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
@@ -308,8 +308,8 @@ public class TestStoragePolicySatisfierWithStripedFile {
*/
   @Test(timeout = 30)
   public void testSPSWhenFileHasLowRedundancyBlocks() throws Exception {
-// start 10 datanodes
-int numOfDatanodes = 10;
+// start 9 datanodes
+int numOfDatanodes = 9;
 int storagesPerDatanode = 2;
 long capacity = 20 * defaultStripeBlockSize;
 long[][] capacities = new long[numOfDatanodes][storagesPerDatanode];
@@ -338,7 +338,6 @@ public class TestStoragePolicySatisfierWithStripedFile {
 {StorageType.DISK, StorageType.ARCHIVE},
 {StorageType.DISK, StorageType.ARCHIVE},
 {StorageType.DISK, StorageType.ARCHIVE},
-{StorageType.DISK, StorageType.ARCHIVE},
 {StorageType.DISK, StorageType.ARCHIVE}})
 .storageCapacities(capacities)
 .build();
@@ -366,15 +365,16 @@ public class TestStoragePolicySatisfierWithStripedFile {
   }
   cluster.restartNameNodes();
   // Restart half datanodes
-  for (int i = 0; i < numOfDatanodes / 2; i++) {
-cluster.restartDataNode(list.get(i), true);
+  for (int i = 0; i < 5; i++) {
+cluster.restartDataNode(list.get(i), false);
   }
   cluster.waitActive();
   fs.satisfyStoragePolicy(fooFile);
-  Thread.sleep(3000 * 6);
+  DFSTestUtil.waitExpectedStorageType(fooFile.toString(),
+  StorageType.ARCHIVE, 5, 3, cluster.getFileSystem());
   //Start reaming datanodes
-  

[08/50] [abbrv] hadoop git commit: HDFS-12141: [SPS]: Fix checkstyle warnings. Contributed by Rakesh R.

[rakeshr]

HDFS-12141: [SPS]: Fix checkstyle warnings. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3d77345b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3d77345b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3d77345b

Branch: refs/heads/HDFS-10285
Commit: 3d77345b2c507f03bf9c08dd9b624b3bb0a5e8cb
Parents: 48d532c
Author: Uma Maheswara Rao G 
Authored: Mon Jul 17 10:24:06 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:25 2018 +0530

--
 .../hdfs/server/blockmanagement/BlockManager.java   |  2 +-
 .../server/datanode/StoragePolicySatisfyWorker.java |  6 +++---
 .../hdfs/server/namenode/StoragePolicySatisfier.java|  6 +++---
 .../hadoop/hdfs/server/protocol/DatanodeProtocol.java   |  5 ++---
 .../org/apache/hadoop/hdfs/server/mover/TestMover.java  |  7 ---
 .../server/namenode/TestStoragePolicySatisfier.java | 12 ++--
 6 files changed, 19 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/3d77345b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index ddf3f6c..5db2c3a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -427,7 +427,7 @@ public class BlockManager implements BlockStatsMXBean {
 
   private final BlockIdManager blockIdManager;
 
-  /** For satisfying block storage policies */
+  /** For satisfying block storage policies. */
   private final StoragePolicySatisfier sps;
   private final BlockStorageMovementNeeded storageMovementNeeded =
   new BlockStorageMovementNeeded();

http://git-wip-us.apache.org/repos/asf/hadoop/blob/3d77345b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
index f4f97dd..196cd58 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
@@ -329,7 +329,7 @@ public class StoragePolicySatisfyWorker {
   /**
* Block movement status code.
*/
-  public static enum BlockMovementStatus {
+  public enum BlockMovementStatus {
 /** Success. */
 DN_BLK_STORAGE_MOVEMENT_SUCCESS(0),
 /**
@@ -343,7 +343,7 @@ public class StoragePolicySatisfyWorker {
 
 private final int code;
 
-private BlockMovementStatus(int code) {
+BlockMovementStatus(int code) {
   this.code = code;
 }
 
@@ -365,7 +365,7 @@ public class StoragePolicySatisfyWorker {
 private final DatanodeInfo target;
 private final BlockMovementStatus status;
 
-public BlockMovementResult(long trackId, long blockId,
+BlockMovementResult(long trackId, long blockId,
 DatanodeInfo target, BlockMovementStatus status) {
   this.trackId = trackId;
   this.blockId = blockId;

http://git-wip-us.apache.org/repos/asf/hadoop/blob/3d77345b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
index 00b4cd0..af3b7f2 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
@@ -714,10 +714,10 @@ public class StoragePolicySatisfier implements Runnable {
   }
 
   private static class StorageTypeNodePair {
-public StorageType storageType = null;
-public DatanodeDescriptor dn = null;
+private StorageType storageType = null;
+private 

[04/50] [abbrv] hadoop git commit: HDFS-11965: [SPS]: Should give chance to satisfy the low redundant blocks before removing the xattr. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-11965: [SPS]: Should give chance to satisfy the low redundant blocks 
before removing the xattr. Contributed by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/03aa3d20
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/03aa3d20
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/03aa3d20

Branch: refs/heads/HDFS-10285
Commit: 03aa3d2088cb683fe587f40ee815a9e95cc04e64
Parents: 4162d3c
Author: Uma Maheswara Rao G 
Authored: Mon Jul 10 18:00:58 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:09 2018 +0530

--
 .../server/blockmanagement/BlockManager.java|  15 +++
 .../server/namenode/StoragePolicySatisfier.java |  20 +++-
 .../namenode/TestStoragePolicySatisfier.java| 102 ++-
 ...stStoragePolicySatisfierWithStripedFile.java |  90 
 4 files changed, 224 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/03aa3d20/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index 1319a2c..ddf3f6c 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -4324,6 +4324,21 @@ public class BlockManager implements BlockStatsMXBean {
   }
 
   /**
+   * Check file has low redundancy blocks.
+   */
+  public boolean hasLowRedundancyBlocks(BlockCollection bc) {
+boolean result = false;
+for (BlockInfo block : bc.getBlocks()) {
+  short expected = getExpectedRedundancyNum(block);
+  final NumberReplicas n = countNodes(block);
+  if (expected > n.liveReplicas()) {
+result = true;
+  }
+}
+return result;
+  }
+
+  /**
* Check sufficient redundancy of the blocks in the collection. If any block
* is needed reconstruction, insert it into the reconstruction queue.
* Otherwise, if the block is more than the expected replication factor,

http://git-wip-us.apache.org/repos/asf/hadoop/blob/03aa3d20/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
index 1b2afa3..97cbf1b 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
@@ -99,7 +99,10 @@ public class StoragePolicySatisfier implements Runnable {
 // Represents that, the analysis skipped due to some conditions.
 // Example conditions are if no blocks really exists in block collection or
 // if analysis is not required on ec files with unsuitable storage policies
-BLOCKS_TARGET_PAIRING_SKIPPED;
+BLOCKS_TARGET_PAIRING_SKIPPED,
+// Represents that, All the reported blocks are satisfied the policy but
+// some of the blocks are low redundant.
+FEW_LOW_REDUNDANCY_BLOCKS
   }
 
   public StoragePolicySatisfier(final Namesystem namesystem,
@@ -247,6 +250,14 @@ public class StoragePolicySatisfier implements Runnable {
   case FEW_BLOCKS_TARGETS_PAIRED:
 this.storageMovementsMonitor.add(blockCollectionID, false);
 break;
+  case FEW_LOW_REDUNDANCY_BLOCKS:
+if (LOG.isDebugEnabled()) {
+  LOG.debug("Adding trackID " + blockCollectionID
+  + " back to retry queue as some of the blocks"
+  + " are low redundant.");
+}
+this.storageMovementNeeded.add(blockCollectionID);
+break;
   // Just clean Xattrs
   case BLOCKS_TARGET_PAIRING_SKIPPED:
   case BLOCKS_ALREADY_SATISFIED:
@@ -347,11 +358,16 @@ public class StoragePolicySatisfier implements Runnable {
 boolean computeStatus = computeBlockMovingInfos(blockMovingInfos,
 blockInfo, expectedStorageTypes, existing, storages);
 if (computeStatus
-&& status != 

[02/50] [abbrv] hadoop git commit: HDFS-11966. [SPS] Correct the log in BlockStorageMovementAttemptedItems#blockStorageMovementResultCheck. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-11966. [SPS] Correct the log in 
BlockStorageMovementAttemptedItems#blockStorageMovementResultCheck. Contributed 
by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/65094cbd
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/65094cbd
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/65094cbd

Branch: refs/heads/HDFS-10285
Commit: 65094cbdc4145549a25a0c2aaa1461f36b5dd276
Parents: 6457596
Author: Rakesh Radhakrishnan 
Authored: Sun Jun 18 11:00:28 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:01 2018 +0530

--
 .../BlockStorageMovementAttemptedItems.java | 39 ++--
 1 file changed, 20 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/65094cbd/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
index bf7859c..6048986 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
@@ -296,19 +296,17 @@ public class BlockStorageMovementAttemptedItems {
 .next();
 synchronized (storageMovementAttemptedItems) {
   Status status = storageMovementAttemptedResult.getStatus();
+  long trackId = storageMovementAttemptedResult.getTrackId();
   ItemInfo itemInfo;
   switch (status) {
   case FAILURE:
-blockStorageMovementNeeded
-.add(storageMovementAttemptedResult.getTrackId());
+blockStorageMovementNeeded.add(trackId);
 LOG.warn("Blocks storage movement results for the tracking id: {}"
 + " is reported from co-ordinating datanode, but result"
-+ " status is FAILURE. So, added for retry",
-storageMovementAttemptedResult.getTrackId());
++ " status is FAILURE. So, added for retry", trackId);
 break;
   case SUCCESS:
-itemInfo = storageMovementAttemptedItems
-.get(storageMovementAttemptedResult.getTrackId());
+itemInfo = storageMovementAttemptedItems.get(trackId);
 
 // ItemInfo could be null. One case is, before the blocks movements
 // result arrives the attempted trackID became timed out and then
@@ -318,20 +316,23 @@ public class BlockStorageMovementAttemptedItems {
 // following condition. If all the block locations under the 
trackID
 // are attempted and failed to find matching target nodes to 
satisfy
 // storage policy in previous SPS iteration.
-if (itemInfo != null
-&& !itemInfo.isAllBlockLocsAttemptedToSatisfy()) {
-  blockStorageMovementNeeded
-  .add(storageMovementAttemptedResult.getTrackId());
-  LOG.warn("Blocks storage movement is SUCCESS for the track id: 
{}"
-  + " reported from co-ordinating datanode. But adding trackID"
-  + " back to retry queue as some of the blocks couldn't find"
-  + " matching target nodes in previous SPS iteration.",
-  storageMovementAttemptedResult.getTrackId());
+String msg = "Blocks storage movement is SUCCESS for the track id: 
"
++ trackId + " reported from co-ordinating datanode.";
+if (itemInfo != null) {
+  if (!itemInfo.isAllBlockLocsAttemptedToSatisfy()) {
+blockStorageMovementNeeded.add(trackId);
+LOG.warn("{} But adding trackID back to retry queue as some of"
++ " the blocks couldn't find matching target nodes in"
++ " previous SPS iteration.", msg);
+  } else {
+LOG.info(msg);
+// Remove xattr for the track id.
+this.sps.postBlkStorageMovementCleanup(
+storageMovementAttemptedResult.getTrackId());
+  }
 } else {
-  LOG.info("Blocks storage movement is SUCCESS for the track id: 
{}"
-  + " reported from co-ordinating datanode. But the trackID "
-  + "doesn't exists in storageMovementAttemptedItems list",
-  

[05/50] [abbrv] hadoop git commit: HDFS-11264: [SPS]: Double checks to ensure that SPS/Mover are not running together. Contributed by Rakesh R.

[rakeshr]

HDFS-11264: [SPS]: Double checks to ensure that SPS/Mover are not running 
together. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/bc4b9e61
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/bc4b9e61
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/bc4b9e61

Branch: refs/heads/HDFS-10285
Commit: bc4b9e61d2a616efbbef3262ba3617e0fc19712c
Parents: 03aa3d2
Author: Uma Maheswara Rao G 
Authored: Wed Jul 12 17:56:56 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Tue Jul 31 12:09:13 2018 +0530

--
 .../server/namenode/StoragePolicySatisfier.java | 53 +++-
 .../namenode/TestStoragePolicySatisfier.java|  3 +-
 ...stStoragePolicySatisfierWithStripedFile.java |  5 +-
 3 files changed, 34 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/bc4b9e61/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
index 97cbf1b..00b4cd0 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
@@ -128,6 +128,14 @@ public class StoragePolicySatisfier implements Runnable {
*/
   public synchronized void start(boolean reconfigStart) {
 isRunning = true;
+if (checkIfMoverRunning()) {
+  isRunning = false;
+  LOG.error(
+  "Stopping StoragePolicySatisfier thread " + "as Mover ID file "
+  + HdfsServerConstants.MOVER_ID_PATH.toString()
+  + " been opened. Maybe a Mover instance is running!");
+  return;
+}
 if (reconfigStart) {
   LOG.info("Starting StoragePolicySatisfier, as admin requested to "
   + "activate it.");
@@ -211,20 +219,6 @@ public class StoragePolicySatisfier implements Runnable {
 
   @Override
   public void run() {
-boolean isMoverRunning = !checkIfMoverRunning();
-synchronized (this) {
-  isRunning = isMoverRunning;
-  if (!isRunning) {
-// Stopping monitor thread and clearing queues as well
-this.clearQueues();
-this.storageMovementsMonitor.stopGracefully();
-LOG.error(
-"Stopping StoragePolicySatisfier thread " + "as Mover ID file "
-+ HdfsServerConstants.MOVER_ID_PATH.toString()
-+ " been opened. Maybe a Mover instance is running!");
-return;
-  }
-}
 while (namesystem.isRunning() && isRunning) {
   try {
 if (!namesystem.isInSafeMode()) {
@@ -274,25 +268,34 @@ public class StoragePolicySatisfier implements Runnable {
 // we want to check block movements.
 Thread.sleep(3000);
   } catch (Throwable t) {
-synchronized (this) {
+handleException(t);
+  }
+}
+  }
+
+  private void handleException(Throwable t) {
+// double check to avoid entering into synchronized block.
+if (isRunning) {
+  synchronized (this) {
+if (isRunning) {
   isRunning = false;
   // Stopping monitor thread and clearing queues as well
   this.clearQueues();
   this.storageMovementsMonitor.stopGracefully();
-}
-if (!namesystem.isRunning()) {
-  LOG.info("Stopping StoragePolicySatisfier.");
-  if (!(t instanceof InterruptedException)) {
-LOG.info("StoragePolicySatisfier received an exception"
-+ " while shutting down.", t);
+  if (!namesystem.isRunning()) {
+LOG.info("Stopping StoragePolicySatisfier.");
+if (!(t instanceof InterruptedException)) {
+  LOG.info("StoragePolicySatisfier received an exception"
+  + " while shutting down.", t);
+}
+return;
   }
-  break;
 }
-LOG.error("StoragePolicySatisfier thread received runtime exception. "
-+ "Stopping Storage policy satisfier work", t);
-break;
   }
 }
+LOG.error("StoragePolicySatisfier thread received runtime exception. "
++ "Stopping Storage policy satisfier work", t);
+return;
   }
 
   private BlocksMovingAnalysisStatus 
analyseBlocksStorageMovementsAndAssignToDN(

hadoop git commit: HDFS-13076: [SPS]: Addendum. Resolve conflicts after rebasing branch to trunk. Contributed by Rakesh R.

[rakeshr]

Repository: hadoop
Updated Branches:
  refs/heads/HDFS-10285 83552dc0b -> ad0f6ba07


HDFS-13076: [SPS]: Addendum. Resolve conflicts after rebasing branch to trunk. 
Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ad0f6ba0
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ad0f6ba0
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ad0f6ba0

Branch: refs/heads/HDFS-10285
Commit: ad0f6ba07f77e9f7659bb7fea7ae239d38ff49dd
Parents: 83552dc
Author: Rakesh Radhakrishnan 
Authored: Fri Jul 20 10:59:16 2018 +0530
Committer: Rakesh Radhakrishnan 
Committed: Fri Jul 20 10:59:16 2018 +0530

--
 .../java/org/apache/hadoop/hdfs/server/datanode/DataNode.java| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ad0f6ba0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
index 565faec..c2aaea2 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
@@ -3621,8 +3621,8 @@ public class DataNode extends ReconfigurableBase
 }
 return this.diskBalancer;
   }
-}
 
   StoragePolicySatisfyWorker getStoragePolicySatisfyWorker() {
 return storagePolicySatisfyWorker;
-  }}
+  }
+}


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[23/50] [abbrv] hadoop git commit: HDFS-12790: [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and HDFS-11968 commits. Contributed by Rakesh R.

[rakeshr]

HDFS-12790: [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and 
HDFS-11968 commits. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9d5f3854
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9d5f3854
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9d5f3854

Branch: refs/heads/HDFS-10285
Commit: 9d5f385478fab3c5b11903631825e87a4920314e
Parents: d3ddec1
Author: Rakesh Radhakrishnan 
Authored: Fri Nov 10 10:06:43 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Thu Jul 19 22:48:12 2018 +0530

--
 .../federation/router/RouterRpcServer.java  |  19 +++
 .../namenode/TestStoragePolicySatisfier.java|   9 +-
 ...stStoragePolicySatisfierWithStripedFile.java |  21 +--
 .../hdfs/tools/TestStoragePolicyCommands.java   |  57 -
 .../TestStoragePolicySatisfyAdminCommands.java  | 127 +++
 5 files changed, 162 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d5f3854/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
 
b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
index 027db8a..c5458f0 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
@@ -87,6 +87,7 @@ import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.ReencryptAction;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.RollingUpgradeAction;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
 import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
 import org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus;
 import org.apache.hadoop.hdfs.protocol.LastBlockWithStatus;
@@ -2490,4 +2491,22 @@ public class RouterRpcServer extends AbstractService
   public FederationRPCMetrics getRPCMetrics() {
 return this.rpcMonitor.getRPCMetrics();
   }
+
+  @Override
+  public void satisfyStoragePolicy(String path) throws IOException {
+checkOperation(OperationCategory.WRITE, false);
+  }
+
+  @Override
+  public boolean isStoragePolicySatisfierRunning() throws IOException {
+checkOperation(OperationCategory.READ, false);
+return false;
+  }
+
+  @Override
+  public StoragePolicySatisfyPathStatus checkStoragePolicySatisfyPathStatus(
+  String path) throws IOException {
+checkOperation(OperationCategory.READ, false);
+return StoragePolicySatisfyPathStatus.NOT_AVAILABLE;
+  }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d5f3854/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
index f42d911..edd1aca 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
@@ -61,6 +61,7 @@ import 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
 import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
 import org.apache.hadoop.hdfs.server.datanode.DataNode;
 import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
+import org.apache.hadoop.hdfs.server.datanode.InternalDataNodeTestUtils;
 import org.apache.hadoop.test.GenericTestUtils;
 import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
 import org.junit.Assert;
@@ -912,8 +913,6 @@ public class TestStoragePolicySatisfier {
 
 int defaultStripedBlockSize =
 StripedFileTestUtil.getDefaultECPolicy().getCellSize() * 4;
-config.set(DFSConfigKeys.DFS_NAMENODE_EC_POLICIES_ENABLED_KEY,
-StripedFileTestUtil.getDefaultECPolicy().getName());
 config.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, defaultStripedBlockSize);
 config.setLong(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1L);
 

[28/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/67d37661/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
deleted file mode 100644
index 9f733ff..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
+++ /dev/null
@@ -1,1775 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.hdfs.server.namenode;
-
-import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_STORAGE_POLICY_ENABLED_KEY;
-import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.slf4j.LoggerFactory.getLogger;
-
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.TimeoutException;
-
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.conf.ReconfigurationException;
-import org.apache.hadoop.fs.FSDataOutputStream;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.StorageType;
-import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.DFSTestUtil;
-import org.apache.hadoop.hdfs.DistributedFileSystem;
-import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
-import org.apache.hadoop.hdfs.NameNodeProxies;
-import org.apache.hadoop.hdfs.StripedFileTestUtil;
-import org.apache.hadoop.hdfs.client.HdfsAdmin;
-import org.apache.hadoop.hdfs.protocol.ClientProtocol;
-import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
-import org.apache.hadoop.hdfs.protocol.HdfsConstants;
-import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
-import org.apache.hadoop.hdfs.protocol.LocatedBlock;
-import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
-import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
-import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
-import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
-import org.apache.hadoop.hdfs.server.datanode.DataNode;
-import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
-import org.apache.hadoop.hdfs.server.datanode.InternalDataNodeTestUtils;
-import org.apache.hadoop.test.GenericTestUtils;
-import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
-import org.junit.Assert;
-import org.junit.Test;
-import org.mockito.Mockito;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.slf4j.event.Level;
-
-import com.google.common.base.Supplier;
-
-/**
- * Tests that StoragePolicySatisfier daemon is able to check the blocks to be
- * moved and finding its suggested target locations to move.
- */
-public class TestStoragePolicySatisfier {
-
-  {
-GenericTestUtils.setLogLevel(
-getLogger(FSTreeTraverser.class), Level.DEBUG);
-  }
-
-  private static final String ONE_SSD = "ONE_SSD";
-  private static final String COLD = "COLD";
-  private static final Logger LOG =
-  LoggerFactory.getLogger(TestStoragePolicySatisfier.class);
-  private final Configuration config = new HdfsConfiguration();
-  private StorageType[][] allDiskTypes =
-  new StorageType[][]{{StorageType.DISK, StorageType.DISK},
-  {StorageType.DISK, StorageType.DISK},
-  {StorageType.DISK, StorageType.DISK}};
-  private MiniDFSCluster hdfsCluster = null;
-  final private int numOfDatanodes = 3;
-  

[27/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/67d37661/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
deleted file mode 100644
index 6991ad2..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfierWithStripedFile.java
+++ /dev/null
@@ -1,580 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.hdfs.server.namenode;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.TimeoutException;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.StorageType;
-import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.DFSTestUtil;
-import org.apache.hadoop.hdfs.DistributedFileSystem;
-import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
-import org.apache.hadoop.hdfs.NameNodeProxies;
-import org.apache.hadoop.hdfs.StripedFileTestUtil;
-import org.apache.hadoop.hdfs.client.HdfsAdmin;
-import org.apache.hadoop.hdfs.protocol.ClientProtocol;
-import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
-import org.apache.hadoop.hdfs.protocol.HdfsConstants;
-import org.apache.hadoop.hdfs.protocol.LocatedBlock;
-import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
-import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
-import org.apache.hadoop.test.GenericTestUtils;
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.Test;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.base.Supplier;
-
-/**
- * Tests that StoragePolicySatisfier daemon is able to check the striped blocks
- * to be moved and finding its expected target locations in order to satisfy 
the
- * storage policy.
- */
-public class TestStoragePolicySatisfierWithStripedFile {
-
-  private static final Logger LOG = LoggerFactory
-  .getLogger(TestStoragePolicySatisfierWithStripedFile.class);
-
-  private final int stripesPerBlock = 2;
-
-  private ErasureCodingPolicy ecPolicy;
-  private int dataBlocks;
-  private int parityBlocks;
-  private int cellSize;
-  private int defaultStripeBlockSize;
-
-  private ErasureCodingPolicy getEcPolicy() {
-return StripedFileTestUtil.getDefaultECPolicy();
-  }
-
-  /**
-   * Initialize erasure coding policy.
-   */
-  @Before
-  public void init(){
-ecPolicy = getEcPolicy();
-dataBlocks = ecPolicy.getNumDataUnits();
-parityBlocks = ecPolicy.getNumParityUnits();
-cellSize = ecPolicy.getCellSize();
-defaultStripeBlockSize = cellSize * stripesPerBlock;
-  }
-
-  /**
-   * Tests to verify that all the striped blocks(data + parity blocks) are
-   * moving to satisfy the storage policy.
-   */
-  @Test(timeout = 30)
-  public void testMoverWithFullStripe() throws Exception {
-// start 10 datanodes
-int numOfDatanodes = 10;
-int storagesPerDatanode = 2;
-long capacity = 20 * defaultStripeBlockSize;
-long[][] capacities = new long[numOfDatanodes][storagesPerDatanode];
-for (int i = 0; i < numOfDatanodes; i++) {
-  for (int j = 0; j < storagesPerDatanode; j++) {
-capacities[i][j] = capacity;
-  }
-}
-
-final Configuration conf = new HdfsConfiguration();
-conf.setBoolean(DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
-true);
-initConfWithStripe(conf, defaultStripeBlockSize);
-final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
-.numDataNodes(numOfDatanodes)
-.storagesPerDatanode(storagesPerDatanode)
-   

[11/50] [abbrv] hadoop git commit: HDFS-11874. [SPS]: Document the SPS feature. Contributed by Uma Maheswara Rao G

[rakeshr]

HDFS-11874. [SPS]: Document the SPS feature. Contributed by Uma Maheswara Rao G


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f4bc889b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f4bc889b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f4bc889b

Branch: refs/heads/HDFS-10285
Commit: f4bc889b042c18d5d760c6329cbfb04c0d0a1c78
Parents: 18c3709
Author: Rakesh Radhakrishnan 
Authored: Fri Jul 14 22:36:09 2017 +0530
Committer: Rakesh Radhakrishnan 
Committed: Thu Jul 19 22:46:57 2018 +0530

--
 .../src/site/markdown/ArchivalStorage.md| 51 ++--
 1 file changed, 48 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/f4bc889b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
index a56cf8b..9098616 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
@@ -97,8 +97,44 @@ The effective storage policy can be retrieved by the 
"[`storagepolicies -getStor
 
 The default storage type of a datanode storage location will be DISK if it 
does not have a storage type tagged explicitly.
 
-Mover - A New Data Migration Tool
--
+Storage Policy Based Data Movement
+--
+
+Setting a new storage policy on already existing file/dir will change the 
policy in Namespace, but it will not move the blocks physically across storage 
medias.
+Following 2 options will allow users to move the blocks based on new policy 
set. So, once user change/set to a new policy on file/directory, user should 
also perform one of the following options to achieve the desired data movement. 
Note that both options cannot be allowed to run simultaneously.
+
+### Storage Policy Satisfier (SPS)
+
+When user changes the storage policy on a file/directory, user can call 
`HdfsAdmin` API `satisfyStoragePolicy()` to move the blocks as per the new 
policy set.
+The SPS daemon thread runs along with namenode and periodically scans for the 
storage mismatches between new policy set and the physical blocks placed. This 
will only track the files/directories for which user invoked 
satisfyStoragePolicy. If SPS identifies some blocks to be moved for a file, 
then it will schedule block movement tasks to datanodes. A Coordinator 
DataNode(C-DN) will track all block movements associated to a file and notify 
to namenode about movement success/failure. If there are any failures in 
movement, the SPS will re-attempt by sending new block movement task.
+
+SPS can be activated and deactivated dynamically without restarting the 
Namenode.
+
+Detailed design documentation can be found at [Storage Policy Satisfier(SPS) 
(HDFS-10285)](https://issues.apache.org/jira/browse/HDFS-10285)
+
+* **Note**: When user invokes `satisfyStoragePolicy()` API on a directory, SPS 
will consider the files which are immediate to that directory. Sub-directories 
won't be considered for satisfying the policy. Its user responsibility to call 
this API on directories recursively, to track all files under the sub tree.
+
+* HdfsAdmin API :
+`public void satisfyStoragePolicy(final Path path) throws IOException`
+
+* Arguments :
+
+| | |
+|: |: |
+| `path` | A path which requires blocks storage movement. |
+
+Configurations:
+
+*   **dfs.storage.policy.satisfier.activate** - Used to activate or deactivate 
SPS. Configuring true represents SPS is
+   activated and vice versa.
+
+*   **dfs.storage.policy.satisfier.recheck.timeout.millis** - A timeout to 
re-check the processed block storage movement
+   command results from Co-ordinator Datanode.
+
+*   **dfs.storage.policy.satisfier.self.retry.timeout.millis** - A timeout to 
retry if no block movement results reported from
+   Co-ordinator Datanode in this configured timeout.
+
+### Mover - A New Data Migration Tool
 
 A new data migration tool is added for archiving data. The tool is similar to 
Balancer. It periodically scans the files in HDFS to check if the block 
placement satisfies the storage policy. For the blocks violating the storage 
policy, it moves the replicas to a different storage type in order to fulfill 
the storage policy requirement. Note that it always tries to move block 
replicas within the same node whenever possible. If that is not possible (e.g. 
when a node doesn’t have the target storage type) then it will copy the block 
replicas to another node over the network.
 
@@ -115,6 +151,10 @@ A new data migration tool 

[42/50] [abbrv] hadoop git commit: HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by Surendra Singh.

[rakeshr]

HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by 
Surendra Singh.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4b9839c5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4b9839c5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4b9839c5

Branch: refs/heads/HDFS-10285
Commit: 4b9839c57463ecd16ed1c0d08f35dd7308b7adbd
Parents: e5603ae
Author: Uma Maheswara Rao G 
Authored: Wed Feb 7 02:28:23 2018 -0800
Committer: Rakesh Radhakrishnan 
Committed: Thu Jul 19 22:49:43 2018 +0530

--
 .../java/org/apache/hadoop/hdfs/DFSClient.java  |   4 +-
 .../hadoop/hdfs/protocol/ClientProtocol.java|   6 +-
 .../ClientNamenodeProtocolTranslatorPB.java |  14 +-
 .../src/main/proto/ClientNamenodeProtocol.proto |   8 +-
 .../federation/router/RouterRpcServer.java  |   2 +-
 .../java/org/apache/hadoop/hdfs/DFSUtil.java|  61 ---
 ...tNamenodeProtocolServerSideTranslatorPB.java |  16 +-
 .../server/blockmanagement/BlockManager.java| 255 +---
 .../blockmanagement/DatanodeDescriptor.java |  33 +-
 .../hdfs/server/common/HdfsServerConstants.java |   2 +-
 .../datanode/StoragePolicySatisfyWorker.java|  15 +-
 .../apache/hadoop/hdfs/server/mover/Mover.java  |   2 +-
 .../namenode/FSDirSatisfyStoragePolicyOp.java   |  26 +-
 .../server/namenode/FSDirStatAndListingOp.java  |   1 -
 .../hdfs/server/namenode/FSDirXAttrOp.java  |   2 +-
 .../hdfs/server/namenode/FSDirectory.java   |   2 +-
 .../hdfs/server/namenode/FSNamesystem.java  |  46 +--
 .../hadoop/hdfs/server/namenode/NameNode.java   |  30 +-
 .../hdfs/server/namenode/NameNodeRpcServer.java |  21 +-
 .../sps/BlockStorageMovementNeeded.java |   4 +-
 .../namenode/sps/IntraSPSNameNodeContext.java   |   6 +-
 .../hdfs/server/namenode/sps/SPSPathIds.java|  70 
 .../hdfs/server/namenode/sps/SPSService.java|  10 +-
 .../namenode/sps/StoragePolicySatisfier.java| 137 ---
 .../sps/StoragePolicySatisfyManager.java| 399 +++
 .../sps/ExternalStoragePolicySatisfier.java |   2 +-
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   |   2 +-
 .../namenode/TestNameNodeReconfigure.java   |  19 +-
 .../TestPersistentStoragePolicySatisfier.java   |   3 +-
 .../TestStoragePolicySatisfierWithHA.java   |   6 +-
 .../sps/TestStoragePolicySatisfier.java |  35 +-
 ...stStoragePolicySatisfierWithStripedFile.java |   6 +-
 .../sps/TestExternalStoragePolicySatisfier.java |  24 +-
 33 files changed, 665 insertions(+), 604 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/4b9839c5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
index 471ab2c..b6f9bdd 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@@ -3110,8 +3110,8 @@ public class DFSClient implements java.io.Closeable, 
RemotePeerFactory,
 }
   }
 
-  public boolean isStoragePolicySatisfierRunning() throws IOException {
-return namenode.isStoragePolicySatisfierRunning();
+  public boolean isInternalSatisfierRunning() throws IOException {
+return namenode.isInternalSatisfierRunning();
   }
 
   Tracer getTracer() {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/4b9839c5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
index 360fd63..5c51c22 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
@@ -1759,12 +1759,12 @@ public interface ClientProtocol {
   void satisfyStoragePolicy(String path) throws IOException;
 
   /**
-   * Check if StoragePolicySatisfier is running.
-   * @return true if StoragePolicySatisfier is running
+   * Check if internal StoragePolicySatisfier is running.
+   * @return true if internal StoragePolicySatisfier is running
* @throws IOException
*/
   @Idempotent
-  boolean isStoragePolicySatisfierRunning() throws IOException;
+  boolean 

[08/50] [abbrv] hadoop git commit: HDFS-11670: [SPS]: Add CLI command for satisfy storage policy operations. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-11670: [SPS]: Add CLI command for satisfy storage policy operations. 
Contributed by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9e745968
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9e745968
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9e745968

Branch: refs/heads/HDFS-10285
Commit: 9e7459681d88c977b1a85fdcd2c753dfcc11
Parents: 9a27c91
Author: Uma Maheswara Rao G 
Authored: Mon Jun 19 17:16:49 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Thu Jul 19 22:46:36 2018 +0530

--
 .../hadoop/hdfs/tools/StoragePolicyAdmin.java   | 93 +++-
 .../src/site/markdown/ArchivalStorage.md| 21 +
 .../src/site/markdown/HDFSCommands.md   |  2 +
 .../hdfs/tools/TestStoragePolicyCommands.java   | 43 -
 4 files changed, 157 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/9e745968/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
index aeb10d9..662957c 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
@@ -23,6 +23,7 @@ import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.BlockStoragePolicySpi;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
 import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
 import org.apache.hadoop.hdfs.protocol.HdfsConstants;
 import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
@@ -32,6 +33,8 @@ import org.apache.hadoop.util.Tool;
 import org.apache.hadoop.util.ToolRunner;
 
 import java.io.FileNotFoundException;
+import com.google.common.base.Joiner;
+
 import java.io.IOException;
 import java.util.Arrays;
 import java.util.Collection;
@@ -245,6 +248,92 @@ public class StoragePolicyAdmin extends Configured 
implements Tool {
 }
   }
 
+  /** Command to schedule blocks to move based on specified policy. */
+  private static class SatisfyStoragePolicyCommand implements
+  AdminHelper.Command {
+@Override
+public String getName() {
+  return "-satisfyStoragePolicy";
+}
+
+@Override
+public String getShortUsage() {
+  return "[" + getName() + " -path ]\n";
+}
+
+@Override
+public String getLongUsage() {
+  TableListing listing = AdminHelper.getOptionDescriptionListing();
+  listing.addRow("", "The path of the file/directory to satisfy"
+  + " storage policy");
+  return getShortUsage() + "\n" +
+  "Schedule blocks to move based on file/directory policy.\n\n" +
+  listing.toString();
+}
+
+@Override
+public int run(Configuration conf, List args) throws IOException {
+  final String path = StringUtils.popOptionWithArgument("-path", args);
+  if (path == null) {
+System.err.println("Please specify the path for setting the storage " +
+"policy.\nUsage: " + getLongUsage());
+return 1;
+  }
+
+  final DistributedFileSystem dfs = AdminHelper.getDFS(conf);
+  try {
+dfs.satisfyStoragePolicy(new Path(path));
+System.out.println("Scheduled blocks to move based on the current"
++ " storage policy on " + path);
+  } catch (Exception e) {
+System.err.println(AdminHelper.prettifyException(e));
+return 2;
+  }
+  return 0;
+}
+  }
+
+  /** Command to check storage policy satisfier status. */
+  private static class IsSPSRunningCommand implements AdminHelper.Command {
+@Override
+public String getName() {
+  return "-isSPSRunning";
+}
+
+@Override
+public String getShortUsage() {
+  return "[" + getName() + "]\n";
+}
+
+@Override
+public String getLongUsage() {
+  return getShortUsage() + "\n" +
+  "Check the status of Storage Policy Statisfier.\n\n";
+}
+
+@Override
+public int run(Configuration conf, List args) throws IOException {
+  if (!args.isEmpty()) {
+System.err.print("Can't understand arguments: "
++ Joiner.on(" ").join(args) + "\n");
+System.err.println("Usage is " + getLongUsage());
+return 1;
+  }
+  final DistributedFileSystem dfs = AdminHelper.getDFS(conf);
+  try {
+if(dfs.getClient().isStoragePolicySatisfierRunning()){
+  

[29/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/67d37661/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
new file mode 100644
index 000..5635621
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/BlockStorageMovementNeeded.java
@@ -0,0 +1,572 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode.sps;
+
+import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
+import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
+import org.apache.hadoop.hdfs.server.namenode.FSDirectory;
+import org.apache.hadoop.hdfs.server.namenode.FSTreeTraverser;
+import org.apache.hadoop.hdfs.server.namenode.INode;
+import org.apache.hadoop.hdfs.server.namenode.Namesystem;
+import 
org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier.ItemInfo;
+import org.apache.hadoop.hdfs.server.namenode.FSTreeTraverser.TraverseInfo;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+
+/**
+ * A Class to track the block collection IDs (Inode's ID) for which physical
+ * storage movement needed as per the Namespace and StorageReports from DN.
+ * It scan the pending directories for which storage movement is required and
+ * schedule the block collection IDs for movement. It track the info of
+ * scheduled items and remove the SPS xAttr from the file/Directory once
+ * movement is success.
+ */
+@InterfaceAudience.Private
+public class BlockStorageMovementNeeded {
+
+  public static final Logger LOG =
+  LoggerFactory.getLogger(BlockStorageMovementNeeded.class);
+
+  private final Queue storageMovementNeeded =
+  new LinkedList();
+
+  /**
+   * Map of startId and number of child's. Number of child's indicate the
+   * number of files pending to satisfy the policy.
+   */
+  private final Map pendingWorkForDirectory =
+  new HashMap();
+
+  private final Map spsStatus =
+  new ConcurrentHashMap<>();
+
+  private final Namesystem namesystem;
+
+  // List of pending dir to satisfy the policy
+  private final Queue spsDirsToBeTraveresed = new LinkedList();
+
+  private final StoragePolicySatisfier sps;
+
+  private Daemon inodeIdCollector;
+
+  private final int maxQueuedItem;
+
+  // Amount of time to cache the SUCCESS status of path before turning it to
+  // NOT_AVAILABLE.
+  private static long statusClearanceElapsedTimeMs = 30;
+
+  public BlockStorageMovementNeeded(Namesystem namesystem,
+  StoragePolicySatisfier sps, int queueLimit) {
+this.namesystem = namesystem;
+this.sps = sps;
+this.maxQueuedItem = queueLimit;
+  }
+
+  /**
+   * Add the candidate to tracking list for which storage movement
+   * expected if necessary.
+   *
+   * @param trackInfo
+   *  - track info for satisfy the policy
+   */
+  public synchronized void add(ItemInfo trackInfo) {
+spsStatus.put(trackInfo.getStartId(),
+new StoragePolicySatisfyPathStatusInfo(
+StoragePolicySatisfyPathStatus.IN_PROGRESS));
+storageMovementNeeded.add(trackInfo);
+  }
+
+  /**
+   * Add the itemInfo to tracking list for which storage movement
+   * expected if necessary.
+   * @param startId
+   *- start id
+  

[35/50] [abbrv] hadoop git commit: HDFS-13033: [SPS]: Implement a mechanism to do file block movements for external SPS. Contributed by Rakesh R.

[rakeshr]

HDFS-13033: [SPS]: Implement a mechanism to do file block movements for 
external SPS. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/941180d8
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/941180d8
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/941180d8

Branch: refs/heads/HDFS-10285
Commit: 941180d82fe076ab3f37c1948d129e1f314713e4
Parents: 49cad26
Author: Uma Maheswara Rao G 
Authored: Tue Jan 23 16:19:46 2018 -0800
Committer: Rakesh Radhakrishnan 
Committed: Thu Jul 19 22:48:56 2018 +0530

--
 .../hdfs/server/balancer/NameNodeConnector.java |   8 +
 .../hdfs/server/common/sps/BlockDispatcher.java | 186 +
 .../sps/BlockMovementAttemptFinished.java   |  80 ++
 .../server/common/sps/BlockMovementStatus.java  |  53 
 .../common/sps/BlockStorageMovementTracker.java | 184 +
 .../sps/BlocksMovementsStatusHandler.java   |  95 +++
 .../hdfs/server/common/sps/package-info.java|  27 ++
 .../datanode/BlockStorageMovementTracker.java   | 186 -
 .../datanode/StoragePolicySatisfyWorker.java| 271 ++-
 .../hdfs/server/namenode/FSNamesystem.java  |   4 +-
 .../namenode/sps/BlockMoveTaskHandler.java  |   3 +-
 .../sps/BlockStorageMovementAttemptedItems.java |  12 +-
 .../IntraSPSNameNodeBlockMoveTaskHandler.java   |   3 +-
 .../hdfs/server/namenode/sps/SPSService.java|  14 +-
 .../namenode/sps/StoragePolicySatisfier.java|  30 +-
 .../sps/ExternalSPSBlockMoveTaskHandler.java| 233 
 .../TestBlockStorageMovementAttemptedItems.java |   2 +-
 .../sps/TestStoragePolicySatisfier.java |   6 +-
 .../sps/TestExternalStoragePolicySatisfier.java |  69 -
 19 files changed, 997 insertions(+), 469 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/941180d8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
index b0dd779..6bfbbb3 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
@@ -269,6 +269,14 @@ public class NameNodeConnector implements Closeable {
 }
   }
 
+  /**
+   * Returns fallbackToSimpleAuth. This will be true or false during calls to
+   * indicate if a secure client falls back to simple auth.
+   */
+  public AtomicBoolean getFallbackToSimpleAuth() {
+return fallbackToSimpleAuth;
+  }
+
   @Override
   public void close() {
 keyManager.close();

http://git-wip-us.apache.org/repos/asf/hadoop/blob/941180d8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/sps/BlockDispatcher.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/sps/BlockDispatcher.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/sps/BlockDispatcher.java
new file mode 100644
index 000..f87fcae
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/sps/BlockDispatcher.java
@@ -0,0 +1,186 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.common.sps;
+
+import static org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import 

[26/50] [abbrv] hadoop git commit: HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/67d37661/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
new file mode 100644
index 000..8dc52dc
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/sps/TestStoragePolicySatisfier.java
@@ -0,0 +1,1779 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode.sps;
+
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_STORAGE_POLICY_ENABLED_KEY;
+import static 
org.apache.hadoop.hdfs.server.common.HdfsServerConstants.XATTR_SATISFY_STORAGE_POLICY;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.slf4j.LoggerFactory.getLogger;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.ReconfigurationException;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties;
+import org.apache.hadoop.hdfs.NameNodeProxies;
+import org.apache.hadoop.hdfs.StripedFileTestUtil;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
+import 
org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfyPathStatus;
+import org.apache.hadoop.hdfs.protocol.LocatedBlock;
+import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
+import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
+import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
+import org.apache.hadoop.hdfs.server.datanode.DataNode;
+import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
+import org.apache.hadoop.hdfs.server.datanode.InternalDataNodeTestUtils;
+import org.apache.hadoop.hdfs.server.namenode.FSDirectory;
+import org.apache.hadoop.hdfs.server.namenode.FSEditLog;
+import org.apache.hadoop.hdfs.server.namenode.FSTreeTraverser;
+import org.apache.hadoop.hdfs.server.namenode.INode;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import com.google.common.base.Supplier;
+
+/**
+ * Tests that StoragePolicySatisfier daemon is able to check the blocks to be
+ * moved and finding its suggested target locations to move.
+ */
+public class TestStoragePolicySatisfier {
+
+  {
+GenericTestUtils.setLogLevel(
+getLogger(FSTreeTraverser.class), Level.DEBUG);
+  }
+
+  private static final String ONE_SSD = "ONE_SSD";
+  private static final String COLD = "COLD";
+  private static final Logger LOG =
+  LoggerFactory.getLogger(TestStoragePolicySatisfier.class);
+  private final Configuration config = new HdfsConfiguration();
+  private StorageType[][] allDiskTypes =
+ 

[16/50] [abbrv] hadoop git commit: HDFS-12225: [SPS]: Optimize extended attributes for tracking SPS movements. Contributed by Surendra Singh Lilhore.

[rakeshr]

HDFS-12225: [SPS]: Optimize extended attributes for tracking SPS movements. 
Contributed by Surendra Singh Lilhore.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ecef680b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ecef680b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ecef680b

Branch: refs/heads/HDFS-10285
Commit: ecef680bfaecef9bc723c0181dcc2ad6b65d04c4
Parents: 98c040b
Author: Uma Maheswara Rao G 
Authored: Wed Aug 23 15:37:03 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Thu Jul 19 22:47:34 2018 +0530

--
 .../server/blockmanagement/BlockManager.java|  21 +-
 .../server/blockmanagement/DatanodeManager.java |  14 +-
 .../hdfs/server/datanode/BPOfferService.java|   1 +
 .../BlockStorageMovementAttemptedItems.java |  95 +---
 .../namenode/BlockStorageMovementNeeded.java| 233 ++-
 .../namenode/FSDirSatisfyStoragePolicyOp.java   |  91 +++-
 .../hdfs/server/namenode/FSDirXAttrOp.java  |  11 +-
 .../hdfs/server/namenode/FSDirectory.java   |   2 +-
 .../hdfs/server/namenode/FSNamesystem.java  |   2 +-
 .../server/namenode/StoragePolicySatisfier.java | 108 ++---
 .../TestStoragePolicySatisfyWorker.java |   5 +-
 .../TestBlockStorageMovementAttemptedItems.java |  34 +--
 .../TestPersistentStoragePolicySatisfier.java   | 104 +
 .../namenode/TestStoragePolicySatisfier.java| 127 +-
 14 files changed, 589 insertions(+), 259 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ecef680b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index a442a92..0ee558a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -89,7 +89,6 @@ import 
org.apache.hadoop.hdfs.server.namenode.INode.BlocksMapUpdateInfo;
 import org.apache.hadoop.hdfs.server.namenode.INodesInPath;
 import org.apache.hadoop.hdfs.server.namenode.NameNode;
 import org.apache.hadoop.hdfs.server.namenode.Namesystem;
-import org.apache.hadoop.hdfs.server.namenode.BlockStorageMovementNeeded;
 import org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier;
 import org.apache.hadoop.hdfs.server.namenode.ha.HAContext;
 import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
@@ -431,9 +430,6 @@ public class BlockManager implements BlockStatsMXBean {
   private final StoragePolicySatisfier sps;
   private final boolean storagePolicyEnabled;
   private boolean spsEnabled;
-  private final BlockStorageMovementNeeded storageMovementNeeded =
-  new BlockStorageMovementNeeded();
-
   /** Minimum live replicas needed for the datanode to be transitioned
* from ENTERING_MAINTENANCE to IN_MAINTENANCE.
*/
@@ -480,8 +476,7 @@ public class BlockManager implements BlockStatsMXBean {
 conf.getBoolean(
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_KEY,
 DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ENABLED_DEFAULT);
-sps = new StoragePolicySatisfier(namesystem, storageMovementNeeded, this,
-conf);
+sps = new StoragePolicySatisfier(namesystem, this, conf);
 blockTokenSecretManager = createBlockTokenSecretManager(conf);
 
 providedStorageMap = new ProvidedStorageMap(namesystem, this, conf);
@@ -5009,20 +5004,6 @@ public class BlockManager implements BlockStatsMXBean {
   }
 
   /**
-   * Set file block collection for which storage movement needed for its 
blocks.
-   *
-   * @param id
-   *  - file block collection id.
-   */
-  public void satisfyStoragePolicy(long id) {
-storageMovementNeeded.add(id);
-if (LOG.isDebugEnabled()) {
-  LOG.debug("Added block collection id {} to block "
-  + "storageMovementNeeded queue", id);
-}
-  }
-
-  /**
* Gets the storage policy satisfier instance.
*
* @return sps

http://git-wip-us.apache.org/repos/asf/hadoop/blob/ecef680b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
 

[13/50] [abbrv] hadoop git commit: HDFS-12141: [SPS]: Fix checkstyle warnings. Contributed by Rakesh R.

[rakeshr]

HDFS-12141: [SPS]: Fix checkstyle warnings. Contributed by Rakesh R.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e0e34f39
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e0e34f39
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e0e34f39

Branch: refs/heads/HDFS-10285
Commit: e0e34f3952ec838d082f92ba5ca776ac7e677c28
Parents: d2ba0a5
Author: Uma Maheswara Rao G 
Authored: Mon Jul 17 10:24:06 2017 -0700
Committer: Rakesh Radhakrishnan 
Committed: Thu Jul 19 22:47:13 2018 +0530

--
 .../hdfs/server/blockmanagement/BlockManager.java   |  2 +-
 .../server/datanode/StoragePolicySatisfyWorker.java |  6 +++---
 .../hdfs/server/namenode/StoragePolicySatisfier.java|  6 +++---
 .../hadoop/hdfs/server/protocol/DatanodeProtocol.java   |  5 ++---
 .../org/apache/hadoop/hdfs/server/mover/TestMover.java  |  7 ---
 .../server/namenode/TestStoragePolicySatisfier.java | 12 ++--
 6 files changed, 19 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/e0e34f39/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index ddf3f6c..5db2c3a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -427,7 +427,7 @@ public class BlockManager implements BlockStatsMXBean {
 
   private final BlockIdManager blockIdManager;
 
-  /** For satisfying block storage policies */
+  /** For satisfying block storage policies. */
   private final StoragePolicySatisfier sps;
   private final BlockStorageMovementNeeded storageMovementNeeded =
   new BlockStorageMovementNeeded();

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e0e34f39/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
index f4f97dd..196cd58 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
@@ -329,7 +329,7 @@ public class StoragePolicySatisfyWorker {
   /**
* Block movement status code.
*/
-  public static enum BlockMovementStatus {
+  public enum BlockMovementStatus {
 /** Success. */
 DN_BLK_STORAGE_MOVEMENT_SUCCESS(0),
 /**
@@ -343,7 +343,7 @@ public class StoragePolicySatisfyWorker {
 
 private final int code;
 
-private BlockMovementStatus(int code) {
+BlockMovementStatus(int code) {
   this.code = code;
 }
 
@@ -365,7 +365,7 @@ public class StoragePolicySatisfyWorker {
 private final DatanodeInfo target;
 private final BlockMovementStatus status;
 
-public BlockMovementResult(long trackId, long blockId,
+BlockMovementResult(long trackId, long blockId,
 DatanodeInfo target, BlockMovementStatus status) {
   this.trackId = trackId;
   this.blockId = blockId;

http://git-wip-us.apache.org/repos/asf/hadoop/blob/e0e34f39/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
index 00b4cd0..af3b7f2 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
@@ -714,10 +714,10 @@ public class StoragePolicySatisfier implements Runnable {
   }
 
   private static class StorageTypeNodePair {
-public StorageType storageType = null;
-public DatanodeDescriptor dn = null;
+private StorageType storageType = null;
+private 

[19/50] [abbrv] hadoop git commit: HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements. Contributed by Rakesh R.

[rakeshr]

http://git-wip-us.apache.org/repos/asf/hadoop/blob/f7e8b1c1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
index d3c5cb1..2f621e6 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
@@ -156,7 +156,7 @@ import 
org.apache.hadoop.hdfs.server.common.IncorrectVersionException;
 import org.apache.hadoop.hdfs.server.namenode.NameNode.OperationCategory;
 import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
 import org.apache.hadoop.hdfs.server.protocol.BlockReportContext;
-import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMovementResult;
+import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMoveAttemptFinished;
 import org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations;
 import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand;
 import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol;
@@ -1517,14 +1517,15 @@ public class NameNodeRpcServer implements 
NamenodeProtocols {
   boolean requestFullBlockReportLease,
   @Nonnull SlowPeerReports slowPeers,
   @Nonnull SlowDiskReports slowDisks,
-  BlocksStorageMovementResult[] blkMovementStatus) throws IOException {
+  BlocksStorageMoveAttemptFinished storageMovementFinishedBlks)
+  throws IOException {
 checkNNStartup();
 verifyRequest(nodeReg);
 return namesystem.handleHeartbeat(nodeReg, report,
 dnCacheCapacity, dnCacheUsed, xceiverCount, xmitsInProgress,
 failedVolumes, volumeFailureSummary, requestFullBlockReportLease,
 slowPeers, slowDisks,
-blkMovementStatus);
+storageMovementFinishedBlks);
   }
 
   @Override // DatanodeProtocol

http://git-wip-us.apache.org/repos/asf/hadoop/blob/f7e8b1c1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
index a4372d5..a28a806 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
@@ -17,6 +17,8 @@
  */
 package org.apache.hadoop.hdfs.server.namenode;
 
+import static org.apache.hadoop.util.Time.monotonicNow;
+
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
@@ -44,7 +46,7 @@ import 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
 import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo;
 import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
 import 
org.apache.hadoop.hdfs.server.protocol.BlockStorageMovementCommand.BlockMovingInfo;
-import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMovementResult;
+import org.apache.hadoop.hdfs.server.protocol.BlocksStorageMoveAttemptFinished;
 import org.apache.hadoop.hdfs.server.protocol.StorageReport;
 import org.apache.hadoop.hdfs.util.StripedBlockUtil;
 import org.apache.hadoop.util.Daemon;
@@ -82,25 +84,38 @@ public class StoragePolicySatisfier implements Runnable {
   /**
* Represents the collective analysis status for all blocks.
*/
-  private enum BlocksMovingAnalysisStatus {
-// Represents that, the analysis skipped due to some conditions. A such
-// condition is if block collection is in incomplete state.
-ANALYSIS_SKIPPED_FOR_RETRY,
-// Represents that, all block storage movement needed blocks found its
-// targets.
-ALL_BLOCKS_TARGETS_PAIRED,
-// Represents that, only fewer or none of the block storage movement needed
-// block found its eligible targets.
-FEW_BLOCKS_TARGETS_PAIRED,
-// Represents that, none of the blocks found for block storage movements.
-BLOCKS_ALREADY_SATISFIED,
-// Represents that, the analysis skipped due to some conditions.
-// Example conditions are if no blocks really exists in block collection or
-// if analysis is not required on ec files with unsuitable storage policies
-BLOCKS_TARGET_PAIRING_SKIPPED,
-// Represents that, All the reported blocks are satisfied the policy but
-// some of the blocks are low redundant.
-FEW_LOW_REDUNDANCY_BLOCKS
+  private