[2/2] hadoop git commit: HDFS-12112. TestBlockManager#testBlockManagerMachinesArray sometimes fails with NPE. Contributed by Wei-Chiu Chuang.

2017-07-15 Thread brahma 
HDFS-12112. TestBlockManager#testBlockManagerMachinesArray sometimes fails with 
NPE. Contributed by Wei-Chiu Chuang.

(cherry picked from commit b778887af59d96f1fac30cae14be1cabbdb74c8b)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6039a2a5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6039a2a5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6039a2a5

Branch: refs/heads/branch-2.8
Commit: 6039a2a55f9ff0a415102f02f880c3808199a68d
Parents: 8b7d1df
Author: Brahma Reddy Battula 
Authored: Sat Jul 15 10:38:31 2017 +0800
Committer: Brahma Reddy Battula 
Committed: Sat Jul 15 11:12:56 2017 +0800

--
 .../hadoop/hdfs/server/blockmanagement/TestBlockManager.java | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/6039a2a5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
index 111bfa8..14b04c0 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
@@ -1149,7 +1149,7 @@ public class TestBlockManager {
 }
   }
 
-  @Test
+  @Test(timeout = 6)
   public void testBlockManagerMachinesArray() throws Exception {
 final Configuration conf = new HdfsConfiguration();
 final MiniDFSCluster cluster =
@@ -1160,6 +1160,8 @@ public class TestBlockManager {
 final Path filePath = new Path("/tmp.txt");
 final long fileLen = 1L;
 DFSTestUtil.createFile(fs, filePath, fileLen, (short) 3, 1L);
+DFSTestUtil.waitForReplication((DistributedFileSystem)fs,
+filePath, (short) 3, 6);
 ArrayList datanodes = cluster.getDataNodes();
 assertEquals(datanodes.size(), 4);
 FSNamesystem ns = cluster.getNamesystem();


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[1/2] hadoop git commit: HDFS-12112. TestBlockManager#testBlockManagerMachinesArray sometimes fails with NPE. Contributed by Wei-Chiu Chuang.

2017-07-15 Thread brahma 
Repository: hadoop
Updated Branches:
  refs/heads/branch-2 5337c2511 -> a33e89931
  refs/heads/branch-2.8 8b7d1df76 -> 6039a2a55


HDFS-12112. TestBlockManager#testBlockManagerMachinesArray sometimes fails with 
NPE. Contributed by Wei-Chiu Chuang.

(cherry picked from commit b778887af59d96f1fac30cae14be1cabbdb74c8b)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a33e8993
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a33e8993
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a33e8993

Branch: refs/heads/branch-2
Commit: a33e89931b543e2bb3c29a6f36d6db16c5f5feed
Parents: 5337c25
Author: Brahma Reddy Battula 
Authored: Sat Jul 15 10:38:31 2017 +0800
Committer: Brahma Reddy Battula 
Committed: Sat Jul 15 10:49:11 2017 +0800

--
 .../hadoop/hdfs/server/blockmanagement/TestBlockManager.java | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/a33e8993/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
index e743a9d..f4fe38c 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
@@ -1149,7 +1149,7 @@ public class TestBlockManager {
 }
   }
 
-  @Test
+  @Test(timeout = 6)
   public void testBlockManagerMachinesArray() throws Exception {
 final Configuration conf = new HdfsConfiguration();
 final MiniDFSCluster cluster =
@@ -1160,6 +1160,8 @@ public class TestBlockManager {
 final Path filePath = new Path("/tmp.txt");
 final long fileLen = 1L;
 DFSTestUtil.createFile(fs, filePath, fileLen, (short) 3, 1L);
+DFSTestUtil.waitForReplication((DistributedFileSystem)fs,
+filePath, (short) 3, 6);
 ArrayList datanodes = cluster.getDataNodes();
 assertEquals(datanodes.size(), 4);
 FSNamesystem ns = cluster.getNamesystem();


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HDFS-12112. TestBlockManager#testBlockManagerMachinesArray sometimes fails with NPE. Contributed by Wei-Chiu Chuang.

2017-07-15 Thread brahma 
Repository: hadoop
Updated Branches:
  refs/heads/trunk 06ece4832 -> b778887af


HDFS-12112. TestBlockManager#testBlockManagerMachinesArray sometimes fails with 
NPE. Contributed by Wei-Chiu Chuang.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b778887a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b778887a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b778887a

Branch: refs/heads/trunk
Commit: b778887af59d96f1fac30cae14be1cabbdb74c8b
Parents: 06ece48
Author: Brahma Reddy Battula 
Authored: Sat Jul 15 10:38:31 2017 +0800
Committer: Brahma Reddy Battula 
Committed: Sat Jul 15 10:38:31 2017 +0800

--
 .../hadoop/hdfs/server/blockmanagement/TestBlockManager.java | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/b778887a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
index 3088b7b..6b1a979 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
@@ -1219,7 +1219,7 @@ public class TestBlockManager {
 }
   }
 
-  @Test
+  @Test(timeout = 6)
   public void testBlockManagerMachinesArray() throws Exception {
 final Configuration conf = new HdfsConfiguration();
 final MiniDFSCluster cluster =
@@ -1230,6 +1230,8 @@ public class TestBlockManager {
 final Path filePath = new Path("/tmp.txt");
 final long fileLen = 1L;
 DFSTestUtil.createFile(fs, filePath, fileLen, (short) 3, 1L);
+DFSTestUtil.waitForReplication((DistributedFileSystem)fs,
+filePath, (short) 3, 6);
 ArrayList datanodes = cluster.getDataNodes();
 assertEquals(datanodes.size(), 4);
 FSNamesystem ns = cluster.getNamesystem();


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[2/2] hadoop git commit: HDFS-12069. Ozone: Create a general abstraction for metadata store. Contributed by Weiwei Yang.

2017-07-15 Thread wwei 
HDFS-12069. Ozone: Create a general abstraction for metadata store. Contributed 
by Weiwei Yang.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8f122a75
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8f122a75
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8f122a75

Branch: refs/heads/HDFS-7240
Commit: 8f122a75054163a2bb0bb8310c2b75016fdf1c3e
Parents: 90f1d58
Author: Weiwei Yang 
Authored: Sun Jul 16 09:34:02 2017 +0800
Committer: Weiwei Yang 
Committed: Sun Jul 16 09:34:02 2017 +0800

--
 .../apache/hadoop/ozone/OzoneConfigKeys.java|   9 +
 .../common/helpers/ContainerUtils.java  |  16 +-
 .../container/common/helpers/KeyUtils.java  |  20 +-
 .../common/impl/ContainerManagerImpl.java   |   6 +-
 .../container/common/impl/KeyManagerImpl.java   |  57 ++--
 .../container/common/interfaces/KeyManager.java |  13 +-
 .../container/common/utils/ContainerCache.java  |  14 +-
 .../hadoop/ozone/ksm/MetadataManager.java   |  27 +-
 .../hadoop/ozone/ksm/MetadataManagerImpl.java   | 146 -
 .../hadoop/ozone/ksm/VolumeManagerImpl.java |  50 ++--
 .../ozone/scm/block/BlockManagerImpl.java   |  67 +++--
 .../org/apache/hadoop/ozone/scm/cli/SQLCLI.java | 112 +++
 .../ozone/scm/container/ContainerMapping.java   |  19 +-
 .../ContainerReplicationManager.java|  14 +-
 .../ozone/scm/exceptions/SCMException.java  |   3 +-
 .../hadoop/ozone/scm/node/NodePoolManager.java  |   5 +-
 .../hadoop/ozone/scm/node/SCMNodeManager.java   |  14 +-
 .../ozone/scm/node/SCMNodePoolManager.java  |  53 ++--
 .../web/localstorage/OzoneMetadataManager.java  | 133 -
 .../org/apache/hadoop/utils/BatchOperation.java |  90 ++
 .../org/apache/hadoop/utils/EntryConsumer.java  |  38 +++
 .../apache/hadoop/utils/LevelDBKeyFilters.java  |  65 
 .../org/apache/hadoop/utils/LevelDBStore.java   | 182 +++-
 .../apache/hadoop/utils/MetadataKeyFilters.java |  65 
 .../org/apache/hadoop/utils/MetadataStore.java  | 152 ++
 .../hadoop/utils/MetadataStoreBuilder.java  |  96 ++
 .../src/main/resources/ozone-default.xml|  11 +
 .../apache/hadoop/ozone/TestLevelDBStore.java   | 165 ---
 .../apache/hadoop/ozone/TestMetadataStore.java  | 296 +++
 .../common/impl/TestContainerPersistence.java   |  12 +-
 .../TestContainerReplicationManager.java|   5 +-
 .../hadoop/ozone/ksm/TestKeySpaceManager.java   |   2 +
 32 files changed, 1247 insertions(+), 710 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/8f122a75/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
index cebb1b0..1822a2a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
@@ -61,6 +61,15 @@ public final class OzoneConfigKeys {
   public static final String OZONE_CONTAINER_METADATA_DIRS =
   "ozone.container.metadata.dirs";
 
+  public static final String OZONE_METADATA_STORE_IMPL =
+  "ozone.metastore.impl";
+  public static final String OZONE_METADATA_STORE_IMPL_LEVELDB =
+  "LevelDB";
+  public static final String OZONE_METADATA_STORE_IMPL_ROCKSDB =
+  "RocksDB";
+  public static final String OZONE_METADATA_STORE_IMPL_DEFAULT =
+  OZONE_METADATA_STORE_IMPL_LEVELDB;
+
   public static final String OZONE_KEY_CACHE = "ozone.key.cache.size";
   public static final int OZONE_KEY_CACHE_DEFAULT = 1024;
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8f122a75/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java
index 7d0e756..0a360e2 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java
@@ -27,8 +27,9 @@ import org.apache.hadoop.hdfs.protocol.DatanodeID;
 import org.apache.hadoop.hdfs.protocol.proto.HdfsProtos;
 import

[1/2] hadoop git commit: HDFS-12069. Ozone: Create a general abstraction for metadata store. Contributed by Weiwei Yang.

2017-07-15 Thread wwei 
Repository: hadoop
Updated Branches:
  refs/heads/HDFS-7240 90f1d5854 -> 8f122a750


http://git-wip-us.apache.org/repos/asf/hadoop/blob/8f122a75/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBKeyFilters.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBKeyFilters.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBKeyFilters.java
deleted file mode 100644
index f813d77..000
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBKeyFilters.java
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- *  with the License.  You may obtain a copy of the License at
- *
- *  http://www.apache.org/licenses/LICENSE-2.0
- *
- *  Unless required by applicable law or agreed to in writing, software
- *  distributed under the License is distributed on an "AS IS" BASIS,
- *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- *  See the License for the specific language governing permissions and
- *  limitations under the License.
- */
-package org.apache.hadoop.utils;
-
-import com.google.common.base.Strings;
-import org.apache.hadoop.hdfs.DFSUtil;
-
-/**
- * An utility class to filter levelDB keys.
- */
-public class LevelDBKeyFilters {
-
-  /**
-   * Interface for levelDB key filters.
-   */
-  public interface LevelDBKeyFilter {
-/**
- * Filter levelDB key with a certain condition.
- *
- * @param preKey previous key.
- * @param currentKey current key.
- * @param nextKeynext key.
- * @return true if a certain condition satisfied, return false otherwise.
- */
-boolean filterKey(byte[] preKey, byte[] currentKey, byte[] nextKey);
-  }
-
-  /**
-   * Utility class to filter key by a string prefix. This filter
-   * assumes keys can be parsed to a string.
-   */
-  public static class KeyPrefixFilter implements LevelDBKeyFilter {
-
-private String keyPrefix = null;
-
-public KeyPrefixFilter(String keyPrefix) {
-  this.keyPrefix = keyPrefix;
-}
-
-@Override public boolean filterKey(byte[] preKey, byte[] currentKey,
-byte[] nextKey) {
-  if (Strings.isNullOrEmpty(keyPrefix)) {
-return true;
-  } else {
-return currentKey != null &&
-DFSUtil.bytes2String(currentKey).startsWith(keyPrefix);
-  }
-}
-  }
-}

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8f122a75/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java
index e6f6fa6..415b788 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java
@@ -18,7 +18,8 @@
 
 package org.apache.hadoop.utils;
 
-import org.apache.hadoop.utils.LevelDBKeyFilters.LevelDBKeyFilter;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.hadoop.utils.MetadataKeyFilters.MetadataKeyFilter;
 import org.fusesource.leveldbjni.JniDBFactory;
 import org.iq80.leveldb.WriteBatch;
 import org.iq80.leveldb.DB;
@@ -30,7 +31,6 @@ import org.iq80.leveldb.ReadOptions;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.io.Closeable;
 import java.io.File;
 import java.io.IOException;
 import java.util.List;
@@ -41,7 +41,7 @@ import java.util.Map.Entry;
 /**
  * LevelDB interface.
  */
-public class LevelDBStore implements Closeable {
+public class LevelDBStore implements MetadataStore {
 
   private static final Logger LOG =
   LoggerFactory.getLogger(LevelDBStore.class);
@@ -51,23 +51,13 @@ public class LevelDBStore implements Closeable {
   private final Options dbOptions;
   private final WriteOptions writeOptions;
 
-  /**
-   * Opens a DB file.
-   *
-   * @param dbPath  - DB File path
-   * @param createIfMissing - Create if missing
-   * @throws IOException
-   */
-  public LevelDBStore(File dbPath, boolean createIfMissing) throws
-  IOException {
+  public LevelDBStore(File dbPath, boolean createIfMissing)
+  throws IOException {
 dbOptions = new Options();
 dbOptions.createIfMissing(createIfMissing);
-db = JniDBFactory.factory.open(dbPath, dbOptions);
-if (db == null) {
-  throw new IOException("Db

[hadoop] Git Push Summary

2017-07-15 Thread asuresh 
Repository: hadoop
Updated Branches:
  refs/heads/YARN-6828 [created] 06ece4832

-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14662. Update azure-storage sdk to version 5.4.0. Contributed by Thomas Marquardt.

2017-07-15 Thread stevel 
Repository: hadoop
Updated Branches:
  refs/heads/trunk 0a6d5c0cf -> 06ece4832


HADOOP-14662. Update azure-storage sdk to version 5.4.0.
Contributed by Thomas Marquardt.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/06ece483
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/06ece483
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/06ece483

Branch: refs/heads/trunk
Commit: 06ece483222b82404ee198159c6866db89043459
Parents: 0a6d5c0
Author: Steve Loughran 
Authored: Sat Jul 15 16:27:17 2017 +0100
Committer: Steve Loughran 
Committed: Sat Jul 15 16:27:17 2017 +0100

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/06ece483/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 3969474..b9819b4 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1118,7 +1118,7 @@
   
 com.microsoft.azure
 azure-storage
-5.3.0
+5.4.0
  
 
   


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org