saintstack commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r436432869
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/AsyncMetaTableAccessor.java
##########
@@ -170,26 +170,27 @@
/**
* Used to get all region locations for the specific table.
- * @param metaTable
* @param tableName table we're looking for, can be null for getting all
regions
* @return the list of region locations. The return value will be wrapped by
a
* {@link CompletableFuture}.
*/
public static CompletableFuture<List<HRegionLocation>>
getTableHRegionLocations(
- AsyncTable<AdvancedScanResultConsumer> metaTable, TableName tableName) {
+ AsyncTable<AdvancedScanResultConsumer> metaTable, TableName tableName,
+ boolean excludeOfflinedSplitParents) {
CompletableFuture<List<HRegionLocation>> future = new
CompletableFuture<>();
- addListener(getTableRegionsAndLocations(metaTable, tableName, true),
(locations, err) -> {
- if (err != null) {
- future.completeExceptionally(err);
- } else if (locations == null || locations.isEmpty()) {
- future.complete(Collections.emptyList());
- } else {
- List<HRegionLocation> regionLocations =
- locations.stream().map(loc -> new HRegionLocation(loc.getFirst(),
loc.getSecond()))
- .collect(Collectors.toList());
- future.complete(regionLocations);
- }
- });
+ addListener(getTableRegionsAndLocations(metaTable, tableName,
excludeOfflinedSplitParents),
Review comment:
Why we need this? The default is not to return offlined locations (as
per the the original method impl). You ask for raw scan if you do want all
including offlined.
s/excludeOfflinedSplitParents/excludeSplitParents/ because split parents are
offlined by definition?
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/AsyncMetaTableAccessor.java
##########
@@ -170,26 +170,27 @@
/**
* Used to get all region locations for the specific table.
- * @param metaTable
* @param tableName table we're looking for, can be null for getting all
regions
* @return the list of region locations. The return value will be wrapped by
a
* {@link CompletableFuture}.
*/
public static CompletableFuture<List<HRegionLocation>>
getTableHRegionLocations(
- AsyncTable<AdvancedScanResultConsumer> metaTable, TableName tableName) {
+ AsyncTable<AdvancedScanResultConsumer> metaTable, TableName tableName,
+ boolean excludeOfflinedSplitParents) {
CompletableFuture<List<HRegionLocation>> future = new
CompletableFuture<>();
- addListener(getTableRegionsAndLocations(metaTable, tableName, true),
(locations, err) -> {
- if (err != null) {
- future.completeExceptionally(err);
- } else if (locations == null || locations.isEmpty()) {
- future.complete(Collections.emptyList());
- } else {
- List<HRegionLocation> regionLocations =
- locations.stream().map(loc -> new HRegionLocation(loc.getFirst(),
loc.getSecond()))
- .collect(Collectors.toList());
- future.complete(regionLocations);
- }
- });
+ addListener(getTableRegionsAndLocations(metaTable, tableName,
excludeOfflinedSplitParents),
Review comment:
This is async version of
```
// What happens here when 1M regions in hbase:meta? This won't scale?
public static List<Pair<RegionInfo, ServerName>>
getTableRegionsAndLocations(
```
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractAsyncTableRegionLocator.java
##########
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.HConstants.EMPTY_END_ROW;
+import static org.apache.hadoop.hbase.client.AsyncRegionLocatorHelper.isGood;
+import static
org.apache.hadoop.hbase.client.ConnectionUtils.createClosestRowAfter;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.commons.lang3.ObjectUtils;
+import org.apache.hadoop.hbase.HBaseIOException;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The base class for locating region of a table.
+ */
[email protected]
+abstract class AbstractAsyncTableRegionLocator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(AbstractAsyncTableRegionLocator.class);
+
+ protected final AsyncConnectionImpl conn;
+
+ protected final TableName tableName;
+
+ protected final int maxConcurrent;
+
+ protected final TableRegionLocationCache cache;
+
+ protected static final class LocateRequest {
Review comment:
Has to be an abstract? If so, why?
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
##########
@@ -2185,4 +2206,27 @@ private static Put addSequenceNum(Put p, long
openSeqNum, int replicaId) throws
.setValue(Bytes.toBytes(openSeqNum))
.build());
}
+
+ private static byte[] buildLocateRegionStartRow(TableName tableName, byte[]
row,
+ RegionLocateType locateType) {
+ if (locateType.equals(RegionLocateType.BEFORE)) {
+ if (Bytes.equals(row, HConstants.EMPTY_END_ROW)) {
+ byte[] binaryTableName = tableName.getName();
+ return Arrays.copyOf(binaryTableName, binaryTableName.length + 1);
+ } else {
+ return createRegionName(tableName, row, ZEROES, false);
+ }
+ } else {
+ return createRegionName(tableName, row, NINES, false);
+ }
+ }
+
+ public static Scan createLocateRegionScan(TableName tableName, byte[] row,
+ RegionLocateType locateType, int prefetchLimit) {
Review comment:
'abstraction'? IIRC, this class is the access point for hbase:meta. It
is all about Scans that take Visitors, Puts, and Deletes against hbase:meta;
i.e. not 'abstractions'.
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
##########
@@ -2185,4 +2206,27 @@ private static Put addSequenceNum(Put p, long
openSeqNum, int replicaId) throws
.setValue(Bytes.toBytes(openSeqNum))
.build());
}
+
+ private static byte[] buildLocateRegionStartRow(TableName tableName, byte[]
row,
+ RegionLocateType locateType) {
+ if (locateType.equals(RegionLocateType.BEFORE)) {
+ if (Bytes.equals(row, HConstants.EMPTY_END_ROW)) {
+ byte[] binaryTableName = tableName.getName();
+ return Arrays.copyOf(binaryTableName, binaryTableName.length + 1);
+ } else {
+ return createRegionName(tableName, row, ZEROES, false);
+ }
+ } else {
+ return createRegionName(tableName, row, NINES, false);
+ }
+ }
+
+ public static Scan createLocateRegionScan(TableName tableName, byte[] row,
+ RegionLocateType locateType, int prefetchLimit) {
Review comment:
hmmm... Went back and saw the locateInMeta. This replicates the head of
that method? It should use this?
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##########
@@ -1105,23 +1097,7 @@ public void run(PRESP resp) {
* List all region locations for the specific table.
*/
private CompletableFuture<List<HRegionLocation>>
getTableHRegionLocations(TableName tableName) {
- if (TableName.META_TABLE_NAME.equals(tableName)) {
- CompletableFuture<List<HRegionLocation>> future = new
CompletableFuture<>();
- addListener(connection.registry.getMetaRegionLocations(), (metaRegions,
err) -> {
- if (err != null) {
- future.completeExceptionally(err);
- } else if (metaRegions == null || metaRegions.isEmpty() ||
- metaRegions.getDefaultRegionLocation() == null) {
- future.completeExceptionally(new IOException("meta region does not
found"));
- } else {
-
future.complete(Collections.singletonList(metaRegions.getDefaultRegionLocation()));
- }
- });
- return future;
- } else {
- // For non-meta table, we fetch all locations by scanning hbase:meta
table
- return AsyncMetaTableAccessor.getTableHRegionLocations(metaTable,
tableName);
- }
+ return connection.getRegionLocator(tableName).getAllRegionLocations();
Review comment:
Having trouble following what is going on. Excuse me.
This looks better. Gets RegionLocator from Connection.
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
##########
@@ -186,24 +186,6 @@ public long getNonceGroup() {
return Arrays.copyOf(row, row.length + 1);
}
- /**
- * Create a row before the specified row and very close to the specified row.
- */
- static byte[] createCloseRowBefore(byte[] row) {
- if (row.length == 0) {
- return MAX_BYTE_ARRAY;
- }
- if (row[row.length - 1] == 0) {
- return Arrays.copyOf(row, row.length - 1);
- } else {
- byte[] nextRow = new byte[row.length + MAX_BYTE_ARRAY.length];
- System.arraycopy(row, 0, nextRow, 0, row.length - 1);
- nextRow[row.length - 1] = (byte) ((row[row.length - 1] & 0xFF) - 1);
- System.arraycopy(MAX_BYTE_ARRAY, 0, nextRow, row.length,
MAX_BYTE_ARRAY.length);
- return nextRow;
- }
- }
-
Review comment:
Not used any more?
##########
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
##########
@@ -1763,4 +1765,44 @@ default void
preHasUserPermissions(ObserverContext<MasterCoprocessorEnvironment>
default void
postHasUserPermissions(ObserverContext<MasterCoprocessorEnvironment> ctx,
String userName, List<Permission> permissions) throws IOException {
}
+
+ /**
+ * Called before locating meta region.
+ * @param ctx ctx the coprocessor instance's environment
+ * @param row the row key to locate
+ * @param locateType the direction of the locate operation
+ */
+ default void
preLocateMetaRegion(ObserverContext<MasterCoprocessorEnvironment> ctx, byte[]
row,
+ RegionLocateType locateType) throws IOException {
+ }
+
+ /**
+ * Called after locating meta region.
+ * @param ctx ctx the coprocessor instance's environment
+ * @param row the row key to locate
+ * @param locateType the direction of the locate operation
+ * @param locs the locations of the given meta region, including meta
replicas if any.
+ */
+ default void
postLocateMetaRegion(ObserverContext<MasterCoprocessorEnvironment> ctx, byte[]
row,
+ RegionLocateType locateType, List<HRegionLocation> locs) throws
IOException {
+ }
+
+ /**
+ * Called before getting all locations for meta regions.
+ * @param ctx ctx the coprocessor instance's environment
+ * @param excludeOfflinedSplitParents don't return split parents
+ */
+ default void
preGetAllMetaRegionLocations(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ boolean excludeOfflinedSplitParents) {
+ }
+
+ /**
+ * Called after getting all locations for meta regions.
+ * @param ctx ctx the coprocessor instance's environment
+ * @param excludeOfflinedSplitParents don't return split parents
+ * @param locs the locations of all meta regions, including meta replicas if
any.
+ */
+ default void
postGetAllMetaRegionLocations(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ boolean excludeOfflinedSplitParents, List<HRegionLocation> locs) {
+ }
Review comment:
This is hard-coding Master as host of hbase:meta locations?
We have MasterRegistry and the ZKRegistry and but Master is host for all
meta locations? This skirts Registry?
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##########
@@ -1105,23 +1097,7 @@ public void run(PRESP resp) {
* List all region locations for the specific table.
*/
private CompletableFuture<List<HRegionLocation>>
getTableHRegionLocations(TableName tableName) {
- if (TableName.META_TABLE_NAME.equals(tableName)) {
- CompletableFuture<List<HRegionLocation>> future = new
CompletableFuture<>();
- addListener(connection.registry.getMetaRegionLocations(), (metaRegions,
err) -> {
- if (err != null) {
- future.completeExceptionally(err);
- } else if (metaRegions == null || metaRegions.isEmpty() ||
- metaRegions.getDefaultRegionLocation() == null) {
- future.completeExceptionally(new IOException("meta region does not
found"));
- } else {
-
future.complete(Collections.singletonList(metaRegions.getDefaultRegionLocation()));
- }
- });
- return future;
- } else {
- // For non-meta table, we fetch all locations by scanning hbase:meta
table
- return AsyncMetaTableAccessor.getTableHRegionLocations(metaTable,
tableName);
- }
+ return connection.getRegionLocator(tableName).getAllRegionLocations();
Review comment:
RegionLocator does the right thing based off table name?
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractAsyncTableRegionLocator.java
##########
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.HConstants.EMPTY_END_ROW;
+import static org.apache.hadoop.hbase.client.AsyncRegionLocatorHelper.isGood;
+import static
org.apache.hadoop.hbase.client.ConnectionUtils.createClosestRowAfter;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.commons.lang3.ObjectUtils;
+import org.apache.hadoop.hbase.HBaseIOException;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The base class for locating region of a table.
+ */
[email protected]
+abstract class AbstractAsyncTableRegionLocator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(AbstractAsyncTableRegionLocator.class);
+
+ protected final AsyncConnectionImpl conn;
+
+ protected final TableName tableName;
+
+ protected final int maxConcurrent;
+
+ protected final TableRegionLocationCache cache;
+
+ protected static final class LocateRequest {
Review comment:
I see, abstract to share code between user-space and meta-space table
locators.
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaTableRegionLocator.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.hadoop.hbase.AsyncMetaTableAccessor;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNotFoundException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The class for locating region for table other than meta.
+ */
[email protected]
+class AsyncNonMetaTableRegionLocator extends AbstractAsyncTableRegionLocator {
Review comment:
Ok. I'm reacting to our Interface => Abstract => Impl tiering.... We do
it too often.
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionLocator.java
##########
@@ -44,25 +51,51 @@
@InterfaceAudience.Private
class AsyncRegionLocator {
- private static final Logger LOG =
LoggerFactory.getLogger(AsyncRegionLocator.class);
+ @VisibleForTesting
+ static final String MAX_CONCURRENT_LOCATE_REQUEST_PER_TABLE =
+ "hbase.client.meta.max.concurrent.locate.per.table";
+
+ private static final int DEFAULT_MAX_CONCURRENT_LOCATE_REQUEST_PER_TABLE = 8;
+
+ @VisibleForTesting
+ static final String MAX_CONCURRENT_LOCATE_META_REQUEST =
+ "hbase.client.meta.max.concurrent.locate";
+
+ @VisibleForTesting
+ static String LOCATE_PREFETCH_LIMIT = "hbase.client.locate.prefetch.limit";
+
+ private static final int DEFAULT_LOCATE_PREFETCH_LIMIT = 10;
private final HashedWheelTimer retryTimer;
private final AsyncConnectionImpl conn;
- private final AsyncMetaRegionLocator metaRegionLocator;
+ private final int maxConcurrentLocateRequestPerTable;
+
+ private final int maxConcurrentLocateMetaRequest;
+
+ private final int locatePrefetchLimit;
- private final AsyncNonMetaRegionLocator nonMetaRegionLocator;
+ private final boolean useMetaReplicas;
- AsyncRegionLocator(AsyncConnectionImpl conn, HashedWheelTimer retryTimer) {
+ private final ConcurrentMap<TableName, AbstractAsyncTableRegionLocator>
table2Locator =
Review comment:
Whats this? One locator for meta and another for user-space Tables?
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncMetaTableRegionLocator.java
##########
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.util.FutureUtils.addListener;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.MasterNotRunningException;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.exceptions.ClientExceptionsUtil;
+import org.apache.hadoop.hbase.ipc.HBaseRpcController;
+import org.apache.hadoop.hbase.ipc.ServerNotRunningYetException;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.protobuf.ByteString;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.ClientMetaService;
Review comment:
Still this new Service? Not ConnectionRegistry?
##########
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##########
@@ -3921,4 +3996,85 @@ public MetaRegionLocationCache
getMetaRegionLocationCache() {
public RSGroupInfoManager getRSGroupInfoManager() {
return rsGroupInfoManager;
}
+
+ public RegionLocations locateMeta(byte[] row, RegionLocateType locateType)
throws IOException {
+ if (locateType == RegionLocateType.AFTER) {
+ // as we know the exact row after us, so we can just create the new row,
and use the same
+ // algorithm to locate it.
+ row = Arrays.copyOf(row, row.length + 1);
+ locateType = RegionLocateType.CURRENT;
+ }
Review comment:
Don't understand.
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
##########
@@ -3537,4 +3538,30 @@ public static RSGroupInfo
toGroupInfo(RSGroupProtos.RSGroupInfo proto) {
return
RSGroupProtos.RSGroupInfo.newBuilder().setName(pojo.getName()).addAllServers(hostports)
.addAllTables(tables).build();
}
+
+ public static MasterProtos.RegionLocateType
toProtoRegionLocateType(RegionLocateType pojo) {
+ switch (pojo) {
+ case BEFORE:
+ return MasterProtos.RegionLocateType.REGION_LOCATE_TYPE_BEFORE;
+ case CURRENT:
+ return MasterProtos.RegionLocateType.REGION_LOCATE_TYPE_CURRENT;
+ case AFTER:
+ return MasterProtos.RegionLocateType.REGION_LOCATE_TYPE_AFTER;
+ default:
+ throw new IllegalArgumentException("Unknown RegionLocateType: " +
pojo);
+ }
+ }
+
+ public static RegionLocateType
toRegionLocateType(MasterProtos.RegionLocateType proto) {
+ switch (proto) {
+ case REGION_LOCATE_TYPE_BEFORE:
+ return RegionLocateType.BEFORE;
+ case REGION_LOCATE_TYPE_CURRENT:
+ return RegionLocateType.CURRENT;
+ case REGION_LOCATE_TYPE_AFTER:
+ return RegionLocateType.AFTER;
+ default:
+ throw new IllegalArgumentException("Unknown proto RegionLocateType: "
+ proto);
+ }
+ }
Review comment:
Why is this wanted in proto now if we had it since async came in?
##########
File path: hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
##########
@@ -1284,4 +1307,16 @@ service ClientMetaService {
* Get current meta replicas' region locations.
*/
rpc GetMetaRegionLocations(GetMetaRegionLocationsRequest)
returns(GetMetaRegionLocationsResponse);
+
+ /**
+ * Get meta region locations for a given row
+ */
+ rpc LocateMetaRegion(LocateMetaRegionRequest)
+ returns(LocateMetaRegionResponse);
+
+ /**
+ * Get all meta regions locations
+ */
+ rpc GetAllMetaRegionLocations(GetAllMetaRegionLocationsRequest)
+ returns(GetAllMetaRegionLocationsResponse);
Review comment:
ClientMetaService backed ConnectionRegistry?
So, we get one Region or all of them... nothing in-between?
No Scan support here because we want 'dumbed-down API' in ConnectionRegistry
or in exposed ClientMetaService? We want to be able to implement hbase:meta
Region locations in cache that we can serve from ZK for your
ZKConnectionRegistry impl?
##########
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionRegistry.java
##########
@@ -31,11 +30,6 @@
@InterfaceAudience.Private
interface ConnectionRegistry extends Closeable {
- /**
- * Get the location of meta region(s).
- */
- CompletableFuture<RegionLocations> getMetaRegionLocations();
Review comment:
Just removed. We trying to add api in here
https://docs.google.com/document/d/1OYrCfpmmLPkSa5-AepQw8hv09zNcNPaFzSXezhcMS7E/edit?ts=5ed4e463#heading=h.kxnc4cup53tk
##########
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##########
@@ -868,8 +879,71 @@ protected void initializeZKBasedSystemTrackers()
// Will be overriden in test to inject customized AssignmentManager
@VisibleForTesting
- protected AssignmentManager createAssignmentManager(MasterServices master) {
- return new AssignmentManager(master);
+ protected AssignmentManager createAssignmentManager(MasterServices master,
+ MasterRegion masterRegion) {
+ return new AssignmentManager(master, masterRegion);
+ }
+
+ /**
+ * Load the meta region state from the meta region server ZNode.
+ * @param zkw reference to the {@link ZKWatcher} which also contains
configuration and operation
+ * @param replicaId the ID of the replica
+ * @return regionstate
+ * @throws KeeperException if a ZooKeeper operation fails
+ */
+ private static RegionState getMetaRegionState(ZKWatcher zkw, int replicaId)
Review comment:
Region State is in zk?
##########
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##########
@@ -868,8 +879,71 @@ protected void initializeZKBasedSystemTrackers()
// Will be overriden in test to inject customized AssignmentManager
@VisibleForTesting
- protected AssignmentManager createAssignmentManager(MasterServices master) {
- return new AssignmentManager(master);
+ protected AssignmentManager createAssignmentManager(MasterServices master,
+ MasterRegion masterRegion) {
+ return new AssignmentManager(master, masterRegion);
+ }
+
+ /**
+ * Load the meta region state from the meta region server ZNode.
+ * @param zkw reference to the {@link ZKWatcher} which also contains
configuration and operation
+ * @param replicaId the ID of the replica
+ * @return regionstate
+ * @throws KeeperException if a ZooKeeper operation fails
+ */
+ private static RegionState getMetaRegionState(ZKWatcher zkw, int replicaId)
Review comment:
Perhaps this is old state? If so, maybe say so in doc?
The @return regionstate can be removed.
##########
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##########
@@ -3921,4 +3996,85 @@ public MetaRegionLocationCache
getMetaRegionLocationCache() {
public RSGroupInfoManager getRSGroupInfoManager() {
return rsGroupInfoManager;
}
+
+ public RegionLocations locateMeta(byte[] row, RegionLocateType locateType)
throws IOException {
+ if (locateType == RegionLocateType.AFTER) {
+ // as we know the exact row after us, so we can just create the new row,
and use the same
+ // algorithm to locate it.
+ row = Arrays.copyOf(row, row.length + 1);
+ locateType = RegionLocateType.CURRENT;
+ }
+ Scan scan =
+ MetaTableAccessor.createLocateRegionScan(TableName.META_TABLE_NAME, row,
locateType, 1);
+ try (RegionScanner scanner = masterRegion.getScanner(scan)) {
+ boolean moreRows;
+ List<Cell> cells = new ArrayList<>();
+ do {
+ moreRows = scanner.next(cells);
+ if (cells.isEmpty()) {
+ continue;
+ }
+ Result result = Result.create(cells);
+ cells.clear();
+ RegionLocations locs = MetaTableAccessor.getRegionLocations(result);
+ if (locs == null || locs.getDefaultRegionLocation() == null) {
+ LOG.warn("No location found when locating meta region with row='{}',
locateType={}",
+ Bytes.toStringBinary(row), locateType);
+ return null;
+ }
+ HRegionLocation loc = locs.getDefaultRegionLocation();
+ RegionInfo info = loc.getRegion();
+ if (info == null) {
+ LOG.warn("HRegionInfo is null when locating meta region with
row='{}', locateType={}",
+ Bytes.toStringBinary(row), locateType);
+ return null;
+ }
+ if (info.isSplitParent()) {
+ continue;
+ }
+ return locs;
+ } while (moreRows);
+ LOG.warn("No location available when locating meta region with row='{}',
locateType={}",
+ Bytes.toStringBinary(row), locateType);
+ return null;
+ }
+ }
+
+ public List<RegionLocations> getAllMetaRegionLocations(boolean
excludeOfflinedSplitParents)
+ throws IOException {
+ Scan scan = new Scan().addFamily(HConstants.CATALOG_FAMILY);
+ List<RegionLocations> list = new ArrayList<>();
+ try (RegionScanner scanner = masterRegion.getScanner(scan)) {
+ boolean moreRows;
+ List<Cell> cells = new ArrayList<>();
+ do {
+ moreRows = scanner.next(cells);
+ if (cells.isEmpty()) {
+ continue;
+ }
+ Result result = Result.create(cells);
+ cells.clear();
+ RegionLocations locs = MetaTableAccessor.getRegionLocations(result);
+ if (locs == null) {
+ LOG.warn("No locations in {}", result);
+ continue;
+ }
+ HRegionLocation loc = locs.getRegionLocation();
+ if (loc == null) {
+ LOG.warn("No non null location in {}", result);
+ continue;
+ }
+ RegionInfo info = loc.getRegion();
+ if (info == null) {
+ LOG.warn("No serialized RegionInfo in {}", result);
+ continue;
+ }
+ if (excludeOfflinedSplitParents && info.isSplitParent()) {
+ continue;
+ }
+ list.add(locs);
+ } while (moreRows);
+ }
+ return list;
+ }
Review comment:
This stuff is inline in HMaster class? Not out in Meta Table Accessor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
