Wellington Chevreuil created HBASE-15860:
--------------------------------------------
Summary: Improvements for HBASE-14280 - Fixing Bulkload for HDFS
HA Clusters
Key: HBASE-15860
URL: https://issues.apache.org/jira/browse/HBASE-15860
Project: HBase
Issue Type: Improvement
Components: util
Affects Versions: 1.0.0
Reporter: Wellington Chevreuil
Priority: Minor
HBASE-14280 introduced fix for bulkload failures when referring a remote
cluster name service id if "bulkloading" from a HA cluster.
HBASE-14280 solution on *FSHDFSUtils.getNNAddresses* was to invoke
*DFSUtil.getNNServiceRpcAddressesForCluster* instead of
*DFSUtil.getNNServiceRpcAddresses*. This works for hadoop 2.6 and above.
Proposed change here is to use "*DFSUtil.getRpcAddressesForNameserviceId*"
instead, which already returns only addresses for specific nameservice
informed. This is available since hadoop 2.4.
Sample proposal on FSHDFSUtils.getNNAddresses:
...
{noformat}
String nameServiceId = serviceName.split(":")[1];
if (dfsUtilClazz == null) {
dfsUtilClazz = Class.forName("org.apache.hadoop.hdfs.DFSUtil");
}
if (getNNAddressesMethod == null) {
getNNAddressesMethod =
dfsUtilClazz.getMethod("getRpcAddressesForNameserviceId",
Configuration.class,
String.class, String.class);
}
Map<String, InetSocketAddress> nnMap =
(Map<String, InetSocketAddress>) getNNAddressesMethod
.invoke(null, conf, nameServiceId, null);
for (Map.Entry<String, InetSocketAddress> e2 : nnMap.entrySet()) {
InetSocketAddress addr = e2.getValue();
addresses.add(addr);
}
...
{noformat}
Will also add test conditions for *FSHDFSUtils.isSameHdfs* to verify scenario
when multiple name service ids are defined.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
