[
https://issues.apache.org/jira/browse/KYLIN-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3874:
--------------------------------
Fix Version/s: v2.6.2
> "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS
> clusters
> -----------------------------------------------------------------------------------
>
> Key: KYLIN-3874
> URL: https://issues.apache.org/jira/browse/KYLIN-3874
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Reporter: Guangxu Cheng
> Assignee: Guangxu Cheng
> Priority: Major
> Fix For: v2.6.2
>
>
> {panel:title=exception}
> 2019-03-14 16:33:45,113 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
> Couldn't create proxy provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> ...
> Caused by: java.lang.RuntimeException: Could not find any configured
> addresses for URI
> hdfs://{color:#ff0000}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:116)
> ... 31 more
> {panel}
> {color:#ff0000}hbase-ns-test{color} is the nameservice of HDFS which is used
> by HBase.
> In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
> {code:java|title=CubeHFileJob.java|borderStyle=solid}
> // use current hbase configuration
> Configuration configuration = new
> Configuration(HBaseConnection.getCurrentHBaseConfiguration());
> merge(configuration, getConf());
> {code}
> "getConf()" contains only the HDFS configuration of MR, and "configuration"
> contains the HDFS configuration both HBase and MR. See:
> [HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
> So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration
> to be covered. Mainly parameter "dfs.nameservices"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
