[
https://issues.apache.org/jira/browse/KYLIN-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792555#comment-16792555
]
ASF GitHub Bot commented on KYLIN-3874:
---------------------------------------
guangxuCheng commented on pull request #516: KYLIN-3874 "Convert Cuboid Data to
HFile" failed when HBase and MR use different HDFS clusters
URL: https://github.com/apache/kylin/pull/516
Detail: https://issues.apache.org/jira/browse/KYLIN-3874
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS
> clusters
> -----------------------------------------------------------------------------------
>
> Key: KYLIN-3874
> URL: https://issues.apache.org/jira/browse/KYLIN-3874
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Reporter: Guangxu Cheng
> Assignee: Guangxu Cheng
> Priority: Major
>
> {panel:title=exception}
> 2019-03-14 16:33:45,113 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
> Couldn't create proxy provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> ...
> Caused by: java.lang.RuntimeException: Could not find any configured
> addresses for URI
> hdfs://{color:#ff0000}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:116)
> ... 31 more
> {panel}
> {color:#ff0000}hbase-ns-test{color} is the nameservice of HDFS which is used
> by HBase.
> In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
> {code:java|title=CubeHFileJob.java|borderStyle=solid}
> // use current hbase configuration
> Configuration configuration = new
> Configuration(HBaseConnection.getCurrentHBaseConfiguration());
> merge(configuration, getConf());
> {code}
> "getConf()" contains only the HDFS configuration of MR, and "configuration"
> contains the HDFS configuration both HBase and MR. See:
> [HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
> So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration
> to be covered. Mainly parameter "dfs.nameservices"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
