Hi,
I am enabling HA for hdfs and yarn on my hdp2.6 cluster. HDFS can start but
yarn cannot due to error:
2018-07-18 18:11:23,967 FATAL
applicationhistoryservice.ApplicationHistoryServer
(ApplicationHistoryServer.java:launchAppHistoryServer(177)) - Error
starting ApplicationHistoryServer
java.lang.IllegalArgumentException: java.net.UnknownHostException:
test-cluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(
SecurityUtil.java:439)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(
NameNodeProxies.java:321)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(
NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:690)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:631)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(
DistributedFileSystem.java:160)
at org.apache.hadoop.fs.FileSystem.createFileSystem(
FileSystem.java:2795)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(
FileSystem.java:2829)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:179)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:374)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.hadoop.yarn.server.timeline.
EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:209)
at org.apache.hadoop.service.AbstractService.init(
AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(
CompositeService.java:107)
at org.apache.hadoop.yarn.server.applicationhistoryservice.
ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
at org.apache.hadoop.service.AbstractService.init(
AbstractService.java:163)
at org.apache.hadoop.yarn.server.applicationhistoryservice.
ApplicationHistoryServer.launchAppHistoryServer(
ApplicationHistoryServer.java:174)
at org.apache.hadoop.yarn.server.applicationhistoryservice.
ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
My current fs.defaultFS is "hdfs://test-cluster" - the HA hdfs uri. Looks
like yarn cannot recognize this uri.
I found an example at https://cwiki.apache.org/confluence/display/AMBARI/
Blueprint+Support+for+HA+Clusters which uses
fs.defaultFS=hdfs://%HOSTGROUP::master_1%:8020
which is not the HA hdfs uri. Plus, this example uses different blueprints
for HA hdfs and HA yarn while I use one blueprint for both.
My questions:
1. should I separate the blueprints for HA Hdfs and HA yarn as above
example? Or one blueprint for both that I am currently using is fine?
2. can yarn use the HA hdfs uri instead of pointing to a specific namenode?
The latter seems to break HA.
Appreciate your hints.
