Thanks Jeff. But the link above has the HA URI fs.defaultFS <https://cwiki.apache.org/confluence/display/AMBARI/fs.defaultFS>=hdfs://mycluster in the HA hdfs blueprint. Any idea why it does not have port? Thanks.
On Wed, Jul 18, 2018 at 2:07 PM, Jeff Hubbs <jhubbsl...@att.net> wrote: > Lian - > > Your value of fs.defaultFS is supposed to have a port number, e.g. > hdfs://test-cluster:9000. > > On 7/18/18 4:28 PM, Lian Jiang wrote: > > Hi, > > I am enabling HA for hdfs and yarn on my hdp2.6 cluster. HDFS can start > but yarn cannot due to error: > > 2018-07-18 18:11:23,967 FATAL > applicationhistoryservice.ApplicationHistoryServer > (ApplicationHistoryServer.java:launchAppHistoryServer(177)) - Error > starting ApplicationHistoryServer > java.lang.IllegalArgumentException: java.net.UnknownHostException: > test-cluster > at org.apache.hadoop.security.SecurityUtil.buildTokenService(Se > curityUtil.java:439) > at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(Name > NodeProxies.java:321) > at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeP > roxies.java:176) > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:690) > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:631) > at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Dist > ributedFileSystem.java:160) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem. > java:2795) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem > .java:2829) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:179) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:374) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) > at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimeline > Store.serviceInit(EntityGroupFSTimelineStore.java:209) > at org.apache.hadoop.service.AbstractService.init(AbstractServi > ce.java:163) > at org.apache.hadoop.service.CompositeService.serviceInit(Compo > siteService.java:107) > at org.apache.hadoop.yarn.server.applicationhistoryservice.Appl > icationHistoryServer.serviceInit(ApplicationHistoryServer.java:111) > at org.apache.hadoop.service.AbstractService.init(AbstractServi > ce.java:163) > at org.apache.hadoop.yarn.server.applicationhistoryservice.Appl > icationHistoryServer.launchAppHistoryServer(ApplicationHisto > ryServer.java:174) > at org.apache.hadoop.yarn.server.applicationhistoryservice.Appl > icationHistoryServer.main(ApplicationHistoryServer.java:184) > > > My current fs.defaultFS is "hdfs://test-cluster" - the HA hdfs uri. Looks > like yarn cannot recognize this uri. > > I found an example at https://cwiki.apache.org/confl > uence/display/AMBARI/Blueprint+Support+for+HA+Clusters which uses > fs.defaultFS=hdfs://%HOSTGROUP::master_1%:8020 which is not the HA hdfs > uri. Plus, this example uses different blueprints for HA hdfs and HA yarn > while I use one blueprint for both. > > My questions: > 1. should I separate the blueprints for HA Hdfs and HA yarn as above > example? Or one blueprint for both that I am currently using is fine? > > 2. can yarn use the HA hdfs uri instead of pointing to a specific > namenode? The latter seems to break HA. > > Appreciate your hints. > > >
