[ https://issues.apache.org/jira/browse/YARN-9029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohith Sharma K S resolved YARN-9029. ------------------------------------- Resolution: Invalid This is vendor specific deployment issue. Please contact the vendor! > yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle > ---------------------------------------------------------------------------- > > Key: YARN-9029 > URL: https://issues.apache.org/jira/browse/YARN-9029 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 > Affects Versions: 3.0.1 > Environment: Ambari 2.7.1 > HDP 3.0.1 > Isilon OneFS 8.1.2 > Reporter: Kirankumar Bhusanurmath > Priority: Major > > *Background on ats-hbase start success*:: We had setStoragePolicy exception > on OneFS for yarn-ats hbase region server and below messages respectively on > HDP and OneFS cluster. > *HDP Host* *è* */var/log/hadoop-yarn/embedded-yarn-ats-hbase/* > 2018-11-02 05:07:21,027 WARN [StoreOpener-1588230740-1] util.CommonFSUtils: > Unable to set storagePolicy=HOT for > path=hdfs://kb-hdp-1.west.isilon.com:8020/atsv2/hbase/data/data/hbase/meta/1588230740/info. > DEBUG log level might have more details. > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > *OneFS HDFS log for (unsecured)* > 2018-11-01T10:25:19Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 > user: yarn-ats exception: org.apache.hadoop.ipc.RpcNoSuchMethodException > cause: Unknown RPC: setStoragePolicy > > *OneFS HDFS log for (secured)* > 2018-11-01T12:39:56Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 > user: > yarn-ats-hbase/[krb-hdp301.solarch.emc....@solarch.emc.com|mailto:krb-hdp301.solarch.emc....@solarch.emc.com] > exception: > org.apache.hadoop.ipc.RpcNoSuchMethodException cause: Unknown RPC: > setStoragePolicy > > > As a resolution, we set below 2 properties in custom hbase-site.xml file and > setstoragePolicy exception got resolved. > *Custom yarn-hbase-site.xml* > 'hbase.wal.storage.policy' to ‘NONE’ > ‘hbase.hstore.block.storage.policy’ to ‘NONE’ > > > ********* > *NOW*, we have a new issue that is OneFS HDFS log and > /var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-****.log > gets flooded with below error messages.*OneFS HDFS Log* > ------------------- > 2018-11-06T06:25:43Z <30.6> kbhusan-edl9tdc-1 hdfs[2542]: [hdfs] RPC V9 > complete user: > yarn-ats-hbase/[rduvnode343393.west.isilon....@west.isilon.com|mailto:rduvnode343393.west.isilon....@west.isilon.com] > exception: > java.io.FileNotFoundException cause: File not found: > /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta > <FS Exception at CreateHandleFromPath():448 status: > STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 Path: > atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta> > --------------- > > */var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-****.log* > -------------- > 2018-11-06 04:01:12,863 INFO [LruBlockCacheStatsExecutor] > hfile.LruBlockCache: totalSize=1.12 MB, freeSize=1.25 GB, max=1.25 GB, > blockCount=24, accesses=287, hits=263, hitRatio=91.64%, , > cachingAccesses=287, cachingHits=263, cachingHitsRatio=91.64%, > evictions=419, evicted=0, evictedPerRun=0.0 > > 2018-11-06 04:01:48,763 WARN > [LeaseRenewer:[yarn-...@kb-hdp301-2.west.isilon.com|mailto:yarn-...@kb-hdp301-2.west.isilon.com]:8020] > impl.LeaseRenewer: > Failed to renew lease for [DFSClient_NONMAPREDUCE_-533598397_1] for 30 > seconds. Will retry shortly ... > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > Client auth method 'SIMPLE', mechanism '' is not supported on zone:1 > auth:'kerberos_only' > > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) > > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > > “ > “ > “ > “ > 2018-11-06 12:51:02,843 WARN [Close-WAL-Writer-1] > asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file > /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541488822956/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541488822956.1541492429802 > failed, retry = 637 > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File > not found: > /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541488822956/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541488822956.1541492429802 > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > > -------------- > On checking the ats-hbase app logs, found hbase region server is trying to > flush the old edits to free the WAL and it fails due to authentication issue. > *Why yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn > principle?* -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org