Can you pastebin RS log hosting meta around Nov 03 13:15?
On Tue, Nov 4, 2014 at 3:15 AM, Jignesh Patel <[email protected]> wrote: > Hi Team, > We are using hbase 0.98.1 and hoop-2.2.0 in our system. We notice following > issue very frequently in our site. Once this issue starts it keeps coming > until we don't restart our application server. How do we fix this issue. > > ) Encountered problems when prefetch hbase:meta table: : > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=10, exceptions: > > Mon Nov 03 13:14:56 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:14:57 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:14:57 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:14:58 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:15:00 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:15:04 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:15:14 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:15:24 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:15:34 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > Mon Nov 03 13:15:44 EST 2014, > org.apache.hadoop.hbase.client.RpcRetryingCaller@361f53dd, > java.io.IOException: com.google.protobuf.ServiceException: > java.lang.NullPointerException > > > at > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:129) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:710) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1158) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1222) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1110) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1067) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:908) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > > org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:72) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:113) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:773) > [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] > > at > > org.datanucleus.store.hbase.HBaseUtils.getResultForObject(HBaseUtils.java:645) > [datanucleus-hbase-3.1.1.jar:] > > at > > org.datanucleus.store.hbase.HBasePersistenceHandler.fetchObject(HBasePersistenceHandler.java:687) > [datanucleus-hbase-3.1.1.jar:] > > at > > org.datanucleus.state.JDOStateManager.loadFieldsFromDatastore(JDOStateManager.java:1634) > > at > org.datanucleus.state.JDOStateManager.validate(JDOStateManager.java:3507) > > at > org.datanucleus.ObjectManagerImpl.findObject(ObjectManagerImpl.java:3379) > [datanucleus-core-3.1.2.jar:] > > at > > org.datanucleus.api.jdo.JDOPersistenceManager.getObjectById(JDOPersistenceManager.java:1722) > [datanucleus-api-jdo-3.1.2.jar:] > > at > > org.datanucleus.api.jdo.JDOPersistenceManager.getObjectById(JDOPersistenceManager.java:1740) > [datanucleus-api-jdo-3.1.2.jar:] > > at >
