Re: Getting intermittent errors while insertind data into HBase
Hi,
Thanks for the replies. Sorry for late response from my side as I was on
vacation.
My HBase version: HBase 0.98.6-cdh5.3.1
We are interacting with HBase from our rest webservices which we developed
using Spring Jersey. Hence Spring and DAO.
Below I have pasted the code where we are trying to create the connection
objects with some overridden parameters. Also placed the service code flow
where we try to create sensor readings. If you go through the code you
might be seeing primary and seconday cluster connection objects and that is
because we are using HBase replication and hence there are primary and
secondary connections in the code and secondary is used when primary fails.
But in our case primary cluster is healthy and accessible.
We are overriding few default configurations as below while creating
connection. Will this has any side affect?
HBASE.CLIENT.PAUSE=50
HBASE.CLIENT.RETRIES.NUMBER=1
ZOOKEEPER.RECOVERY.RETRY=3
ZOOKEEPER.RECOVERY.RETRY.INTERVALMILL=1000
*and the connection object is created like below. We are intializing the
connections while tomcat startup.*
static {
rowKeyDelimiter =
PropUtils.getProperty(ADConstants.ROWKEY_DELIMITER);
try {
SensorDataDAO.primaryClusterConnection = getConnection(true);
} catch (IOException e) {
ExceptionUtils.logException(, , , e);
}
try {
backupClusterConnection = getConnection(false);
} catch (IOException e) {
ExceptionUtils.logException(, , , e);
}
}
private static HConnection getConnection(boolean connectToPrimary) throws
IOException {
Configuration conf = HBaseConfiguration.create();
if (connectToPrimary) {
conf.addResource(PropUtils.getProperty(ADConstants.PRIMARY_CLUSTER_HBASE_SITE_FILE_NAME));
conf.addResource(PropUtils.getProperty(ADConstants.PRIMARY_CLUSTER_CORE_SITE_FILE_NAME));
conf.addResource(PropUtils.getProperty(ADConstants.PRIMARY_CLUSTER_HDFS_SITE_FILE_NAME));
conf.set(ADConstants.HBASE_CLIENT_PAUSE.toLowerCase(),
PropUtils.getProperty(ADConstants.HBASE_CLIENT_PAUSE));
conf.set(ADConstants.HBASE_CLIENT_RETRIES_NUMBER.toLowerCase(),
PropUtils.getProperty(ADConstants.HBASE_CLIENT_RETRIES_NUMBER));
conf.set(ADConstants.ZOOKEEPER_RECOVERY_RETRY.toLowerCase(),
PropUtils.getProperty(ADConstants.ZOOKEEPER_RECOVERY_RETRY));
conf.set(ADConstants.ZOOKEEPER_RECOVERY_RETRY_INTERVALMILL.toLowerCase(),
PropUtils.getProperty(ADConstants.ZOOKEEPER_RECOVERY_RETRY_INTERVALMILL));
} else {
conf.addResource(PropUtils.getProperty(ADConstants.BACKUP_CLUSTER_HBASE_SITE_FILE_NAME));
conf.addResource(PropUtils.getProperty(ADConstants.BACKUP_CLUSTER_CORE_SITE_FILE_NAME));
conf.addResource(PropUtils.getProperty(ADConstants.BACKUP_CLUSTER_HDFS_SITE_FILE_NAME));
conf.set(ADConstants.HBASE_CLIENT_PAUSE.toLowerCase(),
PropUtils.getProperty(ADConstants.HBASE_CLIENT_PAUSE));
conf.set(ADConstants.HBASE_CLIENT_RETRIES_NUMBER.toLowerCase(),
PropUtils.getProperty(ADConstants.HBASE_CLIENT_RETRIES_NUMBER));
conf.set(ADConstants.ZOOKEEPER_RECOVERY_RETRY.toLowerCase(),
PropUtils.getProperty(ADConstants.ZOOKEEPER_RECOVERY_RETRY));
conf.set(ADConstants.ZOOKEEPER_RECOVERY_RETRY_INTERVALMILL.toLowerCase(),
PropUtils.getProperty(ADConstants.ZOOKEEPER_RECOVERY_RETRY_INTERVALMILL));
}
return HConnectionManager.createConnection(conf);
}
*Service Class: Below is our service class where it starts serving the
client request.*
/**
* Service Class to handle all the REST WebService requests for the Reading.
@Path(/reading)
public class ReadingService extends SensorDataService {
public ReadingProcessor getProcessor() {
return (ReadingProcessor) getContext().getBean(readingProcessor);
}
/**
* WebService EndPoint to create the Reading. If reading is created
* successfully , reading Information is returned. Supported request
* formats: XML,JSON, Protobuf
*
* @param format
* @param authorizationHeader
* @param uri
* @param readingCreate
* @param xForwardedProto
* @return
* @throws ADException
*/
@Timer
@POST
@Path(/v1/create.{format: (protobuf|xml|json)})
@Consumes({ MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON })
@Produces({ MediaType.APPLICATION_XML,
MediaType.APPLICATION_JSON,
ADConstants.APPLICATION_PROTOBUF_MEDIA_TYPE })
public Response createReading(@PathParam(format) String format,
@HeaderParam(authorization) String
authorizationHeader,
@Context UriInfo uri,
ReadingCreateRequestV1 readingCreate,
@HeaderParam(X-Forwarded-Proto) String
xForwardedProto,
@HeaderParam(userId)
Re: Getting intermittent errors while insertind data into HBase
Maybe you share HTable instance across several threads? Can you share the code: 1. HTable initialization 2. Htable.put(something) 2015-05-21 11:57 GMT+03:00 Jithender Boreddy jithen1...@gmail.com: Hi, I am inserting data from my java application into two HBase tables back to back. And I am running my application sequentially as part of stress testing. I am getting strange error intermittently. It is passing many times but failing by throwing below error few times. Can someone point me to the correct direction here by letting me know what going wrong ? Pasted below partial stack trace: Stack Trace: java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:953) java.util.LinkedList$ListItr.remove(LinkedList.java:919) org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:319) org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:965) org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1281) org.apache.hadoop.hbase.client.HTable.put(HTable.java:925) com.autodesk.dao.SensorDataDAO.insertRecords(Unknown Source) com.autodesk.dao.SensorDataDAO.insertRecords(Unknown Source) com.autodesk.dao.SensorDataDAO$$FastClassByCGLIB$$36f4c9d9.invoke(generated) net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) com.autodesk.utils.aspects.TimerAspect.log(Unknown Source) sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) com.autodesk.dao.ReadingDAO$$EnhancerByCGLIB$$fa7dd7e1.insertRecords(generated) com.autodesk.business.ReadingProcessor.createReadings(Unknown Source)
Getting intermittent errors while insertind data into HBase
Hi, I am inserting data from my java application into two HBase tables back to back. And I am running my application sequentially as part of stress testing. I am getting strange error intermittently. It is passing many times but failing by throwing below error few times. Can someone point me to the correct direction here by letting me know what going wrong ? Pasted below partial stack trace: Stack Trace: java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:953) java.util.LinkedList$ListItr.remove(LinkedList.java:919) org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:319) org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:965) org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1281) org.apache.hadoop.hbase.client.HTable.put(HTable.java:925) com.autodesk.dao.SensorDataDAO.insertRecords(Unknown Source) com.autodesk.dao.SensorDataDAO.insertRecords(Unknown Source) com.autodesk.dao.SensorDataDAO$$FastClassByCGLIB$$36f4c9d9.invoke(generated) net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) com.autodesk.utils.aspects.TimerAspect.log(Unknown Source) sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) com.autodesk.dao.ReadingDAO$$EnhancerByCGLIB$$fa7dd7e1.insertRecords(generated) com.autodesk.business.ReadingProcessor.createReadings(Unknown Source)
Re: Getting intermittent errors while insertind data into HBase
Why spring? Why a DAO? I’m not suggesting that using Spring or a DAO is wrong, however, you really should justify it. Since it looks like you’re trying to insert sensor data (based on the naming convention), what’s the velocity of the inserts? Are you manually flushing commits or are you waiting until your memstore is full. (Actually commits is the wrong term because you don’t have transactions in HBase, but that’s another issue in terms of HBase naming) You’re going to need to provide a bit more background. On May 21, 2015, at 4:57 AM, Jithender Boreddy jithen1...@gmail.com wrote: Hi, I am inserting data from my java application into two HBase tables back to back. And I am running my application sequentially as part of stress testing. I am getting strange error intermittently. It is passing many times but failing by throwing below error few times. Can someone point me to the correct direction here by letting me know what going wrong ? Pasted below partial stack trace: Stack Trace: java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:953) java.util.LinkedList$ListItr.remove(LinkedList.java:919) org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:319) org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:965) org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1281) org.apache.hadoop.hbase.client.HTable.put(HTable.java:925) com.autodesk.dao.SensorDataDAO.insertRecords(Unknown Source) com.autodesk.dao.SensorDataDAO.insertRecords(Unknown Source) com.autodesk.dao.SensorDataDAO$$FastClassByCGLIB$$36f4c9d9.invoke(generated) net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) com.autodesk.utils.aspects.TimerAspect.log(Unknown Source) sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) com.autodesk.dao.ReadingDAO$$EnhancerByCGLIB$$fa7dd7e1.insertRecords(generated) com.autodesk.business.ReadingProcessor.createReadings(Unknown Source)
Re: Getting intermittent errors while insertind data into HBase
bq. Stack Trace: java.util.LinkedList$ListItr.checkForComodification( LinkedList.java:953) What was the error prior to the above line ? Please let us know the hbase release you're using. Cheers On Thu, May 21, 2015 at 2:08 AM, Serega Sheypak serega.shey...@gmail.com wrote: Maybe you share HTable instance across several threads? Can you share the code: 1. HTable initialization 2. Htable.put(something) 2015-05-21 11:57 GMT+03:00 Jithender Boreddy jithen1...@gmail.com: Hi, I am inserting data from my java application into two HBase tables back to back. And I am running my application sequentially as part of stress testing. I am getting strange error intermittently. It is passing many times but failing by throwing below error few times. Can someone point me to the correct direction here by letting me know what going wrong ? Pasted below partial stack trace: Stack Trace: java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:953) java.util.LinkedList$ListItr.remove(LinkedList.java:919) org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:319) org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:965) org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1281) org.apache.hadoop.hbase.client.HTable.put(HTable.java:925) com.autodesk.dao.SensorDataDAO.insertRecords(Unknown Source) com.autodesk.dao.SensorDataDAO.insertRecords(Unknown Source) com.autodesk.dao.SensorDataDAO$$FastClassByCGLIB$$36f4c9d9.invoke(generated) net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) com.autodesk.utils.aspects.TimerAspect.log(Unknown Source) sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) com.autodesk.dao.ReadingDAO$$EnhancerByCGLIB$$fa7dd7e1.insertRecords(generated) com.autodesk.business.ReadingProcessor.createReadings(Unknown Source)
