[ 
https://issues.apache.org/jira/browse/KYLIN-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046227#comment-15046227
 ] 

liyang commented on KYLIN-1205:
-------------------------------

[http-bio-7070-exec-10]:[2015-12-07 
02:27:29,111][INFO][org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:203)]
 - Got exception making request scanner_id: 418418710023042957 number_of_rows: 
1024 close_scanner: false next_call_seq: 0 to 
region=KYLIN_PBEVCU523G,\x00\x00\x00\x00\x00\x00\x11\xFF\x0B:To\x1E\x06\xBF\x95}\x00\xE7\x0D\xE0\xD6\x01\x02,1447961804989.ab03605758e28c9591146509be5d75b5.,
 hostname=phxaishdc9dn0210.phx.ebay.com,60020,1445848741773, seqNum=25
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: 
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 
418418710023042957 number_of_rows: 1024 close_scanner: false next_call_seq: 0
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3109)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:28861)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2017)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
        at 
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
        at 
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
        at 
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
        at java.lang.Thread.run(Thread.java:745)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at 
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
        at 
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
        at 
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:300)
        at 
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:198)
        at 
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
        at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)
        at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
        at 
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:336)
        at 
org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:91)
        at 
org.apache.kylin.storage.hbase.CubeSegmentTupleIterator.hasNext(CubeSegmentTupleIterator.java:156)
        at 
org.apache.kylin.storage.hbase.SerializedHBaseTupleIterator.next(SerializedHBaseTupleIterator.java:112)
        at 
org.apache.kylin.query.enumerator.CubeEnumerator.moveNext(CubeEnumerator.java:81)
        at Baz$1$1.moveNext(Unknown Source)
        at 
org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:737)
        at 
org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:677)
        at 
org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:301)
        at Baz.bind(Unknown Source)
        at 
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:281)
        at 
org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:235)
        at 
org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:533)
        at 
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:184)
        at 
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:63)
        at 
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:42)
        at 
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
        at 
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:566)
        at 
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
        at 
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
        at 
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
        at 
org.apache.kylin.rest.service.QueryService.execute(QueryService.java:354)
        at 
org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:268)
        at 
org.apache.kylin.rest.service.QueryService.query(QueryService.java:114)
        at 
org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at 
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
        at 
org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$7c676529.query(<generated>)
        at 
org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:177)
        at 
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:84)
        at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213)
        at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
        at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
        at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
        at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
        at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
        at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
        at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
        at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
        at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
        at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at 
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
        at 
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
        at 
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
        at 
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at 
org.apache.kylin.rest.filter.KylinApiFilter.doFilterInternal(KylinApiFilter.java:66)
        at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at 
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:205)
        at 
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:266)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
        at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
        at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
        at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at 
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
 org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 
418418710023042957 number_of_rows: 1024 close_scanner: false next_call_seq: 0
        at 
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3109)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:28861)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2017)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
        at 
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
        at 
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
        at 
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
        at java.lang.Thread.run(Thread.java:745)

        at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1508)
        at 
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1694)
        at 
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1747)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:29264)
        at 
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:168)
        ... 106 more


> hbase RpcClient java.io.IOException: Unexpected closed connection
> -----------------------------------------------------------------
>
>                 Key: KYLIN-1205
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1205
>             Project: Kylin
>          Issue Type: Bug
>            Reporter: liyang
>            Assignee: liyang
>
> during a scan of many rows(each row may be more than 64k), we encountered 
> "RpcClient java.io.IOException: Unexpected closed connection" a lot, this is 
> breaking kylin's normal functions.
> if you open my log attached and grep "http-bio-7070-exec-11", you will find 
> this thread is:
> 1. launching several scans, this is because the kylin cube has several 
> segments, each segment correspond to a htable. A complete cube visit requires 
> to visit every htable related, so there're many scans established
> 2. One of the scanner(scanner_id: 538064847515747456 ) failed due 
> to:java.io.IOException: Unexpected closed connection
> 3. the error in step 2 lead to hbase client side/server side out of sync, 
> casuing org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException
> 4. hbase client retried with a different scanner id (still same htable, same 
> region) but still failed
> can you please check at server side on what on earth is causing the " 
> Unexpected closed connection" it seems to be the root cause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to