Thanks for the response, Jerry. I created a patch:
http://pastebin.com/xisGVHt8 All REST tests passed. I know Ben logged a JIRA on this subject already. Not sure if that should be re-opened or, a new JIRA should be created. Once we have an open JIRA, I will attach my patch there. Cheers On Thu, Dec 3, 2015 at 8:16 PM, Jerry He <[email protected]> wrote: > From HBase 0.98, there have been changes going into the Rest gateway, > mainly more scan support. > > There seems to be a change in the way the url table/rowkey* is executed > on the Rest gateway. > in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte of > 255 on Rest gateway in the Scan sent to the servers. > In the newer versions, table/rowkey* seems to treated with a prefixFilter. > This may explain why you are seeing slower response. > > Jerry > > > > On Thu, Dec 3, 2015 at 1:12 PM, Sutton, Ben <[email protected]> > wrote: > > > We are currently experiencing an issue with HBase 1.1.1 through the REST > > API whereby we are getting a very slow (> 2minutes) /non-existent > response > > when hitting the Rest API with a globbing option i.e. http:// > > <HBase_Rest>:<HBase_Rest_Port/table/key*<http:// > > %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*> > > > > > > > > We are able to return each individual row for the key with no problem > i.e. > > http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http:// > %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01> > > or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http:// > %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02> > > etc. > > > > We have also found if we load in a third of the data to hbase we have no > > trouble with the above. > > > > Facing this issue with 1.7billion row hbase table - we are able to > > retrieve data for initial rows keys through to 1/10 of the region with a > > slowly degrading performance after this point until no result. > > > > This is with 19 regionservers with each having at least 5 regions. > > > > Seeing the below error: > > Failed after attempts=36, exceptions: > > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: > > callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at > > region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36., > > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093 > > (through reference chain: > > org.apache.hadoop.hbase.rest.CellSetModelStream["Row"]) > > 2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213] > mortbay.log: > > /$TABLE/KEY.* > > java.lang.RuntimeException: > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > > attempts=36, exceptions: > > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: > > callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at > > > region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36., > > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093 > > > > at > > > org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97) > > at > > > org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82) > > at > > > com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285) > > at > > > com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137) > > at > > > com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159) > > at > > > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344) > > at > > > com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597) > > at > > > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328) > > at > > > com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498) > > at > > com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320) > > at > > > com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249) > > at > > > javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95) > > at > > > com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179) > > at > > > com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157) > > at > > > com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) > > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437) > > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > > at > > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > at > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > > at > > > org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76) > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > > at > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > > at > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > at > > > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > > at > > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) > > at > > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > at org.mortbay.jetty.Server.handle(Server.java:326) > > at > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > at > > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > > at > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > > at > org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > at > > > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > > at > > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: > > Failed after attempts=36, exceptions: > > > > Anyone have any ideas? > > > > Thanks, > > > > Ben > > > > > > -------------------------------------------------------------------- > > Save Paper - Do you really need to print this e-mail? > > > > Visit www.virginmedia.com for more information, and more fun. > > > > This email and any attachments are or may be confidential and legally > > privileged > > and are sent solely for the attention of the addressee(s). If you have > > received this > > email in error, please delete it from your system: its use, disclosure or > > copying is > > unauthorised. Statements and opinions expressed in this email may not > > represent > > those of Virgin Media. Any representations or commitments in this email > are > > subject to contract. > > > > Registered office: Media House, Bartley Wood Business Park, Hook, > > Hampshire, RG27 9UP > > Registered in England and Wales with number 2591237 > > >
