Created HBASE-14928 and attached patch there. FYI
On Thu, Dec 3, 2015 at 9:05 PM, Ted Yu <[email protected]> wrote: > Thanks for the response, Jerry. > > I created a patch: > > http://pastebin.com/xisGVHt8 > > All REST tests passed. > > I know Ben logged a JIRA on this subject already. > Not sure if that should be re-opened or, a new JIRA should be created. > > Once we have an open JIRA, I will attach my patch there. > > Cheers > > On Thu, Dec 3, 2015 at 8:16 PM, Jerry He <[email protected]> wrote: > >> From HBase 0.98, there have been changes going into the Rest gateway, >> mainly more scan support. >> >> There seems to be a change in the way the url table/rowkey* is executed >> on the Rest gateway. >> in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte >> of >> 255 on Rest gateway in the Scan sent to the servers. >> In the newer versions, table/rowkey* seems to treated with a prefixFilter. >> This may explain why you are seeing slower response. >> >> Jerry >> >> >> >> On Thu, Dec 3, 2015 at 1:12 PM, Sutton, Ben <[email protected] >> > >> wrote: >> >> > We are currently experiencing an issue with HBase 1.1.1 through the REST >> > API whereby we are getting a very slow (> 2minutes) /non-existent >> response >> > when hitting the Rest API with a globbing option i.e. http:// >> > <HBase_Rest>:<HBase_Rest_Port/table/key*<http:// >> > %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*> >> > >> > >> > >> > We are able to return each individual row for the key with no problem >> i.e. >> > http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http:// >> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01> >> > or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http:// >> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02> >> > etc. >> > >> > We have also found if we load in a third of the data to hbase we have no >> > trouble with the above. >> > >> > Facing this issue with 1.7billion row hbase table - we are able to >> > retrieve data for initial rows keys through to 1/10 of the region with a >> > slowly degrading performance after this point until no result. >> > >> > This is with 19 regionservers with each having at least 5 regions. >> > >> > Seeing the below error: >> > Failed after attempts=36, exceptions: >> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: >> > callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at >> > region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36., >> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093 >> > (through reference chain: >> > org.apache.hadoop.hbase.rest.CellSetModelStream["Row"]) >> > 2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213] >> mortbay.log: >> > /$TABLE/KEY.* >> > java.lang.RuntimeException: >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after >> > attempts=36, exceptions: >> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: >> > callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at >> > >> region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36., >> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093 >> > >> > at >> > >> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97) >> > at >> > >> org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82) >> > at >> > >> com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285) >> > at >> > >> com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137) >> > at >> > >> com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159) >> > at >> > >> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344) >> > at >> > >> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597) >> > at >> > >> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328) >> > at >> > >> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498) >> > at >> > >> com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320) >> > at >> > >> com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249) >> > at >> > >> javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95) >> > at >> > >> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179) >> > at >> > >> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157) >> > at >> > >> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) >> > at >> > >> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437) >> > at >> > >> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) >> > at >> > >> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) >> > at >> > >> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) >> > at >> > >> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) >> > at >> > >> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> > at >> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> > at >> > >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >> > at >> > >> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76) >> > at >> > >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> > at >> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >> > at >> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >> > at >> > >> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> > at >> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) >> > at >> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> > at org.mortbay.jetty.Server.handle(Server.java:326) >> > at >> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >> > at >> > >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >> > at >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >> > at >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >> > at >> > >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >> > at >> > >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >> > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: >> > Failed after attempts=36, exceptions: >> > >> > Anyone have any ideas? >> > >> > Thanks, >> > >> > Ben >> > >> > >> > -------------------------------------------------------------------- >> > Save Paper - Do you really need to print this e-mail? >> > >> > Visit www.virginmedia.com for more information, and more fun. >> > >> > This email and any attachments are or may be confidential and legally >> > privileged >> > and are sent solely for the attention of the addressee(s). If you have >> > received this >> > email in error, please delete it from your system: its use, disclosure >> or >> > copying is >> > unauthorised. Statements and opinions expressed in this email may not >> > represent >> > those of Virgin Media. Any representations or commitments in this email >> are >> > subject to contract. >> > >> > Registered office: Media House, Bartley Wood Business Park, Hook, >> > Hampshire, RG27 9UP >> > Registered in England and Wales with number 2591237 >> > >> > >
