[
https://issues.apache.org/jira/browse/KNOX-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Larry McCay reopened KNOX-475:
------------------------------
Assignee: Larry McCay
Reopening in response to [[email protected]]'s observations.
[[email protected]] - where did you come up with the &buffersize query
parameter idea? I'm just not familiar with it.
I will investigate the CappedBufferHttpEntity code to see where the problem is.
Thank you for the observations as this seems to need to be reopened.
> "Hit replay buffer max limit" when uploading greater than 4KB file to HDFS
> --------------------------------------------------------------------------
>
> Key: KNOX-475
> URL: https://issues.apache.org/jira/browse/KNOX-475
> Project: Apache Knox
> Issue Type: Bug
> Components: Server
> Affects Versions: 0.4.0
> Environment: CentOS 6.5
> HttpFS 2.4.0, Kerberized
> Oracle JDK 1.7.0_45 64bit
> Reporter: jaehoon ko
> Assignee: Larry McCay
>
> My cluster has a KNOX gateway that forwards WebHDFS REST API to HttpFS.
> HttpFS and HDFS are Kerberized.
> If I try uploading a file bigger than 4K, I get 500 error.
> {code}
> $ curl -u cersei:cersei -ik -X PUT
> "https://node-02-04:8443/gateway/gc/webhdfs/v1/user/cersei/bigfile?op=create";
> HTTP/1.1 307 Temporary Redirect
> Set-Cookie:
> JSESSIONID=wlsp2k09p1mj15i0n1tfzdai0;Path=/gateway/gc;Secure;HttpOnly
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Server: Apache-Coyote/1.1
> Location:
> https://node-02-04:8443/gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile?_=AAAACAAAABAAAABQ0N5SioA7O6sdYwm-BuvUtz3Ov6BvONMosqdDZf865RRUEsOmLDbvjh6vwMCr9kRebZ-_98ksmWnbOQkEMDbbZzYD_vrSCLTbGS9fkz1OLSqd4dOUaoIjZDsXj9QDgdbwOa6BPsjSUmg
> Date: Tue, 02 Dec 2014 11:01:26 GMT
> Content-Type: application/json
> Content-Length: 0
> curl -ik -H "Content-Type:application/octet-stream" -u cersei:cersei -T
> bigfile -X PUT
> "https://node-02-04:8443/gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile?_=AAAACAAAABAAAABQ0N5SioA7O6sdYwm-BuvUtz3Ov6BvONMosqdDZf865RRUEsOmLDbvjh6vwMCr9kRebZ-_98ksmWnbOQkEMDbbZzYD_vrSCLTbGS9fkz1OLSqd4dOUaoIjZDsXj9QDgdbwOa6BPsjSUmg";
> HTTP/1.1 100 Continue
> HTTP/1.1 500 Server Error
> Set-Cookie:
> JSESSIONID=1t8dm6cn9rlw31mzureff6nqum;Path=/gateway/gc;Secure;HttpOnly
> Content-Length: 0
> Server: Jetty(8.1.14.v20131031)
> {code}
> Knox log is:
> {panel}
> 2014-12-02 20:01:26,411 INFO realm.AuthorizingRealm
> (AuthorizingRealm.java:getAuthorizationCacheLazy(248)) - No cache or
> cacheManager properties have been set. Authorization cache cannot be
> obtained.
> 2014-12-02 20:02:08,094 INFO realm.AuthorizingRealm
> (AuthorizingRealm.java:getAuthorizationCacheLazy(248)) - No cache or
> cacheManager properties have been set. Authorization cache cannot be
> obtained.
> 2014-12-02 20:02:08,395 INFO client.DefaultHttpClient
> (DefaultRequestDirector.java:tryExecute(726)) - I/O exception
> (java.io.IOException) caught when processing request: Hit replay buffer max
> limit
> 2014-12-02 20:02:08,395 INFO client.DefaultHttpClient
> (DefaultRequestDirector.java:tryExecute(733)) - Retrying request
> 2014-12-02 20:02:08,405 INFO client.DefaultHttpClient
> (DefaultRequestDirector.java:tryExecute(726)) - I/O exception
> (java.io.IOException) caught when processing request: Hit replay buffer max
> limit
> 2014-12-02 20:02:08,407 INFO client.DefaultHttpClient
> (DefaultRequestDirector.java:tryExecute(733)) - Retrying request
> 2014-12-02 20:02:08,415 INFO client.DefaultHttpClient
> (DefaultRequestDirector.java:tryExecute(726)) - I/O exception
> (java.io.IOException) caught when processing request: Hit replay buffer max
> limit
> 2014-12-02 20:02:08,416 INFO client.DefaultHttpClient
> (DefaultRequestDirector.java:tryExecute(733)) - Retrying request
> 2014-12-02 20:02:08,426 WARN hadoop.gateway
> (HttpClientDispatch.java:executeRequest(124)) - Connection exception
> dispatching request:
> http://node-02-04.gc.net:14000/webhdfs/v1/user/cersei/bigfile?op=CREATE&data=true&doAs=cersei
> java.io.IOException: Hit replay buffer max limit
> java.io.IOException: Hit replay buffer max limit
> at
> org.apache.hadoop.gateway.dispatch.CappedBufferHttpEntity$ReplayStream.read(CappedBufferHttpEntity.java:143)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
> at
> org.apache.hadoop.gateway.dispatch.CappedBufferHttpEntity.writeTo(CappedBufferHttpEntity.java:93)
> at
> org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
> at
> org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
> at
> org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
> at
> org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
> at
> org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
> at
> org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at
> org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeKerberosDispatch(HttpClientDispatch.java:193)
> at
> org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:120)
> at
> org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273)
> at
> org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148)
> at
> org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209)
> at
> org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157)
> at
> org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77)
> at
> org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
> at
> org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
> at
> org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
> at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
> at
> org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
> at
> org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128)
> at
> org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:370)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:701)
> 2014-12-02 20:02:08,433 ERROR hadoop.gateway
> (AbstractGatewayFilter.java:doFilter(62)) - Failed to execute filter:
> java.io.IOException: Service connectivity error.
> 2014-12-02 20:02:08,433 ERROR hadoop.gateway
> (AbstractGatewayFilter.java:doFilter(62)) - Failed to execute filter:
> java.io.IOException: Service connectivity error.
> 2014-12-02 20:02:08,434 ERROR hadoop.gateway
> (AbstractGatewayFilter.java:doFilter(65)) - Failed to execute filter:
> javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException:
> java.security.PrivilegedActionException: java.io.IOException: Service
> connectivity error.
> 2014-12-02 20:02:08,434 ERROR hadoop.gateway
> (GatewayFilter.java:doFilter(134)) - Failed to execute filter:
> javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException:
> java.security.PrivilegedActionException: java.io.IOException: Service
> connectivity error.
> 2014-12-02 20:02:08,435 ERROR hadoop.gateway
> (GatewayServlet.java:service(122)) - Failed to execute filter:
> javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException:
> java.security.PrivilegedActionException: java.io.IOException: Service
> connectivity error.
> 2014-12-02 20:02:08,435 WARN servlet.ServletHandler
> (ServletHandler.java:doHandle(536)) -
> javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException:
> java.security.PrivilegedActionException: java.io.IOException: Service
> connectivity error.
> at
> org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
> at
> org.apache.shiro.web.filter.authc.AuthenticatingFilter.cleanup(AuthenticatingFilter.java:155)
> at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
> at
> org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
> at
> org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128)
> at
> org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:370)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:701)
> Caused by: org.apache.shiro.subject.ExecutionException:
> java.security.PrivilegedActionException: java.io.IOException: Service
> connectivity error.
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:385)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
> at
> org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
> at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
> ... 42 more
> Caused by: java.security.PrivilegedActionException: java.io.IOException:
> Service connectivity error.
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77)
> at
> org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
> at
> org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
> ... 48 more
> Caused by: java.io.IOException: Service connectivity error.
> at
> org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:126)
> at
> org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273)
> at
> org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148)
> at
> org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209)
> at
> org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157)
> at
> org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90)
> ... 55 more
> 2014-12-02 20:02:08,451 WARN servlet.ServletHandler
> (ServletHandler.java:doHandle(561)) -
> /gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile
> org.apache.shiro.subject.ExecutionException:
> java.security.PrivilegedActionException: java.io.IOException: Service
> connectivity error.
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:385)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
> at
> org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
> at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
> at
> org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
> at
> org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128)
> at
> org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:370)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:701)
> Caused by: java.security.PrivilegedActionException: java.io.IOException:
> Service connectivity error.
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77)
> at
> org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
> at
> org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
> at
> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
> ... 48 more
> Caused by: java.io.IOException: Service connectivity error.
> at
> org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:126)
> at
> org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273)
> at
> org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148)
> at
> org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
> at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209)
> at
> org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157)
> at
> org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55)
> at
> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
> at
> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93)
> at
> org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90)
> ... 55 more
> {panel}
> I examined the code and discovered it is related to capped replay buffer.
> When Hadoop cluster is secured and delegation token does not exist, KNOX
> tries to buffer the current request for later replay in case Hadoop cluster
> challenges with WWW-Authenticate
> ([KNOX-174|https://issues.apache.org/jira/browse/KNOX-174]) The default
> buffer size is 4K. When I commented out that part, I could upload 10GB file
> without problem.
> I'm not sure whether this is a bug. If it's not a bug, how can I upload big
> file?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
