Did you resolve the issue? On Thu, Aug 30, 2018 at 7:10 PM Guang Yang <k...@uber.com> wrote:
> Ok, the issue seems goes away after I upgrade jetty to 9.3.10 on Knox > 1.1.0. But there was some issue on Knox 0.13.0 + Jetty 9.3.10. > > On Tue, Aug 28, 2018 at 1:23 PM, Guang Yang <k...@uber.com> wrote: > >> Hey Larry, >> >> We're using 0.13.0 and running on Linux version 4.4.92 (Debian 4.9.2-10), >> the JDK version is 8. >> >> On Tue, Aug 28, 2018 at 1:01 PM, larry mccay <lmc...@apache.org> wrote: >> >>> Hi Guang - >>> >>> I do recall this FD issue from looong ago. >>> Not sure what was done to address it but I haven't seen it in a few >>> years. >>> >>> What version of Knox are you using? >>> What OS and JDK versions are you using? >>> >>> We generally upgrade jetty based on identification of CVEs on current >>> version but also try not to jump too many versions as it is often painful >>> to make larger moves due to compatibility issues. >>> >>> thanks, >>> >>> --larry >>> >>> On Tue, Aug 28, 2018 at 3:03 PM, Guang Yang <k...@uber.com> wrote: >>> >>>> Hi guys, >>>> >>>> We're using Knox to proxy WEBHDFS request. The problem is when the >>>> client shut down the connection, Knox will throw EofException. According to >>>> our dashboard, tons of file descriptors are leaked. When the open files >>>> reach to the limit, Knox can't receive any request because there is not >>>> socket available. >>>> >>>> I dig a little bit and found there is a bug in lower version of Jetty >>>> https://github.com/eclipse/jetty.project/issues/634. According that >>>> link, the issue has been fixed at Jetty 9.3.10, so I upgrade Jetty in Knox. >>>> But after sometime, Knox throw some other issue >>>> >>>> 2018-08-28 18:45:57,214 INFO hadoop.gateway >>>> (DefaultHaDispatch.java:executeRequest(93)) - Could not connect to server: >>>> http://datanode/webhdfs/v1/sompath org.eclipse.jetty.io.EofException >>>> org.eclipse.jetty.io.EofException >>>> at org.eclipse.jetty.io >>>> .ChannelEndPoint.flush(ChannelEndPoint.java:197) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:839) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shutdownOutput(SslConnection.java:928) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(SslConnection.java:967) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:893) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shutdownOutput(SslConnection.java:928) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(SslConnection.java:967) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:893) >>>> >>>> And then Knox stops working again. My question is did anybody see the >>>> FD leak issue before? and have we ever tried to update Jetty to a higher >>>> version? >>>> >>>> Thanks, >>>> Guang >>>> >>> >>> >> >
