Hi Kevin, In my humble opinion, this has to do with cpu processing encryption in general based on which cipher being used. Couldn’t the same type of principals/improvements (hdfs encryption improvements) be done here for let’s say for AES cipher suites? If the main bottleneck here is CPU couldn’t you enhance encryption though hardware acceleration and you may see better performance numbers?
https://calomel.org/aesni_ssl_performance.html Try forcing a less secure cipher to be used in your environment. Do you then see better numbers? dav From: Kevin Risden <kris...@apache.org> Reply-To: "user@knox.apache.org" <user@knox.apache.org> Date: Tuesday, October 9, 2018 at 1:05 PM To: "user@knox.apache.org" <user@knox.apache.org> Subject: Re: WebHDFS performance issue in Knox @David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked. @Guang - NP just took some time to sit down and look at it. Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty. * https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions * https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements * https://webtide.com/conscrypting-native-ssl-for-jetty/ Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting (https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further. Kevin Risden On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <k...@uber.com<mailto:k...@uber.com>> wrote: Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL. On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <dvillarr...@hortonworks.com<mailto:dvillarr...@hortonworks.com>> wrote: I bring this up because HDFS encryption saw an increase in performance. https://issues.apache.org/jira/browse/HDFS-6606 https://issues.apache.org/jira/browse/HADOOP-10768 Maybe Knox can make some enhancements in this area? From: David Villarreal <dvillarr...@hortonworks.com<mailto:dvillarr...@hortonworks.com>> Date: Tuesday, October 9, 2018 at 10:34 AM To: "user@knox.apache.org<mailto:user@knox.apache.org>" <user@knox.apache.org<mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox Hi Kevin, Now increase your CPU processing power and show me the numbers. Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration? libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.) From: Kevin Risden <kris...@apache.org<mailto:kris...@apache.org>> Reply-To: "user@knox.apache.org<mailto:user@knox.apache.org>" <user@knox.apache.org<mailto:user@knox.apache.org>> Date: Tuesday, October 9, 2018 at 10:26 AM To: "user@knox.apache.org<mailto:user@knox.apache.org>" <user@knox.apache.org<mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox Writes look to have performance impact as well: * directly to webhdfs - ~2.6 seconds * knox no ssl - ~29 seconds * knox ssl - ~49.6 seconds Kevin Risden On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kris...@apache.org<mailto:kris...@apache.org>> wrote: If I run two downloads concurrently: 1,073,741,824 46.1MB/s in 22s 1,073,741,824 51.3MB/s in 22s So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow. Kevin Risden On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kris...@apache.org<mailto:kris...@apache.org>> wrote: So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop. Rough timings with wget for a 1GB random file: * directly to webhdfs - 1,073,741,824 252MB/s in 3.8s * knox no ssl - 1,073,741,824 264MB/s in 3.6s * knox ssl - 1,073,741,824 54.3MB/s in 20s There is a significant decrease with Knox SSL for some reason. Kevin Risden On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lmc...@apache.org<mailto:lmc...@apache.org>> wrote: SSL handshake will likely happen at least twice. Once for the request through Knox to the NN then the redirect from the NN to the DN goes all the way back to the client. So they have to follow the redirect and do the handshake to the DN. On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kris...@apache.org<mailto:kris...@apache.org>> wrote: So I found this in the Knox issues list in JIRA: https://issues.apache.org/jira/browse/KNOX-1221 It sounds familiar in terms of a slowdown when going through Knox. Kevin Risden On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kris...@apache.org<mailto:kris...@apache.org>> wrote: Hmmm yea curl for a single file should do the handshake once. What are the system performance statistics during the SSL vs non SSL testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here if using that. Otherwise watching top may be helpful. It would be help to determine if the Knox is working harder during the SSL transfer. Kevin Risden On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <k...@uber.com<mailto:k...@uber.com>> wrote: I'm just using curl to download a single large file. So I suspect SSL handshake just happens once? On Tue, Sep 11, 2018 at 12:02 PM Kevin Risden <kris...@apache.org<mailto:kris...@apache.org>> wrote: What client are you using to connect Knox? Is this for a single file or a bunch of files? The SSL handshake can be slow if the client doesn't keep the connection open. Kevin Risden On Tue, Sep 11, 2018, 14:51 Guang Yang <k...@uber.com<mailto:k...@uber.com>> wrote: Thanks Larry. But the only difference is this part in my gateway-site.xml. <property> <name>ssl.enabled</name> <value>false</value> <description>Indicates whether SSL is enabled.</description> </property> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lmc...@apache.org<mailto:lmc...@apache.org>> wrote: I really don't think that kind of difference should be expected from merely SSL overhead. I don't however have any metrics to contradict it either since I do not run Knox without SSL. Given the above, I am struggling coming up with a meaningful response to this. :( I don't think you should see a 10 fold increase in speed by disabling SSL though. On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <k...@uber.com<mailto:k...@uber.com>> wrote: Any idea guys? On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <k...@uber.com<mailto:k...@uber.com>> wrote: Thanks guys! The issue seems exactly what David pointed out, which is because of encrypted over SSL. Without Knox, the download speed can reach to 400M/s if I call Namenode directly. And with disabling SSL, the speed can reach to ~400M/s as well through Knox. But with SSL, the speed drops significantly to ~40M/s. I know it's because of encrypted, but it does surprised me with such a difference. Is it normal from your perspective? Thanks, Guang On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <dvillarr...@hortonworks.com<mailto:dvillarr...@hortonworks.com>> wrote: Hi Guang, Keep in mind the data is being encrypted over SSL. If you disable SSL you will most likely see a very significant boost in throughput. Some people have used more powerful computers to make encryption quicker. Thanks, David From: Sean Roberts <srobe...@hortonworks.com<mailto:srobe...@hortonworks.com>> Reply-To: "user@knox.apache.org<mailto:user@knox.apache.org>" <user@knox.apache.org<mailto:user@knox.apache.org>> Date: Tuesday, September 4, 2018 at 1:53 AM To: "user@knox.apache.org<mailto:user@knox.apache.org>" <user@knox.apache.org<mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox Guang – This is somewhat to be expected. When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source. With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you. Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity. -- Sean Roberts From: Guang Yang <k...@uber.com<mailto:k...@uber.com>> Reply-To: "user@knox.apache.org<mailto:user@knox.apache.org>" <user@knox.apache.org<mailto:user@knox.apache.org>> Date: Tuesday, 4 September 2018 at 07:46 To: "user@knox.apache.org<mailto:user@knox.apache.org>" <user@knox.apache.org<mailto:user@knox.apache.org>> Subject: WebHDFS performance issue in Knox Hi, We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least. Are you guys aware of this problem? Any suggestion? Thanks, Guang
