For reference I summarized from this thread and put the results here: KNOX-1221
The write specific performance improvement is here: KNOX-1521 Kevin Risden On Wed, Oct 10, 2018 at 3:48 PM Kevin Risden <kris...@apache.org> wrote: > I tried disabling GCM ciphers based on the following information: > * https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8 > * > https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20 > > The results for the read were: > * knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s > * knox ssl - 1,073,741,824 54.3MB/s in 20s > > This is a little more than a 2x speedup. There is also information in the > links above that there should be more performance improvements with JDK 9+. > > For the write side slow down, I found an issue with how Knox is handing > the streaming data on writes only. I am looking into fixing this to get the > write performance for HDFS improved. > > Kevin Risden > > > On Wed, Oct 10, 2018 at 1:20 PM David Villarreal < > dvillarr...@hortonworks.com> wrote: > >> I believe Curl has an option of what cipher to use.. You may also be >> able to force it at the server jvm level using >> /jre/lib/security/java.security >> >> >> >> >> >> *From: *Sandeep Moré <moresand...@gmail.com> >> *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> >> *Date: *Tuesday, October 9, 2018 at 6:39 PM >> *To: *"user@knox.apache.org" <user@knox.apache.org> >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> I think this would be a good test, worth a try, not sure how we can force >> a certain cipher to be used perhaps a permutation combination of >> >> ssl.include.ciphers, ssl.exclude.ciphers. >> >> >> >> Best, >> >> Sandeep >> >> >> >> >> >> On Tue, Oct 9, 2018 at 5:29 PM David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> Hi Kevin, >> >> >> >> In my humble opinion, this has to do with cpu processing encryption in >> general based on which cipher being used. Couldn’t the same type of >> principals/improvements (hdfs encryption improvements) be done here for >> let’s say for AES cipher suites? If the main bottleneck here is CPU >> couldn’t you enhance encryption though hardware acceleration and you may >> see better performance numbers? >> >> >> >> https://calomel.org/aesni_ssl_performance.html >> >> >> >> Try forcing a less secure cipher to be used in your environment. Do you >> then see better numbers? >> >> >> >> dav >> >> >> >> >> >> >> >> >> >> >> *From:Kevin Risden <kris...@apache.org <kris...@apache.org>> Reply-To: >> "user@knox.apache.org <user@knox.apache.org>" <user@knox.apache.org >> <user@knox.apache.org>> Date: Tuesday, October 9, 2018 at 1:05 PM To: >> "user@knox.apache.org <user@knox.apache.org>" <user@knox.apache.org >> <user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox* >> >> >> >> @David - Not sure what you mean since this is SSL/TLS and not related to >> RPC encryption like the two JIRAs that you linked. >> >> @Guang - NP just took some time to sit down and look at it. >> >> >> >> Some preliminary investigation shows this may be the JDK implementation >> of TLS/SSL that is slowing down the read path. I need to dig into it >> further but found a few references showing that Java slowness for TLS/SSL >> affects Jetty. >> >> - >> https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions >> - >> https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements >> - https://webtide.com/conscrypting-native-ssl-for-jetty/ >> >> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to >> enable conscrypting ( >> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). >> With that I was able to get read performance on par with non ssl and native >> webhdfs. The write side of the equation still has some performance >> differences that need to be looked at further. >> >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <k...@uber.com> wrote: >> >> Thanks Kevin conducting such experiment! This is exactly what I saw >> before. It doesn't look right the download speed is 10x slower when >> enabling SSL. >> >> >> >> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> I bring this up because HDFS encryption saw an increase in performance. >> >> https://issues.apache.org/jira/browse/HDFS-6606 >> >> >> >> https://issues.apache.org/jira/browse/HADOOP-10768 >> >> >> >> Maybe Knox can make some enhancements in this area? >> >> >> >> *From: *David Villarreal <dvillarr...@hortonworks.com> >> *Date: *Tuesday, October 9, 2018 at 10:34 AM >> *To: *"user@knox.apache.org" <user@knox.apache.org> >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> Hi Kevin, >> >> Now increase your CPU processing power and show me the numbers. >> >> >> >> Do we support AES-NI optimization with extended CPU instruction set for >> AES hardware acceleration? >> >> libcrypto.so library that supports hardware acceleration, such as >> OpenSSL 1.0.1e. (Many OS versions have an older version of the library that >> does not support AES-NI.) >> >> >> >> >> >> >> *From: * >> >> *Kevin Risden* >> >> >> >> >> >> *<kris...@apache.org <kris...@apache.org>> Reply-To: >> "user@knox.apache.org <user@knox.apache.org>" <user@knox.apache.org >> <user@knox.apache.org>> Date: Tuesday, October 9, 2018 at 10:26 AM To: >> "user@knox.apache.org <user@knox.apache.org>" <user@knox.apache.org >> <user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox* >> >> >> >> Writes look to have performance impact as well: >> >> - directly to webhdfs - ~2.6 seconds >> - knox no ssl - ~29 seconds >> - knox ssl - ~49.6 seconds >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kris...@apache.org> wrote: >> >> If I run two downloads concurrently: >> >> >> >> 1,073,741,824 46.1MB/s in 22s >> >> 1,073,741,824 51.3MB/s in 22s >> >> >> >> So it isn't a limitation of the Knox gateway itself in total bandwidth >> but a per connection limitation somehow. >> >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kris...@apache.org> wrote: >> >> So I was able to reproduce a slowdown with SSL with a pseudo distributed >> HDFS setup on a single node with Knox running on the same node. This was >> setup in Virtualbox on my laptop. >> >> >> >> Rough timings with wget for a 1GB random file: >> >> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s >> - knox no ssl - 1,073,741,824 264MB/s in 3.6s >> - knox ssl - 1,073,741,824 54.3MB/s in 20s >> >> There is a significant decrease with Knox SSL for some reason. >> >> >> >> Kevin Risden >> >> >> >> >> >> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lmc...@apache.org> wrote: >> >> SSL handshake will likely happen at least twice. >> >> Once for the request through Knox to the NN then the redirect from the NN >> to the DN goes all the way back to the client. >> >> So they have to follow the redirect and do the handshake to the DN. >> >> >> >> >> >> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kris...@apache.org> wrote: >> >> So I found this in the Knox issues list in JIRA: >> >> >> >> https://issues.apache.org/jira/browse/KNOX-1221 >> >> >> >> It sounds familiar in terms of a slowdown when going through Knox. >> >> >> Kevin Risden >> >> >> >> >> >> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kris...@apache.org> wrote: >> >> Hmmm yea curl for a single file should do the handshake once. >> >> >> >> What are the system performance statistics during the SSL vs non SSL >> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here >> if using that. Otherwise watching top may be helpful. It would be help to >> determine if the Knox is working harder during the SSL transfer. >> >> >> Kevin Risden >> >> >> >> >> >> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <k...@uber.com> wrote: >> >> I'm just using curl to download a single large file. So I suspect SSL >> handshake just happens once? >> >> >> >> On Tue, Sep 11, 2018 at 12:02 PM >> >> Kevin Risden >> >> <kris...@apache.org> wrote: >> >> What client are you using to connect Knox? Is this for a single file or a >> bunch of files? >> >> >> >> The SSL handshake can be slow if the client doesn't keep the connection >> open. >> >> Kevin Risden >> >> >> >> On Tue, Sep 11, 2018, 14:51 Guang Yang <k...@uber.com> wrote: >> >> Thanks Larry. But the only difference is this part in my gateway-site.xml. >> >> >> >> *<property>* >> >> * <name>ssl.enabled</name>* >> >> * <value>false</value>* >> >> * <description>Indicates whether SSL is enabled.</description>* >> >> *</property>* >> >> >> >> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lmc...@apache.org> wrote: >> >> I really don't think that kind of difference should be expected from >> merely SSL overhead. >> >> I don't however have any metrics to contradict it either since I do not >> run Knox without SSL. >> >> >> >> Given the above, I am struggling coming up with a meaningful response to >> this. :( >> >> I don't think you should see a 10 fold increase in speed by disabling SSL >> though. >> >> >> >> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <k...@uber.com> wrote: >> >> Any idea guys? >> >> >> >> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <k...@uber.com> wrote: >> >> Thanks guys! The issue seems exactly what David pointed out, which is >> because of encrypted over SSL. >> >> >> >> Without Knox, the download speed can reach to *400M/s* if I call >> Namenode directly. And with disabling SSL, the speed can reach to >> *~400M/s* as well through Knox. But with SSL, the speed drops >> significantly to *~40M/s*. I know it's because of encrypted, but it does >> surprised me with such a difference. Is it normal from your perspective? >> >> >> >> Thanks, >> >> Guang >> >> >> >> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> Hi Guang, >> >> >> >> Keep in mind the data is being encrypted over SSL. If you disable SSL >> you will most likely see a very significant boost in throughput. Some >> people have used more powerful computers to make encryption quicker. >> >> >> >> Thanks, >> >> >> >> David >> >> >> >> *From: *Sean Roberts <srobe...@hortonworks.com> >> *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> >> *Date: *Tuesday, September 4, 2018 at 1:53 AM >> *To: *"user@knox.apache.org" <user@knox.apache.org> >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> Guang – This is somewhat to be expected. >> >> >> >> When you talk to WebHDFS directly, the client can distribute the request >> across many data nodes. Also, you are getting data directly from the source. >> >> With Knox, all traffic goes through the single Knox host. Knox is >> responsible for fetching from the datanodes and consolidating to send to >> you. This means overhead as it’s acting as a middle man, and lower network >> capacity since only 1 host is serving data to you. >> >> >> >> Also, if running on a cloud provider, the Knox host may be a smaller >> instance size with lower network capacity. >> >> -- >> >> Sean Roberts >> >> >> >> *From: *Guang Yang <k...@uber.com> >> *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> >> *Date: *Tuesday, 4 September 2018 at 07:46 >> *To: *"user@knox.apache.org" <user@knox.apache.org> >> *Subject: *WebHDFS performance issue in Knox >> >> >> >> Hi, >> >> >> >> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file >> through WebHDFS in Knox, the download speed is just about 11M/s. However, >> if we download directly from datanode, the speed is about 40M/s at least. >> >> >> >> Are you guys aware of this problem? Any suggestion? >> >> >> >> Thanks, >> >> Guang >> >> >> >> >> >> >> >>
