Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL.
On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < dvillarr...@hortonworks.com> wrote: > I bring this up because HDFS encryption saw an increase in performance. > > https://issues.apache.org/jira/browse/HDFS-6606 > > > > https://issues.apache.org/jira/browse/HADOOP-10768 > > > > Maybe Knox can make some enhancements in this area? > > > > *From: *David Villarreal <dvillarr...@hortonworks.com> > *Date: *Tuesday, October 9, 2018 at 10:34 AM > *To: *"user@knox.apache.org" <user@knox.apache.org> > *Subject: *Re: WebHDFS performance issue in Knox > > > > Hi Kevin, > > Now increase your CPU processing power and show me the numbers. > > > > Do we support AES-NI optimization with extended CPU instruction set for > AES hardware acceleration? > > libcrypto.so library that supports hardware acceleration, such as OpenSSL > 1.0.1e. (Many OS versions have an older version of the library that does > not support AES-NI.) > > > > > > *From: *Kevin Risden <kris...@apache.org> > *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> > *Date: *Tuesday, October 9, 2018 at 10:26 AM > *To: *"user@knox.apache.org" <user@knox.apache.org> > *Subject: *Re: WebHDFS performance issue in Knox > > > > Writes look to have performance impact as well: > > - directly to webhdfs - ~2.6 seconds > - knox no ssl - ~29 seconds > - knox ssl - ~49.6 seconds > > Kevin Risden > > > > > > On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kris...@apache.org> wrote: > > If I run two downloads concurrently: > > > > 1,073,741,824 46.1MB/s in 22s > > 1,073,741,824 51.3MB/s in 22s > > > > So it isn't a limitation of the Knox gateway itself in total bandwidth but > a per connection limitation somehow. > > > Kevin Risden > > > > > > On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kris...@apache.org> wrote: > > So I was able to reproduce a slowdown with SSL with a pseudo distributed > HDFS setup on a single node with Knox running on the same node. This was > setup in Virtualbox on my laptop. > > > > Rough timings with wget for a 1GB random file: > > - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s > - knox no ssl - 1,073,741,824 264MB/s in 3.6s > - knox ssl - 1,073,741,824 54.3MB/s in 20s > > There is a significant decrease with Knox SSL for some reason. > > > > Kevin Risden > > > > > > On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lmc...@apache.org> wrote: > > SSL handshake will likely happen at least twice. > > Once for the request through Knox to the NN then the redirect from the NN > to the DN goes all the way back to the client. > > So they have to follow the redirect and do the handshake to the DN. > > > > > > On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kris...@apache.org> wrote: > > So I found this in the Knox issues list in JIRA: > > > > https://issues.apache.org/jira/browse/KNOX-1221 > > > > It sounds familiar in terms of a slowdown when going through Knox. > > > Kevin Risden > > > > > > On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kris...@apache.org> wrote: > > Hmmm yea curl for a single file should do the handshake once. > > > > What are the system performance statistics during the SSL vs non SSL > testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here > if using that. Otherwise watching top may be helpful. It would be help to > determine if the Knox is working harder during the SSL transfer. > > > Kevin Risden > > > > > > On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <k...@uber.com> wrote: > > I'm just using curl to download a single large file. So I suspect SSL > handshake just happens once? > > > > On Tue, Sep 11, 2018 at 12:02 PM > > Kevin Risden > > <kris...@apache.org> wrote: > > What client are you using to connect Knox? Is this for a single file or a > bunch of files? > > > > The SSL handshake can be slow if the client doesn't keep the connection > open. > > Kevin Risden > > > > On Tue, Sep 11, 2018, 14:51 Guang Yang <k...@uber.com> wrote: > > Thanks Larry. But the only difference is this part in my gateway-site.xml. > > > > *<property>* > > * <name>ssl.enabled</name>* > > * <value>false</value>* > > * <description>Indicates whether SSL is enabled.</description>* > > *</property>* > > > > On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lmc...@apache.org> wrote: > > I really don't think that kind of difference should be expected from > merely SSL overhead. > > I don't however have any metrics to contradict it either since I do not > run Knox without SSL. > > > > Given the above, I am struggling coming up with a meaningful response to > this. :( > > I don't think you should see a 10 fold increase in speed by disabling SSL > though. > > > > On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <k...@uber.com> wrote: > > Any idea guys? > > > > On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <k...@uber.com> wrote: > > Thanks guys! The issue seems exactly what David pointed out, which is > because of encrypted over SSL. > > > > Without Knox, the download speed can reach to *400M/s* if I call Namenode > directly. And with disabling SSL, the speed can reach to *~400M/s* as > well through Knox. But with SSL, the speed drops significantly to *~40M/s*. > I know it's because of encrypted, but it does surprised me with such a > difference. Is it normal from your perspective? > > > > Thanks, > > Guang > > > > On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < > dvillarr...@hortonworks.com> wrote: > > Hi Guang, > > > > Keep in mind the data is being encrypted over SSL. If you disable SSL you > will most likely see a very significant boost in throughput. Some people > have used more powerful computers to make encryption quicker. > > > > Thanks, > > > > David > > > > *From: *Sean Roberts <srobe...@hortonworks.com> > *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> > *Date: *Tuesday, September 4, 2018 at 1:53 AM > *To: *"user@knox.apache.org" <user@knox.apache.org> > *Subject: *Re: WebHDFS performance issue in Knox > > > > Guang – This is somewhat to be expected. > > > > When you talk to WebHDFS directly, the client can distribute the request > across many data nodes. Also, you are getting data directly from the source. > > With Knox, all traffic goes through the single Knox host. Knox is > responsible for fetching from the datanodes and consolidating to send to > you. This means overhead as it’s acting as a middle man, and lower network > capacity since only 1 host is serving data to you. > > > > Also, if running on a cloud provider, the Knox host may be a smaller > instance size with lower network capacity. > > -- > > Sean Roberts > > > > *From: *Guang Yang <k...@uber.com> > *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> > *Date: *Tuesday, 4 September 2018 at 07:46 > *To: *"user@knox.apache.org" <user@knox.apache.org> > *Subject: *WebHDFS performance issue in Knox > > > > Hi, > > > > We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file > through WebHDFS in Knox, the download speed is just about 11M/s. However, > if we download directly from datanode, the speed is about 40M/s at least. > > > > Are you guys aware of this problem? Any suggestion? > > > > Thanks, > > Guang > > > > > > > >
