Thanks guys! The issue seems exactly what David pointed out, which is because of encrypted over SSL.
Without Knox, the download speed can reach to *400M/s* if I call Namenode directly. And with disabling SSL, the speed can reach to *~400M/s* as well through Knox. But with SSL, the speed drops significantly to *~40M/s*. I know it's because of encrypted, but it does surprised me with such a difference. Is it normal from your perspective? Thanks, Guang On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < dvillarr...@hortonworks.com> wrote: > Hi Guang, > > > > Keep in mind the data is being encrypted over SSL. If you disable SSL you > will most likely see a very significant boost in throughput. Some people > have used more powerful computers to make encryption quicker. > > > > Thanks, > > > > David > > > > *From: *Sean Roberts <srobe...@hortonworks.com> > *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> > *Date: *Tuesday, September 4, 2018 at 1:53 AM > *To: *"user@knox.apache.org" <user@knox.apache.org> > *Subject: *Re: WebHDFS performance issue in Knox > > > > Guang – This is somewhat to be expected. > > > > When you talk to WebHDFS directly, the client can distribute the request > across many data nodes. Also, you are getting data directly from the source. > > With Knox, all traffic goes through the single Knox host. Knox is > responsible for fetching from the datanodes and consolidating to send to > you. This means overhead as it’s acting as a middle man, and lower network > capacity since only 1 host is serving data to you. > > > > Also, if running on a cloud provider, the Knox host may be a smaller > instance size with lower network capacity. > > -- > > Sean Roberts > > > > *From: *Guang Yang <k...@uber.com> > *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> > *Date: *Tuesday, 4 September 2018 at 07:46 > *To: *"user@knox.apache.org" <user@knox.apache.org> > *Subject: *WebHDFS performance issue in Knox > > > > Hi, > > > > We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file > through WebHDFS in Knox, the download speed is just about 11M/s. However, > if we download directly from datanode, the speed is about 40M/s at least. > > > > Are you guys aware of this problem? Any suggestion? > > > > Thanks, > > Guang >
