Any idea guys? On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <k...@uber.com> wrote:
> Thanks guys! The issue seems exactly what David pointed out, which is > because of encrypted over SSL. > > Without Knox, the download speed can reach to *400M/s* if I call Namenode > directly. And with disabling SSL, the speed can reach to *~400M/s* as > well through Knox. But with SSL, the speed drops significantly to *~40M/s*. > I know it's because of encrypted, but it does surprised me with such a > difference. Is it normal from your perspective? > > Thanks, > Guang > > On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < > dvillarr...@hortonworks.com> wrote: > >> Hi Guang, >> >> >> >> Keep in mind the data is being encrypted over SSL. If you disable SSL >> you will most likely see a very significant boost in throughput. Some >> people have used more powerful computers to make encryption quicker. >> >> >> >> Thanks, >> >> >> >> David >> >> >> >> *From: *Sean Roberts <srobe...@hortonworks.com> >> *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> >> *Date: *Tuesday, September 4, 2018 at 1:53 AM >> *To: *"user@knox.apache.org" <user@knox.apache.org> >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> Guang – This is somewhat to be expected. >> >> >> >> When you talk to WebHDFS directly, the client can distribute the request >> across many data nodes. Also, you are getting data directly from the source. >> >> With Knox, all traffic goes through the single Knox host. Knox is >> responsible for fetching from the datanodes and consolidating to send to >> you. This means overhead as it’s acting as a middle man, and lower network >> capacity since only 1 host is serving data to you. >> >> >> >> Also, if running on a cloud provider, the Knox host may be a smaller >> instance size with lower network capacity. >> >> -- >> >> Sean Roberts >> >> >> >> *From: *Guang Yang <k...@uber.com> >> *Reply-To: *"user@knox.apache.org" <user@knox.apache.org> >> *Date: *Tuesday, 4 September 2018 at 07:46 >> *To: *"user@knox.apache.org" <user@knox.apache.org> >> *Subject: *WebHDFS performance issue in Knox >> >> >> >> Hi, >> >> >> >> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file >> through WebHDFS in Knox, the download speed is just about 11M/s. However, >> if we download directly from datanode, the speed is about 40M/s at least. >> >> >> >> Are you guys aware of this problem? Any suggestion? >> >> >> >> Thanks, >> >> Guang >> > >
