Re: WebHDFS performance issue in Knox
Awesome, I had seen GCM suck big time in the past. Great work ! On Wed, Oct 10, 2018 at 3:48 PM Kevin Risden wrote: > I tried disabling GCM ciphers based on the following information: > * https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8 > * > https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20 > > The results for the read were: > * knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s > * knox ssl - 1,073,741,824 54.3MB/s in 20s > > This is a little more than a 2x speedup. There is also information in the > links above that there should be more performance improvements with JDK 9+. > > For the write side slow down, I found an issue with how Knox is handing > the streaming data on writes only. I am looking into fixing this to get the > write performance for HDFS improved. > > Kevin Risden > > > On Wed, Oct 10, 2018 at 1:20 PM David Villarreal < > dvillarr...@hortonworks.com> wrote: > >> I believe Curl has an option of what cipher to use.. You may also be >> able to force it at the server jvm level using >> /jre/lib/security/java.security >> >> >> >> >> >> *From: *Sandeep Moré >> *Reply-To: *"user@knox.apache.org" >> *Date: *Tuesday, October 9, 2018 at 6:39 PM >> *To: *"user@knox.apache.org" >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> I think this would be a good test, worth a try, not sure how we can force >> a certain cipher to be used perhaps a permutation combination of >> >> ssl.include.ciphers, ssl.exclude.ciphers. >> >> >> >> Best, >> >> Sandeep >> >> >> >> >> >> On Tue, Oct 9, 2018 at 5:29 PM David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> Hi Kevin, >> >> >> >> In my humble opinion, this has to do with cpu processing encryption in >> general based on which cipher being used. Couldn’t the same type of >> principals/improvements (hdfs encryption improvements) be done here for >> let’s say for AES cipher suites? If the main bottleneck here is CPU >> couldn’t you enhance encryption though hardware acceleration and you may >> see better performance numbers? >> >> >> >> https://calomel.org/aesni_ssl_performance.html >> >> >> >> Try forcing a less secure cipher to be used in your environment. Do you >> then see better numbers? >> >> >> >> dav >> >> >> >> >> >> >> *From:* >> *Kevin Risden* >> >> >> >> >> * > Reply-To: >> "user@knox.apache.org " > > Date: Tuesday, October 9, 2018 at 1:05 PM To: >> "user@knox.apache.org " > > Subject: Re: WebHDFS performance issue in Knox* >> >> >> >> @David - Not sure what you mean since this is SSL/TLS and not related to >> RPC encryption like the two JIRAs that you linked. >> >> @Guang - NP just took some time to sit down and look at it. >> >> >> >> Some preliminary investigation shows this may be the JDK implementation >> of TLS/SSL that is slowing down the read path. I need to dig into it >> further but found a few references showing that Java slowness for TLS/SSL >> affects Jetty. >> >>- >>https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions >>- >>https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements >>- https://webtide.com/conscrypting-native-ssl-for-jetty/ >> >> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to >> enable conscrypting ( >> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). >> With that I was able to get read performance on par with non ssl and native >> webhdfs. The write side of the equation still has some performance >> differences that need to be looked at further. >> >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang wrote: >> >> Thanks Kevin conducting such experiment! This is exactly what I saw >> before. It doesn't look right the download speed is 10x slower when >> enabling SSL. >> >> >> >> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> I bring this up because HDFS encryption saw an increase in performance. >> >> https://issues.apache.org/jira/browse/HDFS-6606 >> >> >> >
Re: WebHDFS performance issue in Knox
Interesting. Nice work. 2x improvement is great! From: Kevin Risden Reply-To: "user@knox.apache.org" Date: Wednesday, October 10, 2018 at 12:48 PM To: "user@knox.apache.org" Subject: Re: WebHDFS performance issue in Knox I tried disabling GCM ciphers based on the following information: * https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8 * https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20 The results for the read were: * knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s * knox ssl - 1,073,741,824 54.3MB/s in 20s This is a little more than a 2x speedup. There is also information in the links above that there should be more performance improvements with JDK 9+. For the write side slow down, I found an issue with how Knox is handing the streaming data on writes only. I am looking into fixing this to get the write performance for HDFS improved. Kevin Risden On Wed, Oct 10, 2018 at 1:20 PM David Villarreal mailto:dvillarr...@hortonworks.com>> wrote: I believe Curl has an option of what cipher to use.. You may also be able to force it at the server jvm level using /jre/lib/security/java.security From: Sandeep Moré mailto:moresand...@gmail.com>> Reply-To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Date: Tuesday, October 9, 2018 at 6:39 PM To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox I think this would be a good test, worth a try, not sure how we can force a certain cipher to be used perhaps a permutation combination of ssl.include.ciphers, ssl.exclude.ciphers. Best, Sandeep On Tue, Oct 9, 2018 at 5:29 PM David Villarreal mailto:dvillarr...@hortonworks.com>> wrote: Hi Kevin, In my humble opinion, this has to do with cpu processing encryption in general based on which cipher being used. Couldn’t the same type of principals/improvements (hdfs encryption improvements) be done here for let’s say for AES cipher suites? If the main bottleneck here is CPU couldn’t you enhance encryption though hardware acceleration and you may see better performance numbers? https://calomel.org/aesni_ssl_performance.html Try forcing a less secure cipher to be used in your environment. Do you then see better numbers? dav From: Kevin Risden mailto:kris...@apache.org>> Reply-To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Date: Tuesday, October 9, 2018 at 1:05 PM To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox @David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked. @Guang - NP just took some time to sit down and look at it. Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty. * https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions * https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements * https://webtide.com/conscrypting-native-ssl-for-jetty/ Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting (https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further. Kevin Risden On Tue, Oct 9, 2018 at 2:01 PM Guang Yang mailto:k...@uber.com>> wrote: Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL. On Tue, Oct 9, 2018 at 10:40 AM David Villarreal mailto:dvillarr...@hortonworks.com>> wrote: I bring this up because HDFS encryption saw an increase in performance. https://issues.apache.org/jira/browse/HDFS-6606 https://issues.apache.org/jira/browse/HADOOP-10768 Maybe Knox can make some enhancements in this area? From: David Villarreal mailto:dvillarr...@hortonworks.com>> Date: Tuesday, October 9, 2018 at 10:34 AM To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox Hi Kevin, Now increase your CPU processing power and show me the numbers. Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration? libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not
Re: WebHDFS performance issue in Knox
For reference I summarized from this thread and put the results here: KNOX-1221 The write specific performance improvement is here: KNOX-1521 Kevin Risden On Wed, Oct 10, 2018 at 3:48 PM Kevin Risden wrote: > I tried disabling GCM ciphers based on the following information: > * https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8 > * > https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20 > > The results for the read were: > * knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s > * knox ssl - 1,073,741,824 54.3MB/s in 20s > > This is a little more than a 2x speedup. There is also information in the > links above that there should be more performance improvements with JDK 9+. > > For the write side slow down, I found an issue with how Knox is handing > the streaming data on writes only. I am looking into fixing this to get the > write performance for HDFS improved. > > Kevin Risden > > > On Wed, Oct 10, 2018 at 1:20 PM David Villarreal < > dvillarr...@hortonworks.com> wrote: > >> I believe Curl has an option of what cipher to use.. You may also be >> able to force it at the server jvm level using >> /jre/lib/security/java.security >> >> >> >> >> >> *From: *Sandeep Moré >> *Reply-To: *"user@knox.apache.org" >> *Date: *Tuesday, October 9, 2018 at 6:39 PM >> *To: *"user@knox.apache.org" >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> I think this would be a good test, worth a try, not sure how we can force >> a certain cipher to be used perhaps a permutation combination of >> >> ssl.include.ciphers, ssl.exclude.ciphers. >> >> >> >> Best, >> >> Sandeep >> >> >> >> >> >> On Tue, Oct 9, 2018 at 5:29 PM David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> Hi Kevin, >> >> >> >> In my humble opinion, this has to do with cpu processing encryption in >> general based on which cipher being used. Couldn’t the same type of >> principals/improvements (hdfs encryption improvements) be done here for >> let’s say for AES cipher suites? If the main bottleneck here is CPU >> couldn’t you enhance encryption though hardware acceleration and you may >> see better performance numbers? >> >> >> >> https://calomel.org/aesni_ssl_performance.html >> >> >> >> Try forcing a less secure cipher to be used in your environment. Do you >> then see better numbers? >> >> >> >> dav >> >> >> >> >> >> >> >> >> >> >> *From:Kevin Risden > Reply-To: >> "user@knox.apache.org " > > Date: Tuesday, October 9, 2018 at 1:05 PM To: >> "user@knox.apache.org " > > Subject: Re: WebHDFS performance issue in Knox* >> >> >> >> @David - Not sure what you mean since this is SSL/TLS and not related to >> RPC encryption like the two JIRAs that you linked. >> >> @Guang - NP just took some time to sit down and look at it. >> >> >> >> Some preliminary investigation shows this may be the JDK implementation >> of TLS/SSL that is slowing down the read path. I need to dig into it >> further but found a few references showing that Java slowness for TLS/SSL >> affects Jetty. >> >>- >>https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions >>- >>https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements >>- https://webtide.com/conscrypting-native-ssl-for-jetty/ >> >> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to >> enable conscrypting ( >> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). >> With that I was able to get read performance on par with non ssl and native >> webhdfs. The write side of the equation still has some performance >> differences that need to be looked at further. >> >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang wrote: >> >> Thanks Kevin conducting such experiment! This is exactly what I saw >> before. It doesn't look right the download speed is 10x slower when >> enabling SSL. >> >> >> >> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> I bring this up because HDFS encryption saw an increase in performance. >> >> https://issues.a
Re: WebHDFS performance issue in Knox
I believe Curl has an option of what cipher to use.. You may also be able to force it at the server jvm level using /jre/lib/security/java.security From: Sandeep Moré Reply-To: "user@knox.apache.org" Date: Tuesday, October 9, 2018 at 6:39 PM To: "user@knox.apache.org" Subject: Re: WebHDFS performance issue in Knox I think this would be a good test, worth a try, not sure how we can force a certain cipher to be used perhaps a permutation combination of ssl.include.ciphers, ssl.exclude.ciphers. Best, Sandeep On Tue, Oct 9, 2018 at 5:29 PM David Villarreal mailto:dvillarr...@hortonworks.com>> wrote: Hi Kevin, In my humble opinion, this has to do with cpu processing encryption in general based on which cipher being used. Couldn’t the same type of principals/improvements (hdfs encryption improvements) be done here for let’s say for AES cipher suites? If the main bottleneck here is CPU couldn’t you enhance encryption though hardware acceleration and you may see better performance numbers? https://calomel.org/aesni_ssl_performance.html Try forcing a less secure cipher to be used in your environment. Do you then see better numbers? dav From: Kevin Risden mailto:kris...@apache.org>> Reply-To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Date: Tuesday, October 9, 2018 at 1:05 PM To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox @David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked. @Guang - NP just took some time to sit down and look at it. Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty. * https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions * https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements * https://webtide.com/conscrypting-native-ssl-for-jetty/ Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting (https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further. Kevin Risden On Tue, Oct 9, 2018 at 2:01 PM Guang Yang mailto:k...@uber.com>> wrote: Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL. On Tue, Oct 9, 2018 at 10:40 AM David Villarreal mailto:dvillarr...@hortonworks.com>> wrote: I bring this up because HDFS encryption saw an increase in performance. https://issues.apache.org/jira/browse/HDFS-6606 https://issues.apache.org/jira/browse/HADOOP-10768 Maybe Knox can make some enhancements in this area? From: David Villarreal mailto:dvillarr...@hortonworks.com>> Date: Tuesday, October 9, 2018 at 10:34 AM To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox Hi Kevin, Now increase your CPU processing power and show me the numbers. Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration? libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.) From: Kevin Risden mailto:kris...@apache.org>> Reply-To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Date: Tuesday, October 9, 2018 at 10:26 AM To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox Writes look to have performance impact as well: * directly to webhdfs - ~2.6 seconds * knox no ssl - ~29 seconds * knox ssl - ~49.6 seconds Kevin Risden On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden mailto:kris...@apache.org>> wrote: If I run two downloads concurrently: 1,073,741,824 46.1MB/s in 22s 1,073,741,824 51.3MB/s in 22s So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow. Kevin Risden On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden mailto:kris...@apache.org>> wrote: So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop. Rough timings with wget for a 1GB random file: * directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
Re: WebHDFS performance issue in Knox
I think this would be a good test, worth a try, not sure how we can force a certain cipher to be used perhaps a permutation combination of ssl.include.ciphers, ssl.exclude.ciphers. Best, Sandeep On Tue, Oct 9, 2018 at 5:29 PM David Villarreal wrote: > Hi Kevin, > > > > In my humble opinion, this has to do with cpu processing encryption in > general based on which cipher being used. Couldn’t the same type of > principals/improvements (hdfs encryption improvements) be done here for > let’s say for AES cipher suites? If the main bottleneck here is CPU > couldn’t you enhance encryption though hardware acceleration and you may > see better performance numbers? > > > > https://calomel.org/aesni_ssl_performance.html > > > > Try forcing a less secure cipher to be used in your environment. Do you > then see better numbers? > > > > dav > > > > > > *From: *Kevin Risden > *Reply-To: *"user@knox.apache.org" > *Date: *Tuesday, October 9, 2018 at 1:05 PM > *To: *"user@knox.apache.org" > *Subject: *Re: WebHDFS performance issue in Knox > > > > @David - Not sure what you mean since this is SSL/TLS and not related to > RPC encryption like the two JIRAs that you linked. > > @Guang - NP just took some time to sit down and look at it. > > > > Some preliminary investigation shows this may be the JDK implementation of > TLS/SSL that is slowing down the read path. I need to dig into it further > but found a few references showing that Java slowness for TLS/SSL affects > Jetty. > >- >https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions >- >https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements >- https://webtide.com/conscrypting-native-ssl-for-jetty/ > > Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to > enable conscrypting ( > https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). > With that I was able to get read performance on par with non ssl and native > webhdfs. The write side of the equation still has some performance > differences that need to be looked at further. > > > Kevin Risden > > > > > > On Tue, Oct 9, 2018 at 2:01 PM Guang Yang wrote: > > Thanks Kevin conducting such experiment! This is exactly what I saw > before. It doesn't look right the download speed is 10x slower when > enabling SSL. > > > > On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < > dvillarr...@hortonworks.com> wrote: > > I bring this up because HDFS encryption saw an increase in performance. > > https://issues.apache.org/jira/browse/HDFS-6606 > > > > https://issues.apache.org/jira/browse/HADOOP-10768 > > > > Maybe Knox can make some enhancements in this area? > > > > *From: *David Villarreal > *Date: *Tuesday, October 9, 2018 at 10:34 AM > *To: *"user@knox.apache.org" > *Subject: *Re: WebHDFS performance issue in Knox > > > > Hi Kevin, > > Now increase your CPU processing power and show me the numbers. > > > > Do we support AES-NI optimization with extended CPU instruction set for > AES hardware acceleration? > > libcrypto.so library that supports hardware acceleration, such as OpenSSL > 1.0.1e. (Many OS versions have an older version of the library that does > not support AES-NI.) > > > > > > > *From: * > > *Kevin Risden* > > > > > > *> Reply-To: "user@knox.apache.org > " > Date: > Tuesday, October 9, 2018 at 10:26 AM To: "user@knox.apache.org > " > > Subject: Re: WebHDFS performance issue in Knox* > > > > Writes look to have performance impact as well: > >- directly to webhdfs - ~2.6 seconds >- knox no ssl - ~29 seconds >- knox ssl - ~49.6 seconds > > Kevin Risden > > > > > > On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden wrote: > > If I run two downloads concurrently: > > > > 1,073,741,824 46.1MB/s in 22s > > 1,073,741,824 51.3MB/s in 22s > > > > So it isn't a limitation of the Knox gateway itself in total bandwidth but > a per connection limitation somehow. > > > Kevin Risden > > > > > > On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden wrote: > > So I was able to reproduce a slowdown with SSL with a pseudo distributed > HDFS setup on a single node with Knox running on the same node. This was > setup in Virtualbox on my laptop. > > > > Rough timings with wget for a 1GB random file: > >- directly to webhdfs - 1,073,741,824 252MB/s in 3.8s >- knox no ssl - 1,073,741,824 264MB/s in 3.6s >- knox ss
Re: WebHDFS performance issue in Knox
Hi Kevin, In my humble opinion, this has to do with cpu processing encryption in general based on which cipher being used. Couldn’t the same type of principals/improvements (hdfs encryption improvements) be done here for let’s say for AES cipher suites? If the main bottleneck here is CPU couldn’t you enhance encryption though hardware acceleration and you may see better performance numbers? https://calomel.org/aesni_ssl_performance.html Try forcing a less secure cipher to be used in your environment. Do you then see better numbers? dav From: Kevin Risden Reply-To: "user@knox.apache.org" Date: Tuesday, October 9, 2018 at 1:05 PM To: "user@knox.apache.org" Subject: Re: WebHDFS performance issue in Knox @David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked. @Guang - NP just took some time to sit down and look at it. Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty. * https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions * https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements * https://webtide.com/conscrypting-native-ssl-for-jetty/ Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting (https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further. Kevin Risden On Tue, Oct 9, 2018 at 2:01 PM Guang Yang mailto:k...@uber.com>> wrote: Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL. On Tue, Oct 9, 2018 at 10:40 AM David Villarreal mailto:dvillarr...@hortonworks.com>> wrote: I bring this up because HDFS encryption saw an increase in performance. https://issues.apache.org/jira/browse/HDFS-6606 https://issues.apache.org/jira/browse/HADOOP-10768 Maybe Knox can make some enhancements in this area? From: David Villarreal mailto:dvillarr...@hortonworks.com>> Date: Tuesday, October 9, 2018 at 10:34 AM To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox Hi Kevin, Now increase your CPU processing power and show me the numbers. Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration? libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.) From: Kevin Risden mailto:kris...@apache.org>> Reply-To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Date: Tuesday, October 9, 2018 at 10:26 AM To: "user@knox.apache.org<mailto:user@knox.apache.org>" mailto:user@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox Writes look to have performance impact as well: * directly to webhdfs - ~2.6 seconds * knox no ssl - ~29 seconds * knox ssl - ~49.6 seconds Kevin Risden On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden mailto:kris...@apache.org>> wrote: If I run two downloads concurrently: 1,073,741,824 46.1MB/s in 22s 1,073,741,824 51.3MB/s in 22s So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow. Kevin Risden On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden mailto:kris...@apache.org>> wrote: So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop. Rough timings with wget for a 1GB random file: * directly to webhdfs - 1,073,741,824 252MB/s in 3.8s * knox no ssl - 1,073,741,824 264MB/s in 3.6s * knox ssl - 1,073,741,824 54.3MB/s in 20s There is a significant decrease with Knox SSL for some reason. Kevin Risden On Sun, Sep 23, 2018 at 8:53 PM larry mccay mailto:lmc...@apache.org>> wrote: SSL handshake will likely happen at least twice. Once for the request through Knox to the NN then the redirect from the NN to the DN goes all the way back to the client. So they have to follow the redirect and do the handshake to the DN. On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden mailto:kris...@apache.org>> wrote: So I found this in the Knox issues list in JIRA: https://issues.apache.org/jira/browse/KNOX-1221 It sounds familiar in terms of a slowdown when going through Knox. Kevin Risden On Sat, Sep 15, 2018 at 10:17 PM Kevin Ris
Re: WebHDFS performance issue in Knox
@David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked. @Guang - NP just took some time to sit down and look at it. Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty. - https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions - https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements - https://webtide.com/conscrypting-native-ssl-for-jetty/ Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting ( https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further. Kevin Risden On Tue, Oct 9, 2018 at 2:01 PM Guang Yang wrote: > Thanks Kevin conducting such experiment! This is exactly what I saw > before. It doesn't look right the download speed is 10x slower when > enabling SSL. > > On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < > dvillarr...@hortonworks.com> wrote: > >> I bring this up because HDFS encryption saw an increase in performance. >> >> https://issues.apache.org/jira/browse/HDFS-6606 >> >> >> >> https://issues.apache.org/jira/browse/HADOOP-10768 >> >> >> >> Maybe Knox can make some enhancements in this area? >> >> >> >> *From: *David Villarreal >> *Date: *Tuesday, October 9, 2018 at 10:34 AM >> *To: *"user@knox.apache.org" >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> Hi Kevin, >> >> Now increase your CPU processing power and show me the numbers. >> >> >> >> Do we support AES-NI optimization with extended CPU instruction set for >> AES hardware acceleration? >> >> libcrypto.so library that supports hardware acceleration, such as >> OpenSSL 1.0.1e. (Many OS versions have an older version of the library that >> does not support AES-NI.) >> >> >> >> >> >> >> >> >> >> >> *From:Kevin Risden > Reply-To: >> "user@knox.apache.org " > > Date: Tuesday, October 9, 2018 at 10:26 AM To: >> "user@knox.apache.org " > > Subject: Re: WebHDFS performance issue in Knox* >> >> >> >> Writes look to have performance impact as well: >> >>- directly to webhdfs - ~2.6 seconds >>- knox no ssl - ~29 seconds >>- knox ssl - ~49.6 seconds >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden wrote: >> >> If I run two downloads concurrently: >> >> >> >> 1,073,741,824 46.1MB/s in 22s >> >> 1,073,741,824 51.3MB/s in 22s >> >> >> >> So it isn't a limitation of the Knox gateway itself in total bandwidth >> but a per connection limitation somehow. >> >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden wrote: >> >> So I was able to reproduce a slowdown with SSL with a pseudo distributed >> HDFS setup on a single node with Knox running on the same node. This was >> setup in Virtualbox on my laptop. >> >> >> >> Rough timings with wget for a 1GB random file: >> >>- directly to webhdfs - 1,073,741,824 252MB/s in 3.8s >>- knox no ssl - 1,073,741,824 264MB/s in 3.6s >>- knox ssl - 1,073,741,824 54.3MB/s in 20s >> >> There is a significant decrease with Knox SSL for some reason. >> >> >> >> Kevin Risden >> >> >> >> >> >> On Sun, Sep 23, 2018 at 8:53 PM larry mccay wrote: >> >> SSL handshake will likely happen at least twice. >> >> Once for the request through Knox to the NN then the redirect from the NN >> to the DN goes all the way back to the client. >> >> So they have to follow the redirect and do the handshake to the DN. >> >> >> >> >> >> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden wrote: >> >> So I found this in the Knox issues list in JIRA: >> >> >> >> https://issues.apache.org/jira/browse/KNOX-1221 >> >> >> >> It sounds familiar in terms of a slowdown when going through Knox. >> >> >> Kevin Risden >> >> >>
Re: WebHDFS performance issue in Knox
Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL. On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < dvillarr...@hortonworks.com> wrote: > I bring this up because HDFS encryption saw an increase in performance. > > https://issues.apache.org/jira/browse/HDFS-6606 > > > > https://issues.apache.org/jira/browse/HADOOP-10768 > > > > Maybe Knox can make some enhancements in this area? > > > > *From: *David Villarreal > *Date: *Tuesday, October 9, 2018 at 10:34 AM > *To: *"user@knox.apache.org" > *Subject: *Re: WebHDFS performance issue in Knox > > > > Hi Kevin, > > Now increase your CPU processing power and show me the numbers. > > > > Do we support AES-NI optimization with extended CPU instruction set for > AES hardware acceleration? > > libcrypto.so library that supports hardware acceleration, such as OpenSSL > 1.0.1e. (Many OS versions have an older version of the library that does > not support AES-NI.) > > > > > > *From: *Kevin Risden > *Reply-To: *"user@knox.apache.org" > *Date: *Tuesday, October 9, 2018 at 10:26 AM > *To: *"user@knox.apache.org" > *Subject: *Re: WebHDFS performance issue in Knox > > > > Writes look to have performance impact as well: > >- directly to webhdfs - ~2.6 seconds >- knox no ssl - ~29 seconds >- knox ssl - ~49.6 seconds > > Kevin Risden > > > > > > On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden wrote: > > If I run two downloads concurrently: > > > > 1,073,741,824 46.1MB/s in 22s > > 1,073,741,824 51.3MB/s in 22s > > > > So it isn't a limitation of the Knox gateway itself in total bandwidth but > a per connection limitation somehow. > > > Kevin Risden > > > > > > On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden wrote: > > So I was able to reproduce a slowdown with SSL with a pseudo distributed > HDFS setup on a single node with Knox running on the same node. This was > setup in Virtualbox on my laptop. > > > > Rough timings with wget for a 1GB random file: > >- directly to webhdfs - 1,073,741,824 252MB/s in 3.8s >- knox no ssl - 1,073,741,824 264MB/s in 3.6s >- knox ssl - 1,073,741,824 54.3MB/s in 20s > > There is a significant decrease with Knox SSL for some reason. > > > > Kevin Risden > > > > > > On Sun, Sep 23, 2018 at 8:53 PM larry mccay wrote: > > SSL handshake will likely happen at least twice. > > Once for the request through Knox to the NN then the redirect from the NN > to the DN goes all the way back to the client. > > So they have to follow the redirect and do the handshake to the DN. > > > > > > On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden wrote: > > So I found this in the Knox issues list in JIRA: > > > > https://issues.apache.org/jira/browse/KNOX-1221 > > > > It sounds familiar in terms of a slowdown when going through Knox. > > > Kevin Risden > > > > > > On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden wrote: > > Hmmm yea curl for a single file should do the handshake once. > > > > What are the system performance statistics during the SSL vs non SSL > testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here > if using that. Otherwise watching top may be helpful. It would be help to > determine if the Knox is working harder during the SSL transfer. > > > Kevin Risden > > > > > > On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: > > I'm just using curl to download a single large file. So I suspect SSL > handshake just happens once? > > > > On Tue, Sep 11, 2018 at 12:02 PM > > Kevin Risden > > wrote: > > What client are you using to connect Knox? Is this for a single file or a > bunch of files? > > > > The SSL handshake can be slow if the client doesn't keep the connection > open. > > Kevin Risden > > > > On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: > > Thanks Larry. But the only difference is this part in my gateway-site.xml. > > > > ** > > *ssl.enabled* > > *false* > > *Indicates whether SSL is enabled.* > > ** > > > > On Tue, Sep 11, 2018 at 11:42 AM, larry mccay wrote: > > I really don't think that kind of difference should be expected from > merely SSL overhead. > > I don't however have any metrics to contradict it either since I do not > run Knox without SSL. > > > > Given the above, I am struggling coming up with a
Re: WebHDFS performance issue in Knox
If I run two downloads concurrently: 1,073,741,824 46.1MB/s in 22s 1,073,741,824 51.3MB/s in 22s So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow. Kevin Risden On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden wrote: > So I was able to reproduce a slowdown with SSL with a pseudo distributed > HDFS setup on a single node with Knox running on the same node. This was > setup in Virtualbox on my laptop. > > Rough timings with wget for a 1GB random file: > >- directly to webhdfs - 1,073,741,824 252MB/s in 3.8s >- knox no ssl - 1,073,741,824 264MB/s in 3.6s >- knox ssl - 1,073,741,824 54.3MB/s in 20s > > There is a significant decrease with Knox SSL for some reason. > > Kevin Risden > > > On Sun, Sep 23, 2018 at 8:53 PM larry mccay wrote: > >> SSL handshake will likely happen at least twice. >> Once for the request through Knox to the NN then the redirect from the NN >> to the DN goes all the way back to the client. >> So they have to follow the redirect and do the handshake to the DN. >> >> >> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden wrote: >> >>> So I found this in the Knox issues list in JIRA: >>> >>> https://issues.apache.org/jira/browse/KNOX-1221 >>> >>> It sounds familiar in terms of a slowdown when going through Knox. >>> >>> Kevin Risden >>> >>> >>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden >>> wrote: >>> >>>> Hmmm yea curl for a single file should do the handshake once. >>>> >>>> What are the system performance statistics during the SSL vs non SSL >>>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here >>>> if using that. Otherwise watching top may be helpful. It would be help to >>>> determine if the Knox is working harder during the SSL transfer. >>>> >>>> Kevin Risden >>>> >>>> >>>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: >>>> >>>>> I'm just using curl to download a single large file. So I suspect SSL >>>>> handshake just happens once? >>>>> >>>>> On Tue, Sep 11, 2018 at 12:02 PM >>>>> Kevin Risden >>>>> wrote: >>>>> >>>>>> What client are you using to connect Knox? Is this for a single file >>>>>> or a bunch of files? >>>>>> >>>>>> The SSL handshake can be slow if the client doesn't keep the >>>>>> connection open. >>>>>> >>>>>> Kevin Risden >>>>>> >>>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >>>>>> >>>>>>> Thanks Larry. But the only difference is this part in my >>>>>>> gateway-site.xml. >>>>>>> >>>>>>> ** >>>>>>> *ssl.enabled* >>>>>>> *false* >>>>>>> *Indicates whether SSL is >>>>>>> enabled.* >>>>>>> ** >>>>>>> >>>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay >>>>>>> wrote: >>>>>>> >>>>>>>> I really don't think that kind of difference should be expected >>>>>>>> from merely SSL overhead. >>>>>>>> I don't however have any metrics to contradict it either since I do >>>>>>>> not run Knox without SSL. >>>>>>>> >>>>>>>> Given the above, I am struggling coming up with a meaningful >>>>>>>> response to this. :( >>>>>>>> I don't think you should see a 10 fold increase in speed by >>>>>>>> disabling SSL though. >>>>>>>> >>>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>>>>>> >>>>>>>>> Any idea guys? >>>>>>>>> >>>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>>>>>> >>>>>>>>>> Thanks guys! The issue seems exactly what David pointed out, >>>>>>>>>> which is because of encrypted over SSL. >>>>>>>>>> >>>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>>>>>>
Re: WebHDFS performance issue in Knox
So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop. Rough timings with wget for a 1GB random file: - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s - knox no ssl - 1,073,741,824 264MB/s in 3.6s - knox ssl - 1,073,741,824 54.3MB/s in 20s There is a significant decrease with Knox SSL for some reason. Kevin Risden On Sun, Sep 23, 2018 at 8:53 PM larry mccay wrote: > SSL handshake will likely happen at least twice. > Once for the request through Knox to the NN then the redirect from the NN > to the DN goes all the way back to the client. > So they have to follow the redirect and do the handshake to the DN. > > > On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden wrote: > >> So I found this in the Knox issues list in JIRA: >> >> https://issues.apache.org/jira/browse/KNOX-1221 >> >> It sounds familiar in terms of a slowdown when going through Knox. >> >> Kevin Risden >> >> >> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden wrote: >> >>> Hmmm yea curl for a single file should do the handshake once. >>> >>> What are the system performance statistics during the SSL vs non SSL >>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here >>> if using that. Otherwise watching top may be helpful. It would be help to >>> determine if the Knox is working harder during the SSL transfer. >>> >>> Kevin Risden >>> >>> >>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: >>> >>>> I'm just using curl to download a single large file. So I suspect SSL >>>> handshake just happens once? >>>> >>>> On Tue, Sep 11, 2018 at 12:02 PM >>>> Kevin Risden >>>> wrote: >>>> >>>>> What client are you using to connect Knox? Is this for a single file >>>>> or a bunch of files? >>>>> >>>>> The SSL handshake can be slow if the client doesn't keep the >>>>> connection open. >>>>> >>>>> Kevin Risden >>>>> >>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >>>>> >>>>>> Thanks Larry. But the only difference is this part in my >>>>>> gateway-site.xml. >>>>>> >>>>>> ** >>>>>> *ssl.enabled* >>>>>> *false* >>>>>> *Indicates whether SSL is enabled.* >>>>>> ** >>>>>> >>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay >>>>>> wrote: >>>>>> >>>>>>> I really don't think that kind of difference should be expected from >>>>>>> merely SSL overhead. >>>>>>> I don't however have any metrics to contradict it either since I do >>>>>>> not run Knox without SSL. >>>>>>> >>>>>>> Given the above, I am struggling coming up with a meaningful >>>>>>> response to this. :( >>>>>>> I don't think you should see a 10 fold increase in speed by >>>>>>> disabling SSL though. >>>>>>> >>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>>>>> >>>>>>>> Any idea guys? >>>>>>>> >>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>>>>> >>>>>>>>> Thanks guys! The issue seems exactly what David pointed out, which >>>>>>>>> is because of encrypted over SSL. >>>>>>>>> >>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>>>>> Namenode directly. And with disabling SSL, the speed can reach to >>>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but >>>>>>>>> it does surprised me with such a difference. Is it normal from your >>>>>>>>> perspective? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Guang >>>>>>>>> >>>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>>
Re: WebHDFS performance issue in Knox
SSL handshake will likely happen at least twice. Once for the request through Knox to the NN then the redirect from the NN to the DN goes all the way back to the client. So they have to follow the redirect and do the handshake to the DN. On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden wrote: > So I found this in the Knox issues list in JIRA: > > https://issues.apache.org/jira/browse/KNOX-1221 > > It sounds familiar in terms of a slowdown when going through Knox. > > Kevin Risden > > > On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden wrote: > >> Hmmm yea curl for a single file should do the handshake once. >> >> What are the system performance statistics during the SSL vs non SSL >> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here >> if using that. Otherwise watching top may be helpful. It would be help to >> determine if the Knox is working harder during the SSL transfer. >> >> Kevin Risden >> >> >> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: >> >>> I'm just using curl to download a single large file. So I suspect SSL >>> handshake just happens once? >>> >>> On Tue, Sep 11, 2018 at 12:02 PM >>> Kevin Risden >>> wrote: >>> >>>> What client are you using to connect Knox? Is this for a single file or >>>> a bunch of files? >>>> >>>> The SSL handshake can be slow if the client doesn't keep the connection >>>> open. >>>> >>>> Kevin Risden >>>> >>>> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >>>> >>>>> Thanks Larry. But the only difference is this part in my >>>>> gateway-site.xml. >>>>> >>>>> ** >>>>> *ssl.enabled* >>>>> *false* >>>>> *Indicates whether SSL is enabled.* >>>>> ** >>>>> >>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay >>>>> wrote: >>>>> >>>>>> I really don't think that kind of difference should be expected from >>>>>> merely SSL overhead. >>>>>> I don't however have any metrics to contradict it either since I do >>>>>> not run Knox without SSL. >>>>>> >>>>>> Given the above, I am struggling coming up with a meaningful response >>>>>> to this. :( >>>>>> I don't think you should see a 10 fold increase in speed by disabling >>>>>> SSL though. >>>>>> >>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>>>> >>>>>>> Any idea guys? >>>>>>> >>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>>>> >>>>>>>> Thanks guys! The issue seems exactly what David pointed out, which >>>>>>>> is because of encrypted over SSL. >>>>>>>> >>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>>>> Namenode directly. And with disabling SSL, the speed can reach to >>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but >>>>>>>> it does surprised me with such a difference. Is it normal from your >>>>>>>> perspective? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Guang >>>>>>>> >>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>>>>>> dvillarr...@hortonworks.com> wrote: >>>>>>>> >>>>>>>>> Hi Guang, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Keep in mind the data is being encrypted over SSL. If you disable >>>>>>>>> SSL you will most likely see a very significant boost in throughput. >>>>>>>>> Some >>>>>>>>> people have used more powerful computers to make encryption quicker. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> David
Re: WebHDFS performance issue in Knox
So I found this in the Knox issues list in JIRA: https://issues.apache.org/jira/browse/KNOX-1221 It sounds familiar in terms of a slowdown when going through Knox. Kevin Risden On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden wrote: > Hmmm yea curl for a single file should do the handshake once. > > What are the system performance statistics during the SSL vs non SSL > testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here > if using that. Otherwise watching top may be helpful. It would be help to > determine if the Knox is working harder during the SSL transfer. > > Kevin Risden > > > On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: > >> I'm just using curl to download a single large file. So I suspect SSL >> handshake just happens once? >> >> On Tue, Sep 11, 2018 at 12:02 PM >> Kevin Risden >> wrote: >> >>> What client are you using to connect Knox? Is this for a single file or >>> a bunch of files? >>> >>> The SSL handshake can be slow if the client doesn't keep the connection >>> open. >>> >>> Kevin Risden >>> >>> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >>> >>>> Thanks Larry. But the only difference is this part in my >>>> gateway-site.xml. >>>> >>>> ** >>>> *ssl.enabled* >>>> *false* >>>> *Indicates whether SSL is enabled.* >>>> ** >>>> >>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay >>>> wrote: >>>> >>>>> I really don't think that kind of difference should be expected from >>>>> merely SSL overhead. >>>>> I don't however have any metrics to contradict it either since I do >>>>> not run Knox without SSL. >>>>> >>>>> Given the above, I am struggling coming up with a meaningful response >>>>> to this. :( >>>>> I don't think you should see a 10 fold increase in speed by disabling >>>>> SSL though. >>>>> >>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>>> >>>>>> Any idea guys? >>>>>> >>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>>> >>>>>>> Thanks guys! The issue seems exactly what David pointed out, which >>>>>>> is because of encrypted over SSL. >>>>>>> >>>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>>> Namenode directly. And with disabling SSL, the speed can reach to >>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but it >>>>>>> does surprised me with such a difference. Is it normal from your >>>>>>> perspective? >>>>>>> >>>>>>> Thanks, >>>>>>> Guang >>>>>>> >>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>>>>> dvillarr...@hortonworks.com> wrote: >>>>>>> >>>>>>>> Hi Guang, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Keep in mind the data is being encrypted over SSL. If you disable >>>>>>>> SSL you will most likely see a very significant boost in throughput. >>>>>>>> Some >>>>>>>> people have used more powerful computers to make encryption quicker. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> David >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *From: *Sean Roberts >>>>>>>> *Reply-To: *"user@knox.apache.org" >>>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>>>>>>> *To: *"user@knox.apache.org" >>>>>>>> *Subject: *Re: WebHDFS performance issue in Knox >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Guang – This is somewhat to be expected. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> When you talk to WebHDFS directly, the client can distribute the >>>>>>>> request across many data nodes. Also, you are getting data directly >>>>>>>> from >>>>>>>> the source. >>>>>>>> >>>>>>>> With Knox, all traffic goes through the single Knox host. Knox is >>>>>>>> responsible for fetching from the datanodes and consolidating to send >>>>>>>> to >>>>>>>> you. This means overhead as it’s acting as a middle man, and lower >>>>>>>> network >>>>>>>> capacity since only 1 host is serving data to you. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Also, if running on a cloud provider, the Knox host may be a >>>>>>>> smaller instance size with lower network capacity. >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Sean Roberts >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *From: *Guang Yang >>>>>>>> *Reply-To: *"user@knox.apache.org" >>>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46 >>>>>>>> *To: *"user@knox.apache.org" >>>>>>>> *Subject: *WebHDFS performance issue in Knox >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a >>>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s. >>>>>>>> However, if we download directly from datanode, the speed is about >>>>>>>> 40M/s at >>>>>>>> least. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Are you guys aware of this problem? Any suggestion? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Guang >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>
Re: WebHDFS performance issue in Knox
Hmmm yea curl for a single file should do the handshake once. What are the system performance statistics during the SSL vs non SSL testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here if using that. Otherwise watching top may be helpful. It would be help to determine if the Knox is working harder during the SSL transfer. Kevin Risden On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: > I'm just using curl to download a single large file. So I suspect SSL > handshake just happens once? > > On Tue, Sep 11, 2018 at 12:02 PM > Kevin Risden > wrote: > >> What client are you using to connect Knox? Is this for a single file or a >> bunch of files? >> >> The SSL handshake can be slow if the client doesn't keep the connection >> open. >> >> Kevin Risden >> >> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >> >>> Thanks Larry. But the only difference is this part in my >>> gateway-site.xml. >>> >>> ** >>> *ssl.enabled* >>> *false* >>> *Indicates whether SSL is enabled.* >>> ** >>> >>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay wrote: >>> >>>> I really don't think that kind of difference should be expected from >>>> merely SSL overhead. >>>> I don't however have any metrics to contradict it either since I do not >>>> run Knox without SSL. >>>> >>>> Given the above, I am struggling coming up with a meaningful response >>>> to this. :( >>>> I don't think you should see a 10 fold increase in speed by disabling >>>> SSL though. >>>> >>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>> >>>>> Any idea guys? >>>>> >>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>> >>>>>> Thanks guys! The issue seems exactly what David pointed out, which is >>>>>> because of encrypted over SSL. >>>>>> >>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>> Namenode directly. And with disabling SSL, the speed can reach to >>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>>>> significantly to *~40M/s*. I know it's because of encrypted, but it >>>>>> does surprised me with such a difference. Is it normal from your >>>>>> perspective? >>>>>> >>>>>> Thanks, >>>>>> Guang >>>>>> >>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>>>> dvillarr...@hortonworks.com> wrote: >>>>>> >>>>>>> Hi Guang, >>>>>>> >>>>>>> >>>>>>> >>>>>>> Keep in mind the data is being encrypted over SSL. If you disable >>>>>>> SSL you will most likely see a very significant boost in throughput. >>>>>>> Some >>>>>>> people have used more powerful computers to make encryption quicker. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> >>>>>>> >>>>>>> David >>>>>>> >>>>>>> >>>>>>> >>>>>>> *From: *Sean Roberts >>>>>>> *Reply-To: *"user@knox.apache.org" >>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>>>>>> *To: *"user@knox.apache.org" >>>>>>> *Subject: *Re: WebHDFS performance issue in Knox >>>>>>> >>>>>>> >>>>>>> >>>>>>> Guang – This is somewhat to be expected. >>>>>>> >>>>>>> >>>>>>> >>>>>>> When you talk to WebHDFS directly, the client can distribute the >>>>>>> request across many data nodes. Also, you are getting data directly from >>>>>>> the source. >>>>>>> >>>>>>> With Knox, all traffic goes through the single Knox host. Knox is >>>>>>> responsible for fetching from the datanodes and consolidating to send to >>>>>>> you. This means overhead as it’s acting as a middle man, and lower >>>>>>> network >>>>>>> capacity since only 1 host is serving data to you. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Also, if running on a cloud provider, the Knox host may be a smaller >>>>>>> instance size with lower network capacity. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Sean Roberts >>>>>>> >>>>>>> >>>>>>> >>>>>>> *From: *Guang Yang >>>>>>> *Reply-To: *"user@knox.apache.org" >>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46 >>>>>>> *To: *"user@knox.apache.org" >>>>>>> *Subject: *WebHDFS performance issue in Knox >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a >>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s. >>>>>>> However, if we download directly from datanode, the speed is about >>>>>>> 40M/s at >>>>>>> least. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Are you guys aware of this problem? Any suggestion? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Guang >>>>>>> >>>>>> >>>>>> >>>>> >>>
Re: WebHDFS performance issue in Knox
I'm just using curl to download a single large file. So I suspect SSL handshake just happens once? On Tue, Sep 11, 2018 at 12:02 PM Kevin Risden wrote: > What client are you using to connect Knox? Is this for a single file or a > bunch of files? > > The SSL handshake can be slow if the client doesn't keep the connection > open. > > Kevin Risden > > On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: > >> Thanks Larry. But the only difference is this part in my gateway-site.xml. >> >> ** >> *ssl.enabled* >> *false* >> *Indicates whether SSL is enabled.* >> ** >> >> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay wrote: >> >>> I really don't think that kind of difference should be expected from >>> merely SSL overhead. >>> I don't however have any metrics to contradict it either since I do not >>> run Knox without SSL. >>> >>> Given the above, I am struggling coming up with a meaningful response to >>> this. :( >>> I don't think you should see a 10 fold increase in speed by disabling >>> SSL though. >>> >>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>> >>>> Any idea guys? >>>> >>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>> >>>>> Thanks guys! The issue seems exactly what David pointed out, which is >>>>> because of encrypted over SSL. >>>>> >>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>> Namenode directly. And with disabling SSL, the speed can reach to >>>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>>> significantly to *~40M/s*. I know it's because of encrypted, but it >>>>> does surprised me with such a difference. Is it normal from your >>>>> perspective? >>>>> >>>>> Thanks, >>>>> Guang >>>>> >>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>>> dvillarr...@hortonworks.com> wrote: >>>>> >>>>>> Hi Guang, >>>>>> >>>>>> >>>>>> >>>>>> Keep in mind the data is being encrypted over SSL. If you disable >>>>>> SSL you will most likely see a very significant boost in throughput. >>>>>> Some >>>>>> people have used more powerful computers to make encryption quicker. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> >>>>>> >>>>>> David >>>>>> >>>>>> >>>>>> >>>>>> *From: *Sean Roberts >>>>>> *Reply-To: *"user@knox.apache.org" >>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>>>>> *To: *"user@knox.apache.org" >>>>>> *Subject: *Re: WebHDFS performance issue in Knox >>>>>> >>>>>> >>>>>> >>>>>> Guang – This is somewhat to be expected. >>>>>> >>>>>> >>>>>> >>>>>> When you talk to WebHDFS directly, the client can distribute the >>>>>> request across many data nodes. Also, you are getting data directly from >>>>>> the source. >>>>>> >>>>>> With Knox, all traffic goes through the single Knox host. Knox is >>>>>> responsible for fetching from the datanodes and consolidating to send to >>>>>> you. This means overhead as it’s acting as a middle man, and lower >>>>>> network >>>>>> capacity since only 1 host is serving data to you. >>>>>> >>>>>> >>>>>> >>>>>> Also, if running on a cloud provider, the Knox host may be a smaller >>>>>> instance size with lower network capacity. >>>>>> >>>>>> -- >>>>>> >>>>>> Sean Roberts >>>>>> >>>>>> >>>>>> >>>>>> *From: *Guang Yang >>>>>> *Reply-To: *"user@knox.apache.org" >>>>>> *Date: *Tuesday, 4 September 2018 at 07:46 >>>>>> *To: *"user@knox.apache.org" >>>>>> *Subject: *WebHDFS performance issue in Knox >>>>>> >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a >>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s. >>>>>> However, if we download directly from datanode, the speed is about 40M/s >>>>>> at >>>>>> least. >>>>>> >>>>>> >>>>>> >>>>>> Are you guys aware of this problem? Any suggestion? >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Guang >>>>>> >>>>> >>>>> >>>> >>
Re: WebHDFS performance issue in Knox
What client are you using to connect Knox? Is this for a single file or a bunch of files? The SSL handshake can be slow if the client doesn't keep the connection open. Kevin Risden On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: > Thanks Larry. But the only difference is this part in my gateway-site.xml. > > ** > *ssl.enabled* > *false* > *Indicates whether SSL is enabled.* > ** > > On Tue, Sep 11, 2018 at 11:42 AM, larry mccay wrote: > >> I really don't think that kind of difference should be expected from >> merely SSL overhead. >> I don't however have any metrics to contradict it either since I do not >> run Knox without SSL. >> >> Given the above, I am struggling coming up with a meaningful response to >> this. :( >> I don't think you should see a 10 fold increase in speed by disabling SSL >> though. >> >> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >> >>> Any idea guys? >>> >>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>> >>>> Thanks guys! The issue seems exactly what David pointed out, which is >>>> because of encrypted over SSL. >>>> >>>> Without Knox, the download speed can reach to *400M/s* if I call >>>> Namenode directly. And with disabling SSL, the speed can reach to >>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>> significantly to *~40M/s*. I know it's because of encrypted, but it >>>> does surprised me with such a difference. Is it normal from your >>>> perspective? >>>> >>>> Thanks, >>>> Guang >>>> >>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>> dvillarr...@hortonworks.com> wrote: >>>> >>>>> Hi Guang, >>>>> >>>>> >>>>> >>>>> Keep in mind the data is being encrypted over SSL. If you disable SSL >>>>> you will most likely see a very significant boost in throughput. Some >>>>> people have used more powerful computers to make encryption quicker. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> >>>>> David >>>>> >>>>> >>>>> >>>>> *From: *Sean Roberts >>>>> *Reply-To: *"user@knox.apache.org" >>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>>>> *To: *"user@knox.apache.org" >>>>> *Subject: *Re: WebHDFS performance issue in Knox >>>>> >>>>> >>>>> >>>>> Guang – This is somewhat to be expected. >>>>> >>>>> >>>>> >>>>> When you talk to WebHDFS directly, the client can distribute the >>>>> request across many data nodes. Also, you are getting data directly from >>>>> the source. >>>>> >>>>> With Knox, all traffic goes through the single Knox host. Knox is >>>>> responsible for fetching from the datanodes and consolidating to send to >>>>> you. This means overhead as it’s acting as a middle man, and lower network >>>>> capacity since only 1 host is serving data to you. >>>>> >>>>> >>>>> >>>>> Also, if running on a cloud provider, the Knox host may be a smaller >>>>> instance size with lower network capacity. >>>>> >>>>> -- >>>>> >>>>> Sean Roberts >>>>> >>>>> >>>>> >>>>> *From: *Guang Yang >>>>> *Reply-To: *"user@knox.apache.org" >>>>> *Date: *Tuesday, 4 September 2018 at 07:46 >>>>> *To: *"user@knox.apache.org" >>>>> *Subject: *WebHDFS performance issue in Knox >>>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file >>>>> through WebHDFS in Knox, the download speed is just about 11M/s. However, >>>>> if we download directly from datanode, the speed is about 40M/s at least. >>>>> >>>>> >>>>> >>>>> Are you guys aware of this problem? Any suggestion? >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Guang >>>>> >>>> >>>> >>> >
Re: WebHDFS performance issue in Knox
Thanks Larry. But the only difference is this part in my gateway-site.xml. ** *ssl.enabled* *false* *Indicates whether SSL is enabled.* ** On Tue, Sep 11, 2018 at 11:42 AM, larry mccay wrote: > I really don't think that kind of difference should be expected from > merely SSL overhead. > I don't however have any metrics to contradict it either since I do not > run Knox without SSL. > > Given the above, I am struggling coming up with a meaningful response to > this. :( > I don't think you should see a 10 fold increase in speed by disabling SSL > though. > > On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: > >> Any idea guys? >> >> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >> >>> Thanks guys! The issue seems exactly what David pointed out, which is >>> because of encrypted over SSL. >>> >>> Without Knox, the download speed can reach to *400M/s* if I call >>> Namenode directly. And with disabling SSL, the speed can reach to >>> *~400M/s* as well through Knox. But with SSL, the speed drops >>> significantly to *~40M/s*. I know it's because of encrypted, but it >>> does surprised me with such a difference. Is it normal from your >>> perspective? >>> >>> Thanks, >>> Guang >>> >>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>> dvillarr...@hortonworks.com> wrote: >>> >>>> Hi Guang, >>>> >>>> >>>> >>>> Keep in mind the data is being encrypted over SSL. If you disable SSL >>>> you will most likely see a very significant boost in throughput. Some >>>> people have used more powerful computers to make encryption quicker. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> David >>>> >>>> >>>> >>>> *From: *Sean Roberts >>>> *Reply-To: *"user@knox.apache.org" >>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>>> *To: *"user@knox.apache.org" >>>> *Subject: *Re: WebHDFS performance issue in Knox >>>> >>>> >>>> >>>> Guang – This is somewhat to be expected. >>>> >>>> >>>> >>>> When you talk to WebHDFS directly, the client can distribute the >>>> request across many data nodes. Also, you are getting data directly from >>>> the source. >>>> >>>> With Knox, all traffic goes through the single Knox host. Knox is >>>> responsible for fetching from the datanodes and consolidating to send to >>>> you. This means overhead as it’s acting as a middle man, and lower network >>>> capacity since only 1 host is serving data to you. >>>> >>>> >>>> >>>> Also, if running on a cloud provider, the Knox host may be a smaller >>>> instance size with lower network capacity. >>>> >>>> -- >>>> >>>> Sean Roberts >>>> >>>> >>>> >>>> *From: *Guang Yang >>>> *Reply-To: *"user@knox.apache.org" >>>> *Date: *Tuesday, 4 September 2018 at 07:46 >>>> *To: *"user@knox.apache.org" >>>> *Subject: *WebHDFS performance issue in Knox >>>> >>>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file >>>> through WebHDFS in Knox, the download speed is just about 11M/s. However, >>>> if we download directly from datanode, the speed is about 40M/s at least. >>>> >>>> >>>> >>>> Are you guys aware of this problem? Any suggestion? >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Guang >>>> >>> >>> >>
Re: WebHDFS performance issue in Knox
I really don't think that kind of difference should be expected from merely SSL overhead. I don't however have any metrics to contradict it either since I do not run Knox without SSL. Given the above, I am struggling coming up with a meaningful response to this. :( I don't think you should see a 10 fold increase in speed by disabling SSL though. On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: > Any idea guys? > > On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: > >> Thanks guys! The issue seems exactly what David pointed out, which is >> because of encrypted over SSL. >> >> Without Knox, the download speed can reach to *400M/s* if I call >> Namenode directly. And with disabling SSL, the speed can reach to >> *~400M/s* as well through Knox. But with SSL, the speed drops >> significantly to *~40M/s*. I know it's because of encrypted, but it does >> surprised me with such a difference. Is it normal from your perspective? >> >> Thanks, >> Guang >> >> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >>> Hi Guang, >>> >>> >>> >>> Keep in mind the data is being encrypted over SSL. If you disable SSL >>> you will most likely see a very significant boost in throughput. Some >>> people have used more powerful computers to make encryption quicker. >>> >>> >>> >>> Thanks, >>> >>> >>> >>> David >>> >>> >>> >>> *From: *Sean Roberts >>> *Reply-To: *"user@knox.apache.org" >>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>> *To: *"user@knox.apache.org" >>> *Subject: *Re: WebHDFS performance issue in Knox >>> >>> >>> >>> Guang – This is somewhat to be expected. >>> >>> >>> >>> When you talk to WebHDFS directly, the client can distribute the request >>> across many data nodes. Also, you are getting data directly from the source. >>> >>> With Knox, all traffic goes through the single Knox host. Knox is >>> responsible for fetching from the datanodes and consolidating to send to >>> you. This means overhead as it’s acting as a middle man, and lower network >>> capacity since only 1 host is serving data to you. >>> >>> >>> >>> Also, if running on a cloud provider, the Knox host may be a smaller >>> instance size with lower network capacity. >>> >>> -- >>> >>> Sean Roberts >>> >>> >>> >>> *From: *Guang Yang >>> *Reply-To: *"user@knox.apache.org" >>> *Date: *Tuesday, 4 September 2018 at 07:46 >>> *To: *"user@knox.apache.org" >>> *Subject: *WebHDFS performance issue in Knox >>> >>> >>> >>> Hi, >>> >>> >>> >>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file >>> through WebHDFS in Knox, the download speed is just about 11M/s. However, >>> if we download directly from datanode, the speed is about 40M/s at least. >>> >>> >>> >>> Are you guys aware of this problem? Any suggestion? >>> >>> >>> >>> Thanks, >>> >>> Guang >>> >> >> >
Re: WebHDFS performance issue in Knox
Thanks guys! The issue seems exactly what David pointed out, which is because of encrypted over SSL. Without Knox, the download speed can reach to *400M/s* if I call Namenode directly. And with disabling SSL, the speed can reach to *~400M/s* as well through Knox. But with SSL, the speed drops significantly to *~40M/s*. I know it's because of encrypted, but it does surprised me with such a difference. Is it normal from your perspective? Thanks, Guang On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < dvillarr...@hortonworks.com> wrote: > Hi Guang, > > > > Keep in mind the data is being encrypted over SSL. If you disable SSL you > will most likely see a very significant boost in throughput. Some people > have used more powerful computers to make encryption quicker. > > > > Thanks, > > > > David > > > > *From: *Sean Roberts > *Reply-To: *"user@knox.apache.org" > *Date: *Tuesday, September 4, 2018 at 1:53 AM > *To: *"user@knox.apache.org" > *Subject: *Re: WebHDFS performance issue in Knox > > > > Guang – This is somewhat to be expected. > > > > When you talk to WebHDFS directly, the client can distribute the request > across many data nodes. Also, you are getting data directly from the source. > > With Knox, all traffic goes through the single Knox host. Knox is > responsible for fetching from the datanodes and consolidating to send to > you. This means overhead as it’s acting as a middle man, and lower network > capacity since only 1 host is serving data to you. > > > > Also, if running on a cloud provider, the Knox host may be a smaller > instance size with lower network capacity. > > -- > > Sean Roberts > > > > *From: *Guang Yang > *Reply-To: *"user@knox.apache.org" > *Date: *Tuesday, 4 September 2018 at 07:46 > *To: *"user@knox.apache.org" > *Subject: *WebHDFS performance issue in Knox > > > > Hi, > > > > We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file > through WebHDFS in Knox, the download speed is just about 11M/s. However, > if we download directly from datanode, the speed is about 40M/s at least. > > > > Are you guys aware of this problem? Any suggestion? > > > > Thanks, > > Guang >
Re: WebHDFS performance issue in Knox
Hi Guang, Keep in mind the data is being encrypted over SSL. If you disable SSL you will most likely see a very significant boost in throughput. Some people have used more powerful computers to make encryption quicker. Thanks, David From: Sean Roberts Reply-To: "user@knox.apache.org" Date: Tuesday, September 4, 2018 at 1:53 AM To: "user@knox.apache.org" Subject: Re: WebHDFS performance issue in Knox Guang – This is somewhat to be expected. When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source. With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you. Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity. -- Sean Roberts From: Guang Yang Reply-To: "user@knox.apache.org" Date: Tuesday, 4 September 2018 at 07:46 To: "user@knox.apache.org" Subject: WebHDFS performance issue in Knox Hi, We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least. Are you guys aware of this problem? Any suggestion? Thanks, Guang
Re: WebHDFS performance issue in Knox
Guang – This is somewhat to be expected. When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source. With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you. Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity. -- Sean Roberts From: Guang Yang Reply-To: "user@knox.apache.org" Date: Tuesday, 4 September 2018 at 07:46 To: "user@knox.apache.org" Subject: WebHDFS performance issue in Knox Hi, We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least. Are you guys aware of this problem? Any suggestion? Thanks, Guang
WebHDFS performance issue in Knox
Hi, We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least. Are you guys aware of this problem? Any suggestion? Thanks, Guang