Re: WebHDFS performance issue in Knox

2018-10-09 Thread Sandeep Moré 
I think this would be a good test, worth a try, not sure how we can force a
certain cipher to be used perhaps a permutation combination of
ssl.include.ciphers, ssl.exclude.ciphers.

Best,
Sandeep


On Tue, Oct 9, 2018 at 5:29 PM David Villarreal 
wrote:

> Hi Kevin,
>
>
>
> In my humble opinion, this has to do with cpu processing encryption in
> general based on which cipher being used.  Couldn’t the same type of
> principals/improvements (hdfs encryption improvements) be done here for
> let’s say for AES cipher suites?  If the main bottleneck here is CPU
> couldn’t you enhance encryption though hardware acceleration and you may
> see better performance numbers?
>
>
>
> https://calomel.org/aesni_ssl_performance.html
>
>
>
> Try forcing a less secure cipher to be used in your environment.  Do you
> then see better numbers?
>
>
>
> dav
>
>
>
>
>
> *From: *Kevin Risden 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Tuesday, October 9, 2018 at 1:05 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> @David - Not sure what you mean since this is SSL/TLS and not related to
> RPC encryption like the two JIRAs that you linked.
>
> @Guang - NP just took some time to sit down and look at it.
>
>
>
> Some preliminary investigation shows this may be the JDK implementation of
> TLS/SSL that is slowing down the read path. I need to dig into it further
> but found a few references showing that Java slowness for TLS/SSL affects
> Jetty.
>
>-
>https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
>-
>https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
>- https://webtide.com/conscrypting-native-ssl-for-jetty/
>
> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
> enable conscrypting (
> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
> With that I was able to get read performance on par with non ssl and native
> webhdfs. The write side of the equation still has some performance
> differences that need to be looked at further.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang  wrote:
>
> Thanks Kevin conducting such experiment! This is exactly what I saw
> before. It doesn't look right the download speed is 10x slower when
> enabling SSL.
>
>
>
> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
> dvillarr...@hortonworks.com> wrote:
>
> I bring this up because HDFS encryption saw an increase in performance.
>
> https://issues.apache.org/jira/browse/HDFS-6606
>
>
>
> https://issues.apache.org/jira/browse/HADOOP-10768
>
>
>
> Maybe Knox can make some enhancements in this area?
>
>
>
> *From: *David Villarreal 
> *Date: *Tuesday, October 9, 2018 at 10:34 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Hi Kevin,
>
> Now increase your CPU processing power and show me the numbers.
>
>
>
> Do we support AES-NI optimization with extended CPU instruction set for
> AES hardware acceleration?
>
> libcrypto.so library that supports hardware acceleration, such as OpenSSL
> 1.0.1e. (Many OS versions have an older version of the library that does
> not support AES-NI.)
>
>
>
>
>
>
> *From: *
>
> *Kevin Risden*
>
>
>
>
>
> *> Reply-To: "user@knox.apache.org
> " > Date:
> Tuesday, October 9, 2018 at 10:26 AM To: "user@knox.apache.org
> " >
> Subject: Re: WebHDFS performance issue in Knox*
>
>
>
> Writes look to have performance impact as well:
>
>- directly to webhdfs - ~2.6 seconds
>- knox no ssl - ~29 seconds
>- knox ssl - ~49.6 seconds
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden  wrote:
>
> If I run two downloads concurrently:
>
>
>
> 1,073,741,824 46.1MB/s   in 22s
>
> 1,073,741,824 51.3MB/s   in 22s
>
>
>
> So it isn't a limitation of the Knox gateway itself in total bandwidth but
> a per connection limitation somehow.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden  wrote:
>
> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
>
>
> Rough timings with wget for a 1GB random file:
>
>- directly to webhdfs - 1,073,741,824  252MB/s   in 3.8s
>- knox no ssl - 1,073,741,824  264MB/s   in 3.6s
>- knox ssl - 1,073,741,824 54.3MB/s   in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
>
>
> Kevin Risden
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay  wrote:
>
> SSL handshake will likely happen at least twice.
>
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
>
> So they have to follow the redirect and do the handshake to the DN.
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden  wrote:
>
> So I found this in the Knox issues list in JIRA:
>
>
>
>

Re: Spark History UI Error WARN HttpParser: Header is too large >8192

2018-10-09 Thread David Villarreal 
Hi Theyaa,

Change the size of gateway.httpserver.requestHeaderBuffer property.  I think 
the default is 8912  (8k) change to 16384. See if that helps.

For the second problem Request is a replay (34))] this message is often seen 
when the timing of one of the servers is off.  Make sure you use NTPD on all 
servers and they are all in sync.  If everything is in sync you can work around 
this issue by turning off krb5 replay cache. With the following parameter
-Dsun.security.krb5.rcache=none

dav


On 10/9/18, 9:01 AM, "Theyaa Matti"  wrote:

Hi,
   I am getting this error message "WARN HttpParser: Header is too large
>8192" when trying to access the spark history ui through knox. Any idea
please?

Also when trying to load the executors page, I get : GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism 
level:
Request is a replay (34))]

when knox is requesting executorspage-template.html

appreciate any help here.

Re: WebHDFS performance issue in Knox

2018-10-09 Thread David Villarreal 
Hi Kevin,

In my humble opinion, this has to do with cpu processing encryption in general 
based on which cipher being used.  Couldn’t the same type of 
principals/improvements (hdfs encryption improvements) be done here for let’s 
say for AES cipher suites?  If the main bottleneck here is CPU couldn’t you 
enhance encryption though hardware acceleration and you may see better 
performance numbers?

https://calomel.org/aesni_ssl_performance.html

Try forcing a less secure cipher to be used in your environment.  Do you then 
see better numbers?

dav


From: Kevin Risden 
Reply-To: "user@knox.apache.org" 
Date: Tuesday, October 9, 2018 at 1:05 PM
To: "user@knox.apache.org" 
Subject: Re: WebHDFS performance issue in Knox

@David - Not sure what you mean since this is SSL/TLS and not related to RPC 
encryption like the two JIRAs that you linked.
@Guang - NP just took some time to sit down and look at it.

Some preliminary investigation shows this may be the JDK implementation of 
TLS/SSL that is slowing down the read path. I need to dig into it further but 
found a few references showing that Java slowness for TLS/SSL affects Jetty.

  *   https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
  *   https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
  *   https://webtide.com/conscrypting-native-ssl-for-jetty/
Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable 
conscrypting 
(https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
 With that I was able to get read performance on par with non ssl and native 
webhdfs. The write side of the equation still has some performance differences 
that need to be looked at further.

Kevin Risden


On Tue, Oct 9, 2018 at 2:01 PM Guang Yang mailto:k...@uber.com>> 
wrote:
Thanks Kevin conducting such experiment! This is exactly what I saw before. It 
doesn't look right the download speed is 10x slower when enabling SSL.

On Tue, Oct 9, 2018 at 10:40 AM David Villarreal 
mailto:dvillarr...@hortonworks.com>> wrote:
I bring this up because HDFS encryption saw an increase in performance.
https://issues.apache.org/jira/browse/HDFS-6606

https://issues.apache.org/jira/browse/HADOOP-10768

Maybe Knox can make some enhancements in this area?

From: David Villarreal 
mailto:dvillarr...@hortonworks.com>>
Date: Tuesday, October 9, 2018 at 10:34 AM
To: "user@knox.apache.org" 
mailto:user@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox

Hi Kevin,
Now increase your CPU processing power and show me the numbers.

Do we support AES-NI optimization with extended CPU instruction set for AES 
hardware acceleration?
libcrypto.so library that supports hardware acceleration, such as OpenSSL 
1.0.1e. (Many OS versions have an older version of the library that does not 
support AES-NI.)


From:
Kevin Risden
mailto:kris...@apache.org>>
Reply-To: "user@knox.apache.org" 
mailto:user@knox.apache.org>>
Date: Tuesday, October 9, 2018 at 10:26 AM
To: "user@knox.apache.org" 
mailto:user@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox

Writes look to have performance impact as well:

  *   directly to webhdfs - ~2.6 seconds
  *   knox no ssl - ~29 seconds
  *   knox ssl - ~49.6 seconds
Kevin Risden


On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden 
mailto:kris...@apache.org>> wrote:
If I run two downloads concurrently:

1,073,741,824 46.1MB/s   in 22s
1,073,741,824 51.3MB/s   in 22s

So it isn't a limitation of the Knox gateway itself in total bandwidth but a 
per connection limitation somehow.

Kevin Risden


On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden 
mailto:kris...@apache.org>> wrote:
So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS 
setup on a single node with Knox running on the same node. This was setup in 
Virtualbox on my laptop.

Rough timings with wget for a 1GB random file:

  *   directly to webhdfs - 1,073,741,824  252MB/s   in 3.8s
  *   knox no ssl - 1,073,741,824  264MB/s   in 3.6s
  *   knox ssl - 1,073,741,824 54.3MB/s   in 20s
There is a significant decrease with Knox SSL for some reason.

Kevin Risden


On Sun, Sep 23, 2018 at 8:53 PM larry mccay 
mailto:lmc...@apache.org>> wrote:
SSL handshake will likely happen at least twice.
Once for the request through Knox to the NN then the redirect from the NN to 
the DN goes all the way back to the client.
So they have to follow the redirect and do the handshake to the DN.


On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden 
mailto:kris...@apache.org>> wrote:
So I found this in the Knox issues list in JIRA:

https://issues.apache.org/jira/browse/KNOX-1221

It sounds familiar in terms of a slowdown when going through Knox.

Kevin Risden


On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden 
mailto:kris...@apache.org>> wrote:
Hmmm yea curl for a single file should do the handshake once.

What are the system performance

Re: WebHDFS performance issue in Knox

2018-10-09 Thread Kevin Risden 
@David - Not sure what you mean since this is SSL/TLS and not related to
RPC encryption like the two JIRAs that you linked.
@Guang - NP just took some time to sit down and look at it.

Some preliminary investigation shows this may be the JDK implementation of
TLS/SSL that is slowing down the read path. I need to dig into it further
but found a few references showing that Java slowness for TLS/SSL affects
Jetty.

   - https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
   -
   https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
   - https://webtide.com/conscrypting-native-ssl-for-jetty/

Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
enable conscrypting (
https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
With that I was able to get read performance on par with non ssl and native
webhdfs. The write side of the equation still has some performance
differences that need to be looked at further.

Kevin Risden


On Tue, Oct 9, 2018 at 2:01 PM Guang Yang  wrote:

> Thanks Kevin conducting such experiment! This is exactly what I saw
> before. It doesn't look right the download speed is 10x slower when
> enabling SSL.
>
> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
> dvillarr...@hortonworks.com> wrote:
>
>> I bring this up because HDFS encryption saw an increase in performance.
>>
>> https://issues.apache.org/jira/browse/HDFS-6606
>>
>>
>>
>> https://issues.apache.org/jira/browse/HADOOP-10768
>>
>>
>>
>> Maybe Knox can make some enhancements in this area?
>>
>>
>>
>> *From: *David Villarreal 
>> *Date: *Tuesday, October 9, 2018 at 10:34 AM
>> *To: *"user@knox.apache.org" 
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Hi Kevin,
>>
>> Now increase your CPU processing power and show me the numbers.
>>
>>
>>
>> Do we support AES-NI optimization with extended CPU instruction set for
>> AES hardware acceleration?
>>
>> libcrypto.so library that supports hardware acceleration, such as
>> OpenSSL 1.0.1e. (Many OS versions have an older version of the library that
>> does not support AES-NI.)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From:Kevin Risden > Reply-To:
>> "user@knox.apache.org " > > Date: Tuesday, October 9, 2018 at 10:26 AM To:
>> "user@knox.apache.org " > > Subject: Re: WebHDFS performance issue in Knox*
>>
>>
>>
>> Writes look to have performance impact as well:
>>
>>- directly to webhdfs - ~2.6 seconds
>>- knox no ssl - ~29 seconds
>>- knox ssl - ~49.6 seconds
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden  wrote:
>>
>> If I run two downloads concurrently:
>>
>>
>>
>> 1,073,741,824 46.1MB/s   in 22s
>>
>> 1,073,741,824 51.3MB/s   in 22s
>>
>>
>>
>> So it isn't a limitation of the Knox gateway itself in total bandwidth
>> but a per connection limitation somehow.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden  wrote:
>>
>> So I was able to reproduce a slowdown with SSL with a pseudo distributed
>> HDFS setup on a single node with Knox running on the same node. This was
>> setup in Virtualbox on my laptop.
>>
>>
>>
>> Rough timings with wget for a 1GB random file:
>>
>>- directly to webhdfs - 1,073,741,824  252MB/s   in 3.8s
>>- knox no ssl - 1,073,741,824  264MB/s   in 3.6s
>>- knox ssl - 1,073,741,824 54.3MB/s   in 20s
>>
>> There is a significant decrease with Knox SSL for some reason.
>>
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Sun, Sep 23, 2018 at 8:53 PM larry mccay  wrote:
>>
>> SSL handshake will likely happen at least twice.
>>
>> Once for the request through Knox to the NN then the redirect from the NN
>> to the DN goes all the way back to the client.
>>
>> So they have to follow the redirect and do the handshake to the DN.
>>
>>
>>
>>
>>
>> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden  wrote:
>>
>> So I found this in the Knox issues list in JIRA:
>>
>>
>>
>> https://issues.apache.org/jira/browse/KNOX-1221
>>
>>
>>
>> It sounds familiar in terms of a slowdown when going through Knox.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden  wrote:
>>
>> Hmmm yea curl for a single file should do the handshake once.
>>
>>
>>
>> What are the system performance statistics during the SSL vs non SSL
>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>> if using that. Otherwise watching top may be helpful. It would be help to
>> determine if the Knox is working harder during the SSL transfer.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang  wrote:
>>
>> I'm just using curl to download a single large file. So I suspect SSL
>> handshake just happens once?
>>
>>
>>
>> On Tue, Sep 11, 2018 at 12:02 PM
>>
>> Kevin Risden
>>
>>  wrote:
>>
>> What client are you using to connect Knox? Is this for a single file or a
>> bunch of files?
>>
>>
>>
>> The SSL handshake can be slow if the client doesn't keep the connection
>>

Re: WebHDFS performance issue in Knox

2018-10-09 Thread Guang Yang 
Thanks Kevin conducting such experiment! This is exactly what I saw before.
It doesn't look right the download speed is 10x slower when enabling SSL.

On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
dvillarr...@hortonworks.com> wrote:

> I bring this up because HDFS encryption saw an increase in performance.
>
> https://issues.apache.org/jira/browse/HDFS-6606
>
>
>
> https://issues.apache.org/jira/browse/HADOOP-10768
>
>
>
> Maybe Knox can make some enhancements in this area?
>
>
>
> *From: *David Villarreal 
> *Date: *Tuesday, October 9, 2018 at 10:34 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Hi Kevin,
>
> Now increase your CPU processing power and show me the numbers.
>
>
>
> Do we support AES-NI optimization with extended CPU instruction set for
> AES hardware acceleration?
>
> libcrypto.so library that supports hardware acceleration, such as OpenSSL
> 1.0.1e. (Many OS versions have an older version of the library that does
> not support AES-NI.)
>
>
>
>
>
> *From: *Kevin Risden 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Tuesday, October 9, 2018 at 10:26 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Writes look to have performance impact as well:
>
>- directly to webhdfs - ~2.6 seconds
>- knox no ssl - ~29 seconds
>- knox ssl - ~49.6 seconds
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden  wrote:
>
> If I run two downloads concurrently:
>
>
>
> 1,073,741,824 46.1MB/s   in 22s
>
> 1,073,741,824 51.3MB/s   in 22s
>
>
>
> So it isn't a limitation of the Knox gateway itself in total bandwidth but
> a per connection limitation somehow.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden  wrote:
>
> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
>
>
> Rough timings with wget for a 1GB random file:
>
>- directly to webhdfs - 1,073,741,824  252MB/s   in 3.8s
>- knox no ssl - 1,073,741,824  264MB/s   in 3.6s
>- knox ssl - 1,073,741,824 54.3MB/s   in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
>
>
> Kevin Risden
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay  wrote:
>
> SSL handshake will likely happen at least twice.
>
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
>
> So they have to follow the redirect and do the handshake to the DN.
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden  wrote:
>
> So I found this in the Knox issues list in JIRA:
>
>
>
> https://issues.apache.org/jira/browse/KNOX-1221
>
>
>
> It sounds familiar in terms of a slowdown when going through Knox.
>
>
> Kevin Risden
>
>
>
>
>
> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden  wrote:
>
> Hmmm yea curl for a single file should do the handshake once.
>
>
>
> What are the system performance statistics during the SSL vs non SSL
> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
> if using that. Otherwise watching top may be helpful. It would be help to
> determine if the Knox is working harder during the SSL transfer.
>
>
> Kevin Risden
>
>
>
>
>
> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang  wrote:
>
> I'm just using curl to download a single large file. So I suspect SSL
> handshake just happens once?
>
>
>
> On Tue, Sep 11, 2018 at 12:02 PM
>
> Kevin Risden
>
>  wrote:
>
> What client are you using to connect Knox? Is this for a single file or a
> bunch of files?
>
>
>
> The SSL handshake can be slow if the client doesn't keep the connection
> open.
>
> Kevin Risden
>
>
>
> On Tue, Sep 11, 2018, 14:51 Guang Yang  wrote:
>
> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>
>
>
> **
>
> *ssl.enabled*
>
> *false*
>
> *Indicates whether SSL is enabled.*
>
> **
>
>
>
> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay  wrote:
>
> I really don't think that kind of difference should be expected from
> merely SSL overhead.
>
> I don't however have any metrics to contradict it either since I do not
> run Knox without SSL.
>
>
>
> Given the above, I am struggling coming up with a meaningful response to
> this. :(
>
> I don't think you should see a 10 fold increase in speed by disabling SSL
> though.
>
>
>
> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang  wrote:
>
> Any idea guys?
>
>
>
> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang  wrote:
>
> Thanks guys! The issue seems exactly what David pointed out, which is
> because of encrypted over SSL.
>
>
>
> Without Knox, the download speed can reach to *400M/s* if I call Namenode
> directly. And with disabling SSL, the speed can reach to *~400M/s* as
> well through Knox. But with SSL, the speed drops significantly to *~40M/s*.
> I know it's because of encrypted, but it does surprised me with such a
>

Re: Spark History UI Error WARN HttpParser: Header is too large >8192

2018-10-09 Thread Theyaa Matti 
Apologies Kevin, I will make sure to send it only to the user group and
thank you for the quick response. I know about the hive configuration but I
do not see similar ones for the spark side. I did apply the knox ones but
did not work since I believe the issue is on the spark side. Any pointers
would be appreciated.

Best

On Tue, Oct 9, 2018 at 12:35 PM Kevin Risden  wrote:

> The issue for the header could be on the Spark side as well.
>
> Some more background details:
> https://risdenk.github.io/2018/03/04/http-413-full-head-kerberos-spnego-authentication.html
>
> Kevin Risden
>
>
> On Tue, Oct 9, 2018 at 12:32 PM Kevin Risden  wrote:
>
>> Please don't send to two different mailing lists at the same time.
>> Removed the dev list.
>>
>> It sounds like you are describing the same issue as KNOX-624 [1]. The two
>> configurations are:
>>
>> gateway.httpserver.requestHeaderBuffer
>> gateway.httpserver.responseHeaderBuffer
>>
>> 1. https://issues.apache.org/jira/browse/KNOX-624
>>
>> Kevin Risden
>>
>>
>> On Tue, Oct 9, 2018 at 12:01 PM Theyaa Matti 
>> wrote:
>>
>>> Hi,
>>>I am getting this error message "WARN HttpParser: Header is too
>>> large >8192" when trying to access the spark history ui through knox. Any
>>> idea please?
>>>
>>> Also when trying to load the executors page, I get : GSS initiate
>>> failed [Caused by GSSException: Failure unspecified at GSS-API level
>>> (Mechanism level: Request is a replay (34))]
>>>
>>> when knox is requesting executorspage-template.html
>>>
>>> appreciate any help here.
>>>
>>

Re: WebHDFS performance issue in Knox

2018-10-09 Thread Kevin Risden 
If I run two downloads concurrently:

1,073,741,824 46.1MB/s   in 22s
1,073,741,824 51.3MB/s   in 22s

So it isn't a limitation of the Knox gateway itself in total bandwidth but
a per connection limitation somehow.

Kevin Risden


On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden  wrote:

> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
> Rough timings with wget for a 1GB random file:
>
>- directly to webhdfs - 1,073,741,824  252MB/s   in 3.8s
>- knox no ssl - 1,073,741,824  264MB/s   in 3.6s
>- knox ssl - 1,073,741,824 54.3MB/s   in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
> Kevin Risden
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay  wrote:
>
>> SSL handshake will likely happen at least twice.
>> Once for the request through Knox to the NN then the redirect from the NN
>> to the DN goes all the way back to the client.
>> So they have to follow the redirect and do the handshake to the DN.
>>
>>
>> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden  wrote:
>>
>>> So I found this in the Knox issues list in JIRA:
>>>
>>> https://issues.apache.org/jira/browse/KNOX-1221
>>>
>>> It sounds familiar in terms of a slowdown when going through Knox.
>>>
>>> Kevin Risden
>>>
>>>
>>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden 
>>> wrote:
>>>
 Hmmm yea curl for a single file should do the handshake once.

 What are the system performance statistics during the SSL vs non SSL
 testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
 if using that. Otherwise watching top may be helpful. It would be help to
 determine if the Knox is working harder during the SSL transfer.

 Kevin Risden


 On Wed, Sep 12, 2018 at 2:52 PM Guang Yang  wrote:

> I'm just using curl to download a single large file. So I suspect SSL
> handshake just happens once?
>
> On Tue, Sep 11, 2018 at 12:02 PM
> Kevin Risden
>  wrote:
>
>> What client are you using to connect Knox? Is this for a single file
>> or a bunch of files?
>>
>> The SSL handshake can be slow if the client doesn't keep the
>> connection open.
>>
>> Kevin Risden
>>
>> On Tue, Sep 11, 2018, 14:51 Guang Yang  wrote:
>>
>>> Thanks Larry. But the only difference is this part in my
>>> gateway-site.xml.
>>>
>>> **
>>> *ssl.enabled*
>>> *false*
>>> *Indicates whether SSL is
>>> enabled.*
>>> **
>>>
>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay 
>>> wrote:
>>>
 I really don't think that kind of difference should be expected
 from merely SSL overhead.
 I don't however have any metrics to contradict it either since I do
 not run Knox without SSL.

 Given the above, I am struggling coming up with a meaningful
 response to this. :(
 I don't think you should see a 10 fold increase in speed by
 disabling SSL though.

 On Tue, Sep 11, 2018 at 2:35 PM Guang Yang  wrote:

> Any idea guys?
>
> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang  wrote:
>
>> Thanks guys! The issue seems exactly what David pointed out,
>> which is because of encrypted over SSL.
>>
>> Without Knox, the download speed can reach to *400M/s* if I call
>> Namenode directly. And with disabling SSL, the speed can reach to
>> *~400M/s* as well through Knox. But with SSL, the speed drops
>> significantly to *~40M/s*. I know it's because of encrypted, but
>> it does surprised me with such a difference. Is it normal from your
>> perspective?
>>
>> Thanks,
>> Guang
>>
>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>> dvillarr...@hortonworks.com> wrote:
>>
>>> Hi Guang,
>>>
>>>
>>>
>>> Keep in mind the data is being encrypted over SSL.  If you
>>> disable SSL you will most likely see a very significant boost in
>>> throughput.  Some people have used more powerful computers to make
>>> encryption quicker.
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> David
>>>
>>>
>>>
>>> *From: *Sean Roberts 
>>> *Reply-To: *"user@knox.apache.org" 
>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>> *To: *"user@knox.apache.org" 
>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>
>>>
>>>
>>> Guang – This is somewhat to be expected.
>>>
>>>
>>>
>>> When you talk to WebHDFS directly, the client can distribute the
>>> request across many data nodes.

Re: Spark History UI Error WARN HttpParser: Header is too large >8192

2018-10-09 Thread Kevin Risden 
Please don't send to two different mailing lists at the same time. Removed
the dev list.

It sounds like you are describing the same issue as KNOX-624 [1]. The two
configurations are:

gateway.httpserver.requestHeaderBuffer
gateway.httpserver.responseHeaderBuffer

1. https://issues.apache.org/jira/browse/KNOX-624

Kevin Risden


On Tue, Oct 9, 2018 at 12:01 PM Theyaa Matti  wrote:

> Hi,
>I am getting this error message "WARN HttpParser: Header is too large
> >8192" when trying to access the spark history ui through knox. Any idea
> please?
>
> Also when trying to load the executors page, I get : GSS initiate failed
> [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism 
> level:
> Request is a replay (34))]
>
> when knox is requesting executorspage-template.html
>
> appreciate any help here.
>

Re: WebHDFS performance issue in Knox

2018-10-09 Thread Kevin Risden 
So I was able to reproduce a slowdown with SSL with a pseudo distributed
HDFS setup on a single node with Knox running on the same node. This was
setup in Virtualbox on my laptop.

Rough timings with wget for a 1GB random file:

   - directly to webhdfs - 1,073,741,824  252MB/s   in 3.8s
   - knox no ssl - 1,073,741,824  264MB/s   in 3.6s
   - knox ssl - 1,073,741,824 54.3MB/s   in 20s

There is a significant decrease with Knox SSL for some reason.

Kevin Risden


On Sun, Sep 23, 2018 at 8:53 PM larry mccay  wrote:

> SSL handshake will likely happen at least twice.
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
> So they have to follow the redirect and do the handshake to the DN.
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden  wrote:
>
>> So I found this in the Knox issues list in JIRA:
>>
>> https://issues.apache.org/jira/browse/KNOX-1221
>>
>> It sounds familiar in terms of a slowdown when going through Knox.
>>
>> Kevin Risden
>>
>>
>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden  wrote:
>>
>>> Hmmm yea curl for a single file should do the handshake once.
>>>
>>> What are the system performance statistics during the SSL vs non SSL
>>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>>> if using that. Otherwise watching top may be helpful. It would be help to
>>> determine if the Knox is working harder during the SSL transfer.
>>>
>>> Kevin Risden
>>>
>>>
>>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang  wrote:
>>>
 I'm just using curl to download a single large file. So I suspect SSL
 handshake just happens once?

 On Tue, Sep 11, 2018 at 12:02 PM
 Kevin Risden
  wrote:

> What client are you using to connect Knox? Is this for a single file
> or a bunch of files?
>
> The SSL handshake can be slow if the client doesn't keep the
> connection open.
>
> Kevin Risden
>
> On Tue, Sep 11, 2018, 14:51 Guang Yang  wrote:
>
>> Thanks Larry. But the only difference is this part in my
>> gateway-site.xml.
>>
>> **
>> *ssl.enabled*
>> *false*
>> *Indicates whether SSL is enabled.*
>> **
>>
>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay 
>> wrote:
>>
>>> I really don't think that kind of difference should be expected from
>>> merely SSL overhead.
>>> I don't however have any metrics to contradict it either since I do
>>> not run Knox without SSL.
>>>
>>> Given the above, I am struggling coming up with a meaningful
>>> response to this. :(
>>> I don't think you should see a 10 fold increase in speed by
>>> disabling SSL though.
>>>
>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang  wrote:
>>>
 Any idea guys?

 On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang  wrote:

> Thanks guys! The issue seems exactly what David pointed out, which
> is because of encrypted over SSL.
>
> Without Knox, the download speed can reach to *400M/s* if I call
> Namenode directly. And with disabling SSL, the speed can reach to
> *~400M/s* as well through Knox. But with SSL, the speed drops
> significantly to *~40M/s*. I know it's because of encrypted, but
> it does surprised me with such a difference. Is it normal from your
> perspective?
>
> Thanks,
> Guang
>
> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
> dvillarr...@hortonworks.com> wrote:
>
>> Hi Guang,
>>
>>
>>
>> Keep in mind the data is being encrypted over SSL.  If you
>> disable SSL you will most likely see a very significant boost in
>> throughput.  Some people have used more powerful computers to make
>> encryption quicker.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> David
>>
>>
>>
>> *From: *Sean Roberts 
>> *Reply-To: *"user@knox.apache.org" 
>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>> *To: *"user@knox.apache.org" 
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Guang – This is somewhat to be expected.
>>
>>
>>
>> When you talk to WebHDFS directly, the client can distribute the
>> request across many data nodes. Also, you are getting data directly 
>> from
>> the source.
>>
>> With Knox, all traffic goes through the single Knox host. Knox is
>> responsible for fetching from the datanodes and consolidating to 
>> send to
>> you. This means overhead as it’s acting as a middle man, and lower 
>> network
>> capacity since only 1 host is serving data to you.
>>
>>
>>

Parcel for Knox

2018-10-09 Thread Lars Francke 
Hi,

I'm wondering if anyone here has ever built a Parcel for Cloudera's CDH for
Knox?

Cheers,
Lars