Re: WebHDFS performance issue in Knox

[Guang Yang]

Thanks Kevin conducting such experiment! This is exactly what I saw before.
It doesn't look right the download speed is 10x slower when enabling SSL.

On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
dvillarr...@hortonworks.com> wrote:

> I bring this up because HDFS encryption saw an increase in performance.
>
> https://issues.apache.org/jira/browse/HDFS-6606
>
>
>
> https://issues.apache.org/jira/browse/HADOOP-10768
>
>
>
> Maybe Knox can make some enhancements in this area?
>
>
>
> *From: *David Villarreal 
> *Date: *Tuesday, October 9, 2018 at 10:34 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Hi Kevin,
>
> Now increase your CPU processing power and show me the numbers.
>
>
>
> Do we support AES-NI optimization with extended CPU instruction set for
> AES hardware acceleration?
>
> libcrypto.so library that supports hardware acceleration, such as OpenSSL
> 1.0.1e. (Many OS versions have an older version of the library that does
> not support AES-NI.)
>
>
>
>
>
> *From: *Kevin Risden 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Tuesday, October 9, 2018 at 10:26 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Writes look to have performance impact as well:
>
>- directly to webhdfs - ~2.6 seconds
>- knox no ssl - ~29 seconds
>- knox ssl - ~49.6 seconds
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden  wrote:
>
> If I run two downloads concurrently:
>
>
>
> 1,073,741,824 46.1MB/s   in 22s
>
> 1,073,741,824 51.3MB/s   in 22s
>
>
>
> So it isn't a limitation of the Knox gateway itself in total bandwidth but
> a per connection limitation somehow.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden  wrote:
>
> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
>
>
> Rough timings with wget for a 1GB random file:
>
>- directly to webhdfs - 1,073,741,824  252MB/s   in 3.8s
>- knox no ssl - 1,073,741,824  264MB/s   in 3.6s
>- knox ssl - 1,073,741,824 54.3MB/s   in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
>
>
> Kevin Risden
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay  wrote:
>
> SSL handshake will likely happen at least twice.
>
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
>
> So they have to follow the redirect and do the handshake to the DN.
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden  wrote:
>
> So I found this in the Knox issues list in JIRA:
>
>
>
> https://issues.apache.org/jira/browse/KNOX-1221
>
>
>
> It sounds familiar in terms of a slowdown when going through Knox.
>
>
> Kevin Risden
>
>
>
>
>
> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden  wrote:
>
> Hmmm yea curl for a single file should do the handshake once.
>
>
>
> What are the system performance statistics during the SSL vs non SSL
> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
> if using that. Otherwise watching top may be helpful. It would be help to
> determine if the Knox is working harder during the SSL transfer.
>
>
> Kevin Risden
>
>
>
>
>
> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang  wrote:
>
> I'm just using curl to download a single large file. So I suspect SSL
> handshake just happens once?
>
>
>
> On Tue, Sep 11, 2018 at 12:02 PM
>
> Kevin Risden
>
>  wrote:
>
> What client are you using to connect Knox? Is this for a single file or a
> bunch of files?
>
>
>
> The SSL handshake can be slow if the client doesn't keep the connection
> open.
>
> Kevin Risden
>
>
>
> On Tue, Sep 11, 2018, 14:51 Guang Yang  wrote:
>
> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>
>
>
> **
>
> *ssl.enabled*
>
> *false*
>
> *    Indicates whether SSL is enabled.*
>
> **
>
>
>
> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay  wrote:
>
> I really don't think that kind of difference should be expected from
> merely SSL overhead.
>
> I don't however have any metrics to contradict it either since I do not
> run Knox without SSL.
>
>
>
> Given the above, I am struggling coming up with a

Re: WebHDFS performance issue in Knox

[Guang Yang]

I'm just using curl to download a single large file. So I suspect SSL
handshake just happens once?

On Tue, Sep 11, 2018 at 12:02 PM Kevin Risden  wrote:

> What client are you using to connect Knox? Is this for a single file or a
> bunch of files?
>
> The SSL handshake can be slow if the client doesn't keep the connection
> open.
>
> Kevin Risden
>
> On Tue, Sep 11, 2018, 14:51 Guang Yang  wrote:
>
>> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>>
>> **
>> *ssl.enabled*
>> *false*
>> *Indicates whether SSL is enabled.*
>> **
>>
>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay  wrote:
>>
>>> I really don't think that kind of difference should be expected from
>>> merely SSL overhead.
>>> I don't however have any metrics to contradict it either since I do not
>>> run Knox without SSL.
>>>
>>> Given the above, I am struggling coming up with a meaningful response to
>>> this. :(
>>> I don't think you should see a 10 fold increase in speed by disabling
>>> SSL though.
>>>
>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang  wrote:
>>>
>>>> Any idea guys?
>>>>
>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang  wrote:
>>>>
>>>>> Thanks guys! The issue seems exactly what David pointed out, which is
>>>>> because of encrypted over SSL.
>>>>>
>>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>> significantly to *~40M/s*. I know it's because of encrypted, but it
>>>>> does surprised me with such a difference. Is it normal from your
>>>>> perspective?
>>>>>
>>>>> Thanks,
>>>>> Guang
>>>>>
>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>> dvillarr...@hortonworks.com> wrote:
>>>>>
>>>>>> Hi Guang,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Keep in mind the data is being encrypted over SSL.  If you disable
>>>>>> SSL you will most likely see a very significant boost in throughput.  
>>>>>> Some
>>>>>> people have used more powerful computers to make encryption quicker.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>>
>>>>>> David
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From: *Sean Roberts 
>>>>>> *Reply-To: *"user@knox.apache.org" 
>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>>> *To: *"user@knox.apache.org" 
>>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>>
>>>>>>
>>>>>>
>>>>>> Guang – This is somewhat to be expected.
>>>>>>
>>>>>>
>>>>>>
>>>>>> When you talk to WebHDFS directly, the client can distribute the
>>>>>> request across many data nodes. Also, you are getting data directly from
>>>>>> the source.
>>>>>>
>>>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>>>> responsible for fetching from the datanodes and consolidating to send to
>>>>>> you. This means overhead as it’s acting as a middle man, and lower 
>>>>>> network
>>>>>> capacity since only 1 host is serving data to you.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Also, if running on a cloud provider, the Knox host may be a smaller
>>>>>> instance size with lower network capacity.
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Sean Roberts
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From: *Guang Yang 
>>>>>> *Reply-To: *"user@knox.apache.org" 
>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>>> *To: *"user@knox.apache.org" 
>>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a
>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s.
>>>>>> However, if we download directly from datanode, the speed is about 40M/s 
>>>>>> at
>>>>>> least.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Guang
>>>>>>
>>>>>
>>>>>
>>>>
>>

Re: WebHDFS performance issue in Knox

[Guang Yang]

Thanks Larry. But the only difference is this part in my gateway-site.xml.

**
*ssl.enabled*
*false*
*Indicates whether SSL is enabled.*
**

On Tue, Sep 11, 2018 at 11:42 AM, larry mccay  wrote:

> I really don't think that kind of difference should be expected from
> merely SSL overhead.
> I don't however have any metrics to contradict it either since I do not
> run Knox without SSL.
>
> Given the above, I am struggling coming up with a meaningful response to
> this. :(
> I don't think you should see a 10 fold increase in speed by disabling SSL
> though.
>
> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang  wrote:
>
>> Any idea guys?
>>
>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang  wrote:
>>
>>> Thanks guys! The issue seems exactly what David pointed out, which is
>>> because of encrypted over SSL.
>>>
>>> Without Knox, the download speed can reach to *400M/s* if I call
>>> Namenode directly. And with disabling SSL, the speed can reach to
>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>> significantly to *~40M/s*. I know it's because of encrypted, but it
>>> does surprised me with such a difference. Is it normal from your
>>> perspective?
>>>
>>> Thanks,
>>> Guang
>>>
>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>> dvillarr...@hortonworks.com> wrote:
>>>
>>>> Hi Guang,
>>>>
>>>>
>>>>
>>>> Keep in mind the data is being encrypted over SSL.  If you disable SSL
>>>> you will most likely see a very significant boost in throughput.  Some
>>>> people have used more powerful computers to make encryption quicker.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> *From: *Sean Roberts 
>>>> *Reply-To: *"user@knox.apache.org" 
>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>> *To: *"user@knox.apache.org" 
>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>
>>>>
>>>>
>>>> Guang – This is somewhat to be expected.
>>>>
>>>>
>>>>
>>>> When you talk to WebHDFS directly, the client can distribute the
>>>> request across many data nodes. Also, you are getting data directly from
>>>> the source.
>>>>
>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>> responsible for fetching from the datanodes and consolidating to send to
>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>> capacity since only 1 host is serving data to you.
>>>>
>>>>
>>>>
>>>> Also, if running on a cloud provider, the Knox host may be a smaller
>>>> instance size with lower network capacity.
>>>>
>>>> --
>>>>
>>>> Sean Roberts
>>>>
>>>>
>>>>
>>>> *From: *Guang Yang 
>>>> *Reply-To: *"user@knox.apache.org" 
>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>> *To: *"user@knox.apache.org" 
>>>> *Subject: *WebHDFS performance issue in Knox
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>>>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>>>> if we download directly from datanode, the speed is about 40M/s at least.
>>>>
>>>>
>>>>
>>>> Are you guys aware of this problem? Any suggestion?
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Guang
>>>>
>>>
>>>
>>

Re: WebHDFS performance issue in Knox

[Guang Yang]

Thanks guys! The issue seems exactly what David pointed out, which is
because of encrypted over SSL.

Without Knox, the download speed can reach to *400M/s* if I call Namenode
directly. And with disabling SSL, the speed can reach to *~400M/s* as well
through Knox. But with SSL, the speed drops significantly to *~40M/s*. I
know it's because of encrypted, but it does surprised me with such a
difference. Is it normal from your perspective?

Thanks,
Guang

On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
dvillarr...@hortonworks.com> wrote:

> Hi Guang,
>
>
>
> Keep in mind the data is being encrypted over SSL.  If you disable SSL you
> will most likely see a very significant boost in throughput.  Some people
> have used more powerful computers to make encryption quicker.
>
>
>
> Thanks,
>
>
>
> David
>
>
>
> *From: *Sean Roberts 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Tuesday, September 4, 2018 at 1:53 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Guang – This is somewhat to be expected.
>
>
>
> When you talk to WebHDFS directly, the client can distribute the request
> across many data nodes. Also, you are getting data directly from the source.
>
> With Knox, all traffic goes through the single Knox host. Knox is
> responsible for fetching from the datanodes and consolidating to send to
> you. This means overhead as it’s acting as a middle man, and lower network
> capacity since only 1 host is serving data to you.
>
>
>
> Also, if running on a cloud provider, the Knox host may be a smaller
> instance size with lower network capacity.
>
> --
>
> Sean Roberts
>
>
>
> *From: *Guang Yang 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Tuesday, 4 September 2018 at 07:46
> *To: *"user@knox.apache.org" 
> *Subject: *WebHDFS performance issue in Knox
>
>
>
> Hi,
>
>
>
> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
> through WebHDFS in Knox, the download speed is just about 11M/s. However,
> if we download directly from datanode, the speed is about 40M/s at least.
>
>
>
> Are you guys aware of this problem? Any suggestion?
>
>
>
> Thanks,
>
> Guang
>

Re: turn off debug logging from org.apache.http.wire

[Guang Yang]

ent.File=${app.log.dir}/${launcher.name
> }-http-client.log
> #log4j.appender.httpclient.DatePattern=.-MM-dd
> #log4j.appender.httpclient.layout=org.apache.log4j.PatternLayout
> #log4j.appender.httpclient.layout.ConversionPattern=%d{ISO8601}|%t|%m%n
>
> # Apache Shiro Related logging - KNOX-757
> #log4j.logger.org.springframework=DEBUG
> #log4j.logger.net.sf.ehcache=DEBUG
> #log4j.logger.org.apache.shiro.util.ThreadContext=DEBUG
>
>
> On Wed, Sep 5, 2018 at 2:07 PM Guang Yang  wrote:
>
>> Hi Larry,
>>
>> Here is my gateway-log4j.properties.
>>
>> *app.log.dir=/var/log/knox*
>> *app.log.file=${launcher.name <http://launcher.name>}.log*
>> *app.audit.file=${launcher.name <http://launcher.name>}-audit.log*
>>
>> *log4j.rootLogger=ERROR*
>>
>> *log4j.logger.org.apache.knox.gateway=ERROR*
>>
>> *log4j.logger.org.eclipse.jetty=ERROR*
>>
>> *log4j.logger.org.apache.shiro=ERROR*
>> *log4j.logger.org.apache.http=ERROR*
>> *log4j.logger.org.apache.http.client=ERROR*
>> *log4j.logger.org.apache.http.headers=ERROR*
>> *log4j.logger.org.apache.http.wire=ERROR*
>>
>> Even I changed it like this, I can still see lots of DEBUG log in
>> *gateway.out*. Seems it only affects* gateway.log*, not *gateway.out.*
>>
>> On Wed, Sep 5, 2018 at 10:48 AM, larry mccay  wrote:
>>
>>> Hi Guang -
>>>
>>> This certainly sounds frustrating.
>>> I have never had trouble turning it off.
>>> Can you share your gatway-log4j.properties file - just make sure there
>>> isn't anything sensitive in there?
>>>
>>> thanks,
>>>
>>> --larry
>>>
>>> On Wed, Sep 5, 2018 at 1:40 PM Guang Yang  wrote:
>>>
>>>> And we're not using Ambari. We just deploy manually.
>>>>
>>>> On Tue, Sep 4, 2018 at 11:02 PM, Guang Yang  wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I'm working with Wei and we still don't figure it out. Let me clarify
>>>>> the question.
>>>>>
>>>>> Currently, we're seeing lots of DEBUG logs in file *gateway.out*,
>>>>> which is from here https://github.com/apache/knox/blob/master/gateway-
>>>>> release/home/bin/gateway.sh#L127. On the one hand, it prints the file
>>>>> content just like Wei talked about before, on the other hand we suspect it
>>>>> might be related to the performance issue when download a file through
>>>>> WEBHDFS. So we're trying to disable all these DEBUG logs. We tried simply
>>>>> removing this part *>>$APP_OUT_FILE*, although there is no such
>>>>> output file, but actually Knox still prints logs to console. So what we
>>>>> want to do is to disable all the DEBUG log thoroughly, so the service 
>>>>> won't
>>>>> print logs to anywhere.
>>>>>
>>>>> We almost tried everything in *gateway-log4j.properties*, but it
>>>>> seems it only affects app.log.file=${launcher.name}.*log* instead of
>>>>> *gateway.out*. So, any idea guys?
>>>>>
>>>>> Thanks,
>>>>> Guang
>>>>>
>>>>> On Sun, Apr 15, 2018 at 11:08 AM, larry mccay 
>>>>> wrote:
>>>>>
>>>>>> +1 to Kevin's point.
>>>>>> Ambari rewrites all configs on server restart.
>>>>>>
>>>>>> On Sun, Apr 15, 2018 at 1:16 PM, Kevin Risden 
>>>>>> wrote:
>>>>>>
>>>>>>> Are you using Ambari or deploying Knox manually?
>>>>>>>
>>>>>>> If you using Ambari, then Ambari will force overwrite the log4j
>>>>>>> configs during a restart. You must update the log4j settings in Ambari.
>>>>>>> Another option if using Ambari is to find the debug setting and set it 
>>>>>>> to
>>>>>>> false (I don't have a cluster in front of me so can't look up the 
>>>>>>> setting).
>>>>>>>
>>>>>>> Kevin Risden
>>>>>>>
>>>>>>> On Sun, Apr 15, 2018 at 10:56 AM, Wei Han  wrote:
>>>>>>>
>>>>>>>> Interesting. Thanks Larry. I'll dig more on my side.
>>>>>>>>
>>>>>>>> On Sun, Apr 15, 2018 at 4:54 AM, larry mccay 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> No, I cannot reproduce it.
>

Re: turn off debug logging from org.apache.http.wire

[Guang Yang]

Hi Larry,

Here is my gateway-log4j.properties.

*app.log.dir=/var/log/knox*
*app.log.file=${launcher.name <http://launcher.name>}.log*
*app.audit.file=${launcher.name <http://launcher.name>}-audit.log*

*log4j.rootLogger=ERROR*

*log4j.logger.org.apache.knox.gateway=ERROR*

*log4j.logger.org.eclipse.jetty=ERROR*

*log4j.logger.org.apache.shiro=ERROR*
*log4j.logger.org.apache.http=ERROR*
*log4j.logger.org.apache.http.client=ERROR*
*log4j.logger.org.apache.http.headers=ERROR*
*log4j.logger.org.apache.http.wire=ERROR*

Even I changed it like this, I can still see lots of DEBUG log in
*gateway.out*. Seems it only affects* gateway.log*, not *gateway.out.*

On Wed, Sep 5, 2018 at 10:48 AM, larry mccay  wrote:

> Hi Guang -
>
> This certainly sounds frustrating.
> I have never had trouble turning it off.
> Can you share your gatway-log4j.properties file - just make sure there
> isn't anything sensitive in there?
>
> thanks,
>
> --larry
>
> On Wed, Sep 5, 2018 at 1:40 PM Guang Yang  wrote:
>
>> And we're not using Ambari. We just deploy manually.
>>
>> On Tue, Sep 4, 2018 at 11:02 PM, Guang Yang  wrote:
>>
>>> Hi guys,
>>>
>>> I'm working with Wei and we still don't figure it out. Let me clarify
>>> the question.
>>>
>>> Currently, we're seeing lots of DEBUG logs in file *gateway.out*, which
>>> is from here https://github.com/apache/knox/blob/master/gateway-
>>> release/home/bin/gateway.sh#L127. On the one hand, it prints the file
>>> content just like Wei talked about before, on the other hand we suspect it
>>> might be related to the performance issue when download a file through
>>> WEBHDFS. So we're trying to disable all these DEBUG logs. We tried simply
>>> removing this part *>>$APP_OUT_FILE*, although there is no such output
>>> file, but actually Knox still prints logs to console. So what we want to do
>>> is to disable all the DEBUG log thoroughly, so the service won't print logs
>>> to anywhere.
>>>
>>> We almost tried everything in *gateway-log4j.properties*, but it seems
>>> it only affects app.log.file=${launcher.name}.*log* instead of
>>> *gateway.out*. So, any idea guys?
>>>
>>> Thanks,
>>> Guang
>>>
>>> On Sun, Apr 15, 2018 at 11:08 AM, larry mccay  wrote:
>>>
>>>> +1 to Kevin's point.
>>>> Ambari rewrites all configs on server restart.
>>>>
>>>> On Sun, Apr 15, 2018 at 1:16 PM, Kevin Risden 
>>>> wrote:
>>>>
>>>>> Are you using Ambari or deploying Knox manually?
>>>>>
>>>>> If you using Ambari, then Ambari will force overwrite the log4j
>>>>> configs during a restart. You must update the log4j settings in Ambari.
>>>>> Another option if using Ambari is to find the debug setting and set it to
>>>>> false (I don't have a cluster in front of me so can't look up the 
>>>>> setting).
>>>>>
>>>>> Kevin Risden
>>>>>
>>>>> On Sun, Apr 15, 2018 at 10:56 AM, Wei Han  wrote:
>>>>>
>>>>>> Interesting. Thanks Larry. I'll dig more on my side.
>>>>>>
>>>>>> On Sun, Apr 15, 2018 at 4:54 AM, larry mccay 
>>>>>> wrote:
>>>>>>
>>>>>>> No, I cannot reproduce it.
>>>>>>> If you are modifying the correct gateway-log4j.properties and
>>>>>>> restarting the server you should not see that.
>>>>>>>
>>>>>>> In fact, turning on DEBUG for wire via:
>>>>>>> log4j.logger.org.apache.http.wire=DEBUG
>>>>>>>
>>>>>>> Doesn't result in output in gateway.out for me but instead
>>>>>>> gateway.log and turning it on and off certainly works for me.
>>>>>>>
>>>>>>> You may have enabled TRACE logging if you are seeing body content -
>>>>>>> those settings are like the following:
>>>>>>>
>>>>>>> #log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
>>>>>>> #log4j.additivity.org.apache.knox.gateway.access=false
>>>>>>>
>>>>>>> #log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
>>>>>>> #log4j.additivity.org.apache.knox.gateway.http=false
>>>>>>> ##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
>>>>>>> ##log4j.logger.org.apache.knox.gateway.http.response.hea

Re: turn off debug logging from org.apache.http.wire

[Guang Yang]

And we're not using Ambari. We just deploy manually.

On Tue, Sep 4, 2018 at 11:02 PM, Guang Yang  wrote:

> Hi guys,
>
> I'm working with Wei and we still don't figure it out. Let me clarify the
> question.
>
> Currently, we're seeing lots of DEBUG logs in file *gateway.out*, which
> is from here https://github.com/apache/knox/blob/master/gateway-
> release/home/bin/gateway.sh#L127. On the one hand, it prints the file
> content just like Wei talked about before, on the other hand we suspect it
> might be related to the performance issue when download a file through
> WEBHDFS. So we're trying to disable all these DEBUG logs. We tried simply
> removing this part *>>$APP_OUT_FILE*, although there is no such output
> file, but actually Knox still prints logs to console. So what we want to do
> is to disable all the DEBUG log thoroughly, so the service won't print logs
> to anywhere.
>
> We almost tried everything in *gateway-log4j.properties*, but it seems it
> only affects app.log.file=${launcher.name}.*log* instead of *gateway.out*.
> So, any idea guys?
>
> Thanks,
> Guang
>
> On Sun, Apr 15, 2018 at 11:08 AM, larry mccay  wrote:
>
>> +1 to Kevin's point.
>> Ambari rewrites all configs on server restart.
>>
>> On Sun, Apr 15, 2018 at 1:16 PM, Kevin Risden  wrote:
>>
>>> Are you using Ambari or deploying Knox manually?
>>>
>>> If you using Ambari, then Ambari will force overwrite the log4j configs
>>> during a restart. You must update the log4j settings in Ambari. Another
>>> option if using Ambari is to find the debug setting and set it to false (I
>>> don't have a cluster in front of me so can't look up the setting).
>>>
>>> Kevin Risden
>>>
>>> On Sun, Apr 15, 2018 at 10:56 AM, Wei Han  wrote:
>>>
>>>> Interesting. Thanks Larry. I'll dig more on my side.
>>>>
>>>> On Sun, Apr 15, 2018 at 4:54 AM, larry mccay  wrote:
>>>>
>>>>> No, I cannot reproduce it.
>>>>> If you are modifying the correct gateway-log4j.properties and
>>>>> restarting the server you should not see that.
>>>>>
>>>>> In fact, turning on DEBUG for wire via:
>>>>> log4j.logger.org.apache.http.wire=DEBUG
>>>>>
>>>>> Doesn't result in output in gateway.out for me but instead gateway.log
>>>>> and turning it on and off certainly works for me.
>>>>>
>>>>> You may have enabled TRACE logging if you are seeing body content -
>>>>> those settings are like the following:
>>>>>
>>>>> #log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
>>>>> #log4j.additivity.org.apache.knox.gateway.access=false
>>>>>
>>>>> #log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
>>>>> #log4j.additivity.org.apache.knox.gateway.http=false
>>>>> ##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
>>>>> ##log4j.logger.org.apache.knox.gateway.http.response.headers=OFF
>>>>> ##log4j.logger.org.apache.knox.gateway.http.request.body=OFF
>>>>> ##log4j.logger.org.apache.knox.gateway.http.response.body=OFF
>>>>>
>>>>> I suggest you back up to the gateway-log4j.properties from the
>>>>> original install and remove any other log4j config that you may have
>>>>> elsewhere.
>>>>>
>>>>> On Sun, Apr 15, 2018 at 1:58 AM, Wei Han  wrote:
>>>>>
>>>>>> Hi Larry - Thanks a lot for getting back to me.
>>>>>>
>>>>>> Yes I made sure all DEBUG level is turned off in my 
>>>>>> gateway-log4j.properties
>>>>>> file, but that doesn't seem to be working. I also tried to
>>>>>> explicitly set log4j.logger.httpclient.wire.header to WARN (as
>>>>>> suggested in post
>>>>>> <https://stackoverflow.com/questions/4915414/disable-httpclient-logging>),
>>>>>> but that also didn't help.
>>>>>>
>>>>>> Actually it's very easy to reproduce this(at least on my side). If
>>>>>> you call knox with a webhdfs request (like 
>>>>>> webhdfs/v1/tmp/weihan/small.txt?op=OPEN),
>>>>>> you should be able to see a bunch of below logs in gateway.out. In fact 
>>>>>> it
>>>>>> outputs the actual content on the wire(security hole?)
>>>>>>
>>>>>>   06:52:49.751 [qtp1473205473-61] DEBUG org.a

Re: turn off debug logging from org.apache.http.wire

[Guang Yang]

Hi guys,

I'm working with Wei and we still don't figure it out. Let me clarify the
question.

Currently, we're seeing lots of DEBUG logs in file *gateway.out*, which is
from here
https://github.com/apache/knox/blob/master/gateway-release/home/bin/gateway.sh#L127.
On the one hand, it prints the file content just like Wei talked about
before, on the other hand we suspect it might be related to the performance
issue when download a file through WEBHDFS. So we're trying to disable all
these DEBUG logs. We tried simply removing this part *>>$APP_OUT_FILE*,
although there is no such output file, but actually Knox still prints logs
to console. So what we want to do is to disable all the DEBUG log
thoroughly, so the service won't print logs to anywhere.

We almost tried everything in *gateway-log4j.properties*, but it seems it
only affects app.log.file=${launcher.name}.*log* instead of *gateway.out*.
So, any idea guys?

Thanks,
Guang

On Sun, Apr 15, 2018 at 11:08 AM, larry mccay  wrote:

> +1 to Kevin's point.
> Ambari rewrites all configs on server restart.
>
> On Sun, Apr 15, 2018 at 1:16 PM, Kevin Risden  wrote:
>
>> Are you using Ambari or deploying Knox manually?
>>
>> If you using Ambari, then Ambari will force overwrite the log4j configs
>> during a restart. You must update the log4j settings in Ambari. Another
>> option if using Ambari is to find the debug setting and set it to false (I
>> don't have a cluster in front of me so can't look up the setting).
>>
>> Kevin Risden
>>
>> On Sun, Apr 15, 2018 at 10:56 AM, Wei Han  wrote:
>>
>>> Interesting. Thanks Larry. I'll dig more on my side.
>>>
>>> On Sun, Apr 15, 2018 at 4:54 AM, larry mccay  wrote:
>>>
 No, I cannot reproduce it.
 If you are modifying the correct gateway-log4j.properties and
 restarting the server you should not see that.

 In fact, turning on DEBUG for wire via:
 log4j.logger.org.apache.http.wire=DEBUG

 Doesn't result in output in gateway.out for me but instead gateway.log
 and turning it on and off certainly works for me.

 You may have enabled TRACE logging if you are seeing body content -
 those settings are like the following:

 #log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
 #log4j.additivity.org.apache.knox.gateway.access=false

 #log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
 #log4j.additivity.org.apache.knox.gateway.http=false
 ##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
 ##log4j.logger.org.apache.knox.gateway.http.response.headers=OFF
 ##log4j.logger.org.apache.knox.gateway.http.request.body=OFF
 ##log4j.logger.org.apache.knox.gateway.http.response.body=OFF

 I suggest you back up to the gateway-log4j.properties from the original
 install and remove any other log4j config that you may have elsewhere.

 On Sun, Apr 15, 2018 at 1:58 AM, Wei Han  wrote:

> Hi Larry - Thanks a lot for getting back to me.
>
> Yes I made sure all DEBUG level is turned off in my 
> gateway-log4j.properties
> file, but that doesn't seem to be working. I also tried to explicitly
> set log4j.logger.httpclient.wire.header to WARN (as suggested in post
> ),
> but that also didn't help.
>
> Actually it's very easy to reproduce this(at least on my side). If you
> call knox with a webhdfs request (like 
> webhdfs/v1/tmp/weihan/small.txt?op=OPEN),
> you should be able to see a bunch of below logs in gateway.out. In fact it
> outputs the actual content on the wire(security hole?)
>
>   06:52:49.751 [qtp1473205473-61] DEBUG org.apache.http.wire -
> http-outgoing-2 << "[0x0][0x0
>
> Let me know if you're able to repro this.
>
> Thanks.
>
> On Sat, Apr 14, 2018 at 7:11 AM, larry mccay 
> wrote:
>
>> Hi Wei -
>>
>> If you look at your gateway-log4j.properties file, you should see
>> something like the following near the top:
>>
>> app.log.dir=${launcher.dir}/../logs
>> app.log.file=${launcher.name}.log
>> app.audit.file=${launcher.name}-audit.log
>>
>> log4j.rootLogger=ERROR, drfa
>>
>> log4j.logger.org.apache.knox.gateway=INFO
>> #log4j.logger.org.apache.knox.gateway=DEBUG
>>
>> #log4j.logger.org.eclipse.jetty=DEBUG
>> #log4j.logger.org.apache.shiro=DEBUG
>> #log4j.logger.org.apache.http=DEBUG
>> #log4j.logger.org.apache.http.client=DEBUG
>> #log4j.logger.org.apache.http.headers=DEBUG
>> #log4j.logger.org.apache.http.wire=DEBUG
>>
>> Note that all of the DEBUG settings are commented out.
>> Also note that the rootLogger is set to ERROR and not DEBUG.
>>
>> Can you compare and share with us what yours are set to?
>>
>> thanks,
>>
>> --larry
>>
>> On Sat, Apr 14, 2018 at 2:56 AM, Wei Han  wrote:
>>
>>> Hi 

WebHDFS performance issue in Knox

[Guang Yang]

Hi,

We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
through WebHDFS in Knox, the download speed is just about 11M/s. However,
if we download directly from datanode, the speed is about 40M/s at least.

Are you guys aware of this problem? Any suggestion?

Thanks,
Guang

Re: Reason why we don't use higher version of Jetty

[Guang Yang]

I think so. I will send out a patch later.

On Thu, Aug 30, 2018 at 4:17 PM, larry mccay  wrote:

> Did you resolve the issue?
>
> On Thu, Aug 30, 2018 at 7:10 PM Guang Yang  wrote:
>
>> Ok, the issue seems goes away after I upgrade jetty to 9.3.10 on Knox
>> 1.1.0. But there was some issue on Knox 0.13.0 + Jetty 9.3.10.
>>
>> On Tue, Aug 28, 2018 at 1:23 PM, Guang Yang  wrote:
>>
>>> Hey Larry,
>>>
>>> We're using 0.13.0 and running on Linux version 4.4.92 (Debian
>>> 4.9.2-10), the JDK version is 8.
>>>
>>> On Tue, Aug 28, 2018 at 1:01 PM, larry mccay  wrote:
>>>
>>>> Hi Guang -
>>>>
>>>> I do recall this FD issue from looong ago.
>>>> Not sure what was done to address it but I haven't seen it in a few
>>>> years.
>>>>
>>>> What version of Knox are you using?
>>>> What OS and JDK versions are you using?
>>>>
>>>> We generally upgrade jetty based on identification of CVEs on current
>>>> version but also try not to jump too many versions as it is often painful
>>>> to make larger moves due to compatibility issues.
>>>>
>>>> thanks,
>>>>
>>>> --larry
>>>>
>>>> On Tue, Aug 28, 2018 at 3:03 PM, Guang Yang  wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> We're using Knox to proxy WEBHDFS request. The problem is when the
>>>>> client shut down the connection, Knox will throw EofException. According 
>>>>> to
>>>>> our dashboard, tons of file descriptors are leaked. When the open files
>>>>> reach to the limit, Knox can't receive any request because there is not
>>>>> socket available.
>>>>>
>>>>> I dig a little bit and found there is a bug in lower version of Jetty
>>>>> https://github.com/eclipse/jetty.project/issues/634. According that
>>>>> link, the issue has been fixed at Jetty 9.3.10, so I upgrade Jetty in 
>>>>> Knox.
>>>>> But after sometime, Knox throw some other issue
>>>>>
>>>>> 2018-08-28 18:45:57,214 INFO  hadoop.gateway 
>>>>> (DefaultHaDispatch.java:executeRequest(93))
>>>>> - Could not connect to server: http://datanode/webhdfs/v1/sompath
>>>>> org.eclipse.jetty.io.EofException
>>>>> org.eclipse.jetty.io.EofException
>>>>> at org.eclipse.jetty.io.ChannelEndPoint.flush(
>>>>> ChannelEndPoint.java:197)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(
>>>>> SslConnection.java:839)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
>>>>> shutdownOutput(SslConnection.java:928)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(
>>>>> SslConnection.java:967)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(
>>>>> SslConnection.java:893)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
>>>>> shutdownOutput(SslConnection.java:928)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(
>>>>> SslConnection.java:967)
>>>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(
>>>>> SslConnection.java:893)
>>>>>
>>>>> And then Knox stops working again. My question is did anybody see the
>>>>> FD leak issue before? and have we ever tried to update Jetty to a higher
>>>>> version?
>>>>>
>>>>> Thanks,
>>>>> Guang
>>>>>
>>>>
>>>>
>>>
>>

Re: Reason why we don't use higher version of Jetty

[Guang Yang]

Ok, the issue seems goes away after I upgrade jetty to 9.3.10 on Knox
1.1.0. But there was some issue on Knox 0.13.0 + Jetty 9.3.10.

On Tue, Aug 28, 2018 at 1:23 PM, Guang Yang  wrote:

> Hey Larry,
>
> We're using 0.13.0 and running on Linux version 4.4.92 (Debian 4.9.2-10),
> the JDK version is 8.
>
> On Tue, Aug 28, 2018 at 1:01 PM, larry mccay  wrote:
>
>> Hi Guang -
>>
>> I do recall this FD issue from looong ago.
>> Not sure what was done to address it but I haven't seen it in a few years.
>>
>> What version of Knox are you using?
>> What OS and JDK versions are you using?
>>
>> We generally upgrade jetty based on identification of CVEs on current
>> version but also try not to jump too many versions as it is often painful
>> to make larger moves due to compatibility issues.
>>
>> thanks,
>>
>> --larry
>>
>> On Tue, Aug 28, 2018 at 3:03 PM, Guang Yang  wrote:
>>
>>> Hi guys,
>>>
>>> We're using Knox to proxy WEBHDFS request. The problem is when the
>>> client shut down the connection, Knox will throw EofException. According to
>>> our dashboard, tons of file descriptors are leaked. When the open files
>>> reach to the limit, Knox can't receive any request because there is not
>>> socket available.
>>>
>>> I dig a little bit and found there is a bug in lower version of Jetty
>>> https://github.com/eclipse/jetty.project/issues/634. According that
>>> link, the issue has been fixed at Jetty 9.3.10, so I upgrade Jetty in Knox.
>>> But after sometime, Knox throw some other issue
>>>
>>> 2018-08-28 18:45:57,214 INFO  hadoop.gateway
>>> (DefaultHaDispatch.java:executeRequest(93)) - Could not connect to
>>> server: http://datanode/webhdfs/v1/sompath org.eclipse.jetty.io
>>> .EofException
>>> org.eclipse.jetty.io.EofException
>>> at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.j
>>> ava:197)
>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flu
>>> sh(SslConnection.java:839)
>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shu
>>> tdownOutput(SslConnection.java:928)
>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.clo
>>> se(SslConnection.java:967)
>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flu
>>> sh(SslConnection.java:893)
>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shu
>>> tdownOutput(SslConnection.java:928)
>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.clo
>>> se(SslConnection.java:967)
>>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flu
>>> sh(SslConnection.java:893)
>>>
>>> And then Knox stops working again. My question is did anybody see the FD
>>> leak issue before? and have we ever tried to update Jetty to a higher
>>> version?
>>>
>>> Thanks,
>>> Guang
>>>
>>
>>
>

Re: Reason why we don't use higher version of Jetty

[Guang Yang]

Hey Larry,

We're using 0.13.0 and running on Linux version 4.4.92 (Debian 4.9.2-10),
the JDK version is 8.

On Tue, Aug 28, 2018 at 1:01 PM, larry mccay  wrote:

> Hi Guang -
>
> I do recall this FD issue from looong ago.
> Not sure what was done to address it but I haven't seen it in a few years.
>
> What version of Knox are you using?
> What OS and JDK versions are you using?
>
> We generally upgrade jetty based on identification of CVEs on current
> version but also try not to jump too many versions as it is often painful
> to make larger moves due to compatibility issues.
>
> thanks,
>
> --larry
>
> On Tue, Aug 28, 2018 at 3:03 PM, Guang Yang  wrote:
>
>> Hi guys,
>>
>> We're using Knox to proxy WEBHDFS request. The problem is when the client
>> shut down the connection, Knox will throw EofException. According to our
>> dashboard, tons of file descriptors are leaked. When the open files reach
>> to the limit, Knox can't receive any request because there is not socket
>> available.
>>
>> I dig a little bit and found there is a bug in lower version of Jetty
>> https://github.com/eclipse/jetty.project/issues/634. According that
>> link, the issue has been fixed at Jetty 9.3.10, so I upgrade Jetty in Knox.
>> But after sometime, Knox throw some other issue
>>
>> 2018-08-28 18:45:57,214 INFO  hadoop.gateway
>> (DefaultHaDispatch.java:executeRequest(93)) - Could not connect to
>> server: http://datanode/webhdfs/v1/sompath org.eclipse.jetty.io.EofExcept
>> ion
>> org.eclipse.jetty.io.EofException
>> at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.
>> java:197)
>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
>> flush(SslConnection.java:839)
>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shu
>> tdownOutput(SslConnection.java:928)
>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
>> close(SslConnection.java:967)
>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
>> flush(SslConnection.java:893)
>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shu
>> tdownOutput(SslConnection.java:928)
>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
>> close(SslConnection.java:967)
>> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
>> flush(SslConnection.java:893)
>>
>> And then Knox stops working again. My question is did anybody see the FD
>> leak issue before? and have we ever tried to update Jetty to a higher
>> version?
>>
>> Thanks,
>> Guang
>>
>
>

Reason why we don't use higher version of Jetty

[Guang Yang]

Hi guys,

We're using Knox to proxy WEBHDFS request. The problem is when the client
shut down the connection, Knox will throw EofException. According to our
dashboard, tons of file descriptors are leaked. When the open files reach
to the limit, Knox can't receive any request because there is not socket
available.

I dig a little bit and found there is a bug in lower version of Jetty
https://github.com/eclipse/jetty.project/issues/634. According that link,
the issue has been fixed at Jetty 9.3.10, so I upgrade Jetty in Knox. But
after sometime, Knox throw some other issue

2018-08-28 18:45:57,214 INFO  hadoop.gateway
(DefaultHaDispatch.java:executeRequest(93)) - Could not connect to server:
http://datanode/webhdfs/v1/sompath org.eclipse.jetty.io.EofException
org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:197)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:839)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shutdownOutput(SslConnection.java:928)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(SslConnection.java:967)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:893)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shutdownOutput(SslConnection.java:928)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(SslConnection.java:967)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:893)

And then Knox stops working again. My question is did anybody see the FD
leak issue before? and have we ever tried to update Jetty to a higher
version?

Thanks,
Guang

Facing Unknown SSL protocol error when calling Knox

[Guang Yang]

Hi guys,

Our Knox service was running well for a long time, but suddenly starting
from this afternoon, we kept seeing this error *curl: (35) Unknown SSL
protocol error in connection to localhost:19943* when curl Knox. Do you
guys have any idea what's the possible reason?


Thanks,

Guang

Re: Job history ui rewrite rule in yarnui

[Guang Yang]

Hi guys,

About the job urls, I found three more non-functional links. They are links
for jobmanager.(err/log/out) in the page of node container logs, like
http://hadoopworker001:8042/node/containerlogs/container_xxx/user_xxx.

Their original urls are like this http://hadoopworker1677-sjc1.prod.uber.internal:8042/node/containerlogs/container_e535_1521074600645_578887_01_01/kedar/jobmanager.err/?start=-4096>
">jobmanager.err : Total file length is 0 bytes.. As there is no node
host and port info in this uri, so when doing the rewrite, it can't be
written to the right url. Any suggestion about how to fix this issue?

Another interesting finding is, actually we have the node host and port
info in the web page url which contains these links, which is like "
https://knox/gateway/sandbox/yarn/nodemanager/node/containerlogs/container_xxx/user_xxx?scheme=http=hadoopworker001-sjc1.prod.uber.internal=8042;.
So that means if there is a way we can get the parameter from the root url
when rewrite some other urls, we can probably fix that problem.

Do you have some suggestion? Appreciated.

Best,
Guang

On Mon, Mar 5, 2018 at 5:07 PM, Guang Yang <k...@uber.com> wrote:

> Any suggestion, guys?
>
> On Fri, Mar 2, 2018 at 1:28 PM, Guang Yang <k...@uber.com> wrote:
>
>> Hi Sandeep,
>>
>> Thanks for looking into it.
>>
>> Yes, I tried that before, the value can be replaced correctly, but the
>> webpage is just stuck there, no redirect happens.
>>
>> The result in html is > content="https://host:port/something;>, which however, should be > http-equiv="refresh" content="*1; url=*https://host:port/something;>. It
>> seems the html meta refresh format requires *'1; url='*, where the 1
>> means stop for 1 seconds before refresh.
>>
>> Thanks,
>> Guang
>>
>> On Fri, Mar 2, 2018 at 6:49 AM, Sandeep Moré <moresand...@gmail.com>
>> wrote:
>>
>>> Hello Guang,
>>>
>>> This does look like a bug, after some digging it appears it was as a
>>> result of KNOX-973.
>>>
>>> Have you tried using
>>>
>>> 
>>>
>>> I am curious to see what you get.
>>>
>>>
>>> Best,
>>> Sandeep
>>>
>>>
>>>
>>>
>>> On Thu, Mar 1, 2018 at 4:30 PM, Guang Yang <k...@uber.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm currently working on the Map Reduce Job History UI rewrite rules,
>>>> and found several potential bugs here.
>>>>
>>>> 
>>>> For this rewrite template
>>>> <https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/yarnui/2.7.0/rewrite.xml#L104>,
>>>> let's not say what the `jobstory` is here for now. I think the target
>>>> url should start with something like {$frontend[url]}, just like other OUT
>>>> rules, because the previous one doesn't specify the deployment/environment
>>>> after word `gateway`.
>>>>
>>>> But after I change it to , the variable
>>>> {$frontend[url]} is not replaced with the right value, it's just literal
>>>> `{$frontend[url]}` in the target url. And I found that only when the
>>>> variable following the double quotes, it can be replaced, otherwise it just
>>>> stays there as literal text.
>>>>
>>>> My question is, anyone knows how to fix this bug? Or how to get 
>>>> {$frontend[url]}
>>>> replaced with right value even it's not at the beginning of the template?
>>>>
>>>> Btw, I think the right template should be .
>>>>
>>>> Appreciate for any help.
>>>>
>>>> Thanks,
>>>> Guang
>>>>
>>>>
>>>>
>>>
>>
>

Re: Job history ui rewrite rule in yarnui

[Guang Yang]

Any suggestion, guys?

On Fri, Mar 2, 2018 at 1:28 PM, Guang Yang <k...@uber.com> wrote:

> Hi Sandeep,
>
> Thanks for looking into it.
>
> Yes, I tried that before, the value can be replaced correctly, but the
> webpage is just stuck there, no redirect happens.
>
> The result in html is https://host:port
> /something">, which however, should be  content="*1; url=*https://host:port/something;>. It seems the html meta
> refresh format requires *'1; url='*, where the 1 means stop for 1 seconds
> before refresh.
>
> Thanks,
> Guang
>
> On Fri, Mar 2, 2018 at 6:49 AM, Sandeep Moré <moresand...@gmail.com>
> wrote:
>
>> Hello Guang,
>>
>> This does look like a bug, after some digging it appears it was as a
>> result of KNOX-973.
>>
>> Have you tried using
>>
>> 
>>
>> I am curious to see what you get.
>>
>>
>> Best,
>> Sandeep
>>
>>
>>
>>
>> On Thu, Mar 1, 2018 at 4:30 PM, Guang Yang <k...@uber.com> wrote:
>>
>>> Hi,
>>>
>>> I'm currently working on the Map Reduce Job History UI rewrite rules,
>>> and found several potential bugs here.
>>>
>>> 
>>> For this rewrite template
>>> <https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/yarnui/2.7.0/rewrite.xml#L104>,
>>> let's not say what the `jobstory` is here for now. I think the target
>>> url should start with something like {$frontend[url]}, just like other OUT
>>> rules, because the previous one doesn't specify the deployment/environment
>>> after word `gateway`.
>>>
>>> But after I change it to , the variable
>>> {$frontend[url]} is not replaced with the right value, it's just literal
>>> `{$frontend[url]}` in the target url. And I found that only when the
>>> variable following the double quotes, it can be replaced, otherwise it just
>>> stays there as literal text.
>>>
>>> My question is, anyone knows how to fix this bug? Or how to get 
>>> {$frontend[url]}
>>> replaced with right value even it's not at the beginning of the template?
>>>
>>> Btw, I think the right template should be .
>>>
>>> Appreciate for any help.
>>>
>>> Thanks,
>>> Guang
>>>
>>>
>>>
>>
>

Re: Job history ui rewrite rule in yarnui

[Guang Yang]

Hi Sandeep,

Thanks for looking into it.

Yes, I tried that before, the value can be replaced correctly, but the
webpage is just stuck there, no redirect happens.

The result in html is https://host:port/something;>,
which however, should be https://host:port/something;>. It seems the html meta refresh format
requires *'1; url='*, where the 1 means stop for 1 seconds before refresh.

Thanks,
Guang

On Fri, Mar 2, 2018 at 6:49 AM, Sandeep Moré <moresand...@gmail.com> wrote:

> Hello Guang,
>
> This does look like a bug, after some digging it appears it was as a
> result of KNOX-973.
>
> Have you tried using
>
> 
>
> I am curious to see what you get.
>
>
> Best,
> Sandeep
>
>
>
>
> On Thu, Mar 1, 2018 at 4:30 PM, Guang Yang <k...@uber.com> wrote:
>
>> Hi,
>>
>> I'm currently working on the Map Reduce Job History UI rewrite rules, and
>> found several potential bugs here.
>>
>> 
>> For this rewrite template
>> <https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/yarnui/2.7.0/rewrite.xml#L104>,
>> let's not say what the `jobstory` is here for now. I think the target
>> url should start with something like {$frontend[url]}, just like other OUT
>> rules, because the previous one doesn't specify the deployment/environment
>> after word `gateway`.
>>
>> But after I change it to , the variable
>> {$frontend[url]} is not replaced with the right value, it's just literal
>> `{$frontend[url]}` in the target url. And I found that only when the
>> variable following the double quotes, it can be replaced, otherwise it just
>> stays there as literal text.
>>
>> My question is, anyone knows how to fix this bug? Or how to get 
>> {$frontend[url]}
>> replaced with right value even it's not at the beginning of the template?
>>
>> Btw, I think the right template should be .
>>
>> Appreciate for any help.
>>
>> Thanks,
>> Guang
>>
>>
>>
>

Job history ui rewrite rule in yarnui

[Guang Yang]

Hi,

I'm currently working on the Map Reduce Job History UI rewrite rules, and
found several potential bugs here.


For this rewrite template
,
let's not say what the `jobstory` is here for now. I think the target url
should start with something like {$frontend[url]}, just like other OUT
rules, because the previous one doesn't specify the deployment/environment
after word `gateway`.

But after I change it to , the variable {$frontend[url]}
is not replaced with the right value, it's just literal `{$frontend[url]}`
in the target url. And I found that only when the variable following the
double quotes, it can be replaced, otherwise it just stays there as literal
text.

My question is, anyone knows how to fix this bug? Or how to get
{$frontend[url]}
replaced with right value even it's not at the beginning of the template?

Btw, I think the right template should be .

Appreciate for any help.

Thanks,
Guang