Re: turn off debug logging from org.apache.http.wire

[Guang Yang]

Ok guys, I finally figure out the reason.

It's because we introduced logback from one of our internal packages, so
Knox was using logback instead of log4j underlying, and that's why no
matter how we modify gateway-log4j.properties, nothing changed. By simply
removing that package, Knox is using log4j now and all the DEBUG logs go
away.

Thanks for all the help guys, and this page

basically has all the possible reasons of failing to turn off DEBUG logs.

On Wed, Sep 5, 2018 at 11:34 AM, larry mccay  wrote:

> Change it back to this and see what happens.
> Another thing to check is that - if you are in a kerberized environment is
> whether any of the kerberos DEBUG settings are set via environment
> variables or krb5.conf, etc.
>
> Lastly, make sure that you are actually using the log4j.propoerties file
> that you think that you are.
> If you are using init.d to start the gateway make sure that you haven't
> changed where the gateway-log4j.properties is being found.
>
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements.  See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership.  The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License.  You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> app.log.dir=${launcher.dir}/../logs
> app.log.file=${launcher.name}.log
> app.audit.file=${launcher.name}-audit.log
>
> log4j.rootLogger=ERROR, drfa
>
> log4j.logger.org.apache.knox.gateway=INFO
> #log4j.logger.org.apache.knox.gateway=DEBUG
>
> #log4j.logger.org.eclipse.jetty=DEBUG
> #log4j.logger.org.apache.shiro=DEBUG
> #log4j.logger.org.apache.http=DEBUG
> #log4j.logger.org.apache.http.client=DEBUG
> #log4j.logger.org.apache.http.headers=DEBUG
> #log4j.logger.org.apache.http.wire=DEBUG
>
> log4j.appender.stdout=org.apache.log4j.ConsoleAppender
> log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
> log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
> %c{2}: %m%n
>
> log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.drfa.File=${app.log.dir}/${app.log.file}
> log4j.appender.drfa.DatePattern=.-MM-dd
> log4j.appender.drfa.layout=org.apache.log4j.PatternLayout
> log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2}
> (%F:%M(%L)) - %m%n
>
> log4j.logger.audit=INFO, auditfile
> log4j.appender.auditfile=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.auditfile.File=${app.log.dir}/${app.audit.file}
> log4j.appender.auditfile.Append = true
> log4j.appender.auditfile.DatePattern = '.'-MM-dd
> log4j.appender.auditfile.layout = org.apache.knox.gateway.audit.
> log4j.layout.AuditLayout
>
> #log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
> #log4j.additivity.org.apache.knox.gateway.access=false
>
> #log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
> #log4j.additivity.org.apache.knox.gateway.http=false
> ##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
> ##log4j.logger.org.apache.knox.gateway.http.response.headers=OFF
> ##log4j.logger.org.apache.knox.gateway.http.request.body=OFF
> ##log4j.logger.org.apache.knox.gateway.http.response.body=OFF
>
> #log4j.logger.org.apache.http.wire=DEBUG,httpclient
> #log4j.additivity.org.apache.http.wire=false
>
> #log4j.appender.httpaccess=org.apache.log4j.DailyRollingFileAppender
> #log4j.appender.httpaccess.File=${app.log.dir}/${launcher.name
> }-http-access.log
> #log4j.appender.httpaccess.DatePattern=.-MM-dd
> #log4j.appender.httpaccess.layout=org.apache.log4j.PatternLayout
> #log4j.appender.httpaccess.layout.ConversionPattern=%d{ISO8601}|%t|%m%n
>
> #log4j.appender.httpserver=org.apache.log4j.DailyRollingFileAppender
> #log4j.appender.httpserver.File=${app.log.dir}/${launcher.name
> }-http-server.log
> #log4j.appender.httpserver.DatePattern=.-MM-dd
> #log4j.appender.httpserver.layout=org.apache.log4j.PatternLayout
> #log4j.appender.httpserver.layout.ConversionPattern=%d{ISO8601}|%t|%m%n
>
> #log4j.appender.httpclient=org.apache.log4j.DailyRollingFileAppender
> #log4j.appender.httpclient.File=${app.log.dir}/${launcher.name
> }-http-client.log
> #log4j.appender.httpclient.DatePattern=.-MM-dd
> #log4j.appender.httpclient.layout=org.apache.log4j.PatternLayout
> #log4j.appender.httpclient.layout.ConversionPattern=%d{ISO8601}|%t|%m%n
>
> # Apache Shiro Related logging - KNOX-757
> 

Re: Knox wrong query param encoding help

[Theyaa Matti]

Thank you Sandeep for the quick reply. I created the new class and modified
services.xml for yarnui to point to it.

I could see the class being loaded in the logs but the url encoding is the
same, which tells me that the url is not passing through the new class.

Here is the dispatch tag from service.xml




and Here is the code for that classs


package org.apache.knox.gateway.dispatch;

import org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch;

import javax.servlet.http.HttpServletRequest;
import java.net.URI;
import java.net.URLDecoder;
import javax.servlet.ServletException;

/**
 * Dispatch which decodes the outgoing URLs (to services).
 * This is useful in cases where the url is picked up
 * from the query parameter and is already encoded.
 *
 * @since 1.1.0
 */
public class RMHaURLDecodingDispatch extends RMUIHaDispatch {

public RMHaURLDecodingDispatch() throws ServletException{
super();
}

@Override
public URI getDispatchUrl(final HttpServletRequest request) {
String decoded;

try {
decoded =
URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
} catch (final Exception e) {
  /* fall back in case of exception */
decoded = request.getRequestURL().toString();
}

final StringBuffer str = new StringBuffer(decoded);
final String query = request.getQueryString();
if ( query != null ) {
str.append('?');
str.append(query);
}
final URI url = URI.create(str.toString());
return url;
}
}


Regards,



On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré  wrote:

> You can extend RMUIHaDispatch class and override getDispatchUrl() method
> from URLDecodingDispatch
>
> Best,
> Sandeep
>
> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti  wrote:
>
>> Yes I am using HA for resource manager and in that case what should I do
>> to the fix this issue?
>>
>> Best,
>>
>> Theyaa.
>>
>>
>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré 
>> wrote:
>>
>>> Are you using HA setup ? in which case org.apache.
>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>
>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti 
>>> wrote:
>>>
 I created the class as follows:

 package org.apache.knox.gateway.dispatch;

 import javax.servlet.http.HttpServletRequest;
 import java.net.URI;
 import java.net.URLDecoder;
 import org.apache.hadoop.gateway.dispatch.DefaultDispatch;

 /**
  * Dispatch which decodes the outgoing URLs (to services).
  * This is useful in cases where the url is picked up
  * from the query parameter and is already encoded.
  *
  * @since 1.1.0
  */
 public class URLDecodingDispatch extends DefaultDispatch {

 public URLDecodingDispatch() {
 super();
 }

 @Override
 public URI getDispatchUrl(final HttpServletRequest request) {
 String decoded;

 try {
 decoded = 
 URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
 } catch (final Exception e) {
   /* fall back in case of exception */
 decoded = request.getRequestURL().toString();
 }

 final StringBuffer str = new StringBuffer(decoded);
 final String query = request.getQueryString();
 if ( query != null ) {
 str.append('?');
 str.append(query);
 }
 final URI url = URI.create(str.toString());
 return url;
 }
 }


 packaged it into a jar and uploaded it to the knox lib dir. I can see
 Knox loading the jar file at start:
 resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
 abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
 can=/knox/lib/knox-1.0-SNAPSHOT.jar

 I modified the service.xml for yarnui as follows.

 >>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>



 Deleted everything under deployments and restarted knox.


 I do not see any changes in the behavior and do not see that class
 being called. Do you please know what I am missing?


 Regards,






 On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré 
 wrote:

> URLDecodingDispatch extends DefaultDispatch so you should be there.
>
> Best,
> Sandeep
>
> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti 
> wrote:
>
>> That should be a good idea, but I still have an issue with adding the
>> custom dispatch to the yarnui service.xml. I already have an existing 
>> class
>> there > classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>
>> Should I replace the DefaultDispatch or I can have more than one
>> there?

Re: Knox wrong query param encoding help

[Sandeep Moré]

You can extend RMUIHaDispatch class and override getDispatchUrl() method
from URLDecodingDispatch

Best,
Sandeep

On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti  wrote:

> Yes I am using HA for resource manager and in that case what should I do
> to the fix this issue?
>
> Best,
>
> Theyaa.
>
>
> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré  wrote:
>
>> Are you using HA setup ? in which case org.apache.
>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>
>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti 
>> wrote:
>>
>>> I created the class as follows:
>>>
>>> package org.apache.knox.gateway.dispatch;
>>>
>>> import javax.servlet.http.HttpServletRequest;
>>> import java.net.URI;
>>> import java.net.URLDecoder;
>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>
>>> /**
>>>  * Dispatch which decodes the outgoing URLs (to services).
>>>  * This is useful in cases where the url is picked up
>>>  * from the query parameter and is already encoded.
>>>  *
>>>  * @since 1.1.0
>>>  */
>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>
>>> public URLDecodingDispatch() {
>>> super();
>>> }
>>>
>>> @Override
>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>> String decoded;
>>>
>>> try {
>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), 
>>> "UTF-8" );
>>> } catch (final Exception e) {
>>>   /* fall back in case of exception */
>>> decoded = request.getRequestURL().toString();
>>> }
>>>
>>> final StringBuffer str = new StringBuffer(decoded);
>>> final String query = request.getQueryString();
>>> if ( query != null ) {
>>> str.append('?');
>>> str.append(query);
>>> }
>>> final URI url = URI.create(str.toString());
>>> return url;
>>> }
>>> }
>>>
>>>
>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>> Knox loading the jar file at start:
>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>
>>> I modified the service.xml for yarnui as follows.
>>>
>>> >> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>
>>>
>>>
>>> Deleted everything under deployments and restarted knox.
>>>
>>>
>>> I do not see any changes in the behavior and do not see that class being
>>> called. Do you please know what I am missing?
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré 
>>> wrote:
>>>
 URLDecodingDispatch extends DefaultDispatch so you should be there.

 Best,
 Sandeep

 On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti 
 wrote:

> That should be a good idea, but I still have an issue with adding the
> custom dispatch to the yarnui service.xml. I already have an existing 
> class
> there  classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>
> Should I replace the DefaultDispatch or I can have more than one there?
>
> Regards,
>
>
> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com>
> wrote:
>
>> In that case you might have to build the jar yourself or preferably
>> use your custom dispatch similar to URLDecodingDispatch.
>>
>> Regards
>> Dhruv
>>
>> On Wednesday, September 5, 2018, Theyaa Matti 
>> wrote:
>>
>>> Sorry 0.12.0
>>>
>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti 
>>> wrote:
>>>
 I am using Knox 1.12.0

 On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré 
 wrote:

> You can try
> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch 
> in
> service.xml ( Knox 1.1.0)
>
> Best,
> Sandeep
>
> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti 
> wrote:
>
>> I am having issues with Knox encoding the following URL when
>> parsing html content.
>>
>>
>> /proxy/application_33323323_0001/stages/stage?id=0attempt=0
>>
>>
>> This URL is generated by Yarn to track a running spark job. When
>> applying regular rules to the above URL, Knox encodes the URL to:
>>
>>
>> /proxy/application_33323323
>> _0001/stages/stage?amp%3Battempt=0=0
>>
>>
>> Which makes the URL unusable and leads to a 404. To make the URL
>> work I have to remove "amp%3B" and then it works.
>>
>>
>> Below are the filter and the rule I am using for this URL and I
>> appreciate your help/comments for a remediation.
>>
>>
>> 
>> 
>> > 

Re: Knox wrong query param encoding help

[Theyaa Matti]

Yes I am using HA for resource manager and in that case what should I do to
the fix this issue?

Best,

Theyaa.


On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré  wrote:

> Are you using HA setup ? in which case org.apache.
> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>
> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti  wrote:
>
>> I created the class as follows:
>>
>> package org.apache.knox.gateway.dispatch;
>>
>> import javax.servlet.http.HttpServletRequest;
>> import java.net.URI;
>> import java.net.URLDecoder;
>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>
>> /**
>>  * Dispatch which decodes the outgoing URLs (to services).
>>  * This is useful in cases where the url is picked up
>>  * from the query parameter and is already encoded.
>>  *
>>  * @since 1.1.0
>>  */
>> public class URLDecodingDispatch extends DefaultDispatch {
>>
>> public URLDecodingDispatch() {
>> super();
>> }
>>
>> @Override
>> public URI getDispatchUrl(final HttpServletRequest request) {
>> String decoded;
>>
>> try {
>> decoded = URLDecoder.decode(request.getRequestURL().toString(), 
>> "UTF-8" );
>> } catch (final Exception e) {
>>   /* fall back in case of exception */
>> decoded = request.getRequestURL().toString();
>> }
>>
>> final StringBuffer str = new StringBuffer(decoded);
>> final String query = request.getQueryString();
>> if ( query != null ) {
>> str.append('?');
>> str.append(query);
>> }
>> final URI url = URI.create(str.toString());
>> return url;
>> }
>> }
>>
>>
>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>> Knox loading the jar file at start:
>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>
>> I modified the service.xml for yarnui as follows.
>>
>> > ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>
>>
>>
>> Deleted everything under deployments and restarted knox.
>>
>>
>> I do not see any changes in the behavior and do not see that class being
>> called. Do you please know what I am missing?
>>
>>
>> Regards,
>>
>>
>>
>>
>>
>>
>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré 
>> wrote:
>>
>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti 
>>> wrote:
>>>
 That should be a good idea, but I still have an issue with adding the
 custom dispatch to the yarnui service.xml. I already have an existing class
 there >>> classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
 ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>

 Should I replace the DefaultDispatch or I can have more than one there?

 Regards,


 On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com>
 wrote:

> In that case you might have to build the jar yourself or preferably
> use your custom dispatch similar to URLDecodingDispatch.
>
> Regards
> Dhruv
>
> On Wednesday, September 5, 2018, Theyaa Matti 
> wrote:
>
>> Sorry 0.12.0
>>
>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti 
>> wrote:
>>
>>> I am using Knox 1.12.0
>>>
>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré 
>>> wrote:
>>>
 You can try
 using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch 
 in
 service.xml ( Knox 1.1.0)

 Best,
 Sandeep

 On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti 
 wrote:

> I am having issues with Knox encoding the following URL when
> parsing html content.
>
>
> /proxy/application_33323323_0001/stages/stage?id=0attempt=0
>
>
> This URL is generated by Yarn to track a running spark job. When
> applying regular rules to the above URL, Knox encodes the URL to:
>
>
> /proxy/application_33323323
> _0001/stages/stage?amp%3Battempt=0=0
>
>
> Which makes the URL unusable and leads to a 404. To make the URL
> work I have to remove "amp%3B" and then it works.
>
>
> Below are the filter and the rule I am using for this URL and I
> appreciate your help/comments for a remediation.
>
>
> 
> 
>  rule="YARNUI/yarn/outbound/apps/history3"/>
> 
> 
>
>
> 
> 
>  template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
> 
>
>
> Thank you for your help.
>
>
>

Re: Knox wrong query param encoding help

[Sandeep Moré]

Are you using HA setup ? in which case org.apache.
hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.

On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti  wrote:

> I created the class as follows:
>
> package org.apache.knox.gateway.dispatch;
>
> import javax.servlet.http.HttpServletRequest;
> import java.net.URI;
> import java.net.URLDecoder;
> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>
> /**
>  * Dispatch which decodes the outgoing URLs (to services).
>  * This is useful in cases where the url is picked up
>  * from the query parameter and is already encoded.
>  *
>  * @since 1.1.0
>  */
> public class URLDecodingDispatch extends DefaultDispatch {
>
> public URLDecodingDispatch() {
> super();
> }
>
> @Override
> public URI getDispatchUrl(final HttpServletRequest request) {
> String decoded;
>
> try {
> decoded = URLDecoder.decode(request.getRequestURL().toString(), 
> "UTF-8" );
> } catch (final Exception e) {
>   /* fall back in case of exception */
> decoded = request.getRequestURL().toString();
> }
>
> final StringBuffer str = new StringBuffer(decoded);
> final String query = request.getQueryString();
> if ( query != null ) {
> str.append('?');
> str.append(query);
> }
> final URI url = URI.create(str.toString());
> return url;
> }
> }
>
>
> packaged it into a jar and uploaded it to the knox lib dir. I can see Knox
> loading the jar file at start:
> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>
> I modified the service.xml for yarnui as follows.
>
>  ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>
>
>
> Deleted everything under deployments and restarted knox.
>
>
> I do not see any changes in the behavior and do not see that class being
> called. Do you please know what I am missing?
>
>
> Regards,
>
>
>
>
>
>
> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré  wrote:
>
>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>
>> Best,
>> Sandeep
>>
>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti 
>> wrote:
>>
>>> That should be a good idea, but I still have an issue with adding the
>>> custom dispatch to the yarnui service.xml. I already have an existing class
>>> there >> classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>
>>> Should I replace the DefaultDispatch or I can have more than one there?
>>>
>>> Regards,
>>>
>>>
>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com> wrote:
>>>
 In that case you might have to build the jar yourself or preferably use
 your custom dispatch similar to URLDecodingDispatch.

 Regards
 Dhruv

 On Wednesday, September 5, 2018, Theyaa Matti 
 wrote:

> Sorry 0.12.0
>
> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti 
> wrote:
>
>> I am using Knox 1.12.0
>>
>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré 
>> wrote:
>>
>>> You can try
>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>> service.xml ( Knox 1.1.0)
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti 
>>> wrote:
>>>
 I am having issues with Knox encoding the following URL when
 parsing html content.

 /proxy/application_33323323_0001/stages/stage?id=0attempt=0


 This URL is generated by Yarn to track a running spark job. When
 applying regular rules to the above URL, Knox encodes the URL to:


 /proxy/application_33323323
 _0001/stages/stage?amp%3Battempt=0=0


 Which makes the URL unusable and leads to a 404. To make the URL
 work I have to remove "amp%3B" and then it works.


 Below are the filter and the rule I am using for this URL and I
 appreciate your help/comments for a remediation.


 
 
 >>> rule="YARNUI/yarn/outbound/apps/history3"/>
 
 


 
 
 >>> template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
 


 Thank you for your help.

Re: Knox wrong query param encoding help

[Theyaa Matti]

I created the class as follows:

package org.apache.knox.gateway.dispatch;

import javax.servlet.http.HttpServletRequest;
import java.net.URI;
import java.net.URLDecoder;
import org.apache.hadoop.gateway.dispatch.DefaultDispatch;

/**
 * Dispatch which decodes the outgoing URLs (to services).
 * This is useful in cases where the url is picked up
 * from the query parameter and is already encoded.
 *
 * @since 1.1.0
 */
public class URLDecodingDispatch extends DefaultDispatch {

public URLDecodingDispatch() {
super();
}

@Override
public URI getDispatchUrl(final HttpServletRequest request) {
String decoded;

try {
decoded =
URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
} catch (final Exception e) {
  /* fall back in case of exception */
decoded = request.getRequestURL().toString();
}

final StringBuffer str = new StringBuffer(decoded);
final String query = request.getQueryString();
if ( query != null ) {
str.append('?');
str.append(query);
}
final URI url = URI.create(str.toString());
return url;
}
}


packaged it into a jar and uploaded it to the knox lib dir. I can see Knox
loading the jar file at start:
resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
can=/knox/lib/knox-1.0-SNAPSHOT.jar

I modified the service.xml for yarnui as follows.





Deleted everything under deployments and restarted knox.


I do not see any changes in the behavior and do not see that class being
called. Do you please know what I am missing?


Regards,






On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré  wrote:

> URLDecodingDispatch extends DefaultDispatch so you should be there.
>
> Best,
> Sandeep
>
> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti  wrote:
>
>> That should be a good idea, but I still have an issue with adding the
>> custom dispatch to the yarnui service.xml. I already have an existing class
>> there > classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>
>> Should I replace the DefaultDispatch or I can have more than one there?
>>
>> Regards,
>>
>>
>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com> wrote:
>>
>>> In that case you might have to build the jar yourself or preferably use
>>> your custom dispatch similar to URLDecodingDispatch.
>>>
>>> Regards
>>> Dhruv
>>>
>>> On Wednesday, September 5, 2018, Theyaa Matti 
>>> wrote:
>>>
 Sorry 0.12.0

 On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti 
 wrote:

> I am using Knox 1.12.0
>
> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré 
> wrote:
>
>> You can try
>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>> service.xml ( Knox 1.1.0)
>>
>> Best,
>> Sandeep
>>
>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti 
>> wrote:
>>
>>> I am having issues with Knox encoding the following URL when parsing
>>> html content.
>>>
>>> /proxy/application_33323323_0001/stages/stage?id=0attempt=0
>>>
>>>
>>> This URL is generated by Yarn to track a running spark job. When
>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>
>>>
>>> /proxy/application_33323323
>>> _0001/stages/stage?amp%3Battempt=0=0
>>>
>>>
>>> Which makes the URL unusable and leads to a 404. To make the URL
>>> work I have to remove "amp%3B" and then it works.
>>>
>>>
>>> Below are the filter and the rule I am using for this URL and I
>>> appreciate your help/comments for a remediation.
>>>
>>>
>>> 
>>> 
>>> >> rule="YARNUI/yarn/outbound/apps/history3"/>
>>> 
>>> 
>>>
>>>
>>> 
>>> 
>>> >> template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>> 
>>>
>>>
>>> Thank you for your help.
>>>
>>>
>>>

Re: Knox wrong query param encoding help

[Theyaa Matti]

That should be a good idea, but I still have an issue with adding the
custom dispatch to the yarnui service.xml. I already have an existing class
there 

Should I replace the DefaultDispatch or I can have more than one there?

Regards,


On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com> wrote:

> In that case you might have to build the jar yourself or preferably use
> your custom dispatch similar to URLDecodingDispatch.
>
> Regards
> Dhruv
>
> On Wednesday, September 5, 2018, Theyaa Matti 
> wrote:
>
>> Sorry 0.12.0
>>
>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti 
>> wrote:
>>
>>> I am using Knox 1.12.0
>>>
>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré 
>>> wrote:
>>>
 You can try
 using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
 service.xml ( Knox 1.1.0)

 Best,
 Sandeep

 On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti 
 wrote:

> I am having issues with Knox encoding the following URL when parsing
> html content.
>
> /proxy/application_33323323_0001/stages/stage?id=0attempt=0
>
>
> This URL is generated by Yarn to track a running spark job. When
> applying regular rules to the above URL, Knox encodes the URL to:
>
>
> /proxy/application_33323323_0001/stages/stage?amp%3Battempt=0=0
>
>
> Which makes the URL unusable and leads to a 404. To make the URL work
> I have to remove "amp%3B" and then it works.
>
>
> Below are the filter and the rule I am using for this URL and I
> appreciate your help/comments for a remediation.
>
>
> 
> 
>  rule="YARNUI/yarn/outbound/apps/history3"/>
> 
> 
>
>
> 
> 
>  template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
> 
>
>
> Thank you for your help.
>
>
>