Sure thank you David. The issue seemed to be that the request object gets
corrupted and becomes null by the time it reaches my custom dispatch class.
You can find my code above for your reference.
On Tue, Sep 25, 2018 at 5:24 PM David Morin <morin.david....@gmail.com>
wrote:
> ok :(
> So, I'll work on it on my side this week
> I keep you in touch if I succeed in fixing that issue
>
> Le mar. 25 sept. 2018 à 23:10, Theyaa Matti <theyaama...@gmail.com> a
> écrit :
>
>> No, I am waiting for the Knox dev team to provide a solution or
>> suggestion to try.
>>
>> I followed all provided suggestions and nothing worked
>>
>> Sent from my iPhone
>>
>> On Sep 25, 2018, at 4:41 PM, David Morin <morin.david....@gmail.com>
>> wrote:
>>
>> I've faced to the same issue with the execution of Spark jobs on Yarn.
>> 404 error with url that contains this: "amp%3B"
>> Have you succeeded in resolving that ?
>>
>> Regards,
>> David
>>
>> Le ven. 7 sept. 2018 à 18:32, Theyaa Matti <theyaama...@gmail.com> a
>> écrit :
>>
>>> I have made the necessary modifications to use my custom dispatcher.
>>> However I am getting NPE from HttpServletRequest.getRequestURL() when the
>>> queryString contains special characters like "amp%3B". In this case the
>>> request object is null.
>>>
>>> Any suggestions?
>>>
>>> Regards,
>>>
>>> On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <moresand...@gmail.com>
>>> wrote:
>>>
>>>> You can extend RMUIHaDispatch class and override getDispatchUrl()
>>>> method from URLDecodingDispatch
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <theyaama...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yes I am using HA for resource manager and in that case what should I
>>>>> do to the fix this issue?
>>>>>
>>>>> Best,
>>>>>
>>>>> Theyaa.
>>>>>
>>>>>
>>>>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <moresand...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Are you using HA setup ? in which case org.apache.
>>>>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>>>>
>>>>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <theyaama...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I created the class as follows:
>>>>>>>
>>>>>>> package org.apache.knox.gateway.dispatch;
>>>>>>>
>>>>>>> import javax.servlet.http.HttpServletRequest;
>>>>>>> import java.net.URI;
>>>>>>> import java.net.URLDecoder;
>>>>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>>>>
>>>>>>> /**
>>>>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>>>>> * This is useful in cases where the url is picked up
>>>>>>> * from the query parameter and is already encoded.
>>>>>>> *
>>>>>>> * @since 1.1.0
>>>>>>> */
>>>>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>>>>
>>>>>>> public URLDecodingDispatch() {
>>>>>>> super();
>>>>>>> }
>>>>>>>
>>>>>>> @Override
>>>>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>>>>> String decoded;
>>>>>>>
>>>>>>> try {
>>>>>>> decoded =
>>>>>>> URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>>>>> } catch (final Exception e) {
>>>>>>> /* fall back in case of exception */
>>>>>>> decoded = request.getRequestURL().toString();
>>>>>>> }
>>>>>>>
>>>>>>> final StringBuffer str = new StringBuffer(decoded);
>>>>>>> final String query = request.getQueryString();
>>>>>>> if ( query != null ) {
>>>>>>> str.append('?');
>>>>>>> str.append(query);
>>>>>>> }
>>>>>>> final URI url = URI.create(str.toString());
>>>>>>> return url;
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> packaged it into a jar and uploaded it to the knox lib dir. I can
>>>>>>> see Knox loading the jar file at start:
>>>>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) -
>>>>>>> ALIAS abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>>>>
>>>>>>> I modified the service.xml for yarnui as follows.
>>>>>>>
>>>>>>> <dispatch
>>>>>>> classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Deleted everything under deployments and restarted knox.
>>>>>>>
>>>>>>>
>>>>>>> I do not see any changes in the behavior and do not see that class
>>>>>>> being called. Do you please know what I am missing?
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <moresand...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Sandeep
>>>>>>>>
>>>>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <theyaama...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> That should be a good idea, but I still have an issue with adding
>>>>>>>>> the custom dispatch to the yarnui service.xml. I already have an
>>>>>>>>> existing
>>>>>>>>> class there <dispatch
>>>>>>>>> classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>>>>> ha-classname=
>>>>>>>>> "org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>>>>
>>>>>>>>> Should I replace the DefaultDispatch or I can have more than one
>>>>>>>>> there?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> In that case you might have to build the jar yourself or
>>>>>>>>>> preferably use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Dhruv
>>>>>>>>>>
>>>>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <
>>>>>>>>>> theyaama...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry 0.12.0
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <
>>>>>>>>>>> theyaama...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <
>>>>>>>>>>>> moresand...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> You can try
>>>>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>>>>>>>>>>> dispatch in
>>>>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <
>>>>>>>>>>>>> theyaama...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>>>>> parsing html content.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This URL is generated by Yarn to track a running spark job.
>>>>>>>>>>>>>> When applying regular rules to the above URL, Knox encodes the
>>>>>>>>>>>>>> URL to:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the
>>>>>>>>>>>>>> URL work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Below are the filter and the rule I am using for this URL and
>>>>>>>>>>>>>> I appreciate your help/comments for a remediation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*"
>>>>>>>>>>>>>> rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>>>>> </content>
>>>>>>>>>>>>>> </filter>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>>> <rewrite
>>>>>>>>>>>>>> template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>>> </rule>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
