Re: Dispatcher class for Grafana

[Divya Narayan]

Hi Sandeep,

In gateway.log I get this:

2018-08-17 04:49:38,760 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) 
- Received request: GET /grafana/api/datasources/proxy/1/ws/v1/timeline/metrics
2018-08-17 04:49:38,763 DEBUG hadoop.gateway 
(UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: 
https://192.168.134.214:8443/gateway/default/grafana/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=30.op=*.TotalCount=%=namenode==1534470578=1534481378,
 direction: IN via explicit rule: GRAFANAUI/grafana/inbound/queryv to URL: 
http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs%3D30.op%3D*.TotalCount=%2525=namenode=1534470578=1534481378
2018-08-17 04:49:38,764 DEBUG hadoop.gateway 
(UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: 
https://192.168.134.214:8443/gateway/default/grafana/dashboard/db/hdfs-topn, 
direction: IN via implicit rule: GRAFANAUI/grafana/inbound/query to URL: 
http://platacc002-mgt-01.gvs.ggn:3000/dashboard/db/hdfs-topn
2018-08-17 04:49:38,764 DEBUG hadoop.gateway 
(DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET 
http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames%3Ddfs.NNTopUserOpCounts.windowMs%3D30.op%3D*.TotalCount%26hostname%3D%2525%26appId%3Dnamenode%26startTime%3D1534470578%26endTime%3D1534481378
2018-08-17 04:49:38,770 DEBUG hadoop.gateway 
(DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 
400
2018-08-17 04:49:38,771 DEBUG hadoop.gateway 
(DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 
400

There are some specific pattern of URLs that are breaking like these:

https://192.168.134.214:8443/gateway/default/grafana/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=30.op=*.TotalCount=%=namenode==1534470578=1534481378

https://192.168.134.214:8443/gateway/default/grafana/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=*.TotalCount=%=namenode==1534470578=1534481378

And there are many such patterns. If not URLDecodingDispatch , could you tell 
me which one is the right dispatcher for such URLs.

Thanks
Divya



From: Sandeep Moré mailto:moresand...@gmail.com>>
Reply-To: "user@knox.apache.org" 
mailto:user@knox.apache.org>>
Date: Thursday, 16 August 2018 at 7:07 PM
To: "user@knox.apache.org" 
mailto:user@knox.apache.org>>
Subject: Re: Dispatcher class for Grafana

Hello Divya,

Looking at the URL you posted looks like you are using the wrong dispatch 
class, URLDecodingDispatch is used for URLs that are already encoded, looking 
at your example URL it looks like it is partially encoded (e.g. the query 
params &), in-fact looks like it is not properly encoded.

What error are you getting in gateway.log corresponding to 400 (Bad Request) ?

Best,
Sandeep



On Thu, Aug 16, 2018 at 6:06 AM Divya Narayan 
mailto:divya.nara...@guavus.com>> wrote:
Hi ,

I tried to integrate grafana UI with knox gateway. I used URLDecodingDispatch 
class but there were some calls which were failing like:

http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=__%.user=%=%25=namenode=1534390641=1534401441


The above call on grafana UI failed with below error:

Caused by: java.lang.IllegalArgumentException: Malformed escape pair at index 
141: 
http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=__%.user=%=%25=namenode=1534390641=1534401441
at java.net.URI.create(URI.java:852)
at knoxdisp.URLDecodingDispatch.getDispatchUrl(URLDecodingDispatch.java:49)
at 
org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:132)
at 
org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:115)
at 
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
... 76 more


Now instead of making rewrite rules for each of such URL, I decided to change 
the dispatcher class itself to handle requests with special characters. In 
dispatcher class I added URL encoding to encode all the special characters.
Below is the code that I tried:


@Override

  public URI getDispatchUrl(final HttpServletRequest request) {

String decoded;

String encodedURL = null;


try {

  decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );

} catch (final Exception e) {

  /* fall back in case of exception */

  decoded = request.getRequestURL().toString();

}


final StringBuffer str = new StringBuffer(decoded);

final String query = request.getQueryString();

if ( query != null ) {

 

Knox SSO - throwing null pointer exception on first time login

[Ravikumar, Praveen Krishnamoorthy]

Greetings,

I’m facing a weird issue on enabling knox-sso functionality for exposing UI’s 
running in AWS EMR. On enabling knox-sso, I tried to access the YARNUI site – 
after authentication its throwing java.lang.null.pointer exception but on 
submitting the url again (without clearing the cookies), the application is 
getting landed to the proper page.

When I clear the cookies and access the URL again, Its throwing null pointer 
exception. We are unable to figure whether it is an issue on the Identity 
provider site or in the knox as Im not seeing any error log messages in both 
the ends.

I have attached the log messages. Could any please help me in fixing this?


Gatway.log -> After the first time login


2018-08-16 22:30:14,369 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - 
Received request: GET /yarn/

2018-08-16 22:30:14,370 DEBUG federation.jwt 
(SSOCookieFederationFilter.java:doFilter(114)) - Sending redirect to: 
https://knox.idf-emr-prd.a..com:8446/gateway/knoxsso/api/v1/websso?originalUrl=https://knox.idf-emr-prd.a.x.com:8446/gateway/gate1/yarn/

2018-08-16 22:30:14,493 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - 
Received request: GET /api/v1/websso
















[cid:image001.png@01D43592.A1449230]

Gateway.log -> Log in second time without clearing browser cookies

Second time:


2018-08-16 22:34:15,242 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - 
Received request: GET /yarn/

2018-08-16 22:34:15,242 DEBUG federation.jwt 
(SSOCookieFederationFilter.java:doFilter(114)) - Sending redirect to: 
https://knox.idf-emr-prd.a..com:8446/gateway/knoxsso/api/v1/websso?originalUrl=https://knox.idf-emr-prd.a..com:8446/gateway/gate1/yarn/

2018-08-16 22:34:15,362 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - 
Received request: GET /api/v1/websso

2018-08-16 22:34:17,055 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - 
Received request: POST /api/v1/websso

2018-08-16 22:34:17,279 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - 
Received request: GET /api/v1/websso

2018-08-16 22:34:17,279 DEBUG filter.Pac4jIdentityAdapter 
(Pac4jIdentityAdapter.java:doFilter(85)) - User authenticated as: 
#SAML2Profile# | id: 
praveen_raviku...@.com | attributes: 
{Mail=[praveen_raviku...@.com], 
UserID=[pravikumar], notOnOrAfter=2018-08-16T22:39:16.775Z, 
sessionindex=MdgrQer1EH_J2n1l.OX54WupIWl, notBefore=2018-08-16T22:29:16.775Z} | 
roles: [] | permissions: [] | isRemembered: false | clientName: SAML2Client | 
linkedId: null |

2018-08-16 22:34:17,282 INFO  service.knoxsso (WebSSOResource.java:init(113)) - 
The cookie max age is being set to: session.

2018-08-16 22:34:17,282 WARN  service.knoxsso (WebSSOResource.java:init(117)) - 
The SSO cookie max age configuration is invalid: session - using default.

2018-08-16 22:34:17,282 INFO  service.knoxsso 
(WebSSOResource.java:getCookieValue(330)) - Unable to find cookie with name: 
original-url

2018-08-16 22:34:17,284 DEBUG service.knoxsso 
(WebSSOResource.java:addJWTHadoopCookie(288)) - Adding the following JWT token 
as a cookie: 
eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJQcmF2ZWVuX1JhdmlrdW1hckBpbnR1aXQuY29tIiwiaXNzIjoiS05PWFNTTyIsImV4cCI6MTUzNDQ1ODk1N30.CKWaSVrxyW18O-SJ7mc8ZieehS8izd90bAdQmTd38y-FkuTXhTMSTDy2_RqERLklgeiPGsE1zQ7TdxRA9wgQ3NBYIaiAfBKE1epbmgFZ_MyAlmHvaSafXKF4LI49HNHJoUE1m2r4wRXsS1uysAwSMJ2r_TGs9obB4z6uYRC09YU

2018-08-16 22:34:17,284 INFO  service.knoxsso 
(WebSSOResource.java:addJWTHadoopCookie(304)) - JWT cookie successfully added.

2018-08-16 22:34:17,284 INFO  service.knoxsso 
(WebSSOResource.java:getAuthenticationToken(214)) - About to redirect to 
original URL: https://knox.idf-emr-prd.a.intuit.com:8446/gateway/gate1/yarn/

2018-08-16 22:34:17,399 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - 
Received request: GET /yarn/

2018-08-16 22:34:17,399 DEBUG federation.jwt 
(SSOCookieFederationFilter.java:getJWTFromCookie(161)) - hadoop-jwt Cookie has 
been found and is being processed.

2018-08-16 22:34:17,400 DEBUG knox.gateway 
(UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: 
https://knox.idf-emr-prd.a.xx.com:8446/gateway/gate1/yarn/, direction: IN 
via implicit rule: YARNUI/yarn/inbound/root to URL: 
http://ip-10-87-66-201.vpc.internal:8088/cluster

2018-08-16 22:34:17,401 DEBUG knox.gateway 
(DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET 
http://ip-xx-xx-xx-xxx.vpc.internal:8088/cluster?user.name=Praveen_Ravikumar%xxx.com

2018-08-16 22:34:17,405 DEBUG knox.gateway 
(DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 
200


Thanks,
Praveen.

Re: Knox Integration with CDAP UI

[Sandeep Moré]

Hello Priyanka,

I am not aware of CDAP UI support for Knox, we would be glad if you could
contribute !
About the filters they can be a bit tricky, try to turn on the debug log
and see what rewrite rules are failing that can narrow down a lot of errors.
For JS filters you can use regex to rewrite all *.js files, Knox also has
AND / OR logic in rules (called Flows) if you want any exceptions to your
rules.

You can look at this Wiki page
https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JavaScriptURLRewriteFilter

it should help you.

Good luck.
Sandeep

On Thu, Aug 16, 2018 at 7:24 AM Priyanka Gupta 
wrote:

> Hi,
>
> I am trying to integrate CDAP UI with Knox. CDAP UI main page internally 
> calls other urls like:
>
> https://192.168.134.119:8443/cdap_assets/img/company_logo.png
>
> https://192.168.134.119:8443/cdap_assets/fonts/icomoon.woff
>
> https://192.168.134.119:8443/_sock/info?t=1534418001305
>
> https://192.168.134.119:8443/backendstatus
>
>
> I tried working with filters and rewrite rules to change the request but that 
> seems not feasible as there are too many URL calls from within cdap.js file.
>
> How we can resolve this?Also the right approach to integrate CDAP UI with 
> Knox.
>
> Thanks,
>
> Priyanka
>
>
>
>
>
>
>
>
>
>
>
>
>

Re: Dispatcher class for Grafana

[Sandeep Moré]

Hello Divya,

Looking at the URL you posted looks like you are using the wrong dispatch
class, URLDecodingDispatch is used for URLs that are already encoded,
looking at your example URL it looks like it is partially encoded (e.g. the
query params &), in-fact looks like it is not properly encoded.

What error are you getting in gateway.log corresponding to 400 (Bad
Request) ?

Best,
Sandeep



On Thu, Aug 16, 2018 at 6:06 AM Divya Narayan 
wrote:

> Hi ,
>
> I tried to integrate grafana UI with knox gateway. I used
> URLDecodingDispatch class but there were some calls which were failing like:
>
>
> *http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=__%.user=%=%25=namenode=1534390641=1534401441*
>
>
> The above call on grafana UI failed with below error:
>
> *Caused by: java.lang.IllegalArgumentException: Malformed escape pair at
> index
> 141: 
> http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=__%.user=%=%25=namenode=1534390641=1534401441*
> *at java.net.URI.create(URI.java:852)*
> *at
> knoxdisp.URLDecodingDispatch.getDispatchUrl(URLDecodingDispatch.java:49)*
> *at
> org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:132)*
> *at
> org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:115)*
> *at org.apache.ha*
> *doop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)*
> *... 76 more*
>
>
> Now instead of making rewrite rules for each of such URL, I decided to
> change the dispatcher class itself to handle requests with special
> characters. In dispatcher class I added URL encoding to encode all the
> special characters.
> Below is the code that I tried:
>
> @Override
>
>   public URI getDispatchUrl(final HttpServletRequest request) {
>
> String decoded;
>
> String encodedURL = null;
>
>
> try {
>
>   decoded = URLDecoder.decode(request.getRequestURL().toString(),
> "UTF-8" );
>
> } catch (final Exception e) {
>
>   /* fall back in case of exception */
>
>   decoded = request.getRequestURL().toString();
>
> }
>
>
> final StringBuffer str = new StringBuffer(decoded);
>
> final String query = request.getQueryString();
>
> if ( query != null ) {
>
>   try {
>
> encodedURL = URLEncoder.encode(query,"UTF-8");
>
> } catch (UnsupportedEncodingException e) {
>
> e.printStackTrace();
>
> }
>
>
>   str.append('?');
>
>   str.append(encodedURL);
>
>
>
> }
>
> final URI url = URI.create(str.toString());
>
> return url;
>
>   }
>
> }
>
>
>
> Now after making these changes, malformed error is resolved and all URLs
> gets encoded like :
>
>
>  *Dispatch request: GET
> http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames%3Ddfs.NNTopUserOpCounts.windowMs%3D150.op%3D__%25.user%3D%25%26hostname%3D%2525%26appId%3Dnamenode%26startTime%3D1534400120%26endTime%3D1534410920
> *
>
>
>
> But the calls still doesn’t work. It throws HTTP Error code: 400 (Bad
> Request).  Can anyone help me to resolve these calls.
>
>
>
> Thanks
>
> Divya
>
>

Re: Need help on rewrite rules

[Sandeep Moré]

Hello Priyanka,

Sometimes the content types do not match, can you check if the response
type you are getting back is indeed "application/javascript" ?
Also, try using a broader rule on service.xml for testing something like /
Do you see any failures in gateway.log, i.e. failure to apply rules etc.

On Thu, Aug 16, 2018 at 3:40 AM Priyanka Gupta 
wrote:

> Hi,
>
> I need some help on a rewrite rule.I have a java script which is internally 
> calling other java scripts.So I tried to modify the url of those javascript 
> using filters.
>
> Java scripts url : 
>
>
> But these requests are not changing.
>
>
> 
> 
> 
>
> 
> 
> 
>
> 
> 
>  rule="CDAPUI/cdapui/outbound/img"/>
> 
> 
>
>
> Also, I tied the rule to service.xml like:
>
> 
> 
> 
> 
> 
>
>
> Thanks in advance!
>
>
>
>
>
>
>
>

Dispatcher class for Grafana

[Divya Narayan]

Hi ,

I tried to integrate grafana UI with knox gateway. I used URLDecodingDispatch 
class but there were some calls which were failing like:

http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=__%.user=%=%25=namenode=1534390641=1534401441


The above call on grafana UI failed with below error:

Caused by: java.lang.IllegalArgumentException: Malformed escape pair at index 
141: 
http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=__%.user=%=%25=namenode=1534390641=1534401441
at java.net.URI.create(URI.java:852)
at knoxdisp.URLDecodingDispatch.getDispatchUrl(URLDecodingDispatch.java:49)
at 
org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:132)
at 
org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:115)
at 
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
... 76 more


Now instead of making rewrite rules for each of such URL, I decided to change 
the dispatcher class itself to handle requests with special characters. In 
dispatcher class I added URL encoding to encode all the special characters.
Below is the code that I tried:


@Override

  public URI getDispatchUrl(final HttpServletRequest request) {

String decoded;

String encodedURL = null;


try {

  decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );

} catch (final Exception e) {

  /* fall back in case of exception */

  decoded = request.getRequestURL().toString();

}


final StringBuffer str = new StringBuffer(decoded);

final String query = request.getQueryString();

if ( query != null ) {

  try {

encodedURL = URLEncoder.encode(query,"UTF-8");

} catch (UnsupportedEncodingException e) {

e.printStackTrace();

}


  str.append('?');

  str.append(encodedURL);



}

final URI url = URI.create(str.toString());

return url;

  }

}



Now after making these changes, malformed error is resolved and all URLs gets 
encoded like :


 Dispatch request: GET 
http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames%3Ddfs.NNTopUserOpCounts.windowMs%3D150.op%3D__%25.user%3D%25%26hostname%3D%2525%26appId%3Dnamenode%26startTime%3D1534400120%26endTime%3D1534410920



But the calls still doesn’t work. It throws HTTP Error code: 400 (Bad Request). 
 Can anyone help me to resolve these calls.



Thanks

Divya

Need help on rewrite rules

[Priyanka Gupta]

Hi,

I need some help on a rewrite rule.I have a java script which is internally 
calling other java scripts.So I tried to modify the url of those javascript 
using filters.

Java scripts url : 


But these requests are not changing.

















Also, I tied the rule to service.xml like:








Thanks in advance!