Re: Problem with Context Path Whitelisting

[Swarup Karavadi]

Thank you for your response Kevin & Jeff.

I could not figure out a way to get the original request that was sent by
Nginx to NiFi. So I'm going to try and provide you with other information
that might give you some insights into the headers that are being forwarded
to NiFi's embedded web server -

1. Request log from Nginx (the format is as specified here

)

10.32.0.1 - [10.32.0.1] - - [14/Oct/2019:05:18:41 +] "GET /pie/ip/
HTTP/2.0" 200 402 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36"
310 0.003 [default-nifi-external-80] [] 10.100.35.29:80 402 0.004 200
e0cded9d89aa2a8be4fc4856489dcc67

To sum up the above log - the NiFi k8s service which is on 10.100.35.29:80
(and forwards traffic to NiFi pod's port 8080) has returned a 200 HTTP OK.

2. The Nginx Ingress config that is specific to NiFi
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header X-ProxyScheme http;
  proxy_set_header X-ProxyHost .
elb.amazonaws.com
  proxy_set_header X-ProxyPort 80;
  proxy_set_header X-ProxyContextPath "/pie/ip";
nginx.ingress.kubernetes.io/cors-allow-methods:
PUT, GET, POST, DELETE, OPTIONS
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/proxy-body-size: 20m
nginx.ingress.kubernetes.io/ssl-passthrough: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
  name: nifi-ingress
  namespace: default
  selfLink:
/apis/extensions/v1beta1/namespaces/default/ingresses/nifi-ingress
spec:
  rules:
- http:
paths:
  - backend:
  serviceName: nifi-external
  servicePort: 80
path: /pie/ip(/|$)(.*)


3. The actual Nginx configuration for the /pie/IP location (extracted from
the nginx.conf file from the Nginx ingress pod)
location ~* "^/pie/ip(/|$)(.*)" {

set $namespace  "default";
set $ingress_name   "nifi-ingress";
set $service_name   "";
set $service_port   "80";
set $location_path  "/pie/ip(/|${literal_dollar})(.*)";

rewrite_by_lua_block {
lua_ingress.rewrite({
force_ssl_redirect = false,
use_port_in_redirects = false,
})
balancer.rewrite()
plugins.run()
}

header_filter_by_lua_block {
plugins.run()
}
body_filter_by_lua_block {
}

log_by_lua_block {
balancer.log()
monitor.call()
plugins.run()
}

if ($scheme = https) {
more_set_headers
"Strict-Transport-Security: max-age=15724800; includeSubDomains";
}

port_in_redirect off;

set $balancer_ewma_score -1;
set $proxy_upstream_name"default-nifi-external-80";
set $proxy_host $proxy_upstream_name;

set $proxy_alternative_upstream_name "";


# Cors Preflight methods needs additional options and different Return Code
if ($request_method = 'OPTIONS') {
more_set_headers 'Access-Control-Allow-Origin: *';
more_set_headers 'Access-Control-Allow-Credentials: true';
more_set_headers
'Access-Control-Allow-Methods: PUT, GET, POST, DELETE, OPTIONS';
more_set_headers
'Access-Control-Allow-Headers:
DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization'
;
more_set_headers 'Access-Control-Max-Age: 1728000';
more_set_headers 'Content-Type: text/plain charset=UTF-8';
more_set_headers 'Content-Length: 0';
return 204;
}

more_set_headers 'Access-Control-Allow-Origin: *';
more_set_headers 'Access-Control-Allow-Credentials: true';
more_set_headers
'Access-Control-Allow-Methods: PUT, GET, POST, DELETE, OPTIONS';
more_set_headers
'Access-Control-Allow-Headers:
DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization'
;

client_max_body_size20m;
proxy_set_header Host   $best_http_host;

# Pass the extracted client certificate to the backend

# Allow websocket connections
proxy_set_headerUpgrade   $http_upgrade
;
proxy_set_headerConnection
$connection_upgrade
;
proxy_set_header X-Request-ID   $req_id;
proxy_set_header X-Real-IP  $the_real_ip;
proxy_set_header X-Forwarded-For$the_real_ip;
proxy_set_header X-Forwarded-Host   $best_http_host;
proxy_set_header X-Forwarded-Port   $pass_port;
proxy_set_header X-Forwarded-Proto  $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme   

Re: Problem with Context Path Whitelisting

[Jeff]

Swarup,

Agreed with Kevin, very nice write-up on the scenario!

Would you please provide the original request as sent by Nginx, along with
your configuration pertaining to NiFi in Nginx?  We can set up some test
cases to reproduce what's happening and get a JIRA filed if there's an edge
case not being handled by NiFi.

On Fri, Oct 11, 2019 at 9:30 AM Kevin Doran  wrote:

> Swarup,
>
> First, thanks for the great email. Nice job troubleshooting this and
> sharing your findings with the community.
>
> I'm more familiar with how these types of things get configured on
> NiFi Registry than NiFi, so I'm not as much help as others. But I did
> take a look and one thing I noticed was a difference between the
> startup config and the per-request config.
>
> On Startup, the whitelisted context paths are coming from the
> ServletContext FilterConfig [1].
>
> During request handling, the whitelisted context paths are coming from
> the ApplicationContext, directly from NiFi Properties [2]
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/filter/SanitizeContextPathFilter.java#L41
> [2]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ApplicationResource.java#L165
>
> Ultimately, my assumption is that both of these property values
> *should* be backed by the same nifi.properties file. But it appears
> something is happening in your case/environment/situation that is
> causing the ServletContext and ApplicationContext to get
> configured/initialized differently. This could be something specific
> to your environment or it could be uncovering an edge-case bug in
> NiFi.
>
> I think others on this mailing list who are more familiar with how the
> ServletContext gets setup in NiFi might be able to help further on
> this and determine if there is a solution/workaround or bug that needs
> patching.
>
> Thanks,
> Kevin
>
> On Fri, Oct 11, 2019 at 4:55 AM Swarup Karavadi  wrote:
> >
> > Greetings,
> >
> > I have deployed a single node unsecured NiFi cluster (I say cluster
> because nifi.cluster.is.node is set to "true") as a stateful set on
> Kubernetes (AWS EKS to be specific). The NiFi cluster sits behind an Nginx
> ingress. I have configured the Nginx ingress to forward the appropriate
> headers to NiFi (when deployed behind a reverse proxy) as described in the
> documentation.
> >
> > The path on the Nginx ingress which proxies traffic to the NiFi UI is
> "/pie/ip". This same path has been whitelisted by setting the
> "nifi.web.proxy.context.path" property to "/pie/ip". The way I am expecting
> this setup to work is that when users navigate to http://foo.com/pie/ip
> in the browser, they are shown a simple HTML page with redirect info and
> then automatically redirected to http://foo.com/pie/ip/nifi where they
> can view the NiFi canvas. Instead, the users are being redirected to
> http://foo.com/nifi which results in a 404 response because there is no
> '/nifi' path that has been configured on the Nginx ingress.
> >
> > I set the NiFi and Jetty Server log levels to DEBUG to understand what
> was happening under the hood and this is what I got -
> >
> > On Startup (when the SanitizeContextPathFilter is initialized) -
> > 2019-10-11 06:07:26,206 DEBUG [main]
> o.a.n.w.filter.SanitizeContextPathFilter SanitizeContextPathFilter received
> provided whitelisted context paths from NiFi properties: /pie/ip
> >
> > On Request (when the actual request is made) -
> > 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23]
> org.apache.nifi.web.util.WebUtils Context path:
> > 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23]
> org.apache.nifi.web.util.WebUtils On the request, the following context
> paths were parsed from headers:
> >  X-ProxyContextPath: /pie/ip
> > X-Forwarded-Context: null
> > X-Forwarded-Prefix: null
> > 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23]
> org.apache.nifi.web.util.WebUtils Determined context path: /pie/ip
> > 2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23]
> org.apache.nifi.web.util.WebUtils The provided context path [/pie/ip] was
> not whitelisted []
> > 2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23]
> org.apache.nifi.web.util.WebUtils Error determining context path on JSP
> page: The provided context path [/pie/ip] was not whitelisted []
> > 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23]
> o.a.n.w.filter.SanitizeContextPathFilter SanitizeContextPathFilter set
> contextPath:
> >
> > You will notice from the above log entries that the path '/pie/ip' was
> successfully whitelisted. Yet, when handling the request, the whitelisted
> context paths array is empty and this causes the wrong redirect to happen
> on the browser - and I can't figure out why this is happening or how I can
> fix it. Has anyone come across this kind of problem before? Any help on
> this is much appreciated.
> >
> > Cheers,
> > Swarup.
>

Re: Problem with Context Path Whitelisting

[Kevin Doran]

Swarup,

First, thanks for the great email. Nice job troubleshooting this and
sharing your findings with the community.

I'm more familiar with how these types of things get configured on
NiFi Registry than NiFi, so I'm not as much help as others. But I did
take a look and one thing I noticed was a difference between the
startup config and the per-request config.

On Startup, the whitelisted context paths are coming from the
ServletContext FilterConfig [1].

During request handling, the whitelisted context paths are coming from
the ApplicationContext, directly from NiFi Properties [2]

[1] 
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/filter/SanitizeContextPathFilter.java#L41
[2] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ApplicationResource.java#L165

Ultimately, my assumption is that both of these property values
*should* be backed by the same nifi.properties file. But it appears
something is happening in your case/environment/situation that is
causing the ServletContext and ApplicationContext to get
configured/initialized differently. This could be something specific
to your environment or it could be uncovering an edge-case bug in
NiFi.

I think others on this mailing list who are more familiar with how the
ServletContext gets setup in NiFi might be able to help further on
this and determine if there is a solution/workaround or bug that needs
patching.

Thanks,
Kevin

On Fri, Oct 11, 2019 at 4:55 AM Swarup Karavadi  wrote:
>
> Greetings,
>
> I have deployed a single node unsecured NiFi cluster (I say cluster because 
> nifi.cluster.is.node is set to "true") as a stateful set on Kubernetes (AWS 
> EKS to be specific). The NiFi cluster sits behind an Nginx ingress. I have 
> configured the Nginx ingress to forward the appropriate headers to NiFi (when 
> deployed behind a reverse proxy) as described in the documentation.
>
> The path on the Nginx ingress which proxies traffic to the NiFi UI is 
> "/pie/ip". This same path has been whitelisted by setting the 
> "nifi.web.proxy.context.path" property to "/pie/ip". The way I am expecting 
> this setup to work is that when users navigate to http://foo.com/pie/ip in 
> the browser, they are shown a simple HTML page with redirect info and then 
> automatically redirected to http://foo.com/pie/ip/nifi where they can view 
> the NiFi canvas. Instead, the users are being redirected to 
> http://foo.com/nifi which results in a 404 response because there is no 
> '/nifi' path that has been configured on the Nginx ingress.
>
> I set the NiFi and Jetty Server log levels to DEBUG to understand what was 
> happening under the hood and this is what I got -
>
> On Startup (when the SanitizeContextPathFilter is initialized) -
> 2019-10-11 06:07:26,206 DEBUG [main] o.a.n.w.filter.SanitizeContextPathFilter 
> SanitizeContextPathFilter received provided whitelisted context paths from 
> NiFi properties: /pie/ip
>
> On Request (when the actual request is made) -
> 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils Context path:
> 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils On the request, the following context paths 
> were parsed from headers:
>  X-ProxyContextPath: /pie/ip
> X-Forwarded-Context: null
> X-Forwarded-Prefix: null
> 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils Determined context path: /pie/ip
> 2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils The provided context path [/pie/ip] was not 
> whitelisted []
> 2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils Error determining context path on JSP page: 
> The provided context path [/pie/ip] was not whitelisted []
> 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
> o.a.n.w.filter.SanitizeContextPathFilter SanitizeContextPathFilter set 
> contextPath:
>
> You will notice from the above log entries that the path '/pie/ip' was 
> successfully whitelisted. Yet, when handling the request, the whitelisted 
> context paths array is empty and this causes the wrong redirect to happen on 
> the browser - and I can't figure out why this is happening or how I can fix 
> it. Has anyone come across this kind of problem before? Any help on this is 
> much appreciated.
>
> Cheers,
> Swarup.

Problem with Context Path Whitelisting

[Swarup Karavadi]

Greetings,

I have deployed a single node unsecured NiFi cluster (I say cluster because 
nifi.cluster.is.node is set to "true") as a stateful set on Kubernetes (AWS EKS 
to be specific). The NiFi cluster sits behind an Nginx ingress. I have 
configured the Nginx ingress to forward the appropriate headers to NiFi (when 
deployed behind a reverse proxy) as described in the documentation 
.
 

The path on the Nginx ingress which proxies traffic to the NiFi UI is 
"/pie/ip". This same path has been whitelisted by setting the 
"nifi.web.proxy.context.path" property to "/pie/ip". The way I am expecting 
this setup to work is that when users navigate to http://foo.com/pie/ip 
 in the browser, they are shown a simple HTML page with 
redirect info and then automatically redirected to http://foo.com/pie/ip/nifi 
 where they can view the NiFi canvas. Instead, the 
users are being redirected to http://foo.com/nifi  which 
results in a 404 response because there is no '/nifi' path that has been 
configured on the Nginx ingress.

I set the NiFi and Jetty Server log levels to DEBUG to understand what was 
happening under the hood and this is what I got - 

On Startup (when the SanitizeContextPathFilter is initialized) - 
2019-10-11 06:07:26,206 DEBUG [main] o.a.n.w.filter.SanitizeContextPathFilter 
SanitizeContextPathFilter received provided whitelisted context paths from NiFi 
properties: /pie/ip

On Request (when the actual request is made) - 
2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils Context path: 
2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils On the request, the following context paths 
were parsed from headers:
 X-ProxyContextPath: /pie/ip
X-Forwarded-Context: null
X-Forwarded-Prefix: null
2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils Determined context path: /pie/ip
2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils The provided context path [/pie/ip] was not 
whitelisted []
2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils Error determining context path on JSP page: 
The provided context path [/pie/ip] was not whitelisted []
2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
o.a.n.w.filter.SanitizeContextPathFilter SanitizeContextPathFilter set 
contextPath: 

You will notice from the above log entries that the path '/pie/ip' was 
successfully whitelisted. Yet, when handling the request, the whitelisted 
context paths array is empty and this causes the wrong redirect to happen on 
the browser - and I can't figure out why this is happening or how I can fix it. 
Has anyone come across this kind of problem before? Any help on this is much 
appreciated.

Cheers,
Swarup.