Re: NiFi Kubernetes question

[Swarup Karavadi]

Hey Joe,

I'm glad that the article was of some help. I, unfortunately, do not
recollect running into that specific error scenario. At the risk of stating
the obvious, can you check if the nifi-ca-cs service is reachable from your
nifi pod?

On the CRD bit, we are considering going this route with NiFi stateless. We
are experimenting to see if we can have something like "NiFi K" that can be
inline with the way "Camel K" works.  Based on our current (limited)
understanding of NiFi stateless, we think the CRD approach will help scale
NiFi stateless horizontally with much more ease.

Cheers,
Swarup.

On Sat, Dec 28, 2019 at 4:32 AM Mike Thomsen  wrote:

> If you don't see a value when you run "echo %JAVA_HOME%" then you need to
> check to see if it was set globally in Windows, and if it was, you need to
> open a new command shell.
>
> On Mon, Oct 21, 2019 at 12:37 PM  wrote:
>
>> Any suggestions?
>>
>>
>>
>> I downloaded NiFi, but when I run the runnifi from the bin folder nothing
>> happens. I get the following message. The JAVA_HOME environment variable is
>> not defined correctly. I downloaded the latest JRE, but still get the same
>> error message.
>>
>>
>>
>>
>>
>> *From:* Swarup Karavadi 
>> *Sent:* Monday, October 21, 2019 12:19 PM
>> *To:* users@nifi.apache.org
>> *Cc:* jgres...@gmail.com
>> *Subject:* Re: NiFi Kubernetes question
>>
>>
>>
>> If you are hosting on the cloud, I'd recommend going for dedicated worker
>> nodes for the NiFi cluster. There might be rare (or not) occasions when a
>> worker node is under high load and needs to evict pods. If your NiFi
>> deployment's pod disruption budget allows for eviction of pods then there
>> are always chances that an evicted NiFi pod can be rescheduled on a
>> different node that is tainted (tainted because the node may not meet the
>> pod's volume affinity requirements). Your best case scenario when this
>> happens is that the pod will keep getting rescheduled on different nodes
>> until it starts up again. The worst case scenario is that it'll be stuck in
>> a CrashLoopBackoff limbo.
>>
>>
>>
>> Disclaimer - I speak from my experience on a non production environment.
>> Our NiFi clusters will be deployed to a production k8s environment in a few
>> weeks from now. I am only sharing some learnings I've had w.r.t. k8s
>> statefulsets along the way.
>>
>>
>>
>> Hope this helps,
>>
>> Swarup.
>>
>>
>>
>> On Mon, Oct 21, 2019, 9:32 PM Wyllys Ingersoll <
>> wyllys.ingers...@keepertech.com> wrote:
>>
>>
>>
>> We had success running  a 3-node cluster in kubernetes using modified
>> configuration scripts from the AlexJones github repo -
>> https://github.com/AlexsJones/nifi
>>
>> Ours is on an internal bare-metal k8s lab configuration, not in a public
>> cloud at this time, but the basics are the same either way.
>>
>>
>>
>> - setup nifi as a stateful set so you can scale up or down as needed.
>> When a pod fails, k8s will spawn another to take its place and zookeeper
>> will manage the election of the master during transitions.
>>
>> - manage your certs as K8S secrets.
>>
>> - you also need to also have a stateful set of zookeeper pods for
>> managing the nifi servers.
>>
>> - use persistent volume mounts to hold the flowfile, database, content,
>> and provenance _repository directories
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Oct 21, 2019 at 11:21 AM Joe Gresock  wrote:
>>
>> Apologies if this has been answered on the list already..
>>
>>
>>
>> Does anyone have knowledge of the latest in the realm of nifi kubernetes
>> support?  I see some pages like https://hub.helm.sh/charts/cetic/nifi,
>> and https://github.com/AlexsJones/nifi but am unsure which example to
>> pick to start with.
>>
>>
>>
>> I'm curious how well kubernetes maintains the nifi cluster state with pod
>> failures.  I.e., do any of the k8s implementations play well with the nifi
>> cluster list so that we don't have dangling downed nodes in the cluster?
>> Also, I'm wondering how certs are managed in a secured cluster.
>>
>>
>>
>> Appreciate any nudge in the right direction,
>>
>> Joe
>>
>>
>>
>> On Mon, Oct 21, 2019, 9:32 PM Wyllys Ingersoll <
>> wyllys.ingers...@keepertech.com> wrote:
>>
>>
>>
>> We had success running  a 3-node cluster in kubernetes using modi

Re: NiFi Kubernetes question

[Swarup Karavadi]

If you are hosting on the cloud, I'd recommend going for dedicated worker
nodes for the NiFi cluster. There might be rare (or not) occasions when a
worker node is under high load and needs to evict pods. If your NiFi
deployment's pod disruption budget allows for eviction of pods then there
are always chances that an evicted NiFi pod can be rescheduled on a
different node that is tainted (tainted because the node may not meet the
pod's volume affinity requirements). Your best case scenario when this
happens is that the pod will keep getting rescheduled on different nodes
until it starts up again. The worst case scenario is that it'll be stuck in
a CrashLoopBackoff limbo.

Disclaimer - I speak from my experience on a non production environment.
Our NiFi clusters will be deployed to a production k8s environment in a few
weeks from now. I am only sharing some learnings I've had w.r.t. k8s
statefulsets along the way.

Hope this helps,
Swarup.

On Mon, Oct 21, 2019, 9:32 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

>
> We had success running  a 3-node cluster in kubernetes using modified
> configuration scripts from the AlexJones github repo -
> https://github.com/AlexsJones/nifi
> Ours is on an internal bare-metal k8s lab configuration, not in a public
> cloud at this time, but the basics are the same either way.
>
> - setup nifi as a stateful set so you can scale up or down as needed. When
> a pod fails, k8s will spawn another to take its place and zookeeper will
> manage the election of the master during transitions.
> - manage your certs as K8S secrets.
> - you also need to also have a stateful set of zookeeper pods for managing
> the nifi servers.
> - use persistent volume mounts to hold the flowfile, database, content,
> and provenance _repository directories
>
>
>
> On Mon, Oct 21, 2019 at 11:21 AM Joe Gresock  wrote:
>
>> Apologies if this has been answered on the list already..
>>
>> Does anyone have knowledge of the latest in the realm of nifi kubernetes
>> support?  I see some pages like https://hub.helm.sh/charts/cetic/nifi,
>> and https://github.com/AlexsJones/nifi but am unsure which example to
>> pick to start with.
>>
>> I'm curious how well kubernetes maintains the nifi cluster state with pod
>> failures.  I.e., do any of the k8s implementations play well with the nifi
>> cluster list so that we don't have dangling downed nodes in the cluster?
>> Also, I'm wondering how certs are managed in a secured cluster.
>>
>> Appreciate any nudge in the right direction,
>> Joe
>>
>
On Mon, Oct 21, 2019, 9:32 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

>
> We had success running  a 3-node cluster in kubernetes using modified
> configuration scripts from the AlexJones github repo -
> https://github.com/AlexsJones/nifi
> Ours is on an internal bare-metal k8s lab configuration, not in a public
> cloud at this time, but the basics are the same either way.
>
> - setup nifi as a stateful set so you can scale up or down as needed. When
> a pod fails, k8s will spawn another to take its place and zookeeper will
> manage the election of the master during transitions.
> - manage your certs as K8S secrets.
> - you also need to also have a stateful set of zookeeper pods for managing
> the nifi servers.
> - use persistent volume mounts to hold the flowfile, database, content,
> and provenance _repository directories
>
>
>
> On Mon, Oct 21, 2019 at 11:21 AM Joe Gresock  wrote:
>
>> Apologies if this has been answered on the list already..
>>
>> Does anyone have knowledge of the latest in the realm of nifi kubernetes
>> support?  I see some pages like https://hub.helm.sh/charts/cetic/nifi,
>> and https://github.com/AlexsJones/nifi but am unsure which example to
>> pick to start with.
>>
>> I'm curious how well kubernetes maintains the nifi cluster state with pod
>> failures.  I.e., do any of the k8s implementations play well with the nifi
>> cluster list so that we don't have dangling downed nodes in the cluster?
>> Also, I'm wondering how certs are managed in a secured cluster.
>>
>> Appreciate any nudge in the right direction,
>> Joe
>>
>

Re: Problem with Context Path Whitelisting

[Swarup Karavadi]

rded-Port   $pass_port;
proxy_set_header X-Forwarded-Proto  $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme   $pass_access_scheme;

# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

# mitigate HTTPoxy Vulnerability
#
https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy  "";

# Custom headers to proxied server
proxy_connect_timeout   5s;
proxy_send_timeout  60s;
proxy_read_timeout  60s;

proxy_buffering off;
proxy_buffer_size   4k;
proxy_buffers   4 4k;
proxy_request_buffering on;

proxy_http_version  1.1;

proxy_cookie_domain off;
proxy_cookie_path   off;


# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0;
proxy_next_upstream_tries   3;
proxy_set_header X-ProxyScheme http;
proxy_set_header X-ProxyHost .
elb.amazonaws.com;
proxy_set_header X-ProxyPort 80;
proxy_set_header X-ProxyContextPath "/pie/ip";
proxy_pass http://upstream_balancer;
proxy_redirect  off;

}

Please do let me know if this is information is useful at all.

Cheers,
Swarup.

On Fri, Oct 11, 2019 at 9:52 PM Jeff  wrote:

> Swarup,
>
> Agreed with Kevin, very nice write-up on the scenario!
>
> Would you please provide the original request as sent by Nginx, along with
> your configuration pertaining to NiFi in Nginx?  We can set up some test
> cases to reproduce what's happening and get a JIRA filed if there's an edge
> case not being handled by NiFi.
>
> On Fri, Oct 11, 2019 at 9:30 AM Kevin Doran  wrote:
>
>> Swarup,
>>
>> First, thanks for the great email. Nice job troubleshooting this and
>> sharing your findings with the community.
>>
>> I'm more familiar with how these types of things get configured on
>> NiFi Registry than NiFi, so I'm not as much help as others. But I did
>> take a look and one thing I noticed was a difference between the
>> startup config and the per-request config.
>>
>> On Startup, the whitelisted context paths are coming from the
>> ServletContext FilterConfig [1].
>>
>> During request handling, the whitelisted context paths are coming from
>> the ApplicationContext, directly from NiFi Properties [2]
>>
>> [1]
>> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/filter/SanitizeContextPathFilter.java#L41
>> [2]
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ApplicationResource.java#L165
>>
>> Ultimately, my assumption is that both of these property values
>> *should* be backed by the same nifi.properties file. But it appears
>> something is happening in your case/environment/situation that is
>> causing the ServletContext and ApplicationContext to get
>> configured/initialized differently. This could be something specific
>> to your environment or it could be uncovering an edge-case bug in
>> NiFi.
>>
>> I think others on this mailing list who are more familiar with how the
>> ServletContext gets setup in NiFi might be able to help further on
>> this and determine if there is a solution/workaround or bug that needs
>> patching.
>>
>> Thanks,
>> Kevin
>>
>> On Fri, Oct 11, 2019 at 4:55 AM Swarup Karavadi  wrote:
>> >
>> > Greetings,
>> >
>> > I have deployed a single node unsecured NiFi cluster (I say cluster
>> because nifi.cluster.is.node is set to "true") as a stateful set on
>> Kubernetes (AWS EKS to be specific). The NiFi cluster sits behind an Nginx
>> ingress. I have configured the Nginx ingress to forward the appropriate
>> headers to NiFi (when deployed behind a reverse proxy) as described in the
>> documentation.
>> >
>> > The path on the Nginx ingress which proxies traffic to the NiFi UI is
>> "/pie/ip". This same path has been whitelisted by setting the
>> "nifi.web.proxy.context.path" property to "/pie/ip". The way I am expecting
>> this setup to work is that when users navigate to http://foo.com/pie/ip
>> in the browser, they are shown a simple HTML page with redirect info and
>> then automa

Problem with Context Path Whitelisting

[Swarup Karavadi]

Greetings,

I have deployed a single node unsecured NiFi cluster (I say cluster because 
nifi.cluster.is.node is set to "true") as a stateful set on Kubernetes (AWS EKS 
to be specific). The NiFi cluster sits behind an Nginx ingress. I have 
configured the Nginx ingress to forward the appropriate headers to NiFi (when 
deployed behind a reverse proxy) as described in the documentation 
.
 

The path on the Nginx ingress which proxies traffic to the NiFi UI is 
"/pie/ip". This same path has been whitelisted by setting the 
"nifi.web.proxy.context.path" property to "/pie/ip". The way I am expecting 
this setup to work is that when users navigate to http://foo.com/pie/ip 
 in the browser, they are shown a simple HTML page with 
redirect info and then automatically redirected to http://foo.com/pie/ip/nifi 
 where they can view the NiFi canvas. Instead, the 
users are being redirected to http://foo.com/nifi  which 
results in a 404 response because there is no '/nifi' path that has been 
configured on the Nginx ingress.

I set the NiFi and Jetty Server log levels to DEBUG to understand what was 
happening under the hood and this is what I got - 

On Startup (when the SanitizeContextPathFilter is initialized) - 
2019-10-11 06:07:26,206 DEBUG [main] o.a.n.w.filter.SanitizeContextPathFilter 
SanitizeContextPathFilter received provided whitelisted context paths from NiFi 
properties: /pie/ip

On Request (when the actual request is made) - 
2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils Context path: 
2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils On the request, the following context paths 
were parsed from headers:
 X-ProxyContextPath: /pie/ip
X-Forwarded-Context: null
X-Forwarded-Prefix: null
2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils Determined context path: /pie/ip
2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils The provided context path [/pie/ip] was not 
whitelisted []
2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23] 
org.apache.nifi.web.util.WebUtils Error determining context path on JSP page: 
The provided context path [/pie/ip] was not whitelisted []
2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
o.a.n.w.filter.SanitizeContextPathFilter SanitizeContextPathFilter set 
contextPath: 

You will notice from the above log entries that the path '/pie/ip' was 
successfully whitelisted. Yet, when handling the request, the whitelisted 
context paths array is empty and this causes the wrong redirect to happen on 
the browser - and I can't figure out why this is happening or how I can fix it. 
Has anyone come across this kind of problem before? Any help on this is much 
appreciated.

Cheers,
Swarup.