OIDC with Authorization?

[Jon Logan]

Hi,

I am trying to use OIDC for Authentication, but it seems to not support any
form of Authorization -- is there any way to avoid having to manually list
every user permitted after installation? ex. allow all authenticated users,
or support groups from the OIDC provider?

Thanks!

Re: Determine cluster primary node using REST API

[Jon Logan]

The best way I've found for finding the REST API for various information is
to use the Browser Network tab when using the UI. It's usually fairly
obvious when poking around that, and you can recreate the API call. You can
also "copy as bash" out of it in Chrome, but you may have to clean it up a
bit.

On Mon, Jan 25, 2021 at 1:45 PM James McMahon  wrote:

> Hello Bryan. We run on Primary only because we are doing an end-to-end
> verification that our pipeline is available at the top of each hour, across
> several nifi links in a lengthy processing chain. We only want that done
> through one Node, not all N nodes in the cluster. It generates an alert
> email to me and others each hour, and we don’t need N email alerts. Let me
> know if you have any other questions.
>
> Can you provide me with an example of the Rest API call in bash via curl
> where you parse the primary node out of the returned JSON structure?
>
> On Mon, Jan 25, 2021 at 1:37 PM Bryan Bende  wrote:
>
>> I know this doesn't really answer your question, but is there a reason
>> you are setting ListenHttp to run on primary node only and not all
>> nodes?
>>
>> Typically you'd use "primary node only" for a source processing that
>> is pulling data from somewhere and you only want it to happen once,
>> otherwise you'd pull the same data multiple times. In this case,
>> ListenHTTP is just going to be sitting there waiting for something to
>> send data to it, so why not listen on all nodes?
>>
>> The processor is going to be started on all nodes anyway, so the
>> embedded Jetty is already started and listening on all nodes, the
>> "Primary Node Only" just means the onTrigger method will only be
>> called for the processor on the primary node, so for ListenHTTP that
>> just means it will only process the requests on the primary.
>>
>> On Sun, Jan 24, 2021 at 6:49 PM James McMahon 
>> wrote:
>> >
>> > I have a NiFi cluster, nifi version 1.8.n. I need to use curl from a
>> bash shell script on a remote host to query for the primary node of the
>> cluster at that moment. I understand there may be a NiFi REST API call I
>> can make to do this, but have little experience integrating such a call in
>> bash. Does anyone have an example that does this?
>> >
>> > Why do I want to do this? I have a ListenHttp running as an entry point
>> in a flow on the cluster, and that processor runs in “Primary node” only
>> configuration. Since the external zookeeper can change the primary at any
>> time, I need to precede this curl call with a curl that returns to me the
>> primary node.
>> >
>> > Thanks in advance for your help.
>>
>

Re: High CPU consumption

[Jon Logan]

That isn't logback, that lists all jars on your classpath, the first of
which happens to be logback.

If you send a SIGKILL3 (you can send it in HTOP) it will dump the thread
stacks to stdout (probably the bootstrap log)...but that's just for one
instant, and may or may not be helpful.

On Sun, Oct 13, 2019 at 8:58 PM Luis Carmona 
wrote:

> hi Aldrin,
>
> thanks a  lot, by now I'm trying to learn how to make the profiling you
> mentioned.
>
> One more question: Is it normal that the father java process has very low
> consumption while the child process related to logback jar is the one that
> is eating up all the CPU ?
> Please take a look at the attached image.
>
> Thanks,
>
> LC
>
> --
> *From: *"Aldrin Piri" 
> *To: *"users" 
> *Sent: *Sunday, October 13, 2019 9:30:47 PM
> *Subject: *Re: High CPU consumption
>
> Luis, please feel free to give us some information on your flow so we can
> help you track down problematic areas as well.
>
> On Sun, Oct 13, 2019 at 3:56 PM Jon Logan  wrote:
>
>> You should put a profiler on it to be sure.
>> Just because your processors aren't processing data doesn't mean they are
>> idle though -- many have to poll for new data, especially sources -- ex.
>> connecting to Kafka, etc, will itself consume some CPU.
>>
>> But again, you should attach a profiler before participating in a wild
>> goose chase of performance issues.
>>
>> On Sun, Oct 13, 2019 at 12:20 PM Luis Carmona 
>> wrote:
>>
>>> HI,
>>>
>>> I've struggling to reduce my nifi installation CPU consumption. Even in
>>> idle state, all processors running but no data flowing, it is beyond 60%
>>> CPU consumption, with peaks of 200%.
>>>
>>> What I've done so far
>>> - Read and followed every instruction/post about tuning NIFI I've found
>>> googling.
>>> - Verify scheduling is 1s for most consuming processors: http
>>> processors, wait/notify, jolt, etc.
>>> - Scratch my head...
>>>
>>> But nothing seems to have a major effect on the issue.
>>>
>>> Can anyone give me some precise directions or tips about how to solve
>>> this please ?
>>> Is this the regular situation, I mean this is the minimun NIFI
>>> consumption.
>>>
>>> The server is configure with 4 CPU's, 8 GB RAM - 4 of them dedicated to
>>> heap at bootstrap.conf.
>>>
>>> Thanks in advance.
>>>
>>> LC
>>>
>>
>

Re: High CPU consumption

[Jon Logan]

You should put a profiler on it to be sure.

Just because your processors aren't processing data doesn't mean they are
idle though -- many have to poll for new data, especially sources -- ex.
connecting to Kafka, etc, will itself consume some CPU.

But again, you should attach a profiler before participating in a wild
goose chase of performance issues.

On Sun, Oct 13, 2019 at 12:20 PM Luis Carmona 
wrote:

> HI,
>
> I've struggling to reduce my nifi installation CPU consumption. Even in
> idle state, all processors running but no data flowing, it is beyond 60%
> CPU consumption, with peaks of 200%.
>
> What I've done so far
> - Read and followed every instruction/post about tuning NIFI I've found
> googling.
> - Verify scheduling is 1s for most consuming processors: http processors,
> wait/notify, jolt, etc.
> - Scratch my head...
>
> But nothing seems to have a major effect on the issue.
>
> Can anyone give me some precise directions or tips about how to solve this
> please ?
> Is this the regular situation, I mean this is the minimun NIFI consumption.
>
> The server is configure with 4 CPU's, 8 GB RAM - 4 of them dedicated to
> heap at bootstrap.conf.
>
> Thanks in advance.
>
> LC
>

Re: node died unexpectedly

[Jon Logan]

Have you looked into this error ? It seems a bit concerning to me.

Executable command /usr/bin/python3 ended in an error:
/usr/local/lib/python3.7/site-packages/requests/__init__.py:80:
RequestsDependencyWarning: urllib3 (1.23) or chardet (3.0.4) doesn't match
a supported version!
  RequestsDependencyWarning)

On Mon, Sep 23, 2019 at 6:08 AM Jean-Sebastien Vachon <
jsvac...@brizodata.com> wrote:

> Hi,
>
> I am running a cluster of five nodes and one of them just quit for no
> apparent reason... at least I could not find anything wrong on the system.
> Here is a section of the logs. You can clearly see that there is a 8
> minutes gap before I manually had to restart Nifi.  I've looked in
> /var/log/messages and Nifi's log files but could
> not find any reason for this to happen...
>
> any thoughts/ideas?
>
> ---
>
> 2019-09-23 12:51:54,567 ERROR [Timer-Driven Process Thread-40]
> o.a.n.p.standard.ExecuteStreamCommand
> ExecuteStreamCommand[id=8b748c9a-7d7d-30cd-86a3-4ea3c4ccd3b4] Transferring
> flow file
> StandardFlowFileRecord[uuid=60d29ce2-088a-4829-a0ae-17672f1d86e6,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id
> =1569243036019-4497, container=default, section=401], offset=591157,
> length=-1],offset=0,name=186373,size=0] to nonzero status. Executable
> command /usr/bin/python3 ended in an error:
> 2019-09-23 12:51:54,570 ERROR [Timer-Driven Process Thread-4]
> o.a.n.p.standard.ExecuteStreamCommand
> ExecuteStreamCommand[id=272623bc-a371-3ff9-adaf-d8363b91a5ed] Transferring
> flow file
> StandardFlowFileRecord[uuid=a18485c8-9b03-43a8-8c13-e76803df9290,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1569242553862-4458,
> container=default, section=362], offset=780733,
> length=21229],offset=0,name=33287,size=21229] to nonzero status. Executable
> command /usr/bin/python3 ended in an error:
> /usr/local/lib/python3.7/site-packages/requests/__init__.py:80:
> RequestsDependencyWarning: urllib3 (1.23) or chardet (3.0.4) doesn't match
> a supported version!
>   RequestsDependencyWarning)
>
> 2019-09-23 12:51:54,570 ERROR [Thread-130947]
> o.a.n.p.standard.ExecuteStreamCommand
> ExecuteStreamCommand[id=a7cb604a-465d-3440-beab-b9554e1ba4bb] Failed to
> write flow file to stdin due to java.io.IOException: Broken pipe:
> java.io.IOException: Broken pipe
> java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:36)
> at
> org.apache.nifi.processors.standard.ExecuteStreamCommand$2.run(ExecuteStreamCommand.java:519)
> at java.lang.Thread.run(Thread.java:748)
> 2019-09-23 12:51:54,570 ERROR [Timer-Driven Process Thread-27]
> o.a.n.p.standard.ExecuteStreamCommand
> ExecuteStreamCommand[id=a7cb604a-465d-3440-beab-b9554e1ba4bb] Transferring
> flow file
> StandardFlowFileRecord[uuid=e5a82bb6-23a9-4a16-825a-1d976cd114cc,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1569242062433-4406,
> container=default, section=310], offset=680454,
> length=-1],offset=0,name=164088,size=0] to nonzero status. Executable
> command /usr/bin/python3 ended in an error:
>
> 2019-09-23 12:59:56,012 INFO [main] org.apache.nifi.NiFi Launching NiFi...
> 2019-09-23 12:59:56,179 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties
> path to be '/opt/nifi/./conf/nifi.properties'
> 2019-09-23 12:59:56,181 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 149 properties from
> /opt/nifi/./conf/nifi.properties
> 2019-09-23 12:59:56,186 INFO [main] org.apache.nifi.NiFi Loaded 149
> properties
>

Re: clean shutdown

[Jon Logan]

Remember that spot instances are given shutdown notifications on a
best-effort basis[1]. You would have to disconnect the node, drain it, then
shut it down after draining, and hope you do so before you get killed. You
could also consider the new hibernation feature -- it'll hibernate your
node instead of terminating, and then rehydrate it at a later time. Your
cluster would have a disconnected node in the mean time though. All of
these scenarios introduce a significant potential of data loss, you should
be sure you could reproduce the data from a durable source if needed (ex.
Kafka, etc), or be accepting of the data loss.


[1]
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html
While
we make every effort to provide this warning as soon as possible, it is
possible that your Spot Instance is terminated before the warning can be
made available. Test your application to ensure that it handles an
unexpected instance termination gracefully, even if you are testing for
interruption notices. You can do so by running the application using an
On-Demand Instance and then terminating the On-Demand Instance yourself.

On Wed, Aug 28, 2019 at 8:57 PM Jean-Sebastien Vachon <
jsvac...@brizodata.com> wrote:

> Hi Craig,
>
> I made some additional tests and I am afraid I lost flows... I used the
> same flow I described earlier, generated around 30k flows and load balanced
> them on the three nodes forming my cluster.
> I then shutdown one of the machine. The result is that I lost 10k flows
> that were scheduled to be processed on this machine. This is a problem I
> need to address and I'll be looking for ideas shortly.
>
> For those interested in automating the removal of a spot instance from a
> cluster... here is something to get you started.
> AWS recommend to monitor the URL found in the if statement every 5s (or
> so)... Since cron only supports 1 minute intervals and nothing smaller,
> I accomplish what I wanted by adding multiple crons and sleeping for a
> variable amount of time.
>
> You will need jq and curl to be installed on your machine for this to
> work.
> The basic idea is to wait until the web page appears to exist and then
> trigger a series of actions.
>
> ---
>
> #!/bin/bash
> sleep $1
>
> NODE_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
> 
> NODE_ID=`curl -s "http://${NODE_IP}:8088/nifi-api/controller/cluster; |
> jq --arg IP "${NODE_IP}" -r '.cluster.nodes[] | select('.address' ==
> $IP).nodeId'`
> OTHER_NODE=`curl -s "http://${NODE_IP}:8088/nifi-api/controller/cluster;
> | jq --arg IP "${NODE_IP}"  -r '.cluster.nodes[] | select('.address' !=
> $IP).address' | head -1`
>
> if [ -z $(curl -Is
> http://169.254.169.254/latest/meta-data/spot/termination-time | head -1 |
> grep 404 | cut -d' ' -f 2) ]
> then
> echo "Running shutdown hook."
> systemctl stop nifi
> sleep 5
> curl -s -X DELETE "http://
> ${OTHER_NODE}:8088/nifi-api/controller/cluster/nodes/$NODE_ID"
> fi
>
> --
> *From:* Jean-Sebastien Vachon 
> *Sent:* Wednesday, August 28, 2019 7:39 PM
> *To:* users@nifi.apache.org 
> *Subject:* Re: clean shutdown
>
> Hi Craig,
>
> First the generic stuff...
>
> according to the tests I made, no flows are lost when a machine is removed
> from the cluster.  They seem to be requeued.
> However, I only tested with a very basic flow and not with my whole flow
> which involves a lot of things.
> Basically, I used a GenerateFlow to generate some data and a dummy Python
> process to do something with it. The queue between the two
> processors was configured to do load balancing using a round robin. I must
> admit that I haven't look if the item was requeued and dispatched to
> another node.
> The output of the python module was split between success and failure and
> no single flow reached the failure state.
>
> then to AWS specific stuff...
>
> I had to script a few things to cleanup within the two minutes warning AWS
> is giving me.
> Since I am using spot instances, I know the instance will not come back so
> I had to automate the clean up of the cluster by
> using an API call to remove the machine from the cluster. In order to
> remove the machine from the cluster, I need to stop Nifi first and then
> remove the machine through
> a call to the API on a second node. I am still polishing the script to
> accomplish this. I may share it once it is working as expected in case
> someone else has this issue.
>
> Let me know if you need more details about anything...
> --
> *From:* Craig Knell 
> *Sent:* Wednesday, August 28, 2019 6:52 PM
> *To:* users@nifi.apache.org 
> *Subject:* Re: clean shutdown
>
> Hi Jean-Sebastien,
>
> I’d be interested to hear how this performs
>
> Best regards
>
> Craig
>
> On 28 Aug 2019, at 22:28, Jean-Sebastien Vachon 
> wrote:
>
> Hi Pierre,
>
> thanks for your input.
>
> I am already intercepting AWS termination 

Re: Using NiFi Expression Language outside of NiFi

[Jon Logan]

Have you considered the Spring Expression Language? It might be more
general purpose and segmentable than the NiFi EL. I am not aware of any
capability that NiFi EL has that SpEL does not, but perhaps there is some.

On Fri, Mar 15, 2019 at 11:25 AM Joe Witt  wrote:

> Tim,
>
> It is certainly feasible/interesting but it would require a lot of legwork
> to extract and make it into its own thing that then is used by NiFi and
> then other things.  We could spawn it into its own subproject to facilitate
> perhaps..
>
> Such ideas are often tough to pursue though as it might be too tied to
> NiFi specifically (not sure that it is) and it requires spending a good bit
> of time not specific to NiFi (not always easy to build that
> momentum/community).
>
> Definitely understand your idea and I think it is cool you'd suggest it.
> I suspect though you'll have to be in position to help do heavy lifting to
> make it a thing.
>
> Thanks
> Joe
>
> On Fri, Mar 15, 2019 at 2:07 PM Tim Zimmerman 
> wrote:
>
>> The NiFi Expression Language is quite powerful. I am working on another
>> project where I would love to have this sort of functionality and am
>> wondering how hard it would be to use nifi-expression-language outside of
>> NiFi, in some other Java app.
>>
>> I have looked at some other alternatives (e.g. JEXL, JUEL) but we are
>> already using NiFi and so are familiar with the syntax. And it seems more
>> powerful than some of the alternatives.
>>
>> How feasible is this idea?
>>
>>
>>
>> --
>> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>>
>

Re: Different NiFi Node sizes within same cluster

[Jon Logan]

Is there a way to specify the number of processing threads not in a global
setting? (ie. if a machine has 24-cores, give it a different number of
threads than the machine with 4-cores)

On Wed, Mar 6, 2019 at 3:51 PM Joe Witt  wrote:

> This is generally workable (heterogenous node capabilities) in NiFi
> clustering.  But you do want to leverage back-pressure and load balanced
> connections so that faster nodes will have an opportunity to take on the
> workload for slower nodes.
>
> Thanks
>
> On Wed, Mar 6, 2019 at 3:48 PM James Srinivasan <
> james.sriniva...@gmail.com> wrote:
>
>> Yes, we hit this with the new load balanced queues (which, to be fair, we
>> also had with remote process groups previously). Two "old" nodes got
>> saturated and their queues filled while three "new" nodes were fine.
>>
>> My "solution" was to move everything to new hardware which we had inbound
>> anyway.
>>
>> On Wed, 6 Mar 2019, 20:40 Jon Logan,  wrote:
>>
>>> You may run into issues with different processing power, as some
>>> machines may be overwhelmed in order to saturate other machines.
>>>
>>> On Wed, Mar 6, 2019 at 3:34 PM Mark Payne  wrote:
>>>
>>>> Chad,
>>>>
>>>> This should not be a problem, given that all nodes have enough storage
>>>> available to handle the influx of data.
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>>
>>>> > On Mar 6, 2019, at 1:44 PM, Chad Woodhead 
>>>> wrote:
>>>> >
>>>> > Are there any negative effects of having filesystem mounts (dedicated
>>>> mounts for each repo) used by the different NiFi repositories differ in
>>>> size on NiFi nodes within the same cluster? For instance, if some nodes
>>>> have a content_repo mount of 130 GB and other nodes have a content_repo
>>>> mount of 125 GB, could that cause any problems or cause one node to be used
>>>> more since it has more space? What about if the difference was larger, by
>>>> say a 100 GB difference?
>>>> >
>>>> > Trying to repurpose old nodes and add them as NiFi nodes, but their
>>>> mount sizes are different than my current cluster’s nodes and I’ve noticed
>>>> I can’t set the max size limit to use of a particular mount for a repo.
>>>> >
>>>> > -Chad
>>>>
>>>>

Re: Different NiFi Node sizes within same cluster

[Jon Logan]

You may run into issues with different processing power, as some machines
may be overwhelmed in order to saturate other machines.

On Wed, Mar 6, 2019 at 3:34 PM Mark Payne  wrote:

> Chad,
>
> This should not be a problem, given that all nodes have enough storage
> available to handle the influx of data.
>
> Thanks
> -Mark
>
>
> > On Mar 6, 2019, at 1:44 PM, Chad Woodhead 
> wrote:
> >
> > Are there any negative effects of having filesystem mounts (dedicated
> mounts for each repo) used by the different NiFi repositories differ in
> size on NiFi nodes within the same cluster? For instance, if some nodes
> have a content_repo mount of 130 GB and other nodes have a content_repo
> mount of 125 GB, could that cause any problems or cause one node to be used
> more since it has more space? What about if the difference was larger, by
> say a 100 GB difference?
> >
> > Trying to repurpose old nodes and add them as NiFi nodes, but their
> mount sizes are different than my current cluster’s nodes and I’ve noticed
> I can’t set the max size limit to use of a particular mount for a repo.
> >
> > -Chad
>
>

Re: How do you use a custom region?

[Jon Logan]

Andrew,

I know there's a few regions not in the list. I'm not sure which region
you're targeting, but at least for the case of one of the new regions, I
submitted a PR for this. I haven't dug into it deeply, but it seems like a
better way to do this might be to remove the enum entirely and get the
region list via the AWS API, or allow a free-form entry.

https://github.com/apache/nifi/pull/3187


Jon

On Wed, Nov 28, 2018 at 10:35 AM Andrew McDonald  wrote:

> I'm trying to upgrade from 1.4.0 to 1.7.1 but the s3 processors can not
> be initialized.
>
> Nifi 1.7.1 uses 1.11.319 and is throwing an IllegalArgumentException: no
> region provided
>
> The region I'm using isn't in the enum, so is it possible to use a
> custom region?
>
> Regards,Andrew
>
>
>

Re: Whitelisting Proxy Host values in a Container Environment?

[Jon Logan]

Thanks Andy. Would a solution involving only checking the hostname and not
the port be considered? That's where our heartache is coming from. The
documentation hints that this is supported by referencing the
format host[:port] but the port is not currently optional (unless using
80/443 I suppose).

On Thu, Oct 18, 2018 at 4:58 AM Andy LoPresto  wrote:

> Jon,
>
> Sorry to hear that securing NiFi is proving difficult for you.
>
> The reason this feature was introduced is because prior to this check
> being enforced, a malicious Host header in requests could poison the server
> and return untrusted resources. Please see the Admin Guide [1] section on
> Proxy Configuration for more details. When not running behind a proxy, this
> feature is usually very simple to configure — the hostname of the NiFi
> service is automatically included in the whitelist, so requests are handled
> successfully. The original feature request (for proxy whitelisting) is in
> NIFI-4761 [2] (unit tests [3]) if you want more detail on why it was
> implemented, or see this article [4] for more details on host header
> attacks.
>
> If your dynamic hostnames fall in a list that is assigned on demand, you
> could provide the full list (comma-separated) as the whitelisted host value
> in all nodes, and this would allow each to be verified. If not, you could
> request a feature to allow regex-parsing of a pattern for that value as
> well, but I am opposed to this as it usually introduces more problems than
> it solves.
>
> I guess you could request an explicit disabling of this header check via
> nifi.properties, but I would certainly encourage you to find a solution
> that maintains the header check if possible.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration
> [2] https://issues.apache.org/jira/browse/NIFI-4761
> [3] https://github.com/apache/nifi/pull/2279/files
> [4] https://dzone.com/articles/what-is-a-host-header-attack
>
>
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Oct 15, 2018, at 11:06 PM, Peter Wilcsinszky <
> peterwilcsins...@gmail.com> wrote:
>
> Jon,
>
> as for the port-forward you can bind NiFi to lo(required by the
> port-forward)+eth0 (I beleive eth0 is not provider dependent):
> nifi.web.https.network.interface.lo=lo
> nifi.web.https.network.interface.eth0=eth0
>
> Peter
>
> On Mon, Oct 15, 2018 at 3:46 PM Jon Logan  wrote:
>
>> Thanks Peter. I briefly was playing around with connecting to localhost
>> via the port-forwarding late last week but was having issues where Nifi was
>> internally trying to connect to 0.0.0.0 and failing...I'll toy with it some
>> more this morning (I had set the https listen property as 0.0.0.0 so that
>> it'd bind to localhost). Or I think the NodePort option will work -- but
>> will require mucking with the yaml file every deployment to avoid port
>> clashing between deployments. Mildly inconvenient, but certainly is doable.
>>
>> If this feature were disabled, certificates would still work -- in this
>> case, we know our entrypoint, but do not know the port. We could have certs
>> dynamically issued including the entrypoint hostname into the cert as an
>> alternate name. Our issue hasn't been knowing the hostname in advance, but
>> knowing the port, which isn't part of the certificate entry. We haven't
>> made it to that point yet, but it should work, and is on our roadmap.
>>
>> Again, thanks a lot -- securing this is turning out to be a lot more
>> complicated than originally anticipated.
>>
>>
>>
>>
>> On Mon, Oct 15, 2018 at 3:57 AM Peter Wilcsinszky <
>> peterwilcsins...@gmail.com> wrote:
>>
>>> Hey,
>>>
>>> I can't tell about the original intent and motivations, but this is the
>>> Jira that introduced this check [1].
>>>
>>> What I can tell is mutual TLS is not the only option to authenticate
>>> against NiFi. You can set up LDAP for example to authenticate the client
>>> and in that case MITM is possible I beleive.
>>>
>>> You will need a stable network identity that you can use to configure as
>>> your "proxy" in advance. For example in a testing scenario where you have
>>> access to the kubernetes cluster you can simply use "localhost" as the name
>>> of the proxy and use kubernetes port forward to tunnel requests from the
>>> localhost to your individual nodes (only one node at a time).
>>>
>>> Another option that could better work for non-local u

Re: Whitelisting Proxy Host values in a Container Environment?

[Jon Logan]

Thanks Peter. I briefly was playing around with connecting to localhost via
the port-forwarding late last week but was having issues where Nifi was
internally trying to connect to 0.0.0.0 and failing...I'll toy with it some
more this morning (I had set the https listen property as 0.0.0.0 so that
it'd bind to localhost). Or I think the NodePort option will work -- but
will require mucking with the yaml file every deployment to avoid port
clashing between deployments. Mildly inconvenient, but certainly is doable.

If this feature were disabled, certificates would still work -- in this
case, we know our entrypoint, but do not know the port. We could have certs
dynamically issued including the entrypoint hostname into the cert as an
alternate name. Our issue hasn't been knowing the hostname in advance, but
knowing the port, which isn't part of the certificate entry. We haven't
made it to that point yet, but it should work, and is on our roadmap.

Again, thanks a lot -- securing this is turning out to be a lot more
complicated than originally anticipated.




On Mon, Oct 15, 2018 at 3:57 AM Peter Wilcsinszky <
peterwilcsins...@gmail.com> wrote:

> Hey,
>
> I can't tell about the original intent and motivations, but this is the
> Jira that introduced this check [1].
>
> What I can tell is mutual TLS is not the only option to authenticate
> against NiFi. You can set up LDAP for example to authenticate the client
> and in that case MITM is possible I beleive.
>
> You will need a stable network identity that you can use to configure as
> your "proxy" in advance. For example in a testing scenario where you have
> access to the kubernetes cluster you can simply use "localhost" as the name
> of the proxy and use kubernetes port forward to tunnel requests from the
> localhost to your individual nodes (only one node at a time).
>
> Another option that could better work for non-local use cases is to use a
> LoadBalancer service in front of the nodes and configure DNS to point to
> your LoadBalancer IP. If you want to do this in advance it is possible to
> create floating IPs and preconfigure DNS for it at almost any cloud
> provider. Then add the configured DNS to nifi.web.proxy.host property when
> starting your cluster. If setting up DNS is not an option you can use the
> IP directly. If setting up the IP in advance is not an option you may use
> an arbitrary hostname as the proxy host and add that hostname to your hosts
> file (or dnsmasq or company dns) to point to the dynamically generated
> LoadBalancer IP after NiFi started up.
>
> Yet another option could be using Knox proxy that can act as an
> authentication gateway to your NiFi cluster. Knox can use a fully
> independent authentication scheme in front of the clients and at the same
> time authenticate against NiFi using NiFi's supported mechanisms. To get a
> grasp of how that works here is a detailed post on that matter [2].
>
> Maybe you try to connect to you NiFi nodes through Kubernetes provided
> NodePorts and want to access them directly. You can set up dedicated
> Kubernetes services if your node list is static but then you have to
> configure all the services with IP/DNS individually. Maybe you can leverage
> Knox here as well to route your request to the individual nodes.
>
> As for disabling this feature I'm not sure how that would work. You would
> still need to add the host as a SAN entry to your node certificates and
> that is something that must be set in advance. Of course you can decide to
> avoid TLS verification of the server, but then you cannot really trust the
> connection anymore. (You may think wildcard cert could be used there, but
> it is discouraged for several reasons)
>
> It would be good to know more about your general and most specifically
> about your security requirements to know what would be the easiest way to
> setup your cluster.
>
> [1] https://issues.apache.org/jira/browse/NIFI-4501
> [2]
> https://risdenk.github.io/2018/03/18/apache-knox-proxying-apache-nifi.html
>
> Cheers,
> Peter
>
> On Sun, Oct 14, 2018 at 11:24 PM Jon Logan  wrote:
>
>> Thanks Jeff. We're not using Helm (I don't think at least!) and are using
>> kubectl directly with yaml files. We are trying to avoid binding to
>> explicit external ports so that we can deploy multiple times, and I am not
>> sure how to determine the arbitrary external port assigned to be able to do
>> the property substitution. Is there a way to do this or am I approaching
>> this wrong?
>>
>> As an aside, I'm not really sure what this is accomplishing, as it seems
>> like the 2-way SSL is preventing any sort of MITM attack. Is there a way to
>> just turn off or bypass this check?
>>
>> On Sun, Oct 14, 2018 at 4:

Re: Whitelisting Proxy Host values in a Container Environment?

[Jon Logan]

Thanks Jeff. We're not using Helm (I don't think at least!) and are using
kubectl directly with yaml files. We are trying to avoid binding to
explicit external ports so that we can deploy multiple times, and I am not
sure how to determine the arbitrary external port assigned to be able to do
the property substitution. Is there a way to do this or am I approaching
this wrong?

As an aside, I'm not really sure what this is accomplishing, as it seems
like the 2-way SSL is preventing any sort of MITM attack. Is there a way to
just turn off or bypass this check?

On Sun, Oct 14, 2018 at 4:33 PM Jeff  wrote:

> Jon,
>
> I'll start off by saying that I'm still new to k8s, and don't know the
> specifics of most of this.  You should be able to inject the host and port
> of the proxy into the NiFi configuration before starting the NiFi
> instances, with a Helm chart, for instance.  I've done similar things with
> docker compose.
>
> You are correct, though, that nifi.web.proxy.host and
> nifi.web.proxy.context.path need to be set in nifi.properties before
> starting NiFi.  This is required for security purposes, and allows NiFi
> return responses which are based on the proxy host and context path values
> so that references to resources hosted by NiFi are made through the proxy,
> instead of exposing the internal URI to a resource.
>
> - Jeff
>
> On Fri, Oct 12, 2018 at 12:42 PM Jon Logan  wrote:
>
>> We are running into issues with NiFi not allowing secure connections in a
>> container due to the proxy...the only documentation we've found on this
>> involves whitelisting specific proxy addresses. Is this the only solution?
>> Specifically, we're concerned about the fact that we don't know the proxy
>> address ahead of time to whitelist -- the port is an arbitrary assigned at
>> runtime port, and the proxy name could be any of the nodes of our
>> Kubernetes cluster.
>>
>> Are we missing something?
>>
>>
>> Thanks!
>>
>

Whitelisting Proxy Host values in a Container Environment?

[Jon Logan]

We are running into issues with NiFi not allowing secure connections in a
container due to the proxy...the only documentation we've found on this
involves whitelisting specific proxy addresses. Is this the only solution?
Specifically, we're concerned about the fact that we don't know the proxy
address ahead of time to whitelist -- the port is an arbitrary assigned at
runtime port, and the proxy name could be any of the nodes of our
Kubernetes cluster.

Are we missing something?


Thanks!

Re: Certificate Issue when connecting to NiFi Registry

[Jon Logan]

There’s an extended key usage section on the cert that is missing the
client auth usage. You can reissue the cert with client and server uses.

https://knowledge.digicert.com/solution/SO18140.html#EKU


On Thu, Sep 13, 2018 at 5:36 AM Nikhil Chaudhary 
wrote:

> Hey Guys,
>
>
>
> Been stumped with a certificate issue.
>
> A bit of info on the deployment strategy.
>
>
>
> NiFi is running with a wildcard certificate in its keystore (*.domain.com)
> – It’s a self signed certificate.
>
> We’ve added the Root CA in the truststore of NiFi.
>
>
>
> We’ve used the same keystore to run NiFi registry.
>
>
>
> So installing the Root CA on my laptop, I can access NiFi on HTTPS with no
> errors or warnings.
>
> In theory the Root CA within the NiFi truststore should do the same when
> accessing NiFi registry, shouldn’t it?
>
>
>
> I enabled debug logs and the error that came up was: *Caused by:
> sun.security.validator.ValidatorException: Extended key usage does not
> permit use for TLS client authentication*
>
>
>
> The certificate only has serverAuth in it’s extended key usage but
> shouldn’t that be enough?
>
> I’ve seen emails and posts online regarding NiFi clustering in which case
> clientAuth needs to be enabled but this case seems different?
>
>
>
> ClientAuth in NiFi registry properties file is set as false.
>
> *nifi.registry.security.needClientAuth=false*
>
>
>
> Is there something I’m missing or not doing correctly?
>
>
>
> *Stack Trace:*
>
>
>
> 2018-09-12 11:02:22,581 DEBUG [NiFi Registry Web Server-15]
> org.eclipse.jetty.server.HttpConnection
>
> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>
>at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
> ~[na:1.8.0_181]
>
>at
> sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
> ~[na:1.8.0_181]
>
>at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
> ~[na:1.8.0_181]
>
>at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_181]
>
>at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
> ~[na:1.8.0_181]
>
>at
> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:621)
> ~[jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:322)
> [jetty-server-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:231)
> [jetty-server-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
> [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
> [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
> [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
> [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
> [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
> [jetty-io-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
> [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
> [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)
> [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
> [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
> [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
> [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
>at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
>
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>
>at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> ~[na:1.8.0_181]
>
>at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_181]
>
>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> ~[na:1.8.0_181]
>
>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> ~[na:1.8.0_181]
>
>at
> sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979)
> ~[na:1.8.0_181]
>
>at
> 

Re: Cluster Crashing When Nodes Unresponsive?

[Jon Logan]

We're on 1.4.0. I suspect it's because our cluster timeouts were too
high...we know some nodes were dying, but it seemed bizarre that it would
take down the whole cluster. ex. we restarted one problematic node and the
entire cluster came back to life. Should that unresponsive node failure not
be contained to just that node?

On Tue, May 1, 2018 at 8:29 PM, Juan Sequeiros <helloj...@gmail.com> wrote:

> Hi what version of NIFI are you on?
> In my experience it’s usually resource issue and most of the time it’s
> disk I/O.
>
>
> On Tue, May 1, 2018 at 4:43 PM Jon Logan <jmlo...@buffalo.edu> wrote:
>
>> We are running into issues where if a node is unresponsive for whatever
>> reason, the entire cluster seems to grind to a halt. I suspect that this is
>> because it can't replicate properly due to the node crashing, etc...is this
>> known/expected behavior? We are connecting a Nifi cluster back to itself
>> via a RPG to distribute the load, which I think is a standard design
>> practice?
>>
>> I figured that node would be penalized and/or disconnected, and life
>> would move on for the rest of the cluster?
>>
>>
>> Thanks!
>>
>

Re: Clustering Questions

[Jon Logan]

Thanks Joe, just a few follow-up questions:

re:durability -- is this something that people have just been accepting as
a risk and hoping for the best? Or is this something people build their
applications around -- ie. using durability outside of the Nifi system
boundary and push it into a database, etc?

re:heterogenous -- you can join nodes of differing hardware specs, but it
seems like you will end up causing your lighter-weight nodes to explode as
there's no way to configure how many tasks and how much to have processing
"in-flight" on the node different than the other nodes? ie. if I know my
large nodes can handle 3 of a cpu-intensive task, that's going to cause
issues for smaller nodes. This is an even bigger problem for differing
memory sizes.

And an unrelated question to the previous -- is there a way to skew or
influence how a RPG distributes its tasks? Say, you wanted to do a group-by
type distribution?


Thanks again!
Jon


On Fri, Apr 13, 2018 at 2:17 PM, Joe Witt <joe.w...@gmail.com> wrote:

> Jon,
>
> Node Failure:
>  You have to care about two things generally speaking.  First is the
> flow execution and second is data in-flight
>  For flow execution nifi clustering will take care of re-assigning the
> primary node and cluster coordinator as needed.
>  For data we do not at present offer distributed data durability.  The
> current model is predicated on using reliable storage such as RAID,
> EBS, etc..
>   There is a very clear and awesome looking K8S based path though that
> will make this work really nicely with persistent volumes and elastic
> scaling.  No clear timeline but discussions/JIRA/contributions i hope
> to start or participate in soon.
>
> How scalable is the NiFi scaling model:
>   Usually NiFi clusters are a few nodes to maybe 10-20 or so.  Some
> have been larger but generally if you're needing that much flow
> management then often it makes more sense to have clusters dedicated
> along various domains of expertise anyway.  So say 3-10 nodes with
> each handling 100,000 events per second around say 100MB per second
> (conservatively) and you can see why a single fairly small cluster can
> handle pretty massive volumes.
>
> RPGs feeding back:
> - This caused issues previously but I believe in recent releases has
> improved significantly.
>
> UI Actions Causing issues:
> There have been reports similar to this especially for some of the
> really massive flows we've seen in terms of number of components and
> concurrent users.  These JIRAs when sorted will help a lot [1], [2],
> [3].
>
> Heterogenous cluster nodes:
> - This should work quite well actually and is a major reason why NiFi
> and the S2S protocol supports/honors backpressure.  Nodes that can
> take on more work take on more work and nodes that cannot pushback.
> You also want to ensure you're using good and scalable protocols to
> source data into the cluster.  If you find you're using a lot of
> protocols requiring you to make many data sourcing steps run 'primary
> node only' then that will require that primary node to do more work
> than others and I have seen uneven behavior in such cases.  Yes, you
> can then route using S2S/RPG which we recommend but still...try to
> design away from 'primary node only' when possible.
>
>
> Thanks
> Joe
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-950
> [2] https://issues.apache.org/jira/browse/NIFI-5064
> [3] https://issues.apache.org/jira/browse/NIFI-5066
>
> On Fri, Apr 13, 2018 at 5:49 PM, Jon Logan <jmlo...@buffalo.edu> wrote:
> > All, I had a few general questions regarding Clustering, and was looking
> for
> > any sort of advice or best-practices information --
> >
> > - documentation discusses failure handling primarily from a NiFi crash
> > scenario, but I don't recall seeing any information on entire
> node-failure
> > scenarios. Is there a way that this is supposed to be handled?
> > - at what point should we expect pain in scaling? I am particularly
> > concerned about the all-to-all relationship that seems to exist if you
> > connect a cluster RPG to itself, as all nodes need to distribute all
> data to
> > all other nodes. We have been also been having some issues when things
> are
> > not as responsive as NiFi would like -- namely, the UI seems to get very
> > upset and crash
> > - do UI actions (incl read-only) require delegation to all nodes
> underneath?
> > I suspect this is the case as otherwise you wouldn't be able to determine
> > queue sizes?
> > - is there a way to have a cluster with heterogeneous node sizes?
> >
> >
> > Thanks in advance!
>

Clustering Questions

[Jon Logan]

All, I had a few general questions regarding Clustering, and was looking
for any sort of advice or best-practices information --

- documentation discusses failure handling primarily from a NiFi crash
scenario, but I don't recall seeing any information on entire node-failure
scenarios. Is there a way that this is supposed to be handled?
- at what point should we expect pain in scaling? I am particularly
concerned about the all-to-all relationship that seems to exist if you
connect a cluster RPG to itself, as all nodes need to distribute all data
to all other nodes. We have been also been having some issues when things
are not as responsive as NiFi would like -- namely, the UI seems to get
very upset and crash
- do UI actions (incl read-only) require delegation to all nodes
underneath? I suspect this is the case as otherwise you wouldn't be able to
determine queue sizes?
- is there a way to have a cluster with heterogeneous node sizes?


Thanks in advance!

Reporting Specific Queue Sizes?

[Jon Logan]

Hi All,

I am trying to get more specific queue sizes in a reporting task for metric
reporting, but I can't seem to find out a way to do it -- the only methods
exposed seem to be showing the total of all queue sizes. Is there a way
that I am missing? I am trying to find a specific queue in front of one of
my processors and grab that size.

I also considered trying to do this via the Rest API but I couldn't quite
find a way to get some of the IDs required to get some of the hops required
through the API (namely, how to list components doesn't seem clear).


Thanks in advance,
Jon