Re: Too many open files

[Abdou B]

Hello,

It seems to me that for some distribution, you should modify those values
in the Cluster management tool.
For example in HDF, with Ambari, you should change the parameter
: nifi_user_nofile_limit. for the change to take effect.

Best regards
Abdou

Le mer. 18 sept. 2019 à 16:34, Jean-Sebastien Vachon 
a écrit :

> Does not seem to help... processes are still limited to 4096 fds
> --
> *From:* Jean-Sebastien Vachon 
> *Sent:* Wednesday, September 18, 2019 10:31 AM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Too many open files
>
> Oups.. just saw the following:
>
> Your distribution may require an edit to
> */etc/security/limits.d/90-nproc.conf* by adding:
> * soft nproc 1
>
> I will try this
> --
> *From:* Jean-Sebastien Vachon 
> *Sent:* Wednesday, September 18, 2019 10:30 AM
> *To:* users@nifi.apache.org 
> *Subject:* Too many open files
>
> Hi all,
>
> I've started to see "Too many open files" error messages in Nifi. I
> checked https://nifi.apache.org/quickstart.html to see the recommended
> values to fix this
> and made the required changes to /etc/security/limits.conf, exited my
> shell and restarted Nifi. When I check the limits of the Java processes I
> can still see the limits to be at 4096
>
> I've added the following to /etc/security/limits.conf
> *  hard  nofile  5
> *  soft  nofile  5
>
> but the processes show this:
>
>  cat /proc/26861/limits
> ...
> Max open files4096 4096 files
> ...
>
> Any idea where this 4096 comes from? I tried grepping in the init scripts,
> nifi configuration and nifi-env.sh but could not find this anywhere
>
> thanks
>

Re: How to only get the newest flowfile from a queue

[Abdou B]

Hello,
Did you tried the standard processors getfile ?
Minimum File Age + Maximuin File Age / Polling Interval  should get the job
done ?

Best Regards.


Le jeu. 13 sept. 2018 à 13:50, DEHAY Aurelien 
a écrit :

> Hello.
>
>
>
> Simpliest solution in my opinion, rather than trying to do that entirely
> in nifi, is to use an shell script and run it via ExecuteStreamCommand as
> an input for FetchFile.
>
>
>
> -- Aurélien
>
>
>
> *From:* scott [mailto:tcots8...@gmail.com]
> *Sent:* mercredi 12 septembre 2018 16:56
> *To:* users@nifi.apache.org
> *Subject:* How to only get the newest flowfile from a queue
>
>
>
> Hello community,
>
> I'm trying to figure out a flow that will run periodically to process the
> latest file in a directory. If the flow doesn't run for a while, there may
> be multiple files found. My logic does not need to process the older files
> because the newest file always trumps the older ones, and older files can
> be discarded. Is there a way to filter the content of a queue this way?
>
>
>
> Thanks,
>
> Scott
>
> This electronic transmission (and any attachments thereto) is intended
> solely for the use of the addressee(s). It may contain confidential or
> legally privileged information. If you are not the intended recipient of
> this message, you must delete it immediately and notify the sender. Any
> unauthorized use or disclosure of this message is strictly prohibited.
> Faurecia does not guarantee the integrity of this transmission and shall
> therefore never be liable if the message is altered or falsified nor for
> any virus, interception or damage to your system.
>

Multi Domains Nifi connection and UI acces

[Abdou B]

Hello,

 I use Nifi 1.5 and I would like to use the functionnality that enables
Nifi to use various network interface.
So as stated in the offficial documentation, i created the following
properties

nifi.web.https.network.interface.default=0.0.0.0

nifi.web.https.network.interface.eth0=eth0
nifi.web.https.network.interface.eth1=eth1
nifi.web.https.network.interface.eth2=eth2
nifi.web.https.network.interface.eth3=eth3
nifi.web.https.network.interface.eth4=eth4

I use a certificate with SAN, the SAN includes : FQDN eth0 +  FQDN eth1 + FQDN
eth2  + FQDN eth3 +  FQDN eth4

I have to use eth1 to connect to the UI
so i set up  :
nifi.web.https.host=the FQDN of eth0
but I have an error stating that
"Cluster is still in the process of voting on the appropriate Data Flow. "

if I use the  eth0, I can't connect anymore (I got a message about header
not allowed) but the cluster seems to work.
I saw that jetty seems to be more restrictive than before and allow only
host that use the  nifi.web.https.host to connect to UI


would you have any advice ?

Best Regards

Multi Domains Nifi connectoon to UI

[Abdou B]

Hello,

 I use Nifi 1.5 and I would like to use the functionnality that enables
Nifi to use various network interface.
So as stated in the offficial documentation, i created the following
properties

nifi.web.https.network.interface.default=0.0.0.0

nifi.web.https.network.interface.eth0=eth0
nifi.web.https.network.interface.eth1=eth1
nifi.web.https.network.interface.eth2=eth2
nifi.web.https.network.interface.eth3=eth3
nifi.web.https.network.interface.eth4=eth4

I use a certificate with SAN, the SAN includes : FQDN eth0 +  FQDN eth1 + FQDN
eth2  + FQDN eth3 +  FQDN eth4

I have to use eth1 to connect to the UI
so i set up  :
nifi.web.https.host=the FQDN of eth0
but I have an error stating that
"Cluster is still in the process of voting on the appropriate Data Flow. "

if I use the  eth0, I can't connect anymore (I got a message about header
not allowed) but the cluster seems to work.
I saw that jetty seems to be more restrictive than before and allow only
host that use the  nifi.web.https.host to connect to UI


would you have any advice ?

Best Regards
Abdou

Re: Reporting Specific Queue Sizes?

[Abdou B]

Hello guys,

I am also interested in this discussion.
I saw the blog post of Pierre :
https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/



Unfortunately , I have versions of Nifi older than 1.2.
You may view/analyze Nifi built monitoring.

If you go close to logout. Click on Summary.
Go to the Connections Tab.
You may sort by Queue / Size Thresold


But i did not find any SiteToSiteStatusReportingTask...
So I would like to know if there is any way to make some reporting on Nifi
1.1.0 ?


Best Regards
Abdou




2018-03-08 7:31 GMT+01:00 Pierre Villard :

> Hi Jon,
>
> You could use the S2S Status reporting task and filter the data on
> Connection and, if need, on a specific connection name. It'll send you JSON
> data containing the information about the queue (stats, size, backpressure,
> etc).
>
> Pierre
>
> 2018-03-08 6:12 GMT+01:00 Jon Logan :
>
>> Hi All,
>>
>> I am trying to get more specific queue sizes in a reporting task for
>> metric reporting, but I can't seem to find out a way to do it -- the only
>> methods exposed seem to be showing the total of all queue sizes. Is there a
>> way that I am missing? I am trying to find a specific queue in front of one
>> of my processors and grab that size.
>>
>> I also considered trying to do this via the Rest API but I couldn't quite
>> find a way to get some of the IDs required to get some of the hops required
>> through the API (namely, how to list components doesn't seem clear).
>>
>>
>> Thanks in advance,
>> Jon
>>
>
>