Re: ExecuteStateless processor

[Mark Payne]

Hello,

I recommend running:

lsof -p 

This will give you a listing of all file handles that NiFi has open. From that, 
you should be able to better understand what files (or sockets) are open.

Thanks
-Mark


On Jun 12, 2023, at 11:47 AM, Fields, Bradley E CTR (USA) 
 wrote:

We have been trying to use the ExecuteStatless processor with our kafka 
dataflow but we are having major issues. We have followed the only youtube 
video that we have found to make sure our setup is good. The issue we have is 
as soon as this processor is turned on, we start seeing the errors of “too many 
open files handles”. We are running this on a RHEL 7 environment and have set 
our limiteNOFILE to 10 in our 
/usr/lib/system/system/confluent-nifi.service. This seems to work perfect with 
the dataflow without the executestateless processor.

I will try to provide everything I can, but we are running this on a private 
system so some things will be harder to provide. Any help will be appreciated.

Bradley Fields
JRSS IT Data Engineer, TN11 JRSS
Contractor, Leidos Inc.

ExecuteStateless processor

[Fields, Bradley E CTR (USA)]

We have been trying to use the ExecuteStatless processor with our kafka
dataflow but we are having major issues. We have followed the only youtube
video that we have found to make sure our setup is good. The issue we have
is as soon as this processor is turned on, we start seeing the errors of
"too many open files handles". We are running this on a RHEL 7 environment
and have set our limiteNOFILE to 10 in our
/usr/lib/system/system/confluent-nifi.service. This seems to work perfect
with the dataflow without the executestateless processor. 

 

I will try to provide everything I can, but we are running this on a private
system so some things will be harder to provide. Any help will be
appreciated. 

 

Bradley Fields

JRSS IT Data Engineer, TN11 JRSS

Contractor, Leidos Inc.

 



smime.p7s
Description: S/MIME cryptographic signature

RE: DeleteHDFS behavior when idle

[Isha Lamboo]

Thanks Bryan and Mark,

I've created a Jira ticket describing things as well as I can. I see Bryan 
already has a better idea of the cause than my guess in the ticket, I will add 
that as a comment.

Regards,

Isha

-Oorspronkelijk bericht-
Van: Bryan Bende  
Verzonden: maandag 12 juni 2023 16:10
Aan: dev@nifi.apache.org
Onderwerp: Re: DeleteHDFS behavior when idle

The processor has @TriggerWhenEmpty so it is going to keep executing regardless 
of whether the incoming queue has data or not. I believe this was done early on 
for some processors that used Kerberos in order to allow the processor to have 
a chance to renew the Kerberos ticket, however we since moved away from need to 
do this, so unless there is another reason for having that, I would think it 
can be removed.

On Mon, Jun 12, 2023 at 9:25 AM Mark Payne  wrote:

> Isha,
>
> If you have an incoming connection, and you’re seeing this, then it’s 
> a bug. If there is no incoming connection and this processor is used 
> as a source processor, it’s normal. Either way, it has rather little 
> overhead, and you can further reduce the overhead by increasing the 
> Yield Duration in settings. This is how long it will wait between 
> invocations if there’s nothing for it to do.
>
> Either way, best to file a Jira, though, to address the behavior for 
> running unnecessarily when there’s an incoming Connection.
>
> Thanks
> -Mark
>
>
> > On Jun 12, 2023, at 8:36 AM, Isha Lamboo 
> > 
> wrote:
> >
> > Hi all,
> >
> > I have a question about behavior I see on one of our NiFi 1.18 
> > clusters
> that has a lot of xHDFS processors. When I look at the number of tasks 
> in the summary, the DeleteHDFS processors have a very high number 
> (800-1000+) of tasks even if they have nothing in their incoming 
> queues. The PutHDFS and FetchHDFS in contrast have no tasks listed 
> when they have no files in the incoming queues. Even though the tasks 
> take very little time (less than
> 100 millis per 5 mins), I’m wondering whether this causes problems 
> when the cluster is heavily loaded during peak hours.
> >
> > Is this a bug or some feature related to deleting files? Should I 
> > submit
> a ticket?
> >
> > Thanks,
> >
> > Isha
>
>

Re: [DISCUSS] Release NiFi NAR Maven Plugin 1.5.1

[Bryan Bende]

+1 Thanks Nandor, I agree it would be nice to get that fix out.

On Mon, Jun 12, 2023 at 10:05 AM Kevin Doran  wrote:

>  +1
>
> On Jun 12, 2023 at 05:35:57, Nandor Soma Abonyi
> 
> wrote:
>
> > Hello Apache NiFi Community,
> >
> > I'd like to initiate a discussion about the next release of
> > nifi-nar-maven-plugin.
> >
> > Currently, nifi-nar-maven-plugin doesn’t use remote repositories, which
> was
> > fixed in [1]. Though it is the only change since the last release, I
> > consider it significant
> > enough to suggest releasing nifi-nar-maven-plugin as 1.5.1.
> >
> > I would be happy to take on RM duties for this release.
> >
> > Do you agree it is time for a new release?
> >
> > Thanks,
> > Nandor Soma Abonyi
> >
> > [1] NIFI-11324  NAR
> > Plugin extension docs not correctly resolving overridden versions from
> > system properties
> >
> >
>

Re: DeleteHDFS behavior when idle

[Bryan Bende]

The processor has @TriggerWhenEmpty so it is going to keep executing
regardless of whether the incoming queue has data or not. I believe this
was done early on for some processors that used Kerberos in order to allow
the processor to have a chance to renew the Kerberos ticket, however we
since moved away from need to do this, so unless there is another reason
for having that, I would think it can be removed.

On Mon, Jun 12, 2023 at 9:25 AM Mark Payne  wrote:

> Isha,
>
> If you have an incoming connection, and you’re seeing this, then it’s a
> bug. If there is no incoming connection and this processor is used as a
> source processor, it’s normal. Either way, it has rather little overhead,
> and you can further reduce the overhead by increasing the Yield Duration in
> settings. This is how long it will wait between invocations if there’s
> nothing for it to do.
>
> Either way, best to file a Jira, though, to address the behavior for
> running unnecessarily when there’s an incoming Connection.
>
> Thanks
> -Mark
>
>
> > On Jun 12, 2023, at 8:36 AM, Isha Lamboo 
> wrote:
> >
> > Hi all,
> >
> > I have a question about behavior I see on one of our NiFi 1.18 clusters
> that has a lot of xHDFS processors. When I look at the number of tasks in
> the summary, the DeleteHDFS processors have a very high number (800-1000+)
> of tasks even if they have nothing in their incoming queues. The PutHDFS
> and FetchHDFS in contrast have no tasks listed when they have no files in
> the incoming queues. Even though the tasks take very little time (less than
> 100 millis per 5 mins), I’m wondering whether this causes problems when the
> cluster is heavily loaded during peak hours.
> >
> > Is this a bug or some feature related to deleting files? Should I submit
> a ticket?
> >
> > Thanks,
> >
> > Isha
>
>

Re: [DISCUSS] Release NiFi NAR Maven Plugin 1.5.1

[Kevin Doran]

 +1

On Jun 12, 2023 at 05:35:57, Nandor Soma Abonyi 
wrote:

> Hello Apache NiFi Community,
>
> I'd like to initiate a discussion about the next release of
> nifi-nar-maven-plugin.
>
> Currently, nifi-nar-maven-plugin doesn’t use remote repositories, which was
> fixed in [1]. Though it is the only change since the last release, I
> consider it significant
> enough to suggest releasing nifi-nar-maven-plugin as 1.5.1.
>
> I would be happy to take on RM duties for this release.
>
> Do you agree it is time for a new release?
>
> Thanks,
> Nandor Soma Abonyi
>
> [1] NIFI-11324  NAR
> Plugin extension docs not correctly resolving overridden versions from
> system properties
>
>

Re: DeleteHDFS behavior when idle

[Mark Payne]

Isha,

If you have an incoming connection, and you’re seeing this, then it’s a bug. If 
there is no incoming connection and this processor is used as a source 
processor, it’s normal. Either way, it has rather little overhead, and you can 
further reduce the overhead by increasing the Yield Duration in settings. This 
is how long it will wait between invocations if there’s nothing for it to do.

Either way, best to file a Jira, though, to address the behavior for running 
unnecessarily when there’s an incoming Connection.

Thanks
-Mark


> On Jun 12, 2023, at 8:36 AM, Isha Lamboo  
> wrote:
> 
> Hi all,
> 
> I have a question about behavior I see on one of our NiFi 1.18 clusters that 
> has a lot of xHDFS processors. When I look at the number of tasks in the 
> summary, the DeleteHDFS processors have a very high number (800-1000+) of 
> tasks even if they have nothing in their incoming queues. The PutHDFS and 
> FetchHDFS in contrast have no tasks listed when they have no files in the 
> incoming queues. Even though the tasks take very little time (less than 100 
> millis per 5 mins), I’m wondering whether this causes problems when the 
> cluster is heavily loaded during peak hours.
> 
> Is this a bug or some feature related to deleting files? Should I submit a 
> ticket?
> 
> Thanks,
> 
> Isha

DeleteHDFS behavior when idle

[Isha Lamboo]

Hi all,

I have a question about behavior I see on one of our NiFi 1.18 clusters that 
has a lot of xHDFS processors. When I look at the number of tasks in the 
summary, the DeleteHDFS processors have a very high number (800-1000+) of tasks 
even if they have nothing in their incoming queues. The PutHDFS and FetchHDFS 
in contrast have no tasks listed when they have no files in the incoming 
queues. Even though the tasks take very little time (less than 100 millis per 5 
mins), I’m wondering whether this causes problems when the cluster is heavily 
loaded during peak hours.

Is this a bug or some feature related to deleting files? Should I submit a 
ticket?

Thanks,

Isha

[DISCUSS] Release NiFi NAR Maven Plugin 1.5.1

[Nandor Soma Abonyi]

Hello Apache NiFi Community,

I'd like to initiate a discussion about the next release of 
nifi-nar-maven-plugin.

Currently, nifi-nar-maven-plugin doesn’t use remote repositories, which was
fixed in [1]. Though it is the only change since the last release, I consider 
it significant
enough to suggest releasing nifi-nar-maven-plugin as 1.5.1.

I would be happy to take on RM duties for this release.

Do you agree it is time for a new release?

Thanks,
Nandor Soma Abonyi

[1] NIFI-11324  NAR Plugin 
extension docs not correctly resolving overridden versions from system 
properties