Re: Maximum load

2020-09-20 Thread Phil H 
Hi there,

The issue I have isn’t my processor, it’s the fact I can’t get Nifi to run
more threads. I’ve noticed that if I bump the max thread counts in the
controller settings to total over 256, then nifi as a whole locks up and I
have to clear the state and restart it.

Thanks,
Phil

On Mon, 21 Sep 2020 at 11:01, Mark Payne  wrote:

> Phil,
>
>
>
> It sounds like you’re dealing with a large amount of lock contention.
> Either using explicit java.util.concurrency.Lock or synchronized blocks.
> Would review the code for lock contention or run a profiler against nifi
> and see what that shows.
>
>
>
> Sent from my iPhone
>
>
>
> > On Sep 20, 2020, at 8:30 PM, Phil H  wrote:
>
> >
>
> > Various numbers - max of 96 so far.
>
> >
>
> >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard <
> pierre.villard...@gmail.com>
>
> >> wrote:
>
> >>
>
> >> How many concurrent tasks did you set on the two custom processors?
>
> >>
>
> >>
>
> >>
>
> >> Le sam. 19 sept. 2020 à 02:16, Phil H  a écrit :
>
> >>
>
> >>
>
> >>
>
> >>> Basically I have two custom processors that are slow (data intensive)
>
> >> that
>
> >>
>
> >>> create backlogs. All other aspects of the flow work well - I am running
>
> >> SSD
>
> >>
>
> >>> disks so that sort of I/O is not biting me. The process itself is
> highly
>
> >>
>
> >>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the
>
> >> CPU
>
> >>
>
> >>> load. This is the only purpose this machine has - I want it to work
> much
>
> >>
>
> >>> harder!
>
> >>
>
> >>>
>
> >>
>
> >>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard <
>
> >> pierre.villard...@gmail.com>
>
> >>
>
> >>> wrote:
>
> >>
>
> >>>
>
> >>
>
>  Hi,
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
>  Do you have backpressure with flow files accumulating somewhere in
> your
>
> >>
>
> 
>
> >>
>
>  flow? If not, then you probably don't need more threads.
>
> >>
>
> 
>
> >>
>
>  If you do find bottlenecks in your flow design, then you can look at
>
> >>
>
> 
>
> >>
>
>  increasing the concurrent tasks on specific processors to have
>
> >>
>
> 
>
> >>
>
>  multi-threading. Note that it's not always the solution.
>
> >>
>
> 
>
> >>
>
>  Changing the pool size does not mean NiFi will use more resources.
>
> >>
>
> 
>
> >>
>
>  Actually, I usually recommend changing the pool size only when you see
>
> >>
>
> >>> that
>
> >>
>
> 
>
> >>
>
>  all the threads available are used.
>
> >>
>
> 
>
> >>
>
>  Overall we'd need more details about your flow and observations.
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
>  Thanks,
>
> >>
>
> 
>
> >>
>
>  Pierre
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
>  Le ven. 18 sept. 2020 à 03:49, Phil H  a écrit :
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
> > Hi there,
>
> >>
>
> 
>
> >>
>
> >
>
> >>
>
> 
>
> >>
>
> > Regardless of what I set in the maximum thread count settings, I
>
> >> never
>
> >>
>
>  see
>
> >>
>
> 
>
> >>
>
> > more than 128 active threads. The CPU load of the system is only 5-8%
>
> >>
>
>  when
>
> >>
>
> 
>
> >>
>
> > NiFi is running at this rate. It’s a Solaris system with 64 cores.
>
> >>
>
> 
>
> >>
>
> >
>
> >>
>
> 
>
> >>
>
> > Any help getting more performance would be appreciated!
>
> >>
>
> 
>
> >>
>
> >
>
> >>
>
> 
>
> >>
>
> > Regards,
>
> >>
>
> 
>
> >>
>
> > Phil
>
> >>
>
> 
>
> >>
>
> >
>
> >>
>
> 
>
> >>
>
> 
>
> >>
>
> >>>
>
> >>
>
> >>
>
>

Re: Maximum load

2020-09-20 Thread Mark Payne 
Phil,

It sounds like you’re dealing with a large amount of lock contention. Either 
using explicit java.util.concurrency.Lock or synchronized blocks. Would review 
the code for lock contention or run a profiler against nifi and see what that 
shows.

Sent from my iPhone

> On Sep 20, 2020, at 8:30 PM, Phil H  wrote:
> 
> Various numbers - max of 96 so far.
> 
>> On Sat, 19 Sep 2020 at 21:44, Pierre Villard 
>> wrote:
>> 
>> How many concurrent tasks did you set on the two custom processors?
>> 
>> 
>> 
>> Le sam. 19 sept. 2020 à 02:16, Phil H  a écrit :
>> 
>> 
>> 
>>> Basically I have two custom processors that are slow (data intensive)
>> that
>> 
>>> create backlogs. All other aspects of the flow work well - I am running
>> SSD
>> 
>>> disks so that sort of I/O is not biting me. The process itself is highly
>> 
>>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the
>> CPU
>> 
>>> load. This is the only purpose this machine has - I want it to work much
>> 
>>> harder!
>> 
>>> 
>> 
>>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard <
>> pierre.villard...@gmail.com>
>> 
>>> wrote:
>> 
>>> 
>> 
 Hi,
>> 
 
>> 
 
>> 
 
>> 
 Do you have backpressure with flow files accumulating somewhere in your
>> 
 
>> 
 flow? If not, then you probably don't need more threads.
>> 
 
>> 
 If you do find bottlenecks in your flow design, then you can look at
>> 
 
>> 
 increasing the concurrent tasks on specific processors to have
>> 
 
>> 
 multi-threading. Note that it's not always the solution.
>> 
 
>> 
 Changing the pool size does not mean NiFi will use more resources.
>> 
 
>> 
 Actually, I usually recommend changing the pool size only when you see
>> 
>>> that
>> 
 
>> 
 all the threads available are used.
>> 
 
>> 
 Overall we'd need more details about your flow and observations.
>> 
 
>> 
 
>> 
 
>> 
 Thanks,
>> 
 
>> 
 Pierre
>> 
 
>> 
 
>> 
 
>> 
 Le ven. 18 sept. 2020 à 03:49, Phil H  a écrit :
>> 
 
>> 
 
>> 
 
>> 
> Hi there,
>> 
 
>> 
> 
>> 
 
>> 
> Regardless of what I set in the maximum thread count settings, I
>> never
>> 
 see
>> 
 
>> 
> more than 128 active threads. The CPU load of the system is only 5-8%
>> 
 when
>> 
 
>> 
> NiFi is running at this rate. It’s a Solaris system with 64 cores.
>> 
 
>> 
> 
>> 
 
>> 
> Any help getting more performance would be appreciated!
>> 
 
>> 
> 
>> 
 
>> 
> Regards,
>> 
 
>> 
> Phil
>> 
 
>> 
> 
>> 
 
>> 
 
>> 
>>> 
>> 
>>

Re: Maximum load

2020-09-20 Thread Phil H 
Various numbers - max of 96 so far.

On Sat, 19 Sep 2020 at 21:44, Pierre Villard 
wrote:

> How many concurrent tasks did you set on the two custom processors?
>
>
>
> Le sam. 19 sept. 2020 à 02:16, Phil H  a écrit :
>
>
>
> > Basically I have two custom processors that are slow (data intensive)
> that
>
> > create backlogs. All other aspects of the flow work well - I am running
> SSD
>
> > disks so that sort of I/O is not biting me. The process itself is highly
>
> > ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the
> CPU
>
> > load. This is the only purpose this machine has - I want it to work much
>
> > harder!
>
> >
>
> > On Fri, 18 Sep 2020 at 20:56, Pierre Villard <
> pierre.villard...@gmail.com>
>
> > wrote:
>
> >
>
> > > Hi,
>
> > >
>
> > >
>
> > >
>
> > > Do you have backpressure with flow files accumulating somewhere in your
>
> > >
>
> > > flow? If not, then you probably don't need more threads.
>
> > >
>
> > > If you do find bottlenecks in your flow design, then you can look at
>
> > >
>
> > > increasing the concurrent tasks on specific processors to have
>
> > >
>
> > > multi-threading. Note that it's not always the solution.
>
> > >
>
> > > Changing the pool size does not mean NiFi will use more resources.
>
> > >
>
> > > Actually, I usually recommend changing the pool size only when you see
>
> > that
>
> > >
>
> > > all the threads available are used.
>
> > >
>
> > > Overall we'd need more details about your flow and observations.
>
> > >
>
> > >
>
> > >
>
> > > Thanks,
>
> > >
>
> > > Pierre
>
> > >
>
> > >
>
> > >
>
> > > Le ven. 18 sept. 2020 à 03:49, Phil H  a écrit :
>
> > >
>
> > >
>
> > >
>
> > > > Hi there,
>
> > >
>
> > > >
>
> > >
>
> > > > Regardless of what I set in the maximum thread count settings, I
> never
>
> > > see
>
> > >
>
> > > > more than 128 active threads. The CPU load of the system is only 5-8%
>
> > > when
>
> > >
>
> > > > NiFi is running at this rate. It’s a Solaris system with 64 cores.
>
> > >
>
> > > >
>
> > >
>
> > > > Any help getting more performance would be appreciated!
>
> > >
>
> > > >
>
> > >
>
> > > > Regards,
>
> > >
>
> > > > Phil
>
> > >
>
> > > >
>
> > >
>
> > >
>
> >
>
>

Re: APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX OS

2020-09-20 Thread Russell Bateman 

Google is your friend.

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-configuration-best-practices/content/basic-cluster-setup.html
https://bryanbende.com/development/2018/10/23/apache-nifi-secure-cluster-setup
https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/
https://www.nifi.rocks/apache-nifi-docker-compose-cluster/

etc.

On 9/19/20 7:33 PM, Abiodun Adegbile wrote:

*/Hello Team,/*

Good to write to you.

I am planning to implement a NIFI cluster installation for two 
separate physical linux servers(to-be-clustered) for my test lab. I 
was hoping you could send a walkthrough documentation for this.


Many thanks for your help. Will appreciate your prompt response.

*/
/*

*/
/*

*/KEEP SAFE, STAY SAFE & STAY STRONG/*

/
/

/I celebrate you.../


/*/ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP)/*/

*/Database Administrator/Infrastructure Services/*

*/Bluechip Technologies Limited/*

*/Plot 9B, Onikoyi Lane/*

*/Ikoyi. Lagos State./*

*/Mobile : +234 806 206 9970/*

*/Website : www.bluechiptech.biz /*

*/
/*

.../*2016 Winner Oracle Transformational Deal of the Year - Nigeria*./