Re: 答复: Re: 答复: Re: NiFi starts very slowly

[Michael Moser]

I was thinking that with 16000 processors, you would need a large amount of
heap space allocated to the JVM.  A small heap would slow NiFi's ability to
do its work, because the JVM garbage collector would be consuming more CPU
time than what is healthy.  A 64 GB heap size, allocated in the NiFi
boostrap.conf, can work.  Hopefully the system you are running this NiFi
has adequate CPU and RAM resources.

-- Mike


On Fri, Dec 29, 2017 at 9:28 AM, Mike Thomsen 
wrote:

> How many of these 1600 processor groups represent distinct functionality
> that cannot be genericized?
>
> Thanks,
>
> Mike
>
> On Thu, Dec 28, 2017 at 8:54 PM,  wrote:
>
> > Thanks for your help.
> >
> > Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the
> > work.
> >
> > There are about 10 processors in each group, so there will be 16000
> > processors in total.
> >
> > I understand that the processor should be validate before it starts, but
> > could this validation be defered until the processor (or the group it
> > belongs to) is to start for the first time
> > instead of doing this when NiFi service starts up?
> >
> >
> >
> > 发件人:
> > Joe Witt 
> > 收件人:
> > dev@nifi.apache.org
> > 日期:
> > 2017/12/29 09:20
> > 主题:
> > Re: 答复: Re: NiFi starts very slowly
> >
> >
> >
> > hello
> >
> > the total number of processors is probably a factor.  how many are there?
> > can you please list how many processors of each type there are?  Any
> > custom
> > processors?
> >
> > creating 1600 disjoint flows is probably going against the grain of nifi.
> > cant the flows be designed to operate as a directed graph amd leverage
> > attributes?
> >
> > keep in mind to start each processor we call validate.  this could be the
> > cause of the slowness.
> >
> > help us collect the above info and we can give pointers.  if you could
> > share your flow.xml.gz that would be great.
> >
> > thanks
> > joe
> >
> > On Dec 28, 2017 8:13 PM,  wrote:
> >
> > > Thanks for your reply.
> > >
> > > We are not using a cluster, our NiFi 1.3 runs in the single node mode.
> > >
> > >
> > >
> > > 发件人:
> > > "Mohammed Nadeem" 
> > > 收件人:
> > > dev@nifi.apache.org
> > > 日期:
> > > 2017/12/28 20:11
> > > 主题:
> > > Re: NiFi starts very slowly
> > >
> > >
> > >
> > > Hi Luby,
> > >
> > > If you are using a cluster based setup then Nifi is unstable due to
> > nodes
> > > getting disconnected by the cluster cordinator. Cluster coordinator
> > > disconnects nodes from the cluster when the nodes take a little longer
> > to
> > > respond ( beyond timeout) . The cluster coordinator timeout based on
> the
> > > settings in the nifi.properties file.
> > >
> > > We need to adjust the following properties
> > >
> > > *nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
> > > *nifi.cluster.node.read.timeout *=  5 sec ( by default)
> > >
> > > By Increasing the timeout ,from 5 sec to higher value would probably
> > solve
> > > a
> > > problem
> > >
> > >
> > > Thanks,
> > > Nadeem
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> > >
> > >
> > >
> > >
> > >
> > >
> > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对
> > 外
> > > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发
> > 件
> > > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
> > >
> > >
> > > This email message may contain confidential and/or privileged
> > information.
> > > If you are not the intended recipient, please do not read, save,
> > forward,
> > > disclose or copy the contents of this email or open any file attached
> to
> > > this email. We will be grateful if you could advise the sender
> > immediately
> > > by replying this email, and delete this email and any attachment or
> > links
> > > to this email completely and immediately from your computer system.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
> > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外
> > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件
> > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
> >
> >
> > This email message may contain confidential and/or privileged
> information.
> > If you are not the intended recipient, please do not read, save, forward,
> > disclose or copy the contents of this email or open any file attached to
> > this email. We will be grateful if you could advise the sender
> immediately
> > by replying this email, and delete this email and any attachment or links
> > to this email completely and immediately from your computer system.
> >
> >
> >
> >
>

Re: 答复: Re: 答复: Re: NiFi starts very slowly

[Mike Thomsen]

How many of these 1600 processor groups represent distinct functionality
that cannot be genericized?

Thanks,

Mike

On Thu, Dec 28, 2017 at 8:54 PM,  wrote:

> Thanks for your help.
>
> Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the
> work.
>
> There are about 10 processors in each group, so there will be 16000
> processors in total.
>
> I understand that the processor should be validate before it starts, but
> could this validation be defered until the processor (or the group it
> belongs to) is to start for the first time
> instead of doing this when NiFi service starts up?
>
>
>
> 发件人:
> Joe Witt 
> 收件人:
> dev@nifi.apache.org
> 日期:
> 2017/12/29 09:20
> 主题:
> Re: 答复: Re: NiFi starts very slowly
>
>
>
> hello
>
> the total number of processors is probably a factor.  how many are there?
> can you please list how many processors of each type there are?  Any
> custom
> processors?
>
> creating 1600 disjoint flows is probably going against the grain of nifi.
> cant the flows be designed to operate as a directed graph amd leverage
> attributes?
>
> keep in mind to start each processor we call validate.  this could be the
> cause of the slowness.
>
> help us collect the above info and we can give pointers.  if you could
> share your flow.xml.gz that would be great.
>
> thanks
> joe
>
> On Dec 28, 2017 8:13 PM,  wrote:
>
> > Thanks for your reply.
> >
> > We are not using a cluster, our NiFi 1.3 runs in the single node mode.
> >
> >
> >
> > 发件人:
> > "Mohammed Nadeem" 
> > 收件人:
> > dev@nifi.apache.org
> > 日期:
> > 2017/12/28 20:11
> > 主题:
> > Re: NiFi starts very slowly
> >
> >
> >
> > Hi Luby,
> >
> > If you are using a cluster based setup then Nifi is unstable due to
> nodes
> > getting disconnected by the cluster cordinator. Cluster coordinator
> > disconnects nodes from the cluster when the nodes take a little longer
> to
> > respond ( beyond timeout) . The cluster coordinator timeout based on the
> > settings in the nifi.properties file.
> >
> > We need to adjust the following properties
> >
> > *nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
> > *nifi.cluster.node.read.timeout *=  5 sec ( by default)
> >
> > By Increasing the timeout ,from 5 sec to higher value would probably
> solve
> > a
> > problem
> >
> >
> > Thanks,
> > Nadeem
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> >
> >
> >
> >
> >
> >
> > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对
> 外
> > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发
> 件
> > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
> >
> >
> > This email message may contain confidential and/or privileged
> information.
> > If you are not the intended recipient, please do not read, save,
> forward,
> > disclose or copy the contents of this email or open any file attached to
> > this email. We will be grateful if you could advise the sender
> immediately
> > by replying this email, and delete this email and any attachment or
> links
> > to this email completely and immediately from your computer system.
> >
> >
> >
> >
>
>
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外
> 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件
> 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
>
>
> This email message may contain confidential and/or privileged information.
> If you are not the intended recipient, please do not read, save, forward,
> disclose or copy the contents of this email or open any file attached to
> this email. We will be grateful if you could advise the sender immediately
> by replying this email, and delete this email and any attachment or links
> to this email completely and immediately from your computer system.
>
>
>
>

答复: Re: 答复: Re: NiFi starts very slowly

[luby]

Thanks for your help.

Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the 
work.

There are about 10 processors in each group, so there will be 16000 
processors in total.

I understand that the processor should be validate before it starts, but 
could this validation be defered until the processor (or the group it 
belongs to) is to start for the first time
instead of doing this when NiFi service starts up?



发件人: 
Joe Witt 
收件人:
dev@nifi.apache.org
日期:
2017/12/29 09:20
主题:
Re: 答复: Re: NiFi starts very slowly



hello

the total number of processors is probably a factor.  how many are there?
can you please list how many processors of each type there are?  Any 
custom
processors?

creating 1600 disjoint flows is probably going against the grain of nifi.
cant the flows be designed to operate as a directed graph amd leverage
attributes?

keep in mind to start each processor we call validate.  this could be the
cause of the slowness.

help us collect the above info and we can give pointers.  if you could
share your flow.xml.gz that would be great.

thanks
joe

On Dec 28, 2017 8:13 PM,  wrote:

> Thanks for your reply.
>
> We are not using a cluster, our NiFi 1.3 runs in the single node mode.
>
>
>
> 发件人:
> "Mohammed Nadeem" 
> 收件人:
> dev@nifi.apache.org
> 日期:
> 2017/12/28 20:11
> 主题:
> Re: NiFi starts very slowly
>
>
>
> Hi Luby,
>
> If you are using a cluster based setup then Nifi is unstable due to 
nodes
> getting disconnected by the cluster cordinator. Cluster coordinator
> disconnects nodes from the cluster when the nodes take a little longer 
to
> respond ( beyond timeout) . The cluster coordinator timeout based on the
> settings in the nifi.properties file.
>
> We need to adjust the following properties
>
> *nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
> *nifi.cluster.node.read.timeout *=  5 sec ( by default)
>
> By Increasing the timeout ,from 5 sec to higher value would probably 
solve
> a
> problem
>
>
> Thanks,
> Nadeem
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
>
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对
外
> 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发
件
> 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
>
>
> This email message may contain confidential and/or privileged 
information.
> If you are not the intended recipient, please do not read, save, 
forward,
> disclose or copy the contents of this email or open any file attached to
> this email. We will be grateful if you could advise the sender 
immediately
> by replying this email, and delete this email and any attachment or 
links
> to this email completely and immediately from your computer system.
>
>
>
>





 
本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外
披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件
人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。

 
This email message may contain confidential and/or privileged information. 
If you are not the intended recipient, please do not read, save, forward, 
disclose or copy the contents of this email or open any file attached to 
this email. We will be grateful if you could advise the sender immediately 
by replying this email, and delete this email and any attachment or links 
to this email completely and immediately from your computer system.