Hi,
Been away for a bit from this community due to other work pressures, but
picking up Nifi again and successfully upgraded to 1.1.2 (apart from screwing
up one of the nodes temporarily).
So, with the renewed interest in log processing our infrastructure team has put
in an F5 load balancer to distribute the syslog traffic I am collecting to my 6
node cluster. This is to stop one node being the only workhorse for receiving
syslog traffic. I had previously used the ‘standard’ pattern of having the
ListenSyslog processor connect to a RPG and then the rest of my data processing
flow receive via a local port – to effectively distribute the processing load.
I was finding though that the single node was getting too many warnings about
buffer, sockets being full etc. – hence the external load balancing.
I am no load balancing expert, but what I believe happens is the F5 load
balancer receives syslog traffic (over UDP) then distributes this load to all
Nifi nodes (gives a bit of syslog traffic to each I believe). All appears fine,
but then I start getting NPE in my node logs thus:
2017-04-25 17:16:34,832 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.standard.ListenSyslog
ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf]
ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] failed to process due to
java.lang.NullPointerException; rolling back session:
java.lang.NullPointerException
2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.standard.ListenSyslog
java.lang.NullPointerException: null
at
org.apache.nifi.processors.standard.ListenSyslog.onTrigger(ListenSyslog.java:431)
~[nifi-standard-processors-1.1.2.jar:1.1.2]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
[nifi-framework-core-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.2.jar:1.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_51]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_51]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_51]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_51]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.standard.ListenSyslog
ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf]
ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] failed to process session
due to java.lang.NullPointerException: java.lang.NullPointerException
2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.standard.ListenSyslog
java.lang.NullPointerException: null
at
org.apache.nifi.processors.standard.ListenSyslog.onTrigger(ListenSyslog.java:431)
~[na:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
~[nifi-framework-core-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.2.jar:1.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_51]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_51]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_51]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_51]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
2017-04-25 17:16:34,833 WARN [Timer-Driven Process Thread-7]
o.a.n.processors.standard.ListenSyslog
ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] Processor
Administratively Yielded for 1 sec due to processing failure
2017-04-25 17:16:34,833 WARN [Timer-Driven Process Thread-7]
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] due to uncaught
Exception: java.lang.NullPointerException
2017-04-25 17:16:34,834 WARN [Timer-Driven Process Thread-7]
o.a.n.c.t.ContinuallyRunProcessorTask
is what I’m trying to do screwing up the processor? If so, is this a) something
that can be done with a load balancer – I’ve seen blog posts about putting an
HA proxy in front of Nifi for other ListXX processors but not ListenSyslog, and
b) if can’t be done is there suggestions on how I can better distribute load
other than the standard pattern using RPG (which only distributes the
processing of the data once it is in).
Many thanks
Conrad
SecureData, combating cyber threats
______________________________________________________________________
The information contained in this message or any of its attachments may be
privileged and confidential and intended for the exclusive use of the intended
recipient. If you are not the intended recipient any disclosure, reproduction,
distribution or other dissemination or use of this communications is strictly
prohibited. The views expressed in this email are those of the individual and
not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if
followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered
Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent,
ME16 9NT
