F5 was sending zero length messages, now it’s not and not seeing NPE any more. Despite this, excellent response to the issue from the community yet again – less than 12 hours to from report to fix! Thank you
From: Andrew Grande <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, 25 April 2017 at 23:08 To: "[email protected]" <[email protected]> Subject: Re: NPE in ListenSyslog processor I wonder if the cause of zero length messages is the health check from the f5 balancer. Worth verifying with your team. Andrew On Tue, Apr 25, 2017, 3:15 PM Andy LoPresto <[email protected]<mailto:[email protected]>> wrote: PR 1694 [1] is available for this issue. [1] https://github.com/apache/nifi/pull/1694 Andy LoPresto [email protected]<mailto:[email protected]> [email protected]<mailto:[email protected]> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Apr 25, 2017, at 10:07 AM, Conrad Crampton <[email protected]<mailto:[email protected]>> wrote: Hi, Thanks for the swift reply (as usual). NIFI-3738 created [1]. I have passed over to infrastructure to try and establish cause of the zero length datagrams, but at least I now know there isn’t anything fundamentally wrong here and can (safely) ignore the errors. Thanks Conrad [1] https://issues.apache.org/jira/browse/NIFI-3738 On 25/04/2017, 17:46, "Bryan Bende" <[email protected]<mailto:[email protected]>> wrote: Hi Conrad, Line 431 of ListenSyslog has the following code: if (!valid || !event.isValid()) So to get an NPE there means event must be null, and event comes from this code: boolean valid = true; try { event = parser.parseEvent(rawSyslogEvent.getData(), sender); } catch (final ProcessException pe) { getLogger().warn("Failed to parse Syslog event; routing to invalid"); valid = false; } The parser returns null if the bytes sent in are null or length 0. We should be checking if (!valid || event == null || !event.isValid()) to avoid this case, and I think a similar situation exists in the ParseSyslog processor. It appears this would only happen if parsing messages is enabled in ListenSyslog. Do you want to create a JIRA for this? The other question is why you are ending up with these 0 length messages, but that one I am not sure about. In the case of UDP, its just reading from a datagram channel into a byte buffer and passing those bytes a long, so I think it means its receiving a 0 byte datagram from the sender. Thanks, Bryan On Tue, Apr 25, 2017 at 12:31 PM, Conrad Crampton <[email protected]<mailto:[email protected]>> wrote: Hi, Been away for a bit from this community due to other work pressures, but picking up Nifi again and successfully upgraded to 1.1.2 (apart from screwing up one of the nodes temporarily). So, with the renewed interest in log processing our infrastructure team has put in an F5 load balancer to distribute the syslog traffic I am collecting to my 6 node cluster. This is to stop one node being the only workhorse for receiving syslog traffic. I had previously used the ‘standard’ pattern of having the ListenSyslog processor connect to a RPG and then the rest of my data processing flow receive via a local port – to effectively distribute the processing load. I was finding though that the single node was getting too many warnings about buffer, sockets being full etc. – hence the external load balancing. I am no load balancing expert, but what I believe happens is the F5 load balancer receives syslog traffic (over UDP) then distributes this load to all Nifi nodes (gives a bit of syslog traffic to each I believe). All appears fine, but then I start getting NPE in my node logs thus: 2017-04-25 17:16:34,832 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.ListenSyslog ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] failed to process due to java.lang.NullPointerException; rolling back session: java.lang.NullPointerException 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.ListenSyslog java.lang.NullPointerException: null at org.apache.nifi.processors.standard.ListenSyslog.onTrigger(ListenSyslog.java:431) ~[nifi-standard-processors-1.1.2.jar:1.1.2] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.2.jar:1.1.2] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.2.jar:1.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_51] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_51] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.ListenSyslog ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] failed to process session due to java.lang.NullPointerException: java.lang.NullPointerException 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.ListenSyslog java.lang.NullPointerException: null at org.apache.nifi.processors.standard.ListenSyslog.onTrigger(ListenSyslog.java:431) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.2.jar:1.1.2] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) ~[nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.2.jar:1.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_51] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_51] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] 2017-04-25 17:16:34,833 WARN [Timer-Driven Process Thread-7] o.a.n.processors.standard.ListenSyslog ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] Processor Administratively Yielded for 1 sec due to processing failure 2017-04-25 17:16:34,833 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] due to uncaught Exception: java.lang.NullPointerException 2017-04-25 17:16:34,834 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask is what I’m trying to do screwing up the processor? If so, is this a) something that can be done with a load balancer – I’ve seen blog posts about putting an HA proxy in front of Nifi for other ListXX processors but not ListenSyslog, and b) if can’t be done is there suggestions on how I can better distribute load other than the standard pattern using RPG (which only distributes the processing of the data once it is in). Many thanks Conrad SecureData, combating cyber threats ________________________________ The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote. SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT ***This email originated outside SecureData*** Click https://www.mailcontrol.com/sr/Ua65DeG3YiLGX2PQPOmvUqEZpHzWxmiIfINdU0qyFhoA5x+dEtpBxXgn6ouHwVqxHTbTRgxltt8YNBM7q0TQEQ== to report this email as spam.
