PR 1694 [1] is available for this issue. [1] https://github.com/apache/nifi/pull/1694 <https://github.com/apache/nifi/pull/1694>
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 25, 2017, at 10:07 AM, Conrad Crampton <[email protected]> > wrote: > > Hi, > Thanks for the swift reply (as usual). > NIFI-3738 created [1]. > > I have passed over to infrastructure to try and establish cause of the zero > length datagrams, but at least I now know there isn’t anything fundamentally > wrong here and can (safely) ignore the errors. > > Thanks > Conrad > > > [1] https://issues.apache.org/jira/browse/NIFI-3738 > > On 25/04/2017, 17:46, "Bryan Bende" <[email protected]> wrote: > > Hi Conrad, > > Line 431 of ListenSyslog has the following code: > > if (!valid || !event.isValid()) > > So to get an NPE there means event must be null, and event comes from this > code: > > boolean valid = true; > try { > event = parser.parseEvent(rawSyslogEvent.getData(), sender); > } catch (final ProcessException pe) { > getLogger().warn("Failed to parse Syslog event; routing to invalid"); > valid = false; > } > > The parser returns null if the bytes sent in are null or length 0. > > We should be checking if (!valid || event == null || !event.isValid()) > to avoid this case, and I think a similar situation exists in the > ParseSyslog processor. It appears this would only happen if parsing > messages is enabled in ListenSyslog. > > Do you want to create a JIRA for this? > > The other question is why you are ending up with these 0 length > messages, but that one I am not sure about. In the case of UDP, its > just reading from a datagram channel into a byte buffer and passing > those bytes a long, so I think it means its receiving a 0 byte > datagram from the sender. > > Thanks, > > Bryan > > > On Tue, Apr 25, 2017 at 12:31 PM, Conrad Crampton > <[email protected]> wrote: >> Hi, >> >> Been away for a bit from this community due to other work pressures, but >> picking up Nifi again and successfully upgraded to 1.1.2 (apart from >> screwing up one of the nodes temporarily). >> >> So, with the renewed interest in log processing our infrastructure team has >> put in an F5 load balancer to distribute the syslog traffic I am collecting >> to my 6 node cluster. This is to stop one node being the only workhorse for >> receiving syslog traffic. I had previously used the ‘standard’ pattern of >> having the ListenSyslog processor connect to a RPG and then the rest of my >> data processing flow receive via a local port – to effectively distribute >> the processing load. I was finding though that the single node was getting >> too many warnings about buffer, sockets being full etc. – hence the external >> load balancing. >> >> >> >> I am no load balancing expert, but what I believe happens is the F5 load >> balancer receives syslog traffic (over UDP) then distributes this load to >> all Nifi nodes (gives a bit of syslog traffic to each I believe). All >> appears fine, but then I start getting NPE in my node logs thus: >> >> >> >> 2017-04-25 17:16:34,832 ERROR [Timer-Driven Process Thread-7] >> o.a.n.processors.standard.ListenSyslog >> ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] >> ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] failed to process due >> to java.lang.NullPointerException; rolling back session: >> java.lang.NullPointerException >> >> 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7] >> o.a.n.processors.standard.ListenSyslog >> >> java.lang.NullPointerException: null >> >> at >> org.apache.nifi.processors.standard.ListenSyslog.onTrigger(ListenSyslog.java:431) >> ~[nifi-standard-processors-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> ~[nifi-api-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) >> [nifi-framework-core-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) >> [nifi-framework-core-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) >> [nifi-framework-core-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) >> [nifi-framework-core-1.1.2.jar:1.1.2] >> >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_51] >> >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> [na:1.8.0_51] >> >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> [na:1.8.0_51] >> >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >> [na:1.8.0_51] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_51] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_51] >> >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] >> >> 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7] >> o.a.n.processors.standard.ListenSyslog >> ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] >> ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] failed to process >> session due to java.lang.NullPointerException: >> java.lang.NullPointerException >> >> 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7] >> o.a.n.processors.standard.ListenSyslog >> >> java.lang.NullPointerException: null >> >> at >> org.apache.nifi.processors.standard.ListenSyslog.onTrigger(ListenSyslog.java:431) >> ~[na:na] >> >> at >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> ~[nifi-api-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) >> ~[nifi-framework-core-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) >> [nifi-framework-core-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) >> [nifi-framework-core-1.1.2.jar:1.1.2] >> >> at >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) >> [nifi-framework-core-1.1.2.jar:1.1.2] >> >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_51] >> >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> [na:1.8.0_51] >> >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> [na:1.8.0_51] >> >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >> [na:1.8.0_51] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_51] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_51] >> >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] >> >> 2017-04-25 17:16:34,833 WARN [Timer-Driven Process Thread-7] >> o.a.n.processors.standard.ListenSyslog >> ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] Processor >> Administratively Yielded for 1 sec due to processing failure >> >> 2017-04-25 17:16:34,833 WARN [Timer-Driven Process Thread-7] >> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding >> ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] due to uncaught >> Exception: java.lang.NullPointerException >> >> 2017-04-25 17:16:34,834 WARN [Timer-Driven Process Thread-7] >> o.a.n.c.t.ContinuallyRunProcessorTask >> >> >> >> is what I’m trying to do screwing up the processor? If so, is this a) >> something that can be done with a load balancer – I’ve seen blog posts about >> putting an HA proxy in front of Nifi for other ListXX processors but not >> ListenSyslog, and b) if can’t be done is there suggestions on how I can >> better distribute load other than the standard pattern using RPG (which only >> distributes the processing of the data once it is in). >> >> >> >> Many thanks >> >> Conrad >> >> >> >> SecureData, combating cyber threats >> >> ________________________________ >> >> The information contained in this message or any of its attachments may be >> privileged and confidential and intended for the exclusive use of the >> intended recipient. If you are not the intended recipient any disclosure, >> reproduction, distribution or other dissemination or use of this >> communications is strictly prohibited. The views expressed in this email are >> those of the individual and not necessarily of SecureData Europe Ltd. Any >> prices quoted are only valid if followed up by a formal written quote. >> >> SecureData Europe Limited. Registered in England & Wales 04365896. >> Registered Address: SecureData House, Hermitage Court, Hermitage Lane, >> Maidstone, Kent, ME16 9NT > > > ***This email originated outside SecureData*** > > Click > https://www.mailcontrol.com/sr/Ua65DeG3YiLGX2PQPOmvUqEZpHzWxmiIfINdU0qyFhoA5x+dEtpBxXgn6ouHwVqxHTbTRgxltt8YNBM7q0TQEQ== > to report this email as spam. > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
