RE: Content repository filling up

Hi Mark, Thanks so much for looking into this. I think you are correct in that my large files are being trapped by the smaller ones in the claim. If nifi is not honouring nifi.content.claim.max.flow.files, is it honouring nifi.content.caim.max.appendable.size? If it is, then what is the

Re: NPE in ListenSyslog processor

I wonder if the cause of zero length messages is the health check from the f5 balancer. Worth verifying with your team. Andrew On Tue, Apr 25, 2017, 3:15 PM Andy LoPresto wrote: > PR 1694 [1] is available for this issue. > > [1] https://github.com/apache/nifi/pull/1694 >

Re: How to identify MiNiFi source edge devices

Jeff, That’s a great explanation and a common thought exercise scenario we’ve used when planning other features of MiNiFi. I think what Andre suggested below would be the easiest and most successful way to accomplish what you are looking for. UpdateAttribute will let you get as specific as you

Re: How to identify MiNiFi source edge devices

Aldrin, To simplify it, the situation is analogous to a deployment of temperature sensors. Each sensor has a unique ID that is assigned by us at deployment time and each sensor periodically adds a new row to a database table that is stored on the sensor. Each sensor uses the same database schema

Re: NPE in ListenSyslog processor

Hi, Thanks for the swift reply (as usual). NIFI-3738 created [1]. I have passed over to infrastructure to try and establish cause of the zero length datagrams, but at least I now know there isn’t anything fundamentally wrong here and can (safely) ignore the errors. Thanks Conrad [1]

NPE in ListenSyslog processor

Hi, Been away for a bit from this community due to other work pressures, but picking up Nifi again and successfully upgraded to 1.1.2 (apart from screwing up one of the nodes temporarily). So, with the renewed interest in log processing our infrastructure team has put in an F5 load balancer to

Re: NPE in ListenSyslog processor

Hi Conrad, Line 431 of ListenSyslog has the following code: if (!valid || !event.isValid()) So to get an NPE there means event must be null, and event comes from this code: boolean valid = true; try { event = parser.parseEvent(rawSyslogEvent.getData(), sender); } catch (final

Re: NPE in ListenSyslog processor

PR 1694 [1] is available for this issue. [1] https://github.com/apache/nifi/pull/1694 Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 25, 2017, at 10:07 AM,

Re: Content repository filling up

Olav, It does look like we are no longer honoring those properties. I created a JIRA [1] to address this. The way that it is working right now, it will write to a single Content Claim until that Content Claim exceeds 1 MB. It will then stop 'recycling' the content claim. What you may be seeing,

How to identify MiNiFi source edge devices

When processing data in NiFi that was received via MiNiFi edge devices I need to be able to identify the source of the data. All of the data on the edge devices will be pulled from a database and will not contain any data that self-identifies the source. My attempt to solve this was to write a

Re: How to identify MiNiFi source edge devices

Jeff, Would if be feasible for you use UpdateAttribute (which I believe is part of MiNiFi core processors) and use the ${hostname(true)} Expression language function? More about it can be found here: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#hostname Cheers On

Re: How to identify MiNiFi source edge devices

It is possible. I will take a look to see if the hostname is sufficient for the device ID. I just learned about the Variable Registry. It seems if I use the Variable Registry to store the device ID it would be available to the UpdateAttribute processor. Is that correct? Thanks, Jeff On Tue,

Re: How to identify MiNiFi source edge devices

Jeff, That would be next suggestion. :-) Cheers On Wed, Apr 26, 2017 at 1:04 AM, Jeff Zemerick wrote: > It is possible. I will take a look to see if the hostname is sufficient > for the device ID. > > I just learned about the Variable Registry. It seems if I use the

Re: How to identify MiNiFi source edge devices

Jeff, Could you expand upon what a device id is in your case? Something intrinsic to the device? The agent? Are these generated and assigned during provisioning? How are you making use of these when the data arrives at its desired destination? What you are expressing is certainly a common