Hi Shaun, Thanks for the additional information. Apologies for not being able to follow up yesterday.
As you mentioned, combining all those discrete entries into a shared header would not fall under the capabilities of MergeContent nor do we have something that maps directly short of the ReplaceText option Andy mentioned but could work out to be a messy regex. If you have scripting savvy, InvokeScriptedProcessor/ExecuteScript could be good candidates to do a script in one of the supported options to take one of the files from MergeContent as an alternative to the ReplaceText. There is also the ability to extend the framework with custom extensions for cases like these where users start getting into particular formats of data for their systems. On Wed, Jun 22, 2016 at 1:22 PM, Andy LoPresto <[email protected]> wrote: > Shaun, > > Following the MergeContent, you can do a ReplaceText transform to remove > the duplicate message headers by matching on a regular expression. > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jun 22, 2016, at 9:43 AM, Shaun McAdams <[email protected]> > wrote: > > Aldrin, > > I checked it out. Appears that merge will concat all the lines not just > portions of it as my request would desire. > > -- > Shaun McAdams > > > From: Shaun McAdams > Reply-To: "[email protected]" > Date: Tuesday, June 21, 2016 at 10:30 AM > To: "[email protected]" > Subject: Re: Nifi Combiner Processor? > > Thanks. I’ll look through this information. > > To answer your question, the format is here: > https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/tmos_management_guide_10_1/tmos_logging.html > > And this is what I have that was sent to me: > > --------------- > These events always come through in a burst. There won’t be much, if any, > lookahead or lookbehind required. Here is an example of a few events and > how we believe the transform would look: > > *BEFORE* > 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: > Session variable 'session.policy.result' set to 'allow' > 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: > f42579b7: Session variable 'session.logon.page.errorcode' set to '0' > 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: > f42579b7: Session variable 'session.assigned.webtop' set to > '/Common/mobilevpn_webtop' > 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: > f42579b7: Session variable 'session.assigned.uuid' set to > 'tmm.uuid./Common/mobilevpn_access-policy.C069850' > 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: > f42579b7: Session variable 'session.assigned.resources.na' set to > '/Common/mobilevpn_network-acl' > 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: > f42579b7: Session variable 'session.assigned.acls' set to > '/Common/mobilevpn_acl' > > *AFTER* > 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: > Session variable*s* 'session.policy.result' set to ‘allow' > 'session.logon.page.errorcode' set to ‘0' 'session.assigned.webtop' set to > '/Common/mobilevpn_webtop’ 'session.assigned.uuid' set to > 'tmm.uuid./Common/mobilevpn_access-policy.C069850’ ' > session.assigned.resources.na' set to '/Common/mobilevpn_network-acl’ > 'session.assigned.acls' set to '/Common/mobilevpn_acl' > > ---------------- > > Thanks. > -- > Shaun McAdams > > From: Aldrin Piri > Reply-To: "[email protected]" > Date: Tuesday, June 21, 2016 at 10:04 AM > To: "[email protected]" > Subject: Re: Nifi Combiner Processor? > > Hi Shaun, > > While there is no explicit processor that will carry this out in one > action, I believe we have the tools in place for you to accomplish the > functionality with our standard processors. > > Not sure I have your exact case, but the way I approached this is through > the following sample data: > > 1,Session1,<other data> > 2,Session1,<other data> > 3,Session2,<other data> > 4,Session4,<other data> > > transforming to 3 resultant groupings: > > 1,Session1,<other data> > 2,Session1,<other data> > > 3,Session2,<other data> > > 4,Session4,<other data> > > I think SplitText[1] and ExtractText[2] with MergeContent[3] (optionally) > may be able to help you with your case. SplitText would break incoming > data into a single event line. ExtractText would be able to find your > session variable from the line and promote it to an attribute. This > attribute could then be used for the 'Correlation Attribute Name' to group > each of the separate lines together. I am a little unclear on the "want > multiple lines carrying a session variable to be group one session > variables line," but this probably gets us close if the interpretation was > incorrect. > > Feel free to provide some sample data (I'm not familiar with the F5 log > format) or some additional details if this comes up a bit short. > > [1] > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitText/index.html > [2] > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html > [3] > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html > > On Tue, Jun 21, 2016 at 9:48 AM, Shaun McAdams <[email protected]> > wrote: > >> Hey users, >> >> I was sent a request for a splunk use case to lower some of the volume >> going to enterprise spunk. Data is from an F5 (log). Easily enough they >> want some data dropped, however they also want multiple lines carrying a >> session variable to be group one session variables line. I don’t see a >> implementation of such a combiner in Nifi itself and want to make sure I’m >> not overlooking something. It appears I need to site-to-site this to a >> spark instance running the combiner. (as one possible solution for them). >> Wondered if anyone else had implemented such a use case. >> >> Thanks. >> -- >> Shaun McAdams >> >> ------------------------------ >> [image: Moser Consulting] <http://www.moserit.com/> >> >> Web <http://www.moserit.com/> | Twitter <http://j.mp/mosertwitter> | >> Facebook <http://j.mp/moserfacebook> | LinkedIn >> <http://j.mp/moserlinkedin> | Google+ <http://j.mp/mosergoogle> >> ------------------------------ >> >> CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, >> is for the sole use of the intended recipient(s) and may contain >> confidential and privileged information or may otherwise be protected by >> law. Any unauthorized review, use, disclosure or distribution is >> prohibited. If you are not the intended recipient, please contact the >> sender by reply e-mail and destroy all copies of the original message and >> any attachment thereto. >> > > >
