Aldrin, I checked it out. Appears that merge will concat all the lines not just portions of it as my request would desire.
-- Shaun McAdams From: Shaun McAdams Reply-To: "[email protected]<mailto:[email protected]>" Date: Tuesday, June 21, 2016 at 10:30 AM To: "[email protected]<mailto:[email protected]>" Subject: Re: Nifi Combiner Processor? Thanks. I’ll look through this information. To answer your question, the format is here: https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/tmos_management_guide_10_1/tmos_logging.html And this is what I have that was sent to me: --------------- These events always come through in a burst. There won’t be much, if any, lookahead or lookbehind required. Here is an example of a few events and how we believe the transform would look: BEFORE 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: Session variable 'session.policy.result' set to 'allow' 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: Session variable 'session.logon.page.errorcode' set to '0' 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: Session variable 'session.assigned.webtop' set to '/Common/mobilevpn_webtop' 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: Session variable 'session.assigned.uuid' set to 'tmm.uuid./Common/mobilevpn_access-policy.C069850' 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: Session variable 'session.assigned.resources.na' set to '/Common/mobilevpn_network-acl' 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: Session variable 'session.assigned.acls' set to '/Common/mobilevpn_acl' AFTER 2016-06-08T15:54:44+00:00 xxxxxxxxxxxxxx.com apd[7764] 01490007:6: f42579b7: Session variables 'session.policy.result' set to ‘allow' 'session.logon.page.errorcode' set to ‘0' 'session.assigned.webtop' set to '/Common/mobilevpn_webtop’ 'session.assigned.uuid' set to 'tmm.uuid./Common/mobilevpn_access-policy.C069850’ 'session.assigned.resources.na' set to '/Common/mobilevpn_network-acl’ 'session.assigned.acls' set to '/Common/mobilevpn_acl' ---------------- Thanks. -- Shaun McAdams From: Aldrin Piri Reply-To: "[email protected]<mailto:[email protected]>" Date: Tuesday, June 21, 2016 at 10:04 AM To: "[email protected]<mailto:[email protected]>" Subject: Re: Nifi Combiner Processor? Hi Shaun, While there is no explicit processor that will carry this out in one action, I believe we have the tools in place for you to accomplish the functionality with our standard processors. Not sure I have your exact case, but the way I approached this is through the following sample data: 1,Session1,<other data> 2,Session1,<other data> 3,Session2,<other data> 4,Session4,<other data> transforming to 3 resultant groupings: 1,Session1,<other data> 2,Session1,<other data> 3,Session2,<other data> 4,Session4,<other data> I think SplitText[1] and ExtractText[2] with MergeContent[3] (optionally) may be able to help you with your case. SplitText would break incoming data into a single event line. ExtractText would be able to find your session variable from the line and promote it to an attribute. This attribute could then be used for the 'Correlation Attribute Name' to group each of the separate lines together. I am a little unclear on the "want multiple lines carrying a session variable to be group one session variables line," but this probably gets us close if the interpretation was incorrect. Feel free to provide some sample data (I'm not familiar with the F5 log format) or some additional details if this comes up a bit short. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitText/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html [3] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html On Tue, Jun 21, 2016 at 9:48 AM, Shaun McAdams <[email protected]<mailto:[email protected]>> wrote: Hey users, I was sent a request for a splunk use case to lower some of the volume going to enterprise spunk. Data is from an F5 (log). Easily enough they want some data dropped, however they also want multiple lines carrying a session variable to be group one session variables line. I don’t see a implementation of such a combiner in Nifi itself and want to make sure I’m not overlooking something. It appears I need to site-to-site this to a spark instance running the combiner. (as one possible solution for them). Wondered if anyone else had implemented such a use case. Thanks. -- Shaun McAdams ________________________________ [Moser Consulting]<http://www.moserit.com> Web<http://www.moserit.com> | Twitter<http://j.mp/mosertwitter> | Facebook<http://j.mp/moserfacebook> | LinkedIn<http://j.mp/moserlinkedin> | Google+<http://j.mp/mosergoogle> ________________________________ CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information or may otherwise be protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and any attachment thereto.
