Thanks for all the help, I am going to look into Metron but I think to get around my current problem of netflow sensor collection from multiple points, I am going to use a listenUDP, with an execute stream running nfdump to structure the data into a json format. I don't usually work with netflow data but couldn't a processor be built that was something like "ExtractNetFlowAttributes" Which would then load the netflow fields into attributes and allow you to use something like attributestoJSON processor to map to the proper structures? I am guessing you could even look in the file to determine version or have a drop down to specify.
Again, I am not a developer, nor do I use Netflow data enough to really know the best course. Thanks for all the help! You guys are awesome as always! Later! On Mon, Apr 17, 2017 at 7:28 PM, Andre <[email protected]> wrote: > Corey, > > I am currently assigned with a JIRA ticket to create this processor but I > have a few other tickets I am working one before I touch this one, so > unless someone has something to contribute I don't expect to get it ready > in time for 1.2. > > Cheers > > On Tue, Apr 18, 2017 at 12:57 AM, Corey Flowers <[email protected]> > wrote: > >> Good morning everyone, >> >> Was there ever a netflow parser processor built? I am currently >> working on a netflow issue and have seen several people discuss netflow >> parsing in threads but I haven't seen any information about the release of >> a netflow processor. The version I am currently working with is v9. >> >> Thanks! >> >> On Mon, Aug 8, 2016 at 10:00 AM, Madhukar Thota <[email protected] >> > wrote: >> >>> Hi Joe, >>> >>> we have bunch of cisco router that has netflow feature built in. Netflow >>> allows us to collect the network traffic from the devices and able to send >>> them to udp destination for processing. As the data comes in raw, we need >>> to parse the incoming data and do transformation and send to them hdfs, >>> kafka or elasticseach or some other destination for analytics. >>> >>> Today we are using logstash as netflow collector and able to do >>> transformation and write them into elasticsearch for visualization. As we >>> are moving all the logstash processing work to nifi, we want to move the >>> netflow parsing to nifi too. >>> >>> https://github.com/logstash-plugins/logstash-codec-netflow >>> http://blogs.cisco.com/security/step-by-step-setup-of-elk-fo >>> r-netflow-analytics >>> https://streamsets.com/documentation/datacollector/latest/he >>> lp/#Origins/UDP.html#concept_rst_2y5_1s >>> >>> >>> -Madhu >>> >>> On Mon, Aug 8, 2016 at 9:42 AM, Joe Witt <[email protected]> wrote: >>> >>>> Hello >>>> >>>> There are no processors included in the apache release that >>>> specifically operate on netflow so you'd need to have a custom >>>> processor to deal with it until one is included. >>>> >>>> Netflow is often flowing through NiFi typically for things like >>>> content merging and loading into HDFS. Parsing is a good use case and >>>> presumably after that you'd want to make some routing decisions or do >>>> some sort of enrichment? Can you describe in more detail what you'd >>>> like to be able to accomplish in NiFi and what systems it would >>>> deliver the netflow to? Also, what type of Netflow is of interest (it >>>> can be frustratingly proprietary)? >>>> >>>> Thanks >>>> Joe >>>> >>>> On Mon, Aug 8, 2016 at 9:27 AM, Madhukar Thota < >>>> [email protected]> wrote: >>>> > Is there any Processor available for Netflow? If not what is the best >>>> way to >>>> > get Netflow data parsed using nifi? >>>> > >>>> > >>>> >>> >>> >> >> >> -- >> Corey Flowers >> Vice President, Onyx Point, Inc >> (410) 541-6699 >> [email protected] >> >> -- This account not approved for unencrypted proprietary information -- >> > > -- Corey Flowers Vice President, Onyx Point, Inc (410) 541-6699 [email protected] -- This account not approved for unencrypted proprietary information --
