Andre, The proposed processors all sound like nice functionality. I did have a couple of questions.
Concerning the ParseKV, are you aware of the getDelimitedField[1] function in Expression Language? I think this may take care of this case for handling these items. With the QueryBulkWhois API, does it make sense to roll this into the QueryDNS as a configurable property to do batch? Performing a cursory review of the PR, it looks like this would potentially be targeting those same servers. Are batch lookups to more web service oriented endpoints as opposed to just querying DNS? --aldrin [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#getdelimitedfield On Sat, Jun 18, 2016 at 11:14 PM, Andre <[email protected]> wrote: > Johny, > > I haven't used graylog heavily so would you mind clarifying what do you > mean by > > "Would those work like graylog also" > > > > Cheers > > On Sun, Jun 19, 2016 at 12:48 PM, johny casanova < > [email protected] > > wrote: > > > Great idea! Would those work like graylog also? > > On Jun 18, 2016 9:30 PM, "Andre" <[email protected]> wrote: > > > > > Devs, > > > > > > I am continuing to drive the migration of our logging pipeline to NiFi > > and > > > in the process identified some areas of log processing that could be > > > improved by the introduction of new processors. > > > > > > I wonder Would anyone oppose the idea of introducing the following > > > processors: > > > > > > > > > 1. ParseCEF (think of it like logstash-codec-cef) > > > Processor to parse CEF format - ( > > > https://www.protect724.hpe.com/docs/DOC-1072); > > > CEF attributes would be converted into NiFi FlowFiles attributes; > > > > > > > > > 2. ParseKV (think of it like Splunk's kv parser) > > > A processor to split strings by keys and values (delimiter based) would > > be > > > added to FlowFIle attributes; > > > Parser would support extracting multiple instances of the same key via > > > attributes like parse.kv.key_name.0 , parse.kv.key_name.1, etc) > > > > > > > > > 3. QueryBulkWhoisAPI > > > This processor would read a batch of Flowfiles, extract the appropriate > > > field (e.g. ip address), make the batch whois query, parse results and > > then > > > append results to individual FlowFiles. > > > > > > This processor would complement QueryDNS (PR#496). QueryDNS only makes > > > individual queries and depending on API access conditions it may lead > to > > > blacklisting. Some providers will license access (e.g. Spamhaus RBLs), > > > while others (e.g. SHadowServer) suggest instead the use of bulk > queries. > > > > > > > > > Keen to hear your opinion > > > > > >
