Aldrin, Would adding support for grok be more advantageous then specific processors to convert a particular format. Grok is very popular for such use cases.
Ryan On Wed, Jun 22, 2016 at 4:17 PM, Aldrin Piri <[email protected]> wrote: > Andre, > > Thanks for the insights and context, the information regarding DNS was > certainly helpful and I think that seems like a pretty good breakdown of > functionality and unique purpose. All seem like they would provide some > neat applications for enrichment. > > ParseKV definitely seems to fulfill a certain need but am curious as to how > we might make it provide a bit more coverage of similar formats. With > configuration that allows specification of both the key value separators > (in the example you provided a space, or new line) and the delimiter of the > pairings, this would possibly provide support for types of files as well, > such as Properties/ini file formats. I do find myself uncertain of how > much it would apply to the latter cases though. I can see how the format > would map pretty nicely to columnar type stores. > > Might you be able to expand on how this information would typically be > handled downstream? > > On Wed, Jun 22, 2016 at 10:35 AM, Andre <[email protected]> wrote: > > > Aldrin, > > > > On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <[email protected]> > > wrote: > > > > > Concerning the ParseKV, are you aware of the getDelimitedField[1] > > function > > > in Expression Language? I think this may take care of this case for > > > handling these items. > > > > > > > I am aware of getDelimitedField but I found a few cases where using > becomes > > a bit challenging: > > > > * Multiple instances of the same key and poorly defined format(note how > > just one field uses quotes): > > > > [email protected] [email protected] [email protected] subject="I had > enough > > of this" > > > > * Variable set of keys (tag wasn't present, now it is): > > > > [email protected] [email protected] [email protected] [email protected] > > tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of this > > > > If you think if reasonably doable I am happy to reconsider. > > > > > > > > For the security folks like me, QueryBulkWhois and QueryDNS are very > > different beasts: > > > > * QueryDNS does what a normal DNS resolver does, but because of the > parsing > > mechanism it can be used to handle responses in a smart way. As such one > > can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco > > SenderBase [1]), RBLs (Spamhaus, etc). > > > > * Enters QueryBulkWhois: batching optimises queries by allowing a large > > number of subjects to be submitted using a single request. > > > > Yes, it may BulkWhois may be offered by providers that may also provide > API > > but these are note restricted to overlapping offerings, however projects > > like "Prefix WhoIs Project" only offer Whois with no DNS API available at > > all. > > > > > > [1] > > > > > http://stackoverflow.com/questions/14145886/how-to-programmatically-query-senderbase-org > > > > > > > With the QueryBulkWhois API, does it make sense to roll this into the > > > QueryDNS as a configurable property to do batch? Performing a cursory > > > review of the PR, it looks like this would potentially be targeting > those > > > same servers. Are batch lookups to more web service oriented endpoints > > as > > > opposed to just querying DNS? > > > > > > --aldrin > > > > > > > > > > > >
