Andre, Thanks for the insights and context, the information regarding DNS was certainly helpful and I think that seems like a pretty good breakdown of functionality and unique purpose. All seem like they would provide some neat applications for enrichment.
ParseKV definitely seems to fulfill a certain need but am curious as to how we might make it provide a bit more coverage of similar formats. With configuration that allows specification of both the key value separators (in the example you provided a space, or new line) and the delimiter of the pairings, this would possibly provide support for types of files as well, such as Properties/ini file formats. I do find myself uncertain of how much it would apply to the latter cases though. I can see how the format would map pretty nicely to columnar type stores. Might you be able to expand on how this information would typically be handled downstream? On Wed, Jun 22, 2016 at 10:35 AM, Andre <[email protected]> wrote: > Aldrin, > > On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <[email protected]> > wrote: > > > Concerning the ParseKV, are you aware of the getDelimitedField[1] > function > > in Expression Language? I think this may take care of this case for > > handling these items. > > > > I am aware of getDelimitedField but I found a few cases where using becomes > a bit challenging: > > * Multiple instances of the same key and poorly defined format(note how > just one field uses quotes): > > [email protected] [email protected] [email protected] subject="I had enough > of this" > > * Variable set of keys (tag wasn't present, now it is): > > [email protected] [email protected] [email protected] [email protected] > tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of this > > If you think if reasonably doable I am happy to reconsider. > > > > For the security folks like me, QueryBulkWhois and QueryDNS are very > different beasts: > > * QueryDNS does what a normal DNS resolver does, but because of the parsing > mechanism it can be used to handle responses in a smart way. As such one > can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco > SenderBase [1]), RBLs (Spamhaus, etc). > > * Enters QueryBulkWhois: batching optimises queries by allowing a large > number of subjects to be submitted using a single request. > > Yes, it may BulkWhois may be offered by providers that may also provide API > but these are note restricted to overlapping offerings, however projects > like "Prefix WhoIs Project" only offer Whois with no DNS API available at > all. > > > [1] > > http://stackoverflow.com/questions/14145886/how-to-programmatically-query-senderbase-org > > > > With the QueryBulkWhois API, does it make sense to roll this into the > > QueryDNS as a configurable property to do batch? Performing a cursory > > review of the PR, it looks like this would potentially be targeting those > > same servers. Are batch lookups to more web service oriented endpoints > as > > opposed to just querying DNS? > > > > --aldrin > > > > > > >
