Would it make sense to lean on something like Apache NiFi for this? It seems a
good fit to handle getting data from wherever (web service, poll, push etc,
streams etc). If we were to build a processor which encapsulated the threat
intel loader logic, that would provide a granular route to update threat intel
entries in a more streaming manner. We could of course do the same thing in
code with storm topologies, but I would wonder whether threat intel feeds would
have enough volume to require that.
> On 16 Feb 2018, at 07:11, Ali Nazemian <alinazem...@gmail.com> wrote:
> I think one of the challenges is where the scope of threat intel ends from
> the Metron roadmap? Does it gonna relly on supporting a standard format and
> a loader to send it to HBase for the later threat intel use cases?
> In my opinion, it would be better to have a separate topology (sort of
> similar to the profiler approach) to get the feeds (maybe from Kafka) and
> load it into HBase frequently based on what criteria we want to have. Maybe
> we need to have some normalizations for the threat feeds (either aggregated
> or single feed) as an example (or any other transformation by using
> Stellar). Maybe we need to tailor row_key in a way that can be utilised
> based on the threat intel look up we want to have further from the
> enrichment topology. The problem I see with different loaders in Metron is
> we can mostly use them for the purpose of POC, but if you want to build an
> actual use case for a production platform then it will be out of the
> flexibility of a loader, so we will end up feeding data to HBase based on
> our use case.
> In this case, maybe it won't be very important we want to use an aggregator
> X or aggregator Y, we can integrate it with Metron based on integration
> On Wed, Feb 14, 2018 at 11:28 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>> We used to install soltra edge in the old ansible builds (which have
>> thankfully now been pared back in the interests of stability in full dev).
>> Soltra has not been a good option since they went proprietary, so since
>> then we’ve included opentaxii (BSD 3) as a discovery and aggregator.
>> Most of the challenges are around licensing. Hippocampe is part of The
>> Hive Project, which is AGPL, which is an apache category X license so can’t
>> be included.
>> Mindmeld is much better license-wise (Apache 2) so would be well worth
>> community consideration. I kinda like it as a framework, but
>> I for one would be very pleased to hear a broader community discussion
>> around which platforms we should have integrations with via the threat
>> intel loader, or even through a direct to hbase streaming connector.
>>> On 14 Feb 2018, at 03:13, Ali Nazemian <alinazem...@gmail.com> wrote:
>>> Hi All,
>>> I would like to understand Metron community view on Threat Intel
>>> aggregators as well as the roadmap of threat intelligence and threat
>>> hunting. There are some open source options available regarding threat
>>> intel aggregator such as Minemeld, Hippocampe, etc. Is there any plan to
>>> build that as a part of Metron in future? Is there any specific
>>> you think would be more aligned with Metron roadmap?