[
https://issues.apache.org/jira/browse/METRON-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361537#comment-15361537
]
James Sirota commented on METRON-265:
-------------------------------------
Casey, great first step in defining the architecture. A couple of quick
questions/clarifications on my side.
1. We need to build in logic into a bolt that it discovers and uses the model
service that is local to it (on the same node). So there should be a
requirement for every node that carries a storm ML bolt to also have the ML
scoring service
2. We would want caching on the bolt itself so a cache hit means there is no
call to the model service at all. If you put a cache on the model side that
means that the call still needs to be made
3. I think the way this needs to work is that you train your model once on a
global data set to come up with a global model and then push the model out to
the scoring service which is local to the ML scoring bolt. The bolt needs to
reference zookeeper to figure out which service is local to it or alternatively
assume that it always has a local service and connect to it
4. If we are using Stellar for scoring then we may be able to get away with
something faster than REST. REST is good for abstracting the complexity of the
call to the model. But if we use Stellar to abstract that call we can
potentially get away with something as basic as a socket threat pool to the
service. Thoughts? We probably still need to provide a REST endpoint for
testing and validating the model
> Provide Model as a Service infrastructure to Metron
> ---------------------------------------------------
>
> Key: METRON-265
> URL: https://issues.apache.org/jira/browse/METRON-265
> Project: Metron
> Issue Type: New Feature
> Reporter: Casey Stella
> Assignee: Casey Stella
> Attachments: Model Management Infrastructure in Metron.docx
>
>
> One of the main features envisioned and requested is the ability to augment
> the threat intelligence and enrichment processes with insights derived from
> machine learning or statistical models. The challenges with this sort of
> infrastructure are
> • Applying the model may be sufficiently computationally/resource
> intensive that we need to support scaling via load balancing, which will
> require service discovery and management.
> • Models require out of band and frequent training to react to growing
> threats and new patterns that emerge.
> • Models should be language/environment agnostic as much as possible.
> These should include small-data and big-data libraries and languages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
