Simon, What do you think a good example of python, spark and MaaS would look like?
On December 7, 2017 at 07:56:00, Simon Elliston Ball ( [email protected]) wrote: I would recommend starting out with something like Spark, but the short answer is that anything that will run inside a yarn container, so the answer is most ML libraries. Using Spark to train models on the historical store is a good bet, and then using the trained models with model as a service. See https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service for information on models and some sample boilerplate for deploying your own python based models. You could as some have suggested use spark streaming, but to be honest, the spark ML models are not well suited to streaming use cases, and you would be very much breaking the metron flow rather than benefitting from elements like MaaS (you’d basically be building a 100% custom side project, which would be fine, but you’re missing a lot of the benefits of Metron that way). If you do go down that route I would strong recommend having the output of your streaming jobs feed back into a Metron sensor. To be honest though, you’re much better off training in batch and scoring / inferring via the Model as a Service approach. Simon On 6 Dec 2017, at 07:45, moshe jarusalem <[email protected]> wrote: Hi All, Would you please suggest some documentation about machine learning libraries can be used in metron architecture? and how ? any examples appretiated. regards,
