Hi Sanket, thanks for sharing! Can you elaborate a bit more on your experience and challenges with model deployment?
> We have another interesting use case where we kind of started prototyping Metron – financial fraud. Although it might sound a very different and unrelated domain, the “technical architectural pattern” is astonishingly similar. TBH, you could probably view Metron even more broadly than that. Fundamentally, it's a streaming analytics platform with some emphasis on cybersecurity to keep things a bit more focused. But I see absolutely no reason why you couldn't replace terminology like "sensor/parser" with something more generalized such as "data source." We get data into the system, normalize it, provide hooks for enhancing (enriching) that data via a variety of sources including machine learning models, and flag records and provide a highly configurable method to score them. I mean, why not use this for genomics? Or dynamic live traffic adjustments? Or stock trading? Etc... On Wed, Nov 27, 2019, 4:01 PM Sanket Sharma <sanket.sha...@dukstra.com> wrote: > Hi, > > > > Thank you for starting a great discussion! We started exploring Metron in > June this for networking monitoring. We are piloting it with an objective > of replacing Splunk in certain or perhaps all scenarios. We’re looking at > about 2 TB of data per day. > > 1. Features we are currently considering: > 1. Enrichments > 2. Streaming enhancements: We are using Spark to do some > enrichments but need to explore this further. > 3. Profiler: Not using it at the moment > 4. Pcap: Not using it at the moment. > 5. Flatfile summarizer: Not using it at the moment. > 6. MaaS: IMHO this needs serious usability enhancements, especially > for data scientists. Deploying models seems like a common issue that > most > data scientist struggle with (at least in our area, unless they have > serious python/engineering skills.). > 7. Meta alerts: Not using it at the moment > 8. Parser aggregation: Limited use > 9. Config UI: Using it extensively to configure sensors and rules. > 10. Alert UI: Using it extensive to view alerts. > 11. Elastic search: Using it extensively to index alerts and other > data. > 12. Stellar: Not using it at the moment, except for creating rules > with scores in the config UI. > 13. Stellar REPL: Not using it at all > 14. REST API: Not using it explicitly. > 15. Other? > 2. Many features around usability can be improved: > 1. Model deployment can reconsidered as a whole. > 2. Ability to compare models > 3. Config UI field configuration could be improved > 4. General ease of use/deployment, documentation > 5. Templates for common use cases > 6. Reports – we just can’t do without reporting in the enterprise ☺ > 3. Alerts UI, Stellar and pipelines I suppose. > 4. I would love to contribute ☺, just in the middle of a big > relocation. Hopefully, I will be able to resume and join the community in > next 2-3 months. > > > > We have another interesting use case where we kind of started prototyping > Metron – financial fraud. Although it might sound a very different and > unrelated domain, the “technical architectural pattern” is astonishingly > similar. We receive streaming and batch data from various channels over > kafka, gets enriched and the based on certain rules we assign a score to > it. It then makes it to the alert UI where investigators can further > examine the transactions. This is obviously an oversimplification, but I > hope you get the idea. > > > > I was thinking of proposing a fork or perhaps a different “flavour” of > metron that caters for finance domain and can be built as a separate > project, although not sure how to go about it. Is that something the > community/project owners might be interested in considering or supporting? > > > > Best regards, > > Sanket > > > > *From: *Michael Miklavcic <michael.miklav...@gmail.com> > *Reply to: *"u...@metron.apache.org" <u...@metron.apache.org> > *Date: *Thursday, 17 October 2019 at 18:22 > *To: *"dev@metron.apache.org" <dev@metron.apache.org>, " > u...@metron.apache.org" <u...@metron.apache.org> > *Subject: *[DISCUSS] How are you using in Metron? > > > > I'd like to kick off a discussion to get a sense of how the broader > community is currently using Metron. > > 1. What features are you using or seriously considering? e.g. > > 1. enrichments > > 2. streaming enrichments > > 3. profiler > > 4. pcap > > 5. flatfile summarizer > > 6. MaaS > > 7. Meta alerts > > 8. parser aggregation > > 9. config UI > > 10. alert UI > > 11. solr, ES > > 12. Stellar > > 13. Stellar REPL > > 14. REST API > > 15. other? > > 2. What features would you like to see added or improved? > > 3. What features do you consider to be core to Metron as a platform? > > 4. If you're using Metron, but not an active community contributor, > what would it take to get you more involved in the project? > > We are close to finishing up a feature branch around upgrading to HDP 3.1, > and subsequently on the doorstep of a 1.0 release. This is a huge milestone > for the project. I think it's time to take some lessons learned over the > past several years and consider what the next phase of Metron will be. > Whether you've participated in community discussions before or not, we'd > love to hear from you. > > > > Best, > > Mike Miklavcic > > PMC Apache Metron >
