I agree with that wholeheartedly. The more integrations in and out of the stack at various points, the better.
Jon On Mon, Aug 27, 2018, 02:44 Ali Nazemian <alinazem...@gmail.com> wrote: > One thing that we could imagine for v1.0 might be an ability to extend > Metron from adding more pipelines to it. For example, being able to extend > Metron to be integrated with other endpoints more easily from Storm > perspective. For example, what if we would like to create other topologies > to write files in ORC directly rather than HDFS or index it to Druid. What > if we would like to build an automatic security response and move to become > SOAR. All these integrations can be done even now and there are other users > that may have done it already. However, providing a clear extension point > can make it easier to contribute other pipelines to the community. I think > by adding this level of extendability, Metron community will grow way > faster by adding more integration that can be available. > > Cheers, > Ali > > On Mon, Aug 20, 2018 at 10:50 PM Casey Stella <ceste...@gmail.com> wrote: > > > I completely agree, Mike. Our docs are either very high level or very > low > > level (and possibly stale) and, worse, aren't aimed at the actors that > > you've stated. > > I think that the HBase project does a good job of providing coherent and > > useable documentation in their "HBase Book" (see > > https://hbase.apache.org/book.html). > > It's not actor-specific, but it is coherent advice for the practical > > practitioner of HBase (both admin and developer) and speaks with one > > voice. I think Metron's need > > is a bit different, but at the minimum some coherent docs that speaks > with > > one voice and has a coherent pitch about what Metron is used for and what > > it isn't used for > > is well needed. > > > > On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Apologies for any spelling mishaps as I'm writing from my phone. > > > > > > I'm for improving our docs. I'd like to see us guide our various > profiles > > > of user towards the specific documentation for the abstraction levels > > > they'll be most interested in working from. I think we should have > > platform > > > docs about how we're a broadly useful, extensible streaming analytics > > > platform for cyber security as well as docs that emphasize more narrow > > and > > > specific use cases. > > > > > > Personally, I think I see 3 potential tiers or classifications of docs. > > > These are just observations and ideas I had, not necessarily a > > prescription > > > for organizing docs: > > > - Low level tool instructions, eg > > > - how do I run the pcap toplogy and then query with the CLI and UI? > > > - Platform docs about building on top of Metron, e.g. > > > - writing custom parsers, enrichment, and threat Intel (imho we > > should > > > start to take a more opinionated view of leveraging Stellar as this > > > extension point rather than implementing new parser classes in Java) > > > - using the profiler for constructing outlier analysis use cases > > > - using MAAS for building and deploying models for use in > enrichment > > > - Docs around more specific use cases that solve specific as opposed to > > > more general problems, similar to those we have in the use-cases > folder. > > > > > > I think one of our challenges currently is that our docs could be > better > > > tailored to the "actors" we've talked about in the past. An individual > > SOC > > > analyst will have a very different set of interests than would a > reseller > > > that wants to build on top of our platform to expose new modules and > > > functionality to those SOC analyst. And we can, and do, currently > support > > > both. > > > > > > > > > On Sat, Aug 18, 2018, 9:34 AM Nick Allen <n...@nickallen.org> wrote: > > > > > > > Yes, I imagine just a separate top level directory which would > contain > > > the > > > > docs. > > > > > > > > We would need someone to survey what doc tools are out there and > > provide > > > > some advice. > > > > > > > > Maybe we could look around at other open source projects that have > done > > > > their docs well and emulate them. > > > > > > > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson < > > > kylerichards...@gmail.com> > > > > wrote: > > > > > > > > > +1 to separating developer docs and user docs. How should we > approach > > > > that. > > > > > Have a separate doc book? I haven’t had a ton of time to contribute > > to > > > > code > > > > > lately but I’d be happy to help write some of these. > > > > > > > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen <n...@nickallen.org> > > wrote: > > > > > > > > > > > Personally, I think the state of our docs and web presence is an > > > > > inhibitor > > > > > > to growing the Metron community. Unless we can offer concise, > > > > compelling > > > > > > answers to the basic questions (What can I do with Metron? Who > > does > > > it > > > > > > help? How do I do that?), potential users and contributors are > > unable > > > > to > > > > > > see the value of Metron. > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen <n...@nickallen.org> > > > > wrote: > > > > > > > > > > > > > I'd like to see us focus on improving our docs before a version > > > 1.0. > > > > > > > Right now we just stitch together a bunch of READMEs, which is > a > > > > great > > > > > > > stride from where we started, but is not ideal. > > > > > > > > > > > > > > Our docs should focused on the user and use cases; What can I > do > > > with > > > > > > > Metron? Who does it help? How do I do that? > > > > > > > > > > > > > > The docs should be separate from the code base to allow for an > > > > > > > organization that is focused on the user rather than the > > > > > implementation. > > > > > > > This allows the READMEs to focus on the developer and the > > > > > implementation, > > > > > > > which should make them more digestible too. The docs should be > > > > version > > > > > > > controlled and maintained through PRs, just like the code. We > > > should > > > > > > take > > > > > > > just as much pride in our docs as we do in our code. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball < > > > > > > > si...@simonellistonball.com> wrote: > > > > > > > > > > > > > >> Agreed, should we add TDE by default, and get the ranger > > policies > > > on > > > > > by > > > > > > >> default? That leaves secured in Kafka, which would have to be > > > built > > > > > into > > > > > > >> the consumers and producers to encrypt into the on disk Kafka > > > > topics. > > > > > > Does > > > > > > >> that seem necessary to people? It would have performance > > > > implications > > > > > > for > > > > > > >> sure. > > > > > > >> > > > > > > >> Simon > > > > > > >> > > > > > > >> > On 15 Aug 2018, at 21:26, Otto Fowler < > > ottobackwa...@gmail.com> > > > > > > wrote: > > > > > > >> > > > > > > > >> > Well, I look at it like this. > > > > > > >> > > > > > > > >> > The Secure Vault was part of the original metron pitch, and > > many > > > > may > > > > > > >> have used that as part of their evaluations. > > > > > > >> > “Look, it is going to have a security vault type thing, it > is > > on > > > > the > > > > > > >> roadmap”. > > > > > > >> > > > > > > > >> > Regardless of the implementation, conceptually, security of > > data > > > > at > > > > > > >> rest is important, and is a major outstanding item or the core > > > > metron > > > > > > >> proposition. > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball ( > > > > > > >> si...@simonellistonball.com) wrote: > > > > > > >> >> > > > > > > >> >> That’s going back a way. I always saw that concept as begin > > > about > > > > > the > > > > > > >> formats, e.g. Orc, and meta data around it plus the data > service > > > api > > > > > to > > > > > > get > > > > > > >> at it. I’m all for that too, but think it needs more thought > > than > > > > the > > > > > > >> ticket captures. > > > > > > >> >> > > > > > > >> >> Simon > > > > > > >> >> > > > > > > >> >> On 15 Aug 2018, at 20:53, Otto Fowler < > > ottobackwa...@gmail.com > > > > > > > > > > wrote: > > > > > > >> >> > > > > > > >> >>> https://issues.apache.org/jira/browse/METRON-343 > > > > > > >> >>> > > > > > > >> >>>> On August 15, 2018 at 15:47:24, Simon Elliston Ball ( > > > > > > >> si...@simonellistonball.com) wrote: > > > > > > >> >>>> > > > > > > >> >>>> What would you see as secure? I’ve seen people use TDE > for > > > the > > > > > HDFS > > > > > > >> store, but it’s harder to encrypt storage with solr / es. > > > Something > > > > I > > > > > > was > > > > > > >> thinking of doing to follow up on the Knox Feature was to add > > > Ranger > > > > > > >> integration for securing and auditing configs, and potentially > > > > > > extending to > > > > > > >> the index destinations. Do you think that would cover the > secure > > > > > storage > > > > > > >> concept? > > > > > > >> >>>> > > > > > > >> >>>> Simon > > > > > > >> >>>> > > > > > > >> >>>> > On 15 Aug 2018, at 20:39, Otto Fowler < > > > > ottobackwa...@gmail.com > > > > > > > > > > > > >> wrote: > > > > > > >> >>>> > > > > > > > >> >>>> > Secure storage off the top of my head > > > > > > >> >>>> > > > > > > > >> >>>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com ( > > > > > > zeo...@gmail.com) > > > > > > >> wrote: > > > > > > >> >>>> > > > > > > > >> >>>> > So, as has been discussed in a few > > > > > > >> >>>> > < > > > > > > >> >>>> > > > > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a > > > > > > >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org > %3E> > > > > > > >> >>>> > > > > > > > >> >>>> > other > > > > > > >> >>>> > < > > > > > > >> >>>> > > > > https://lists.apache.org/thread.html/427a20c22207e84331b94e8 > > > > > > >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org > %3E> > > > > > > >> >>>> > > > > > > > >> >>>> > recent dev list threads, I would like to discuss what a > > > > Metron > > > > > > 1.0 > > > > > > >> release > > > > > > >> >>>> > looks like. > > > > > > >> >>>> > > > > > > > >> >>>> > In order to kick off the conversation, I would like to > > > make a > > > > > few > > > > > > >> >>>> > suggestions regarding "what 1.0 means to me," but I'm > > very > > > > > > >> interested to > > > > > > >> >>>> > hear everybody else's opinions. > > > > > > >> >>>> > > > > > > > >> >>>> > In order to go 1.0 I believe we should have: > > > > > > >> >>>> > 1. A clear, supported method of upgrading from one > > version > > > of > > > > > > >> Metron to the > > > > > > >> >>>> > next. We have attempted > > > > > > >> >>>> > < > > https://github.com/apache/metron/blob/master/Upgrading.md > > > > > > > > to > > > > > > >> make this > > > > > > >> >>>> > easier in the past, but it is currently not > > > > > > >> >>>> > < > > > > > > >> >>>> > > > > https://github.com/apache/metron/tree/master/metron-deployme > > > > > > >> nt/packaging/ambari/metron-mpack#limitations> > > > > > > >> >>>> > > > > > > > >> >>>> > supported > > > > > > >> >>>> > < > > > > > > >> >>>> > > > > https://github.com/apache/metron/tree/master/metron-deployme > > > > > > >> nt/packaging/ambari/elasticsearch-mpack#limitations> > > > > > > >> >>>> > > > > > > > >> >>>> > . > > > > > > >> >>>> > 2. Authentication for all of the UIs and APIs should be > > > > secure > > > > > > and > > > > > > >> support > > > > > > >> >>>> > SSO. I believe this is in progress via METRON-1663 > > > > > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1663>. > > > > > > >> >>>> > 3. Each of our personas > > > > > > >> >>>> > < > > > > > > >> >>>> > > > https://cwiki.apache.org/confluence/display/METRON/Metron+ > > > > > > >> User+Personas+And+Benefits> > > > > > > >> >>>> > > > > > > > >> >>>> > should > > > > > > >> >>>> > be well documented, understood, and supported. > > > > > > >> >>>> > - The current state of documentation is, in my opinion, > > > > > > inadequate > > > > > > >> and I > > > > > > >> >>>> > admit I am partially to blame for this. I suggest we > > > define a > > > > > > >> strict > > > > > > >> >>>> > approach for documentation, align to it (such as > perhaps > > > > > > migrating > > > > > > >> all > > > > > > >> >>>> > useful wiki documentation to git), and enforce it. > > > > > > >> >>>> > - I would consider METRON-1699 > > > > > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1699> > as a > > > > > > critical > > > > > > >> item for > > > > > > >> >>>> > a Security Data Scientist, but it is currently not > clear > > to > > > > me > > > > > > >> where the > > > > > > >> >>>> > line exists between some of the other personas, or that > > > each > > > > > > >> persona has > > > > > > >> >>>> > been sufficiently implemented. > > > > > > >> >>>> > 4. A performance tuning guide should be available for > all > > > of > > > > > the > > > > > > >> main > > > > > > >> >>>> > components, whether as an independent document or as a > > part > > > > of > > > > > a > > > > > > >> larger > > > > > > >> >>>> > document. > > > > > > >> >>>> > 5. Simple data ingest. > > > > > > >> >>>> > - Similar to the ongoing conversation for NiFi > > integration > > > > > > >> >>>> > < > > > > > > >> >>>> > > > > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f269 > > > > > > >> 73f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org > %3E>, > > > > > > >> >>>> > > > > > > > >> >>>> > we should be able to say that we have broken down the > > > > barriers > > > > > to > > > > > > >> getting > > > > > > >> >>>> > data into a Metron cluster in easy and efficient ways. > In > > > > > > addition > > > > > > >> to > > > > > > >> >>>> > NiFi, having support for other popular tools such as > > beats > > > > > > >> >>>> > <https://www.elastic.co/products/beats>, fluentd < > > > > > > >> https://www.fluentd.org/>, > > > > > > >> >>>> > > > > > > > >> >>>> > etc. > > > > > > >> >>>> > - Parsers should be pluggable, with independent tests > and > > > the > > > > > > >> ability to > > > > > > >> >>>> > make versioned modifications with roll-backs. > > > > > > >> >>>> > > > > > > > >> >>>> > What else? Are any of these items not necessary for a > > 1.0? > > > > > > >> >>>> > > > > > > > >> >>>> > Jon > > > > > > >> >>>> > -- > > > > > > >> >>>> > > > > > > > >> >>>> > Jon > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > A.Nazemian > -- Jon