Per our prior conversations, I prefer option 2 - treating third party and built-in the same way. I would love to see signing of extensions in the future as a potential follow-on so we could verify the Metron built-ins (and even third parties).
Jon On Wed, Sep 20, 2017 at 10:22 AM Otto Fowler <[email protected]> wrote: > Simon, I’m sorry, I may not have answered your question. > I use parser and sensors as the same thing, but from what you say I think I > mean parser. > > > On September 20, 2017 at 10:08:25, Otto Fowler ([email protected]) > wrote: > > So, > The distinction between ‘sensor’ and ‘parser’ is not clear to me either, if > it is defined somewhere and I have missed it, please point me in the right > direction. > While I don’t want to fork the discussion, the question is where you find > it so to speak, so about bundles and configurations. > > With the extension system you have two options for ‘config’ only > parser/sensor installs. > > 1. The previous mechanism of creating the configurations and manually > pushing them zookeeper and possibly patterns still works, although they > will not be managed as extensions. This I believe is a feature gap that > already exists and I did not address it as part of this effort. > 2. Create a new parser from the archetype, but remove all the src/main code > and make it configuration only. This I think should be the recommended way > to create configuration only parsers, since such parsers will still have > the unit and integration tests. Flows where you start with the archetype > and refine through tests or manually deploy, refine and then build from the > archetype can be imagined. > > > The main question for this thread however, is should metron parsers and > 3rd party parsers be treated the same -> they are all extensions, > manageable through the extensions ui. > > I can demo for you whenever you are free if you don’t want to wait for the > community demo. > > > On September 20, 2017 at 09:52:56, Simon Elliston Ball ( > [email protected]) wrote: > > Otto, > > Can you just clarify what you mean by parsers in this instance. To my mind > parsers in metron are be classes, and do not have any configuration > settings. Instances of parsers are referred to in the ui as sensors, and > are essentially concrete instances of parsers and as such do have config. > > The parser vs sensor distinction feels like a valuable one to make here. > I'd really like a clearer understanding of the role of bundles in config, > and how we maintain the ability to control config without the need for > files in the bundle. > > Maybe some samples / a demo would really clear this up, but to be honest > right now I'm a little confused about how this would work for an average > SOC team who do not have maven available. > > Thanks, > Simon > > Sent from my iPhone > > > On 20 Sep 2017, at 23:43, Otto Fowler <[email protected]> wrote: > > > > Note: Grok, CSV and JSONMap would be ‘always there’, as they are still > > part of the system and not installed individually. > > > > > > > > On September 20, 2017 at 09:39:38, Otto Fowler ([email protected]) > > wrote: > > > > The question has come up about the metron parsers installation vs. parser > > extension installation differences, and I’d like to get some comments. > > > > Right now ( let’s pretend the UI PR get’s merged to the feature branch > for > > a minute ) in the original take on this the metron parsers ( bro, yaf, > > snort etc ) get installed > > into the system basically as before with regards to zookeeper ( although > > they ALL get installed since they all have demo compatible default > configs > > now). The parser extensions, installed after the fact through ui however > > have a new parser extension configuration > > that is registered into a new zookeeper area, and are listed and managed > in > > the new UI. They can be installed or uninstalled basically. > > > > The question is if the parsers installed by metron should *also* be > > registered in the same way, such that the parser extension ui lists all > of > > the parsers. > > > > This would allow removal and installation of metron parsers installed by > > the system. Also, following on we can customize the install to not > install > > everything. It may also be more simple concept wise. > > > > That is the basic thing. So the question is if we want to go in this > > direction. > > > > The not so basic thing is to still deploy the extensions into the system > as > > packages, but not install them, such that you can add an extension from a > > file *and also* from a ‘repository’ of > > extensions. Down the line we can support local and remote repositories > etc. > > > > So the options are: > > > > 1. As it is now on the feature branch ( still pretending ;)), the system > > installed parsers are not the same as the extension installed parsers. > > While the project can uninstall and replace these parsers, the user/ui > > cannot. > > 2. All parsers are installed as extensions and can be removed, but are > all > > initially installed > > 3. All parsers are extensions, but not installed during the original > > deployment, rather they are put into a repository that the ui lets you > > browse to select, in addition to allowing > > install from file ( like intellij and other plugin systems do ). **This > > would have implications for the demo system, since we would want to still > > install the bro, snort and maybe yaf parsers. > > > > > > In considering these questions, we need to keep in mind where we want to > > go, and how much of this is required for first release of the extension > > system. There is a lot of ‘while we are already doing xxx, we might as > > well do yyyy since it will be harder later’ in this. > > > > Thanks for your time and your timely responses. > > > > ottO > -- Jon
