Great! Since it looks like I’ll be restating and refactoring things on master in chunks this is still a ways off. When I get to the ui/rest area I will refactor to this. When that time comes, I would like for us to talk about having the concept of a local repository.
On top of what we just talked about…. We would install extensions to the repository in hdfs. When you went to install an extension, you could upload or pick from the repository. We would then get into the flow we were just talking about. So the user would only install the parser extensions they need. Also, we could then introduce remote repositories etc. Another question is signing extensions. We will have to think about that before landing. On September 26, 2017 at 07:06:23, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Hi Otto, Sounds like we’re really getting somewhere there. I really like that idea. If we have configs as a kind of default, which we can then use to bootstrap any new sensor based on a parser from the bundles, that gives us the best of every world. Also, I’d agree with you that we should move all the parsers to the bundle method (as you have in the code base) and that the UI should therefore treat the parsers we supply as a project (CEF, bro, snort etc) as no different from a parser an end user writes and bundles. The only area I would say we want to distinguish, as you already have I believe, is the base parsers (Java, Grok) since those would be a start point for inheritance. If we can get the configs as default bootstrap rather than source of truth, I think we will have a great system. Let me know if there is anything around that I can help with on the implementation side. Simon On 26 Sep 2017, at 11:53, Otto Fowler <ottobackwa...@gmail.com> wrote: Hi Simon 1. At this moment, in the unreviewed PR ( although this kind of counts ;) ) it will overwrite. 2. You just create a new sensor, based on that type and give it a different name, same as you would now. You would have to start each configuration from nothing however, because the UI doesn’t copy an existing sensor -or- copy from the extension default ( where they exist ) More on 1. I have mentioned in the PR and other threads about these changes being the start of a lifecycle for parsers/sensors. The idea being that you install the parsers, and their default configurations but doesn’t create the sensor by default as we do now ( both in master and fb ), and then when you create a parser, it copies the default configuration, using that to pre- populate the configuration ui. Thus, the install would never touch the deployed configurations. I did not go so far as to implement that, but I think that is where we can go. Short of that, I can change the extensions to not install default configs or not to overwrite if they exist. BTW. The issue you are speaking about I believe is present in metron in general before this change. If someone could clarify the upgrade stuff in ambari it would help. We don’t check if there are already configurations in zookeeper ever before pushing configurations out. This also brings me back to my discuss question from the other day: The extension list you see in the demo, only has the user installed parser extensions, not the system ( metron ) ones, like bro, yaf, snort etc. Thus they are not managed the same way, and they are not registered the same way with the default configurations in a separate place. I think what we want is all parsers installed as extensions, not to install the parsers as sensors by default as we do now, and the configuration ui to use the default from the extension registration when creating new. I think the ‘demo’ system should be a separate ambari install all together. IE> The install of bro, snort, the dashboard, es templates. We are now a long ways off from landing the UI pr. I was always expecting refactoring around the configuration management and install part, so please, let me know what you think and I can take a shot at it. Don’t forget about irc. On September 26, 2017 at 05:17:32, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Hi Otto, This is a great demo, nice and clear, many thanks. Two questions remain for me: 1. how I would change configuration outside of the bundle? i.e. I install a bundle and that gives me enrichment and indexing config, but I then want to tune indexing for the characteristics of the actual sensor, or I want to add use cases in enrichment which are customised to my org, what happens if I update the bundle, does it overwrite the zookeeper config changes? Do I have to add all my enrichment config in the bundle? That would be an absolute blocker for SOC ops users who do not have access to maven for example and would render the management ui null and void. 2. At the moment it seems bundle implies sensor. How would I have a parser (e.g. CEF parser) which was used against multiple topics (i.e. one parser bundle for a data type, multiple sensors based on multiple topics feeding that type)? Really keen to get this in as an extension mechanism, as I think it will be a huge help to people developing custom parsers, but I really want to make sure it will not clobber the use cases people add with enrichment config in particular. Simon > On 25 Sep 2017, at 16:27, Otto Fowler <ottobackwa...@gmail.com> wrote: > > https://youtu.be/-ISycoP3TVA > > The video is short and simple. Hopefully it is what you are looking for. > > > On September 21, 2017 at 16:54:13, zeo...@gmail.com (zeo...@gmail.com) > wrote: > > I won't be able to make it and would really like to make sure there's a > recording for this one, if possible. I'm unavailable until Thursday of > next week, but not necessarily suggesting this gets moved. > > Jon > > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ottobackwa...@gmail.com> wrote: > >> I can’t make that time, can we make it later in the day? >> >> >> On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org) >> wrote: >> >> https://hortonworks.webex.com/meet/jsirota >> > -- > > Jon
