Hi Otto, Sounds like we’re really getting somewhere there. I really like that idea. If we have configs as a kind of default, which we can then use to bootstrap any new sensor based on a parser from the bundles, that gives us the best of every world.
Also, I’d agree with you that we should move all the parsers to the bundle method (as you have in the code base) and that the UI should therefore treat the parsers we supply as a project (CEF, bro, snort etc) as no different from a parser an end user writes and bundles. The only area I would say we want to distinguish, as you already have I believe, is the base parsers (Java, Grok) since those would be a start point for inheritance. If we can get the configs as default bootstrap rather than source of truth, I think we will have a great system. Let me know if there is anything around that I can help with on the implementation side. Simon > On 26 Sep 2017, at 11:53, Otto Fowler <ottobackwa...@gmail.com> wrote: > > Hi Simon > > 1. At this moment, in the unreviewed PR ( although this kind of counts ;) ) > it will overwrite. > 2. You just create a new sensor, based on that type and give it a different > name, same as you would now. You would have to start each configuration from > nothing however, because the UI doesn’t copy an existing sensor -or- copy > from the extension default ( where they exist ) > > More on 1. I have mentioned in the PR and other threads about these changes > being the start of a lifecycle for parsers/sensors. The idea being that you > install the parsers, and their default configurations but doesn’t create the > sensor by default as we do now ( both in master and fb ), and then when you > create a parser, it copies the default configuration, using that to pre- > populate the configuration ui. Thus, the install would never touch the > deployed configurations. I did not go so far as to implement that, but I > think that is where we can go. Short of that, I can change the extensions to > not install default configs or not to overwrite if they exist. BTW. The > issue you are speaking about I believe is present in metron in general before > this change. If someone could clarify the upgrade stuff in ambari it would > help. We don’t check if there are already configurations in zookeeper ever > before pushing configurations out. > > This also brings me back to my discuss question from the other day: The > extension list you see in the demo, only has the user installed parser > extensions, not the system ( metron ) ones, like bro, yaf, snort etc. Thus > they are not managed the same way, and they are not registered the same way > with the default configurations in a separate place. > > I think what we want is all parsers installed as extensions, not to install > the parsers as sensors by default as we do now, and the configuration ui to > use the default from the extension registration when creating new. I think > the ‘demo’ system should be a separate ambari install all together. IE> The > install of bro, snort, the dashboard, es templates. > > We are now a long ways off from landing the UI pr. I was always expecting > refactoring around the configuration management and install part, so please, > let me know what you think and I can take a shot at it. Don’t forget about > irc. > > > > > On September 26, 2017 at 05:17:32, Simon Elliston Ball > (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote: > >> Hi Otto, >> >> This is a great demo, nice and clear, many thanks. >> >> Two questions remain for me: >> >> 1. how I would change configuration outside of the bundle? i.e. I install a >> bundle and that gives me enrichment and indexing config, but I then want to >> tune indexing for the characteristics of the actual sensor, or I want to add >> use cases in enrichment which are customised to my org, what happens if I >> update the bundle, does it overwrite the zookeeper config changes? Do I have >> to add all my enrichment config in the bundle? That would be an absolute >> blocker for SOC ops users who do not have access to maven for example and >> would render the management ui null and void. >> 2. At the moment it seems bundle implies sensor. How would I have a parser >> (e.g. CEF parser) which was used against multiple topics (i.e. one parser >> bundle for a data type, multiple sensors based on multiple topics feeding >> that type)? >> >> Really keen to get this in as an extension mechanism, as I think it will be >> a huge help to people developing custom parsers, but I really want to make >> sure it will not clobber the use cases people add with enrichment config in >> particular. >> >> Simon >> >> > On 25 Sep 2017, at 16:27, Otto Fowler <ottobackwa...@gmail.com >> > <mailto:ottobackwa...@gmail.com>> wrote: >> > >> > https://youtu.be/-ISycoP3TVA <https://youtu.be/-ISycoP3TVA> >> > >> > The video is short and simple. Hopefully it is what you are looking for. >> > >> > >> > On September 21, 2017 at 16:54:13, zeo...@gmail.com (zeo...@gmail.com >> > <mailto:zeo...@gmail.com>) >> > wrote: >> > >> > I won't be able to make it and would really like to make sure there's a >> > recording for this one, if possible. I'm unavailable until Thursday of >> > next week, but not necessarily suggesting this gets moved. >> > >> > Jon >> > >> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ottobackwa...@gmail.com >> > <mailto:ottobackwa...@gmail.com>> wrote: >> > >> >> I can’t make that time, can we make it later in the day? >> >> >> >> >> >> On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org >> >> <mailto:jsir...@apache.org>) >> >> wrote: >> >> >> >> https://hortonworks.webex.com/meet/jsirota >> >> <https://hortonworks.webex.com/meet/jsirota> >> >> >> > -- >> > >> > Jon
