Otto, Happy to help around this.
Couple of questions and things to maybe think about… REST seems fine for sending the payload, but the unpacking job should probably be async (triggered by the job, but not responsible for replying, i.e. the initial call gets an ACCEPTED response immediately before the heavy lifting is done. I’m still not 100% sold on the idea of ES templates being included in the bundle, and would prefer that to be a broader “register the schema for the parser” somewhere from which the Metron framework can generate templates, this allows us to account for things like enrichments in templates, but the principle you describe holds if we make it non-ES specific (some people use solr for instance). If we do this as a REST call, we can easily incorporate it to the UI, or maybe add an ambari view later, so I’d say the REST approach is definitely the right first step. One other thing to think about is whether this belongs in the metron-rest package (personally i think it probably does) or whether it is a separate micro-service. Anyway, happy to help in any way I can on this. Simon > On 27 Apr 2017, at 15:09, Otto Fowler <[email protected]> wrote: > > Also, if you want to help that would be great ;) > > > On April 27, 2017 at 09:30:59, Otto Fowler ([email protected]) wrote: > > So, assuming ( I know I know ) that METRON-777 eventually lands, it will > have lain the framework for the extension system for parsers out, including > the capability to create parsers outside the metron tree. > > The next step is METRON-258, side loading of parsers. This will be the > effort to actually install 3rd party parser and other extensions ( stellar > libs for example ) into metron. > > My first inclination is to add new capabilities to the REST services to > accomplish this. We would use the current services, or orchestrate them to > accomplish this. > > The ‘process’ required would be the following: > > I’ll pseudo code it out: > > [rest endpoint] > installExtension(FILE extensionTgz, string extensionType) > > workingDir = unpackExtensionAssembly(tmpDir, extensionTgz) > > deployExtensionBundleToHdfs(workingDir) > > if(parser == extenstionType) > pushConfigurationToZK(workingDir) > pushESTemplate(workingDir) *possibly depending on 777 review > setupLogRotate(workingDIr) * possibly depending on 777 review > saveExtensionTgzSomewhere(extensionTgz) > > > Then, the configuration ui would be extended to front the new api. > > * there is still a question of how we get a parser to the ambari > configuration, such that when starting parsers it starts a parser - unless > that happens and I don’t see it > > > I would like some feedback on this approach. > > * Is rest the right way? Should we do an ambari view instead? > * Is this too much to do in a rest call? Will it timeout etc? > ??? > > > Any ideas would be appreciated.
