As previously discussed here, I have been working on side loading of parsers. The goals of this work are: * Make it possible of developers to create, maintain and deploy parsers outside of the Metron code tree and not have to fork * Create maven archetype support for developers of parsers * Introduce a parser ‘lifecycle’ to support multiple instances and configurations, states of being installed, under configuration, and deployed etc.
I would like to have some discussion based on where I am after rebasing onto METRON-671 which revamps deployment to be totally ambari based. Packaging and Deployment I have not change the packaging methodology that was already there, ie. All the parsers are still shaded uber jars, and all are package into tar.gz to include /lib /config /pattens They are all explicitly called out in the copy resources portion of the rpm-docker pom, and explicitly configured in the metron.spec for rpm generation. When deployed, they are deployed to a new directory - telemetry under the metron home ( /usr/metron/0.3.1/telemetry ). Each parser has it’s own directory, which gives it an isolated environment: telemetry/asa /config /lib /patterns I could see adding a version here as well. Also, this directory structure could change, if not by review then by other follow on deployment options due to ‘thinning' All the scripts, ambari services have been changed to account for this, and the start parser topology script is changed to find the right parser jar to use for the -s option ( as opposed to only loading metron-parsers jar which is the root of the issue ). The packaging issues here: * 10 or so new Uber jars increases the build size x2 * Travis needs to be changed from a container build to a vm build to get a bigger space to work in The rpm issues here: * explicitly listing like items that *could* be iterated from a list is a code smell to me. With ansible I was able to define a list and use with_items to get a nice clean, maintainable flow. With rpm and maven resources we have to have explicit entries. My rpm and maven foo was not good enough to sort this out, so I just bit the bullet and did it. I think we should explore copying using some kind of script or iteration in maven, and possibly generating the metron spec from a template ( this too is easier in ansible ). Options here: 1) Accept this as mvp for the PR with improvements to packaging and deployment as a follow on 2) Delay the mvp and go for a possibly smaller - optimized deployment Going for a smaller deployment, that is slimming the jars is something I’ll talk about in another email :)
