Hi, We have been discussing ways to simplify how our containers get built and deployed and something we hope to do is to stop using Ansible and custom templates for configuration files because they increase the container image sizes (lots of dependencies), break whenever the vanilla config files change and make troubleshooting more difficult.
We created some new containers to demonstrate these wins and used the default config files with no changes to simulate an ideal situation where we wouldn't be making changes to config files and would instead be using environment variables, command line options, etc, to affect how the components runs. These are some preliminary numbers: - Container image size: 542-972MB vs. 55-308MB - Build times: 120-240sec vs. 30-180sec - Lines of automation code required: 726 vs. 11 This work with the gpii-launcher seems critical for our ability to achieve those goals. Thank you, Giovanni On 10/24/2016 05:37 PM, Tony Atkins wrote: > Hi, All: > > I finally had a few minutes to run the tests and confirm that the new > kettle-ist approach works as expected. I have submitted a PR and created a > retroactive ticket as well: > > https://github.com/GPII/gpii-launcher/pull/1 > https://issues.gpii.net/browse/GPII-2103 > > Cheers, > > > Tony > > > On Mon, Oct 24, 2016 at 2:34 PM, Tony Atkins <[email protected] > <mailto:[email protected]>> wrote: > > Hi, All: > > I was able to migrate to using kettle.config.loadConfig without any > difficulty. To confirm my theory about the config code being portable, I > included it using /fluid.require("%kettle/lib/KettleConfigLoader.js")/. It > works fine without any other parts of the kettle codebase. > > Once I finish tidying up, I will reply to this thread with a link to the > PR for anyone who wants to follow the in-depth review of the launcher. > > Cheers, > > > Tony > > On Mon, Oct 24, 2016 at 11:55 AM, Tony Atkins <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Antranig: > > Thanks for closing the loop. In my brief review, > kettle.config.loadConfig > <https://github.com/fluid-project/kettle/blob/master/lib/KettleConfigLoader.js#L157> > looks very helpful, and shouldn't be difficult to integrate. I should be > able to at least highlight potential sticking points before the next > architecture meeting. > > I can add kettle as a dependency for now, but as the uses for this > are not unique to kettle, and since the config loading seems pretty > self-contained, we should talk about moving that code and its tests over to > gpii-launcher longer term. > > Cheers, > > > Tony > > On Sun, Oct 23, 2016 at 3:09 AM, Antranig Basman > <[email protected] <mailto:[email protected]>> wrote: > > Thanks - this is a very helpful package. However, we should make > energetic efforts not to end up with duplicate functionality across our > projects. This is a feature that we have often discussed both in the GPII and > Fluid - here is the paper trail: > > > http://lists.gpii.net/pipermail/architecture/2016-August/004248.html > <http://lists.gpii.net/pipermail/architecture/2016-August/004248.html> > > http://lists.gpii.net/pipermail/architecture/2016-August/004251.html > <http://lists.gpii.net/pipermail/architecture/2016-August/004251.html> > > I'm sorry that this didn't seem to get written up into a JIRA so > far, which would be a great first step. > > Do you think you could refactor your system so that it can be > used for resolution with the existing Kettle config system that we use in the > core architecture? This would be super-helpful, since this is a feature that > has been requested by Giovanni and others for some time - Cheers! > > Use of the Kettle config loader would also spare you some work > related to taking control of file resolution and component instantiation. > > Antranig. > > On 21/10/2016 16:19, Tony Atkins wrote: > > Hi, All: > > As mentioned in the last meeting, I have been working on a > "launcher" to > make it easier to launch components with customized options. > Here is > the micromodule I created: > > https://github.com/the-t-in-rtf/gpii-launcher > <https://github.com/the-t-in-rtf/gpii-launcher> > > It's based on "yargs <http://yargs.js.org/docs/>", which > gives us lots > of really nice stuff out of the box: > > 1. Declarative configuration options. > 2. Support for both environment variables and command line > parameters. > 3. Support for using your own functions to check the > validity of > supplied params. > 4. Inherent support for deep "dot notation" references, i.e. > --deep.param 45 > 5. Ability to declaratively specify the "type" for a given > parameter. > > In addition, the wrapper I wrote adds an implicit > /optionsFile/ > parameter that can be use to load multiple options from a > JSON file. > Yargs has a similar built-in mechanism, but in talking with > others, > their order of inheritance was sub-optimal, so we needed our > own > mechanism. The /optionsFile/ option supports relative, > absolute, and > package-relative (%package-name/path/to/file.json) paths, the > last of > which was also not possible with the built-in mechanism. > > The launcher has the ability to load both shallow and deep > references > from (in order of preference): > > 1. Command line parameters. > 2. Environment variables. > 3. Options files. > > This will hopefully give us a lot of flexibility in running > production > services, as well as in testing configuration changes on a > one-off basis. > > The module itself is solely responsible for merging the above > sources of > options, and then emitting an event with the merged options. > Implementers are expected to add a listener to perform actual > work, for > example, (re)launching a new component with the supplied > options. There > is an example included with the package that uses > fluid.construct to > construct a dynamic component when it receives new options. > I have also > been experimenting with more streamlined invoker-based > approaches based > on feedback from Antranig. My hope is that we'll have a > simple pattern > for using this more widely by the time the initial PR is > reviewed. > > Anyway, do please comment if you have ideas, concerns, et > cetera. I'm > also happy to demo the package if there's enough interest. > > Cheers, > > > Tony > > > _______________________________________________ > Architecture mailing list > [email protected] > <mailto:[email protected]> > http://lists.gpii.net/mailman/listinfo/architecture > <http://lists.gpii.net/mailman/listinfo/architecture> > > > _______________________________________________ > Architecture mailing list > [email protected] <mailto:[email protected]> > http://lists.gpii.net/mailman/listinfo/architecture > <http://lists.gpii.net/mailman/listinfo/architecture> > > > > _______________________________________________ Architecture mailing list [email protected] http://lists.gpii.net/mailman/listinfo/architecture
