Interesting points, Aled. I thought about this, and I'm not sure about the handlers part, but would be willing to be convinced further. In the meantime, as an experiment, I decided to make the JcloudsLocationCustomizer an EntityInitializer as well. This means we can write a customizer that installs packages, and then just reference it in the `brooklyn.initializers` section. See the pull request here:
- https://github.com/apache/brooklyn-server/pull/168 This could easily be updated to be referenced in a different section of the YAML later, so it is useful for comment just now, I think. Andrew. On Tue, 24 May 2016 at 10:37 Aled Sage <aled.s...@gmail.com> wrote: > Hi all, > > TL;DR: how about the following, where there is a "handler" registered to > process the brooklyn.packages section? > > - type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess > brooklyn.packages: > yum: curl > apt: curl-whatever=2.3.4 > > Or is this over engineering?! > > --- > To me, the "brooklyn.initializers" and the > "org.apache.brooklyn.initializer.stock.PackageInstaller" are > implementation details that a YAML blueprint author would never want to > type. > > I also agree that we don't want "packages" as a config key: as Duncan > Grant described, that moves VanillaSoftwareProcess towards being a "god > object" with more and more config on that class. > > --- > I think there is an alternative where we add first-class support for > more things in YAML. For example, we should get rid of the need for > "brooklyn.initializers" when defining sensors, effectors and feeds - we > should be able to have a section "brooklyn.sensors:". > > The same could apply for packages. > > However, "packages" are not a first-class concept in Brooklyn (unlike > sensors, effectors, etc). > > --- > To take a step back... > > The use of "brooklyn.initializers" is to declare mixins [1]. The entries > under brooklyn.initializers are the types + config of each mixin. This > gives a very direct translation from YAML to the underlying Java api. > > Instead, one could register a handler for a given YAML key (of the form > "brooklyn.xyz"). This would apply only to top-level entries against an > entity, policy, enricher, etc (e.g. not when embedded inside a > brooklyn.config section). Under-the-covers, the handler would return > something of type EntityInitializer (or equivalent for > policy/enricher/location). Or perhaps it could even act directly on the > EntitySpec that is being created. > > --- > When implementing this, we'd separate it into a number of parts: > > * Implement Andrew's > org.apache.brooklyn.initializer.stock.PackageInstaller > * Add support for registering yaml "handlers" > * Register a yaml handler for "brooklyn.packages", which instantiates > Andrew's PackageInstaller. > > --- > I also worry that for installing packages, things are often more > complicated than these simple use-cases. For example, when installing > Docker on CentOS7, you may first need to populate the file > /etc/yum.repos.d/docker.repo [2]. Or you may need more complicated > install commands, such as first executing yum upgrade ca-certificates > --disablerepo=epel [3]. > > The ordering of executing `brooklyn.packages` (compared to other > commands on VanillaSoftwareProcess) would not be obvious to a blueprint > author. > > For some use-cases, a bash script becomes easier to write and test, > rather than trying to support all these variations. > > However, making simple cases simpler is a good thing. > > --- > Thoughts? > > Aled > > [1] https://en.wikipedia.org/wiki/Mixin > [2] https://docs.docker.com/engine/installation/linux/centos/ > [3] http://serverfault.com/a/654660/323894 > > > On 24/05/2016 09:57, Thomas Bouron wrote: > > I quite like Andrew's approach using an initializer, feels elegant and > > relatively easy to implement. > > > > Regarding the syntax, getting a rid of the `brooklyn.initializer` and > > `type` is good for a discoverability point of view but that means going > > through a config key which feels wrong in that case. I don't have an > > argument for it, it's more a gut feeling. > > > > On Tue, 24 May 2016 at 09:37 John McCabe <j...@johnmccabe.net> wrote: > > > >> Is there any way we could get rid of the initializers/type parts, rather > >> than: > >> > >> - type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess > >> brooklyn.initializers: > >> - type: org.apache.brooklyn.initializer.stock.PackageInstaller > >> packages: > >> yum: curl > >> apt: curl-whatever=2.3.4 > >> > >> Something like: > >> > >> - type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess > >> packages: > >> yum: curl > >> apt: curl-whatever=2.3.4 > >> > >> > >> On Tue, 24 May 2016 at 09:34 Duncan Grant < > duncan.gr...@cloudsoftcorp.com> > >> wrote: > >> > >>> Andrew, > >>> > >>> good point. That's much neater. Should be pretty straightforward to > >>> implement as well. > >>> > >>> Duncan > >>> > >>> On Mon, 23 May 2016 at 21:16 Andrew Kennedy < > >>> andrew.kenn...@cloudsoftcorp.com> wrote: > >>> > >>>> Duncan, > >>>> > >>>> I like the idea here, of having some extra object that does the > package > >>>> installation, but what about a customizer or initializer instead? You > >>> could > >>>> have something like this: > >>>> > >>>> - type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess > >>>> brooklyn.initializers: > >>>> - type: org.apache.brooklyn.initializer.stock.PackageInstaller > >>>> packages: > >>>> yum: curl > >>>> apt: curl-whatever=2.3.4 > >>>> > >>>> I initially thought of a JcloudsLocationCustomizer but that fails when > >> we > >>>> use BYON locations, so I am thinking of something more like a generic > >>>> entity customizer that runs after the entity has been created, and the > >>>> location is available and SSHable. There are lots of other things > these > >>>> initializers could do, like setting up users, configuring networking, > >>>> mounting volumes and so on. I don't think its a big stretch to add > this > >>>> functionality here. > >>>> > >>>> Andrew. > >>>> > >>>> On Fri, 6 May 2016 at 14:40 Duncan Grant < > >> duncan.gr...@cloudsoftcorp.com > >>>> wrote: > >>>> > >>>>> I agree that this is a problem that needs solved but I have > >> misgivings > >>>>> about each of the proposed solutions. > >>>>> > >>>>> Firstly I think that "discovering" how to write brooklyn yaml can be > >>>>> difficult. It can be difficult to find things in the documentation > >> and > >>>> if > >>>>> you don't know to look for them in the first place then your only > >> hope > >>> is > >>>>> coming across an example. > >>>>> > >>>>> So if we pre-install a set of bash scripts before running the install > >>>>> scripts I don't see any obvious way for someone using brooklyn to > >> find > >>>> out > >>>>> about those scripts, and when the scripts change I imagine people > >> won't > >>>>> notice the new functionality either. > >>>>> Another downside to using scripts is that it will become more > >> difficult > >>>> to > >>>>> debug script failures. At the moment I can copy the contents of > >> stdin > >>>> and > >>>>> see the error associated. If I had to step into scripts in other > >> files > >>>> it > >>>>> could become much harder to find a failure. In my opinion this ease > >> of > >>>>> reproducibility and seeing the code I am running is one of the > >>> advantages > >>>>> of using bash over just using chef/puppet/salt/etc. > >>>>> If we could find a widely used, well tested bash library then some of > >>>> these > >>>>> issues might be mitigated but I've yet to find one. > >>>>> > >>>>> There is a similar issue with discovering the behaviour of the DSL > >>>> option. > >>>>> I think that the DSL only works because we keep it to a minimum and > >>>> really > >>>>> it only does one task - which is run-time configuring relationships > >>>> between > >>>>> entities. If we extend it then we have replaced java with dsl and I > >>>> don't > >>>>> think that benefits anyone. One advantage of the DSL option is that > >> it > >>>>> will generate bash so it shouldn't make debugging any harder. > >>>>> > >>>>> The advantage of adding config options to the VanillaSoftwareProcess > >> is > >>>>> that it adds to extra ways of discovering the behaviour - through the > >>>>> composer's autocomplete and in javadoc. I assume that this would > >>>> generate > >>>>> extra tasks so we could again debug the bash used and it might make > >> it > >>>>> easier as we might be able to see exactly which package failed to > >>>> install. > >>>>> However the big downside to this is that we are starting to make a > >> sort > >>>> of > >>>>> god object which contains all brooklyn functionality in one object - > >>> i.e. > >>>>> VanilllaSoftwareProcess and I'm not sure that's where we want to go. > >>>>> > >>>>> After all that I don't really have a good answer. > >>>>> > >>>>> My not too good answer would be to make more use of the brooklyn > >> child > >>>>> relationship so that a) at least the yaml could be composed from > >> small > >>>>> pieces of code b) we could write a InstallPackage entity to simplify > >>> the > >>>>> yaml. > >>>>> > >>>>> This might look something like: > >>>>> > >>>>> - type: brooklyn.entity.basic.VanillaSoftwareProcess > >>>>> install.command: curl somefile > somfile.txt > >>>>> brooklyn.config: > >>>>> children.startable.mode: foreground_early > >>>>> brooklyn.children: > >>>>> - type: brooklyn.entity.basic.VanillaSoftwareProcess > >>>>> install.command: | > >>>>> which curl || \ > >>>>> { sudo apt-get update && sudo apt-get install curl ; } > >> || \ > >>>>> { sudo yum update && sudo yum install curl ; } || \ > >>>>> { echo WARNING: cannot install curl && exit 1 ; } > >>>>> or > >>>>> - type: brooklyn.entity.basic.InstallPackage > >>>>> package: curl > >>>>> or > >>>>> - type: brooklyn.entity.basic.InstallPackage > >>>>> packages: > >>>>> yum: curl > >>>>> apt: someOtherCurl > >>>>> > >>>>> This way everything is discoverable through documentation, javadoc, > >> and > >>>>> autocomplete. Common requirements like curl are re-usable. Generated > >>>> bash > >>>>> is explicit and if curl fails then it will go on fire. > >>>>> On the other hand it requires longer yaml, isn't as readable, and due > >>> to > >>>>> the use of child parent relationships has a bit of a learning hurdle > >>>>> > >>>>> Regards > >>>>> > >>>>> Duncan > >>>>> > >>>>> > >>>>> On Wed, 4 May 2016 at 17:35 Thomas Bouron < > >>>> thomas.bou...@cloudsoftcorp.com > >>>>> wrote: > >>>>> > >>>>>> Hi Guglielmo. > >>>>>> > >>>>>> I'm not sure to follow your idea of "stacks", could you elaborate > >> on > >>>> that > >>>>>> please? > >>>>>> > >>>>>> Best. > >>>>>> > >>>>>> On Tue, 3 May 2016, 13:38 Guglielmo Nigri, < > >>>>>> guglielmo.ni...@cloudsoftcorp.com> wrote: > >>>>>> > >>>>>>> I propose a declarative approach. For example we could add a > >>>> configKey > >>>>> to > >>>>>>> VanillaSoftwareProcess called requiredPackages. > >>>>>>> > >>>>>>> This way, one could just specify the prerequisite packages, sort > >> of > >>>>>>> dependencies for the software process -- this would also open up > >>>>>>> interesting possibilities such as “stacks” as bundled > >> dependencies. > >>>>>>> A declarative approach would also work for Windows (or non-bash > >>>>>>> environments). > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Guglielmo > >>>>>>> > >>>>>>> > >>>>>>> On 3 May 2016 at 14:28, Andrea Turli < > >>> andrea.tu...@cloudsoftcorp.com > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Great discussion guys! > >>>>>>>> > >>>>>>>> It seems that it is a shared need, and I wanted to discuss with > >>> you > >>>>> as > >>>>>> I > >>>>>>>> was not sure about my approach. > >>>>>>>> > >>>>>>>> Andrew, > >>>>>>>> > >>>>>>>> I like your proposal, thanks! > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Andrea > >>>>>>>> > >>>>>>>> > >>>>>>>> On 3 May 2016 at 13:05, Aled Sage <aled.s...@gmail.com> wrote: > >>>>>>>> > >>>>>>>>> I lean towards Andrew's approach, rather than a special > >>>>>>>>> $brooklyn.installPackage. Note that different distros use > >>>> different > >>>>>>>> package > >>>>>>>>> names sometimes, so the parameters to such a function can get > >>>>>> annoying. > >>>>>>>>> I've been hesitant about us going down the road of > >>>>>>> `brooklyn-commands.sh` > >>>>>>>>> for achieving portable blueprints. It feels like we are > >>>> increasing > >>>>>> the > >>>>>>>>> overlap with things like Chef, Salt, Ansible and Puppet > >> (which > >>> we > >>>>> can > >>>>>>>> build > >>>>>>>>> on top of with Brooklyn). > >>>>>>>>> > >>>>>>>>> However, if we keep brooklyn-commands.sh small and focused, > >>> then > >>>>>> maybe > >>>>>>>>> it's a good idea. > >>>>>>>>> > >>>>>>>>> Aled > >>>>>>>>> > >>>>>>>>> p.s. I want to decrease the use of > >>>>>> `BashCommands.installExecutable()`. > >>>>>>>> The > >>>>>>>>> bash command it generates, with all the alternatives and > >>>>> parentheses, > >>>>>>>> does > >>>>>>>>> not look like nice Bash. A brooklyn-commands.sh could do it > >>> much > >>>>>>> cleaner, > >>>>>>>>> with a properly structured multi-line if...elif... (or > >>> whatever). > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 03/05/2016 11:43, Andrew Kennedy wrote: > >>>>>>>>> > >>>>>>>>>> How about an alternative approach - for all SoftwareProcess > >>>>>> entities, > >>>>>>>>>> Brooklyn will copy over a script file called > >>>>> `brooklyn-commands.sh` > >>>>>>> and > >>>>>>>>>> the > >>>>>>>>>> entity commands will soutce this script before running the > >>> rest > >>>> of > >>>>>>> their > >>>>>>>>>> configured commands. The script will contain OS agnostic > >>>> functions > >>>>>>>> written > >>>>>>>>>> in Bash that do things like install packages, download Curl, > >>>> get a > >>>>>>> file > >>>>>>>>>> from a URL etc. Then, the install config might look like > >> this: > >>>>>>>>>> ``` > >>>>>>>>>> install.commands: | > >>>>>>>>>> brooklyn-installPackages apt:openjdk-1.8.0 > >>>>>>>> yum:java-1.8.0-openjdk-devel > >>>>>>>>>> java-1.8.0 > >>>>>>>>>> brooklyn-installPackages curl > >>>>>>>>>> brooklyn-runAsRoot cp /tmp/whatever /etc/hosts > >>>>>>>>>> ``` > >>>>>>>>>> > >>>>>>>>>> Andrew. > >>>>>>>>>> > >>>>>>>>>> On Tue, 3 May 2016 at 11:24 Andrea Turli < > >>>>>>>> andrea.tu...@cloudsoftcorp.com> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> hi, > >>>>>>>>>>> I’ve been thinking about an utility to simplify the YAML > >>>>> blueprint > >>>>>>>>>>> creation: from my experience when using > >>> VanillaSoftwareProcess > >>>> is > >>>>>>>>>>> annoying > >>>>>>>>>>> to write a portable script just to install a package (say, > >>>> java) > >>>>>>> valid > >>>>>>>>>>> for > >>>>>>>>>>> apt, yum, etc so I usually write it (multiple times) just > >> for > >>>> an > >>>>>> OS. > >>>>>>>>>>> To increase the portability of the YAML blueprint I’d like > >> to > >>>>>> suggest > >>>>>>>> we > >>>>>>>>>>> extend the brooklyn DSL with something like: > >>>>>>>>>>> ``` > >>>>>>>>>>> $brooklyn.installPackage(“curl”) > >>>>>>>>>>> $brooklyn:installPackage({"apt", "openjdk-1.8.0", "yum", > >>>>>>>>>>> "java-1.8.0-openjdk-devel"}, "java-1.8.0") > >>>>>>>>>>> ``` > >>>>>>>>>>> instead of things like > >>>>>>>>>>> ``` > >>>>>>>>>>> which curl || \ > >>>>>>>>>>> { sudo apt-get update && sudo apt-get install curl > >>> ; } > >>>>> || > >>>>>> \ > >>>>>>>>>>> { sudo yum update && sudo yum install curl ; } || > >> \ > >>>>>>>>>>> { echo WARNING: cannot install curl && exit 1 ; } > >>>>>>>>>>> ``` > >>>>>>>>>>> I’m not entirely sure this feature fits well on the DSL. > >>>>>>>>>>> > >>>>>>>>>>> Alternatively, we could add a configKey to > >>>> VanillaSoftwareProcess > >>>>>>>> called > >>>>>>>>>>> requiredPackages for a more declarative approach > >>> (@googlielmo's > >>>>>> idea) > >>>>>>>>>>> Wdyt? > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>> -- > >>>>>> Thomas Bouron • Software Engineer @ Cloudsoft Corporation • > >>>>>> http://www.cloudsoftcorp.com/ > >>>>>> Github: https://github.com/tbouron > >>>>>> Twitter: https://twitter.com/eltibouron > >>>>>> > >>>> -- > >>>> > >>>> Andrew Kennedy ; Founder clocker.io project ; @grkvlt ; Cloudsoft > >>>> > > -- Andrew Kennedy ; Founder clocker.io project ; @grkvlt ; Cloudsoft
