Hi all,
TL;DR: how about the following, where there is a "handler" registered to
process the brooklyn.packages section?
- type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess
brooklyn.packages:
yum: curl
apt: curl-whatever=2.3.4
Or is this over engineering?!
---
To me, the "brooklyn.initializers" and the
"org.apache.brooklyn.initializer.stock.PackageInstaller" are
implementation details that a YAML blueprint author would never want to
type.
I also agree that we don't want "packages" as a config key: as Duncan
Grant described, that moves VanillaSoftwareProcess towards being a "god
object" with more and more config on that class.
---
I think there is an alternative where we add first-class support for
more things in YAML. For example, we should get rid of the need for
"brooklyn.initializers" when defining sensors, effectors and feeds - we
should be able to have a section "brooklyn.sensors:".
The same could apply for packages.
However, "packages" are not a first-class concept in Brooklyn (unlike
sensors, effectors, etc).
---
To take a step back...
The use of "brooklyn.initializers" is to declare mixins [1]. The entries
under brooklyn.initializers are the types + config of each mixin. This
gives a very direct translation from YAML to the underlying Java api.
Instead, one could register a handler for a given YAML key (of the form
"brooklyn.xyz"). This would apply only to top-level entries against an
entity, policy, enricher, etc (e.g. not when embedded inside a
brooklyn.config section). Under-the-covers, the handler would return
something of type EntityInitializer (or equivalent for
policy/enricher/location). Or perhaps it could even act directly on the
EntitySpec that is being created.
---
When implementing this, we'd separate it into a number of parts:
* Implement Andrew's
org.apache.brooklyn.initializer.stock.PackageInstaller
* Add support for registering yaml "handlers"
* Register a yaml handler for "brooklyn.packages", which instantiates
Andrew's PackageInstaller.
---
I also worry that for installing packages, things are often more
complicated than these simple use-cases. For example, when installing
Docker on CentOS7, you may first need to populate the file
/etc/yum.repos.d/docker.repo [2]. Or you may need more complicated
install commands, such as first executing yum upgrade ca-certificates
--disablerepo=epel [3].
The ordering of executing `brooklyn.packages` (compared to other
commands on VanillaSoftwareProcess) would not be obvious to a blueprint
author.
For some use-cases, a bash script becomes easier to write and test,
rather than trying to support all these variations.
However, making simple cases simpler is a good thing.
---
Thoughts?
Aled
[1] https://en.wikipedia.org/wiki/Mixin
[2] https://docs.docker.com/engine/installation/linux/centos/
[3] http://serverfault.com/a/654660/323894
On 24/05/2016 09:57, Thomas Bouron wrote:
I quite like Andrew's approach using an initializer, feels elegant and
relatively easy to implement.
Regarding the syntax, getting a rid of the `brooklyn.initializer` and
`type` is good for a discoverability point of view but that means going
through a config key which feels wrong in that case. I don't have an
argument for it, it's more a gut feeling.
On Tue, 24 May 2016 at 09:37 John McCabe <[email protected]> wrote:
Is there any way we could get rid of the initializers/type parts, rather
than:
- type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess
brooklyn.initializers:
- type: org.apache.brooklyn.initializer.stock.PackageInstaller
packages:
yum: curl
apt: curl-whatever=2.3.4
Something like:
- type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess
packages:
yum: curl
apt: curl-whatever=2.3.4
On Tue, 24 May 2016 at 09:34 Duncan Grant <[email protected]>
wrote:
Andrew,
good point. That's much neater. Should be pretty straightforward to
implement as well.
Duncan
On Mon, 23 May 2016 at 21:16 Andrew Kennedy <
[email protected]> wrote:
Duncan,
I like the idea here, of having some extra object that does the package
installation, but what about a customizer or initializer instead? You
could
have something like this:
- type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess
brooklyn.initializers:
- type: org.apache.brooklyn.initializer.stock.PackageInstaller
packages:
yum: curl
apt: curl-whatever=2.3.4
I initially thought of a JcloudsLocationCustomizer but that fails when
we
use BYON locations, so I am thinking of something more like a generic
entity customizer that runs after the entity has been created, and the
location is available and SSHable. There are lots of other things these
initializers could do, like setting up users, configuring networking,
mounting volumes and so on. I don't think its a big stretch to add this
functionality here.
Andrew.
On Fri, 6 May 2016 at 14:40 Duncan Grant <
[email protected]
wrote:
I agree that this is a problem that needs solved but I have
misgivings
about each of the proposed solutions.
Firstly I think that "discovering" how to write brooklyn yaml can be
difficult. It can be difficult to find things in the documentation
and
if
you don't know to look for them in the first place then your only
hope
is
coming across an example.
So if we pre-install a set of bash scripts before running the install
scripts I don't see any obvious way for someone using brooklyn to
find
out
about those scripts, and when the scripts change I imagine people
won't
notice the new functionality either.
Another downside to using scripts is that it will become more
difficult
to
debug script failures. At the moment I can copy the contents of
stdin
and
see the error associated. If I had to step into scripts in other
files
it
could become much harder to find a failure. In my opinion this ease
of
reproducibility and seeing the code I am running is one of the
advantages
of using bash over just using chef/puppet/salt/etc.
If we could find a widely used, well tested bash library then some of
these
issues might be mitigated but I've yet to find one.
There is a similar issue with discovering the behaviour of the DSL
option.
I think that the DSL only works because we keep it to a minimum and
really
it only does one task - which is run-time configuring relationships
between
entities. If we extend it then we have replaced java with dsl and I
don't
think that benefits anyone. One advantage of the DSL option is that
it
will generate bash so it shouldn't make debugging any harder.
The advantage of adding config options to the VanillaSoftwareProcess
is
that it adds to extra ways of discovering the behaviour - through the
composer's autocomplete and in javadoc. I assume that this would
generate
extra tasks so we could again debug the bash used and it might make
it
easier as we might be able to see exactly which package failed to
install.
However the big downside to this is that we are starting to make a
sort
of
god object which contains all brooklyn functionality in one object -
i.e.
VanilllaSoftwareProcess and I'm not sure that's where we want to go.
After all that I don't really have a good answer.
My not too good answer would be to make more use of the brooklyn
child
relationship so that a) at least the yaml could be composed from
small
pieces of code b) we could write a InstallPackage entity to simplify
the
yaml.
This might look something like:
- type: brooklyn.entity.basic.VanillaSoftwareProcess
install.command: curl somefile > somfile.txt
brooklyn.config:
children.startable.mode: foreground_early
brooklyn.children:
- type: brooklyn.entity.basic.VanillaSoftwareProcess
install.command: |
which curl || \
{ sudo apt-get update && sudo apt-get install curl ; }
|| \
{ sudo yum update && sudo yum install curl ; } || \
{ echo WARNING: cannot install curl && exit 1 ; }
or
- type: brooklyn.entity.basic.InstallPackage
package: curl
or
- type: brooklyn.entity.basic.InstallPackage
packages:
yum: curl
apt: someOtherCurl
This way everything is discoverable through documentation, javadoc,
and
autocomplete. Common requirements like curl are re-usable. Generated
bash
is explicit and if curl fails then it will go on fire.
On the other hand it requires longer yaml, isn't as readable, and due
to
the use of child parent relationships has a bit of a learning hurdle
Regards
Duncan
On Wed, 4 May 2016 at 17:35 Thomas Bouron <
[email protected]
wrote:
Hi Guglielmo.
I'm not sure to follow your idea of "stacks", could you elaborate
on
that
please?
Best.
On Tue, 3 May 2016, 13:38 Guglielmo Nigri, <
[email protected]> wrote:
I propose a declarative approach. For example we could add a
configKey
to
VanillaSoftwareProcess called requiredPackages.
This way, one could just specify the prerequisite packages, sort
of
dependencies for the software process -- this would also open up
interesting possibilities such as “stacks” as bundled
dependencies.
A declarative approach would also work for Windows (or non-bash
environments).
Cheers,
Guglielmo
On 3 May 2016 at 14:28, Andrea Turli <
[email protected]
wrote:
Great discussion guys!
It seems that it is a shared need, and I wanted to discuss with
you
as
I
was not sure about my approach.
Andrew,
I like your proposal, thanks!
Thanks,
Andrea
On 3 May 2016 at 13:05, Aled Sage <[email protected]> wrote:
I lean towards Andrew's approach, rather than a special
$brooklyn.installPackage. Note that different distros use
different
package
names sometimes, so the parameters to such a function can get
annoying.
I've been hesitant about us going down the road of
`brooklyn-commands.sh`
for achieving portable blueprints. It feels like we are
increasing
the
overlap with things like Chef, Salt, Ansible and Puppet
(which
we
can
build
on top of with Brooklyn).
However, if we keep brooklyn-commands.sh small and focused,
then
maybe
it's a good idea.
Aled
p.s. I want to decrease the use of
`BashCommands.installExecutable()`.
The
bash command it generates, with all the alternatives and
parentheses,
does
not look like nice Bash. A brooklyn-commands.sh could do it
much
cleaner,
with a properly structured multi-line if...elif... (or
whatever).
On 03/05/2016 11:43, Andrew Kennedy wrote:
How about an alternative approach - for all SoftwareProcess
entities,
Brooklyn will copy over a script file called
`brooklyn-commands.sh`
and
the
entity commands will soutce this script before running the
rest
of
their
configured commands. The script will contain OS agnostic
functions
written
in Bash that do things like install packages, download Curl,
get a
file
from a URL etc. Then, the install config might look like
this:
```
install.commands: |
brooklyn-installPackages apt:openjdk-1.8.0
yum:java-1.8.0-openjdk-devel
java-1.8.0
brooklyn-installPackages curl
brooklyn-runAsRoot cp /tmp/whatever /etc/hosts
```
Andrew.
On Tue, 3 May 2016 at 11:24 Andrea Turli <
[email protected]>
wrote:
hi,
I’ve been thinking about an utility to simplify the YAML
blueprint
creation: from my experience when using
VanillaSoftwareProcess
is
annoying
to write a portable script just to install a package (say,
java)
valid
for
apt, yum, etc so I usually write it (multiple times) just
for
an
OS.
To increase the portability of the YAML blueprint I’d like
to
suggest
we
extend the brooklyn DSL with something like:
```
$brooklyn.installPackage(“curl”)
$brooklyn:installPackage({"apt", "openjdk-1.8.0", "yum",
"java-1.8.0-openjdk-devel"}, "java-1.8.0")
```
instead of things like
```
which curl || \
{ sudo apt-get update && sudo apt-get install curl
; }
||
\
{ sudo yum update && sudo yum install curl ; } ||
\
{ echo WARNING: cannot install curl && exit 1 ; }
```
I’m not entirely sure this feature fits well on the DSL.
Alternatively, we could add a configKey to
VanillaSoftwareProcess
called
requiredPackages for a more declarative approach
(@googlielmo's
idea)
Wdyt?
--
Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
http://www.cloudsoftcorp.com/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
--
Andrew Kennedy ; Founder clocker.io project ; @grkvlt ; Cloudsoft
