Just on the last point of "We could also support the example below", I'd
say let's not even be that flexible - my feeling is that the more flexible
we are, the more confusing it is, and the harder to get right. If we're
going to add this capability let's just have one way to do it; if you are
going to do a ZIP upload, you MUST have a catalog.bom, which MUST contain a
manifest section, which MUST have both a symbolic name and a version. That
way everything's explicit and very clear, every time.
On Thu, 2 Mar 2017 at 11:45 Aled Sage <[email protected]> wrote:
Geoff, all,
I was imagining the manifest version (in the catalog.bom) to be separate
from the item version. The reason is that we support multiple items in
the bom that can be independently versioned.
Somone perverted could write:
brooklyn.catalog:
version: 1.0
manifest:
symbolic_name: com.example.Foo
version: 2.0
items:
- id: foo
version: 3.0
itemType: entity
item:
type: blah
- id: bar
version: 4.0
itemType: entity
item:
type: blah.blah
Here, the "1.0" version is not used by anything; the auto-generated OSGi
bundle would be version 2.0; catalog item foo would be 3.0; and catalog
item bar would be 4.0.
If we were starting from scratch, I'd be tempted to lock things down a
lot more for what we actually support. For now, it feels like asking for
an explicit version is a reasonable thing to do.
---
We could also support the example below, where "1.0" is used by both the
manifest and the item:
brooklyn.catalog:
version: 1.0
manifest:
symbolic_name: com.example.Foo
items:
- id: foo
itemType: entity
item:
type: blah
The general rule would be: if there is a version in the manifest
section, that will be used; if not then a top-level version number will
be used. If that is also missing, then fail.
i.e. we would not accept:
# Fails - manifest has no version.
brooklyn.catalog:
manifest:
symbolic_name: com.example.Foo
items:
- id: foo
version: 1.0
itemType: entity
item:
type: blah
Thoughts?
Aled
On 02/03/2017 10:02, Geoff Macartney wrote:
I take Alex's point about the manifest being Java specific, and I agree
therefore we shouldn't insist on it.
+1 also to preferring explicit name/version in the catalog.bom rather
than
as API params, I agree we do
need to keep the version in source control.
Question on your straw man, does the 'version' below
brooklyn.catalog:
manifest:
symbolic_name: com.example.Foo
version: 1.0.0-SNAPSHOT
items:
_replace_ the 'version' in existing catalogs
brooklyn.catalog:
version: "0.1.0-SNAPSHOT"
or is it an independently variable value, i.e. the version of the bundle,
separate from the version of catalog items that it happens to contain?
(Sounds a bit disquieting, but I guess it's possible).
e.g. could you have
brooklyn.catalog:
version: 0.11.0-SNAPSHOT
manifest:
symbolic_name: com.example.Foo
version: 1.0.0-SNAPSHOT
items:
- id: foo
type: xyz
which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
contain item foo:0.11.0-SNAPSHOT?
If so, what would happen if you left out the second line above? What
version of 'foo' would the catalog contain?
Or does the version within the manifest _mean_ that it is not only the
bundle version you are specifying, but the catalog item versions too? (I
guess unless the item explicitly supplies its own version.)
Geoff
On Wed, 1 Mar 2017 at 17:26 Aled Sage <[email protected]> wrote:
Hi all,
Discussion of the REST api kicked off again in
https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
Alex wrote:
Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
like to appeal to people who write YAML blueprints with co-bundled
scripts and images. They'd wonder why they can't simply make and
upload a ZIP. They could probably be persuaded to supply a name and
optional version on a CLI or UI, and understand why it is needed
(hence supporting those args) but it would not be idiomatic to
anyone but a java programmer to create a META-INF/ dir with a
MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
convert the ZIP to a JAR on the server. Feels like uncontroversial
good UX.
OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
to drop the name/version args if they are in that file, and I see no
reason to forbid that pattern.
I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As
for Geoff's suggestion that we could auto-generate a manifest in the
`br` CLI, I'd prefer a more general solution that works for users of the
REST api as well (i.e. doing it server-side).
---
Svet suggested that the catalog.bom could give the symbolic name +
version, via additional metadata in that file.
What I really like about that is it's in version control, becuase it's
in the file/repo. If alternatively the name/version are just passed as
REST api parameters, then it's not in version control (and is more
susceptible to typos etc).
I'm not sure what we'd want this to look like. As a straw man:
brooklyn.catalog:
manifest:
symbolic_name: com.example.Foo
version: 1.0.0-SNAPSHOT
items:
- ...
If the user built their own real OSGi bundle, then they wouldn't need to
include this "manifest" section. If they did include that and it
contradicted the OSGi bundle's manifest, then we'd fail-fast.
(Note this reminds me of the (unrelated) metadata described in
https://github.com/brooklyncentral/blueprint-repository- there is a
"publish" block that can be added to the catalog.bom.)
---
With Svet's suggestion, if there was no manifest section/file, then
should we auto-generate something from the item(s) in the catalog.bom?
I can see how that could easily work for a .bom file that has a single
item (e.g. we use the catalog item's id + version, possibly with a
special prefix in the symbolic name to avoid conflicts).
However, if there are multiple items then it would get trickier.
I'm inclined to say that for a minimal viable product (MVP) we can
insist on the "manifest" section in the catalog.bom.
Aled
On 20/12/2016 16:34, Svetoslav Neykov wrote:
Svet, if instead we tried to infer it from the catalog.bom, would we
require some additional metadata within the .bom file? Or would we use
the
catalog item's id + version? I'm not convinced by the latter - it would
mean some .bom files would work and others wouldn't (e.g. if the .bom had
multiple items with different versions). Better to support the explicit
approach IMO.
I imagine it would be additional metadata. On the other hand I don't see
a technical reason why we need an explicit symbolicName and version -
they
can be auto-generated.
Svet.
On 20.12.2016 г., at 17:50, Aled Sage <[email protected]> wrote:
Hi all,
+1
(D) sounds good. What version are you imagining the bundle would be, if
one runs `br catalog add ~/my/project/ --name com.example.myproject`?
---
I like the idea of uploading a plain zip (rather than only supporting
OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi
becomes a (hidden) implementation detail to many users.
---
If auto-generating the manifest, I think we need the user to be
explicit
about symbolic name and version. Having these supplied in the REST api
call
(as Alex suggests) would achieve that.
Svet, if instead we tried to infer it from the catalog.bom, would we
require some additional metadata within the .bom file? Or would we use
the
catalog item's id + version? I'm not convinced by the latter - it would
mean some .bom files would work and others wouldn't (e.g. if the .bom had
multiple items with different versions). Better to support the explicit
approach IMO.
---
For E ("have a mechanism whereby deployed entities based on an affected
blueprint are optionally migrated to the new code"), that feels like a
separate discussion. It could equally apply to a pure YAML .bom file that
has been added to the catalog.
I suggest we discuss that in a separate email thread.
---
For (G), it's an interesting suggestion from Svet to make use of Karaf
Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
Brooklyn
node whose VM has died, then it adds big additional requirements for what
constitutes the "persisted state"). On the other hand, it's good to use
well-established technologies rather than re-inventing things!
An alternative ("pure brooklyn") approach could be to write the bundle
to persisted state; on rebind, we'd install + activate those bundles.
---
For "catalogGroupId", I agree with Svet that in the initial use-case
this can be an implementation detail.
It could be set as the bundle's symbolic name + version: everything
from
the bundle should be deleted at once, along with the bundle.
Longer term, I can see how exposing "catalogGroupId" to the user could
support more use-cases (e.g. for several catalog items from different
bundles to work together). I don't think we should try to support that
yet.
Aled
On 19/12/2016 17:19, Geoff Macartney wrote:
hi Alex,
this looks like a good feature to have, I shall look at the PR as soon
as I
can.
The catalog.bom scanner feature was initially enabled by default, but
we
had to
disable it because it turned out not to work properly with rebind. I
don't
think
it should be a lot of work to fix that but it hasn't been something
we've
got round
to yet. This would be a great opportunity to look back at that.
Some random thoughts:
re (C), if we are going to treat the zips as bundles, my gut feel is
that
we
should insist on a manifest and get the metadata from it. It doesn't
feel
to me
like it makes much sense to allow a zip file without a MANIFEST.MF but
convey
the intended bundle metadata to Brooklyn via HTTP headers. And rather
than
infer bundle metadata I think it's better to ask users to be explicit
about
what
their intentions are. To make users lives easier, we could
add a command to br to generate the manifest (locally) with correct
syntax,
so that the manifest is in the right place, rather than have br add
the
data
to the "upload" request headers.
re. (D) will be glad to have a look at it
re. (E) it would certainly need to be optional - maybe keep it as an
explicit
separate command ('upgrade'?)
(F) it does seem like a lot of work but might be nice for users who
are
not
keen on command lines.
G - I: we'll definitely need to pay close attention to persistence
and
rebind;
I wonder also about HA operation, are there any additional
implications?
(J) I think it would be good to treat all the files from a jar, sorry
bundle,
as an atomic group - cleaner that way perhaps than allowing
delete/update
of
individual entries from a bundle on a piecemeal basis. Rest support
on
delete
catalog could warn about related catalog entries being deleted and ask
for
a "--force" param to confirm.
Geoff
On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
[email protected]> wrote:
+1
Some thoughts:
* (A) add a utility class BundleMaker
Sounds very similar to
https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
Looking at the code it's much more focused on zip files so I
guess
there's no much overlap, but worth keeping in mind
* (C) accept bundle symbolic name and version
Why require them at all? Could infer them from the catalog.bom
in
some
way - maybe require those properties to be in there. If not present
are
they really needed?
* (G) Bundles installed via this mechanism are not persisted
currently &
(I) We persist the individual catalog items as YAML, so we end up
with
two
records
Suggest marking the catalog items coming from bundles as
non-persistable. Then try to share the bundles between HA nodes.
(Karaf
Cellar?)
* (J) Introduce a catalogGroupId field on catalog items;
Agree this could be useful and I like the idea of deleting the
bundle
altogether with the catalog items. From user's perspective I don't
see
the
need for an extra field (i.e. it's an implementation detail).
Svet.
On 16.12.2016 г., at 12:50, Alex Heneveld <
[email protected]> wrote:
Hi Brooklyners-
In the code we currently have two routes for users to install new
blueprints:
(1) upload a catalog YAML file to /v1/catalog
(2) install a bundle with catalog.bom in the root
The feature (2) is disabled by default, but I'd like to move towards
enabling it. This will make it easier to create nicely structured
BOM
files because scripts etc can be taken out of the BOM, stored as
files in
the same bundle. (Because URLs of the form
`classpath://scripts/install.sh` use the bundle's classpath to
resolve.)
As a first step in #485 [1] I do a few things:
(A) add a utility class BundleMaker that lets us create and modify
bundles/zips, to make it easier to do things we might want to with
bundles,
especially for testing
(B) add an endpoint to the REST API which allows uploading a bundle
ZIP
(C) accept bundle symbolic name and version in that REST API to
facilitate
uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
automatically
generated
With this PR, if you have a directory on your local file system with
scripts and config files, and a BOM which refers to them, you can
just
ZIP
that up an upload it, specifying the bundle name so that a YAML
blueprint
author never needs to touch any java-isms.
Where I see this going is a development workflow where a user can
edit
files locally and upload the ZIP to have that installed, and if they
make
changes locally they can POST it again to have catalog items updated
(because default version is a SNAPSHOT). We could also:
(D) have `br catalog add ~/my/project/ --name my.project` create a
ZIP
and
POST it, with bundle name metadata, so essentially the user's
process
is
just to run that whenever they make a change
(E) have a mechanism whereby deployed entities based on an affected
blueprint are optionally migrated to the new code, so if you've
changed
an
enricher the changes are picked up, or if say a launch.sh script has
changed, a restart will run the new code
The above are fairly straightforward programmatically (although good
user
interaction with (E) needs some thought). So I think we can pretty
quickly
get to a much smoother dev workflow.
That's the highlight of this message. You can jump to the end,
unless
you're interested in some important but low level details...
I'm also tempted by:
(F) Integration with web-based IDE and/or Brooklyn reading and
writing
straight from GitHub -- but this seems like a lot of work and I'm
not
convinced it's much better than (D) workflow-wise
Before we can change (2) to be the default, or start widely using
the
POST
a ZIP feature, we need to sort out some issues to do with
persistence
and
reloading:
(G) Bundles installed via this mechanism are not persisted
currently,
so
if
you move to a different Brooklyn using the same backing store,
you'll
lose
those bundles
(H) On rebind, bundles aren't always activated when needed, meaning
items
can't be loaded
(I) We persist the individual catalog items as YAML, so we end up
with
two
records — the YAML from the catalog.bom in the bundle, and the YAML
persisted for the item. This isn't a problem per se, but something
to
think about, and some sometimes surprising behaviour. In particular
if
you
delete the persisted YAML, the bundle is still there so the item is
no
longer deleted after a full rebind.
One idea which might be useful is:
(J) Introduce a catalogGroupId field on catalog items; this will do
two
things: if you try to delete an item with such a record, you'll be
encouraged to delete all such items (maybe disallowed to delete an
individual one), with the effect of deleting the bundle if it comes
from
a
bundle; and when resolving types we search first for items with the
same
catalogGroupId (so that e.g. if I install MyCluster:1.0 and
MyNode:1.0 in
the same group, the former can refer simply to "MyNode" but if I
install
a
2.0 version of that group, the 1.0 cluster still loads the 1.0 node
--
this
has bitten people i the past)
There is a related Brooklyn upgrade problem worth mentioning, which
the
above might help with, where:
(K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
longer
have certain entities that were at v10, since we don't include
those;
an
upgrade could include rules that certain groupIds need to be
updated,
or
it
can search and attempt to automatically apply the updates
Quite a lot here and we don't need to solve it but I wanted to:
* Share the current thinking
* Get opinions on the general dev workflow suggested by (D)
Thanks for feedback -- and if we like it help with (D) would be
appreciated!
Best
Alex
[1] . https://github.com/apache/brooklyn-server/pull/485
