It also means clients ('br' at least) can easily validate the upload even
before execution by checking that all the manifest details are present.On Fri, 3 Mar 2017 at 09:57 Geoff Macartney < [email protected]> wrote: > Just on the last point of "We could also support the example below", I'd > say let's not even be that flexible - my feeling is that the more flexible > we are, the more confusing it is, and the harder to get right. If we're > going to add this capability let's just have one way to do it; if you are > going to do a ZIP upload, you MUST have a catalog.bom, which MUST contain a > manifest section, which MUST have both a symbolic name and a version. That > way everything's explicit and very clear, every time. > > > > On Thu, 2 Mar 2017 at 11:45 Aled Sage <[email protected]> wrote: > > Geoff, all, > > I was imagining the manifest version (in the catalog.bom) to be separate > from the item version. The reason is that we support multiple items in > the bom that can be independently versioned. > > Somone perverted could write: > > brooklyn.catalog: > version: 1.0 > manifest: > symbolic_name: com.example.Foo > version: 2.0 > items: > - id: foo > version: 3.0 > itemType: entity > item: > type: blah > - id: bar > version: 4.0 > itemType: entity > item: > type: blah.blah > > Here, the "1.0" version is not used by anything; the auto-generated OSGi > bundle would be version 2.0; catalog item foo would be 3.0; and catalog > item bar would be 4.0. > > If we were starting from scratch, I'd be tempted to lock things down a > lot more for what we actually support. For now, it feels like asking for > an explicit version is a reasonable thing to do. > > --- > We could also support the example below, where "1.0" is used by both the > manifest and the item: > > brooklyn.catalog: > version: 1.0 > manifest: > symbolic_name: com.example.Foo > items: > - id: foo > itemType: entity > item: > type: blah > > The general rule would be: if there is a version in the manifest > section, that will be used; if not then a top-level version number will > be used. If that is also missing, then fail. > > i.e. we would not accept: > > # Fails - manifest has no version. > brooklyn.catalog: > manifest: > symbolic_name: com.example.Foo > items: > - id: foo > version: 1.0 > itemType: entity > item: > type: blah > > Thoughts? > > Aled > > > On 02/03/2017 10:02, Geoff Macartney wrote: > > I take Alex's point about the manifest being Java specific, and I agree > > therefore we shouldn't insist on it. > > > > +1 also to preferring explicit name/version in the catalog.bom rather > than > > as API params, I agree we do > > need to keep the version in source control. > > > > Question on your straw man, does the 'version' below > > > > brooklyn.catalog: > > manifest: > > symbolic_name: com.example.Foo > > version: 1.0.0-SNAPSHOT > > items: > > > > _replace_ the 'version' in existing catalogs > > > > brooklyn.catalog: > > version: "0.1.0-SNAPSHOT" > > > > or is it an independently variable value, i.e. the version of the bundle, > > separate from the version of catalog items that it happens to contain? > > (Sounds a bit disquieting, but I guess it's possible). > > > > e.g. could you have > > > > brooklyn.catalog: > > version: 0.11.0-SNAPSHOT > > manifest: > > symbolic_name: com.example.Foo > > version: 1.0.0-SNAPSHOT > > items: > > - id: foo > > type: xyz > > > > which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to > > contain item foo:0.11.0-SNAPSHOT? > > > > If so, what would happen if you left out the second line above? What > > version of 'foo' would the catalog contain? > > > > Or does the version within the manifest _mean_ that it is not only the > > bundle version you are specifying, but the catalog item versions too? (I > > guess unless the item explicitly supplies its own version.) > > > > Geoff > > > > > > > > > > On Wed, 1 Mar 2017 at 17:26 Aled Sage <[email protected]> wrote: > > > > Hi all, > > > > Discussion of the REST api kicked off again in > > > https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366: > > > > Alex wrote: > > > > Requiring a MANIFEST.MF makes the input strongly java-centric; I'd > > like to appeal to people who write YAML blueprints with co-bundled > > scripts and images. They'd wonder why they can't simply make and > > upload a ZIP. They could probably be persuaded to supply a name and > > optional version on a CLI or UI, and understand why it is needed > > (hence supporting those args) but it would not be idiomatic to > > anyone but a java programmer to create a META-INF/ dir with a > > MANIFEST.MF and its syntax. Meanwhile it is very easy for us to > > convert the ZIP to a JAR on the server. Feels like uncontroversial > > good UX. > > > > OTOH for a java programmer a MANIFEST.MF is natural, and they'd want > > to drop the name/version args if they are in that file, and I see no > > reason to forbid that pattern. > > > > I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As > > for Geoff's suggestion that we could auto-generate a manifest in the > > `br` CLI, I'd prefer a more general solution that works for users of the > > REST api as well (i.e. doing it server-side). > > > > --- > > Svet suggested that the catalog.bom could give the symbolic name + > > version, via additional metadata in that file. > > > > What I really like about that is it's in version control, becuase it's > > in the file/repo. If alternatively the name/version are just passed as > > REST api parameters, then it's not in version control (and is more > > susceptible to typos etc). > > > > I'm not sure what we'd want this to look like. As a straw man: > > > > brooklyn.catalog: > > manifest: > > symbolic_name: com.example.Foo > > version: 1.0.0-SNAPSHOT > > items: > > - ... > > > > If the user built their own real OSGi bundle, then they wouldn't need to > > include this "manifest" section. If they did include that and it > > contradicted the OSGi bundle's manifest, then we'd fail-fast. > > > > (Note this reminds me of the (unrelated) metadata described in > > https://github.com/brooklyncentral/blueprint-repository- there is a > > "publish" block that can be added to the catalog.bom.) > > > > --- > > With Svet's suggestion, if there was no manifest section/file, then > > should we auto-generate something from the item(s) in the catalog.bom? > > > > I can see how that could easily work for a .bom file that has a single > > item (e.g. we use the catalog item's id + version, possibly with a > > special prefix in the symbolic name to avoid conflicts). > > > > However, if there are multiple items then it would get trickier. > > > > I'm inclined to say that for a minimal viable product (MVP) we can > > insist on the "manifest" section in the catalog.bom. > > > > Aled > > > > > > On 20/12/2016 16:34, Svetoslav Neykov wrote: > >>> Svet, if instead we tried to infer it from the catalog.bom, would we > > require some additional metadata within the .bom file? Or would we use > the > > catalog item's id + version? I'm not convinced by the latter - it would > > mean some .bom files would work and others wouldn't (e.g. if the .bom had > > multiple items with different versions). Better to support the explicit > > approach IMO. > >> I imagine it would be additional metadata. On the other hand I don't see > > a technical reason why we need an explicit symbolicName and version - > they > > can be auto-generated. > >> Svet. > >> > >> > >>> On 20.12.2016 г., at 17:50, Aled Sage <[email protected]> wrote: > >>> > >>> Hi all, > >>> > >>> +1 > >>> > >>> (D) sounds good. What version are you imagining the bundle would be, if > > one runs `br catalog add ~/my/project/ --name com.example.myproject`? > >>> --- > >>> I like the idea of uploading a plain zip (rather than only supporting > > OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi > > becomes a (hidden) implementation detail to many users. > >>> --- > >>> If auto-generating the manifest, I think we need the user to be > explicit > > about symbolic name and version. Having these supplied in the REST api > call > > (as Alex suggests) would achieve that. > >>> Svet, if instead we tried to infer it from the catalog.bom, would we > > require some additional metadata within the .bom file? Or would we use > the > > catalog item's id + version? I'm not convinced by the latter - it would > > mean some .bom files would work and others wouldn't (e.g. if the .bom had > > multiple items with different versions). Better to support the explicit > > approach IMO. > >>> --- > >>> For E ("have a mechanism whereby deployed entities based on an affected > > blueprint are optionally migrated to the new code"), that feels like a > > separate discussion. It could equally apply to a pure YAML .bom file that > > has been added to the catalog. > >>> I suggest we discuss that in a separate email thread. > >>> > >>> --- > >>> For (G), it's an interesting suggestion from Svet to make use of Karaf > > Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone > Brooklyn > > node whose VM has died, then it adds big additional requirements for what > > constitutes the "persisted state"). On the other hand, it's good to use > > well-established technologies rather than re-inventing things! > >>> An alternative ("pure brooklyn") approach could be to write the bundle > > to persisted state; on rebind, we'd install + activate those bundles. > >>> --- > >>> For "catalogGroupId", I agree with Svet that in the initial use-case > > this can be an implementation detail. > >>> It could be set as the bundle's symbolic name + version: everything > from > > the bundle should be deleted at once, along with the bundle. > >>> Longer term, I can see how exposing "catalogGroupId" to the user could > > support more use-cases (e.g. for several catalog items from different > > bundles to work together). I don't think we should try to support that > yet. > >>> Aled > >>> > >>> > >>> On 19/12/2016 17:19, Geoff Macartney wrote: > >>>> hi Alex, > >>>> > >>>> this looks like a good feature to have, I shall look at the PR as soon > > as I > >>>> can. > >>>> > >>>> The catalog.bom scanner feature was initially enabled by default, but > we > >>>> had to > >>>> disable it because it turned out not to work properly with rebind. I > > don't > >>>> think > >>>> it should be a lot of work to fix that but it hasn't been something > > we've > >>>> got round > >>>> to yet. This would be a great opportunity to look back at that. > >>>> > >>>> Some random thoughts: > >>>> > >>>> re (C), if we are going to treat the zips as bundles, my gut feel is > > that > >>>> we > >>>> should insist on a manifest and get the metadata from it. It doesn't > > feel > >>>> to me > >>>> like it makes much sense to allow a zip file without a MANIFEST.MF but > >>>> convey > >>>> the intended bundle metadata to Brooklyn via HTTP headers. And rather > > than > >>>> infer bundle metadata I think it's better to ask users to be explicit > > about > >>>> what > >>>> their intentions are. To make users lives easier, we could > >>>> add a command to br to generate the manifest (locally) with correct > > syntax, > >>>> so that the manifest is in the right place, rather than have br add > the > > data > >>>> to the "upload" request headers. > >>>> > >>>> re. (D) will be glad to have a look at it > >>>> > >>>> re. (E) it would certainly need to be optional - maybe keep it as an > >>>> explicit > >>>> separate command ('upgrade'?) > >>>> > >>>> (F) it does seem like a lot of work but might be nice for users who > are > > not > >>>> keen on command lines. > >>>> > >>>> G - I: we'll definitely need to pay close attention to persistence > and > >>>> rebind; > >>>> I wonder also about HA operation, are there any additional > implications? > >>>> > >>>> (J) I think it would be good to treat all the files from a jar, sorry > >>>> bundle, > >>>> as an atomic group - cleaner that way perhaps than allowing > > delete/update > >>>> of > >>>> individual entries from a bundle on a piecemeal basis. Rest support > on > >>>> delete > >>>> catalog could warn about related catalog entries being deleted and ask > > for > >>>> a "--force" param to confirm. > >>>> > >>>> Geoff > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov < > >>>> [email protected]> wrote: > >>>> > >>>>> +1 > >>>>> > >>>>> Some thoughts: > >>>>> * (A) add a utility class BundleMaker > >>>>> Sounds very similar to > >>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles < > >>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles> > >>>>> Looking at the code it's much more focused on zip files so I > guess > >>>>> there's no much overlap, but worth keeping in mind > >>>>> * (C) accept bundle symbolic name and version > >>>>> Why require them at all? Could infer them from the catalog.bom > in > > some > >>>>> way - maybe require those properties to be in there. If not present > are > >>>>> they really needed? > >>>>> * (G) Bundles installed via this mechanism are not persisted > > currently & > >>>>> (I) We persist the individual catalog items as YAML, so we end up > with > > two > >>>>> records > >>>>> Suggest marking the catalog items coming from bundles as > >>>>> non-persistable. Then try to share the bundles between HA nodes. > (Karaf > >>>>> Cellar?) > >>>>> * (J) Introduce a catalogGroupId field on catalog items; > >>>>> Agree this could be useful and I like the idea of deleting the > > bundle > >>>>> altogether with the catalog items. From user's perspective I don't > see > > the > >>>>> need for an extra field (i.e. it's an implementation detail). > >>>>> > >>>>> Svet. > >>>>> > >>>>> > >>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld < > >>>>> [email protected]> wrote: > >>>>>> Hi Brooklyners- > >>>>>> > >>>>>> In the code we currently have two routes for users to install new > >>>>>> blueprints: > >>>>>> > >>>>>> (1) upload a catalog YAML file to /v1/catalog > >>>>>> > >>>>>> (2) install a bundle with catalog.bom in the root > >>>>>> > >>>>>> The feature (2) is disabled by default, but I'd like to move towards > >>>>>> enabling it. This will make it easier to create nicely structured > BOM > >>>>>> files because scripts etc can be taken out of the BOM, stored as > > files in > >>>>>> the same bundle. (Because URLs of the form > >>>>>> `classpath://scripts/install.sh` use the bundle's classpath to > > resolve.) > >>>>>> As a first step in #485 [1] I do a few things: > >>>>>> > >>>>>> (A) add a utility class BundleMaker that lets us create and modify > >>>>>> bundles/zips, to make it easier to do things we might want to with > >>>>> bundles, > >>>>>> especially for testing > >>>>>> > >>>>>> (B) add an endpoint to the REST API which allows uploading a bundle > > ZIP > >>>>>> (C) accept bundle symbolic name and version in that REST API to > >>>>> facilitate > >>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is > automatically > >>>>>> generated > >>>>>> > >>>>>> With this PR, if you have a directory on your local file system with > >>>>>> scripts and config files, and a BOM which refers to them, you can > just > >>>>> ZIP > >>>>>> that up an upload it, specifying the bundle name so that a YAML > > blueprint > >>>>>> author never needs to touch any java-isms. > >>>>>> > >>>>>> Where I see this going is a development workflow where a user can > edit > >>>>>> files locally and upload the ZIP to have that installed, and if they > > make > >>>>>> changes locally they can POST it again to have catalog items updated > >>>>>> (because default version is a SNAPSHOT). We could also: > >>>>>> > >>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a > ZIP > >>>>> and > >>>>>> POST it, with bundle name metadata, so essentially the user's > process > > is > >>>>>> just to run that whenever they make a change > >>>>>> > >>>>>> (E) have a mechanism whereby deployed entities based on an affected > >>>>>> blueprint are optionally migrated to the new code, so if you've > > changed > >>>>> an > >>>>>> enricher the changes are picked up, or if say a launch.sh script has > >>>>>> changed, a restart will run the new code > >>>>>> > >>>>>> The above are fairly straightforward programmatically (although good > > user > >>>>>> interaction with (E) needs some thought). So I think we can pretty > >>>>> quickly > >>>>>> get to a much smoother dev workflow. > >>>>>> > >>>>>> > >>>>>> That's the highlight of this message. You can jump to the end, > unless > >>>>>> you're interested in some important but low level details... > >>>>>> > >>>>>> > >>>>>> I'm also tempted by: > >>>>>> > >>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and > writing > >>>>>> straight from GitHub -- but this seems like a lot of work and I'm > not > >>>>>> convinced it's much better than (D) workflow-wise > >>>>>> > >>>>>> Before we can change (2) to be the default, or start widely using > the > >>>>> POST > >>>>>> a ZIP feature, we need to sort out some issues to do with > persistence > > and > >>>>>> reloading: > >>>>>> > >>>>>> (G) Bundles installed via this mechanism are not persisted > currently, > > so > >>>>> if > >>>>>> you move to a different Brooklyn using the same backing store, > you'll > >>>>> lose > >>>>>> those bundles > >>>>>> > >>>>>> (H) On rebind, bundles aren't always activated when needed, meaning > > items > >>>>>> can't be loaded > >>>>>> > >>>>>> (I) We persist the individual catalog items as YAML, so we end up > with > >>>>> two > >>>>>> records — the YAML from the catalog.bom in the bundle, and the YAML > >>>>>> persisted for the item. This isn't a problem per se, but something > to > >>>>>> think about, and some sometimes surprising behaviour. In particular > > if > >>>>> you > >>>>>> delete the persisted YAML, the bundle is still there so the item is > no > >>>>>> longer deleted after a full rebind. > >>>>>> > >>>>>> One idea which might be useful is: > >>>>>> > >>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do > > two > >>>>>> things: if you try to delete an item with such a record, you'll be > >>>>>> encouraged to delete all such items (maybe disallowed to delete an > >>>>>> individual one), with the effect of deleting the bundle if it comes > > from > >>>>> a > >>>>>> bundle; and when resolving types we search first for items with the > > same > >>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and > > MyNode:1.0 in > >>>>>> the same group, the former can refer simply to "MyNode" but if I > > install > >>>>> a > >>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node > -- > >>>>> this > >>>>>> has bitten people i the past) > >>>>>> > >>>>>> There is a related Brooklyn upgrade problem worth mentioning, which > > the > >>>>>> above might help with, where: > >>>>>> > >>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no > > longer > >>>>>> have certain entities that were at v10, since we don't include > those; > > an > >>>>>> upgrade could include rules that certain groupIds need to be > updated, > > or > >>>>> it > >>>>>> can search and attempt to automatically apply the updates > >>>>>> > >>>>>> > >>>>>> Quite a lot here and we don't need to solve it but I wanted to: > >>>>>> > >>>>>> * Share the current thinking > >>>>>> > >>>>>> * Get opinions on the general dev workflow suggested by (D) > >>>>>> > >>>>>> > >>>>>> Thanks for feedback -- and if we like it help with (D) would be > >>>>> appreciated! > >>>>>> Best > >>>>>> Alex > >>>>>> > >>>>>> > >>>>>> > >>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485 > >
