Hi Wilfried, On Wed, Aug 7, 2013 at 6:28 PM, Sibla Wilfried <[email protected]> wrote: > Hi to all > > In the last days I implemented a utility to derive a ACE feature from a maven > pom project. > I build a simple pom project with all directly required artifacts in the > dependency list. > But I had to enrich the dependencies with a list of exclusions, because some > included projects were only repackaging non-OSGi artifacts as OSGi bundles. > But e.g. if those feature references quite high number of artifacts, and all > artifacts are already available within a maven repo, this works quite good. >
Well there you go! Sounds interesting... tell us more :) Are you doing this deriving in maven or in an ACE utility? And are you talking directly against REST/client API or through an intermediate representation? > @Bram: I could possibly provide this solution in the next months... > Always looking forward to contributions! Enjoy your holiday, grz Bram > > Greetings > Wilfried > > > -----Ursprüngliche Nachricht----- > Von: Bram de Kruijff [mailto:[email protected]] > Gesendet: Mittwoch, 7. August 2013 16:49 > An: [email protected] > Betreff: Re: Where does Ace server store everything? > > On Wed, Aug 7, 2013 at 3:30 PM, Matthew Bishop <[email protected]> wrote: >> We have two problems here: definition of artifact / feature / >> distributions (AFD?), and ACE Server loadout of AFDs. >> >> *AFD definitions:* it makes a lot of sense to me to use the Maven >> project file structure to declare which bundles that define the AFD >> components. >> Maven is a well-known and capable at this task and has considerable >> tooling for transitive dependency declaration and discovery. I can >> imagine the following design: >> >> *Artifact* - maven jar artifact >> *Feature* - POM that lists <dependencies> on artifacts. Builds out a >> pom artifact. >> *Distribution* - POM that lists <dependencies> on features. Builds out >> a pom artifact. >> >> I know you might be thinking that maven isn't good enough for OSGi >> because it doesn't understand the OSGi manifest headers, or OBR, or >> capabilities, and you are right. However, at the end of the day, what >> server boffins want to do is define every single bundle artifact in a >> way that can be understood and managed in a familiar technology. Maven >> fits this bill very well. >> > > Well I did not see that one coming, but I can see the 'familiar technology' > argument ;) However, IMHO we should not require the maven schema in ACE for > this purpose. It is way too heavy (you would only be able to use a small > subset of the schema) and limited in flexibility (associations are 1-1 based > on the dependency graph). But, then again mapping pom to the ACE model should > be pretty simple. That is, once you figured out the mapping. > > Simple example mapping; > > An ACE bundle artifact has a name, description, bsn, version, url (could even > point to maven central), and an arbitrary number of tags. > So a rough mapping (excuse my rusty maven vocabulary); > > name = ${pom.name} > description = ${pom.description} > bsn = ${pom.groupId}.${pom.artifactId} > version = ${pom.version} > url = ${some.repourl} > tags = ${pom.properties} //?? > > An ACE feature basically only has a name/identitfier So simply; > > name = ${pom.name} > > The ACE artifact2FeatureAssociations derived from the feature pom > dependencies would be something like for each dependency; > > leftEndpoint = > &((Bundle-SymbolicName=${dependency.groupId}.${dependency.artifactId})(version=${dependency.version})) > rightEndpoint = (name=${pom.name}) > > And the same pattern applies for distribution and feature2Distribution. > > >> Most of the tech in OSGi that you know and love at dev time, like BND >> Tools, is still applicable. All I am suggesting is that the AFD >> definitions use the maven POM files, not necessarily the 'mvn' >> runtimes to generate the actual bundle jars. Using maven like this >> will populate the maven repositories with AFD definitions that will be >> useful for loadout. >> >> >> *Server Loadout:* The ACE Server (not the targets) would be able to >> consume the various Feature and Distribution POMs. It would be >> configured to look to maven repositories (like maven central) to load >> in the bundles as artifacts. The versioning of the AFDs would be >> managed by the maven version numbering itself. Again, very familiar, >> trusted tech. Easy to roll back if all hell breaks loose. Easy to write >> scripts to automate. >> > > Again I would not want to make the maven shema the default/requirement for > ACE and moving from some poms to a more neutral/generic representation or > even directly to ACE would probably best be handled by maven itself through a > plugin are custom packaging. Specifically when you are talking about > transitive dependencies this will be the easiest by far. > > That small piece of maven tooling wil then be able to talk to the ACE > importer of even REST/client apis directly. So given you multi-module project > for example; > > $ mvn ace:export-model > ace-deployment-descriptor.xml $ import curl > ace-deployment-descriptor.xml (using a future generic ACE importer) > > OR > > $ mvn ace:deploy-model -D serverEndpoint=http://localhost:8080/ace // talk to > the REST api directly > > Actually I wrote the latter using that Amdatu client for a former employer to > provide continuous deployment from a build, but that only covered artifacts. > > >> The mountain I am really trying to climb here is to provide a >> distribution package for OSGi applications, a package that can be >> understood and examined in existing and well-integrated deployment >> technology. >> > > Well if a maven model serves you purpose I think the above approach could > work well. As long as you are aware that you will still need OSGi aware > tooling to answer question as "will this distribution package actually > resolve in an OSGi runtime". > > Anyway, I would be happy to collaborate on coming up with a generic ACE > model/importer that would serve your purpose. > > Regards, > Bram > >> >> On Tue, Aug 6, 2013 at 11:25 PM, Bram de Kruijff <[email protected]>wrote: >> >>> Hi Matthew, >>> >>> On Wed, Aug 7, 2013 at 1:06 AM, Matthew Bishop <[email protected]> >>> wrote: >>> > Hmm, this looks like a lot like nimble's scripting. >>> > >>> > Can ACE generate a deployment script that loads up a set of >>> > artifacts/features/etc? It would go a long ways towards the >>> checkin-ability >>> > of what is manipulated in the UI / API. >>> > >>> >>> Generating scripts does not feel like the way to go for me. In this >>> example script the artifacts come from a declarative model (the R5 >>> repository representation), but obviously the association, feature >>> and distribution do not. As you already state this information should >>> be easy enough to capture in a declarative manner. You would need to >>> do this anyway if you would want to generate a script so why not >>> support that from ACE directly? I am curious to hear what your idea >>> of the 'input format' to a generic import would ideally be. >>> >>> As a reference, some time ago in wrote an Amdatu ACE Client [0] >>> (source repo not online) that talks to the REST API and is also >>> capable of doing basic import/export based on a simple JSON >>> representation of the custom java model that is in there. However, >>> that client does not talk to the OBR, so no uploads. In fact the ACE >>> repository uses a similar XML representation for disk storage just >>> listing the different entities. Also something we could leverage. >>> >>> Another idea that I am toying with is to leverage the OSGi Resource >>> model of the R5 Repository which would allow us to store all this >>> information in a R5 repository. Features and distributions could be >>> modeled as specific resource types, providing meta-data >>> (attributes/tags) through capabilities and modelling association >>> through requirements. This approach would allow capturing all >>> information in one single source and standard representation. >>> >>> So I think there are options :) Again I would like to hear what your >>> preferred approach would be an maybe we can collaborate on coming up >>> with a generic solution. >>> >>> Best Regards, >>> Bram >>> >>> [0] http://repository.amdatu.org/release-util/org.amdatu.ace.client/ >>> >>> > >>> > On Tue, Aug 6, 2013 at 2:34 PM, Marcel Offermans < >>> > [email protected]> wrote: >>> > >>> >> Hello Matthew, >>> >> >>> >> On Aug 6, 2013, at 23:10 PM, Matthew Bishop <[email protected]> >>> wrote: >>> >> >>> >> > On Tue, Aug 6, 2013 at 1:58 PM, Marcel Offermans < >>> >> > [email protected]> wrote: >>> >> > >>> >> >> To give you another example, we recently automated our >>> >> >> continuous >>> build >>> >> >> system so for each succesful build it takes the artifacts that >>> >> >> come >>> out >>> >> of >>> >> >> that build and (through a GoGo Shell script) upload them to a >>> repository >>> >> >> and then create all the metadata in ACE. The end result is that >>> >> >> we >>> can >>> >> >> automatically deploy such code directly to our QA and demo >>> environments, >>> >> >> completely automated. >>> >> > >>> >> > Is this gogo script checked into ACE somewhere? It is definitely >>> >> > an interesting part of the story. I am working my backwards from >>> >> > the end deployment story to the developer, as we did it >>> >> > developer-first and >>> ended >>> >> > up with pax and maven on the production server. Not cool. >>> >> >>> >> It is not, because it is specific to our environment. The GoGo >>> >> commands are something we only very recently added, so they're not >>> >> documented >>> well >>> >> yet. >>> >> >>> >> However, I have adapted the script we use a bit (taking out the >>> >> specific bits). It probably requires some more context to be fully >>> >> understood, >>> and >>> >> probably Bram can answer some more detailed questions about it (he >>> >> wrote the script). Basically it takes the artifacts from a build, >>> >> checks if >>> they >>> >> are really different from the ones we already deployed the >>> >> previous >>> time, >>> >> uploads all changed artifacts and configures ACE. We run this >>> >> script >>> with a >>> >> client jar from the ACE distribution, and configure it as the >>> >> startup script. As soon as the script is done, we shutdown this client >>> >> again. >>> >> >>> >> >>> >> # >>> >> # Continuous deployment gogo script # >>> >> >>> >> # environment variables we import here targetrepourl = >>> >> "$DEPLOYMENT_TARGETOBR_URL" >>> >> releaserepourl = "$DEPLOYMENT_RELEASEOBR_URL" >>> >> autoconfurl = "$DEPLOYMENT_AUTOCONF_URL" >>> >> >>> >> sourceindex = (repo:index /tmp/ace-cdeploy) sourcerepo = >>> >> (repo:repo R5 $sourceindex) targetrepo = (repo:repo OBR >>> >> $targetrepourl) releaserepo = (repo:repo OBR $releaserepourl) >>> >> >>> >> echo "Deploying release resources" >>> >> deployed = repo:cd $releaserepo $sourcerepo $targetrepo >>> >> >>> >> echo "Ensuring autoconf resource" >>> >> autoconf = (coll:first ($targetrepo find >>> >> "(osgi.identity=org.apache.felix.deployment.rp.autoconf)")) >>> >> if { $autoconf } { echo " -> exists" } \ { >>> >> repo:d $targetrepo $autoconfurl >>> >> autoconf = (coll:first ($targetrepo find >>> >> "(osgi.identity=org.apache.felix.deployment.rp.autoconf)")) >>> >> echo " -> deployed" >>> >> } >>> >> >>> >> echo "Opening client workspace" >>> >> workspace = (ace:cw) >>> >> >>> >> echo "Ensuring cdtarget exists" >>> >> target = (coll:first ($workspace lt "(id=cdtarget)")) if { $target >>> >> } { echo " -> exists" } \ { >>> >> $workspace ct "cdtarget"; >>> >> target = (coll:first ($workspace lt "(id=cdtarget)")) >>> >> echo " -> created" >>> >> } >>> >> >>> >> echo "Ensuring cdtarget is registered" >>> >> if { $target isRegistered } { } { $target register } >>> >> >>> >> echo "Ensuring cdtarget is !autoapprove" >>> >> $target setAutoApprove false >>> >> >>> >> # here we set some specific properties used by our configurations >>> >> (anonymized) >>> >> $target addTag "domain" "foo.com" >>> >> $target addTag "ip-address" "127.0.0.1" >>> >> $target addTag "timezone" "Europe/Amsterdam" >>> >> >>> >> echo "Ensuring cdfeature exists" >>> >> if { (coll:first ($workspace lf "(name=cdfeature)")) } { echo " -> >>> exists" >>> >> } { $workspace cf "cdfeature"; echo " -> created" } >>> >> >>> >> echo "Ensuring cddistribution exists" >>> >> if { (coll:first ($workspace ld "(name=cddistribution)")) } { echo >>> >> " -> exists" } { $workspace cd "cddistribution"; echo " -> >>> >> created" } >>> >> >>> >> echo "Ensuring f2d association" >>> >> if { (coll:first ($workspace lf2d >>> >> "(leftEndpoint=*name=cdfeature*)")) } >>> { >>> >> echo " -> exists" } { $workspace cf2d "(name=cdfeature)" >>> >> "(name=cddistribution)" "1" "1"; echo " -> created" } >>> >> >>> >> echo "Ensuring a2f association" >>> >> if { (coll:first ($workspace la2f >>> >> "(rightEndpoint=*name=cdfeature*)")) >>> } { >>> >> echo " -> exists" } { $workspace ca2f "(cdartifact=true)" >>> >> "(name=cdfeature)" "10000" "1"; echo " -> created" } >>> >> >>> >> echo "Ensuring d2t association" >>> >> if { (coll:first ($workspace ld2t >>> "(leftEndpoint=*name=cddistribution*)")) >>> >> } { echo " -> exists" } \ >>> >> { $workspace cd2t "(name=cddistribution)" "(id=*)" "1" "1000"; echo " >>> -> >>> >> created" } >>> >> >>> >> echo "Ensuring autoconf artifact" >>> >> if { (coll:first ($workspace lrp)) } { echo " -> exists" } \ { >>> >> identity = $autoconf getIdentity >>> >> version = $autoconf getVersion >>> >> name = "$identity - $version" >>> >> url = $autoconf getUrl >>> >> mimetype = $autoconf getMimetype >>> >> $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype" >>> >> Bundle-SymbolicName="$identity" Bundle-Version="$version" >>> >> Deployment-ProvidesResourceProcessor="org.osgi.deployment.rp.autoconf" ] >>> >> echo " -> created" >>> >> } >>> >> >>> >> echo "Purging old artifacts" >>> >> artifacts = ($workspace la) >>> >> each $artifacts { >>> >> $workspace da $it >>> >> } >>> >> >>> >> echo "Creating new artifacts" >>> >> each $deployed { >>> >> identity = $it getIdentity >>> >> version = $it getVersion >>> >> name = "$identity - $version" >>> >> url = $it getUrl >>> >> mimetype = $it getMimetype >>> >> if { $mimetype equals "application/xml:osgi-autoconf" } \ >>> >> { $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype" >>> >> filename="$name" processorPid="org.osgi.deployment.rp.autoconf" ] >>> >> [ cdartifact="true" ] } \ >>> >> { $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype" >>> >> Bundle-SymbolicName="$identity" Bundle-Version="$version" ] [ >>> >> cdartifact="true"] } } >>> >> >>> >> echo "Ensuring approval on target" >>> >> if { $target needsApprove } { echo " approving"; $target approve } >>> >> { >>> echo >>> >> " no changes" } >>> >> >>> >> echo "Committing workspace" >>> >> $workspace commit >>> >> >>> >> # Let events settle >>> >> misc:sleep 2000 >>> >> ace:rw $workspace >>> >> >>> >> # Let events settle >>> >> misc:shutdown 2000 >>> >> >>> >> >>> >> Greetings, Marcel >>> >> >>> >> >>>
