Thanks again for your advice. With your help, I have successfully updated my
build process to provision all IUs at once, which is indeed significantly
faster: the full build took 26 minutes instead of over an hour.
Provisioning for the main target platform took a little below 7 minutes:
[p2.dir] Operation completed in 413085 ms.
And the test target platform took about the same time to provision:
[p2.dir] Operation completed in 402634 ms.
It strikes me as odd though that the second provisioning takes the same time
as the first, if it re-uses the first target platform as a source.
I have the following error message in my log when building the test target
platform:
[exec] [p2.dir] !MESSAGE No repository found at
file:/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace/build/N200912100748/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile/.
Does it mean the first target platform couldn't be re-used as a source for
provisioning the test target platform? If so, would you have an idea why?
On Thu, Dec 10, 2009 at 6:02 AM, Nick Boldt <nickbo...@gmail.com> wrote:
> Sadly, at this point it's trial an error... wait for the build fail with a
> message like
>
> Bundle org.foo_0.0.0 failed to resolve.: Missing required plug-in
> org.bar_0.0.0.
>
> And then go looking for the plugin or feature to add to the list of IUs you
> need. Chances are good you can just add the plugin and the Galileo repo and
> you're done... but in some cases, you have to dig deeper - especially if you
> want to be able to run the build offline (eg., using repo zips).
>
> Or use PDE (or the PDE viz plugin) to show the plugins you depend on. Then
> you just need to map those plugins to their containing projects (eg. EMF,
> GEF, WebTools WST / JST, XSD)
>
> What I do is grab an update site zip, unpack it, then iterate through the
> feature jars, unpacking them. From there you can iterate through the
> unjarred features' feature.xml files looking for plugin names that match the
> one you need. For example:
>
> unzip -d wst-buildrepo-R-3.1.1-20090917225226.zip{_,}
> cd wst-buildrepo-R-3.1.1-20090917225226.zip_/features/
> for f in $(find . -maxdepth 1 -type f -name "*.jar"); do unzip -d ${f}{_,};
> done
> find.sh . feature.xml wst.sse
> find.sh . feature.xml org.eclipse.wst.ws_ui.feature
>
> Note that find.sh is a wrapper for find + grep which I wrote (WTFPL
> license), and is available here [1].)
>
> [1]http://divbyzero.com/linux/find.in.files.sh.txt
>
> I'm looking for a better, more discovery-based approach (eg., using
> Buckminster or Tycho) which would be able to crawl though the sources and
> discover what features/plugins are needed, then use that generated list as
> input to the IUsToInstall property (if not otherwise set). If you have any
> ideas, I'm open to 'em.
>
> N
>
> Nicolas Bros wrote:
>
>> Wow! That looks useful.
>> But how do you get the names of the IUs you must put in IUsToInstall?
>>
>> On Wed, Dec 9, 2009 at 8:31 PM, Nick Boldt <nickbo...@gmail.com <mailto:
>> nickbo...@gmail.com>> wrote:
>>
>> If you list the IUsToInstall separated by "+" instead of "," they're
>> all installed in a single step.
>>
>> Requires Eclipse 3.5+ as the base, though, not 3.4.
>>
>> Example from a recent build:
>>
>> [exec] [p2.dir] Command-line arguments: -application
>> org.eclipse.equinox.p2.director -consoleLog -flavor tooling -roaming
>> -profile SDKProfile -installIU org.eclipse.sdk.feature.group
>> -installIU org.eclipse.sdk.ide -installIU org.eclipse.core.net
>> <http://org.eclipse.core.net> -installIU org.eclipse.equinox.common
>>
>> -installIU org.eclipse.core.runtime -installIU
>> org.eclipse.debug.core -installIU org.eclipse.rcp.feature.group
>> -installIU org.eclipse.jst.server.generic.core -installIU
>> org.eclipse.wst.ws_core.feature.feature.group -installIU
>> org.eclipse.wst.web_ui.feature.feature.group -installIU
>> org.eclipse.wst.ws_wsdl15.feature.feature.group -installIU
>> org.eclipse.wst.xml_ui.feature.feature.group -installIU
>> org.eclipse.wst.common_ui.feature.feature.group -installIU
>> org.eclipse.wst.common_core.feature.feature.group -installIU
>> org.eclipse.xsd.feature.group -installIU
>> org.eclipse.gef.feature.group -installIU
>> org.eclipse.gmf.runtime.diagram.ui -installIU
>> org.eclipse.emf.compare.match -installIU
>> org.eclipse.emf.compare.diff -installIU org.eclipse.emf.compare.ui
>> -installIU org.mozilla.xpcom -installIU org.jboss.tools.vpe.resref
>> -installIU org.jboss.tools.jst.web.ui -installIU
>> org.jboss.ide.eclipse.as.core -installIU
>> org.jboss.ide.eclipse.archives.webtools -installIU
>> org.jboss.tools.jmx.feature.feature.group -destination
>>
>>
>> /home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/testing/target/eclipse
>> -bundlepool
>>
>>
>> /home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/testing/target/eclipse
>> -metadataRepository
>>
>>
>> file:/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile,
>> http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/
>> ,
>> -artifactRepository
>>
>>
>> file:/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile,
>> http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/
>> ,
>> -profileProperties org.eclipse.update.install.features=true -os
>> linux -ws gtk -arch x86
>>
>> As run from this releng project [1]:
>>
>> [1]https://anonsvn.jboss.org/repos/jbosstools/trunk/bpel/releng/
>>
>> Which simply says:
>>
>> repositoryURLs=\
>> http://download.jboss.org/jbosstools/updates/nightly/trunk/,\
>> http://download.eclipse.org/releases/galileo/
>>
>> IUsToInstall=org.eclipse.sdk.feature.group+org.eclipse.sdk.ide+
>> org.eclipse.core.net
>> <http://org.eclipse.core.net
>> >+org.eclipse.equinox.common+org.eclipse.core.runtime+org.eclipse.debug.core+org.eclipse.rcp.feature.group+\
>>
>> org.eclipse.jst.server.generic.core+\
>>
>>
>> org.eclipse.wst.ws_core.feature.feature.group+org.eclipse.wst.web_ui.feature.feature.group+org.eclipse.wst.ws_wsdl15.feature.feature.group+\
>>
>>
>> org.eclipse.wst.xml_ui.feature.feature.group+org.eclipse.wst.common_ui.feature.feature.group+org.eclipse.wst.common_core.feature.feature.group+\
>> org.eclipse.xsd.feature.group+org.eclipse.gef.feature.group+\
>> org.eclipse.gmf.runtime.diagram.ui+\
>>
>>
>> org.eclipse.emf.compare.match+org.eclipse.emf.compare.diff+org.eclipse.emf.compare.ui+\
>>
>>
>> org.mozilla.xpcom+org.jboss.tools.vpe.resref+org.jboss.tools.jst.web.ui+org.jboss.ide.eclipse.as.core+org.jboss.ide.eclipse.archives.webtools+org.jboss.tools.jmx.feature.feature.group
>>
>> In this case, even pulling from the Galileo and JBT nightly update
>> sites, the provisioning step only took 4 minutes (248411 ms).
>>
>> If you source from locally cached p2 repo zips such as these [2],
>> it's even faster.
>>
>> [2]http://download.eclipse.org/athena/repos/
>>
>> Nick
>>
>>
>> David Carver wrote:
>>
>> There are ways to do this. And you might be able to do it in
>> your own customTargets.xml file. Basically you would do a two
>> stage build, either all in one Job or in two separate jobs. The
>> first one would be to build and create the target platform and
>> provision it. The second would be to create the necessary
>> tests, and run them based on the input from the first provisioning.
>>
>> I've done this in the past by using a combination of the ZIP
>> files, combining them into a working target platform, and then
>> deploying the tests and running them. I do agree that the P2
>> provisioning process seems to take way longer to get setup than
>> just a straight ZIP provisioning.
>>
>> The main issue I see right now is that with the P2 director
>> steps, it is done in a loop, instead of being able to provide a
>> list of all the items to be provisioned in one provisioning
>> step. Each plugin or feature is provisioned separately. If
>> we could find a way to provide a list of items to be provisioned
>> and have it done in one provisioning step (i.e. not executing P2
>> director multiple times), I think that alone would go a long way
>> to speeding up the build.
>>
>> Also, you may want to look at an alternative cleanUp step if you
>> have a large directory that is created during the build with
>> many sub directories. I've found that trying to avoid wildcard
>> searches and deletions with FileSets greatly speeds up the
>> deletion and cleanup process.
>>
>> Nicolas Bros wrote:
>>
>> In fact, I was not only thinking of re-using the build
>> target platform for testing, but also re-using a previously
>> snapshotted build target platform instead of provisioning a
>> new one for each build. My idea was to be able to skip all
>> the work done in the "postSetup" target in
>> customTargets.xml, if nothing changed in the build
>> configuration since the last build. Would that be doable ?
>>
>> On Wed, Dec 9, 2009 at 3:24 AM, Nick Boldt
>> <nickbo...@gmail.com <mailto:nickbo...@gmail.com>
>> <mailto:nickbo...@gmail.com <mailto:nickbo...@gmail.com>>>
>>
>> wrote:
>>
>> The previously provisioned target used to build is reused
>> when
>> creating the test target - it's added to the list of
>> repos from
>> which IUs are installed.
>>
>> This means (in theory) that instead of refetching from
>> Galileo,
>> IUs will simply be copied from the local disk. So the
>> only thing
>> that slows the process down is file i/o.
>>
>> Yes, we could even go a step further to use the existing
>> eclipse
>> instead of creating a fresh, clean one, but that means a
>> "dirty"
>> test, and I don't like that approach. Having this
>> behaviour be
>> optional is certainly doable, but I'm sure people like
>> Dave C will
>> object violently to the idea of setting up tests w/ chunks
>> of
>> leftover compilation artifacts in them, instead of a clean
>> "provision the target, install the built runtime, install
>> the
>> tests, and run them" process.
>>
>> It may be slower, but it's a more valid test IMO.
>>
>> If you want an optional override to reuse the build
>> environment
>> instead of creating a new one, please open a bug and
>> propose the
>> override property name, eg.,
>> "useDirtyBuildEnvironmentForTesting" :)
>>
>> Nicolas Bros wrote:
>>
>> Hi,
>>
>> I'd like my build to be faster. I have noticed that
>> Athena
>> installs a whole new Eclipse with all the dependencies
>> specified in the build.properties twice for each
>> build (once
>> for the build and a second time for tests). That
>> obviously
>> takes quite some time.
>> Would it be possible to save the Eclipse install once
>> the
>> dependencies are deemed stable, and re-use it for future
>> builds? There could be a property in the
>> build.properties file
>> that indicates the path of an existing Eclipse
>> install to use
>> for the build for example.
>> If not, what would prevent this?
>> -- Nicolas Bros
>> R&D
>> tel: 06 75 09 19 88
>> nb...@mia-software.com
>> <mailto:nb...@mia-software.com>
>> <mailto:nb...@mia-software.com <mailto:nb...@mia-software.com
>> >>
>> <mailto:nb...@mia-software.com
>> <mailto:nb...@mia-software.com>
>> <mailto:nb...@mia-software.com <mailto:nb...@mia-software.com
>> >>>
>> nbros....@gmail.com <mailto:nbros....@gmail.com>
>> <mailto:nbros....@gmail.com <mailto:nbros....@gmail.com>>
>> <mailto:nbros....@gmail.com
>> <mailto:nbros....@gmail.com> <mailto:nbros....@gmail.com
>> <mailto:nbros....@gmail.com>>>
>>
>> Mia-Software, 410 clos de la Courtine
>> 93160 Noisy-le-Grand
>> http://www.mia-software.com
>> .: model driven agility :.
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> dash-dev mailing list
>> dash-dev@eclipse.org <mailto:dash-dev@eclipse.org>
>> <mailto:dash-dev@eclipse.org <mailto:dash-dev@eclipse.org>>
>>
>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>>
>> -- Nick Boldt :: http://nick.divbyzero.com
>> Release Engineer :: Eclipse Modeling & Dash Athena
>> _______________________________________________
>> dash-dev mailing list
>> dash-dev@eclipse.org <mailto:dash-dev@eclipse.org>
>> <mailto:dash-dev@eclipse.org <mailto:dash-dev@eclipse.org>>
>>
>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>>
>>
>>
>> -- Nicolas Bros
>> R&D
>> tel: 06 75 09 19 88
>> nb...@mia-software.com <mailto:nb...@mia-software.com>
>> <mailto:nb...@mia-software.com <mailto:nb...@mia-software.com
>> >>
>> nbros....@gmail.com <mailto:nbros....@gmail.com>
>> <mailto:nbros....@gmail.com <mailto:nbros....@gmail.com>>
>> Mia-Software, 410 clos de la Courtine
>> 93160 Noisy-le-Grand
>> http://www.mia-software.com
>> .: model driven agility :.
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> dash-dev mailing list
>> dash-dev@eclipse.org <mailto:dash-dev@eclipse.org>
>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>>
>> _______________________________________________
>> dash-dev mailing list
>> dash-dev@eclipse.org <mailto:dash-dev@eclipse.org>
>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>>
>> -- Nick Boldt :: http://nick.divbyzero.com
>> Release Engineer :: Eclipse Modeling & Dash Athena
>> _______________________________________________
>> dash-dev mailing list
>> dash-dev@eclipse.org <mailto:dash-dev@eclipse.org>
>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>>
>>
>>
>> --
>> Nicolas Bros
>> R&D
>> tel: 06 75 09 19 88
>> nb...@mia-software.com <mailto:nb...@mia-software.com>
>> nbros....@gmail.com <mailto:nbros....@gmail.com>
>> Mia-Software, 410 clos de la Courtine
>> 93160 Noisy-le-Grand
>> http://www.mia-software.com
>> .: model driven agility :.
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> dash-dev mailing list
>> dash-dev@eclipse.org
>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>
> --
> Nick Boldt :: http://nick.divbyzero.com
> Release Engineer :: Eclipse Modeling & Dash Athena
> _______________________________________________
> dash-dev mailing list
> dash-dev@eclipse.org
> https://dev.eclipse.org/mailman/listinfo/dash-dev
>
--
Nicolas Bros
R&D
tel: 06 75 09 19 88
nb...@mia-software.com
nbros....@gmail.com
Mia-Software, 410 clos de la Courtine
93160 Noisy-le-Grand
http://www.mia-software.com
.: model driven agility :.
_______________________________________________
dash-dev mailing list
dash-dev@eclipse.org
https://dev.eclipse.org/mailman/listinfo/dash-dev
