Re: Incrementals and ATH tests (inside a plugin CI)

2020-08-28 Thread Ullrich Hafner 
> Am 28.08.2020 um 17:21 schrieb Jesse Glick :
> 
> On Thu, Aug 27, 2020 at 5:22 PM Ullrich Hafner  
> wrote:
>> Do we have something existing that can extract the list of dependencies for 
>> a plugin…
> 
> I think it would suffice to inject a new implementation of
> `UpdateCenterMetadataDecorator` using
> `PluginMetadata.LocalOverride.create` based on
> `target/{,test-classes/test-dependencies/}*.hpi`. Or simply use the
> existing implementation and define the environment variable
> `LOCAL_JARS` in your launcher script.
> 

This is a beautiful solution! Thank you for the tip: I now use the following 
properties:

  ../plugin/target/test-classes/test-dependencies
  ${jenkins.version}
  firefox-container

(And before the UI tests start it is required to copy the hpi under test into 
the test-dependencies folder as well) 

Now everything is green (or blue) again! This also improves the runtime of the 
tests (now about 2min per test):
https://ci.jenkins.io/job/Plugins/job/warnings-ng-plugin/job/PR-595/lastCompletedBuild/testReport/io.jenkins.plugins.analysis.warnings/SmokeTests/
 


Old variant (about 4min per test):
https://ci.jenkins.io/job/Plugins/job/warnings-ng-plugin/job/master/988/testReport/io.jenkins.plugins.analysis.warnings/SmokeTests/
 



> Plugins not mentioned in this list will still be retrieved according
> to versions in the online UC (though by default using Maven to
> retrieve local copies of the actual binaries). You could inject an
> alternate `UpdateCenterMetadataProvider` to completely override that,
> so that only dependencies in the test classpath are offered; this
> would match the behavior of `JenkinsRule` (other than when annotated
> by `@WithPlugin`).
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr32vCiP5eJdqpq%2BUc1dATxmf%3DzBRBZkGLx%3Dt3%3DwvdpAKg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/34A7CDC5-0967-44F1-8A62-10F01D18DF50%40gmail.com.

Re: Incrementals and ATH tests (inside a plugin CI)

2020-08-28 Thread Jesse Glick 
On Thu, Aug 27, 2020 at 5:22 PM Ullrich Hafner  wrote:
> Do we have something existing that can extract the list of dependencies for a 
> plugin…

I think it would suffice to inject a new implementation of
`UpdateCenterMetadataDecorator` using
`PluginMetadata.LocalOverride.create` based on
`target/{,test-classes/test-dependencies/}*.hpi`. Or simply use the
existing implementation and define the environment variable
`LOCAL_JARS` in your launcher script.

Plugins not mentioned in this list will still be retrieved according
to versions in the online UC (though by default using Maven to
retrieve local copies of the actual binaries). You could inject an
alternate `UpdateCenterMetadataProvider` to completely override that,
so that only dependencies in the test classpath are offered; this
would match the behavior of `JenkinsRule` (other than when annotated
by `@WithPlugin`).

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr32vCiP5eJdqpq%2BUc1dATxmf%3DzBRBZkGLx%3Dt3%3DwvdpAKg%40mail.gmail.com.

Re: Incrementals and ATH tests (inside a plugin CI)

2020-08-28 Thread Ullrich Hafner 

> Am 28.08.2020 um 07:52 schrieb Daniel Beck :
> 
> 
> On Thu, Aug 27, 2020 at 11:22 PM Ullrich Hafner  > wrote:
> 
> Yes, I had that problem a couple of weeks ago. One of my dependencies 
> upgraded its Jenkins version to a higher version than the version I was using 
> in my UI tests: that broke my build.
> 
> Do you still have the failing build log, or know when exactly this happened?
> 
> There was a bug with the update center tiers I introduced a few months ago 
> that made the tiers not work quite correctly ( 
> https://github.com/jenkins-infra/update-center2/pull/406 
>  ) but I fixed that 
> four weeks ago, so for that time it should have been working without problems.
> 

Hmm, I don’t know the exact time and those builds are already deleted. But it 
was around 4 weeks ago as well. So maybe this was actually caused by the bug 
you are describing… 

> Otherwise, the ATH (WithPlugins) should download plugins using the plugin 
> manager, and that should only install compatible dependencies.

Since no builds fail anymore now I think ATH picks the right versions anyway. 

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/BFB991AC-3294-4107-B453-4400559F2C8A%40gmail.com.

Re: Incrementals and ATH tests (inside a plugin CI)

2020-08-27 Thread Daniel Beck 
On Thu, Aug 27, 2020 at 11:22 PM Ullrich Hafner 
wrote:

>
> Yes, I had that problem a couple of weeks ago. One of my dependencies
> upgraded its Jenkins version to a higher version than the version I was
> using in my UI tests: that broke my build.
>

Do you still have the failing build log, or know when exactly this happened?

There was a bug with the update center tiers I introduced a few months ago
that made the tiers not work quite correctly (
https://github.com/jenkins-infra/update-center2/pull/406 ) but I fixed that
four weeks ago, so for that time it should have been working without
problems.

Otherwise, the ATH (WithPlugins) should download plugins using the plugin
manager, and that should only install compatible dependencies.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAMo7PtLr_H3Hv%3DFHHka4qLErackYfZ27yRdLKKa7msg4QxKyyg%40mail.gmail.com.

Re: Incrementals and ATH tests (inside a plugin CI)

2020-08-27 Thread Ullrich Hafner 
> 
> I suspect this is just the most visible special case of a general
> problem that the ATH tests are using the latest release at the time of
> the test run, regardless of what versions your POM asks to test
> against. This is a bad thing IMO: as one example, it means tests can
> start failing in a PR which did not change sources, leading to
> confusion.

Yes, I had that problem a couple of weeks ago. One of my dependencies upgraded 
its Jenkins version to a higher version than the version I was using in my UI 
tests: that broke my build.

A possible workaround would be to use our plugin-installation-manager to 
download all dependencies in the correct version before the ATH container 
starts the Jenkins instance (the ATH already can load a set of pre-configured 
plugins). That would also have the benefit, that the startup time when running 
several ATH tests will be reduced as well.

Do we have something existing that can extract the list of dependencies for a 
plugin so that the plugin-installation-manager can be feeded with that input?

> Verifying compatibility with recent upstream changes is of
> course valuable but this can be accomplished more deterministically
> with Dependabot.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3qetGKx9g1Rz6xLWL%2BB9SSeWeZm-8SR79HgGMXTyUJ-w%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/D7972A1D-2AC9-44C3-91F3-545FE9A7DCFA%40gmail.com.

Re: Incrementals and ATH tests (inside a plugin CI)

2020-08-27 Thread Jesse Glick 
On Thu, Aug 27, 2020 at 3:07 PM Ullrich Hafner  wrote:
> Incrementals are resolved fine in integration tests but in UI tests the 
> official update center is used

I suspect this is just the most visible special case of a general
problem that the ATH tests are using the latest release at the time of
the test run, regardless of what versions your POM asks to test
against. This is a bad thing IMO: as one example, it means tests can
start failing in a PR which did not change sources, leading to
confusion. Verifying compatibility with recent upstream changes is of
course valuable but this can be accomplished more deterministically
with Dependabot.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3qetGKx9g1Rz6xLWL%2BB9SSeWeZm-8SR79HgGMXTyUJ-w%40mail.gmail.com.

Re: Incrementals and ATH tests (inside a plugin CI)

2020-08-27 Thread Oliver Gondža 

I am not aware that any work have been done to support this.

On 27/08/2020 21.06, Ullrich Hafner wrote:
My warnings plugin is using in our Jenkins CI build also UI tests (using 
the ATH container). Is there a way to start these UI tests (using the 
ATH container) so that depending plugins that have an incremental 
version are also resolved? Incrementals are resolved fine in integration 
tests but in UI tests the official update center is used which does not 
contain the incremental versions.


See: 
https://ci.jenkins.io/job/Plugins/job/warnings-ng-plugin/job/PR-595/5/testReport/io.jenkins.plugins.analysis.warnings/SmokeTests/shouldRecordIssuesInPipelineAndExpandTokens/


java.io.IOException: Failed to load: Warnings Next Generation Plugin 
(8.5.0-SNAPSHOT (private-2e85864e-ubuntu))
master49529| - Plugin is missing: forensics-api (0.8.0-rc587.65fc61fe9be2)


--
You received this message because you are subscribed to the Google 
Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to jenkinsci-dev+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/10362FBD-68D4-4ADD-BEA9-267CB38E6EED%40gmail.com 
.



--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/aca6a70f-861b-9723-bdf2-d034b4dfc452%40gmail.com.

Incrementals and ATH tests (inside a plugin CI)

2020-08-27 Thread Ullrich Hafner 
My warnings plugin is using in our Jenkins CI build also UI tests (using the 
ATH container). Is there a way to start these UI tests (using the ATH 
container) so that depending plugins that have an incremental version are also 
resolved? Incrementals are resolved fine in integration tests but in UI tests 
the official update center is used which does not contain the incremental 
versions.

See: 
https://ci.jenkins.io/job/Plugins/job/warnings-ng-plugin/job/PR-595/5/testReport/io.jenkins.plugins.analysis.warnings/SmokeTests/shouldRecordIssuesInPipelineAndExpandTokens/
 


java.io.IOException: Failed to load: Warnings Next Generation Plugin 
(8.5.0-SNAPSHOT (private-2e85864e-ubuntu))
master49529| - Plugin is missing: forensics-api (0.8.0-rc587.65fc61fe9be2)

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/10362FBD-68D4-4ADD-BEA9-267CB38E6EED%40gmail.com.