Re: Launchpad stable and launchpad unstable

[Sandro Boehme]

Hi Bertrand,

Am 29.11.16 um 13:25 schrieb Bertrand Delacretaz:

Hi Sandro,

On Mon, Nov 28, 2016 at 11:45 PM, Sandro Boehme  wrote:

...If I checkout and build the latest trunk I would expect that the built
projects are available at the runtime of the server. This way I can set
breakpoints in the debugger, examine the inner workings and maybe provide an
improvement


Yes, as a Sling developer that's what you want, and also for testing
the whole thing in automated tests.

OTOH for Sling to be always release ready we need a launchpad that
refers only to released modules, so we need both.

I agree that we need both!


...If the launchpad provides only stable versions I would always first have to
switch to the snapshot versions using the script and I currently I can't do
that, right?...


The launchpad/builder/set-sling-snapshots.sh script should allow you
to do that but when I wrote it it didn't fully work, see comments
inside.

-Bertrand



I tried it with the current trunk and http://localhost:8080/ returned a 
503 with 'AuthenticationSupport service missing. Cannot authenticate 
request.'



'mvn dependency:resolve | grep org.apache.sling | grep -v SNAPSHOT' 
returned:


[INFO] 
org.apache.sling:org.apache.sling.commons.scheduler:jar:2.5.2:provided
[INFO] 
org.apache.sling:org.apache.sling.repoinit.parser:jar:1.1.0:provided
[INFO] 
org.apache.sling:org.apache.sling.commons.classloader:jar:1.3.6:provided

[INFO]org.apache.sling:org.apache.sling.xss:jar:1.0.16:provided
[INFO] 
org.apache.sling:org.apache.sling.provisioning.model:jar:1.7.0:provided

[INFO]org.apache.sling:org.apache.sling.commons.mime:jar:2.1.10:provided
[INFO]org.apache.sling:org.apache.sling.event.dea:jar:1.1.0:provided
[INFO]org.apache.sling:org.apache.sling.auth.core:jar:1.3.22:provided

When I look in the /logs/error.log after startup I see exceptions like
'java.lang.NoClassDefFoundError: javax/jcr/RepositoryException'

I feel like I'm always just complaining but I'm afraid it would take way 
too much time to understand how the launchpad translates the txt files 
into installed Maven artifacts. Let alone fixing something there. OTOH I 
think the aspect I've mentioned in my last email is not unimportant.


Best,

Sandro

Re: Launchpad stable and launchpad unstable

[Bertrand Delacretaz]

Hi Sandro,

On Mon, Nov 28, 2016 at 11:45 PM, Sandro Boehme  wrote:
> ...If I checkout and build the latest trunk I would expect that the built
> projects are available at the runtime of the server. This way I can set
> breakpoints in the debugger, examine the inner workings and maybe provide an
> improvement

Yes, as a Sling developer that's what you want, and also for testing
the whole thing in automated tests.

OTOH for Sling to be always release ready we need a launchpad that
refers only to released modules, so we need both.

> ...If the launchpad provides only stable versions I would always first have to
> switch to the snapshot versions using the script and I currently I can't do
> that, right?...

The launchpad/builder/set-sling-snapshots.sh script should allow you
to do that but when I wrote it it didn't fully work, see comments
inside.

-Bertrand

Re: Launchpad stable and launchpad unstable

[Sandro Boehme]

- the launchpad maintained in trunk should always be the "stable" launchpad, 
referencing only released dependencies of the sling bundles and oft the integration tests 
project
If I checkout and build the latest trunk I would expect that the built 
projects are available at the runtime of the server. This way I can set 
breakpoints in the debugger, examine the inner workings and maybe 
provide an improvement.
If the launchpad provides only stable versions I would always first have 
to switch to the snapshot versions using the script and I currently I 
can't do that, right?


Best,

Sandro

PS: Thanks Bertrand for pointing me to this discussion!

Re: Launchpad stable and launchpad unstable

[Robert Munteanu]

On Wed, 2016-11-02 at 12:02 +0100, Carsten Ziegeler wrote:
> Bertrand Delacretaz wrote
> > On Wed, Nov 2, 2016 at 9:39 AM, Carsten Ziegeler  > org> wrote:
> > > Bertrand Delacretaz wrote
> > > > On Wed, Nov 2, 2016 at 6:56 AM, Carsten Ziegeler  > > > che.org> wrote:
> > > > > ...what problem are we exactly trying to solve?
> > > > 
> > > > ...As described in the original post here - having a stable
> > > > launchpad
> > > > that's always ready to release, while also continuously testing
> > > > the
> > > > integration of the latest snapshots.
> > > > 
> > > 
> > > So if the integration tests always test the latest snapshots then
> > > there
> > > is nothing we have do to, right?...
> > 
> > We need to test the launchpad that we release, don't we?
> 
> Yeah, sorry that I'm a little bit picky here. All I'm trying that we
> get
> a description of the full picture.
> 
> We want to test the *same* integration tests against the current
> launchpad (which has no snapshot deps) and against the same launchpad
> but replacing all sling deps to the latest snapshot versions.
> And that might lead us into the situation you describe below.
> 
> > 
> > Depending on the combination of bundles that release might have
> > different behavior than combining the latest snapshots, although
> > maybe
> > it's not that likely.
> 
> Right, I would argue that it is not very likely.
> 
> > 
> > As a first step we might manually run the integration tests against
> > the launchpad that's about to be released, that's better than
> > nothing
> > but some tests will fail and we need a manual analysis of that -
> > which
> > is not impossible if the goal is to release say every 3 months.
> 
> Instead of coming up with a gigantic approach that will not be used
> or
> only once every two years, I think lets deal with it once we're
> there.
> Something simple like an exclusion list activated by the same
> mechanism
> that switches to snapshot dependencies.

I've tried to summarise the current state of the discussions at 

  https://cwiki.apache.org/confluence/display/SLING/Moving+the+Sling+La
unchpad+to+use+released+artifacts+only

I think currently we need to decide/finalise:

- how to transform a 'stable' launchpad into an 'unstable' one
- how to exclude tests from running on an 'stable' launchpad but keep
the running on an 'unstable' launchpad.

Comment on the wiki page more than welcome, I have probably missed
something :-)

Thanks,

Robert 
> 
>  Carsten
> 

Re: Launchpad stable and launchpad unstable

[Carsten Ziegeler]

Bertrand Delacretaz wrote
> On Wed, Nov 2, 2016 at 9:39 AM, Carsten Ziegeler  wrote:
>> Bertrand Delacretaz wrote
>>> On Wed, Nov 2, 2016 at 6:56 AM, Carsten Ziegeler  
>>> wrote:
 ...what problem are we exactly trying to solve?
>>> ...As described in the original post here - having a stable launchpad
>>> that's always ready to release, while also continuously testing the
>>> integration of the latest snapshots.
>>>
>> So if the integration tests always test the latest snapshots then there
>> is nothing we have do to, right?...
> 
> We need to test the launchpad that we release, don't we?

Yeah, sorry that I'm a little bit picky here. All I'm trying that we get
a description of the full picture.

We want to test the *same* integration tests against the current
launchpad (which has no snapshot deps) and against the same launchpad
but replacing all sling deps to the latest snapshot versions.
And that might lead us into the situation you describe below.

> 
> Depending on the combination of bundles that release might have
> different behavior than combining the latest snapshots, although maybe
> it's not that likely.

Right, I would argue that it is not very likely.

> 
> As a first step we might manually run the integration tests against
> the launchpad that's about to be released, that's better than nothing
> but some tests will fail and we need a manual analysis of that - which
> is not impossible if the goal is to release say every 3 months.

Instead of coming up with a gigantic approach that will not be used or
only once every two years, I think lets deal with it once we're there.
Something simple like an exclusion list activated by the same mechanism
that switches to snapshot dependencies.

 Carsten

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Launchpad stable and launchpad unstable

[Bertrand Delacretaz]

On Wed, Nov 2, 2016 at 9:39 AM, Carsten Ziegeler  wrote:
> Bertrand Delacretaz wrote
>> On Wed, Nov 2, 2016 at 6:56 AM, Carsten Ziegeler  
>> wrote:
>>> ...what problem are we exactly trying to solve?
>> ...As described in the original post here - having a stable launchpad
>> that's always ready to release, while also continuously testing the
>> integration of the latest snapshots.
>>
> So if the integration tests always test the latest snapshots then there
> is nothing we have do to, right?...

We need to test the launchpad that we release, don't we?

Depending on the combination of bundles that release might have
different behavior than combining the latest snapshots, although maybe
it's not that likely.

As a first step we might manually run the integration tests against
the launchpad that's about to be released, that's better than nothing
but some tests will fail and we need a manual analysis of that - which
is not impossible if the goal is to release say every 3 months.

-Bertrand

Re: Launchpad stable and launchpad unstable

[Carsten Ziegeler]

Bertrand Delacretaz wrote
> On Wed, Nov 2, 2016 at 6:56 AM, Carsten Ziegeler  wrote:
>> ...what problem are we exactly trying to solve?
> 
> As described in the original post here - having a stable launchpad
> that's always ready to release, while also continuously testing the
> integration of the latest snapshots.
> 
So if the integration tests always test the latest snapshots then there
is nothing we have do to, right?

 Carsten

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Launchpad stable and launchpad unstable

[Bertrand Delacretaz]

On Wed, Nov 2, 2016 at 6:56 AM, Carsten Ziegeler  wrote:
> ...what problem are we exactly trying to solve?

As described in the original post here - having a stable launchpad
that's always ready to release, while also continuously testing the
integration of the latest snapshots.

-Bertrand

Re: Launchpad stable and launchpad unstable

[Carsten Ziegeler]

Maybe I'm missing something, but what problem are we exactly trying to
solve?

Carsten


Robert Munteanu wrote
> On Mon, 2016-10-31 at 11:17 +0100, Carsten Ziegeler wrote:
>> Bertrand Delacretaz wrote
>>> On Wed, Oct 26, 2016 at 9:08 AM, Robert Munteanu >> g> wrote:
 ...How would we handle launchpad tests that rely on functionality
 introduced in a SNAPSHOT dependency?...
>>>
>>> I suppose we'll need an annotation to indicate the requirements for
>>> a
>>> specific test.
>>>
>>
>> Isn't that a little bit complicated? Who removes the annotation when
>> things are released?
> 
> How about an annotation or an assumption mechanism which looks for a
> specific bundle + version to be present? That would not need further
> maintenance and can be removed at any time, should we want to remove
> it:
> 
>   @NeedsBundle("org.apache.sling.commons.json", "2.1.0")
>   @Test
>   public void validateSomething() { ... }
> 
> or
> 
>   @Test
>   public void validateSomething() {
> assumeThat(bundleIsPresent("org.apache.sling.commons.json",
> "2.1.0"));
> ...
>   }
> 
>>
>> In general, can we come up with a full proposal instead of discussing
>> individual single points? I suggest we setup a wiki where we outline
>> the
>> full process and then can fill in the details
> 
> I can come up with a full proposal but there are some things which are
> not totally clear to me, so that's why I'm asking :-)
> 
> Robert
> 


 

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Launchpad stable and launchpad unstable

[Robert Munteanu]

On Mon, 2016-10-31 at 11:17 +0100, Carsten Ziegeler wrote:
> Bertrand Delacretaz wrote
> > On Wed, Oct 26, 2016 at 9:08 AM, Robert Munteanu  > g> wrote:
> > > ...How would we handle launchpad tests that rely on functionality
> > > introduced in a SNAPSHOT dependency?...
> > 
> > I suppose we'll need an annotation to indicate the requirements for
> > a
> > specific test.
> > 
> 
> Isn't that a little bit complicated? Who removes the annotation when
> things are released?

How about an annotation or an assumption mechanism which looks for a
specific bundle + version to be present? That would not need further
maintenance and can be removed at any time, should we want to remove
it:

  @NeedsBundle("org.apache.sling.commons.json", "2.1.0")
  @Test
  public void validateSomething() { ... }

or

  @Test
  public void validateSomething() {
assumeThat(bundleIsPresent("org.apache.sling.commons.json",
"2.1.0"));
...
  }

> 
> In general, can we come up with a full proposal instead of discussing
> individual single points? I suggest we setup a wiki where we outline
> the
> full process and then can fill in the details

I can come up with a full proposal but there are some things which are
not totally clear to me, so that's why I'm asking :-)

Robert

Re: Launchpad stable and launchpad unstable

[Carsten Ziegeler]

Bertrand Delacretaz wrote
> On Wed, Oct 26, 2016 at 9:08 AM, Robert Munteanu  wrote:
>> ...How would we handle launchpad tests that rely on functionality
>> introduced in a SNAPSHOT dependency?...
> 
> I suppose we'll need an annotation to indicate the requirements for a
> specific test.
> 
Isn't that a little bit complicated? Who removes the annotation when
things are released?

In general, can we come up with a full proposal instead of discussing
individual single points? I suggest we setup a wiki where we outline the
full process and then can fill in the details

 Carsten

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Launchpad stable and launchpad unstable

[Bertrand Delacretaz]

On Wed, Oct 26, 2016 at 9:08 AM, Robert Munteanu  wrote:
> ...How would we handle launchpad tests that rely on functionality
> introduced in a SNAPSHOT dependency?...

I suppose we'll need an annotation to indicate the requirements for a
specific test.

-Bertrand

Re: Launchpad stable and launchpad unstable

[Robert Munteanu]

On Fri, 2016-10-14 at 15:38 +0200, Bertrand Delacretaz wrote:
> I've committed an experimental script [1] which modifies the
> launchpad/builder provisioning model files to remove the version
> numbers for all Sling artifacts. See comments in the script for
> details (and enjoy the sed regexp ;-)
> 
> When doing so, the Slingstart plugin uses the latest snapshot instead
> of a specified version, IIUC.
> 
> The script doesn't fully work yet (see comments in the script -
> patches or hints welcome) but once it does the process for running
> our
> integration tests on this all-snapshots launchpad would be:


For the record, it doesn't work for me either, but misses more SNAPSHOT
versions:

$ mvn dependency:resolve | grep org.apache.sling | grep -v SNAPSHOT
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
[INFO] --- maven-dependency-plugin:2.10:resolve (default-cli) @
org.apache.sling.launchpad ---
[INFO]org.apache.sling:org.apache.sling.servlets.resolver:jar:2.4.6
:provided
[INFO]org.apache.sling:org.apache.sling.launchpad.base:war:webapp:5
.6.0-2.6.12:provided
[INFO]org.apache.sling:org.apache.sling.commons.scheduler:jar:2.5.2
:provided
[INFO]org.apache.sling:org.apache.sling.commons.log.webconsole:jar:
1.0.0:provided
[INFO]org.apache.sling:org.apache.sling.commons.log:jar:5.0.0:provi
ded
[INFO]org.apache.sling:org.apache.sling.resourceresolver:jar:1.5.0:
provided
[INFO]org.apache.sling:org.apache.sling.launchpad.base:jar:5.6.0-
2.6.12:provided
[INFO]org.apache.sling:org.apache.sling.launchpad.base:jar:app:5.6.
0-2.6.12:provided
[INFO]org.apache.sling:org.apache.sling.event.dea:jar:1.1.0:provide
d
[INFO]org.apache.sling:org.apache.sling.api:jar:2.15.0:provided
[INFO]org.apache.sling:org.apache.sling.installer.core:jar:3.7.0:pr
ovided

Robert

Re: Launchpad stable and launchpad unstable

[Robert Munteanu]

On Fri, 2016-10-14 at 15:45 +0200, Carsten Ziegeler wrote:
> Bertrand Delacretaz wrote
> > On Mon, Oct 3, 2016 at 3:20 PM, Robert Munteanu  > > wrote:
> > > ...If we make building an unstable launchpad really simple, e.g.
> > > an addition
> > > profile or a mojo property, I don't think we lose anything...
> > 
> > I've committed an experimental script [1] which modifies the
> > launchpad/builder provisioning model files to remove the version
> > numbers for all Sling artifacts. See comments in the script for
> > details (and enjoy the sed regexp ;-)
> > 
> > When doing so, the Slingstart plugin uses the latest snapshot
> > instead
> > of a specified version, IIUC.
> > 
> > The script doesn't fully work yet (see comments in the script -
> > patches or hints welcome) but once it does the process for running
> > our
> > integration tests on this all-snapshots launchpad would be:
> > 
> > 1) Run this script in launchpad/builder, build and deploy that,
> > with a
> > specific classifier or other marker
> > 2) Run the launchpad/testing build using that special launchpad jar
> > 3) Revert the code changes caused by step 1), unless working in a
> > throwaway folder
> > 
> > Would that work for you guys?
> > 
> 
> Sorry I already replied to your commit as I didn't see this mail :(
> 
> I think a script is fine and thanks for taking it up.
> My initial thoughts were to do this in the slingstart maven plugin
> This would give a little bit more control - if required, like
> enabling
> snapshots only for a single feature or something. Or having a
> blacklist.
> 
> But maybe that's not needed anyways

How would we handle launchpad tests that rely on functionality
introduced in a SNAPSHOT dependency?

These would fail with the stable launchpad. We could make then run
conditionally just with the unstable launchpad but OTOH we need to
remember to remove that check after the bundle which provides the new
functionality ( or the bug fix ) is released and included in the
launchpad.

Robert

Re: Launchpad stable and launchpad unstable

[Carsten Ziegeler]

Bertrand Delacretaz wrote
> On Mon, Oct 3, 2016 at 3:20 PM, Robert Munteanu  wrote:
>> ...If we make building an unstable launchpad really simple, e.g. an addition
>> profile or a mojo property, I don't think we lose anything...
> 
> I've committed an experimental script [1] which modifies the
> launchpad/builder provisioning model files to remove the version
> numbers for all Sling artifacts. See comments in the script for
> details (and enjoy the sed regexp ;-)
> 
> When doing so, the Slingstart plugin uses the latest snapshot instead
> of a specified version, IIUC.
> 
> The script doesn't fully work yet (see comments in the script -
> patches or hints welcome) but once it does the process for running our
> integration tests on this all-snapshots launchpad would be:
> 
> 1) Run this script in launchpad/builder, build and deploy that, with a
> specific classifier or other marker
> 2) Run the launchpad/testing build using that special launchpad jar
> 3) Revert the code changes caused by step 1), unless working in a
> throwaway folder
> 
> Would that work for you guys?
> 
Sorry I already replied to your commit as I didn't see this mail :(

I think a script is fine and thanks for taking it up.
My initial thoughts were to do this in the slingstart maven plugin
This would give a little bit more control - if required, like enabling
snapshots only for a single feature or something. Or having a blacklist.

But maybe that's not needed anyways

Carsten

 

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Launchpad stable and launchpad unstable

[Bertrand Delacretaz]

On Mon, Oct 3, 2016 at 3:20 PM, Robert Munteanu  wrote:
>...If we make building an unstable launchpad really simple, e.g. an addition
> profile or a mojo property, I don't think we lose anything...

I've committed an experimental script [1] which modifies the
launchpad/builder provisioning model files to remove the version
numbers for all Sling artifacts. See comments in the script for
details (and enjoy the sed regexp ;-)

When doing so, the Slingstart plugin uses the latest snapshot instead
of a specified version, IIUC.

The script doesn't fully work yet (see comments in the script -
patches or hints welcome) but once it does the process for running our
integration tests on this all-snapshots launchpad would be:

1) Run this script in launchpad/builder, build and deploy that, with a
specific classifier or other marker
2) Run the launchpad/testing build using that special launchpad jar
3) Revert the code changes caused by step 1), unless working in a
throwaway folder

Would that work for you guys?

-Bertrand

(BTW the resulting launchpad with all snapshots doesn't start, might
be a good indicator that we need this!

[1] 
https://svn.apache.org/repos/asf/sling/trunk/launchpad/builder/set-sling-snapshots.sh

Re: Launchpad stable and launchpad unstable

[Robert Munteanu]

On Wed, 2016-09-28 at 22:06 +, Stefan Seifert wrote:
> discussed at the Sling Committer Round Table @ adaptTo() 2016
> 
> currently we have only one launchpad in trunk, which contains a
> mixture of release dependencies and snapshot dependencies. this makes
> it difficult to release new versions because one has to release all
> snapshot projects first, and perhaps the last relese version is fine
> as well.
> 
> we came up with a new proposal to fulfill the different needs:
> 
> - the launchpad maintained in trunk should always be the "stable"
> launchpad, referencing only released dependencies of the sling
> bundles and oft the integration tests project
> 
> - the "unstable" launchpad is created dynamically from this stable
> launchpad by using the same bundle list, but for each dependency the
> latest snapshot version that is available in the maven repo,
> including the latest snapshot version oft he integration tests.
> perhaps the slingstart plugin could be extented to generate this.
> 
> - i'm not sure what to do with new bundles which are not released yet
> thus are not referenced in the stable launchpad.

I'm +1 on this change, it makes it much simpler to perform releases. If
we make building an unstable launchpad really simple, e.g. an addition
profile or a mojo property, I don't think we lose anything.

Robert