Re: RIP Apache Jenkins!?

[Robert Munteanu]

On Thu, 2016-12-08 at 10:46 +0100, Julian Reschke wrote:
> On 2016-12-07 12:21, Michael Dürig wrote:
> > 
> > In a F2F discussion we agreed to pay more attention to test
> > failures on
> > the Apache Jenkins instance and make an effort to turn it green:
> > 
> > - tests that are constantly failing and also fail on a local
> > checkout
> > should be marked @Ignored along with an issue reference and an bug
> > report in Jira.
> > 
> > - tests that only fail on Jenkins should be marked as such through
> > the
> > CIHelpers facility
> > 
> > In the meanwhile I tried to configure the Jenkins Jira plugin so it
> > would automatically file Jira issues for failures. This didn't seem
> > to
> > successful as the last failing build didn't show up in the Oak
> > Jira.
> > There is no indications of anything going wrong in the build logs
> > neither though. Anyone who wants to take this up, be my guest.
> > 
> > Michael
> 
> +1
> 
> FWIW; we have way too many tests that happen to fail "sometimes".
> The 
> following just happened twice to me when running integration tests:
> 
> Failed tests: 
> externalAddOffline(org.apache.jackrabbit.oak.plugins.blob.datastore.B
> lobIdTrackerTest): 
> expected:<1006> but was:<1001>

There is a plug-in named 'Test results analyser' [1] installed on the
ASF Jenkins instance which exposes test execution results over a number
of jobs.

We have it enabled for Sling and you can see some of the flaky tests
'in action' at

  https://builds.apache.org/view/S-Z/view/Sling-Dashboard/job/sling-ins
taller-it-1.8/test_results_analyzer/

You might want to enable it to track how flaky tests really are.

Robert

[1]: https://wiki.jenkins-ci.org/display/JENKINS/Test+Results+Analyzer+
Plugin

Re: RIP Apache Jenkins!?

[Amit Jain]

>> Failed tests: externalAddOffline(org.apache.
jackrabbit.oak.plugins.blob.datastore.BlobIdTrackerTest): expected:<1006>
but was:<1001>
Was added yesterdaytaking a look. Will Ignore it meantime.

Thanks
Amit

On Thu, Dec 8, 2016 at 3:16 PM, Julian Reschke 
wrote:

> On 2016-12-07 12:21, Michael Dürig wrote:
>
>>
>> In a F2F discussion we agreed to pay more attention to test failures on
>> the Apache Jenkins instance and make an effort to turn it green:
>>
>> - tests that are constantly failing and also fail on a local checkout
>> should be marked @Ignored along with an issue reference and an bug
>> report in Jira.
>>
>> - tests that only fail on Jenkins should be marked as such through the
>> CIHelpers facility
>>
>> In the meanwhile I tried to configure the Jenkins Jira plugin so it
>> would automatically file Jira issues for failures. This didn't seem to
>> successful as the last failing build didn't show up in the Oak Jira.
>> There is no indications of anything going wrong in the build logs
>> neither though. Anyone who wants to take this up, be my guest.
>>
>> Michael
>>
>
> +1
>
> FWIW; we have way too many tests that happen to fail "sometimes". The
> following just happened twice to me when running integration tests:
>
> Failed tests: 
> externalAddOffline(org.apache.jackrabbit.oak.plugins.blob.datastore.BlobIdTrackerTest):
> expected:<1006> but was:<1001>
>
> Best regards, Julian
>
>

Re: RIP Apache Jenkins!?

[Julian Reschke]

On 2016-12-07 12:21, Michael Dürig wrote:


In a F2F discussion we agreed to pay more attention to test failures on
the Apache Jenkins instance and make an effort to turn it green:

- tests that are constantly failing and also fail on a local checkout
should be marked @Ignored along with an issue reference and an bug
report in Jira.

- tests that only fail on Jenkins should be marked as such through the
CIHelpers facility

In the meanwhile I tried to configure the Jenkins Jira plugin so it
would automatically file Jira issues for failures. This didn't seem to
successful as the last failing build didn't show up in the Oak Jira.
There is no indications of anything going wrong in the build logs
neither though. Anyone who wants to take this up, be my guest.

Michael


+1

FWIW; we have way too many tests that happen to fail "sometimes". The 
following just happened twice to me when running integration tests:


Failed tests: 
externalAddOffline(org.apache.jackrabbit.oak.plugins.blob.datastore.BlobIdTrackerTest): 
expected:<1006> but was:<1001>


Best regards, Julian

Re: RIP Apache Jenkins!?

[Michael Dürig]

In a F2F discussion we agreed to pay more attention to test failures on 
the Apache Jenkins instance and make an effort to turn it green:


- tests that are constantly failing and also fail on a local checkout 
should be marked @Ignored along with an issue reference and an bug 
report in Jira.


- tests that only fail on Jenkins should be marked as such through the 
CIHelpers facility


In the meanwhile I tried to configure the Jenkins Jira plugin so it 
would automatically file Jira issues for failures. This didn't seem to 
successful as the last failing build didn't show up in the Oak Jira. 
There is no indications of anything going wrong in the build logs 
neither though. Anyone who wants to take this up, be my guest.


Michael

On 15.11.16 11:01 , Michael Dürig wrote:


Hi,

I'm losing my patience with the Apache Jenkins instance. Again our Job
is not working properly since a week. I'm spending roughly 2 hours a
week trouble shooting stuff and this does neither include actual test
failures nor the Windows instance that never actually worked. This is
going on for too long now and I don't see some real progress.

I would suggest we start looking for other options.

Michael

Re: RIP Apache Jenkins!?

[Bertrand Delacretaz]

Hi,

On Tue, Nov 29, 2016 at 11:15 AM, Thomas Mueller  wrote:
...
> https://issues.apache.org/jira/browse/INFRA-9709
>
> ...This is just crazy. Either issues get resolved, or they don't, in which
> case we should get a notification that they don't...

I agree. The above issue is about Sonar which I suspect is low
priority compared to Jenkins, and the ASF infra team was understaffed
for several months this year AFAIK so I suspect priorities have gotten
in the way.

I haven't worked much on Jenkins lately but I know Robert got great
results for Sling by adapting our build better to how the ASF's
Jenkins works, so maybe there's hope in that direction?

OTOH if people have concrete requests for how
https://builds.apache.org/ can be more useful I'm happy to relay those
concerns to ASF infra. The more concrete the better.

-Bertrand

Re: RIP Apache Jenkins!?

[Thomas Mueller]

Hi,

>And option #4 - donate some computing capacity to run some dedicated
>Jenkins slaves for Oak.

I don't think it's a hardware problem. The problem seems to be turnaround
times from the Apache infra *team*: they seem to be overloaded. It's not
just with Jenkins, see for example:


https://issues.apache.org/jira/browse/INFRA-9709


This issue was created May 2015! With comments from Infra on June and
August 2015, and no activity since then, even after I have asked January
this year. Status: WAITING FOR INFRA

This is just crazy. Either issues get resolved, or they don't, in which
case we should get a notification that they don't.


Regards,
Thomas

Re: RIP Apache Jenkins!?

[Bertrand Delacretaz]

On Tue, Nov 29, 2016 at 10:13 AM, Robert Munteanu  wrote:
> ...There's also option #3, which is even more interesting :-)...

And option #4 - donate some computing capacity to run some dedicated
Jenkins slaves for Oak.

-Bertrand

Re: RIP Apache Jenkins!?

[Robert Munteanu]

On Tue, 2016-11-29 at 09:36 +0100, Michael Dürig wrote:
> 
> On 29.11.16 9:33 , Robert Munteanu wrote:
> > Option #2 is to make the whole server public, but restrict login to
> > Oak
> > committers.
> 
> Hmm, that's gonna be an "interesting" exercise ;-)

There's also option #3, which is even more interesting :-)

Publish only the results as static HTML + whatever is needed, either
using a custom Jenkins plugin or a scraper like wget.

But I would first try to find out how much of the instability is due to
 the Jenkins instance and how much due to the test setup.

For us in Sling a lot of the Jenkins-caused problems are gone, most of
the issues are related to us.

Robert

Re: RIP Apache Jenkins!?

[Michael Dürig]

On 29.11.16 9:33 , Robert Munteanu wrote:

Option #2 is to make the whole server public, but restrict login to Oak
committers.


Hmm, that's gonna be an "interesting" exercise ;-)

Michael

Re: RIP Apache Jenkins!?

[Robert Munteanu]

On Tue, 2016-11-29 at 09:30 +0100, Michael Dürig wrote:
> 
> On 17.11.16 10:56 , Robert Munteanu wrote:
> > Nothing prevents you from setting up a separate Jenkins instance
> > maintained by Adobe and making the results public.
> 
> Interesting idea. How would you make the results public? Just by
> sending 
> the notification mails to the dev list? Or is there a better way?

That's option #1. 

Option #2 is to make the whole server public, but restrict login to Oak
committers.

Robert

Re: RIP Apache Jenkins!?

[Michael Dürig]

On 17.11.16 10:56 , Robert Munteanu wrote:

Nothing prevents you from setting up a separate Jenkins instance
maintained by Adobe and making the results public.


Interesting idea. How would you make the results public? Just by sending 
the notification mails to the dev list? Or is there a better way?


Michael

Re: RIP Apache Jenkins!?

[Robert Munteanu]

Here's a thought from the peanuts gallery :-)

Nothing prevents you from setting up a separate Jenkins instance
maintained by Adobe and making the results public. This would not be
vastly different from the ASF Jenkins instance, I guess that you would
not be in a 'trusted' ASF network anymore and the administrator are
different, but that's about it.

Robert 

On Thu, 2016-11-17 at 14:22 +0100, Michael Dürig wrote:
> I agree in general. And we could as well also keep our current
> Jenkins 
> instance. It is just that it doesn't provide much value unless we
> spent 
> a lot of time on it, which didn't happen in the (extended) past.
> 
> Michael
> 
> On 17.11.16 2:15 , Tommaso Teofili wrote:
> > sorry but I disagree, while Adobe can have its internal
> > infrastructure run
> > whatever kind of tests, I think Oak should have its own continuous
> > build
> > working on the ASF infra, if anything needs to be adjusted we can
> > get in
> > touch with infra@.
> > Other than that I am sorry I could not spend more time on CI setup
> > (and Oak
> > in general) lately.
> > 
> > Regards,
> > Tommaso
> > 
> > Il giorno gio 17 nov 2016 alle ore 13:35 Michael Dürig  > che.org>
> > ha scritto:
> > 
> > > 
> > > 
> > > On 17.11.16 12:31 , Bertrand Delacretaz wrote:
> > > > On Thu, Nov 17, 2016 at 9:27 AM, Michael Dürig  > > > org>
> > > 
> > > wrote:
> > > > > ...I was mostly thinking of using some internal resources as
> > > > > so far the
> > > 
> > > public
> > > > > options didn't work out for us...
> > > > 
> > > > Internal to what?
> > > 
> > > To Adobe
> > > 
> > > Michael
> > > > 
> > > > -Bertrand
> > > > 

Re: RIP Apache Jenkins!?

[Tommaso Teofili]

sorry but I disagree, while Adobe can have its internal infrastructure run
whatever kind of tests, I think Oak should have its own continuous build
working on the ASF infra, if anything needs to be adjusted we can get in
touch with infra@.
Other than that I am sorry I could not spend more time on CI setup (and Oak
in general) lately.

Regards,
Tommaso

Il giorno gio 17 nov 2016 alle ore 13:35 Michael Dürig 
ha scritto:

>
>
> On 17.11.16 12:31 , Bertrand Delacretaz wrote:
> > On Thu, Nov 17, 2016 at 9:27 AM, Michael Dürig 
> wrote:
> >> ...I was mostly thinking of using some internal resources as so far the
> public
> >> options didn't work out for us...
> >
> > Internal to what?
>
> To Adobe
>
> Michael
> >
> > -Bertrand
> >
>

Re: RIP Apache Jenkins!?

[Michael Dürig]

On 17.11.16 12:31 , Bertrand Delacretaz wrote:

On Thu, Nov 17, 2016 at 9:27 AM, Michael Dürig  wrote:

...I was mostly thinking of using some internal resources as so far the public
options didn't work out for us...


Internal to what?


To Adobe

Michael


-Bertrand

Re: RIP Apache Jenkins!?

[Bertrand Delacretaz]

On Thu, Nov 17, 2016 at 9:27 AM, Michael Dürig  wrote:
> ...I was mostly thinking of using some internal resources as so far the public
> options didn't work out for us...

Internal to what?

-Bertrand

Re: RIP Apache Jenkins!?

[Michael Dürig]

On 16.11.16 4:00 , Davide Giannella wrote:

On 15/11/2016 22:01, Michael Dürig wrote:

I would suggest we start looking for other options.

I'm all for it. What options do we have?

Travis, we moved away as of lack of resources IIRC.


I was mostly thinking of using some internal resources as so far the 
public options didn't work out for us.


Michael



Davide

Re: RIP Apache Jenkins!?

[Davide Giannella]

On 15/11/2016 22:01, Michael Dürig wrote:
> I would suggest we start looking for other options.
I'm all for it. What options do we have?

Travis, we moved away as of lack of resources IIRC.

Davide

RIP Apache Jenkins!?

[Michael Dürig]

Hi,

I'm losing my patience with the Apache Jenkins instance. Again our Job 
is not working properly since a week. I'm spending roughly 2 hours a 
week trouble shooting stuff and this does neither include actual test 
failures nor the Windows instance that never actually worked. This is 
going on for too long now and I don't see some real progress.


I would suggest we start looking for other options.

Michael