Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-19 Thread Flavio Percoco
On 08/14/2014 12:35 AM, Russell Bryant wrote:
 On 08/13/2014 06:23 PM, Mark McLoughlin wrote:
 On Wed, 2014-08-13 at 12:05 -0700, James E. Blair wrote:
 cor...@inaugust.com (James E. Blair) writes:

 Sean Dague s...@dague.net writes:

 This has all gone far enough that someone actually wrote a Grease Monkey
 script to purge all the 3rd Party CI content out of Jenkins UI. People
 are writing mail filters to dump all the notifications. Dan Berange
 filters all them out of his gerrit query tools.

 I should also mention that there is a pending change to do something
 similar via site-local Javascript in our Gerrit:

   https://review.openstack.org/#/c/95743/

 I don't think it's an ideal long-term solution, but if it works, we may
 have some immediate relief without all having to install greasemonkey
 scripts.

 You may have noticed that this has merged, along with a further change
 that shows the latest results in a table format.  (You may need to
 force-reload in your browser to see the change.)

 Beautiful! Thank you so much to everyone involved.
 
 +1!  Love this.
 

Yes, finally. One of the things I really wanted.

Thanks guys!

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-14 Thread Christopher Yeoh
On Wed, 13 Aug 2014 18:52:05 -0400
Jay Pipes jaypi...@gmail.com wrote:

 On 08/13/2014 06:35 PM, Russell Bryant wrote:
  On 08/13/2014 06:23 PM, Mark McLoughlin wrote:
  On Wed, 2014-08-13 at 12:05 -0700, James E. Blair wrote:
  cor...@inaugust.com (James E. Blair) writes:
 
  Sean Dague s...@dague.net writes:
 
  This has all gone far enough that someone actually wrote a
  Grease Monkey script to purge all the 3rd Party CI content out
  of Jenkins UI. People are writing mail filters to dump all the
  notifications. Dan Berange filters all them out of his gerrit
  query tools.
 
  I should also mention that there is a pending change to do
  something similar via site-local Javascript in our Gerrit:
 
 https://review.openstack.org/#/c/95743/
 
  I don't think it's an ideal long-term solution, but if it works,
  we may have some immediate relief without all having to install
  greasemonkey scripts.
 
  You may have noticed that this has merged, along with a further
  change that shows the latest results in a table format.  (You may
  need to force-reload in your browser to see the change.)
 
  Beautiful! Thank you so much to everyone involved.
 
  +1!  Love this.
 
 Indeed. Amazeballs.


Agreed! This is a really nice improvement

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-14 Thread Daniel P. Berrange
On Wed, Aug 13, 2014 at 12:05:27PM -0700, James E. Blair wrote:
 cor...@inaugust.com (James E. Blair) writes:
 
  Sean Dague s...@dague.net writes:
 
  This has all gone far enough that someone actually wrote a Grease Monkey
  script to purge all the 3rd Party CI content out of Jenkins UI. People
  are writing mail filters to dump all the notifications. Dan Berange
  filters all them out of his gerrit query tools.
 
  I should also mention that there is a pending change to do something
  similar via site-local Javascript in our Gerrit:
 
https://review.openstack.org/#/c/95743/
 
  I don't think it's an ideal long-term solution, but if it works, we may
  have some immediate relief without all having to install greasemonkey
  scripts.
 
 You may have noticed that this has merged, along with a further change
 that shows the latest results in a table format.  (You may need to
 force-reload in your browser to see the change.)

 Thanks again to Radoslav Gerganov for writing the original change.

These are both a great step forward in usability of the Gerrit web UI.
Thanks for the effort everyone put into these changes.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread James E. Blair
cor...@inaugust.com (James E. Blair) writes:

 Sean Dague s...@dague.net writes:

 This has all gone far enough that someone actually wrote a Grease Monkey
 script to purge all the 3rd Party CI content out of Jenkins UI. People
 are writing mail filters to dump all the notifications. Dan Berange
 filters all them out of his gerrit query tools.

 I should also mention that there is a pending change to do something
 similar via site-local Javascript in our Gerrit:

   https://review.openstack.org/#/c/95743/

 I don't think it's an ideal long-term solution, but if it works, we may
 have some immediate relief without all having to install greasemonkey
 scripts.

You may have noticed that this has merged, along with a further change
that shows the latest results in a table format.  (You may need to
force-reload in your browser to see the change.)

The table only includes CI systems that leave their results in the same
format that we use for Jenkins; we will update the recommendations for
third party CI systems to encourage the use of that format.

This is all still fairly brittle, based mostly on javascript-powered
screen scraping.  However, I'm hoping we can get something like it in a
Gerrit plugin for a more long-term solution to the problem.

Thanks again to Radoslav Gerganov for writing the original change.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread Dan Smith
 You may have noticed that this has merged, along with a further change
 that shows the latest results in a table format.  (You may need to
 force-reload in your browser to see the change.)

Friggin. Awesome.

 Thanks again to Radoslav Gerganov for writing the original change.

Thanks to all involved, as this is a major improvement for everyone!

One thing that we discussed at the nova meetup was having a space for
each CI we *expect* to vote. I haven't looked at the implementation
here, but I assume it just parses the comments to generate the table.
How hard would it be to make the table show all the CI systems we expect
so that it's very obvious that one has gone MIA (as they often do)
before we merge a patch? I think we struggle right now with merging
things that a CI system would have NAKed, but only because we didn't
notice that it hadn't voted.

--Dan



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread James E. Blair
Dan Smith d...@danplanet.com writes:

 You may have noticed that this has merged, along with a further change
 that shows the latest results in a table format.  (You may need to
 force-reload in your browser to see the change.)

 Friggin. Awesome.

 Thanks again to Radoslav Gerganov for writing the original change.

 Thanks to all involved, as this is a major improvement for everyone!

 One thing that we discussed at the nova meetup was having a space for
 each CI we *expect* to vote. I haven't looked at the implementation
 here, but I assume it just parses the comments to generate the table.
 How hard would it be to make the table show all the CI systems we expect
 so that it's very obvious that one has gone MIA (as they often do)
 before we merge a patch? I think we struggle right now with merging
 things that a CI system would have NAKed, but only because we didn't
 notice that it hadn't voted.

I think my preferred method of doing this would be to drive all
third-party CI systems from OpenStack's Zuul.  We are not far from being
able to do that technically, though it's not clear to me that there is
the will for third party CI systems to do that.  However, if we really
are expecting them to report back and are willing to hold up merging a
change because of it, then we really should consider that.

As we look into using a Gerrit plugin for test results, we could see
about adding that as a feature.

Something that we could technically do immediately would be to add
third-party CI systems as reviewers to changes when they are uploaded.
Then they would show up in the list of reviewers at the top of the
change.  This would require maintaining a list of which systems were
expected to vote on which repositories.  Rather than keeping that in
git, we could use group membership for it (and implement something we
have talked about off-and-on, which is using Gerrit group membership to
grant voting privileges to each individual repository).  That would also
allow projects to self-manage both who is permitted to vote on their
repos and who is expected to vote.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread Chmouel Boudjnah
On Wed, Aug 13, 2014 at 3:05 PM, James E. Blair cor...@inaugust.com wrote:

 You may have noticed that this has merged, along with a further change
 that shows the latest results in a table format.  (You may need to
 force-reload in your browser to see the change.)



Very cool!! this is really nice UI, super useful

one litle suggestions for the folk that knows how to do that if that's
possible to do is to sort between the voting and the non-voting so we can
easily spot which one are worthwhile to look at or not.

Chmouel
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread Mark McLoughlin
On Wed, 2014-08-13 at 12:05 -0700, James E. Blair wrote:
 cor...@inaugust.com (James E. Blair) writes:
 
  Sean Dague s...@dague.net writes:
 
  This has all gone far enough that someone actually wrote a Grease Monkey
  script to purge all the 3rd Party CI content out of Jenkins UI. People
  are writing mail filters to dump all the notifications. Dan Berange
  filters all them out of his gerrit query tools.
 
  I should also mention that there is a pending change to do something
  similar via site-local Javascript in our Gerrit:
 
https://review.openstack.org/#/c/95743/
 
  I don't think it's an ideal long-term solution, but if it works, we may
  have some immediate relief without all having to install greasemonkey
  scripts.
 
 You may have noticed that this has merged, along with a further change
 that shows the latest results in a table format.  (You may need to
 force-reload in your browser to see the change.)

Beautiful! Thank you so much to everyone involved.

Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread James E. Blair
Chmouel Boudjnah chmo...@enovance.com writes:

 On Wed, Aug 13, 2014 at 3:05 PM, James E. Blair cor...@inaugust.com wrote:

 You may have noticed that this has merged, along with a further change
 that shows the latest results in a table format.  (You may need to
 force-reload in your browser to see the change.)



 Very cool!! this is really nice UI, super useful

 one litle suggestions for the folk that knows how to do that if that's
 possible to do is to sort between the voting and the non-voting so we can
 easily spot which one are worthwhile to look at or not.

If it is not worth looking at a job that is run by the OpenStack CI
system, please propose a patch to openstack-infra/config to delete it
from the Zuul config.  We only want to run what's useful, and we have
other methods (the silent and experimental queues) to develop new jobs
without creating noise.

If there is a third-party CI system reporting non-voting jobs, um, I
don't know what that means.  If it bothers you, you might ask the
third-party CI system to disable them and if they don't, then ask us to
disable the third-party CI system.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread Russell Bryant
On 08/13/2014 06:23 PM, Mark McLoughlin wrote:
 On Wed, 2014-08-13 at 12:05 -0700, James E. Blair wrote:
 cor...@inaugust.com (James E. Blair) writes:

 Sean Dague s...@dague.net writes:

 This has all gone far enough that someone actually wrote a Grease Monkey
 script to purge all the 3rd Party CI content out of Jenkins UI. People
 are writing mail filters to dump all the notifications. Dan Berange
 filters all them out of his gerrit query tools.

 I should also mention that there is a pending change to do something
 similar via site-local Javascript in our Gerrit:

   https://review.openstack.org/#/c/95743/

 I don't think it's an ideal long-term solution, but if it works, we may
 have some immediate relief without all having to install greasemonkey
 scripts.

 You may have noticed that this has merged, along with a further change
 that shows the latest results in a table format.  (You may need to
 force-reload in your browser to see the change.)
 
 Beautiful! Thank you so much to everyone involved.

+1!  Love this.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread Chmouel Boudjnah
On Wed, Aug 13, 2014 at 6:27 PM, James E. Blair cor...@inaugust.com wrote:

 If it is not worth looking at a job that is run by the OpenStack CI
 system, please propose a patch to openstack-infra/config to delete it
 from the Zuul config.  We only want to run what's useful, and we have
 other methods (the silent and experimental queues) to develop new jobs
 without creating noise.

 If there is a third-party CI system reporting non-voting jobs, um, I
 don't know what that means.  If it bothers you, you might ask the
 third-party CI system to disable them and if they don't, then ask us to
 disable the third-party CI system.



I didn't meant that they were not worthwhile to look at I was just thinking
it could be useful to sort them so we can easily identify from a UI
perspective which one was voting or not.

Chmouel
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread Jay Pipes

On 08/13/2014 06:35 PM, Russell Bryant wrote:

On 08/13/2014 06:23 PM, Mark McLoughlin wrote:

On Wed, 2014-08-13 at 12:05 -0700, James E. Blair wrote:

cor...@inaugust.com (James E. Blair) writes:


Sean Dague s...@dague.net writes:


This has all gone far enough that someone actually wrote a Grease Monkey
script to purge all the 3rd Party CI content out of Jenkins UI. People
are writing mail filters to dump all the notifications. Dan Berange
filters all them out of his gerrit query tools.


I should also mention that there is a pending change to do something
similar via site-local Javascript in our Gerrit:

   https://review.openstack.org/#/c/95743/

I don't think it's an ideal long-term solution, but if it works, we may
have some immediate relief without all having to install greasemonkey
scripts.


You may have noticed that this has merged, along with a further change
that shows the latest results in a table format.  (You may need to
force-reload in your browser to see the change.)


Beautiful! Thank you so much to everyone involved.


+1!  Love this.


Indeed. Amazeballs.

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-08-13 Thread James E. Blair
Chmouel Boudjnah chmo...@enovance.com writes:

 On Wed, Aug 13, 2014 at 6:27 PM, James E. Blair cor...@inaugust.com wrote:

 If it is not worth looking at a job that is run by the OpenStack CI
 system, please propose a patch to openstack-infra/config to delete it
 from the Zuul config.  We only want to run what's useful, and we have
 other methods (the silent and experimental queues) to develop new jobs
 without creating noise.

 If there is a third-party CI system reporting non-voting jobs, um, I
 don't know what that means.  If it bothers you, you might ask the
 third-party CI system to disable them and if they don't, then ask us to
 disable the third-party CI system.

 I didn't meant that they were not worthwhile to look at I was just thinking
 it could be useful to sort them so we can easily identify from a UI
 perspective which one was voting or not.

I think part of why I responded with that is because this is not the
first time I have heard someone say they don't want to see the
non-voting jobs.  If that's a real desire, we should get to the bottom
of it.  We don't actually want the CI system to be annoying.  :)

From my perspective, anything red should represent something that
someone needs to process and either correct or make an informed decision
to ignore.  That is, a failing non-voting job should either cause the
submitter to fix a problem with their code (same as a failing voting
job), or investigate the result and determine that in this case, the
non-voting job can be safely ignored (ie, is an expected result of the
change).

If we have non-voting jobs that don't match that criteria, we should
remove them.  Or make them voting.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-07-04 Thread Duncan Thomas
On 2 July 2014 16:11, Anita Kuno ante...@anteaya.info wrote:
 Hmmm, my first response - given that long chew we had on the ml[0] about
 the use of the word certified as well as the short confirmation we had
 in the tc meeting[1] that the word certified would not be used, but
 rather some version of the word 'tested' - how long until edits can be
 made to the cinder wiki to comply with that agreement?

It's a wiki - anybody who cares enough can go and re-word it...



-- 
Duncan Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-07-04 Thread Anita Kuno
On 07/04/2014 08:11 AM, Duncan Thomas wrote:
 On 2 July 2014 16:11, Anita Kuno ante...@anteaya.info wrote:
 Hmmm, my first response - given that long chew we had on the ml[0] about
 the use of the word certified as well as the short confirmation we had
 in the tc meeting[1] that the word certified would not be used, but
 rather some version of the word 'tested' - how long until edits can be
 made to the cinder wiki to comply with that agreement?
 
 It's a wiki - anybody who cares enough can go and re-word it...
 
 
 
True, but it is a cinder wiki. I don't know what words you want to use
and don't want to lose the intent of the page.

Also given that some folks in other places in cinder are still using
references to certified I think it shows a strong message if someone in
a leadership position in cinder makes the edits.

Thanks Duncan,
Anita.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-07-02 Thread Anita Kuno
On 07/01/2014 12:27 PM, Asselin, Ramy wrote:
 Anita,
 
 This line [1] is effectively a sub-set of tempest-dsm-full,  and what we're 
 currently running manually now. I far as I understood, this is the current 
 minimum. The exact sub-set (or full set, or if additional tests are allowed) 
 is still under discussion. 
 
Fair enough, I need to make some time to attend Cinder weekly meetings.
Wednesdays are tough for me, but I need to make more of an effort.
Thanks for making me aware.
 I created a WIP reference patch [2] for the cinder team that mimics the above 
 script to run these tests based off the similar  Jenkins job-template used by 
 Openstack-Jenkins {pipeline}-tempest-dsvm-*
 
Thanks for sharing the link to your patch, I have given it an initial
review for formatting. I'll be interested to see the discussion from
this, I'm not sure where a design like this fits but I appreciate you
putting up a patch so we can gather some input.

Thanks Ramy,
Anita.

 Ramy
 
 [1] 
 https://github.com/openstack-dev/devstack/blob/master/driver_certs/cinder_driver_cert.sh#L97
 [2] 
 https://review.openstack.org/#/c/93141/1/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml
 
 -Original Message-
 From: Duncan Thomas [mailto:duncan.tho...@gmail.com] 
 Sent: Tuesday, July 01, 2014 7:03 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit
 
 On 1 July 2014 14:44, Anita Kuno ante...@anteaya.info wrote:
 
 On 07/01/2014 05:56 AM, Duncan Thomas wrote:
 For the record, cinder gave a very clear definition of success in our 
 3rd party guidelines: Passes every test in tempest-dsm-full. If that 
 needs documenting somewhere else, please let me know. It may of 
 course change as we learn more about how 3rd party CI works out, so 
 the fewer places it is duplicated the better, maybe?

 Thanks Duncan, I wasn't aware of this. Can we start with a url for 
 those guidelines in your reply to this post and then go from there?
 
 https://wiki.openstack.org/wiki/Cinder/certified-drivers should make it clear 
 but doesn't, I'll get that cleared up.
 
 https://etherpad.openstack.org/p/juno-cinder-3rd-party-cert-and-verification
 mentions it, and various weekly meeting minutes also mention it.
 
 
 --
 Duncan Thomas
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-07-02 Thread Anita Kuno
On 07/01/2014 10:03 AM, Duncan Thomas wrote:
 On 1 July 2014 14:44, Anita Kuno ante...@anteaya.info wrote:
 
 On 07/01/2014 05:56 AM, Duncan Thomas wrote:
 For the record, cinder gave a very clear definition of success in our
 3rd party guidelines: Passes every test in tempest-dsm-full. If that
 needs documenting somewhere else, please let me know. It may of course
 change as we learn more about how 3rd party CI works out, so the fewer
 places it is duplicated the better, maybe?

 Thanks Duncan, I wasn't aware of this. Can we start with a url for those
 guidelines in your reply to this post and then go from there?
 
 https://wiki.openstack.org/wiki/Cinder/certified-drivers should make
 it clear but doesn't, I'll get that cleared up.
 
 https://etherpad.openstack.org/p/juno-cinder-3rd-party-cert-and-verification
 mentions it, and various weekly meeting minutes also mention it.
 
 
Thanks for sharing those links, Duncan.

Hmmm, my first response - given that long chew we had on the ml[0] about
the use of the word certified as well as the short confirmation we had
in the tc meeting[1] that the word certified would not be used, but
rather some version of the word 'tested' - how long until edits can be
made to the cinder wiki to comply with that agreement?

Thanks Duncan,
Anita.

[0] http://lists.openstack.org/pipermail/openstack-dev/2014-June/036933.html
[1]
http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-06-17-20.03.html 2.b

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-07-01 Thread Duncan Thomas
On 30 June 2014 16:49, Anita Kuno ante...@anteaya.info wrote:

 Right now that dashboard introduces more confusion than it alleviates
 since the definition of success in regards to third party ci systems
 has yet to be defined by the community.

For the record, cinder gave a very clear definition of success in our
3rd party guidelines: Passes every test in tempest-dsm-full. If that
needs documenting somewhere else, please let me know. It may of course
change as we learn more about how 3rd party CI works out, so the fewer
places it is duplicated the better, maybe?

-- 
Duncan Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-07-01 Thread Anita Kuno
On 07/01/2014 05:56 AM, Duncan Thomas wrote:
 On 30 June 2014 16:49, Anita Kuno ante...@anteaya.info wrote:
 
 Right now that dashboard introduces more confusion than it alleviates
 since the definition of success in regards to third party ci systems
 has yet to be defined by the community.
 
 For the record, cinder gave a very clear definition of success in our
 3rd party guidelines: Passes every test in tempest-dsm-full. If that
 needs documenting somewhere else, please let me know. It may of course
 change as we learn more about how 3rd party CI works out, so the fewer
 places it is duplicated the better, maybe?
 
Thanks Duncan, I wasn't aware of this. Can we start with a url for those
guidelines in your reply to this post and then go from there?

Thanks,
Anita.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-07-01 Thread Duncan Thomas
On 1 July 2014 14:44, Anita Kuno ante...@anteaya.info wrote:

 On 07/01/2014 05:56 AM, Duncan Thomas wrote:
 For the record, cinder gave a very clear definition of success in our
 3rd party guidelines: Passes every test in tempest-dsm-full. If that
 needs documenting somewhere else, please let me know. It may of course
 change as we learn more about how 3rd party CI works out, so the fewer
 places it is duplicated the better, maybe?

 Thanks Duncan, I wasn't aware of this. Can we start with a url for those
 guidelines in your reply to this post and then go from there?

https://wiki.openstack.org/wiki/Cinder/certified-drivers should make
it clear but doesn't, I'll get that cleared up.

https://etherpad.openstack.org/p/juno-cinder-3rd-party-cert-and-verification
mentions it, and various weekly meeting minutes also mention it.


-- 
Duncan Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-07-01 Thread Asselin, Ramy
Anita,

This line [1] is effectively a sub-set of tempest-dsm-full,  and what we're 
currently running manually now. I far as I understood, this is the current 
minimum. The exact sub-set (or full set, or if additional tests are allowed) is 
still under discussion. 

I created a WIP reference patch [2] for the cinder team that mimics the above 
script to run these tests based off the similar  Jenkins job-template used by 
Openstack-Jenkins {pipeline}-tempest-dsvm-*

Ramy

[1] 
https://github.com/openstack-dev/devstack/blob/master/driver_certs/cinder_driver_cert.sh#L97
[2] 
https://review.openstack.org/#/c/93141/1/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml

-Original Message-
From: Duncan Thomas [mailto:duncan.tho...@gmail.com] 
Sent: Tuesday, July 01, 2014 7:03 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

On 1 July 2014 14:44, Anita Kuno ante...@anteaya.info wrote:

 On 07/01/2014 05:56 AM, Duncan Thomas wrote:
 For the record, cinder gave a very clear definition of success in our 
 3rd party guidelines: Passes every test in tempest-dsm-full. If that 
 needs documenting somewhere else, please let me know. It may of 
 course change as we learn more about how 3rd party CI works out, so 
 the fewer places it is duplicated the better, maybe?

 Thanks Duncan, I wasn't aware of this. Can we start with a url for 
 those guidelines in your reply to this post and then go from there?

https://wiki.openstack.org/wiki/Cinder/certified-drivers should make it clear 
but doesn't, I'll get that cleared up.

https://etherpad.openstack.org/p/juno-cinder-3rd-party-cert-and-verification
mentions it, and various weekly meeting minutes also mention it.


--
Duncan Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-30 Thread Daniel P. Berrange
On Sat, Jun 28, 2014 at 08:26:44AM -0500, Matt Riedemann wrote:
 
 
 On 6/27/2014 7:35 AM, Daniel P. Berrange wrote:
 On Fri, Jun 27, 2014 at 07:40:51AM -0400, Sean Dague wrote:
 It's clear that lots of projects want 3rd Party CI information on
 patches. But it's also clear that 6 months into this experiment with a
 lot of 3rd Party CI systems, the Gerrit UI is really not great for this.
 
 That's an understatement about the UI :-)
 
 It seems what we actually want is a dashboard of these results. We want
 them available when we go to Gerrit, but we don't want them in Gerrit
 itself.
 
 What if 3rd Party CI didn't vote in Gerrit? What if it instead published
 to some 3rd party test reporting site (a thing that doesn't yet exist).
 Gerrit has the facility so that we could inject the dashboard content
 for this in Gerrit in a little table somewhere, but the data would
 fundamentally live outside of Gerrit. It would also mean that all the
 aggregate reporting of 3rd Party CI that's being done in custom gerrit
 scripts, could be integrated directly into such a thing.
 
 Agreed, it would be a great improvement in usability if we stopped all
 CI systems, including our default Jenkins, from ever commenting on
 reviews. At most gating CIs should +1/-1.  Having a table of results
 displayed, pulling the data from an external result tracking system
 would be a great idea.
 
 Even better if this external system had a nice button you can press
 to trigger re-check, so we can stop using comments for that too.
 
 I would disagree with this idea since it's equivalent to 'recheck no bug'
 and that's naughty, because then we don't track race bugs as well.

It could easily have a text field for entering a bug number. The point
is to stop adding comments to gerrit that aren't related to code review.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-30 Thread Sean Dague
On 06/29/2014 09:39 AM, Joshua Hesketh wrote:
 On 6/28/14 10:40 AM, James E. Blair wrote:
 An alternate approach would be to have third-party CI systems register
 jobs with OpenStack's Zuul rather than using their own account.  This
 would mean only a single report of all jobs (upstream and 3rd-party)
 per-patchset.  It significantly reduces clutter and makes results more
 accessible -- but even with one system we've never actually wanted to
 have Jenkins results in comments, so I think one of the other options
 would be preferred.  Nonetheless, this is possible with a little bit of
 work.
 
 I agree this isn't the preferred solution, but I disagree with the
 little bit of work. This would require CI systems registering with
 gearman which would mean security issues. The biggest problem with this
 though is that zuul would be stuck waiting from results from 3rd parties
 which often have very slow return times.

Right, one of the other issues is the quality of the CI results varies
as well.

I think one of the test result burn out issues right now is based on the
fact that they are too rolled up as it is. For instance, a docs only
change gets Tempest results, which humans know are irrelevant, but
Jenkins insists they aren't. I think that if we rolled up more
information, and waited longer, we'd be in a worse state.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-30 Thread James E. Blair
Joshua Hesketh joshua.hesk...@rackspace.com writes:

 On 6/28/14 10:40 AM, James E. Blair wrote:
 An alternate approach would be to have third-party CI systems register
 jobs with OpenStack's Zuul rather than using their own account.  This
 would mean only a single report of all jobs (upstream and 3rd-party)
 per-patchset.  It significantly reduces clutter and makes results more
 accessible -- but even with one system we've never actually wanted to
 have Jenkins results in comments, so I think one of the other options
 would be preferred.  Nonetheless, this is possible with a little bit of
 work.

 I agree this isn't the preferred solution, but I disagree with the
 little bit of work. This would require CI systems registering with
 gearman which would mean security issues. The biggest problem with
 this though is that zuul would be stuck waiting from results from 3rd
 parties which often have very slow return times.

Security issues is a bit vague.  They already register with Gerrit;
I'm only suggesting that the point of aggregation would change.  I'm
anticipating that they would use authenticated SSL, with ACLs scoped to
the names of jobs each system is permitted to run.  From the perspective
of overall security as well as network topology (ie, firewalls), very
little changes.  The main differences are third party CI systems don't
have to run Zuul anymore, and we go back to having a smaller number of
votes/comments.

Part of the little bit of work I was referring to was adding a
timeout.  That should truly be not much work, and work we're planning on
doing anyway to help with the tripleo cloud.

But anyway, it's not important to design this out if we prefer another
solution (and I prefer the table of results separated from comments).

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-30 Thread Kurt Taylor
Dan Smith d...@danplanet.com wrote on 06/27/2014 12:33:48 PM:

  What if 3rd Party CI didn't vote in Gerrit? What if it instead
  published to some 3rd party test reporting site (a thing that
  doesn't yet exist). Gerrit has the facility so that we could inject
  the dashboard content for this in Gerrit in a little table
  somewhere, but the data would fundamentally live outside of Gerrit.
  It would also mean that all the aggregate reporting of 3rd Party CI
  that's being done in custom gerrit scripts, could be integrated
  directly into such a thing.

 If it really does show up right in Gerrit as if it were integrated,
 then that would be fine with me. I think the biggest problem we have
 right now is that a lot of the CI systems are very inconsistent in
 their reporting and we often don't realize when one of them *hasn't*
 voted. If the new thing could fill out the chart based on everything
 we expect to see a vote from, so that it's very clear that one is
 missing, then that's a net win right there.

There is a similar old bug for that, with a good suggestion for how it
could possibly be done:

https://bugs.launchpad.net/openstack-ci/+bug/1251758


Kurt Taylor (krtaylor)
OpenStack Development Lead - PowerKVM CI
IBM Linux Technology Center
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-30 Thread Kurt Taylor

Sean Dague s...@dague.net wrote on 06/30/2014 06:03:50 AM:

 From:

 Sean Dague s...@dague.net

 To:

 OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org,

 Date:

 06/30/2014 06:09 AM

 Subject:

 Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

 On 06/29/2014 09:39 AM, Joshua Hesketh wrote:
  On 6/28/14 10:40 AM, James E. Blair wrote:
  An alternate approach would be to have third-party CI systems register
  jobs with OpenStack's Zuul rather than using their own account.  This
  would mean only a single report of all jobs (upstream and 3rd-party)
  per-patchset.  It significantly reduces clutter and makes results more
  accessible -- but even with one system we've never actually wanted to
  have Jenkins results in comments, so I think one of the other options
  would be preferred.  Nonetheless, this is possible with a little bit
of
  work.
 
  I agree this isn't the preferred solution, but I disagree with the
  little bit of work. This would require CI systems registering with
  gearman which would mean security issues. The biggest problem with this
  though is that zuul would be stuck waiting from results from 3rd
parties
  which often have very slow return times.

 Right, one of the other issues is the quality of the CI results varies
 as well.

Agreed. After last summit, Anita, Jay and I decided to start gathering a
team
of 3rd party testers that have the goal of improving the quality of the
third
party systems. We are starting with gathering global unwritten
requirements,
improving documentation and reaching out to new projects that will have
third
party testing needs. We are still in the early stages but now have weekly
meetings to discuss what needs to be done and track progress.

https://wiki.openstack.org/wiki/Meetings/ThirdParty

 I think one of the test result burn out issues right now is based on the
 fact that they are too rolled up as it is. For instance, a docs only
 change gets Tempest results, which humans know are irrelevant, but
 Jenkins insists they aren't. I think that if we rolled up more
 information, and waited longer, we'd be in a worse state.

Maybe it could promptly timeout and then report the system that did not
complete? That would also have the benefit of enforcing a time limit on
reporting results.


Kurt Taylor (krtaylor)
OpenStack Development Lead - PowerKVM CI
IBM Linux Technology Center
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-30 Thread Ilya Shakhat
2014-06-30 19:17 GMT+04:00 Kurt Taylor krtay...@us.ibm.com:

 Dan Smith d...@danplanet.com wrote on 06/27/2014 12:33:48 PM:
  If it really does show up right in Gerrit as if it were integrated,
  then that would be fine with me. I think the biggest problem we have
  right now is that a lot of the CI systems are very inconsistent in
  their reporting and we often don't realize when one of them *hasn't*
  voted. If the new thing could fill out the chart based on everything
  we expect to see a vote from, so that it's very clear that one is
  missing, then that's a net win right there.


 There is a similar old bug for that, with a good suggestion for how it
 could possibly be done:

 https://bugs.launchpad.net/openstack-ci/+bug/1251758


What about to have report like this:
http://stackalytics.com/report/ci/neutron/7 ?

Thanks,
Ilya
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-30 Thread Anita Kuno
On 06/30/2014 11:41 AM, Ilya Shakhat wrote:
 2014-06-30 19:17 GMT+04:00 Kurt Taylor krtay...@us.ibm.com:
 
 Dan Smith d...@danplanet.com wrote on 06/27/2014 12:33:48 PM:
 If it really does show up right in Gerrit as if it were integrated,
 then that would be fine with me. I think the biggest problem we have
 right now is that a lot of the CI systems are very inconsistent in
 their reporting and we often don't realize when one of them *hasn't*
 voted. If the new thing could fill out the chart based on everything
 we expect to see a vote from, so that it's very clear that one is
 missing, then that's a net win right there.


 There is a similar old bug for that, with a good suggestion for how it
 could possibly be done:

 https://bugs.launchpad.net/openstack-ci/+bug/1251758


 What about to have report like this:
 http://stackalytics.com/report/ci/neutron/7 ?
 
 Thanks,
 Ilya
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
As I responded here, Ilya:
http://lists.openstack.org/pipermail/openstack-dev/2014-June/038933.html

Right now that dashboard introduces more confusion than it alleviates
since the definition of success in regards to third party ci systems
has yet to be defined by the community.

I look forward to discussing this more at today's third party meeting,
as Kurt has already linked to. It would be great if you would be willing
to add an item to the agenda.

Thanks Ilya,
Anita.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-30 Thread Dan Smith
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 There is a similar old bug for that, with a good suggestion for how
 it could possibly be done:
 
 https://bugs.launchpad.net/openstack-ci/+bug/1251758

This isn't what I'm talking about. What we need is, for each new
patchset on a given change, an empty table listing all the answers we
expect to see (i.e. one for each of the usual suspect nova CI
systems). The above bug (AFAICT) is simply for tracking last-status of
each, so that if one stops reporting entirely (as minesweeper often
does), we get some indication that the *system* is broken.

- --Dan
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTsYc+AAoJEBeZxaMESjNVcQYH+wayg4T9QFe4tTvGn24PisCf
5cEaeSkwXl+Adiae5cCfCGSTjlErK4lpFtzFapKukcM0+eEp464toskl7vNC0izp
UWCpcg2gbON6Ef/AMa1+PT8uXR9OYAo+/eU8NUJNM01ajeZqqe3H3jnltgoUau0O
fq3O3+Wa2PxBTAVVGi3HXJl4SWpVdEuYDZYOBOtkDXwhIS/hvBdIRuJwt0CygxHx
78WatFsQ09tBHaQCJbs2E+Oar0rD4sF93qjG8jAFiVB/0SJ6wV7AsLColVId2hbe
Qfua3Q6CufJBO2WHV7JORX2fBSOTUmcPcOM1IE4/lgXGiyu3aw5ataL9e4qxudQ=
=VoHH
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-29 Thread Joshua Hesketh

On 6/28/14 10:40 AM, James E. Blair wrote:

An alternate approach would be to have third-party CI systems register
jobs with OpenStack's Zuul rather than using their own account.  This
would mean only a single report of all jobs (upstream and 3rd-party)
per-patchset.  It significantly reduces clutter and makes results more
accessible -- but even with one system we've never actually wanted to
have Jenkins results in comments, so I think one of the other options
would be preferred.  Nonetheless, this is possible with a little bit of
work.


I agree this isn't the preferred solution, but I disagree with the 
little bit of work. This would require CI systems registering with 
gearman which would mean security issues. The biggest problem with this 
though is that zuul would be stuck waiting from results from 3rd parties 
which often have very slow return times.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-28 Thread James E. Blair
Matt Riedemann mrie...@linux.vnet.ibm.com writes:

 I would be good with Jenkins not reporting on a successful run, or if
 rather than a comment from Jenkins the vote in the table had a link to
 the test results, so if you get a -1 from Jenkins you can follow the
 link from the -1 in the table rather than the comment (to avoid
 cluttering up the review comments, especially if it's a +1).

The problem with that is it makes non-voting jobs very difficult to see,
not to mention ancillary information like job runtime.  Plus it adds
extra clicks to get to jobs whose output are frequently reviewed even
with positive votes, such as docs jobs (on docs-draft.o.o).

I think a new table on the page, separate from the comment stream, with
only the latest results of each job is the ideal presentation.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread Flavio Percoco

On 27/06/14 07:40 -0400, Sean Dague wrote:

It's clear that lots of projects want 3rd Party CI information on
patches. But it's also clear that 6 months into this experiment with a
lot of 3rd Party CI systems, the Gerrit UI is really not great for this.

A couple of things have fallen out of this. 3rd Party CI bots outnumber
Human comments on changes on some projects (Nova / Neutron). That has an
impact on the readability of the votes section (on a neutron change the
files in the change are rarely above the fold), the readability of the
comments.

3rd Party CI systems haven't become all that reliable. They fall into
the same problems that Jenkins hits with cloud networking, race bugs in
OpenStack, but also new bugs around site configs. It's kind of a
testament to how much we've learned about how to keep the machine
running that the upstream CI system, even with all it's faults, still
trends more reliable than most of the 3rd Party systems.

Commenting in Gerrit was to eventually get towards voting in Gerrit. But
my experience at this point is reviewers are at CI fatigue and are
mostly not paying attention to the votes. Heck, when we're dealing with
a bunch of bugs in the gate most reviewers want to ignore the Jenkins
votes too, which is why you get the recheck grinding behavior.

This has all gone far enough that someone actually wrote a Grease Monkey
script to purge all the 3rd Party CI content out of Jenkins UI. People
are writing mail filters to dump all the notifications. Dan Berange
filters all them out of his gerrit query tools.

It seems what we actually want is a dashboard of these results. We want
them available when we go to Gerrit, but we don't want them in Gerrit
itself.

What if 3rd Party CI didn't vote in Gerrit? What if it instead published
to some 3rd party test reporting site (a thing that doesn't yet exist).
Gerrit has the facility so that we could inject the dashboard content
for this in Gerrit in a little table somewhere, but the data would
fundamentally live outside of Gerrit. It would also mean that all the
aggregate reporting of 3rd Party CI that's being done in custom gerrit
scripts, could be integrated directly into such a thing.

I'm not signing up for this particular mission, but I wanted to stick it
out there to figure out if the idea had merrit, and if it did, if it
excited anyone to enough to dive on it.



I agree all those comments cause way to much noise. The most important
bit of the comment is the link that takes you to the CI logs so, it
may be probably enough to just have 1 link that will take someone to
the builds of the 3rd party CI for that specific change.

Along the lines of what you just proposed, what if each 3rd Party CI
username in the review's list just links to this builds-list
resources that a developer can go to to check the logs. Would that be
too much of a hack for gerrit?

I'd like to avoid us to create yet-another-monitoring-resource that
we'll have to go to in order to figure out what's going on.

FWIW, the above could also be done for our Jenkins.

Overall, +1 for the email and idea.
Flavio

--
@flaper87
Flavio Percoco


pgpwpRJnlxUen.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread Daniel P. Berrange
On Fri, Jun 27, 2014 at 07:40:51AM -0400, Sean Dague wrote:
 It's clear that lots of projects want 3rd Party CI information on
 patches. But it's also clear that 6 months into this experiment with a
 lot of 3rd Party CI systems, the Gerrit UI is really not great for this.

That's an understatement about the UI :-)

 It seems what we actually want is a dashboard of these results. We want
 them available when we go to Gerrit, but we don't want them in Gerrit
 itself.

 What if 3rd Party CI didn't vote in Gerrit? What if it instead published
 to some 3rd party test reporting site (a thing that doesn't yet exist).
 Gerrit has the facility so that we could inject the dashboard content
 for this in Gerrit in a little table somewhere, but the data would
 fundamentally live outside of Gerrit. It would also mean that all the
 aggregate reporting of 3rd Party CI that's being done in custom gerrit
 scripts, could be integrated directly into such a thing.

Agreed, it would be a great improvement in usability if we stopped all
CI systems, including our default Jenkins, from ever commenting on
reviews. At most gating CIs should +1/-1.  Having a table of results
displayed, pulling the data from an external result tracking system
would be a great idea.

Even better if this external system had a nice button you can press
to trigger re-check, so we can stop using comments for that too.

To me the ideal world is where the only things adding comments to
reviews are human and their comments are actually about the code
in the patch :-)

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread John Griffith
On Fri, Jun 27, 2014 at 6:35 AM, Daniel P. Berrange berra...@redhat.com
wrote:

 On Fri, Jun 27, 2014 at 07:40:51AM -0400, Sean Dague wrote:
  It's clear that lots of projects want 3rd Party CI information on
  patches. But it's also clear that 6 months into this experiment with a
  lot of 3rd Party CI systems, the Gerrit UI is really not great for this.

 That's an understatement about the UI :-)

  It seems what we actually want is a dashboard of these results. We want
  them available when we go to Gerrit, but we don't want them in Gerrit
  itself.


​Yes please, and no I'm not signing up for it either :)
​

 
  What if 3rd Party CI didn't vote in Gerrit? What if it instead published
  to some 3rd party test reporting site (a thing that doesn't yet exist).
  Gerrit has the facility so that we could inject the dashboard content
  for this in Gerrit in a little table somewhere, but the data would
  fundamentally live outside of Gerrit. It would also mean that all the
  aggregate reporting of 3rd Party CI that's being done in custom gerrit
  scripts, could be integrated directly into such a thing.

 Agreed, it would be a great improvement in usability if we stopped all
 CI systems, including our default Jenkins, from ever commenting on
 reviews. At most gating CIs should +1/-1.  Having a table of results
 displayed, pulling the data from an external result tracking system
 would be a great idea.

 Even better if this external system had a nice button you can press
 to trigger re-check, so we can stop using comments for that too.

 To me the ideal world is where the only things adding comments to
 reviews are human and their comments are actually about the code
 in the patch :-)

 Regards,
 Daniel
 --
 |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/
 :|
 |: http://libvirt.org  -o- http://virt-manager.org
 :|
 |: http://autobuild.org   -o- http://search.cpan.org/~danberr/
 :|
 |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc
 :|

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread Dan Smith
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 What if 3rd Party CI didn't vote in Gerrit? What if it instead
 published to some 3rd party test reporting site (a thing that
 doesn't yet exist). Gerrit has the facility so that we could inject
 the dashboard content for this in Gerrit in a little table
 somewhere, but the data would fundamentally live outside of Gerrit.
 It would also mean that all the aggregate reporting of 3rd Party CI
 that's being done in custom gerrit scripts, could be integrated
 directly into such a thing.

If it really does show up right in Gerrit as if it were integrated,
then that would be fine with me. I think the biggest problem we have
right now is that a lot of the CI systems are very inconsistent in
their reporting and we often don't realize when one of them *hasn't*
voted. If the new thing could fill out the chart based on everything
we expect to see a vote from, so that it's very clear that one is
missing, then that's a net win right there.

But at the end of the day, we need to make sure we're improving
visibility and not hiding all the noise they generate just because
it's annoying. As long as something like what was suggested here is
injected right into the gerrit API, then I'm in favor of something
like this.

- --Dan
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTrar8AAoJEBeZxaMESjNVizsH/RfjoJOlX80N+jviDHG4yMUv
n9dRV2vFfitKmapN8HIWdo6aFjRDHb5t3BWcO+9xP7+w/nZYpC88Y6Eq8c8HeZPz
WIeVVY8/AmL/VVGXNojeIoCdszR8vdlaEaOrlKCjQIr8zqZJrz6LgPXIfoVCVrMg
hHN2+TM4rn9scFHlKSnf8vq7rLj/aTfzG+ITSSksslM2YxWkqDFWWBTrTnqrWHx5
Uk91JSFmWTC2UnbyVzPI88D0a7HGkpyxgFnWb8Y63RBr/JygFisoBLaqbHA9o9ML
ajI+st8ySBEwWGyVkckzX5Go1KWgTP9pH1zwhpZz+qvEZCJA6Pbp16+r+kTFBew=
=NkTB
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread Zaro
Sean, there is a proposal[1] in upstream gerrit to fix this issue.  David's
proposal is to make Gerrit handle multiple notifications channels per
project which would allow us to treat bot notifications differently than
human notifications.  He has the same problem as we do, most of his builds
are done using 'third party' CI so he has many more third party systems
reporting to his gerrit instance than we do.  Please take a look and
provide feedback.

[1] *https://code.google.com/p/gerrit/issues/detail?id=2465
https://code.google.com/p/gerrit/issues/detail?id=2465*
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread Sean Dague
On 06/27/2014 02:19 PM, Zaro wrote:
 
 Sean, there is a proposal[1] in upstream gerrit to fix this issue.
  David's proposal is to make Gerrit handle multiple notifications
 channels per project which would allow us to treat bot notifications
 differently than human notifications.  He has the same problem as we do,
 most of his builds are done using 'third party' CI so he has many more
 third party systems reporting to his gerrit instance than we do.  Please
 take a look and provide feedback.
 
 [1] _https://code.google.com/p/gerrit/issues/detail?id=2465_

That seems to require the new Change Screen?

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread Zaro
David did suggest adding REST api endpoints to get data for each channel so
it doesn't necessarily require you to even use the gerrit web ui.  However
the web UI's old screen is pretty much dead so I assume the presentation
would only be available in new change screen.  I know Openstackers have had
reservations about the new change screen but please don't judge it by
trying the gerrit 2.8.x implementation.  It has improved quite a bit.  I
have been using it on the https://gerrit-review.googlesource.com and do
prefer it over the old change screen :)

-Khai

On Fri, Jun 27, 2014 at 6:38 PM, Sean Dague s...@dague.net wrote:

 On 06/27/2014 02:19 PM, Zaro wrote:
 
  Sean, there is a proposal[1] in upstream gerrit to fix this issue.
   David's proposal is to make Gerrit handle multiple notifications
  channels per project which would allow us to treat bot notifications
  differently than human notifications.  He has the same problem as we do,
  most of his builds are done using 'third party' CI so he has many more
  third party systems reporting to his gerrit instance than we do.  Please
  take a look and provide feedback.
 
  [1] _https://code.google.com/p/gerrit/issues/detail?id=2465_

 That seems to require the new Change Screen?

 -Sean

 --
 Sean Dague
 http://dague.net


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread Sean Dague
I do quite like the conflicts with part, I was thinking about that the
other day. That would be hugely useful.

-Sean

On 06/27/2014 02:56 PM, Zaro wrote:
 David did suggest adding REST api endpoints to get data for each channel
 so it doesn't necessarily require you to even use the gerrit web ui.
  However the web UI's old screen is pretty much dead so I assume the
 presentation would only be available in new change screen.  I know
 Openstackers have had reservations about the new change screen but
 please don't judge it by trying the gerrit 2.8.x implementation.  It has
 improved quite a bit.  I have been using it on
 the https://gerrit-review.googlesource.com and do prefer it over the
 old change screen :)
 
 -Khai
 
 On Fri, Jun 27, 2014 at 6:38 PM, Sean Dague s...@dague.net
 mailto:s...@dague.net wrote:
 
 On 06/27/2014 02:19 PM, Zaro wrote:
 
  Sean, there is a proposal[1] in upstream gerrit to fix this issue.
   David's proposal is to make Gerrit handle multiple notifications
  channels per project which would allow us to treat bot notifications
  differently than human notifications.  He has the same problem as
 we do,
  most of his builds are done using 'third party' CI so he has many more
  third party systems reporting to his gerrit instance than we do.
  Please
  take a look and provide feedback.
 
  [1] _https://code.google.com/p/gerrit/issues/detail?id=2465_
 
 That seems to require the new Change Screen?
 
 -Sean
 
 --
 Sean Dague
 http://dague.net
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 mailto:OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit

2014-06-27 Thread James E. Blair
Sean Dague s...@dague.net writes:

 It seems what we actually want is a dashboard of these results. We want
 them available when we go to Gerrit, but we don't want them in Gerrit
 itself.

I agree with most of what you wrote, particularly that we want them
available in Gerrit and with a sensible presentation.  Though I don't
think that necessarily excludes storing the data in Gerrit, as Khai
points out.

I think one ideal interface is a table in Gerrit of all the jobs run on
a change (from all systems) and their results (with links).  That sounds
like the project David is working on (that Khai pointed out), which
isn't surprising -- we sketched out the initial design with him about
three years ago; our needs are very similar.

David's proposal is a table like:

  Job/Category | started on | ended on | href | status
  foo [...]
  bar [...]
  bat […]

I think that's a great approach because it provides the needed
information to reviewers in an appropriate interface in Gerrit.

I suspect that whether the data are stored in Gerrit or a different
system, the (future) vinz web ui could retrieve it from either.

An alternate approach would be to have third-party CI systems register
jobs with OpenStack's Zuul rather than using their own account.  This
would mean only a single report of all jobs (upstream and 3rd-party)
per-patchset.  It significantly reduces clutter and makes results more
accessible -- but even with one system we've never actually wanted to
have Jenkins results in comments, so I think one of the other options
would be preferred.  Nonetheless, this is possible with a little bit of
work.

If anyone is seriously interested in working on Sean's proposal or
something similar, please write up a spec in the
openstack-infra/infra-specs repo.  If you'd like to help with David's
proposal, the upstream Gerrit project is the best place to collaborate.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev