Re: [openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
On 06/29/2014 07:59 PM, Anita Kuno wrote: On 06/29/2014 07:43 PM, Anita Kuno wrote: On 06/29/2014 03:25 PM, Ilya Shakhat wrote: Hi! During last couple weeks there is an increasing demand on tracking 3rd-party CI statuses. We at Stackalytics decided to be in trend and (with some inspiration from Salvatore's proposal) implemented report that shows summary on external CI status. The initial version is available for Neutron - http://stackalytics.com/report/ci/neutron/7 The report shows summary of all CI jobs during specified period of time, including: * stats of runs on merged patch sets: - total number of runs - success rate (success to total ratio) - time of the latest run - last test result * stats for all patch sets (the same set as for merged) * last test results for every merged patch set grouped by days (useful to see how different CIs correlate with each other and how often they run) Under merged patch set report means the last patch set in the merged change request, thus it is almost the same as the trunk code. CI configuration is taken from DriverLog's default data https://git.openstack.org/cgit/stackforge/driverlog/tree/etc/default_data.json. Standard Stackalytics screen is also available for CIs - http://stackalytics.com/?metric=ci, including votes breakdown and activity log. Since this is the first version there are some open questions: * Currently report shows results per CI id, but there are CIs that run tests against multiple drivers and this case is not supported. What would be more useful: to get stats for a driver or for CI? * Most CIs run tests when patch set is posted. So even if change request is merged within selected time period corresponding CI results may be missing. * Patterns for non-voting CIs need to be verified. For example Cisco CI now runs 5 jobs, but DriverLog data still contains old data. Thanks, Ilya 2014-06-16 17:52 GMT+04:00 Salvatore Orlando sorla...@nicira.com: However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Ilya: I look forward to hearing more about this dashboard and ensuring you or someone else associated with this dashboard are available for questions at the third party meeting tomorrow: https://wiki.openstack.org/wiki/Meetings/ThirdParty We missed you last week. Thanks Ilya, Anita. And one question I will have when we discuss this is regarding this statement: Green cell - tests ran successfully, Currently we don't have community consensus around the use of the statement tests ran successfully regarding third party ci systems. This is as statement, you recall, we had discussed at the third party meeting when we talked about driverlog. 18:46:02 krtaylor but what does CI tested really mean? just running tests? or tested to pass some level of requirements? http://eavesdrop.openstack.org/meetings/third_party/2014/third_party.2014-06-16-18.00.log.html Using a statement to convey success prior to having a definition from the community about what the statement means will create confusion in the community and frustration from folks, including myself, who are willing to have the conversation about what it means, who feel circumvented by this second use of a phrase which implies a decided meaning where none yet exists. Please participate in conversations around the definition of phrases of success and failure regarding third party systems and point to the logs where consensus is reached prior it its use in future. In addition to attending the third party meeting, please attend the infra meeting or the qa meeting, and hopefully meetings that include programs that have interactions with third party ci systems including nova, neutron, and cinder (if there are other programs interacting with third party ci systems please attend the third party meeting so I know about you). Thanks Ilya, I look forward to our future discussions, Anita. As an update, Ilya did attend the third party meeting today, thank you Ilya. I am disappointed to realize that Ilya (or stackalytics, I don't know where this is coming from) is unwilling to cease making up definitions of success
Re: [openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
On 30 June 2014 21:08, Anita Kuno ante...@anteaya.info wrote: I am disappointed to realize that Ilya (or stackalytics, I don't know where this is coming from) is unwilling to cease making up definitions of success for third party ci systems to allow the openstack community to arrive at its own definition. There is indeed a risk that the new dashboards won't give a meaningful view of whether a 3rd party CI is voting correctly or not. However, there is an elephant in the room and a much more important problem: To measure how accurately a CI is voting says much more about a driver author's Gerrit-fu ability to operate a CI system than it does about whether the code they have contributed to OpenStack actually works, and the latter is what is actually important. To my mind the whole 3rd party testing discussion should refocus on helping developers maintain good working code and less on waving you will be kicked out of OpenStack if you don't keep your swarm of nebulous daemons running 24/7. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
On 06/30/2014 03:27 PM, Luke Gorrie wrote: On 30 June 2014 21:08, Anita Kuno ante...@anteaya.info wrote: I am disappointed to realize that Ilya (or stackalytics, I don't know where this is coming from) is unwilling to cease making up definitions of success for third party ci systems to allow the openstack community to arrive at its own definition. There is indeed a risk that the new dashboards won't give a meaningful view of whether a 3rd party CI is voting correctly or not. However, there is an elephant in the room and a much more important problem: To measure how accurately a CI is voting says much more about a driver author's Gerrit-fu ability to operate a CI system than it does about whether the code they have contributed to OpenStack actually works, and the latter is what is actually important. To my mind the whole 3rd party testing discussion should refocus on helping developers maintain good working code and less on waving you will be kicked out of OpenStack if you don't keep your swarm of nebulous daemons running 24/7. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Well I am glad you have reached that conclusion, Luke. Given that I gave you months of my time with that exact approach and you squandered it away, claiming I wasn't doing enough to help you, I am glad you have finally reached a productive place. I hope your direction produces results. Thanks Luke, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
Well, Luke, this is collaborative effort by everybody. By having these CI systems in place ensures that one person's code does not break other person's code and vice versa. Therefore, having these CI systems operational and voting 24x7 is a critical step in achieving this goal. However, the details as to how and what should be tested is definitely debatable and the team has done excellent job in converging on that. Now, as to the issue at hand which Anita is describing, I attended the meeting this morning and was very pleased with the debate that took place and the definition as to Sucess On Mon, Jun 30, 2014 at 12:27 PM, Luke Gorrie l...@tail-f.com wrote: On 30 June 2014 21:08, Anita Kuno ante...@anteaya.info wrote: I am disappointed to realize that Ilya (or stackalytics, I don't know where this is coming from) is unwilling to cease making up definitions of success for third party ci systems to allow the openstack community to arrive at its own definition. There is indeed a risk that the new dashboards won't give a meaningful view of whether a 3rd party CI is voting correctly or not. However, there is an elephant in the room and a much more important problem: To measure how accurately a CI is voting says much more about a driver author's Gerrit-fu ability to operate a CI system than it does about whether the code they have contributed to OpenStack actually works, and the latter is what is actually important. To my mind the whole 3rd party testing discussion should refocus on helping developers maintain good working code and less on waving you will be kicked out of OpenStack if you don't keep your swarm of nebulous daemons running 24/7. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
Sorry, accidentally hit the wrong key and message went out. Was making a mention about the definition of Success. I thought the debate in the meeting was very productive - when a CI posts a +1 that is success, and when a CI posts a -1 (or no vote with comment) is also a success - as this reflects that the CI is doing what it is suppose to do. So, when it comes to stackalytics, it is more critical to show if a given CI is operational or not - and for how long? Another thing we can debate is how to present the +1/-1 votes by a given CI - unless we have some benchmark, it will be hard to consider success/failure. So, I am of the opinion that, initially, we only report on the operational status and duration of the CIs, and a counter of +1 and -1 votes over a period of time. For example, looking at Arista CI, it has casted 7,958 votes so far and it has been operational for past 6 months. This information is not available anywhere - hence, presenting this kind of information on a dashboard created by Ilya would be very useful to the community as well to the vendors.. thoughts? -Sukhdev On Mon, Jun 30, 2014 at 12:49 PM, Sukhdev Kapur sukhdevka...@gmail.com wrote: Well, Luke, this is collaborative effort by everybody. By having these CI systems in place ensures that one person's code does not break other person's code and vice versa. Therefore, having these CI systems operational and voting 24x7 is a critical step in achieving this goal. However, the details as to how and what should be tested is definitely debatable and the team has done excellent job in converging on that. Now, as to the issue at hand which Anita is describing, I attended the meeting this morning and was very pleased with the debate that took place and the definition as to Sucess On Mon, Jun 30, 2014 at 12:27 PM, Luke Gorrie l...@tail-f.com wrote: On 30 June 2014 21:08, Anita Kuno ante...@anteaya.info wrote: I am disappointed to realize that Ilya (or stackalytics, I don't know where this is coming from) is unwilling to cease making up definitions of success for third party ci systems to allow the openstack community to arrive at its own definition. There is indeed a risk that the new dashboards won't give a meaningful view of whether a 3rd party CI is voting correctly or not. However, there is an elephant in the room and a much more important problem: To measure how accurately a CI is voting says much more about a driver author's Gerrit-fu ability to operate a CI system than it does about whether the code they have contributed to OpenStack actually works, and the latter is what is actually important. To my mind the whole 3rd party testing discussion should refocus on helping developers maintain good working code and less on waving you will be kicked out of OpenStack if you don't keep your swarm of nebulous daemons running 24/7. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
Hi! During last couple weeks there is an increasing demand on tracking 3rd-party CI statuses. We at Stackalytics decided to be in trend and (with some inspiration from Salvatore's proposal) implemented report that shows summary on external CI status. The initial version is available for Neutron - http://stackalytics.com/report/ci/neutron/7 The report shows summary of all CI jobs during specified period of time, including: * stats of runs on merged patch sets: - total number of runs - success rate (success to total ratio) - time of the latest run - last test result * stats for all patch sets (the same set as for merged) * last test results for every merged patch set grouped by days (useful to see how different CIs correlate with each other and how often they run) Under merged patch set report means the last patch set in the merged change request, thus it is almost the same as the trunk code. CI configuration is taken from DriverLog's default data https://git.openstack.org/cgit/stackforge/driverlog/tree/etc/default_data.json. Standard Stackalytics screen is also available for CIs - http://stackalytics.com/?metric=ci, including votes breakdown and activity log. Since this is the first version there are some open questions: * Currently report shows results per CI id, but there are CIs that run tests against multiple drivers and this case is not supported. What would be more useful: to get stats for a driver or for CI? * Most CIs run tests when patch set is posted. So even if change request is merged within selected time period corresponding CI results may be missing. * Patterns for non-voting CIs need to be verified. For example Cisco CI now runs 5 jobs, but DriverLog data still contains old data. Thanks, Ilya 2014-06-16 17:52 GMT+04:00 Salvatore Orlando sorla...@nicira.com: However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
On 06/29/2014 03:25 PM, Ilya Shakhat wrote: Hi! During last couple weeks there is an increasing demand on tracking 3rd-party CI statuses. We at Stackalytics decided to be in trend and (with some inspiration from Salvatore's proposal) implemented report that shows summary on external CI status. The initial version is available for Neutron - http://stackalytics.com/report/ci/neutron/7 The report shows summary of all CI jobs during specified period of time, including: * stats of runs on merged patch sets: - total number of runs - success rate (success to total ratio) - time of the latest run - last test result * stats for all patch sets (the same set as for merged) * last test results for every merged patch set grouped by days (useful to see how different CIs correlate with each other and how often they run) Under merged patch set report means the last patch set in the merged change request, thus it is almost the same as the trunk code. CI configuration is taken from DriverLog's default data https://git.openstack.org/cgit/stackforge/driverlog/tree/etc/default_data.json. Standard Stackalytics screen is also available for CIs - http://stackalytics.com/?metric=ci, including votes breakdown and activity log. Since this is the first version there are some open questions: * Currently report shows results per CI id, but there are CIs that run tests against multiple drivers and this case is not supported. What would be more useful: to get stats for a driver or for CI? * Most CIs run tests when patch set is posted. So even if change request is merged within selected time period corresponding CI results may be missing. * Patterns for non-voting CIs need to be verified. For example Cisco CI now runs 5 jobs, but DriverLog data still contains old data. Thanks, Ilya 2014-06-16 17:52 GMT+04:00 Salvatore Orlando sorla...@nicira.com: However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Ilya: I look forward to hearing more about this dashboard and ensuring you or someone else associated with this dashboard are available for questions at the third party meeting tomorrow: https://wiki.openstack.org/wiki/Meetings/ThirdParty We missed you last week. Thanks Ilya, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
On 06/29/2014 07:43 PM, Anita Kuno wrote: On 06/29/2014 03:25 PM, Ilya Shakhat wrote: Hi! During last couple weeks there is an increasing demand on tracking 3rd-party CI statuses. We at Stackalytics decided to be in trend and (with some inspiration from Salvatore's proposal) implemented report that shows summary on external CI status. The initial version is available for Neutron - http://stackalytics.com/report/ci/neutron/7 The report shows summary of all CI jobs during specified period of time, including: * stats of runs on merged patch sets: - total number of runs - success rate (success to total ratio) - time of the latest run - last test result * stats for all patch sets (the same set as for merged) * last test results for every merged patch set grouped by days (useful to see how different CIs correlate with each other and how often they run) Under merged patch set report means the last patch set in the merged change request, thus it is almost the same as the trunk code. CI configuration is taken from DriverLog's default data https://git.openstack.org/cgit/stackforge/driverlog/tree/etc/default_data.json. Standard Stackalytics screen is also available for CIs - http://stackalytics.com/?metric=ci, including votes breakdown and activity log. Since this is the first version there are some open questions: * Currently report shows results per CI id, but there are CIs that run tests against multiple drivers and this case is not supported. What would be more useful: to get stats for a driver or for CI? * Most CIs run tests when patch set is posted. So even if change request is merged within selected time period corresponding CI results may be missing. * Patterns for non-voting CIs need to be verified. For example Cisco CI now runs 5 jobs, but DriverLog data still contains old data. Thanks, Ilya 2014-06-16 17:52 GMT+04:00 Salvatore Orlando sorla...@nicira.com: However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Ilya: I look forward to hearing more about this dashboard and ensuring you or someone else associated with this dashboard are available for questions at the third party meeting tomorrow: https://wiki.openstack.org/wiki/Meetings/ThirdParty We missed you last week. Thanks Ilya, Anita. And one question I will have when we discuss this is regarding this statement: Green cell - tests ran successfully, Currently we don't have community consensus around the use of the statement tests ran successfully regarding third party ci systems. This is as statement, you recall, we had discussed at the third party meeting when we talked about driverlog. 18:46:02 krtaylor but what does CI tested really mean? just running tests? or tested to pass some level of requirements? http://eavesdrop.openstack.org/meetings/third_party/2014/third_party.2014-06-16-18.00.log.html Using a statement to convey success prior to having a definition from the community about what the statement means will create confusion in the community and frustration from folks, including myself, who are willing to have the conversation about what it means, who feel circumvented by this second use of a phrase which implies a decided meaning where none yet exists. Please participate in conversations around the definition of phrases of success and failure regarding third party systems and point to the logs where consensus is reached prior it its use in future. In addition to attending the third party meeting, please attend the infra meeting or the qa meeting, and hopefully meetings that include programs that have interactions with third party ci systems including nova, neutron, and cinder (if there are other programs interacting with third party ci systems please attend the third party meeting so I know about you). Thanks Ilya, I look forward to our future discussions, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
