Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
hi, just so this issue doesn't get lost again. Mehdi's bp seems like a good place to track this issue: https://blueprints.launchpad.net/nova/+spec/usage-data-in-notification. Joe, i agree with you that it's too late for this iteration... maybe it's something we should mark low priority for J cycle. adding participants to the bp just so we get eyes on it. cheers, gordon chung openstack, ibm software standards___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On 02/26/2014 03:33 PM, Gordon Chung wrote: hi, just so this issue doesn't get lost again. Mehdi's bp seems like a good place to track this issue: https://blueprints.launchpad.net/nova/+spec/usage-data-in-notification. Joe, i agree with you that it's too late for this iteration... maybe it's something we should mark low priority for J cycle. adding participants to the bp just so we get eyes on it. cheers, gordon chung openstack, ibm software standards Ceilometer really needs to stop importing server projects in unit tests. By nature this is just going to break all the time. Cross project interactions need to be tested by something in the gate which is cross project gating - like tempest/devstack. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
Ceilometer really needs to stop importing server projects in unit tests. By nature this is just going to break all the time. i believe that was the takeaway from the thread -- it's an old thread and i was just doing some back-reading. Cross project interactions need to be tested by something in the gate which is cross project gating - like tempest/devstack. that said, we currently import swift for unit test as well to test a swift middleware solution which gathers metrics. i've added this as a bug here for discussion: https://bugs.launchpad.net/ceilometer/+bug/1285388 cheers, gordon chung openstack, ibm software standards___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Tue, Feb 04 2014, Joe Gordon wrote: Ceilometer running a plugin in nova is bad (for all the reasons previously discussed), Well, I partially disagree. Are you saying that nobody is allowed to run a plugin in Nova? So what are these plugins in the first place? Or if you're saying that Ceilometer cannot have plugins in Nova, I would like to know why. What is wrong, I agree, is that we have to use and mock nova internals to test our plugins. OTOH anyone writing plugin for Nova will have the same issue. To which extend this is a problem with the plugin system, I'll let everybody thing about it. :) So what can nova do to help this? It sounds like you have a valid use case that nova should support without requiring a plugin. We just need the possibility to run some code before an instance is deleted, in a synchronous manner – i.e. our code needs to be fully executed before Nova can actually destroyes the VM. -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
We don't have to add a new notification, but we have to add some new datas in the nova notifications. At least for the delete instance notification to remove the ceilometer nova notifier. A while ago, I have registered a blueprint that explains which datas are missing in the current nova notifications: https://blueprints.launchpad.net/nova/+spec/usage-data-in-notification https://wiki.openstack.org/wiki/Ceilometer/blueprints/remove-ceilometer-nova-notifier This seems like a much better way to do this. I'm not opposed to a nova plugin, but if it's something that lives outside the nova tree, I think there's going to be a problem of constantly chasing internal API changes. IMHO, a plugin should live (and be tested) in the nova tree and provide/consume a stableish API to/from Ceilometer. So, it seems like we've got the following options: 1. Provide the required additional data in our notifications to avoid the need for a plugin to hook into nova internals. 2. Continue to use a plugin in nova to scrape the additional data needed during certain events, but hopefully in a way that ties the plugin to the internal APIs in a maintainable way. Is that right? Personally, I think #1 is far superior to #2. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Wed, Feb 5, 2014 at 6:57 AM, Dan Smith d...@danplanet.com wrote: We don't have to add a new notification, but we have to add some new datas in the nova notifications. At least for the delete instance notification to remove the ceilometer nova notifier. A while ago, I have registered a blueprint that explains which datas are missing in the current nova notifications: https://blueprints.launchpad.net/nova/+spec/usage-data-in-notification https://wiki.openstack.org/wiki/Ceilometer/blueprints/remove-ceilometer-nova-notifier This seems like a much better way to do this. I'm not opposed to a nova plugin, but if it's something that lives outside the nova tree, I think there's going to be a problem of constantly chasing internal API changes. IMHO, a plugin should live (and be tested) in the nova tree and provide/consume a stableish API to/from Ceilometer. So, it seems like we've got the following options: 1. Provide the required additional data in our notifications to avoid the need for a plugin to hook into nova internals. 2. Continue to use a plugin in nova to scrape the additional data needed during certain events, but hopefully in a way that ties the plugin to the internal APIs in a maintainable way. Is that right? Personally, I think #1 is far superior to #2. ++ to #1. I am concerned about the timing of this and don't think we can do this by icehouse though. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Wed, Feb 5, 2014 at 1:35 AM, Julien Danjou jul...@danjou.info wrote: On Tue, Feb 04 2014, Joe Gordon wrote: Ceilometer running a plugin in nova is bad (for all the reasons previously discussed), Well, I partially disagree. Are you saying that nobody is allowed to run a plugin in Nova? So what are these plugins in the first place? Or if you're saying that Ceilometer cannot have plugins in Nova, I would like to know why. Dan explained it well in another email. Is the plugin run in devstack? I couldn't seem to find the code that enables the ceilometer plugin in nova. What is wrong, I agree, is that we have to use and mock nova internals to test our plugins. OTOH anyone writing plugin for Nova will have the same issue. To which extend this is a problem with the plugin system, I'll let everybody thing about it. :) Yup, the nova plugin system is not a stable API and we don't make any guarantees about plugins working after an upgrade. In other words plugins will inventively break (which is why I would prefer to not use them upstream). So what can nova do to help this? It sounds like you have a valid use case that nova should support without requiring a plugin. We just need the possibility to run some code before an instance is deleted, in a synchronous manner - i.e. our code needs to be fully executed before Nova can actually destroyes the VM. -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Wed, Feb 05 2014, Joe Gordon wrote: Dan explained it well in another email. Is the plugin run in devstack? I couldn't seem to find the code that enables the ceilometer plugin in nova. It was in devstack but got removed because it was hanging IIRC. There was a bug report about that. It didn't get love since them AFAIK. -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Wed, Feb 05 2014, Joe Gordon wrote: […] Is that right? Personally, I think #1 is far superior to #2. ++ to #1. I am concerned about the timing of this and don't think we can do this by icehouse though. #1 has been on the radar for – at least – a year. I don't have the courage to dig into the mailing list archive, but you'll find at least one or two threads about that subject already. -- Julien Danjou // Free Software hacker / independent consultant // http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Mon, Feb 03 2014, Joe Gordon wrote: We know, Ceilometer has been broken several times because of that in the past months. We know we shouldn't do that, but for now we don't have enough work force to work on a batter solution unfortunately. Does this issue mean Ceilometer won't work for the most literal definition of continuous deployment? They are tests intercepting the issue, so I don't think that invalidate the use of CD. Has this ever been a problem in the stable branches? It has been a problem only because we were referencing nova-master in test-requirements, but changing it to the reference the same version as Ceilometer (e.g. havana) fixed it. Whats the underlying problem here? nova notifications aren't versioned? Nova should try to support ceilometer's use case so it sounds like there is may be a nova issue in here as well. Oh you're far from it. Long story short, the problem is that when an instance is detroyed, we need to poll one last time for its CPU, IO, etc statistics to send them to Ceilometer. The only way we found to do that in Nova is to plug a special notification driver that intercepts the deletion notification in Nova, run the pollsters, and then returns to Nova execution. -- Julien Danjou /* Free Software hacker * independent consultant http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
Whats the underlying problem here? nova notifications aren't versioned? Nova should try to support ceilometer's use case so it sounds like there is may be a nova issue in here as well. Oh you're far from it. Long story short, the problem is that when an instance is detroyed, we need to poll one last time for its CPU, IO, etc statistics to send them to Ceilometer. The only way we found to do that in Nova is to plug a special notification driver that intercepts the deletion notification in Nova, run the pollsters, and then returns to Nova execution. Doesn't this just mean that Nova needs to do an extra poll and send an extra notification? Using a special notification driver, catching the delete notification, and polling one last time seems extremely fragile to me. It makes assumptions about the order things happen internally to nova, right? What can be done to make Ceilometer less of a bolt-on? That seems like the thing worth spending time discussing... --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Tue, Feb 4, 2014 at 2:01 AM, Julien Danjou jul...@danjou.info wrote: On Mon, Feb 03 2014, Joe Gordon wrote: We know, Ceilometer has been broken several times because of that in the past months. We know we shouldn't do that, but for now we don't have enough work force to work on a batter solution unfortunately. Does this issue mean Ceilometer won't rwork for the most literal definition of continuous deployment? They are tests intercepting the issue, so I don't think that invalidate the use of CD. Has this ever been a problem in the stable branches? It has been a problem only because we were referencing nova-master in test-requirements, but changing it to the reference the same version as Ceilometer (e.g. havana) fixed it. Whats the underlying problem here? nova notifications aren't versioned? Nova should try to support ceilometer's use case so it sounds like there is may be a nova issue in here as well. Oh you're far from it. Long story short, the problem is that when an instance is detroyed, we need to poll one last time for its CPU, IO, etc statistics to send them to Ceilometer. The only way we found to do that in Nova is to plug a special notification driver that intercepts the deletion notification in Nova, run the pollsters, and then returns to Nova execution. Ceilometer running a plugin in nova is bad (for all the reasons previously discussed), this broke ceilometer for in Havana (https://wiki.openstack.org/wiki/ReleaseNotes/Havana#Known_Issues_7). So what can nova do to help this? It sounds like you have a valid use case that nova should support without requiring a plugin. -- Julien Danjou /* Free Software hacker * independent consultant http://julien.danjou.info */ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On 05/02/14 10:07, Joe Gordon wrote: On Tue, Feb 4, 2014 at 2:01 AM, Julien Danjou jul...@danjou.info wrote: On Mon, Feb 03 2014, Joe Gordon wrote: We know, Ceilometer has been broken several times because of that in the past months. We know we shouldn't do that, but for now we don't have enough work force to work on a batter solution unfortunately. Does this issue mean Ceilometer won't rwork for the most literal definition of continuous deployment? They are tests intercepting the issue, so I don't think that invalidate the use of CD. Has this ever been a problem in the stable branches? It has been a problem only because we were referencing nova-master in test-requirements, but changing it to the reference the same version as Ceilometer (e.g. havana) fixed it. Whats the underlying problem here? nova notifications aren't versioned? Nova should try to support ceilometer's use case so it sounds like there is may be a nova issue in here as well. Oh you're far from it. Long story short, the problem is that when an instance is detroyed, we need to poll one last time for its CPU, IO, etc statistics to send them to Ceilometer. The only way we found to do that in Nova is to plug a special notification driver that intercepts the deletion notification in Nova, run the pollsters, and then returns to Nova execution. Ceilometer running a plugin in nova is bad (for all the reasons previously discussed), this broke ceilometer for in Havana (https://wiki.openstack.org/wiki/ReleaseNotes/Havana#Known_Issues_7). So what can nova do to help this? It sounds like you have a valid use case that nova should support without requiring a plugin. I would like to also point out that if we are handling the terminate case it would be good to handle: suspend, resize, soft_reboot, hard_reboot, shutdown, and rebuild. Pause still allows metric gathering, so that case is fine. Mainly because in all those cases cumulative data is lost because the metrics are reset before the next poll interval. It would be nice to have a poll occur before any shutdown/suspend and also on start/resume so we can final values and be certain that the next ceilometer sample is a reset value. Or in the case of rebuild/reboot/resize, a poll on event.start and event.end so we get final reading, and clear reset value. Otherwise we aren't getting accurate and consistent data. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Tue, Feb 04, 2014 at 01:11:10PM -0800, Dan Smith wrote: Whats the underlying problem here? nova notifications aren't versioned? Nova should try to support ceilometer's use case so it sounds like there is may be a nova issue in here as well. Oh you're far from it. Long story short, the problem is that when an instance is detroyed, we need to poll one last time for its CPU, IO, etc statistics to send them to Ceilometer. The only way we found to do that in Nova is to plug a special notification driver that intercepts the deletion notification in Nova, run the pollsters, and then returns to Nova execution. Doesn't this just mean that Nova needs to do an extra poll and send an extra notification? Using a special notification driver, catching the delete notification, and polling one last time seems extremely fragile to me. It makes assumptions about the order things happen internally to nova, right? What can be done to make Ceilometer less of a bolt-on? That seems like the thing worth spending time discussing... We don't have to add a new notification, but we have to add some new datas in the nova notifications. At least for the delete instance notification to remove the ceilometer nova notifier. A while ago, I have registered a blueprint that explains which datas are missing in the current nova notifications: https://blueprints.launchpad.net/nova/+spec/usage-data-in-notification https://wiki.openstack.org/wiki/Ceilometer/blueprints/remove-ceilometer-nova-notifier Regards, -- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht signature.asc Description: Digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
On Thu, Jan 30, 2014 at 2:28 AM, Julien Danjou jul...@danjou.info wrote: On Thu, Jan 30 2014, Joe Gordon wrote: Hi Joe, While looking at gate failures trying to improve our classification rate I stumbled across this: http://logs.openstack.org/50/67850/5/gate/gate-ceilometer-python26/8fd55b6/console.html It appears that ceilometer is pulling in the nova repo (http://git.openstack.org/cgit/openstack/ceilometer/tree/test-requirements.txt#n8) and running tests against non-contractual (private) APIs. This means that nova will occasionally break ceilometer unit tests. /me pat Joe on the back :) We know, Ceilometer has been broken several times because of that in the past months. We know we shouldn't do that, but for now we don't have enough work force to work on a batter solution unfortunately. Does this issue mean Ceilometer won't work for the most literal definition of continuous deployment? Has this ever been a problem in the stable branches? A fix has been proposed and I approved it. I don't know how to prevent that. We don't want to block Nova for making internal changes, as we're not doing something supported in Ceilometer. Whats the underlying problem here? nova notifications aren't versioned? Nova should try to support ceilometer's use case so it sounds like there is may be a nova issue in here as well. So, until now we lived with that burden and each breakage of our gate reminds us how important it would be to work on that. -- Julien Danjou // Free Software hacker / independent consultant // http://julien.danjou.info ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
It's also worth pointing out a related side effect of that choice... URL: http://logs.openstack.org/14/68714/3/gate/gate-ceilometer-python27/dc7e987/console.html#_2014-01-30_15_57_30_413 Uploads of branch tarballs are not stable and they're also not atomic. If your job tries to retrieve that tarball at the same time that it's being updated from a post-merge branch-tarball job, you will end up with a truncated file and your job will fail. The larger and more complex the tarball (for example nova's), the greater chance you have to catch it at just the wrong moment. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
