Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Thierry Carrez
David Kranz wrote: On 08/27/2014 03:43 PM, Sean Dague wrote: On 08/27/2014 03:33 PM, David Kranz wrote: Race conditions are what makes debugging very hard. I think we are in the process of experimenting with such an idea: asymetric gating by moving functional tests to projects, making them

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Doug Hellmann
On Aug 27, 2014, at 5:56 PM, Sean Dague s...@dague.net wrote: On 08/27/2014 05:27 PM, Doug Hellmann wrote: On Aug 27, 2014, at 2:54 PM, Sean Dague s...@dague.net wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote: On

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Dean Troyer
On Thu, Aug 28, 2014 at 11:48 AM, Doug Hellmann d...@doughellmann.com wrote: In my case, a neutron call failed. Most of the other services seem to have a *-api.log file, but neutron doesn’t. It took a little while to find the API-related messages in screen-q-svc.txt (I’m glad I’ve been around

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Sean Dague
On 08/28/2014 12:48 PM, Doug Hellmann wrote: On Aug 27, 2014, at 5:56 PM, Sean Dague s...@dague.net wrote: On 08/27/2014 05:27 PM, Doug Hellmann wrote: On Aug 27, 2014, at 2:54 PM, Sean Dague s...@dague.net wrote: Note: thread intentionally broken, this is really a different topic. On

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Doug Hellmann
On Aug 28, 2014, at 1:00 PM, Dean Troyer dtro...@gmail.com wrote: On Thu, Aug 28, 2014 at 11:48 AM, Doug Hellmann d...@doughellmann.com wrote: In my case, a neutron call failed. Most of the other services seem to have a *-api.log file, but neutron doesn’t. It took a little while to find the

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Doug Hellmann
On Aug 28, 2014, at 1:17 PM, Sean Dague s...@dague.net wrote: On 08/28/2014 12:48 PM, Doug Hellmann wrote: On Aug 27, 2014, at 5:56 PM, Sean Dague s...@dague.net wrote: On 08/27/2014 05:27 PM, Doug Hellmann wrote: On Aug 27, 2014, at 2:54 PM, Sean Dague s...@dague.net wrote: Note:

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Joe Gordon
On Thu, Aug 28, 2014 at 10:17 AM, Sean Dague s...@dague.net wrote: On 08/28/2014 12:48 PM, Doug Hellmann wrote: On Aug 27, 2014, at 5:56 PM, Sean Dague s...@dague.net wrote: On 08/27/2014 05:27 PM, Doug Hellmann wrote: On Aug 27, 2014, at 2:54 PM, Sean Dague s...@dague.net wrote:

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Dean Troyer
On Thu, Aug 28, 2014 at 12:44 PM, Doug Hellmann d...@doughellmann.com wrote: I usually use the functions for editing ENABLED_SERVICES. Is it still common to edit the variable directly? Not generally. It was looking at it in log files to see what was/was not enabled where I started to think

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Sean Dague
On 08/28/2014 01:48 PM, Doug Hellmann wrote: On Aug 28, 2014, at 1:17 PM, Sean Dague s...@dague.net wrote: On 08/28/2014 12:48 PM, Doug Hellmann wrote: On Aug 27, 2014, at 5:56 PM, Sean Dague s...@dague.net wrote: On 08/27/2014 05:27 PM, Doug Hellmann wrote: On Aug 27, 2014, at 2:54

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Sean Dague
On 08/28/2014 02:07 PM, Joe Gordon wrote: On Thu, Aug 28, 2014 at 10:17 AM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: On 08/28/2014 12:48 PM, Doug Hellmann wrote: On Aug 27, 2014, at 5:56 PM, Sean Dague s...@dague.net mailto:s...@dague.net wrote:

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Joe Gordon
On Thu, Aug 28, 2014 at 5:17 AM, Thierry Carrez thie...@openstack.org wrote: David Kranz wrote: On 08/27/2014 03:43 PM, Sean Dague wrote: On 08/27/2014 03:33 PM, David Kranz wrote: Race conditions are what makes debugging very hard. I think we are in the process of experimenting with

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Doug Hellmann
On Aug 28, 2014, at 2:16 PM, Sean Dague s...@dague.net wrote: On 08/28/2014 02:07 PM, Joe Gordon wrote: On Thu, Aug 28, 2014 at 10:17 AM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: On 08/28/2014 12:48 PM, Doug Hellmann wrote: On Aug 27, 2014, at 5:56 PM, Sean Dague

Re: [openstack-dev] [all] gate debugging

2014-08-28 Thread Doug Hellmann
On Aug 28, 2014, at 2:15 PM, Sean Dague s...@dague.net wrote: On 08/28/2014 01:48 PM, Doug Hellmann wrote: On Aug 28, 2014, at 1:17 PM, Sean Dague s...@dague.net wrote: On 08/28/2014 12:48 PM, Doug Hellmann wrote: On Aug 27, 2014, at 5:56 PM, Sean Dague s...@dague.net wrote: On

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread Jeremy Stanley
On 2014-08-27 14:54:55 -0400 (-0400), Sean Dague wrote: [...] I think we break down on communication when we get into a conversation of I want to learn gate debugging because I don't quite know what that means, or where the starting point of understanding is. So those intentions are well

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread David Kranz
On 08/27/2014 02:54 PM, Sean Dague wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote: On Aug 27, 2014, at 1:30 PM, Chris Dent chd...@redhat.com wrote: On Wed, 27 Aug 2014, Doug Hellmann wrote: I have found it immensely

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread Sean Dague
On 08/27/2014 03:33 PM, David Kranz wrote: On 08/27/2014 02:54 PM, Sean Dague wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote: On Aug 27, 2014, at 1:30 PM, Chris Dent chd...@redhat.com wrote: On Wed, 27 Aug 2014, Doug

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread David Kranz
On 08/27/2014 03:43 PM, Sean Dague wrote: On 08/27/2014 03:33 PM, David Kranz wrote: On 08/27/2014 02:54 PM, Sean Dague wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote: On Aug 27, 2014, at 1:30 PM, Chris Dent

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread Anita Kuno
On 08/27/2014 03:43 PM, Sean Dague wrote: On 08/27/2014 03:33 PM, David Kranz wrote: On 08/27/2014 02:54 PM, Sean Dague wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote: On Aug 27, 2014, at 1:30 PM, Chris Dent

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread Doug Hellmann
On Aug 27, 2014, at 5:27 PM, Doug Hellmann d...@doughellmann.com wrote: On Aug 27, 2014, at 2:54 PM, Sean Dague s...@dague.net wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote: On Aug 27, 2014, at 1:30 PM, Chris

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread Sean Dague
On 08/27/2014 05:27 PM, Doug Hellmann wrote: On Aug 27, 2014, at 2:54 PM, Sean Dague s...@dague.net wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote: On Aug 27, 2014, at 1:30 PM, Chris Dent chd...@redhat.com wrote:

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread Matthew Treinish
On Wed, Aug 27, 2014 at 05:47:09PM -0400, Doug Hellmann wrote: On Aug 27, 2014, at 5:27 PM, Doug Hellmann d...@doughellmann.com wrote: On Aug 27, 2014, at 2:54 PM, Sean Dague s...@dague.net wrote: Note: thread intentionally broken, this is really a different topic. On

Re: [openstack-dev] [all] gate debugging

2014-08-27 Thread Salvatore Orlando
As it has been pointed out previously in this thread debugging gate failures is mostly about chasing race conditions, which in some cases involve the most disparate interactions between Openstack services [1]. Finding the root cause of these races is a mix of knowledge, pragmatism, and luck.