Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet neutron cli json output parsing.

2016-01-06 Thread Sofer Athlan-Guyot 
Mike Dorman <mdor...@godaddy.com> writes:

> I wonder if we should just refactor the Neutron provider to support
> either format? That way we stay independent from whatever the
> particular installation’s cliff/tablib situation is.
>
> We can probably safely assume that none of the Neutron objects will
> have attributes called ‘Field’ or ‘Value’, so it would be fairly easy
> to detect which format is there.

Hum, oki, so you need two parsers as the two outputs are different and a
"format detector" (cliff or clifftab).  Furthermore the clifftab always
returns its stuff in a json array, so "[{" was chosen as a starting tag.
Unfortunately cliff can return a json hash as well.  So you have to
somehow detect the start tag as well.

Considering that the first tag detection was the original problem[1], I
submitted this patch[2] which solves the problem for csv output quite
easily.  So with[3] we can have a not randomly buggy neutron parser.

That sound simpler than a tag detector + format detector + 2 parsers.

[1] https://bugs.launchpad.net/puppet-neutron/+bug/1508511
[2] https://review.openstack.org/#/c/263874/
[3] https://review.openstack.org/262809
>
> Mike
>
> From: Denis Egorenko <degore...@mirantis.com>
> Reply-To: "OpenStack Development Mailing List (not for usage
> questions)" <openstack-dev@lists.openstack.org>
> Date: Thursday, December 31, 2015 at 5:59 AM
> To: "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet
> neutron cli json output parsing.
>
> Last I checked, which was quite a while ago, openstackclient didn't
> support everything we were using from the neutron client.
>
> That's true. Openstack client doesn't support all features of neutron
> client [1].
>
> I would prefer use 3) option, but, unfortunately, i also don't see
> way, how to detect stevedore and cliff.
>
> [1]
> https://github.com/openstack/python-openstackclient/blob/master/setup.cfg#L329-
> L339
>
> 2015-12-30 19:53 GMT+03:00 Colleen Murphy <coll...@gazlene.net>:
>
> 
> 
> 
> 
> On Wed, Dec 30, 2015 at 8:37 AM, Sofer Athlan-Guyot
> <sathl...@redhat.com> wrote:
> 
> Hi,
> 
> I have added neutron people as they may help to find a
> solution.
> 
> After banging my head against the wall for a whole afternoon I
> discovered a serious bug in puppet neutron module.
> 
> I not going to repeat here the detail of the bug report[1].
> Basically:
> - neutron_port
> - neutron_subnet
> - neutron_router
> - neutron_network
> 
> may break idempotency randomly and won't work at all when
> clifftablib is
> removed from the package dependency of python-openstackclient
> (Mitaka[2])
> 
> So the problem is that neutron cli json output on liberty (at
> least, and
> maybe before) is not consistent and may come from cliff or
> clifftablib.
> I didn't test it but the same may apply to openstack cli. As
> we don't
> use the openstack cli json output it's not a issue (for puppet
> modules)
> 
> The available solution I can see are:
> 1. go back to parsing csv, shell output (revert [3])
> 2. find a way to leverage openstacklib parsing for neutron as
> well
> 3. keep json and parse the right output (cliff) and find a way
> to make
> sure that it is always used by stevedore
> 4. ?
> 
> Last I checked, which was quite a while ago, openstackclient didn't
> support everything we were using from the neutron client. I would
> like to reevaluate that and go with option 2 if we can. Otherwise
> option 1 seems reasonable.
> 
> From my point of view 3) is not a option, but other may disagree.
> 
> The problem is tricky and the fact that the CI cannot detect
> this is
> trickier[4].
> 
> So before Mitaka, the json parsing should go. I would love to
> see an
> interface that all puppet modules would use (solution 2). The
> current
> openstacklib parses openstack client well enough. The neutron
> command
> is not that different and I think there is space for code
> reuse. This
> would be a long term solution. It would bring the advantage of
> having
> only one interface to change if it was decided

Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet neutron cli json output parsing.

2016-01-04 Thread Mike Dorman 
I wonder if we should just refactor the Neutron provider to support either 
format?  That way we stay independent from whatever the particular 
installation’s cliff/tablib situation is.

We can probably safely assume that none of the Neutron objects will have 
attributes called ‘Field’ or ‘Value’, so it would be fairly easy to detect 
which format is there.

Mike


From: Denis Egorenko <degore...@mirantis.com<mailto:degore...@mirantis.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Thursday, December 31, 2015 at 5:59 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet neutron 
cli json output parsing.

Last I checked, which was quite a while ago, openstackclient didn't support 
everything we were using from the neutron client.

That's true. Openstack client doesn't support all features of neutron client 
[1].

I would prefer use 3) option, but, unfortunately, i also don't see way, how to 
detect stevedore and cliff.

[1] 
https://github.com/openstack/python-openstackclient/blob/master/setup.cfg#L329-L339

2015-12-30 19:53 GMT+03:00 Colleen Murphy 
<coll...@gazlene.net<mailto:coll...@gazlene.net>>:


On Wed, Dec 30, 2015 at 8:37 AM, Sofer Athlan-Guyot 
<sathl...@redhat.com<mailto:sathl...@redhat.com>> wrote:
Hi,

I have added neutron people as they may help to find a solution.

After banging my head against the wall for a whole afternoon I
discovered a serious bug in puppet neutron module.

I not going to repeat here the detail of the bug report[1].  Basically:
 - neutron_port
 - neutron_subnet
 - neutron_router
 - neutron_network

may break idempotency randomly and won't work at all when clifftablib is
removed from the package dependency of python-openstackclient
(Mitaka[2])

So the problem is that neutron cli json output on liberty (at least, and
maybe before) is not consistent and may come from cliff or clifftablib.
I didn't test it but the same may apply to openstack cli. As we don't
use the openstack cli json output it's not a issue (for puppet modules)

The available solution I can see are:
 1. go back to parsing csv, shell output (revert [3])
 2. find a way to leverage openstacklib parsing for neutron as well
 3. keep json and parse the right output (cliff) and find a way to make
sure that it is always used by stevedore
 4. ?

Last I checked, which was quite a while ago, openstackclient didn't support 
everything we were using from the neutron client. I would like to reevaluate 
that and go with option 2 if we can. Otherwise option 1 seems reasonable.

From my point of view 3) is not a option, but other may disagree.

The problem is tricky and the fact that the CI cannot detect this is
trickier[4].

So before Mitaka, the json parsing should go.  I would love to see an
interface that all puppet modules would use (solution 2).  The current
openstacklib parses openstack client well enough.  The neutron command
is not that different and I think there is space for code reuse.  This
would be a long term solution.  It would bring the advantage of having
only one interface to change if it was decided to use the API directly
for instance[5]

In the meantime, a quick solution to this bug must be found.

Looking forward to your comments.

Regards,

[1] https://bugs.launchpad.net/puppet-neutron/+bug/1530163
[2] https://bugs.launchpad.net/python-neutronclient/+bug/1529914
[3] https://review.openstack.org/#/c/238156/
[4] https://review.openstack.org/#/c/262223/
[5] http://lists.openstack.org/pipermail/openstack-dev/2015-October/076439.html
--
Sofer Athlan-Guyot

Colleen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Best Regards,
Egorenko Denis,
Deployment Engineer
Mirantis
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet neutron cli json output parsing.

2015-12-31 Thread Denis Egorenko 
>
> Last I checked, which was quite a while ago, openstackclient didn't
> support everything we were using from the neutron client.


That's true. Openstack client doesn't support all features of neutron
client [1].

I would prefer use 3) option, but, unfortunately, i also don't see way, how
to detect stevedore and cliff.

[1]
https://github.com/openstack/python-openstackclient/blob/master/setup.cfg#L329-L339

2015-12-30 19:53 GMT+03:00 Colleen Murphy :

>
>
> On Wed, Dec 30, 2015 at 8:37 AM, Sofer Athlan-Guyot 
> wrote:
>
>> Hi,
>>
>> I have added neutron people as they may help to find a solution.
>>
>> After banging my head against the wall for a whole afternoon I
>> discovered a serious bug in puppet neutron module.
>>
>> I not going to repeat here the detail of the bug report[1].  Basically:
>>  - neutron_port
>>  - neutron_subnet
>>  - neutron_router
>>  - neutron_network
>>
>> may break idempotency randomly and won't work at all when clifftablib is
>> removed from the package dependency of python-openstackclient
>> (Mitaka[2])
>>
>> So the problem is that neutron cli json output on liberty (at least, and
>> maybe before) is not consistent and may come from cliff or clifftablib.
>> I didn't test it but the same may apply to openstack cli. As we don't
>> use the openstack cli json output it's not a issue (for puppet modules)
>>
>> The available solution I can see are:
>>  1. go back to parsing csv, shell output (revert [3])
>>  2. find a way to leverage openstacklib parsing for neutron as well
>>  3. keep json and parse the right output (cliff) and find a way to make
>> sure that it is always used by stevedore
>>  4. ?
>>
>> Last I checked, which was quite a while ago, openstackclient didn't
> support everything we were using from the neutron client. I would like to
> reevaluate that and go with option 2 if we can. Otherwise option 1 seems
> reasonable.
>
>
>> From my point of view 3) is not a option, but other may disagree.
>>
>> The problem is tricky and the fact that the CI cannot detect this is
>> trickier[4].
>>
>> So before Mitaka, the json parsing should go.  I would love to see an
>> interface that all puppet modules would use (solution 2).  The current
>> openstacklib parses openstack client well enough.  The neutron command
>> is not that different and I think there is space for code reuse.  This
>> would be a long term solution.  It would bring the advantage of having
>> only one interface to change if it was decided to use the API directly
>> for instance[5]
>>
>> In the meantime, a quick solution to this bug must be found.
>>
>> Looking forward to your comments.
>>
>> Regards,
>>
>> [1] https://bugs.launchpad.net/puppet-neutron/+bug/1530163
>> [2] https://bugs.launchpad.net/python-neutronclient/+bug/1529914
>> [3] https://review.openstack.org/#/c/238156/
>> [4] https://review.openstack.org/#/c/262223/
>> [5]
>> http://lists.openstack.org/pipermail/openstack-dev/2015-October/076439.html
>> --
>> Sofer Athlan-Guyot
>>
>> Colleen
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Best Regards,
Egorenko Denis,
Deployment Engineer
Mirantis
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet neutron cli json output parsing.

2015-12-31 Thread Sofer Athlan-Guyot 
Denis Egorenko  writes:

> Last I checked, which was quite a while ago, openstackclient didn't
> support everything we were using from the neutron client.
>
> That's true. Openstack client doesn't support all features of neutron
> client [1].

What I had in mind was more like using the same parsing for both
"neutron" and "openstack" cli.  This would require quite a lot of
refactoring though.  But if we had a common "parsing backend", this
would be great IMHO.

>
> I would prefer use 3) option, but, unfortunately, i also don't see
> way, how to detect stevedore and cliff.

For 3) it really looks like a impossible mission to me.

To give flesh to option 1), I submitted this patch[1].  I
put it in WIP as
 - more people may have other view when the Christmas holiday are off;
 - https://bugs.launchpad.net/puppet-neutron/+bug/1508511 should be
   fixed in this patch as well.

[1]: https://review.openstack.org/#/c/262809/1

>
> [1]
> https://github.com/openstack/python-openstackclient/blob/master/setup.cfg#L329-
> L339
>
> 2015-12-30 19:53 GMT+03:00 Colleen Murphy :
>
> 
> 
> 
> 
> On Wed, Dec 30, 2015 at 8:37 AM, Sofer Athlan-Guyot
>  wrote:
> 
> Hi,
> 
> I have added neutron people as they may help to find a
> solution.
> 
> After banging my head against the wall for a whole afternoon I
> discovered a serious bug in puppet neutron module.
> 
> I not going to repeat here the detail of the bug report[1].
> Basically:
> - neutron_port
> - neutron_subnet
> - neutron_router
> - neutron_network
> 
> may break idempotency randomly and won't work at all when
> clifftablib is
> removed from the package dependency of python-openstackclient
> (Mitaka[2])
> 
> So the problem is that neutron cli json output on liberty (at
> least, and
> maybe before) is not consistent and may come from cliff or
> clifftablib.
> I didn't test it but the same may apply to openstack cli. As
> we don't
> use the openstack cli json output it's not a issue (for puppet
> modules)
> 
> The available solution I can see are:
> 1. go back to parsing csv, shell output (revert [3])
> 2. find a way to leverage openstacklib parsing for neutron as
> well
> 3. keep json and parse the right output (cliff) and find a way
> to make
> sure that it is always used by stevedore
> 4. ?
> 
> Last I checked, which was quite a while ago, openstackclient didn't
> support everything we were using from the neutron client. I would
> like to reevaluate that and go with option 2 if we can. Otherwise
> option 1 seems reasonable.
> 
> From my point of view 3) is not a option, but other may disagree.
> 
> The problem is tricky and the fact that the CI cannot detect
> this is
> trickier[4].
> 
> So before Mitaka, the json parsing should go. I would love to
> see an
> interface that all puppet modules would use (solution 2). The
> current
> openstacklib parses openstack client well enough. The neutron
> command
> is not that different and I think there is space for code
> reuse. This
> would be a long term solution. It would bring the advantage of
> having
> only one interface to change if it was decided to use the API
> directly
> for instance[5]
> 
> In the meantime, a quick solution to this bug must be found.
> 
> Looking forward to your comments.
> 
> Regards,
> 
> [1] https://bugs.launchpad.net/puppet-neutron/+bug/1530163
> [2]
> https://bugs.launchpad.net/python-neutronclient/+bug/1529914
> [3] https://review.openstack.org/#/c/238156/
> [4] https://review.openstack.org/#/c/262223/
> [5]
> 
> http://lists.openstack.org/pipermail/openstack-dev/2015-October/076439.
> html
> --
> Sofer Athlan-Guyot
> 
> Colleen 
>
> ___
> ___
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Sofer Athlan-Guyot

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] [neutron] Serious bug in puppet neutron cli json output parsing.

2015-12-30 Thread Sofer Athlan-Guyot 
Hi,

I have added neutron people as they may help to find a solution.

After banging my head against the wall for a whole afternoon I
discovered a serious bug in puppet neutron module.

I not going to repeat here the detail of the bug report[1].  Basically:
 - neutron_port
 - neutron_subnet
 - neutron_router
 - neutron_network

may break idempotency randomly and won't work at all when clifftablib is
removed from the package dependency of python-openstackclient
(Mitaka[2])

So the problem is that neutron cli json output on liberty (at least, and
maybe before) is not consistent and may come from cliff or clifftablib.
I didn't test it but the same may apply to openstack cli. As we don't
use the openstack cli json output it's not a issue (for puppet modules)

The available solution I can see are:
 1. go back to parsing csv, shell output (revert [3])
 2. find a way to leverage openstacklib parsing for neutron as well
 3. keep json and parse the right output (cliff) and find a way to make
sure that it is always used by stevedore
 4. ?

>From my point of view 3) is not a option, but other may disagree.

The problem is tricky and the fact that the CI cannot detect this is
trickier[4].

So before Mitaka, the json parsing should go.  I would love to see an
interface that all puppet modules would use (solution 2).  The current
openstacklib parses openstack client well enough.  The neutron command
is not that different and I think there is space for code reuse.  This
would be a long term solution.  It would bring the advantage of having
only one interface to change if it was decided to use the API directly
for instance[5]

In the meantime, a quick solution to this bug must be found.

Looking forward to your comments.

Regards,

[1] https://bugs.launchpad.net/puppet-neutron/+bug/1530163
[2] https://bugs.launchpad.net/python-neutronclient/+bug/1529914
[3] https://review.openstack.org/#/c/238156/
[4] https://review.openstack.org/#/c/262223/
[5] http://lists.openstack.org/pipermail/openstack-dev/2015-October/076439.html
-- 
Sofer Athlan-Guyot

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet neutron cli json output parsing.

2015-12-30 Thread Colleen Murphy 
On Wed, Dec 30, 2015 at 8:37 AM, Sofer Athlan-Guyot 
wrote:

> Hi,
>
> I have added neutron people as they may help to find a solution.
>
> After banging my head against the wall for a whole afternoon I
> discovered a serious bug in puppet neutron module.
>
> I not going to repeat here the detail of the bug report[1].  Basically:
>  - neutron_port
>  - neutron_subnet
>  - neutron_router
>  - neutron_network
>
> may break idempotency randomly and won't work at all when clifftablib is
> removed from the package dependency of python-openstackclient
> (Mitaka[2])
>
> So the problem is that neutron cli json output on liberty (at least, and
> maybe before) is not consistent and may come from cliff or clifftablib.
> I didn't test it but the same may apply to openstack cli. As we don't
> use the openstack cli json output it's not a issue (for puppet modules)
>
> The available solution I can see are:
>  1. go back to parsing csv, shell output (revert [3])
>  2. find a way to leverage openstacklib parsing for neutron as well
>  3. keep json and parse the right output (cliff) and find a way to make
> sure that it is always used by stevedore
>  4. ?
>
> Last I checked, which was quite a while ago, openstackclient didn't
support everything we were using from the neutron client. I would like to
reevaluate that and go with option 2 if we can. Otherwise option 1 seems
reasonable.


> From my point of view 3) is not a option, but other may disagree.
>
> The problem is tricky and the fact that the CI cannot detect this is
> trickier[4].
>
> So before Mitaka, the json parsing should go.  I would love to see an
> interface that all puppet modules would use (solution 2).  The current
> openstacklib parses openstack client well enough.  The neutron command
> is not that different and I think there is space for code reuse.  This
> would be a long term solution.  It would bring the advantage of having
> only one interface to change if it was decided to use the API directly
> for instance[5]
>
> In the meantime, a quick solution to this bug must be found.
>
> Looking forward to your comments.
>
> Regards,
>
> [1] https://bugs.launchpad.net/puppet-neutron/+bug/1530163
> [2] https://bugs.launchpad.net/python-neutronclient/+bug/1529914
> [3] https://review.openstack.org/#/c/238156/
> [4] https://review.openstack.org/#/c/262223/
> [5]
> http://lists.openstack.org/pipermail/openstack-dev/2015-October/076439.html
> --
> Sofer Athlan-Guyot
>
> Colleen
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev