Re: [openstack-dev] [puppet] Config support for oslo.config.cfg.MultiStrOpt

[Martin Mágr]

On 12/04/2015 11:21 PM, Emilien Macchi wrote:


On 12/02/2015 10:32 PM, Cody Herriges wrote:

Martin,

I see no reason this shouldn't just be pushed into puppetlabs-inifile.
I can't actually find a real "spec" for INI file and even the Wiki
link[3] calls out that there is no actual spec.

I suggest:

1/ we land https://review.openstack.org/#/c/234727/
2/ in the meantime, we work on a puppetlabs-inifile patch
3/ once it's done, we switch puppet-openstacklib to use it.

What do you think?
Martin, are you willing to work on it?


Sure, no problem.




On Fri, Nov 27, 2015 at 5:04 AM, Martin Mágr <mm...@redhat.com
<mailto:mm...@redhat.com>> wrote:

 Greetings,

   I've submitted patch to puppet-openstacklib [1] which adds
 provider for parsing INI files containing duplicated variables
 (a.k.a MultiStrOpt [2]). Such parameters are used for example to set
 service_providers/service_provider for Neutron LBaaSv2. There has
 been a thought raised, that the patch should rather be submitted to
 puppetlabs-inifile module instead. The reason I did not submitted
 the patch to inifile module is that IMHO duplicate variables are not
 in the INI file spec [3]. Thoughts?

 Regards,
 Martin


 [1] https://review.openstack.org/#/c/234727/
 [2]
 
https://docs.openstack.org/developer/oslo.config/api/oslo.config.cfg.html#oslo.config.cfg.MultiStrOpt
 [3] https://en.wikipedia.org/wiki/INI_file#Duplicate_names

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] Config support for oslo.config.cfg.MultiStrOpt

[Martin Mágr]

Greetings,

  I've submitted patch to puppet-openstacklib [1] which adds provider 
for parsing INI files containing duplicated variables (a.k.a MultiStrOpt 
[2]). Such parameters are used for example to set 
service_providers/service_provider for Neutron LBaaSv2. There has been a 
thought raised, that the patch should rather be submitted to 
puppetlabs-inifile module instead. The reason I did not submitted the 
patch to inifile module is that IMHO duplicate variables are not in the 
INI file spec [3]. Thoughts?


Regards,
Martin


[1] https://review.openstack.org/#/c/234727/
[2] 
https://docs.openstack.org/developer/oslo.config/api/oslo.config.cfg.html#oslo.config.cfg.MultiStrOpt

[3] https://en.wikipedia.org/wiki/INI_file#Duplicate_names

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] [Puppet] Potential critical issue, due Puppet mix stderr and stdout while execute commands

[Martin Mágr]

On 10/23/2015 02:17 PM, Dmitry Ilyin wrote:
Here is the implementation of the puppet "command" that outputs only 
stdout and drops the stderr unless an error have happened.

https://github.com/dmitryilyin/puppet-neutron/commit/b55f36a8da62fc207a91b358c396c03c8c58981b

+1

I believe such logic should be in puppet-openstacklib and all providers 
in puppet-openstack should inherit from it.


Regards,
Martin



2015-10-22 17:59 GMT+03:00 Matt Fischer >:


On Thu, Oct 22, 2015 at 12:52 AM, Sergey Vasilenko
> wrote:


On Thu, Oct 22, 2015 at 6:16 AM, Matt Fischer
> wrote:

I thought we had code in other places that split out
stderr and only logged it if there was an actual error but
I cannot find the reference now. I think that matches the
original proposal. Not sure I like idea #3.


Matthew, this topic not about SSL. ANY warnings, ANY output to
stderr from CLI may corrupt work of providers from puppet-*
modules for openstack components.

IMHO it's a very serious bug, that potential affect openstack
puppet modules.

I see 3 way for fix it:

 1. Long way. big patch to puppet core for add ability to
collect stderr and stdout separately. But most of existing
puppet providers waits that stderr and stdout was mixed
when handle errors of execution (non-zero return code).
Such patch will broke backward compatibility, if will be
enabled by default.
 2. Middle way. We should write code for redefine 'commands'
method. New commands should collect stderr and stdout
separately, but if error happens return stderr (with
ability access to stdout too).
 3. Short way. Modify existing providers to use json output
instead plain-text or csv. JSON output may be separated
from any garbage (warnings) slightly. I make this patch as
example of this way:
https://review.openstack.org/#/c/238156/ . Anyway json
more formalized format for data exchange, than plain text.

IMHO way #1 is a best solution, but not easy.


I must confess that I'm a bit confused about this. It wasn't a
secret that we're calling out to commands and parsing the output.
It's been discussed over and over on this list as recently as last
week, so this has been a known possible issue for quite a long
time. In my original email I was agreeing with you, so I'm not
sure why we're arguing now. Anyway...

I think we need to split stderr and stdout and log stderr on
errors, your idea #2. Using json like openstack-client can do does
not solve this problem for us, you still can end up with a bunch
of junk on stderr.

This would be a good quick discussion in Tokyo if you guys will be
there.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] [Puppet] Potential critical issue, due Puppet mix stderr and stdout while execute commands

[Martin Mágr]

On 10/22/2015 05:16 AM, Matt Fischer wrote:
I thought we had code in other places that split out stderr and only 
logged it if there was an actual error but I cannot find the reference 
now.

https://github.com/openstack/puppet-glance/blob/stable/kilo/lib/puppet/provider/glance.rb#L184

But it was removed in master branch for some reason.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] Incompatible default parameters

[Martin Mágr]

Greetings,

  I'd like to discuss bug 1506061 ([1]). The main problem I'm trying to 
solve here is that using default parameters of classes cinder::api and 
nova::keystone::auth migration won't work, because Cinder will search 
for endpoint with different name ('Compute service' instead of 'nova'). 
I've submitted fix for that in first patchset of [2], but it was denied 
as backward incompatible change, which I agree with, and instead I just 
added warnings about rename in next cycle [3].


  So the question is if we should start following endpoint naming 
according to Keystone's default catalog or just change default value of 
$nova_catalog_.*info parameters in cinder::api to match endpoint created 
by nova::keystone::auth? I don't mind going one way or another, but I do 
think that default parameters has to create fully functional environment.


Thanks in advance for answers,
Martin

[1] https://bugs.launchpad.net/puppet-nova/+bug/1506061
[2] https://review.openstack.org/#/c/222120/1
[3] 
https://review.openstack.org/#/q/status:open+branch:master+topic:bug/1506061,n,z

[4] https://review.openstack.org/#/c/219284/2/manifests/api.pp

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] service default value functions

[Martin Mágr]

On 09/23/2015 02:17 AM, Cody Herriges wrote:

Alex Schultz wrote:

Hey puppet folks,

Based on the meeting yesterday[0], I had proposed creating a parser
function called is_service_default[1] to validate if a variable matched
our agreed upon value of ''.  This got me thinking
about how can we maybe not use the arbitrary string throughout the
puppet that can not easily be validated.  So I tested creating another
puppet function named service_default[2] to replace the use of '' throughout all the puppet modules.  My tests seemed to
indicate that you can use a parser function as parameter default for
classes.

I wanted to send a note to gather comments around the second function.
When we originally discussed what to use to designate for a service's
default configuration, I really didn't like using an arbitrary string
since it's hard to parse and validate. I think leveraging a function
might be better since it is something that can be validated via tests
and a syntax checker.  Thoughts?


Thanks,
-Alex

[0] 
http://eavesdrop.openstack.org/meetings/puppet_openstack/2015/puppet_openstack.2015-09-15-15.00.html
[1] https://review.openstack.org/#/c/223672
[2] https://review.openstack.org/#/c/224187


I've been mulling this over the last several days and I just can't
accept an entire ruby function which would be ran for every parameter
with the desired static value of "" when the class is
declared and parsed.  I am not generally against using functions as a
parameter default just not a fan in this case because running ruby just
to return a static string seems inappropriate and not optimal.

In this specific case I think the params pattern and inheritance can
obtain us the same goals.  I also find this a valid us of inheritance
cross module namespaces but...only because all our modules must depend
on puppet-openstacklib.

http://paste.openstack.org/show/473655


+1 for implementation in pastebin above. Much better solution than 
running function.


Regards,
Martin





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Parameters possible default value

[Martin Mágr]

Hey Yanis

On 07/17/2015 10:56 AM, Yanis Guenane wrote:

Hello everyone,


if set the value would have been set else it would default to upstream
default.

But Mathieu raised a fair point here[2] is that an empty string for some
settings is a valid value, and hence we can't rely on
it.
  


Since the beginning we are trying to avoid the use of a magic string, but
I am starting to run out of idea here.

Does someone has an idea on which sane value the default could be ?


How about  '*config_default*'? Or whatever similar which says you want 
the default value, but will potentially never be value of any parameter.


Regards,
Martin



Thanks in advance,

[1] https://review.openstack.org/#/c/202488
[2] https://review.openstack.org/#/c/202574
--
Yanis Guenane

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] openstacklib::db::sync proposal

[Martin Mágr]

On 06/03/2015 02:32 PM, Martin Mágr wrote:

is *not* necessary


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Change abandonment policy

[Martin Mágr]

On 06/02/2015 08:39 PM, Colleen Murphy wrote:

4) Auto-abandon after N months/weeks if patch has a -1 or -2

```
If a change is given a -2 and the author has been unresponsive for at 
least 3 months, a script will automatically abandon the change, 
leaving a message about how the author can restore the change and 
attempt to resolve the -2 with the reviewer who left it.

```

We would use a tool like this one[1] to automatically abandon changes 
meeting a certain criteria. We would have to decide whether we want to 
only auto-abandon changes with -2's or go as far as to auto-abandon 
those with -1's. The policy proposal above assumes -2. The tool would 
leave a canned message about how to restore the change.
+1 for auto-abandoning. 3-4 weeks inactivity with -1/-2 seems reasonable 
proof that commiter gave up on patch.


Martin

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] openstacklib::db::sync proposal

[Martin Mágr]

On 06/02/2015 07:05 PM, Mathieu Gagné wrote:

On 2015-06-02 12:41 PM, Yanis Guenane wrote:

The openstacklib::db::sync[2] is currently only a wrapper around an exec
that does the actual db sync, this allow to make any modification to the
exec into a single place. The main advantage IMO is that a contributor
is provided with the same experience as it is not the case today across
all modules.


The amount of possible change to an exec resource is very limited. [1] I
don't see a value in this change which outweighs the code churn and
review load needed to put it in place. Unless we have real use cases or
outrageously genius feature to add to it, I'm not in favor of this change.

Furthermore, any change to the public interface of
openstacklib::db::sync would require changes across all our modules
anyway to benefit from this latest hypothetical feature. I think we are
starting to nitpick over as little generic code we could possibly find
to put in openstacklib.

[1] https://docs.puppetlabs.com/references/latest/type.html#exec



Wearing my consistency hat I must say I like this change. On the other 
hand I agree with Mathieu that delegating single resource from several 
modules to single module is necessary in this case.


Regards,
Martin

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Puppet] Package updates strategy

[Martin Mágr]

On 05/28/2015 05:32 PM, James Slagle wrote:

Then I'm +1 for running Puppet with 'ensure = latest' first and then 'yum
update -y'. Seems ideal solution from my point of view.

How would this solve the library update problem though?. If a new
version of a library is released to address a security issue or what
not, first you'd run puppet, which doesn't see anything new. Then the
yum update -y would pick up the library update, but no services
would get restarted. I don't think we can convince all distros to rev
every package version that might depend on such a library update.

Take something like heartbleed for example, when the updated openssl
was released, there wasn't a corresponding rebuild of every package
out there that requires openssl to set a minimum requires on the new
openssl version (that I know of).


Hmpf, it won't solve the library update problem. I didn't think about 
such case.




I don't know puppet internals at all, but what about some type of
Puppet provider that overrides latest to make Puppet think that it's
actually updated a package even if no such update exists? That could
trigger Puppet to then restart the services b/c it thinks it's updated
something.


Service restart is not triggered by package providers, but by 
dependencies stated in modules.

So using dummy package provider does not seem ideal for me.



SImilar to what TripleO is doing with the norpm provider where the
install is a noop:
https://github.com/stackforge/puppet-tripleo/blob/master/lib/puppet/provider/package/norpm.rb

Could we then swap in this provider when we're triggering the update
via Heat? So we do a yum update -y, and then rerun puppet, and it
thinks it's updated everything, so it restarts as needed.

Each module would need to have parameter implemented to enable such 
behaviour. But anyway
I don't like this solution because I see it as dirty hack. AFAIR all 
service resources in OpenStack modules
are defined as 'refreshonly'. This means that we could implement in 
Tripleo manifests a logic just

to schedule refresh on them when it is required. Thoughts?

Regards,
Martin


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Puppet] Package updates strategy

[Martin Mágr]

On 05/28/2015 09:35 AM, Jan Provaznik wrote:

On 05/28/2015 01:10 AM, Steve Baker wrote:

On 28/05/15 10:54, Richard Raseley wrote:

Zane Bitter wrote:



One solution proposed was to do a yum update first but specifically
exclude any packages that Puppet knows about (the --excludes flag
appears sufficient for this); then follow that up with another Puppet
run using ensure - latest.


My only concern with this approach is how do we collect and maintain the
excludes list. Other than that it sounds reasonable.


Why not swap the order? First run puppet using ensure=latest which 
updates/restarts everything Openstack depends on, then run yum/apt 
update to update any remaining bits. You wouldn't need exclude list then.


Isn't running Puppet with 'ensure = latest' enough? If packaging is 
correct all dependencies which will require update will be updated 
together with packages updated by Puppet. Am I missing something or the 
goal here is to update all packages?




A problem with that approach is that it still fails to restart 
services

which have had libraries updated but have not themselves been updated.
That's no worse than the pure yum approach though. We might need an
additional way to just manually trigger a restart of services.


Maybe this could be handled at the packaging stage by reving the package
version when there is a known fix in a low-level library, thus
triggering a service restart in the puppet phase.



My concern is that then e.g. libc update would mean repackaging 
(bumping version) of zillion of other packages, also zillion of 
packages would be downloaded/upgraded on each system because of a new 
package version.


I think that providing user a manual (script) way to restart services 
after update would be sufficient solution (though not so sophisticated).


Jan

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Regards,
Martin

--

Martin Mágr
Openstack
Red Hat Czech

IRC nick: mmagr / paramite
Freenode channels: #openstack-dev, #packstack-dev, #puppet-openstack, #rdo, 
#rdo-puppet


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Puppet] Package updates strategy

[Martin Mágr]

On 05/28/2015 10:47 AM, Jan Provaznik wrote:

On 05/28/2015 10:25 AM, Martin Mágr wrote:


On 05/28/2015 09:35 AM, Jan Provaznik wrote:

On 05/28/2015 01:10 AM, Steve Baker wrote:

On 28/05/15 10:54, Richard Raseley wrote:

Zane Bitter wrote:



One solution proposed was to do a yum update first but specifically
exclude any packages that Puppet knows about (the --excludes flag
appears sufficient for this); then follow that up with another 
Puppet

run using ensure - latest.

My only concern with this approach is how do we collect and 
maintain the

excludes list. Other than that it sounds reasonable.


Why not swap the order? First run puppet using ensure=latest which
updates/restarts everything Openstack depends on, then run yum/apt
update to update any remaining bits. You wouldn't need exclude list 
then.


Isn't running Puppet with 'ensure = latest' enough? If packaging is
correct all dependencies which will require update will be updated
together with packages updated by Puppet. Am I missing something or the
goal here is to update all packages?



Yes - the goal so to update all packages (so puppet only is not 
sufficient because it will not take care of packages which are not 
listed in puppet modules).



__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Then I'm +1 for running Puppet with 'ensure = latest' first and then 
'yum update -y'. Seems ideal solution from my point of view.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Moving forward with puppet-keystone CI (beaker tests)

[Martin Mágr]

On 04/22/2015 08:03 PM, Emilien Macchi wrote:

* shell testing: yes because it's the way we wrote our providers.
We don't necessary need to shell out, but instead implement resources 
for Serverspec which will use openstackclient to test Keystone/Nova/... 
resources. I'm currently in process of merging Serverspec tests I made 
and will submit patches as soon as I will have something working. What I 
have currently is just Serverspec resource for testing INI file content, 
but openstackclient Serverspec resources are next on my TODO list.


Regards,
Martin

--

Martin Mágr

IRC nick: mmagr / para
Freenode channels: #openstack-dev, #packstack-dev, #puppet-openstack, #rdo, 
#rdo-puppet


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev