Re: [openstack-dev] [tripleo] Proposing Enrique Llorente Pastora as a core reviewer for TripleO

[Cédric Jeanneret]

Of course +1 :).

On 11/15/18 5:37 PM, Emilien Macchi wrote:
> +1 to have him part of TripleO CI core team, thanks for your dedication
> and hard work. I'm glad to see you're learning fast. Keep your
> motivation and thanks again!
> 
> On Thu, Nov 15, 2018 at 11:33 AM Alex Schultz  <mailto:aschu...@redhat.com>> wrote:
> 
> +1
> On Thu, Nov 15, 2018 at 8:51 AM Sagi Shnaidman  <mailto:sshna...@redhat.com>> wrote:
> >
> > Hi,
> > I'd like to propose Quique (@quiquell) as a core reviewer for
> TripleO. Quique is actively involved in improvements and development
> of TripleO and TripleO CI. He also helps in other projects including
> but not limited to Infrastructure.
> > He shows a very good understanding how TripleO and CI works and
> I'd like suggest him as core reviewer of TripleO in CI related code.
> >
> > Please vote!
> > My +1 is here :)
> >
> > Thanks
> > --
> > Best regards
> > Sagi Shnaidman
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> -- 
> Emilien Macchi
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Zuul Queue backlogs and resource usage

[Cédric Jeanneret]

On 11/5/18 11:47 AM, Bogdan Dobrelya wrote:
> Let's also think of removing puppet-tripleo from the base container.
> It really brings the world-in (and yum updates in CI!) each job and each
> container!
> So if we did so, we should then either install puppet-tripleo and co on
> the host and bind-mount it for the docker-puppet deployment task steps
> (bad idea IMO), OR use the magical --volumes-from 
> option to mount volumes from some "puppet-config" sidecar container
> inside each of the containers being launched by docker-puppet tooling.

And, in addition, I'd rather see the "podman" thingy as a bind-mount,
especially since we MUST get the same version in all the calls.

> 
> On 10/31/18 6:35 PM, Alex Schultz wrote:
>>
>> So this is a single layer that is updated once and shared by all the
>> containers that inherit from it. I did notice the same thing and have
>> proposed a change in the layering of these packages last night.
>>
>> https://review.openstack.org/#/c/614371/
>>
>> In general this does raise a point about dependencies of services and
>> what the actual impact of adding new ones to projects is. Especially
>> in the container world where this might be duplicated N times
>> depending on the number of services deployed.  With the move to
>> containers, much of the sharedness that being on a single host
>> provided has been lost at a cost of increased bandwidth, memory, and
>> storage usage.
>>
>> Thanks,
>> -Alex
>>
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] PSA lets use deploy_steps_tasks

[Cédric Jeanneret]

On 11/2/18 2:39 PM, Dan Prince wrote:
> I pushed a patch[1] to update our containerized deployment
> architecture docs yesterday. There are 2 new fairly useful sections we
> can leverage with TripleO's stepwise deployment. They appear to be
> used somewhat sparingly so I wanted to get the word out.

Good thing, it's important to highlight this feature and explain how it
works, big thumb up Dan!

> 
> The first is 'deploy_steps_tasks' which gives you a means to run
> Ansible snippets on each node/role in a stepwise fashion during
> deployment. Previously it was only possible to execute puppet or
> docker commands where as now that we have deploy_steps_tasks we can
> execute ad-hoc ansible in the same manner.

I'm wondering if such a thing could be used for the "inflight
validations" - i.e. a step to validate a service/container is working as
expected once it's deployed, in order to get early failure.
For instance, we deploy a rabbitmq container, and right after it's
deployed, we'd like to ensure it's actually running and works as
expected before going forward in the deploy.

Care to have a look at that spec[1] and see if, instead of adding a new
"validation_tasks" entry, we could "just" use the "deploy_steps_tasks"
with the right step number? That would be really, really cool, and will
probably avoid a lot of code in the end :).

Thank you!

C.

[1] https://review.openstack.org/#/c/602007/

> 
> The second is 'external_deploy_tasks' which allows you to use run
> Ansible snippets on the Undercloud during stepwise deployment. This is
> probably most useful for driving an external installer but might also
> help with some complex tasks that need to originate from a single
> Ansible client.
> 
> The only downside I see to these approaches is that both appear to be
> implemented with Ansible's default linear strategy. I saw shardy's
> comment here [2] that the :free strategy does not yet apparently work
> with the any_errors_fatal option. Perhaps we can reach out to someone
> in the Ansible community in this regard to improve running these
> things in parallel like TripleO used to work with Heat agents.
> 
> This is also how host_prep_tasks is implemented which BTW we should
> now get rid of as a duplicate architectural step since we have
> deploy_steps_tasks anyway.
> 
> [1] https://review.openstack.org/#/c/614822/
> [2] 
> http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/common/deploy-steps.j2#n554
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] quickstart for humans

[Cédric Jeanneret]

On 08/30/2018 04:28 PM, Honza Pokorny wrote:
> Hello!
> 
> Over the last few months, it seems that tripleo-quickstart has evolved
> into a CI tool.  It's primarily used by computers, and not humans.
> tripleo-quickstart is a helpful set of ansible playbooks, and a
> collection of feature sets.  However, it's become less useful for
> setting up development environments by humans.  For example, devmode.sh
> was recently deprecated without a user-friendly replacement. Moreover,
> during some informal irc conversations in #oooq, some developers even
> mentioned the plan to merge tripleo-quickstart and tripleo-ci.
> 
> I think it would be beneficial to create a set of defaults for
> tripleo-quickstart that can be used to spin up new environments; a set
> of defaults for humans.  This can either be a well-maintained script in
> tripleo-quickstart itself, or a brand new project, e.g.
> tripleo-quickstart-humans.  The number of settings, knobs, and flags
> should be kept to a minimum.
> 
> This would accomplish two goals:
> 
> 1.  It would bring uniformity to the team.  Each environment is
> installed the same way.  When something goes wrong, we can
> eliminate differences in setup when debugging.  This should save a
> lot of time.
> 
> 2.  Quicker and more reliable environment setup.  If the set of defaults
> is used by many people, it should container fewer bugs because more
> people using something should translate into more bug reports, and
> more bug fixes.
> 
> These thoughts are coming from the context of tripleo-ui development.  I
> need an environment in order to develop, but I don't necessarily always
> care about how it's installed.  I want something that works for most
> scenarios.
> 
> What do you think?  Does this make sense?  Does something like this
> already exist?

Hello,

As an exercise in order to learn a bit more ansible and refresh my
deploy knowledge, I've create that simple thing:
https://github.com/cjeanner/tripleo-lab
It's "more or less generic", but it was probably never deployed outside
my home infra - its aim is to provide a quick'n'dropable libvirt env,
allowing some tweaking in a convenient way.
That's not at quickstart level - but in order to boostrap an undercloud
or a more complete env, it's more than enough.

The other reason I made this was the feeling quickstart is a beast, not
really easy to master - apparently I'm not the only one """fearing""""
it. I probably didn't dig deep enough. And I wanted to get my own thing,
with some proxy/local mirror support in order to alleviate network
traffic on my home line (it's fast, but still... it's faster on the LAN
;) ).

Cheers,

C.

> 
> Thanks for listening!
> 
> Honza
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Cédric Jeanneret]

On 08/17/2018 12:25 AM, Steve Baker wrote:
> 
> 
> On 15/08/18 21:32, Cédric Jeanneret wrote:
>> Dear Community,
>>
>> As you may know, a move toward Podman as replacement of Docker is starting.
>>
>> One of the issues with podman is the lack of daemon, precisely the lack
>> of a socket allowing to send commands and get a "computer formatted
>> output" (like JSON or YAML or...).
>>
>> In order to work that out, Podman has added support for varlink¹, using
>> the "socket activation" feature in Systemd.
>>
>> On my side, I would like to push forward the integration of varlink in
>> TripleO deployed containers, especially since it will allow the following:
>> # proper interface with Paunch (via python link)
> I'm not sure this would be desirable. If we're going to all container
> management via a socket I think we'd be better supported by using CRI-O.
> One of the advantages I see of podman is being able to manage services
> with systemd again.

Using the socket wouldn't prevent a "per service" systemd unit. Varlink
would just provide another way to manage the containers.
It's NOT like the docker daemon - it will not manage the containers on
startup for example. It's just an API endpoint, without any "automated
powers".

See it as an interesting complement to the CLI, allowing to access
containers data easily with a computer-oriented language like python3.

>> # a way to manage containers from within specific containers (think
>> "healthcheck", "monitoring") by mounting the socket as a shared volume
>>
>> # a way to get container statistics (think "metrics")
>>
>> # a way, if needed, to get an ansible module being able to talk to
>> podman (JSON is always better than plain text)
>>
>> # a way to secure the accesses to Podman management (we have to define
>> how varlink talks to Podman, maybe providing dedicated socket with
>> dedicated rights so that we can have dedicated users for specific tasks)
> Some of these cases might prove to be useful, but I do wonder if just
> making podman calls would be just as simple without the complexity of
> having another host-level service to manage. We can still do podman
> operations inside containers by bind-mounting in the container state.

I wouldn't mount the container state as-is for mainly security reasons.
I'd rather get the varlink abstraction rather than the plain `podman'
CLI - in addition, it is far, far easier for applications to get a
proper JSON instead of some random plain text - even if `podman' seems
to get a "--format" option. I really dislike calling "subprocess" things
when there is a nice API interface - maybe that's just me ;).

In addition, apparently the state is managed by some sqlite DB -
concurrent accesses to that DB isn't really a good idea, we really don't
want a corruption, do we?

> 
>> That said, I have some questions:
>> ° Does any of you have some experience with varlink and podman interface?
>> ° What do you think about that integration wish?
>> ° Does any of you have concern with this possible addition?
> I do worry a bit that it is advocating for a solution before we really
> understand the problems. The biggest unknown for me is what we do about
> healthchecks. Maybe varlink is part of the solution here, or maybe its a
> systemd timer which executes the healthcheck and restarts the service
> when required.

Maybe. My main concern is: would it be interesting to compare both
solutions?
The Healthchecks are clearly docker-specific, no interface exists atm in
the libpod for that. So we have to mimic it in the best way.
Maybe the healthchecks place is in systemd, and varlink would be used
only for external monitoring and metrics. That would also be a nice way
to explore.

I would not focus on only one of the possibilities I've listed. There
are probably even more possibilities I didn't see - once we get a proper
socket, anything is possible, the good and the bad ;).

>> Thank you for your feedback and ideas.
>>
>> Have a great day (or evening, or whatever suits the time you're reading
>> this ;))!
>>
>> C.
>>
>>
>> ¹ https://www.projectatomic.io/blog/2018/05/podman-varlink/
>>
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usa

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Cédric Jeanneret]

>> On my side, I would like to push forward the integration of varlink in
>> TripleO deployed containers, especially since it will allow the
>> following:
>> # proper interface with Paunch (via python link)
> 
> "integration of varlink in TripleO deployed containers" sounds like we'd
> need to make some changes to the containers themselves, but is that the
> case? As i read the docs, it seems like a management API wrapper for
> Podman, so just an alternative interface to Podman CLI. I'd expect we'd
> use varlink from Paunch, but probably not from the containers
> themselves? (Perhaps that's what you meant, just making sure we're on
> the same page.)

In fact, the "podman varlink thing" is already distributed with the
podman package. In order to activate that socket, we just need to
activate a systemd unit that will create the socket - the "service"
itself is activated only when the socket is accessed.
The only thing we might need to add as a package is the libvarlink-util
(provides the "varlink" command) and the python3 binding
(python3-libvarlink iirc).

Varlink "activation" itself doesn't affect the containers.

And yep, it's just an alternative to `podman' CLI, providing a nicer
computer interface with python3 bindings in order to avoid
"subprocess.Popen" and the like, providing a nice JSON output (well.
mostly - I detected at least one output not properly formatted).

> 
>>
>> # a way to manage containers from within specific containers (think
>> "healthcheck", "monitoring") by mounting the socket as a shared volume
> 
> I think healthchecks are currently quite Docker-specific, so we could
> have a Podman-specific alternative here. We should be careful about how
> much container runtime specificity we introduce and keep though, and
> we'll probably have to amend our tools (e.g. pre-upgrade validations
> [2]) to work with both, at least until we decide whether to really make
> a full transition to Podman or not.

Of course - I just listed the possibilities activating varlink would
provide - proper PoCs and tests are to be done ;).

> 
>>
>> # a way to get container statistics (think "metrics")
>>
>> # a way, if needed, to get an ansible module being able to talk to
>> podman (JSON is always better than plain text)
>>
>> # a way to secure the accesses to Podman management (we have to define
>> how varlink talks to Podman, maybe providing dedicated socket with
>> dedicated rights so that we can have dedicated users for specific tasks)
>>
>> That said, I have some questions:
>> ° Does any of you have some experience with varlink and podman interface?
>> ° What do you think about that integration wish?
>> ° Does any of you have concern with this possible addition?
> 
> I like it, but we should probably sync up with Podman community if they
> consider varlink a "supported" interface for controlling Podman, and
> it's not just an experiment which will vanish. To me it certainly looks
> like a much better programmable interface than composing CLI calls and
> parsing their output, but we should make sure Podman folks think so too :)

I think we can say "supported", since they provide the varlink socket
and service directly in podman package. In addition, it was a request:
https://trello.com/c/8RQ6ZF4A/565-8-add-podman-varlink-subcommand
https://github.com/containers/libpod/pull/627

and it's pretty well followed regarding both issues and libpod API updates.

I'll ping them in order to validate that feeling.

> 
> Thanks for looking into this
> 
> Jirka
> 
> [2] https://review.openstack.org/#/c/582502/
> 
>>
>> Thank you for your feedback and ideas.
>>
>> Have a great day (or evening, or whatever suits the time you're reading
>> this ;))!
>>
>> C.
>>
>>
>> ¹ https://www.projectatomic.io/blog/2018/05/podman-varlink/
>>
>>
>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Cédric Jeanneret]

On 08/16/2018 12:10 AM, Jason E. Rist wrote:
> On 08/15/2018 03:32 AM, Cédric Jeanneret wrote:
>> Dear Community,
>>
>> As you may know, a move toward Podman as replacement of Docker is starting.
>>
>> One of the issues with podman is the lack of daemon, precisely the lack
>> of a socket allowing to send commands and get a "computer formatted
>> output" (like JSON or YAML or...).
>>
>> In order to work that out, Podman has added support for varlink¹, using
>> the "socket activation" feature in Systemd.
>>
>> On my side, I would like to push forward the integration of varlink in
>> TripleO deployed containers, especially since it will allow the following:
>> # proper interface with Paunch (via python link)
>>
>> # a way to manage containers from within specific containers (think
>> "healthcheck", "monitoring") by mounting the socket as a shared volume
>>
>> # a way to get container statistics (think "metrics")
>>
>> # a way, if needed, to get an ansible module being able to talk to
>> podman (JSON is always better than plain text)
>>
>> # a way to secure the accesses to Podman management (we have to define
>> how varlink talks to Podman, maybe providing dedicated socket with
>> dedicated rights so that we can have dedicated users for specific tasks)
>>
>> That said, I have some questions:
>> ° Does any of you have some experience with varlink and podman interface?
>> ° What do you think about that integration wish?
>> ° Does any of you have concern with this possible addition?
>>
>> Thank you for your feedback and ideas.
>>
>> Have a great day (or evening, or whatever suits the time you're reading
>> this ;))!
>>
>> C.
>>
>>
>> ¹ https://www.projectatomic.io/blog/2018/05/podman-varlink/
>>
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> How might this effect upgrades?

What exactly? addition of varlink, or the whole podman thingy? The
question was more about "varlink" than "podman" in fact - I should maybe
have worded things otherwise... ?

> 
> -J
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Cédric Jeanneret]

Dear Community,

As you may know, a move toward Podman as replacement of Docker is starting.

One of the issues with podman is the lack of daemon, precisely the lack
of a socket allowing to send commands and get a "computer formatted
output" (like JSON or YAML or...).

In order to work that out, Podman has added support for varlink¹, using
the "socket activation" feature in Systemd.

On my side, I would like to push forward the integration of varlink in
TripleO deployed containers, especially since it will allow the following:
# proper interface with Paunch (via python link)

# a way to manage containers from within specific containers (think
"healthcheck", "monitoring") by mounting the socket as a shared volume

# a way to get container statistics (think "metrics")

# a way, if needed, to get an ansible module being able to talk to
podman (JSON is always better than plain text)

# a way to secure the accesses to Podman management (we have to define
how varlink talks to Podman, maybe providing dedicated socket with
dedicated rights so that we can have dedicated users for specific tasks)

That said, I have some questions:
° Does any of you have some experience with varlink and podman interface?
° What do you think about that integration wish?
° Does any of you have concern with this possible addition?

Thank you for your feedback and ideas.

Have a great day (or evening, or whatever suits the time you're reading
this ;))!

C.


¹ https://www.projectatomic.io/blog/2018/05/podman-varlink/


-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] easily identifying how services are configured

[Cédric Jeanneret]

 loose
>>> nothing important in progress.
>>>
>>> So I'd say please let's do *not* change services' paths/namespaces in
>>> t-h-t
>>> "API" w/o real need to do that, when there is no more alternatives
>>> left to
>>> that.
>>>
>> Ok so it's time to dig this thread back up. I'm currently looking at
>> the chrony support which will require a new service[0][1]. Rather than
>> add it under puppet, we'll likely want to leverage ansible. So I guess
>> the question is where do we put services going forward?  Additionally
>> as we look to truly removing the baremetal deployment options and
>> puppet service deployment, it seems like we need to consolidate under
>> a single structure.  Given that we don't want force too much churn,
>> does this mean that we should align to the docker/services/*.yaml
>> structure or should we be proposing a new structure that we can try to
>> align on.
>>
>> There is outstanding tech-debt around the nested stacks and references
>> within these services when we added the container deployments so it's
>> something that would be beneficial to start tackling sooner rather
>> than later.  Personally I think we're always going to have the issue
>> when we rename files that could have been referenced by custom
>> templates, but I don't think we can continue to carry the outstanding
>> tech debt around these static locations.  Should we be investing in
>> coming up with some sort of mappings that we can use/warn a user on
>> when we move files?
> 
> When Stein development starts, the puppet services will have been
> deprecated for an entire cycle. Can I suggest we use this reorganization
> as the time we delete the puppet services files? This would release us
> of the burden of maintaining a deployment method that we no longer use.
> Also we'll gain a deployment speedup by removing a nested stack for each
> docker based service.
> 
> Then I'd suggest doing an "mv docker/services services" and moving any
> remaining files in the puppet directory into that. This is basically the
> naming that James suggested, except we wouldn't have to suffix the files
> with -puppet.yaml, -docker.yaml unless we still had more than one
> deployment method for that service.

We must be cuatious, as a tree change might prevent backporting things
when we need them in older releases. That was also discussed during the
latter thread regarding reorganization - although I'm also all for a
"simplify that repository" thing, it might become tricky in some cases :/.

> 
> Finally, we could consider symlinking docker/services to services for a
> cycle. I'm not sure how a swift-stored plan would handle this, but this
> would be a great reason to land Ian's plan speedup patch[1] which stores
> tripleo-heat-templates in a tarball :)

Might be worth a try. Might also allow to backport things, as the
"original files" would stay in the "old" location, making the new tree
compatible with older release like Newton (hey, yes, LTS for Red Hat). I
think the templates are aggregated and generated prior the upload,
meaning it should not create new issues. Hopefully.

Maybe shardy can jump in and provide some more info?

> 
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2018-August/132768.html
> 
>> Thanks,
>> -Alex
>>
>> [0] https://review.openstack.org/#/c/586679/
>> [1] https://review.openstack.org/#/c/588111/
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] [tripleo-validations] using using top-level fact vars will deprecated in future Ansible versions

[Cédric Jeanneret]

Hello Sam,

Thanks for the clarifications.

On 07/25/2018 07:46 PM, Sam Doran wrote:
> I spoke with other Ansible Core devs to get some clarity on this change.
> 
> This is not a change that is being made quickly, lightly, or without a
> whole of bunch of reservation. In fact, that PR created by agaffney may
> not be merged any time soon. He just wanted to get something started and
> there is still ongoing discussion on that PR. It is definitely a WIP at
> this point.
> 
> The main reason for this change is that pretty much all of the Ansible
> CVEs to date came from "fact injection", meaning a fact that contains
> executable Python code Jinja will merrily exec(). Vars, hostvars, and
> facts are different in Ansible (yes, this is confusing — sorry). All
> vars go through a templating step. By copying facts to vars, it means
> facts get templated controller side which could lead to controller
> compromise if malicious code exists in facts.
> 
> We created an AnsibleUnsafe class to protect against this, but stopping
> the practice of injecting facts into vars would close the door
> completely. It also alleviates some name collisions if you set a hostvar
> that has the same name as a var. We have some methods that filter out
> certain variables, but keeping facts and vars in separate spaces is much
> cleaner.
> 
> This also does not change how hostvars set via set_fact are referenced.
> (set_fact should really be called set_host_var). Variables set with
> set_fact are not facts and are therefore not inside the ansible_facts
> dict. They are in the hostvars dict, which you can reference as {{
> my_var }} or {{ hostvars['some-host']['my_var'] }} if you need to look
> it up from a different host.

so if, for convenience, we do this:
vars:
  a_mounts: "{{ hostvars[inventory_hostname].ansible_facts.mounts }}"

That's completely acceptable and correct, and won't create any security
issue, right?

> 
> All that being said, the setting to control this behavior as Emilien
> pointed out is inject_facts_as_vars, which defaults to True and will
> remain that way for the foreseeable future. I would not rush into
> changing all the fact references in playbooks. It can be a gradual process.
> 
> Setting inject_facts_as_vars toTrue means ansible_hostname becomes
> ansible_facts.hostname. You do not have to use the hostvars dictionary —
> that is for looking up facts about hosts other than the current host.
> 
> If you wanted to be proactive, you could start using the ansible_facts
> dictionary today since it is compatible with the default setting and
> will not affect others trying to use playbooks that reference ansible_facts.
> 
> In other words, with the default setting of True, you can use either
> ansible_hostname or ansible_facts.hostname. Changing it to False means
> only ansible_facts.hostname is defined.
> 
>> Like, really. I know we can't really have a word about that kind of
>> decision, but... damn, WHY ?!
> 
> That is most certainly not the case. Ansible is developed in the open
> and we encourage community members to attend meetings
> <https://github.com/ansible/community/blob/master/meetings/README.md> and add
> topics to the agenda
> <https://github.com/ansible/community/issues/325> for discussion.
> Ansible also goes through a proposal process for major changes, which
> you can view here
> <https://github.com/ansible/proposals/issues?utf8=%E2%9C%93=is:issue+is:open>.
> 
> You can always go to #ansible-devel on Freenode or start a discussion on
> the mailing list
> <https://groups.google.com/forum/#%21forum/ansible-devel> to speak with
> the Ansible Core devs about these things as well.

And I also have the "Because" linked to my "why" :). big thanks!

Bests,

C.

> 
> ---
> 
> Respectfully,
> 
> Sam Doran
> Senior Software Engineer
> Ansible by Red Hat
> sdo...@redhat.com <mailto:sdo...@redhat.com>
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] PTL candidacy for the Stein Cycle

[Cédric Jeanneret]

+1 :).

On 07/25/2018 02:03 PM, Juan Antonio Osorio wrote:
> Hello folks!
> 
> I'd like to nominate myself for the TripleO PTL role for the Stein cycle.
> 
> Alex has done a great job as a PTL: The project is progressing nicely
> with many
> new, exciting features and uses for TripleO coming to fruition recently.
> It's a
> great time for the project. But, there's more work to be done.
> 
> I have served the TripleO community as a core-reviewer for some years
> now and,
> more recently, by driving the Security Squad. This project has been a
> great learning experience for me, both technically (I got to learn even
> more of
> OpenStack) and community-wise. Now I wish to better serve the community
> further
> by bringing my experiences into PTL role. While I have not yet served as PTL
> for a project before,I'm eager to learn the ropes and help improve the
> community that has been so influential on me.
> 
> For Stein, I would like to focus on:
> 
> * Increasing TripleO's usage in the testing of other projects
>   Now that TripleO can deploy a standalone OpenStack installation, I hope it
>   can be leveraged to add value to other projects' testing efforts. I
> hope this
>   would subsequentially help increase TripleO's testing coverage, and reduce
>   the footprint required for full-deployment testing.
> 
> * Technical Debt & simplification
>   We've been working on simplifying the deployment story and battle
> technical
>   depth -- let’s keep  this momentum going.  We've been running (mostly)
> fully
>   containerized environments for a couple of releases now; I hope we can
> reduce
>   the number of stacks we create, which would in turn simplify the project
>   structure (at least on the t-h-t side). We should also aim for the most
>   convergence we can achieve (e.g. CLI and UI workflows).
> 
> * CI and testing
>   The project has made great progress regarding CI and testing; lets
> keep this
>   moving forward and get developers easier ways to bring up testing
>   environments for them to work on and to be able to reproduce CI jobs.
> 
> Thanks!
> 
> Juan Antonio Osorio Robles
> IRC: jaosorior
> 
> 
> -- 
> Juan Antonio Osorio R.
> e-mail: jaosor...@gmail.com <mailto:jaosor...@gmail.com>
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] [tripleo-validations] using using top-level fact vars will deprecated in future Ansible versions

[Cédric Jeanneret]

On 07/23/2018 08:33 PM, Emilien Macchi wrote:
> Thanks Monty for pointing that out to me today on #ansible-devel.
> 
> Context: https://github.com/ansible/ansible/pull/41811
> The top-level fact vars are currently being deprecated in Ansible, maybe
> 2.7.
> It looks like it only affects tripleo-validations (in a quick look), but
> it could be more.
> See: http://codesearch.openstack.org/?q=ansible_facts=nope==
> 
> An example playbook was written to explain what is deprecated:
> https://github.com/ansible/ansible/pull/41811#issuecomment-399220997
> 
> But it seems like, starting with Ansible 2.5 (what we already have in
> Rocky and beyond), we should encourage the usage of ansible_facts
> dictionary.
> Example:
> var=hostvars[inventory_hostname].ansible_facts.hostname
> instead of:
> var=ansible_hostname

guh I'm sorry, but this is a non-sense, ugly as hell, and will just
make things overcomplicated as sh*t. Like, really. I know we can't
really have a word about that kind of decision, but... damn, WHY ?!

Thanks for the heads-up though - will patch my current disk space
validation update in order to take that into account.

> 
> Can we have someone from TripleO Validations to help, and make sure we
> make it working for future versions of Ansible.
> Also there is a way to test this behavior by disabling the
> 'inject_facts_as_vars' option in ansible.cfg.
> 
> Hope this helps,
> -- 
> Emilien Macchi
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Disk space requirement - any way to lower it a little?

[Cédric Jeanneret]

On 07/20/2018 09:49 AM, Cédric Jeanneret wrote:
> 
> 
> On 07/19/2018 06:55 PM, Paul Belanger wrote:
>> On Thu, Jul 19, 2018 at 05:30:27PM +0200, Cédric Jeanneret wrote:
>>> Hello,
>>>
>>> While trying to get a new validation¹ in the undercloud preflight
>>> checks, I hit an (not so) unexpected issue with the CI:
>>> it doesn't provide flavors with the minimal requirements, at least
>>> regarding the disk space.
>>>
>>> A quick-fix is to disable the validations in the CI - Wes has already
>>> pushed a patch for that in the upstream CI:
>>> https://review.openstack.org/#/c/583275/
>>> We can consider this as a quick'n'temporary fix².
>>>
>>> The issue is on the RDO CI: apparently, they provide instances with
>>> "only" 55G of free space, making the checks fail:
>>> https://logs.rdoproject.org/17/582917/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9cf398/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-17_10_23_46
>>>
>>> So, the question is: would it be possible to lower the requirment to,
>>> let's say, 50G? Where does that 60G³ come from?
>>>
>>> Thanks for your help/feedback.
>>>
>>> Cheers,
>>>
>>> C.
>>>
>>>
>>>
>>> ¹ https://review.openstack.org/#/c/582917/
>>>
>>> ² as you might know, there's a BP for a unified validation framework,
>>> and it will allow to get injected configuration in CI env in order to
>>> lower the requirements if necessary:
>>> https://blueprints.launchpad.net/tripleo/+spec/validation-framework
>>>
>>> ³
>>> http://tripleo.org/install/environments/baremetal.html#minimum-system-requirements
>>>
>> Keep in mind, upstream we don't really have control over partitions of 
>> nodes, in
>> some case it is a single, other multiple. I'd suggest looking more at:
> 
> After some checks on y locally deployed containerized undercloud (hence,
> Rocky) without real activity, here's what I could get:
> - most data are located in /var - this explains the current check.
> 
> If we go a bit deeper, here are the "actually used" directory in /var/lib:
> 20K   alternatives
> 36K   certmonger
> 4.0K  chrony
> 1.2G  config-data
> 4.0K  dhclient
> 6.0G  docker
> 28K   docker-config-scripts
> 92K   docker-container-startup-configs.json
> 44K   docker-puppet
> 592K  heat-config
> 832K  ironic
> 4.0K  ironic-inspector
> 236K  kolla
> 4.0K  logrotate
> 286M  mysql
> 48K   neutron
> 4.0K  ntp
> 4.0K  postfix
> 872K  puppet
> 3.8M  rabbitmq
> 59M   rpm
> 4.0K  rsyslog
> 64K   systemd
> 20K   tripleo
> 236K  tripleo-config
> 9.8M  yum
> 7.5G  total
> 
> Most of the "default installer" partition schema don't go further than
> putting /var, /tmp, /home and /usr in dedicated volumes - of course,
> end-user can chose to ignore that and provide a custom schema.
> 
> That said, we can get the "used" paths. In addition to /var/lib, there's
> obviously /usr.
> 
> We might want to:
> - loop on known locations
> - check if they are on dedicated mount points
> - check the available disk space on those mount points.
> 
> An interesting thing in bash:
> df /var/lib/docker
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sda1  104846316 10188828  94657488  10% /
> 
> This allows to get:
> - actual volume
> - free space on the volume.
> 
> More than that, we might also try to figure out some pattern. For
> instance, "docker" seems to be a pretty good candidate for space, as it
> will get the images and container data. This is probably even the
> biggest eater, at least on the undercloud - as well as the logs (/var/logs).
> 
> We might do a check ensuring we can, at least, DEPLOY the app. This
> would require far less than the required 60G, and with a proper doc
> announcing that, we can get a functional test, aiming on its purpose:
> ensure we can deploy (so asking, let's say, 10G in /var/lib/docker, 5G
> in /var/lib/config-data, 5G in /usr, 1G in /var/log) and, later, upgrade
> (requiring the same amount of *free* space).
> 
> That would require some changes in the validation check of course. But
> at least, we might get a pretty nice covering, while allowing it to run
> smoothly in the CI.
> But, as said: proper documentation should be set, and the "60G minimum
> required" should be rephrased in order to point the locations needing
> space (with the appropriate warning about "none exhaustiven

Re: [openstack-dev] Disk space requirement - any way to lower it a little?

[Cédric Jeanneret]

On 07/19/2018 06:55 PM, Paul Belanger wrote:
> On Thu, Jul 19, 2018 at 05:30:27PM +0200, Cédric Jeanneret wrote:
>> Hello,
>>
>> While trying to get a new validation¹ in the undercloud preflight
>> checks, I hit an (not so) unexpected issue with the CI:
>> it doesn't provide flavors with the minimal requirements, at least
>> regarding the disk space.
>>
>> A quick-fix is to disable the validations in the CI - Wes has already
>> pushed a patch for that in the upstream CI:
>> https://review.openstack.org/#/c/583275/
>> We can consider this as a quick'n'temporary fix².
>>
>> The issue is on the RDO CI: apparently, they provide instances with
>> "only" 55G of free space, making the checks fail:
>> https://logs.rdoproject.org/17/582917/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9cf398/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-17_10_23_46
>>
>> So, the question is: would it be possible to lower the requirment to,
>> let's say, 50G? Where does that 60G³ come from?
>>
>> Thanks for your help/feedback.
>>
>> Cheers,
>>
>> C.
>>
>>
>>
>> ¹ https://review.openstack.org/#/c/582917/
>>
>> ² as you might know, there's a BP for a unified validation framework,
>> and it will allow to get injected configuration in CI env in order to
>> lower the requirements if necessary:
>> https://blueprints.launchpad.net/tripleo/+spec/validation-framework
>>
>> ³
>> http://tripleo.org/install/environments/baremetal.html#minimum-system-requirements
>>
> Keep in mind, upstream we don't really have control over partitions of nodes, 
> in
> some case it is a single, other multiple. I'd suggest looking more at:

After some checks on y locally deployed containerized undercloud (hence,
Rocky) without real activity, here's what I could get:
- most data are located in /var - this explains the current check.

If we go a bit deeper, here are the "actually used" directory in /var/lib:
20K alternatives
36K certmonger
4.0Kchrony
1.2Gconfig-data
4.0Kdhclient
6.0Gdocker
28K docker-config-scripts
92K docker-container-startup-configs.json
44K docker-puppet
592Kheat-config
832Kironic
4.0Kironic-inspector
236Kkolla
4.0Klogrotate
286Mmysql
48K neutron
4.0Kntp
4.0Kpostfix
872Kpuppet
3.8Mrabbitmq
59M rpm
4.0Krsyslog
64K systemd
20K tripleo
236Ktripleo-config
9.8Myum
7.5Gtotal

Most of the "default installer" partition schema don't go further than
putting /var, /tmp, /home and /usr in dedicated volumes - of course,
end-user can chose to ignore that and provide a custom schema.

That said, we can get the "used" paths. In addition to /var/lib, there's
obviously /usr.

We might want to:
- loop on known locations
- check if they are on dedicated mount points
- check the available disk space on those mount points.

An interesting thing in bash:
df /var/lib/docker
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1  104846316 10188828  94657488  10% /

This allows to get:
- actual volume
- free space on the volume.

More than that, we might also try to figure out some pattern. For
instance, "docker" seems to be a pretty good candidate for space, as it
will get the images and container data. This is probably even the
biggest eater, at least on the undercloud - as well as the logs (/var/logs).

We might do a check ensuring we can, at least, DEPLOY the app. This
would require far less than the required 60G, and with a proper doc
announcing that, we can get a functional test, aiming on its purpose:
ensure we can deploy (so asking, let's say, 10G in /var/lib/docker, 5G
in /var/lib/config-data, 5G in /usr, 1G in /var/log) and, later, upgrade
(requiring the same amount of *free* space).

That would require some changes in the validation check of course. But
at least, we might get a pretty nice covering, while allowing it to run
smoothly in the CI.
But, as said: proper documentation should be set, and the "60G minimum
required" should be rephrased in order to point the locations needing
space (with the appropriate warning about "none exhaustiveness" and the
like).

Would that suit better the actual needs, and allow to get a proper disk
space check/validation?

Cheers,

C.


> 
>   https://docs.openstack.org/infra/manual/testing.html
> 
> As for downstream RDO, the same is going to apply once we start adding more
> cloud providers. I would look to see if you actually need that much space for
> deployments, and make try to mock the testing of that logic.
> 
> - Paul
> 
> __

[openstack-dev] Disk space requirement - any way to lower it a little?

[Cédric Jeanneret]

Hello,

While trying to get a new validation¹ in the undercloud preflight
checks, I hit an (not so) unexpected issue with the CI:
it doesn't provide flavors with the minimal requirements, at least
regarding the disk space.

A quick-fix is to disable the validations in the CI - Wes has already
pushed a patch for that in the upstream CI:
https://review.openstack.org/#/c/583275/
We can consider this as a quick'n'temporary fix².

The issue is on the RDO CI: apparently, they provide instances with
"only" 55G of free space, making the checks fail:
https://logs.rdoproject.org/17/582917/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9cf398/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-17_10_23_46

So, the question is: would it be possible to lower the requirment to,
let's say, 50G? Where does that 60G³ come from?

Thanks for your help/feedback.

Cheers,

C.



¹ https://review.openstack.org/#/c/582917/

² as you might know, there's a BP for a unified validation framework,
and it will allow to get injected configuration in CI env in order to
lower the requirements if necessary:
https://blueprints.launchpad.net/tripleo/+spec/validation-framework

³
http://tripleo.org/install/environments/baremetal.html#minimum-system-requirements

-- 
Cédric Jeanneret
Software Engineer
DFG:DF





signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] New "validation" subcommand for "openstack undercloud"

[Cédric Jeanneret]

Dear Stackers,

Seeing the answers on and off-list, we're moving forward!

So, here are the first steps:

A blueprint has been created:
https://blueprints.launchpad.net/tripleo/+spec/validation-framework

I've started a draft of the spec, based on the feedbacks and discussions
I could have:
https://review.openstack.org/#/c/583475/

Please, feel free to comment the spec and add your thoughts - this is a
really great opportunity to get a proper validation framework in
tripleoclient directly.

Thank you for your feedback and attention.

Cheers,

C.


On 07/16/2018 05:27 PM, Cédric Jeanneret wrote:
> Dear Stackers,
> 
> In order to let operators properly validate their undercloud node, I
> propose to create a new subcommand in the "openstack undercloud" "tree":
> `openstack undercloud validate'
> 
> This should only run the different validations we have in the
> undercloud_preflight.py¹
> That way, an operator will be able to ensure all is valid before
> starting "for real" any other command like "install" or "upgrade".
> 
> Of course, this "validate" step is embedded in the "install" and
> "upgrade" already, but having the capability to just validate without
> any further action is something that can be interesting, for example:
> 
> - ensure the current undercloud hardware/vm is sufficient for an update
> - ensure the allocated VM for the undercloud is sufficient for a deploy
> - and so on
> 
> There are probably other possibilities, if we extend the "validation"
> scope outside the "undercloud" (like, tripleo, allinone, even overcloud).
> 
> What do you think? Any pros/cons/thoughts?
> 
> Cheers,
> 
> C.
> 
> 
> 
> ¹
> http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/tripleoclient/v1/undercloud_preflight.py
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] New "validation" subcommand for "openstack undercloud"

[Cédric Jeanneret]

On 07/17/2018 06:57 AM, Udi Kalifon wrote:
> We should also add support for the openstack client to launch the other
> validators that are used in the GUI. There are validators for the
> overcloud as well, and new validators are added all the time.
> 
> These validators are installed under
> /usr/share/openstack-tripleo-validations/validations/ and they're
> launched by the command:
> ansible-playbook -i /usr/bin/tripleo-ansible-inventory
> /usr/share/openstack-tripleo-validations/validations/<>

Hey, funky - I'm currently adding the support for ansible-playbook (in
an "easy, fast and pre-step" way) to the tripleoclient in order to be
able to run validations from that very same location:
https://review.openstack.org/582917

Guess we're on the same track :).

> 
> Cedric, feel free to open an RFE.

Will do once we have the full scope :).

> 
> 
> 
> 
> Regards,
> Udi Kalifon; Senior QE; RHOS-UIAutomation
> 
> 
> On Mon, Jul 16, 2018 at 6:32 PM, Dan Prince  <mailto:dpri...@redhat.com>> wrote:
> 
> On Mon, Jul 16, 2018 at 11:27 AM Cédric Jeanneret
> mailto:cjean...@redhat.com>> wrote:
> >
> > Dear Stackers,
> >
> > In order to let operators properly validate their undercloud node, I
> > propose to create a new subcommand in the "openstack undercloud" "tree":
> > `openstack undercloud validate'
> >
> > This should only run the different validations we have in the
> > undercloud_preflight.py¹
> > That way, an operator will be able to ensure all is valid before
> > starting "for real" any other command like "install" or "upgrade".
> >
> > Of course, this "validate" step is embedded in the "install" and
> > "upgrade" already, but having the capability to just validate without
> > any further action is something that can be interesting, for example:
> >
> > - ensure the current undercloud hardware/vm is sufficient for an update
> > - ensure the allocated VM for the undercloud is sufficient for a deploy
> > - and so on
> >
> > There are probably other possibilities, if we extend the "validation"
> > scope outside the "undercloud" (like, tripleo, allinone, even 
> overcloud).
> >
> > What do you think? Any pros/cons/thoughts?
> 
> I think this command could be very useful. I'm assuming the underlying
> implementation would call a 'heat stack-validate' using an ephemeral
> heat-all instance. If so way we implement it for the undercloud vs the
> 'standalone' use case would likely be a bit different. We can probably
> subclass the implementations to share common code across the efforts
> though.
> 
> For the undercloud you are likely to have a few extra 'local only'
> validations. Perhaps extra checks for things on the client side.
> 
> For the all-in-one I had envisioned using the output from the 'heat
> stack-validate' to create a sample config file for a custom set of
> services. Similar to how tools like Packstack generate a config file
> for example.
> 
> Dan
> 
> >
> > Cheers,
> >
> > C.
> >
> >
> >
> > ¹
> > 
> http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/tripleoclient/v1/undercloud_preflight.py
> 
> <http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/tripleoclient/v1/undercloud_preflight.py>
> > --
> > Cédric Jeanneret
> > Software Engineer
> > DFG:DF
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] New "validation" subcommand for "openstack undercloud"

[Cédric Jeanneret]

Dear Stackers,

In order to let operators properly validate their undercloud node, I
propose to create a new subcommand in the "openstack undercloud" "tree":
`openstack undercloud validate'

This should only run the different validations we have in the
undercloud_preflight.py¹
That way, an operator will be able to ensure all is valid before
starting "for real" any other command like "install" or "upgrade".

Of course, this "validate" step is embedded in the "install" and
"upgrade" already, but having the capability to just validate without
any further action is something that can be interesting, for example:

- ensure the current undercloud hardware/vm is sufficient for an update
- ensure the allocated VM for the undercloud is sufficient for a deploy
- and so on

There are probably other possibilities, if we extend the "validation"
scope outside the "undercloud" (like, tripleo, allinone, even overcloud).

What do you think? Any pros/cons/thoughts?

Cheers,

C.



¹
http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/tripleoclient/v1/undercloud_preflight.py
-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tripleo] New validation: ensure we actually have enough disk space on the undercloud

[Cédric Jeanneret]

On 07/16/2018 12:18 PM, Sergii Golovatiuk wrote:
> Hi,
> 
> On Thu, Jul 12, 2018 at 4:45 PM, Cédric Jeanneret  wrote:
>> Dear Stackers,
>>
>> I'm currently looking for some inputs in order to get a new validation,
>> ran as a "preflight check" on the undercloud.
>>
>> The aim is to ensure we actually have enough disk space for all the
>> files and, most importantly, the registry, being local on the
>> undercloud, or remote (provided the operator has access to it, of course).
>>
>> Although the doc talks about minimum requirements, there's the "never
>> trust the user inputs" law, so it would be great to ensure the user
>> didn't overlook the requirements regarding disk space.
> 
> You may check disk space before undercloud deployment. All we need to
> do is to ensure there is enough space for particular version of
> undercloud. Also there should be upgrade check to validate space
> before running undercloud upgrade.

Yep, that's the intend. According to the requirements, 60G minimum are
required for the undercloud prior to deploy - and there's a check in
tripleo-validations for upgrade, asking for 10G.

> 
>>
>> The "right" way would be to add a new validation directly in the
>> tripleo-validations repository, and run it at an early stage of the
>> undercloud deployment (and maybe once again before the overcloud deploy
>> starts, as disk space will probably change due to the registry and logs
>> and packages and so on).
> 
> Since validations are simple ansible tasks so you may invoke them
> before installing undercloud. However, we may need additional tag for
> undercloud checks

Not really - in fact we can run them as ansible from the
undercloud_preflight "lib" - it's already in the pipe, just have to
ensure it actually works as expected.

There's a tiny thing though: the "openstack-tripleo-validations" package
is not installed prior the undercloud deploy, meaning we don't have
access to its content - I'm currently adding this package as a
dependency of python-tripleoclient:
https://review.rdoproject.org/r/#/c/14847/

Once this one is merged, we will be able to call all the validations we
want from this package - in addition, I've created a small wrapper
allowing to call "ansible-playbook" with the correct options directly
from the undercloud_preflight.py thingy.
That will allow to converge some requirements, for example the RAM check
is wrong in the undercloud_preflight (checking for 8G only) but correct
in the ansible check (asking for 16G).

The card has been updated accordingly.

Thank you for the answers, in and off-list :).

Cheers,

C.

> 
>>
>> There are a few details on this public trello card:
>> https://trello.com/c/QqBsMmP9/89-implement-storage-space-checks
>>
>> What do you think? Care to provide some hints and tips for the correct
>> implementation?
>>
>> Thank you!
>>
>> Bests,
>>
>> C.
>>
>>
>>
>> --
>> Cédric Jeanneret
>> Software Engineer
>> DFG:DF
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Tripleo] New validation: ensure we actually have enough disk space on the undercloud

[Cédric Jeanneret]

Dear Stackers,

I'm currently looking for some inputs in order to get a new validation,
ran as a "preflight check" on the undercloud.

The aim is to ensure we actually have enough disk space for all the
files and, most importantly, the registry, being local on the
undercloud, or remote (provided the operator has access to it, of course).

Although the doc talks about minimum requirements, there's the "never
trust the user inputs" law, so it would be great to ensure the user
didn't overlook the requirements regarding disk space.

The "right" way would be to add a new validation directly in the
tripleo-validations repository, and run it at an early stage of the
undercloud deployment (and maybe once again before the overcloud deploy
starts, as disk space will probably change due to the registry and logs
and packages and so on).

There are a few details on this public trello card:
https://trello.com/c/QqBsMmP9/89-implement-storage-space-checks

What do you think? Care to provide some hints and tips for the correct
implementation?

Thank you!

Bests,

C.



-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] easily identifying how services are configured

[Cédric Jeanneret]

[snip]

> 
> I would almost rather see us organize the directories by service
> name/project instead of implementation.
> 
> Instead of:
> 
> puppet/services/nova-api.yaml
> puppet/services/nova-conductor.yaml
> docker/services/nova-api.yaml
> docker/services/nova-conductor.yaml
> 
> We'd have:
> 
> services/nova/nova-api-puppet.yaml
> services/nova/nova-conductor-puppet.yaml
> services/nova/nova-api-docker.yaml
> services/nova/nova-conductor-docker.yaml

I'd also go for that one - it would be clearer and easier to search when
one wants to see how the service is configured, displaying all implem
for given service.
The current tree is a bit unusual.

> 
> (or perhaps even another level of directories to indicate
> puppet/docker/ansible?)
> 
> Personally, such an organization is something I'm more used to. It
> feels more similar to how most would expect a puppet module or ansible
> role to be organized, where you have the abstraction (service
> configuration) at a higher directory level than specific
> implementations.
> 
> It would also lend itself more easily to adding implementations only
> for specific services, and address the question of if a new top level
> implementation directory needs to be created. For example, adding a
> services/nova/nova-api-chef.yaml seems a lot less contentious than
> adding a top level chef/services/nova-api.yaml.

True. Easier to add new deployment ways, and probably easier to search.

> 
> It'd also be nice if we had a way to mark the default within a given
> service's directory. Perhaps services/nova/nova-api-default.yaml,
> which would be a new template that just consumes the default? Or
> perhaps a symlink, although it was pointed out symlinks don't work in
> swift containers. Still, that could possibly be addressed in our plan
> upload workflows. Then the resource-registry would point at
> nova-api-default.yaml. One could easily tell which is the default
> without having to cross reference with the resource-registry.

+42 for a way to get the default implem - a template that just consume
the right one should be enough and self-explanatory.
Having a tree based on services instead of implem will allow that in an
easy way.

> 
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] improved privilege escalation in py-scripts

[Cédric Jeanneret]

Dear all,

In order to improve the overall security in TripleO, we're currently
creating a couple of specs, aiming Stein version.

The first one concerns calls to "sudo" from shell scripts and the like:
https://review.openstack.org/572760

The second one concerns privilege escalation inside python scripts:
https://review.openstack.org/580033

The short version is "get rid of the NOPASSWD:ALL" scattering the
sudoers for a couple of users.

Both are still Work In Progress, and need a ton of reviews and
discussions in order to get a clear consensus from the community.

Thank you for your time and feedback.

Cheers,

C.

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][tripleoclient] No more global sudo for "stack" on the undercloud

[Cédric Jeanneret]

On 06/06/2018 06:59 AM, Mike Carden wrote:
> 
> \o/ - care to add the links on the doc? Would be really helpful for
> others I guess :).
> 
> 
> Doc? What doc?

This one: https://docs.openstack.org/oslo.privsep/latest/index.html

I just created https://review.openstack.org/#/c/572670/

So. back to business: we need some spec and discussions in order to get
a consensus and implement best practices.

Using privsep will allow to drop the sudo part, as it uses rootwrap
instead. This way also allows to filter out the rights, and we can
ensure we actually don't let people do bad things.

The mentioned blog posts also points to the test process, and shows how
we can ensure we actually mock the calls. It also proposes a directory
structure, and stress on the way to actually call the privileged methods.
All of that makes perfectly sense, as it has a simple logic: if you need
privileges, show them without any hide-and-seek game.

Those advice should be followed, and integrated in any spec/blueprint
we're to write prior the implementation.

Regarding the tripleoclient part: there's currently one annoying issue,
as the generated files aren't owned by the deploy user (usually named
"stack").
This isn't a really urgent correction, but I'm pretty sure we have to
lock any change toward a "quick'n'dirty resolution".

Cheers,

C.

> 
> --
> MC
>  
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][tripleoclient] No more global sudo for "stack" on the undercloud

[Cédric Jeanneret]

On 06/06/2018 06:37 AM, Mike Carden wrote:
> 
> > In regards to your suggested positions within python code such as the
> > client, its worth looking at oslo.privsep [1] where a decorator can be
> > used for when needing to setuid.
> 
> hmm yep, have to understand how to use it - its doc is.. well. kind of
> sparse. Would be good to get examples.
> 
> 
> 
> Examples you say? Michael Still has been at that recently:
> 
> https://www.madebymikal.com/how-to-make-a-privileged-call-with-oslo-privsep/
> https://www.madebymikal.com/adding-oslo-privsep-to-a-new-project-a-worked-example/

\o/ - care to add the links on the doc? Would be really helpful for
others I guess :).

> 
> -- 
> MC
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][tripleoclient] No more global sudo for "stack" on the undercloud

[Cédric Jeanneret]

On 06/05/2018 06:08 PM, Luke Hinds wrote:
> 
> 
> On Tue, Jun 5, 2018 at 3:44 PM, Cédric Jeanneret  <mailto:cjean...@redhat.com>> wrote:
> 
> Hello guys!
> 
> I'm currently working on python-tripleoclient in order to squash the
> dreadful "NOPASSWD:ALL" allowed to the "stack" user.
> 
> The start was an issue with the rights on some files being wrong (owner
> by root instead of stack, in stack home). After some digging and poking,
> it appears the undercloud deployment is called with a "sudo openstack
> tripleo deploy" command - this, of course, creates some major issues
> regarding both security and right management.
> 
> I see a couple of ways to correct that bad situation:
> - let the global "sudo" call, and play with setuid/setgid when we
> actually don't need the root access (as it's mentioned in this comment¹)
> 
> - drop that global sudo call, and replace all the necessary calls by
> some "sudo" when needed. This involves the replacement of native python
> code, like "os.mkdir" and the like.
> 
> The first one isn't a solution - code maintenance will not be possible,
> having to thing "darn, os.setuid() before calling that, because I don't
> need root" is the current way, and it just doesn't apply.
> 
> So I started the second one. It's, of course, longer, not really nice
> and painful, but at least this will end to a good status, and not so bad
> solution.
> 
> This also meets the current work of the Security Squad about "limiting
> sudo rights and accesses".
> 
> For now I don't have a proper patch to show, but it will most probably
> appear shortly, as a Work In Progress (I don't think it will be
> mergeable before some time, due to all the constraints we have regarding
> version portability, new sudoer integration and so on).
> 
> I'll post the relevant review link as an answer of this thread when I
> have something I can show.
> 
> Cheers,
> 
> C.
> 
> 
> Hi Cédric,

Hello Luke,

> 
> Pleased to hear you are willing to take this on.

Well, we have to ;).

> 
> It makes sense we should co-ordinate efforts here as I have been looking
> at the same item, but planned to start with heat-admin over on the
> overcloud.

yep, took part in some discussions already.

> 
> Due to the complexity / level of coverage in the use of sudo, it makes
> sense to have a spec where we can then get community consensus on the
> approach selected. This is important as it looks like we will need to
> have some sort of white list to maintain and make considerations around
> functional test coverage in CI (in case someone writes something new
> wrapped in sudo).

For now, I'm trying to see how's the extend at the code level itself.
This also helps me understanding the different things involved, and I
also make some archaeology in order to understand the current situation.

But indeed, we should push a spec/blueprint in order to get a good idea
of the task and open the discussion on a clear basis.

> 
> In regards to your suggested positions within python code such as the
> client, its worth looking at oslo.privsep [1] where a decorator can be
> used for when needing to setuid.

hmm yep, have to understand how to use it - its doc is.. well. kind of
sparse. Would be good to get examples.

> 
> Let's discuss this also in the squad meeting tomorrow and try to
> synergize approach for all tripleo nix accounts.

You can ping me on #tripleo - I go there by Tengu nick. I'm CET (so
yeah, already up'n'running ;)).

Cheers,

C.

> 
> [1] https://github.com/openstack/oslo.privsep
> 
> Cheers,
> 
> Luke
> 
> 
> ¹
> 
> https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L827-L829
> 
> <https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L827-L829>
> 
> 
> -- 
> Cédric Jeanneret
> Software Engineer
> DFG:DF
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage

[openstack-dev] [tripleo][tripleoclient] No more global sudo for "stack" on the undercloud

[Cédric Jeanneret]

Hello guys!

I'm currently working on python-tripleoclient in order to squash the
dreadful "NOPASSWD:ALL" allowed to the "stack" user.

The start was an issue with the rights on some files being wrong (owner
by root instead of stack, in stack home). After some digging and poking,
it appears the undercloud deployment is called with a "sudo openstack
tripleo deploy" command - this, of course, creates some major issues
regarding both security and right management.

I see a couple of ways to correct that bad situation:
- let the global "sudo" call, and play with setuid/setgid when we
actually don't need the root access (as it's mentioned in this comment¹)

- drop that global sudo call, and replace all the necessary calls by
some "sudo" when needed. This involves the replacement of native python
code, like "os.mkdir" and the like.

The first one isn't a solution - code maintenance will not be possible,
having to thing "darn, os.setuid() before calling that, because I don't
need root" is the current way, and it just doesn't apply.

So I started the second one. It's, of course, longer, not really nice
and painful, but at least this will end to a good status, and not so bad
solution.

This also meets the current work of the Security Squad about "limiting
sudo rights and accesses".

For now I don't have a proper patch to show, but it will most probably
appear shortly, as a Work In Progress (I don't think it will be
mergeable before some time, due to all the constraints we have regarding
version portability, new sudoer integration and so on).

I'll post the relevant review link as an answer of this thread when I
have something I can show.

Cheers,

C.


¹
https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L827-L829


-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Containerized Undercloud deep-dive

[Cédric Jeanneret]

On 05/30/2018 04:10 PM, Dan Prince wrote:
> We are on for this tomorrow (Thursday) at 2pm UTC (10am EST).

meaning 3:00pm CET - conflicting with my squad meeting (3:30pm for squad
red) :(. I'll watch it on youtube then.

> 
> We'll meet here: https://redhat.bluejeans.com/dprince/ and record it
> live. We'll do an overview presentation and then perhaps jump into a
> terminal for some live questions.
> 
> Dan
> 
> On Tue, May 15, 2018 at 10:51 AM, Emilien Macchi  wrote:
>> Dan and I are organizing a deep-dive session focused on the containerized
>> undercloud.
>>
>> https://etherpad.openstack.org/p/tripleo-deep-dive-containerized-undercloud
>>
>> We proposed a date + list of topics but feel free to comment and ask for
>> topics/questions.
>> Thanks,
>> --
>> Emilien & Dan
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] A culture change (nitpicking)

[Cédric Jeanneret]

simple change to take a long
>>>> amount of time and reviews.
>>>>
>>>> I think this is a great addition!
>>>>
>>>> Thanks,
>>>>
>>>> Amy (spotz)
>>>>
>>>> On Tue, May 29, 2018 at 6:55 AM, Julia Kreger
>>>>  wrote:
>>>>>
>>>>> During the Forum, the topic of review culture came up in session after
>>>>> session. During these discussions, the subject of our use of nitpicks
>>>>> were often raised as a point of contention and frustration, especially
>>>>> by community members that have left the community and that were
>>>>> attempting to re-engage the community. Contributors raised the point
>>>>> of review feedback requiring for extremely precise English, or
>>>>> compliance to a particular core reviewer's style preferences, which
>>>>> may not be the same as another core reviewer.
>>>>>
>>>>> These things are not just frustrating, but also very inhibiting for
>>>>> part time contributors such as students who may also be time limited.
>>>>> Or an operator who noticed something that was clearly a bug and that
>>>>> put forth a very minor fix and doesn't have the time to revise it over
>>>>> and over.
>>>>>
>>>>> While nitpicks do help guide and teach, the consensus seemed to be
>>>>> that we do need to shift the culture a little bit. As such, I've
>>>>> proposed a change to our principles[1] in governance that attempts to
>>>>> capture the essence and spirit of the nitpicking topic as a first
>>>>> step.
>>>>>
>>>>> -Julia
>>>>> -
>>>>> [1]: https://review.openstack.org/570940
>>>>>
>>>>>
>>>>> __
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>> __
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] A culture change (nitpicking)

[Cédric Jeanneret]

On 05/30/2018 01:42 AM, Zane Bitter wrote:
> On 29/05/18 16:49, Slawomir Kaplonski wrote:
>> Hi,
>>
>>> Wiadomość napisana przez Jay S Bryant  w dniu
>>> 29.05.2018, o godz. 22:25:
>>> Maybe it would be different now that I am a Core/PTL but in the past
>>> I had been warned to be careful as it could be misinterpreted if I
>>> was changing other people's patches or that it could look like I was
>>> trying to pad my numbers. (I am a nit-picker though I do my best not
>>> to be.
>>
>> Exactly. I remember when I was doing my first patch (or one of first
>> patches) and someone pushed new PS with some very small nits fixed. I
>> was a bit confused because of that and I was thinking why he did it
>> instead of me?
>> Now it’s of course much more clear for me but for someone who is new
>> contributor I think that this might be confusing. Maybe such person
>> should at least remember to explain in comment why he pushed new PS
>> and that’s not „stealing” work of original author :)
> 
> Another issue is that if the original author needs to rev the patch
> again for any reason, they then need to figure out how to check out the
> modified patch. This requires a fairly sophisticated knowledge of both
> git and gerrit, which isn't a problem for those of us who have been
> using them for years but is potentially a nightmarish introduction for a
> relatively new contributor. Sometimes it's the right choice though
> (especially if the patch owner hasn't been seen for a while).

hm, "Download" -> copy/paste, and Voilà. Gerrit interface is pretty nice
with the user (I an "old new contributor", never really struggled with
Gerrit itself.. On the other hand, heat, ansible, that's another story :) ).

> 
> A follow-up patch is a good alternative, unless of course it conflicts
> with another patch in the series.
> 
> +1 with a comment can also get you a long way - it indicates that you've
> reviewed the whole patch and have found only nits to quibble with. If
> you're a core reviewer, another core could potentially +2/+A on a
> subsequent patch set with the nits addressed if they felt it
> appropriate, and even if they don't you'll have an easy re-review when
> you follow up.
> 
> We have lots of tools in our toolbox that are less blunt than -1. Let's
> save -1 for when major work is required and/or the patch as written
> would actually break something.

+1 (not core, can't +2, sorry :D)

"-1" is "aggressive".

Cheers,

C.

> 
> 
> Since I am replying to this thread, Julia also mentioned the situation
> where two core reviewers are asking for opposite changes to a patch. It
> is never ever ever the contributor's responsibility to resolve a dispute
> between two core reviewers! If you see a core reviewer's advice on a
> patch and you want to give the opposite advice, by all means take it up
> immediately - with *the other core reviewer*. NOT the submitter.
> Preferably on IRC and not in the review. You work together every day,
> you can figure it out! A random contributor has no chance of parachuting
> into the middle of that dynamic and walking out unscathed, and they
> should never be asked to.
> 
> cheers,
> Zane.
> 
> ______
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Limiting sudo coverage of heat-admin / stack and other users.

[Cédric Jeanneret]

On 05/22/2018 09:24 AM, Cédric Jeanneret wrote:
> 
> 
> On 05/22/2018 09:08 AM, Luke Hinds wrote:
>>
>>
>> On Tue, May 22, 2018 at 5:27 AM, Cédric Jeanneret <cjean...@redhat.com
>> <mailto:cjean...@redhat.com>> wrote:
>>
>>
>>
>> On 05/21/2018 03:49 PM, Luke Hinds wrote:
>> > A few operators have requested if its possible to limit sudo's coverage
>> > on both the under / overcloud. There is concern over `ALL=(ALL)
>> > NOPASSWD:ALL` , which allows someone to  `sudo su`.
>> > 
>> > This task has come under the care of the tripleo security squad.
>> > 
>> > The work is being tracked and discussed here [0].
>> > 
>> > So far it looks like the approach will be to use regexp within
>> > /etc/sudoers.d/*., to narrow down as close as possible to the specific
>> > commands called. Some services already do this with rootwrap:
>> > 
>> > ironic ALL = (root) NOPASSWD: /usr/bin/ironic-rootwrap
>> > /etc/ironic/rootwrap.conf *   
>> > 
>> > It's fairly easy to pick up a list of all sudo calls using a simple
>> > script [1]
>> > 
>> > The other prolific user of sudo is ansible / stack, for example:
>> > 
>> > /bin/sh -c echo BECOME-SUCCESS-kldpbeueyodisjajjqthpafzadrncdff;
>> > /usr/bin/python
>> > 
>> /home/stack/.ansible/tmp/ansible-tmp-1526579105.0-109863952786117/systemd.py;
>> > rm -rf
>> > "/home/stack/.ansible/tmp/ansible-tmp-1526579105.0-109863952786117/" >
>> > /dev/null 2>&1
>> > 
>> > My feelings here are to again use regexp around the immutable non 
>> random
>> > parts of the command.  cjeanner also made some suggestions in the
>> > etherpad [0].
>>
>> Might be a temporary way to limit the surface indeed, but an upstream
>> change in Ansible would still be really nice. Predictable names is the
>> only "right" way, although this will create a long sudo ruleset. A
>> really long one to be honnest. Maintainability is also to be discussed
>> in either way (maintain a couple of regexp vs 200+ rules.. hmmm).
>>
>>
>> As I understand it, the problem with predicable names is they also
>> become predictable to attackers (this would be the reason ansible adds
>> in the random string). It helps prevent someone creating a race
>> condition to replace the python script with something more nefarious.
>> Its the same reason commands such as mktemp exists.
> 
> Fair enough indeed. Both solution have their pros and cons. In order to
> move on, I think the regexp in sudoers is acceptable for the following
> reasons:
> - limits accesses outside of ansible generated code
> - allows others to still push new content without having to change sudo
> listing (thanks to regexp)
> - still hard to inject bad things in the executed script/code
> - quick to implement (well, fastest than requiring an upstream change
> that will most probably break some internal things before working
> properly, and without adding more security as you explained it)

Small idea: it might be interesting to check if SELinux can't be a ally
for that issue in fact: dedicated context, separation, that's a SELinux
kind of thing isn't it?
I'm no SELinux poweruser¹, but that kind of usage is, to my small
knowledge of this product, a perfect fit.

Would be good to dig in that direction, don't you think?



###
¹ I'm more the poor guy at the end head-banging his desk when SELinux
comes in the way ;). That hurts.

> 
> @Juan do you agree with that statement? As we had some quick chat about it.
> 
> Note: I'm not part of the security squad ;). But I like secured things.
> 
>>
>> > 
>> > However aside to the approach, we need to consider the impact locking
>> > down might have should someone create a develop a new bit of code that
>> > leverages commands wrapped in sudo and assumes ALL with be in place.
>> > This of course will be blocked.
>>
>> This will indeed require some doc, as this is a "major" change. However,
>> the use of regexp should somewhat limit the impact, especially since
>> Ansible pushes its exec script in the same location.
>> Even new parts should be allowed (that might be a bit of concern if we
>> want to really dig in the consequences of a bad template being injected
>> in some way [looking config-download ;)]).
>> But at some p

Re: [openstack-dev] [tripleo] Limiting sudo coverage of heat-admin / stack and other users.

[Cédric Jeanneret]

On 05/22/2018 09:08 AM, Luke Hinds wrote:
> 
> 
> On Tue, May 22, 2018 at 5:27 AM, Cédric Jeanneret <cjean...@redhat.com
> <mailto:cjean...@redhat.com>> wrote:
> 
> 
> 
> On 05/21/2018 03:49 PM, Luke Hinds wrote:
> > A few operators have requested if its possible to limit sudo's coverage
> > on both the under / overcloud. There is concern over `ALL=(ALL)
> > NOPASSWD:ALL` , which allows someone to  `sudo su`.
> > 
> > This task has come under the care of the tripleo security squad.
> > 
> > The work is being tracked and discussed here [0].
> > 
> > So far it looks like the approach will be to use regexp within
> > /etc/sudoers.d/*., to narrow down as close as possible to the specific
> > commands called. Some services already do this with rootwrap:
> > 
> > ironic ALL = (root) NOPASSWD: /usr/bin/ironic-rootwrap
> > /etc/ironic/rootwrap.conf *   
> > 
> > It's fairly easy to pick up a list of all sudo calls using a simple
> > script [1]
> > 
> > The other prolific user of sudo is ansible / stack, for example:
> > 
> > /bin/sh -c echo BECOME-SUCCESS-kldpbeueyodisjajjqthpafzadrncdff;
> > /usr/bin/python
> > 
> /home/stack/.ansible/tmp/ansible-tmp-1526579105.0-109863952786117/systemd.py;
> > rm -rf
> > "/home/stack/.ansible/tmp/ansible-tmp-1526579105.0-109863952786117/" >
> > /dev/null 2>&1
> > 
> > My feelings here are to again use regexp around the immutable non random
> > parts of the command.  cjeanner also made some suggestions in the
> > etherpad [0].
> 
> Might be a temporary way to limit the surface indeed, but an upstream
> change in Ansible would still be really nice. Predictable names is the
> only "right" way, although this will create a long sudo ruleset. A
> really long one to be honnest. Maintainability is also to be discussed
> in either way (maintain a couple of regexp vs 200+ rules.. hmmm).
> 
> 
> As I understand it, the problem with predicable names is they also
> become predictable to attackers (this would be the reason ansible adds
> in the random string). It helps prevent someone creating a race
> condition to replace the python script with something more nefarious.
> Its the same reason commands such as mktemp exists.

Fair enough indeed. Both solution have their pros and cons. In order to
move on, I think the regexp in sudoers is acceptable for the following
reasons:
- limits accesses outside of ansible generated code
- allows others to still push new content without having to change sudo
listing (thanks to regexp)
- still hard to inject bad things in the executed script/code
- quick to implement (well, fastest than requiring an upstream change
that will most probably break some internal things before working
properly, and without adding more security as you explained it)

@Juan do you agree with that statement? As we had some quick chat about it.

Note: I'm not part of the security squad ;). But I like secured things.

> 
> > 
> > However aside to the approach, we need to consider the impact locking
> > down might have should someone create a develop a new bit of code that
> > leverages commands wrapped in sudo and assumes ALL with be in place.
> > This of course will be blocked.
> 
> This will indeed require some doc, as this is a "major" change. However,
> the use of regexp should somewhat limit the impact, especially since
> Ansible pushes its exec script in the same location.
> Even new parts should be allowed (that might be a bit of concern if we
> want to really dig in the consequences of a bad template being injected
> in some way [looking config-download ;)]).
> But at some point, we might also decide to let the OPs ensure their
> infra isn't compromised.
> Always the same thread-of with Security vs The World - convenience vs
> cumbersome management, and so on.
> 
> >
> > Now my guess is that our CI would capture this as the deploy would
> > fail(?) and the developer should work out an entry is needed when
> > testing their patch, but wanted to open this up to others who know
> > testing at gate better much better than myself.  Also encourage any
> > thoughts on the topic to be introduced to the etherpad [0]
> >
> > [0] https://etherpad.openstack.org/p/tripleo-heat-admin-security
> <https://etherpad.openstack.org/p/tripleo-heat-admin-security>
> > [1]
> https://gist.github.com/lukehinds/4cdb1b

Re: [openstack-dev] [tripleo] Limiting sudo coverage of heat-admin / stack and other users.

[Cédric Jeanneret]

On 05/21/2018 03:49 PM, Luke Hinds wrote:
> A few operators have requested if its possible to limit sudo's coverage
> on both the under / overcloud. There is concern over `ALL=(ALL)
> NOPASSWD:ALL` , which allows someone to  `sudo su`.
> 
> This task has come under the care of the tripleo security squad.
> 
> The work is being tracked and discussed here [0].
> 
> So far it looks like the approach will be to use regexp within
> /etc/sudoers.d/*., to narrow down as close as possible to the specific
> commands called. Some services already do this with rootwrap:
> 
> ironic ALL = (root) NOPASSWD: /usr/bin/ironic-rootwrap
> /etc/ironic/rootwrap.conf *   
> 
> It's fairly easy to pick up a list of all sudo calls using a simple
> script [1]
> 
> The other prolific user of sudo is ansible / stack, for example:
> 
> /bin/sh -c echo BECOME-SUCCESS-kldpbeueyodisjajjqthpafzadrncdff;
> /usr/bin/python
> /home/stack/.ansible/tmp/ansible-tmp-1526579105.0-109863952786117/systemd.py;
> rm -rf
> "/home/stack/.ansible/tmp/ansible-tmp-1526579105.0-109863952786117/" >
> /dev/null 2>&1
> 
> My feelings here are to again use regexp around the immutable non random
> parts of the command.  cjeanner also made some suggestions in the
> etherpad [0].

Might be a temporary way to limit the surface indeed, but an upstream
change in Ansible would still be really nice. Predictable names is the
only "right" way, although this will create a long sudo ruleset. A
really long one to be honnest. Maintainability is also to be discussed
in either way (maintain a couple of regexp vs 200+ rules.. hmmm).

> 
> However aside to the approach, we need to consider the impact locking
> down might have should someone create a develop a new bit of code that
> leverages commands wrapped in sudo and assumes ALL with be in place.
> This of course will be blocked.

This will indeed require some doc, as this is a "major" change. However,
the use of regexp should somewhat limit the impact, especially since
Ansible pushes its exec script in the same location.
Even new parts should be allowed (that might be a bit of concern if we
want to really dig in the consequences of a bad template being injected
in some way [looking config-download ;)]).
But at some point, we might also decide to let the OPs ensure their
infra isn't compromised.
Always the same thread-of with Security vs The World - convenience vs
cumbersome management, and so on.

> 
> Now my guess is that our CI would capture this as the deploy would
> fail(?) and the developer should work out an entry is needed when
> testing their patch, but wanted to open this up to others who know
> testing at gate better much better than myself.  Also encourage any
> thoughts on the topic to be introduced to the etherpad [0]
> 
> [0] https://etherpad.openstack.org/p/tripleo-heat-admin-security
> [1] https://gist.github.com/lukehinds/4cdb1bf4de526a049c51f05698b8b04f
> 
> -- 
> Luke Hinds

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] [barbican] [tc] key store in base services

[Cédric Jeanneret]

On 05/17/2018 10:18 AM, Bogdan Dobrelya wrote:
> On 5/17/18 9:58 AM, Thierry Carrez wrote:
>> Jeremy Stanley wrote:
>>> [...]
>>> As a community, we're likely to continue to make imbalanced
>>> trade-offs against relevant security features if we don't move
>>> forward and declare that some sort of standardized key storage
>>> solution is a fundamental component on which OpenStack services can
>>> rely. Being able to just assume that you can encrypt volumes in
>>> Swift, even as a means to further secure a TripleO undercloud, would
>>> be a step in the right direction for security-minded deployments.
>>>
>>> Unfortunately, I'm unable to find any follow-up summary on the
>>> mailing list from the aforementioned session, but recollection from
>>> those who were present (I had a schedule conflict at that time) was
>>> that a Castellan-compatible key store would at least be a candidate
>>> for inclusion in our base services list:
>>>
>>> https://governance.openstack.org/tc/reference/base-services.html
>>
>> Yes, last time this was discussed, there was lazy consensus that
>> adding "a Castellan-compatible secret store" would be a good addition
>> to the base services list if we wanted to avoid proliferation of
>> half-baked keystore implementations in various components.
>>
>> The two blockers were:
>>
>> 1/ castellan had to be made less Barbican-specific, offer at least one
>> other secrets store (Vault), and move under Oslo (done)
> 
> Back to the subject and tripleo underclouds running Barbican, using
> vault as a backend may be a good option, given that openshift supports
> [0] it as well for storing k8s secrets, and kubespray does [1] for
> vanilla k8s deployments, and that we have openshift/k8s-based control
> plane for openstack on the integration roadmap. So we'll highly likely
> end up running Barbican/Vault on undercloud anyway.
> 
> [0]
> https://blog.openshift.com/managing-secrets-openshift-vault-integration/
> [1]
> https://github.com/kubernetes-incubator/kubespray/blob/master/docs/vault.md
> 

That just sounds lovely, especially since this allows to converge
"secure storage" tech between projects.
On my own, I was considering some secure storage (custodia) in the
context of the public TLS certificate storage/update/provisioning.
Having by default a native way to store secrets used by the overcloud
deploy/life is a really good thing, and will prevent leaks, having
ardcoded passwords in files and so on (although, yeah, you'll need
something to access barbican ;)).

>>
>> 2/ some projects (was it Designate ? Octavia ?) were relying on
>> advanced functions of Barbican not generally found in other secrets
>> store, like certificate generation, and so would prefer to depend on
>> Barbican itself, which confuses the messaging around the base service
>> addition a bit ("any Castellan-supported secret store as long as it's
>> Barbican")
>>
> 
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] Meet DoubleNO³

[Cédric Jeanneret]

Dear all,

In the need of a "lab TripleO" in order to validate updates before
pushing to production, I created some ansible receipt, and a whole
isolated network. This isolated network mimic 1:1 the current
production, and is mainly based on libvirt.

I named this "project" DoubleNO³, and it tells a lot of thing regarding
the concepts and architecture:
Double-NATed-tripleO

Some explanations:

On a dedicated baremetal, I've installed:
- CentOS 7
- Libvirt components

The main VMs are:
- virtualized RedHat Satellite (in order to use repository snapshots)
- virtualized undercloud (prod)
- virtualized undercloud (lab)
- virtualized computes (lab, 4 nodes)
- virtualized controllers (lab, 3 nodes)
- virtualized ceph-storages (lab, 2 nodes, with additional volumes for
Ceph setup emulation)

The lab instances share a network bridge (name it lab-trunk), and each
VM has the "right" amount of network interfaces (we use bonding in prod).

In order to isolate the lab, I've created a second layer, with a
dedicated "prenat" instance (lab-prenat). This one has two interfaces:
- one on a libvirt "NAT" network (eth0)
- one on lab-trunk bridge (eth1)

Regarding IPs, eth0 has a private IP in the 192.168.x.y scope, is the
default route to Internet via the libvirt NAT.
Eth1 has multiple IPs:
- one that emulates the public gateway of our production deploy
- all the IPMI addresses of our productive nodes

The second pool allows to keep Ironic configuration 1:1 with production.
In order to make IPMI calls, I've deployed VirtualBMC on the lab-prenat
instance, and am using the qemu+ssh capability of libvirt in order to
talk to the hypervisor and manage the VMs.

All of that is working pretty nicely together. As I decided to use a
virtual Undercloud node instead of baremetal, it was pretty easy to
duplicate the current undercloud node, drop the stack, and redeploy the
virtual lab in a swift way.

Of course, all wasn't as painless as it seems, I had "some" issues with
the network, especially for IPMI: I wanted to have the virtualBMC
directly on the hypervisor, and had many issues with libvirt itpables
rules. In the end, using the qemu+ssh was just the right, easy way to do
things. And this prevent any IPMI listener to be displayed on the outside.

But in the end: we actually have a 1:1 matching lab we can use in order
to validate every updates, and an easy way to rollback to a previous
consistent state using libvirt snapshots.

Would anyone be interested in a more "academic" documentation about the
whole stuff (especially the lab itself - satellite is an (interesting)
option)?
If so, where should I push that doc? Probably somewhere in "advanced
deployment" or something like that.

Unfortunately, I don't think I'll be able to open the ansible code soon,
but at least the concepts can be explained, and configuration example
provided.


Cheers,

C.

-- 
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanne...@camptocamp.com



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tripleo] Some historic digging

[Cédric Jeanneret]

On 12/07/2017 05:23 PM, Steven Hardy wrote:
> On Thu, Dec 7, 2017 at 7:22 AM, Cédric Jeanneret
> <cedric.jeanne...@camptocamp.com> wrote:
>> Hello,
>>
>> While trying to add some unit tests for some resources in the
>> puppet-tripleo repository, I stumbled on a not-so-nice case: the
>> tripleo::firewall::service_rules definition.
>>
>> In there, a resource is dynamically taken from hiera, using the
>> following key structure:
>> tripleo..firewall_rules
>>
>> Although this seems to work properly in the tripleo deploy process, it
>> just doesn't work at all for a unit test.
>>
>> After some more digging, it appears the "dot" (".") in YAML is a
>> reserved char for hashes. In the puppet world, we usually use the double
>> semi-colon, aka "::" as a separator.
>>
>> Sooo… I'm wondering what's the history behind that non-puppet-standard
>> choice of "dot" separator, and what would be required in order to move
>> to a more standard syntax for that kind of resources.
> 
> dprince may be the best person to answer as IIRC he implemented this 
> originally.
> 
> However I suspect the choice of "." was deliberate to differentiate
> from :: which implies the hiera values are consumed by puppet class
> interfaces, vs parsed inside the class.
> 

Hmm, possible, although.. well, a dedicated prefix might be used in
order to avoid weird things

> Can you work around the yaml issue by using quotes to force the
> "tripleo.foo.firewall" to be a string?

That was the first thing I tried - but nope. YAML seems to want to parse
that and kind of "re-cast" it :(

> 
> We don't seem to require that for any templates in
> tripleo-heat-templates though, is that just because the yaml key gets
> cast to a string by hiera?

There might be something in some hiera configuration used in the deploy
process. I'll check the hiera.yaml and try to find something about that
weird string. If we can just report it into the unit tests, it can be
fine. Although... dots... well. ;)

> 
>> I've checked the puppet-tripleo repository, and apparently only two
>> resources are using that kind of syntax:
>> - tripleo::firewall::service_rules (already widely present in the
>> tripleo-heat-templates)
>> - tripleo::haproxy::service_endpoints
>> (at least, those are the only resources using a "$underscore_name" variable)
>>
>> Currently, if anyone wants to change something close to those two
>> resources, it won't be tested against regressions, and it's a really bad
>> situation. Especially since it might break central services (closing all
>> firewall ports isn't good, for example, or dropping an haproxy endpoint
>> by mistake)… Unit tests are needed, especially in such a huge piece of
>> software ;).
> 
> Yeah I think we need to know more about the reasons this syntax won't
> work for unit tests, we could conceivably change it, but as you say
> it's widely used for firewall rules, and could potentially break any
> out of tree service templates that exist, so we'd have to follow a
> deprecation process to change the interface.

Hmmm ok. I didn't check outside of the puppet-tripleo - it indeed might
be some things using this string format. I've already pushed two reviews
that *adds* support for the "::" notation for the said resources. That
way, we have at least a basis for a smooth move. Of course, they are
only "reviews", so they have to be accepted:
https://review.openstack.org/#/c/526589/
https://review.openstack.org/#/c/526292/

I doubt out-of-tree stuff uses those hiera entries, but of course this
would require some grep and, probably, advice from other contributors ;).

Thank you for your feedback!

> 
> Thanks,
> 
> Steve
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Tripleo] Some historic digging

[Cédric Jeanneret]

Hello,

While trying to add some unit tests for some resources in the
puppet-tripleo repository, I stumbled on a not-so-nice case: the
tripleo::firewall::service_rules definition.

In there, a resource is dynamically taken from hiera, using the
following key structure:
tripleo..firewall_rules

Although this seems to work properly in the tripleo deploy process, it
just doesn't work at all for a unit test.

After some more digging, it appears the "dot" (".") in YAML is a
reserved char for hashes. In the puppet world, we usually use the double
semi-colon, aka "::" as a separator.

Sooo… I'm wondering what's the history behind that non-puppet-standard
choice of "dot" separator, and what would be required in order to move
to a more standard syntax for that kind of resources.

I've checked the puppet-tripleo repository, and apparently only two
resources are using that kind of syntax:
- tripleo::firewall::service_rules (already widely present in the
tripleo-heat-templates)
- tripleo::haproxy::service_endpoints
(at least, those are the only resources using a "$underscore_name" variable)

Currently, if anyone wants to change something close to those two
resources, it won't be tested against regressions, and it's a really bad
situation. Especially since it might break central services (closing all
firewall ports isn't good, for example, or dropping an haproxy endpoint
by mistake)… Unit tests are needed, especially in such a huge piece of
software ;).

Thank you for your historical knowledge and its sharing :).

Cheers,

C.

-- 
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanne...@camptocamp.com



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Cédric Jeanneret]

On 11/06/2017 08:36 AM, Juan Antonio Osorio wrote:
> Giving this a bit more thought; I'm slightly more inclined on merely adding
> the JSON formatter option to the standard oslo.log configuration options.

Fine for me.

> 
> This is because we right now have the ability to pass oslo.cfg options via
> the CLI, and we would loose that with the advanced logging configuration
> file. The aforementioned option is something that we're using for
> containerized openstack services.

OK - I'll check on my own if I can get something nice with the
logging.conf file. But that won't be for "tomorrow", as I'm not
full-time on openstack (unfortunately :(). Both solutions have their
pros and cons in the end.

> 
> I'll look into adding the ability to turn that handler on/off from oslo.log.

Ping me if I can help :). And big thanks for that coming change!

> 
> 
> 
> On Mon, Nov 6, 2017 at 8:34 AM, Juan Antonio Osorio <jaosor...@gmail.com>
> wrote:
> 
>>
>>
>> On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret <
>> cedric.jeanne...@camptocamp.com> wrote:
>>
>>> On 11/04/2017 07:08 AM, Doug Hellmann wrote:
>>>> Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47
>>> +0200:
>>>>>> On 3 Nov 2017 19:59, "Doug Hellmann" <d...@doughellmann.com> wrote:
>>>>>>
>>>>>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:
>>>>>>> Dear Stackers,
>>>>>>>
>>>>>>> While working on my locally deployed Openstack (Pike using TripleO),
>>> I
>>>>>>> was a bit struggling with the logging part. Currently, all logs are
>>>>>>> pushed to per-service files, in the standard format "one line per
>>>>>>> entry", plain flat text.
>>>>>>>
>>>>>>> It's nice, but if one is wanting to push and index those logs in an
>>> ELK,
>>>>>>> the current, default format isn't really good.
>>>>>>>
>>>>>>> After some discussions about oslo.log, it appears it provides a nice
>>>>>>> JSONFormatter handler¹ one might want to use for each (python)
>>> service
>>>>>>> using oslo central library.
>>>>>>>
>>>>>>> A JSON format is really cool, as it's easy to parse for machines,
>>> and it
>>>>>>> can be on a multi-line without any bit issue - this is especially
>>>>>>> important for stack traces, as their output is multi-line without
>>> real
>>>>>>> way to have a common delimiter so that we can re-format it and feed
>>> it
>>>>>>> to any log parser (logstash, fluentd, …).
>>>>>>>
>>>>>>> After some more talks, olso.log will not provide a unified interface
>>> in
>>>>>>> order to output all received logs as JSON - this makes sens, as that
>>>>>>> would mean "rewrite almost all the python logging management
>>>>>>> interface"², and that's pretty useless, since (all?) services have
>>> their
>>>>>>> own "logging.conf" file.
>>>>>>>
>>>>>>> That said… to the main purpose of that mail:
>>>>>>>
>>>>>>> - Default format for logs
>>>>>>> A first question would be "are we all OK with the default output
>>> format"
>>>>>>> - I'm pretty sure "humans" are happy with that, as it's really
>>>>>>> convenient to read and grep. But on a "standard" Openstack deploy,
>>> I'm
>>>>>>> pretty sure one does not have only one controller, one ceph node and
>>> one
>>>>>>> compute. Hence comes the log centralization, and with that, the log
>>>>>>> indexation and treatments.
>>>>>>>
>>>>>>> For that, one might argue "I'm using plain files on my logger, and
>>>>>>> grep-ing -r in them". That's a way to do things, and for that, plain,
>>>>>>> flat logs are great.
>>>>>>>
>>>>>>> But… I'm pretty sure I'm not the only one wanting to use some kind of
>>>>>>> ELK cluster for that kind of purpose. So the right question is: what
>>>>>>> about switching the default log format to JSON? On my part, I don't
>>> see
>

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Cédric Jeanneret]

On 11/04/2017 07:08 AM, Doug Hellmann wrote:
> Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200:
>>> On 3 Nov 2017 19:59, "Doug Hellmann"  wrote:
>>>
>>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:
 Dear Stackers,

 While working on my locally deployed Openstack (Pike using TripleO), I
 was a bit struggling with the logging part. Currently, all logs are
 pushed to per-service files, in the standard format "one line per
 entry", plain flat text.

 It's nice, but if one is wanting to push and index those logs in an ELK,
 the current, default format isn't really good.

 After some discussions about oslo.log, it appears it provides a nice
 JSONFormatter handler¹ one might want to use for each (python) service
 using oslo central library.

 A JSON format is really cool, as it's easy to parse for machines, and it
 can be on a multi-line without any bit issue - this is especially
 important for stack traces, as their output is multi-line without real
 way to have a common delimiter so that we can re-format it and feed it
 to any log parser (logstash, fluentd, …).

 After some more talks, olso.log will not provide a unified interface in
 order to output all received logs as JSON - this makes sens, as that
 would mean "rewrite almost all the python logging management
 interface"², and that's pretty useless, since (all?) services have their
 own "logging.conf" file.

 That said… to the main purpose of that mail:

 - Default format for logs
 A first question would be "are we all OK with the default output format"
 - I'm pretty sure "humans" are happy with that, as it's really
 convenient to read and grep. But on a "standard" Openstack deploy, I'm
 pretty sure one does not have only one controller, one ceph node and one
 compute. Hence comes the log centralization, and with that, the log
 indexation and treatments.

 For that, one might argue "I'm using plain files on my logger, and
 grep-ing -r in them". That's a way to do things, and for that, plain,
 flat logs are great.

 But… I'm pretty sure I'm not the only one wanting to use some kind of
 ELK cluster for that kind of purpose. So the right question is: what
 about switching the default log format to JSON? On my part, I don't see
 "cons", only "pros", but my judgment is of course biased, as I'm "alone
 in my corner". But what about you, Community?

 - Provide a way to configure the output format/handler
 While poking around in the puppet modules code, I didn't find any way to
 set the output handler for the logs. For example, in puppet-nova³ we can
 set a lot of things, but not the useful handler for the output.

 It would be really cool to get, for each puppet module, the capability
 to set the handler so that one can just push some stuff in hiera, and
 Voilà, we have JSON logs.

 Doing so would allow people to chose between the default  (current)
 output, and something more "computable".
>>>
>>> Using the JSON formatter currently requires setting a logging
>>> configuration file using the standard library configuration format
>>> and fully specifying things like log levels, handlers, and output
>>> destination. Would it make sense to add an option in oslo.log to
>>> give deployers an easier way to enable the JSON formatter?
>>>
>>
>> This would actually be very useful.
> 
> Great! Let me know if you would like to help figuring out where to do
> that in the library code.

There's already some (not really documented) feature allowing oslo.log
to load a configuration file. In fact, there's already one existing in
the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we
might base something "casual" and "generic" on that one.

I think all wsgi/python services should configure the logging through
that file so that it's clearer and cleaner. But that part is maybe a bit
too big and "aggressive" :). And the logging configuration isn't that
friendly, to be honest, at least I have some issues with its doc ;).

But I think we might come to something nice and cool. It would allow
anyone to push  their own log "formatter", in the end.
So you (oslo.log) wouldn't need to work with format output requests
anymore, just provide two basics (plain and json), and let users play
with the logging configuration (and even output things in XML if they
want) ;).

Regarding oslo.log: apparently, adding the following entry in the
service configuration file should do it:
log-config-append¹

Can anyone confirm that?


¹ https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L44

> 
> Doug
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 

Re: [openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

[Cédric Jeanneret]

On 11/02/2017 05:18 PM, Ben Nemec wrote:
> Adding tags for the relevant projects.  I _think_ this is mostly
> directed at Puppet/TripleO, but Oslo is obviously relevant as well.

Thank you! First mail in there, wasn't really sure how to do that.

> 
> On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:
>> Dear Stackers,
>>
>> While working on my locally deployed Openstack (Pike using TripleO), I
>> was a bit struggling with the logging part. Currently, all logs are
>> pushed to per-service files, in the standard format "one line per
>> entry", plain flat text.
>>
>> It's nice, but if one is wanting to push and index those logs in an ELK,
>> the current, default format isn't really good.
>>
>> After some discussions about oslo.log, it appears it provides a nice
>> JSONFormatter handler¹ one might want to use for each (python) service
>> using oslo central library.
>>
>> A JSON format is really cool, as it's easy to parse for machines, and it
>> can be on a multi-line without any bit issue - this is especially
>> important for stack traces, as their output is multi-line without real
>> way to have a common delimiter so that we can re-format it and feed it
>> to any log parser (logstash, fluentd, …).
>>
>> After some more talks, olso.log will not provide a unified interface in
>> order to output all received logs as JSON - this makes sens, as that
>> would mean "rewrite almost all the python logging management
>> interface"², and that's pretty useless, since (all?) services have their
>> own "logging.conf" file.
>>
>> That said… to the main purpose of that mail:
>>
>> - Default format for logs
>> A first question would be "are we all OK with the default output format"
>> - I'm pretty sure "humans" are happy with that, as it's really
>> convenient to read and grep. But on a "standard" Openstack deploy, I'm
>> pretty sure one does not have only one controller, one ceph node and one
>> compute. Hence comes the log centralization, and with that, the log
>> indexation and treatments.
>>
>> For that, one might argue "I'm using plain files on my logger, and
>> grep-ing -r in them". That's a way to do things, and for that, plain,
>> flat logs are great.
>>
>> But… I'm pretty sure I'm not the only one wanting to use some kind of
>> ELK cluster for that kind of purpose. So the right question is: what
>> about switching the default log format to JSON? On my part, I don't see
>> "cons", only "pros", but my judgment is of course biased, as I'm "alone
>> in my corner". But what about you, Community?
> 
> The major con I see is that we don't require an ELK stack and reading
> JSON logs if you don't have one of those is probably worse, although I
> will admit I haven't spent much time reading our JSON-formatted logs. My
> experience with things that log in JSON has not generally been positive
> though.  It's just not as human-readable.

We're on the same line on that - hence the following proposal. I was
pretty sure switching the default format was a bad thing, but I had to
submit it in order to be complete ;).
Let's focus on the other one, as it's more friendly for everyone.

> 
> The other problem with changing log format defaults is that many people
> have already set up filters and processing based on the existing log
> format.  There are significant user impacts to changing that default.
> 
>>
>> - Provide a way to configure the output format/handler
>> While poking around in the puppet modules code, I didn't find any way to
>> set the output handler for the logs. For example, in puppet-nova³ we can
>> set a lot of things, but not the useful handler for the output.
>>
>> It would be really cool to get, for each puppet module, the capability
>> to set the handler so that one can just push some stuff in hiera, and
>> Voilà, we have JSON logs.
>>
>> Doing so would allow people to chose between the default  (current)
>> output, and something more "computable".
> 
> I think we should do this regardless.  There are valid arguments for
> people to want both log formats IMHO and we should allow people to use
> what they want.

If I understand the things correctly, that would require a "small"
change in every puppet modules so that it configures the service with
the proper log output. Any way to automate something on that? It might
be worth to do some PoC on a specific module maybe?


> 
>>
>> Of course, either proposal will require a nice code change in all puppet
>> modules (add a new paramete

[openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on

[Cédric Jeanneret]

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

- Default format for logs
A first question would be "are we all OK with the default output format"
- I'm pretty sure "humans" are happy with that, as it's really
convenient to read and grep. But on a "standard" Openstack deploy, I'm
pretty sure one does not have only one controller, one ceph node and one
compute. Hence comes the log centralization, and with that, the log
indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

- Provide a way to configure the output format/handler
While poking around in the puppet modules code, I didn't find any way to
set the output handler for the logs. For example, in puppet-nova³ we can
set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default  (current)
output, and something more "computable".

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.


¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp


-- 
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanne...@camptocamp.com




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev