Re: [openstack-dev] [openstack-ansible] Proposing Mohammed Naser as core reviewer

[Carter, Kevin]

+2 from me!


--

Kevin Carter
IRC: Cloudnull

On Wed, Apr 25, 2018 at 4:06 AM, Markos Chandras  wrote:

> On 24/04/18 16:05, Jean-Philippe Evrard wrote:
> > Hi everyone,
> >
> > I’d like to propose Mohammed Naser [1] as a core reviewer for
> OpenStack-Ansible.
> >
>
> +2
>
> --
> markos
>
> SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
> HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] PTL non-candidacy

[Carter, Kevin]

++ Thanks for leadership within Glance and everything else you've done in
the community!


--

Kevin Carter
IRC: Cloudnull

On Mon, Jan 29, 2018 at 1:27 PM, Sean McGinnis 
wrote:

> On Mon, Jan 29, 2018 at 02:18:18PM -0500, Brian Rosmaita wrote:
> > I've been PTL of Glance through some rocky times, but have decided not
> > to stand for election for the Rocky cycle.  My plan is to stick
> > around, attend to my duties as a glance core contributor, and support
> > my successor in whatever way I can to make for a smooth transition.
> > After three consecutive cycles of me, it's time for some new ideas and
> > new approaches.
> >
> > For anyone out there who hasn't contributed to Glance yet, the Glance
> > community is friendly and welcoming, and we've got a backlog of
> > "untargeted" specs ready for you to pick up.  Weekly meetings are
> > 14:00 UTC on Thursdays in #openstack-meeting-4.
> >
> > cheers,
> > brian
> >
>
> Thanks for all your hard work as Glance PTL Brian. Great to hear you are
> not
> going anywhere.
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][deployment][kolla][tripleo][osa] Service diagnostics task force

[Carter, Kevin]

Sounds like a worthwhile goal. Count me in.

--

Kevin Carter
IRC: Cloudnull


On Sep 14, 2017 01:46, "Michał Jastrzębski"  wrote:

Hello my dearest of communities,

During deployment tools session on PTG we discussed need for deep
health checking and metering of running services. It's very relevant
in context of containers (especially k8s) and HA. Things like
watchdog, heartbeats or exposing relative metrics (I don't want to get
into design here, suffice to say it's non-trivial problem to solve).

We would like to start a "task force", few volunteers from both
deployment tool side (ops, ha) and project dev teams. We would like to
design together a proper health check mechanism for one of projects to
create best practices and design, that later could be implemented in
all other services.

We would to ask for volunteer project team to join us and spearhead this
effort.

Regards,
Michal

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Ansible] Not running for Queens PTL

[Carter, Kevin]

Andy,

The last couple of cycles have been great and we appreciate all you've done
for the community.

--

Kevin Carter
IRC: Cloudnull

On Mon, Jul 31, 2017 at 4:48 AM, Andy McCrae  wrote:

> Following on from last week's meeting - I've had 2 cycles as PTL for OSA,
> which has been a really great experience - we've achieved a lot and built
> on the strong base we had, which I'm really proud of. I strongly believe
> that inviting a fresh perspective and new ideas as PTL is a winning
> strategy - it's served us well so far, and in line with previous PTLs I
> won't be standing for a 3rd cycle.
>
> Looking forward to assisting the next PTL, and helping to continue to
> mature and improve the project!
>
> Thanks!
> Andy
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [upgrades][skip-level][leapfrog] - RFC - Skipping releases when upgrading

[Carter, Kevin]

Hello Stackers,

As I'm sure many of you know there was a talk about doing "skip-level"[0]
upgrades at the OpenStack Summit which quite a few folks were interested
in. Today many of the interested parties got together and talked about
doing more of this in a formalized capacity. Essentially we're looking for
cloud upgrades with the possibility of skipping releases, ideally enabling
an N+3 upgrade. In our opinion it would go a very long way to solving cloud
consumer and deployer problems it folks didn't have to deal with an upgrade
every six months. While we talked about various issues and some of the
current approaches being kicked around we wanted to field our general chat
to the rest of the community and request input from folks that may have
already fought such a beast. If you've taken on an adventure like this how
did you approach it? Did it work? Any known issues, gotchas, or things
folks should be generally aware of?


During our chat today we generally landed on an in-place upgrade with known
API service downtime and little (at least as little as possible) data plane
downtime. The process discussed was basically:
a1. Create utility "thing-a-me" (container, venv, etc) which contains the
required code to run a service through all of the required upgrades.
a2. Stop service(s).
a3. Run migration(s)/upgrade(s) for all releases using the utility
"thing-a-me".
a4. Repeat for all services.

b1. Once all required migrations are complete run a deployment using the
target release.
b2. Ensure all services are restarted.
b3. Ensure cloud is functional.
b4. profit!

Obviously, there's a lot of hand waving here but such a process is being
developed by the OpenStack-Ansible project[1]. Currently, the OSA tooling
will allow deployers to upgrade from Juno/Kilo to Newton using Ubuntu
14.04. While this has worked in the lab, it's early in development (YMMV).
Also, the tooling is not very general purpose or portable outside of OSA
but it could serve as a guide or just a general talking point. Are there
other tools out there that solve for the multi-release upgrade? Are there
any folks that might want to share their expertise? Maybe a process outline
that worked? Best practices? Do folks believe tools are the right way to
solve this or would comprehensive upgrade documentation be better for the
general community?

As most of the upgrade issues center around database migrations, we
discussed some of the potential pitfalls at length. One approach was to
roll-up all DB migrations into a single repository and run all upgrades for
a given project in one step. Another was to simply have mutliple python
virtual environments and just run in-line migrations from a version
specific venv (this is what the OSA tooling does). Does one way work better
than the other? Any thoughts on how this could be better? Would having
N+2/3 migrations addressable within the projects, even if they're not
tested any longer, be helpful?

It was our general thought that folks would be interested in having the
ability to skip releases so we'd like to hear from the community to
validate our thinking. Additionally, we'd like to get more minds together
and see if folks are wanting to work on such an initiative, even if this
turns into nothing more than a co-op/channel where we can "phone a friend".
Would it be good to try and secure some PTG space to work on this? Should
we try and create working group going?

If you've made it this far, please forgive my stream of consciousness. I'm
trying to ask a lot of questions and distill long form conversation(s) into
as little text as possible all without writing a novel. With that said, I
hope this finds you well, I look forward to hearing from (and working with)
you soon.

[0] https://etherpad.openstack.org/p/BOS-forum-skip-level-upgrading
[1] https://github.com/openstack/openstack-ansible-ops/tree/
master/leap-upgrades


--

Kevin Carter
IRC: Cloudnull
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-ansible] Monitoring script framework PoC

[Carter, Kevin]

I think this is a great idea. It's fairly straight forward to
contribute to, and with the aim to support multiple output formats I
can see this benefiting lots of folks without being bound to a single
system.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-ansible] So long, farewell, auf wiedersehen

[Carter, Kevin]

It's been great working with you, you've done a lot for our community
and it'll be sad not seeing you around. Congratulations on the new
position and I wish you all the best!

Take care.

--

Kevin Carter
IRC: Cloudnull


On Wed, Feb 22, 2017 at 1:06 PM, Major Hayden  wrote:
> On 02/22/2017 11:48 AM, Truman, Travis wrote:
>> I’ve very much enjoyed being part of the OpenStack community over the past 
>> 14 months. My time in the OpenStack-Ansible community has been one of the 
>> most rewarding experiences of my career. However, my career is changing 
>> directions and I’ll no longer be able to invest the time and energy required 
>> to maintain my involvement in the community.
>
> Thanks for all you've done for the project and for all you've done for the 
> OpenStack-Ansible community members, too.  We wish you the best in your 
> future endeavors! :)
>
> --
> Major Hayden
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][oslo][openstack-ansible] DB deadlocks, Mitaka, and You

[Carter, Kevin]

Hi Matt and thanks for the reply,

We do have that commit as found here: [
https://github.com/openstack/nova/blob/dd30603f91e6fd3d1a4db452f20a51ba8820e1f4/nova/db/sqlalchemy/api.py#L1846
]. If there's anything you'd like to see as we're trying to figure
this out I'd be happy to provide {any,every}thing.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][oslo][openstack-ansible] DB deadlocks, Mitaka, and You

[Carter, Kevin]

Hello all,

As some folks may know the OSIC cloud1 just upgraded to Mitaka last
week (I know what's old is new again, sorry). Since the upgrade things
have been running fairly smoothly however there's been one issue that
has left me scratching my head. When attempting a scale out test we've
run into issues where the nova client was returning a message
indicating it had encountered a DB deadlock [0]. In attempting to gain
more information we enabled debug logging and collected the following
[1] (Here is a pretty version for the tracebacks [4]). This is the
command we're using to build all of the VMs [2] which was being used
to build 3 vms per compute node for 242 compute nodes. Once the
instances are all online we grab the nodes IPv6 address and ensure
we're able to SSH to them. While we're happy to report that almost all
of the VMs came online in one shot we did run into a few of these DB
dead lock messages and would like to see if folks out on the interwebs
have seen or experienced such problems. If so we'd love to know if
there's some remediation that we can use to make this all even
happier. It should be noted that this is not a Major issue at this
point, 3 out of 726 VMs didn't come online due to the problem but it
would be amazing if there was something we could do to generally
resolve this.

Other potentially interesting things:
  * DB is MariaDB "mariadb-galera-server-10.0". The replication system
is using xtrabackup from percona with version "2.3.5-1.trusty".
  * The DB is a 3 node cluster however the connection to the cluster
is using a VIP on our loadbalancer and the services are only ever
connected to 1 of the three nodes; this is for both reads and writes.
Should a node become un-happy the load balancer promotes and demotes
always ensuring only 1 node is being connected to.
  * While reproducing this issue repeatedly we've watched wsrep to see
if nodes we're dropping or otherwise having a bad day. To our dismay
there were no un-happy nodes and the wsrep state seemed to remain
OPERATIONAL with minimal latency (Example [3]).
  * For all of the OpenStack Services we use pymsql as it's DB driver
we were using mysql-python but I believe OpenStack-Ansible switched in
kilo due to DB deadlock issues we were experiencing.
  * Cloud1 is running nova at commit
"dd30603f91e6fd3d1a4db452f20a51ba8820e1f4" which was the HEAD of
"stable/mitaka" on 29-09-2016. This code is in production today with
absolutely no modifications.

Any insight would be greatly appreciated. Thanks again everyone!

[0] http://paste.openstack.org/show/586302/
[1] http://paste.openstack.org/show/586309/
[2] http://paste.openstack.org/show/586298/
[3] http://paste.openstack.org/show/586306/
[4] http://paste.openstack.org/show/586307/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-ansible] Project Mascot

[Carter, Kevin]

A little out of band from the meeting but maybe an "Osa Eucharitid" [
http://www.petsfoto.com/insect-world/#foobox-1/0/Insect-Life1.jpg ]?

However the Cape Buffalo would be good too.

--

Kevin Carter
IRC: Cloudnull


On Fri, Jul 15, 2016 at 8:15 AM, Jesse Pretorius
 wrote:
> Hi everyone,
>
> As discussed in the community meeting yesterday [1] the foundation wishes
> each project to select a mascot for the project [2][3]. The mascot needs to
> be appropriate (please read the links for the constraints) and in some way
> linked to the project. We need to come up with a list of 3-5 options.
>
> Options discussed and agreed as appropriate in the meeting yesterday:
>
> A Bull (links directly to Ansible’s mascot, although ours can be different).
> A Cape Buffalo (distinctive, and reasonably close to a Bull).
>
> Please consider and reply with more options, explaining the link to the
> project so that we can all discuss the options at the next meeting to
> finalise our list.
>
> [1]
> http://eavesdrop.openstack.org/meetings/openstack_ansible/2016/openstack_ansible.2016-07-14-15.58.log.html
> [2] http://lists.openstack.org/pipermail/openstack-dev/2016-July/099046.html
> [3] http://www.openstack.org/project-mascots
>
> Thanks,
>
> Jesse
> IRC: odyssey4me
>
>
> 
> Rackspace Limited is a company registered in England & Wales (company
> registered number 03897010) whose registered office is at 5 Millington Road,
> Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be
> viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may
> contain confidential or privileged information intended for the recipient.
> Any dissemination, distribution or copying of the enclosed material is
> prohibited. If you receive this transmission in error, please notify us
> immediately by e-mail at ab...@rackspace.com and delete the original
> message. Your cooperation is appreciated.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-ansible] Nominating Jean-Philippe Evrard for core in openstack-ansible and all openstack-ansible-* roles

[Carter, Kevin]

+1 from me too. It'll be great to have him on the core team.

On Tue, Jul 12, 2016 at 1:33 PM, Truman, Travis
 wrote:
> Jean-Philippe has been providing great code reviews and patches for some
> time. His recent commitment to running bug triage every week shows his
> willingness to step up and take responsibilities within the community. He’s
> also found an opportunity to innovate by introducing an improved bug triage
> process. He can often be found in #openstack-ansible as evrardjp providing
> support to deployers in a welcoming and friendly manner.
>
> In short, just the kind of contribution our community desires from core
> reviewers.
>
> Thanks for all that you do for the community Jean-Philippe,
> Travis Truman
>
>
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Ansible] Splitting out generic testing components

[Carter, Kevin]

+1 I think this is a great initiative and should go a long way to
unifying and improving our tests within the roles. I too would love to
see this moved to the namespace so that folks can begin working on and
reviewing changes gerrit.

@Jesse, assuming everything is on the up and up, what are the next
steps to getting this setup in our project-config?

--

Kevin Carter
IRC: cloudnull

On Fri, Jul 8, 2016 at 9:18 AM, Truman, Travis
 wrote:
> Nice work Andy. I’m strongly in favor of getting the central test repo into
> Gerrit for broader reviews and once done, getting one or two patches up with
> roles making use of it.
>
> - Travis Truman
>
> From: Andy McCrae 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Date: Friday, June 3, 2016 at 11:41 AM
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Subject: [openstack-dev] [OpenStack-Ansible] Splitting out generic testing
> components
>
> TL;DR We're doing a PoC to split out common testing plays/vars into a
> generic repository, for the purposes of making in-role testing more uniform
> and generic. Looking for feedback, comments, informal reviews, ideas etc!
> https://github.com/andymcc/openstack-ansible-testing-core (Includes a simple
> README)
>
> We now have a lot of duplication after moving to a single role per
> repository structure with specific testing per role. For example, almost all
> repositories require Galera/Rabbit/Keystone in order to deploy testing
> successfully. This has led to a scenario where each repository essentially
> carries the same deployment code.
>
>
> Aims:
> - The primary aim of extracting the testing infrastructure into a single
> repository is to reduce the cases where a simple change is needed, which
> dominoes, causing a patch to each of 15 repositories. Instead, a change to
> the uniform testing repository would resolve the issue for all other
> repository's testing.
> - In the future, we are looking to deploy scenario testing. For example, we
> may want to test glance with a swift backend, or keystone with memcache. If
> the testing play to deploy swift is already in a uniform repository, the
> changes required to get glance testing enabled for that use case would be
> minimal.
> - This allows new projects to consume existing structure/playbooks to deploy
> common components and vars, which should simplify the manner in which new
> openstack-ansible projects set up their testing.
>
>
> Steps taken so far:
> - The deployment plays for each project have been split out into the
> separate testing role.
> - Each role only carries a specific "Test project" play.
> - The test playbooks have been made generic so it uses the inventory
> specified per repository (defining what hosts/roles there are).
> - The test-vars have been put in the testing-repository and moved out of the
> generic role.
> - An override file has been created for each project and included using "-e"
> (the highest precedence) but at present, of the 4 projects converted the
> maximum number of overrides used is 2, so these overrides are minimal.
> - Adjustments to tox.ini and var files references have been made to use the
> new repository.
>
>
> Further work to look into:
> - It may be worth looking into making the tox.ini more generic, if we were
> to make a sweeping change that impacts on tox.ini we would still need to do
> changes to each repository. (I am not sure on how feasible this is though)
>
>
> Raised concerns:
> - This creates a situation where a change may require me to make a change to
> the central testing repository before changing the role repository. For
> example, in order to get the generic testing for a keystone change I would
> have to change the testing repository in advance, and then change the
> keystone repository. Or override the var, adjust the testing repo and then
> remove the keystone override.
> - Changes to the testing-repo can cause all other repo tests (aside from the
> overarching openstack-ansible repository) from breaking.
>
>
> Where to find the PoC, what next?
>
> The repository can be found here:
> https://github.com/andymcc/openstack-ansible-testing-core
>
> This is a proof of concept so in no way is it considered complete or
> perfect, we are asking for more eyes on this; test it, let us know what you
> like/do not like/want changed/additional ideas to improve.
>
> Thanks!
>
> Andy
> irc: andymccr
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [openstack-ansible] When to purge the DB, and when not to purge the DB?

[Carter, Kevin]

+1 to #4 -- As an operator I have archive and audit requirements that
proactive automated DB pruning would likely get in the way of. If we
do produce pruning tools I think they should exist in the OPS repo
and, as a rule, should not be part of the general deployment/upgrade
framework.

On Sat, Jul 2, 2016 at 5:20 PM, Ian Cordasco  wrote:
>
> On Jul 2, 2016 10:37 AM, "Dan Smith"  wrote:
>>
>> > The question is whether we should do something like this:
>> >
>> > 1) As part of the normal execution of the service playbooks;
>> > 2) As part of the automated major upgrade (i.e. The step is not
>> > optional);
>> > 3) As part of the manual major upgrade (i.e. The step is optional);
>> > 4) Never.
>>
>> I am not an operator, but I would think that #4 is the right thing to
>> do. If I want to purge the database, it's going to be based on billing
>> reasons (or lack thereof) and be tied to another archival, audit, etc
>> policy that the "business people" are involved with. Install and
>> configuration of my services shouldn't really ever touch my data other
>> than mechanical upgrade scripts and the like, IMHO.
>>
>> Purging the database only during upgrades is not sufficient for large
>> installs, so why artificially tie it to that process? In Nova we don't
>> do data migrations as part of schema updates anymore, so it's not like a
>> purge is going to make the upgrade any faster...
>
> I agree with this sentiment. If OSA feels like it must provide automation
> for purging databases, it should be in the ops repo mentioned earlier.
>
> I see no reason to over extend upgrades with something not inherently
> necessary or appropriate for upgrades.
>
> --
> Ian
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev