Re: [openstack-dev] [craton] Nomination of Thomas Maddox as Craton core

[Jim Baker]

We can now call this vote: 4 out of 4 of the active cores, all +1. Welcome
to the team, Thomas!

On Thu, Mar 23, 2017 at 11:28 AM, git harry <git-ha...@live.co.uk> wrote:

> +1
>
> ------
> *From:* Jim Baker <jim.ba...@python.org>
> *Sent:* 21 March 2017 20:41
> *To:* OpenStack Development Mailing List
> *Subject:* [openstack-dev] [craton] Nomination of Thomas Maddox as Craton
> core
>
> *I nominate Thomas Maddox as a core reviewer for the Craton project.*
>
> Thomas has shown extensive knowledge of Craton, working across a range of
> issues in the core service, including down to the database modeling; the
> client; and corresponding bugs, blueprints, and specs. Perhaps most notably
> he has contributed a number of end-to-end patches, such as his work with
> project support.
> https://review.openstack.org/#/q/owner:thomas.maddox
>
> He has also expertly helped across a range of reviews, while always being
> amazingly positive with other team members and potential contributors:
> https://review.openstack.org/#/q/reviewer:thomas.maddox
>
> Other details can be found here on his contributions:
> http://stackalytics.com/report/users/thomas-maddox
>
> In my opinion, Thomas has proven that he will make a fantastic addition to
> the core review team. In particular, I'm confident Thomas will help further
> improve the velocity for our project as a whole as a core reviewer. I hope
> others concur with me in this assessment!
>
> - Jim
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [craton] Nomination of Thomas Maddox as Craton core

[Jim Baker]

*I nominate Thomas Maddox as a core reviewer for the Craton project.*

Thomas has shown extensive knowledge of Craton, working across a range of
issues in the core service, including down to the database modeling; the
client; and corresponding bugs, blueprints, and specs. Perhaps most notably
he has contributed a number of end-to-end patches, such as his work with
project support.
https://review.openstack.org/#/q/owner:thomas.maddox

He has also expertly helped across a range of reviews, while always being
amazingly positive with other team members and potential contributors:
https://review.openstack.org/#/q/reviewer:thomas.maddox

Other details can be found here on his contributions:
http://stackalytics.com/report/users/thomas-maddox

In my opinion, Thomas has proven that he will make a fantastic addition to
the core review team. In particular, I'm confident Thomas will help further
improve the velocity for our project as a whole as a core reviewer. I hope
others concur with me in this assessment!

- Jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [craton] Proposing Harry Harrington for Craton core

[Jim Baker]

+1 to both recommendations by Sulo.

Let me also add: Harry has proven himself as a strong contributor to Craton
development. I have been especially impressed by how his work has helped
improve the quality and consistency of our coding; often while removing
large chunks of unnecessary code at the same time by being smarter. I look
forward to more great contributions by Harry as a core team member!

On Wed, Feb 1, 2017 at 4:05 AM, Sulochan Acharya 
wrote:

> Hi everyone,
>
> I would like to propose  Harry Harrington @git-harry to Craton core team.
> He has been doing a great job with both reviewing and proposing meaningful
> changes to craton project. I am confident Harry will be great help as
> craton core member.
>
> https://github.com/openstack/craton/commits/master?author=git-harry
>
>
> I am also proposing to remove Sean Roberts from core team at this time.
> Sean has been great help to get Craton started but has been busy with other
> duties lately. We would be delighted to add Sean back in the future when
> his reviewing picks up again.
>
>
> Thanks,
> Sulo
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Craton] NFV planned host maintenance

[Jim Baker]

On Wed, Nov 16, 2016 at 10:54 AM, Sulochan Acharya 
wrote:

> Hi,
>
> On Wed, Nov 16, 2016 at 2:46 PM, Ian Cordasco 
> wrote:
>
>> -Original Message-
>> From: Juvonen, Tomi (Nokia - FI/Espoo) 
>> Reply: OpenStack Development Mailing List (not for usage questions)
>> 
>> Date: November 11, 2016 at 02:27:19
>> To: OpenStack Development Mailing List (not for usage questions)
>> 
>> Subject:  [openstack-dev] [Craton] NFV planned host maintenance
>>
>> > I have been looking in past two OpenStack summits to have changes
>> needed to
>> > fulfill OPNFV Doctor use case for planned host maintenance and at the
>> same
>> > time trying to find other Ops requirements to satisfy different needs.
>> I was
>> > just about to start a new project (Fenix), but looking Craton, it seems
>> > a good alternative and was proposed to me in Barcelona meetup. Here is
>> some
>> > ideas and would like a comment wither Craton could be used here.
>>
>> Hi Tomi,
>>
>> Thanks for your interest in craton! I'm replying in-line, but please
>> come and join us in #craton on Freenode as well!
>>
>> > OPNFV Doctor / NFV requirements are described here:
>> > http://artifacts.opnfv.org/doctor/docs/requirements/02-use_
>> cases.html#nvfi-maintenance
>> > http://artifacts.opnfv.org/doctor/docs/requirements/03-archi
>> tecture.html#nfvi-maintenance
>> > http://artifacts.opnfv.org/doctor/docs/requirements/05-imple
>> mentation.html#nfvi-maintenance
>> >
>> > My rough thoughts about what would be initially needed (as short as I
>> can):
>> >
>> > - There should be a database of all hosts matching to what is known by
>> Nova.
>>
>> So I think this might be the first problem that you'll run into with
>> Craton.
>>
>> Craton is designed to specifically manage the physical devices in a
>> data centre. At the moment, it only considers the hosts that you'd run
>> Nova on, not the Virtual Machines that Nova is managing on the Compute
>> hosts.
>>
>
Craton's inventory supports the following modeling:

   1. devices, which may have a parent (so a strict tree); we map this
   against such entities as top-of-rack switches; hosts; and containers
   2. logical relationships for these devices, including project, region,
   cell (optional); and arbitrary labels (tags)
   3. key/value variables on most entities, including devices. Variables
   support *resolution* - an override mechanism where values are looked up
   against some chain (for device, that's the device tree, cell, region, in
   that order). Values are typed JSON in the underlying (and default)
   SQLAlchemy model we use.

Craton users synchronize the device inventory from other source of truth
systems, such as an asset database; or perhaps manually. Meanwhile,
variables can reflect desired state configuration (so like Ansible); as
well as captured information.


>> It's plausible that we could add the ability to track virtual
>> machines, but Craton is meant to primarily work underneath the cloud.
>> I think this might be changing since Craton is looking forward to
>> helping manage a multi-cloud environment, so it's possible this won't
>> be an issue for long.
>>
>
Craton's device-focused model, although oriented to hardware, is rather
arbitrary. Recently we have been also looking at what is needed to support
a multi-tenant, multi-cloud inventory, and it seems quite feasible to
manage in Craton's inventory a subset of the resources provided by AWS or
Azure.

Does this mean VMs and similar resources? Maybe. However, our thinking has
been, for relatively fast changing and numerous resources, link to the
source of truth, in this case Nova. In particular, we have a very model for
variables that could be readily extended to support what we call
virtualized variables - dictionary mappings that are implemented by looking
up on a remote service. See https://bugs.launchpad.net/craton/+bug/1606882
- so long as it implements collections.abc.Mapping, we can plug into how
variables are resolved.


>
> So I think there is 2 parts to this. 1. What Ian mentioned is that Craton
> currently does not keep inventory of VM running inside an openstack
> deployment. Nova (and Horizon for UI) is already doing this for the users.
> However, we do run jobs or workload on the VM .. like live migrating VM's
> from a host that might undergo maintenance, or a host monitoring flagged as
> bad etc. This is done using `plugins` that talk to nova etc. So I think
> some of what you are looking for falls into that perhaps ? This can be
> based on some notification the Craton engine receives from another
> application (like monitoring for example).
>

>
>>
>> > - There should by an API for Cloud Admin to set planned maintenance
>> window
>> > for a host (maybe aggregate, group of hosts), when in maintenance and
>> unset
>> > when finished. There might be some optional parameters like target 

Re: [openstack-dev] [craton] branch clean up and other stuff

[Jim Baker]

Sean, sounds good.

Today, among other things, we will be discussing the branch cleanup, which
should be straightforward. We look forward to discussing more about OneOps,
and I will briefly mention our thoughts to date on how this could work.

https://etherpad.openstack.org/p/craton-meetings for the full agenda



On Mon, Jul 18, 2016 at 8:29 AM, sean roberts 
wrote:

> I'm going to have to miss this mornings meeting again. Here out my
> outstanding issues
> - project patch https://review.openstack.org/#/c/332470/9, is pending a
> request to clean up the branches in the craton repo. Once the repo is
> moved, branch cleanup is expensive time wise. Branches are typically minor
> and major releases only.
> - allocating devs to the project. I'm getting spun up around a few teams.
> After the watcher, nova mid cycle this week. I will have a better idea who
> is available.
> - forking OneOps CMS repos as code base start. We are definitely
> interested internally in doing this. I suggest I get my devs to propose how
> the fork would look and what the implications are. Then the craton team as
> whole can discuss the merits and modifications to the approach.
>
>
>
> --
> ~sean
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [craton] Craton developer meeting Thurs July 14 at 1700 UTC

[Jim Baker]

Please join us this Thursday for our weekly Craton developer video meeting.
To better accommodate people from Europe, we have moved the time for this
and subsequent Thursday meetings to 1700 UTC.

Topics to be discussed in this week's call include:

   - Inventory fabric. Recently we have decided this term best describes
   our inventory approach for Craton: a seamless wrapping of both Craton's
   source of truth information about inventory, such as config variables about
   hosts; and externally sourced data, using virtualization (much like
   symbolic links in a filesystem). There's still a lot of work to do to
   complete this vision.
   - Summary of discussion on proposed work with upstream TaskFlow to look
   at using containers for jobs. For Craton, our reference implementation is
   against OpenStack Ansible. Instead of using TaskFlow workers to in turn run
   an Ansible playbook, the idea is to run a container. This moves, and
   hopefully simplifies, the packaging problem for playbooks and any
   dependencies. If Craton has similar qualities to Hadoop map/reduce, then
   containers offers even more than packing jars and using ClassLoader
   approaches :)
   - Outstanding issues, as currently tracked here:
   https://waffle.io/rackerlabs/craton

Video conference calls are held on Vidyo:
https://vc.rackspace.com/flex.html?roomdirect.html=OpmgbcmKWcXB31Tbl7VHgTrytb0

(Please note, these meetings will be recorded and posted to YouTube, so if
you do not wish to be recorded, please do not attend the meeting.)

- Jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [craton] Developer weekly meeting scheduled every Thursday at 2100 UTC

[Jim Baker]

On Wed, Jun 29, 2016 at 2:59 PM, Ian Cordasco <sigmaviru...@gmail.com>
wrote:

> -Original Message-
> From: Jim Baker <jim.ba...@python.org>
> Reply: OpenStack Development Mailing List (not for usage questions)
> <openstack-dev@lists.openstack.org>
> Date: June 29, 2016 at 14:08:25
> To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org>
> Subject:  [openstack-dev] [craton] Developer weekly meeting scheduled
> every Thursday at 2100 UTC
>
> > Please join us for our weekly developer video conference call about
> Craton
> > development every Thursday at 2100 UTC.
> >
> > We meet on Vidyo; please use this link:
> >
> https://vc.rackspace.com/flex.html?roomdirect.html=OpmgbcmKWcXB31Tbl7VHgTrytb0
> >
> > Meetings will be recorded and posted to YouTube, so if you do not wish to
> > be recorded, please do not attend the meeting. The agenda can be found at
> > https://etherpad.openstack.org/p/craton-meetings, but most often we will
> > use a standing agenda: to discuss open development issues.
> >
> > - Jim
>
> Is this replacing the weekly Monday IRC meetings?
>

Good question!

No, this video conference call will not replace our weekly IRC meeting on
#openstack-meeting-4.

Instead, these two meetings will hopefully complement each other:

   - Monday's IRC meeting on #openstack-meeting-4 will set overall
   direction for Craton, including interaction with stakeholders and
   determining action items. Agendas will be published (
   https://etherpad.openstack.org/p/craton-meetings), and minutes recorded
   with meetbot and published on eavesdrop.
   - Thursday's Vidyo call will focus on development issues. We will record
   this call, and I'm sure notes will be taken in relevant etherpads, but it
   will be much more informal because we are simply trying to help each other
   make progress on our issues. Think standup, brainstorming, etc.


I'm sure these weekly meetings will evolve over time, and I welcome all
input on how to make them as effective as possible.

- Jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [craton] Developer weekly meeting scheduled every Thursday at 2100 UTC

[Jim Baker]

Please join us for our weekly developer video conference call about Craton
development every Thursday at 2100 UTC.

We meet on Vidyo; please use this link:
https://vc.rackspace.com/flex.html?roomdirect.html=OpmgbcmKWcXB31Tbl7VHgTrytb0

Meetings will be recorded and posted to YouTube, so if you do not wish to
be recorded, please do not attend the meeting. The agenda can be found at
https://etherpad.openstack.org/p/craton-meetings, but most often we will
use a standing agenda: to discuss open development issues.

- Jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [craton] No IRC meeting on Monday July 4

[Jim Baker]

Many of us will be celebrating the July 4 holiday in the US, so we will not
be having our regularly scheduled meeting in #openstack-meeting-4 this
coming Monday.

Craton meetings are planned here:
https://etherpad.openstack.org/p/craton-meetings

- Jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [craton] Midcycle meetup to discuss fleet management - Aug 23-26 in NYC

[Jim Baker]

The Craton fleet management midcycle meetup will be held in New York City,
August 23-26, using the following two events to provide structure/meeting
support:

   - OpenStack East - developer-focused discussion on Craton core, along
   with further development of integration points;
   http://www.openstackeast.com/
   - Operators midcycle hosted by Bloomberg - get feedback from operators
   and course correction;
   
http://lists.openstack.org/pipermail/openstack-operators/2016-June/010788.html

Please join us! We also meet on #openstack-meeting-4 (1500 UTC Monday) and
hang out on #craton (Freenode).

Craton is a new open source project that we plan to propose for OpenStack
inclusion ("big tent"). Craton supports deploying and operating OpenStack
clouds by providing scalable fleet management. It provides inventory;
scalable workflows for audit and remediation (in progress); and
corresponding REST APIs/CLI/Python client. We are targeting OpenStack
Ansible in terms of reference workflows.

Craton is based on how Rackspace currently operates its public cloud at
scale, but it is also intended to support private clouds, from small to
large. We are building it on SQLAlchemy, TaskFlow, and Flask; along with
*optional* undercloud integration with Keystone, Barbican, and Horizon.
https://github.com/rackerlabs/craton

(I earlier also posted this email on openstack-operators, intending instead
for this list. But in retrospect, this email seems like a reasonable cross
post.)

- Jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [craton] OneOps CMS demo

[Jim Baker]

Just to correct the UTC time based on the calendar invite I got for this
discussion:

We are meeting at *1600 UTC*, or 9am PDT

At some point, we should just set our calendars to only
use Atlantic/Reykjavik (= UTC) and stop doing the timezone arithmetic :)

On Mon, Jun 27, 2016 at 3:11 PM, sean roberts 
wrote:

> Vitaliy is available 9:00 PDT, 15:00 GMT tomorrow. Let's use webex link
> here
> https://walmart.webex.com/walmart/j.php?MTID=m2e306e0b4a7fec980b21cdc5a02f7815
>
> This is about the option to fork CMS for use in Craton. Everyone join in!
>
>
> --
> ~sean
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Reasoning behind my vote on the Go topic

[Jim Baker]

On Tue, Jun 7, 2016 at 4:26 PM, Ben Meyer  wrote:

> On 06/07/2016 06:09 PM, Samuel Merritt wrote:
> > On 6/7/16 12:00 PM, Monty Taylor wrote:
> >> [snip]
> > >
> >> I'd rather see us focus energy on Python3, asyncio and its pluggable
> >> event loops. The work in:
> >>
> >> http://magic.io/blog/uvloop-blazing-fast-python-networking/
> >>
> >> is a great indication in an actual apples-to-apples comparison of what
> >> can be accomplished in python doing IO-bound activities by using modern
> >> Python techniques. I think that comparing python2+eventlet to a fresh
> >> rewrite in Go isn't 100% of the story. A TON of work has gone in to
> >> Python that we're not taking advantage of because we're still supporting
> >> Python2. So what I've love to see in the realm of comparative
> >> experimentation is to see if the existing Python we already have can be
> >> leveraged as we adopt newer and more modern things.
> >
> > Asyncio, eventlet, and other similar libraries are all very good for
> > performing asynchronous IO on sockets and pipes. However, none of them
> > help for filesystem IO. That's why Swift needs a golang object server:
> > the go runtime will keep some goroutines running even though some
> > other goroutines are performing filesystem IO, whereas filesystem IO
> > in Python blocks the entire process, asyncio or no asyncio.
>
> That can be modified. gevent has a tool
> (http://www.gevent.org/gevent.fileobject.html) that enables the File IO
> to be async  as well by putting the file into non-blocking mode. I've
> used it, and it works and scales well.
>
> Sadly, Python doesn't offer this by default; perhaps OpenStack can get
> that changed.
>
> $0.02
>
> Ben
>
>
The uvloop library is very new, but libuv itself uses a thread pool, so it
should work just fine with files once this functionality is implemented;
see https://github.com/MagicStack/uvloop/issues/1 and
https://nikhilm.github.io/uvbook/filesystem.html

I don't see any difference here with how Golang would implement such async
support, in terms of mapping goroutines against its own thread pool. With
respect to latency, there could even be better performance for certain
workloads with uvloop than Go. This is because of CPython's refcounting, vs
Go's GC - even with all the recent improvements in Go (
https://blog.golang.org/go15gc), Go still does stop the world.

A key piece here is that uvloop does not hold the GIL in these ops - a big
advantage that Python C extensions enjoy, including libraries they wrap,
and Cython explicitly enables with nogil support. So in particular we have
the following code in the the core uvloop function, which in turn is just
calling into libuv with its own core function, uv_run:

```
# Although every UVHandle holds a reference to the loop,
# we want to do everything to ensure that the loop will
# never deallocate during the run -- so we do some
# manual refs management.
Py_INCREF(self)
with nogil:
err = uv.uv_run(self.uvloop, mode)
Py_DECREF(self)
```

More here on nogil support in Cython:
http://docs.cython.org/src/userguide/external_C_code.html#declaring-a-function-as-callable-without-the-gil

- Jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev