[openstack-dev] [ironic] [API] Evolving our REST API

2016-10-13 Thread Devananda van der Veen
Hi all,

We discussed a little at the last summit, and discussed further at the midcycle
[1], about how we might go about remedying some of the frustrations and missing
functionality in our API, and I volunteered to work on it during the Newton 
cycle.

As I looked at all of the feedback we collected and thought about these issues,
I became convinced we could make most, if not all, of the changes with small
steps. Together, they bring a lot of new functionality, but without a complete
API rewrite and the accompanying burden of carrying two versions of the API.

So I have finalized five proposals of substantial changes we can make, that
folks agreed were important to work on, and which I believe we can do within the
microversion framework starting immediately. Four of them will, I think, be
fairly straight forward. The fifth, adding a /tasks/ resource, has the most
challenges, and its own session planned.

I have posted the specs here:

https://review.openstack.org/#/q/status:open+project:openstack/ironic-specs+branch:master+topic:api-evolution

Please give them a read, and let's discuss them in Barcelona [2].

It would be great to have someone from the API working group also peruse these
proposals and validate the direction.

Cheers,
Devananda


[1] https://etherpad.openstack.org/p/ironic-newton-midcycle around L390
[2] https://etherpad.openstack.org/p/ironic-ocata-summit-api-evolution

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] [infra] RFC: consolidating and extending Ironic CI jobs

2016-10-12 Thread Devananda van der Veen


On 10/12/2016 05:01 AM, Dmitry Tantsur wrote:
> Hi folks!
> 
> I'd like to propose a plan on how to simultaneously extend the coverage of our
> jobs and reduce their number.
> 
> Currently, we're running one instance per job. This was reasonable when the
> coreos-based IPA image was the default, but now with tinyipa we can run up to 
> 7
> instances (and actually do it in the grenade job). I suggest we use 6 fake bm
> nodes to make a single CI job cover many scenarios.
> 
> The jobs will be grouped based on driver (pxe_ipmitool and agent_ipmitool) to 
> be
> more in sync with how 3rd party CI does it. A special configuration option 
> will
> be used to enable multi-instance testing to avoid breaking 3rd party CI 
> systems
> that are not ready for it.
> 
> To ensure coverage, we'll only leave a required number of nodes "available", 
> and
> deploy all instances in parallel.
> 
> In the end, we'll have these jobs on ironic:
> gate-tempest-ironic-pxe_ipmitool-tinyipa
> gate-tempest-ironic-agent_ipmitool-tinyipa
> 
> Each job will cover the following scenarious:
> * partition images:
> ** with local boot:
> ** 1. msdos partition table and BIOS boot
> ** 2. GPT partition table and BIOS boot
> ** 3. GPT partition table and UEFI boot  <*>
> ** with netboot:
> ** 4. msdos partition table and BIOS boot <**>
> * whole disk images:
> * 5. with msdos partition table embedded and BIOS boot
> * 6. with GPT partition table embedded and UEFI boot  <*>
> 
>  <*> - in the future, when we figure our UEFI testing
>  <**> - we're moving away from defaulting to netboot, hence only one scenario
> 
> I suggest creating the jobs for Newton and Ocata, and starting with Xenial 
> right
> away.
> 
> Any comments, ideas and suggestions are welcome.

Huge +1 on this from me, as well.

I am also in favor of some of the other suggestions on this thread, namely,
moving API testing over to functional tests so that those can be run more
quickly / with less resources / without affecting tempest scenario tests.

I also think we should begin defining additional scenario tests to cover things
we are not covering today  (eg, adopt a running instance), as Vasyl already
pointed out. But I don't think that conflicts or prevents the changes you're
suggesting, Dmitry.

-Deva

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Next API meeting cancelled

2016-09-22 Thread Devananda van der Veen
Considering I'm the only one currently working on it / bringing up new topics, I
think biweekly is a better match to the pace I'm working on this.

However, I hope that changes after the summit / once we start implementing these
changes.

--d

On 09/22/2016 08:38 AM, Jim Rollenhagen wrote:
> We decided in the last meeting to cancel next week's meeting. So we'll
> meet again October 4.
> 
> Side question: should we just make this meeting biweekly always?
> 
> // jim
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Driver removal policies - should we make it softer?

2016-08-22 Thread Devananda van der Veen
I like this approach more than the current "you're in or you're out".

+1

--deva

On Mon, Aug 22, 2016 at 7:02 AM Vladyslav Drok  wrote:

> +1 from me then :)
>
> On Mon, Aug 22, 2016 at 4:52 PM, Jim Rollenhagen 
> wrote:
>
>> On Mon, Aug 22, 2016 at 8:01 AM, Vladyslav Drok 
>> wrote:
>> > On Fri, Aug 19, 2016 at 5:15 PM, Jim Rollenhagen <
>> j...@jimrollenhagen.com>
>> > wrote:
>> >>
>> >> Hi Ironickers,
>> >>
>> >> There was a big thread here[0] about Cinder, driver removal, and
>> standard
>> >> deprecation policy. If you haven't read through it yet, please do
>> before
>> >> continuing here. :)
>> >>
>> >> The outcome of that thread is summarized well here.[1]
>> >>
>> >> I know that I previously had a different opinion on this, but I think
>> we
>> >> should go roughly the same route, for the sake of the users.
>> >>
>> >> 1) A ``supported`` flag for each driver that is True if and only if the
>> >> driver
>> >>is tested in infra or third-party CI (and meets our third party CI
>> >>requirements).
>> >> 2) If the supported flag is False for a driver, deprecation is implied
>> >> (and
>> >>a warning is emitted at load time). A driver may be removed per
>> >> standard
>> >>deprecation policies, with turning the supported flag False to start
>> >> the
>> >>clock.
>> >> 3) Add a ``enable_unsupported_drivers`` config option that allows
>> enabling
>> >>drivers marked supported=False. If a driver is in enabled_drivers,
>> has
>> >>supported=False, and enable_unsupported_drivers=False,
>> ironic-conductor
>> >>will fail to start. Setting enable_unsupported_drivers=True will
>> allow
>> >>ironic-conductor to start with warnings emitted.
>> >
>> >
>> > Just for clarity, its default value is False?
>>
>> Sorry, I meant to add that. Yes, default for
>> enable_unsupported_drivers is False.
>>
>> // jim
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Driver composition defaults call

2016-08-15 Thread Devananda van der Veen


On 08/11/2016 10:37 AM, Julia Kreger wrote:
> Yesterday as a group (jroll, rloo, dtantsur, matt128, devananda,
> vdrok, and myself) discussed defaults for driver composition.
> 
> The options we discussed were:
> 
> * The existing specification[0] - Global and hardware_type
> default_FOO_interface configuration, global enabled_FOO_interfaces in
> configs, supported_FOO_interface in the hardware_type.
> 
> * Sambetts proposal[1] - To base any defaults on the intersection of
> enabled_FOO_interfaces and the supported_FOO_interface lists taking
> the first common option.
> 
> During the discussion the group came to the conclusion that if we were
> to enable the ability to set defaults solely by list ordering, as put
> forth in sambetts proposal, the question would then shift to who knows
> best. The operator, or the vendor via the hardware_type. This further
> evolved into substantial amounts of potential configuration which we
> seemed to agree was confusing and unrealistic. We eventually circled
> back to the original intent of the global configuration
> default_FOO_interface which was to make an operator’s life easier by
> allowing the definition of what would by in large be an explicitly
> chosen environmental or operating default.
> 
> Circling back to the intent allowed us to focus the discussion further
> and decide the following:
> 
> 1. If the client creating the node does not set an explicit
> FOO_interface, we must save whatever is determined as the default, in
> node.FOO_interface.
> 
> 2. To determine a default if one is not explicitly set via a
> default_FOO_interface, the intersection between the hardware_type
> definition supported_FOO_interfaces and the enabled_FOO_interfaces
> global configuration would be used to determine the default.
> 
> Having determined the two preceding items, we reached a consensus that
> the resulting default that is determined, must be present in
> enabled_FOO_interfaces list. The result of this is that there should
> be no implicit enablement of drivers, and the operator should be
> selecting the interfaces possible for their environment in the
> enabled_FOO_interfaces global configuration setting.
> 
> In following up with sambetts this morning and discussing the concerns
> that drove his proposal initially, Sam and I determined that any
> implicit enablement of drivers was not ideal, that an operator
> explicit default override for its intended purpose seemed okay, and
> that the determination of any default should come from the the
> intersection of what is supported versus what is enabled, as the
> larger group reached a consensus on.  That this would ultimately
> result in default_FOO_interface dropping from the hardware_type and
> only being present as global configuration option for an operator to
> use if they so choose to do so.  This seems in-line with what the
> group reached while on the call yesterday.
> 
> Conceptually, this leaves us with something that looks like the
> following when nodes are created without a valid FOO_interface in the
> initial API post.
> 
> 1. The hardware_type's supported_FOO_interfaces is in order of
> preference by the vendor, for example: supported_FOO_interface = [BAR,
> CAR, DAR] this represents: if BAR enabled then use BAR else if CAR
> enabled then use CAR else if DAR enabled then use DAR.
> 
> 2. possible_FOO_interfaces to use for a hardware_type are calculated
> by intersecting enabled_FOO_interfaces and the hardware_type's
> supported_FOO_interfaces, order as in supported_FOO_interface is
> maintained.
> 
> 3. If configuration option default_FOO_interface is set AND
> default_FOO_interface is in possible_FOO_interfaces THEN
> node.FOO_interface is set to default_FOO_interface
> 
> 4. If configuration option default_FOO_interface is set AND
> default_FOO_interface is NOT in possible_FOO_interfaces THEN node
> create fails
> 
> 5. If configuration option default_FOO_interface is NOT set THEN
> node.FOO_interface is set to the first interface in
> possible_FOO_interface
> 

Thanks, Julia, for this excellent summary of the discussion.

This logic appears sound and, I believe, is as simple as possible, while not
unduly limiting operator or vendor choice.

+1

-Deva


> Thank you Sam for typing out the above logic.  I think this means we
> seem to have some consensus on the direction to move forward, at least
> I hope. :)
> 
> -Julia
> 
> [0] 
> http://specs.openstack.org/openstack/ironic-specs/specs/approved/driver-composition-reform.html
> [1] http://lists.openstack.org/pipermail/openstack-dev/2016-July/099257.html
> 
> On Mon, Aug 8, 2016 at 8:51 AM, Julia Kreger
>  wrote:
>> Thank you for sending the corrected link Mathieu!  I thought I fixed it
>> before I sent the email, but... *shrug*
>>
>> Anyway!  Looking at the doodle, the mutually available time is 4 PM UTC on
>> this Wednesday (8/10/16).  If there are no objections, I guess we will hear
>> those seeking to discuss defaults on 

Re: [openstack-dev] [ironic] static Portgroup support.

2016-08-10 Thread Devananda van der Veen
On 08/09/2016 01:28 AM, Vasyl Saienko wrote:
> Hello Ironic'ers!
> 
> We've recorded a demo that shows how static portgroup works at the moment:
> 
> Flat network scenario: https://youtu.be/vBlH0ie6Lm4
> Multitenant network scenario: https://youtu.be/Kk5Cc_K1tV8

Awesome! Thank you for creating & sharing these demos!

--deva

> 
> Sincerely,
> Vasyl Saienko
> 
> On Tue, Jul 19, 2016 at 3:30 PM, Vasyl Saienko  > wrote:
> 
> Hello Community,
> 
> Current portgroup scenario is not fully clear for me. The related spec [3]
> doesn't clearly describe it. And based on implementation [1] and [2] I 
> guess
> it should work in the following fashion for node with 3 NICs, where eth1 
> and
> eth2 are members of Porgroup Po0/1
> 
> Node network connection info:
> eth1 (aa:bb:cc:dd:ee:f1) <---> Gig0/1
> eth2 (aa:bb:cc:dd:ee:f2) <---> Gig0/2
> eth3 (aa:bb:cc:dd:ee:f3) <---> Gig0/3
>  
> For FLAT network scenario:
> 1. Administrator enrol ironic node.
> 2. Administrator creates a 3 ports for each interface, and a portgroup 
> that
> contains eth0 and eth1 ports.
> 3. The ports Gig0/1 and Gig0/2 are added to portgroup Po0/1 manually on 
> the
> switch.
> 4. When user request to boot an instance, Nova randomly picks interface 
> [2],
> it might be a portgroup or single NIC interface. Proposed change [1] do 
> not
> allow to specify what exactly network type we would like to use single NIC
> or portgroup.
> 
> For multitenancy case:
> All looks the same, in addition administrator adds local_link_connection
> information for each port (local_link_connection 'port_id' field is 
> 'Gig0/1'
> for eth1, 'Gig0/2' for eth2 and 'Gig0/3' for eth3, ). Ironic send this
> information to Neutron who plugs ports to needed network.
> 
> The same user-scenario is available at the moment without any changes to
> Nova or Ironic. The difference is that administrator creates one port for
> single interface eth3 with local_link_connection 'port_id'='Gig0/3',  and 
> a
> port that is a logical representation of portgroup (eth1 and eth2) with
> local_link_connection 'port_id'='Po0/1'. 
> 
> Please let me know if I've missed something or misunderstood current
> portgroup scenario.
> 
> Reference:
> [0] https://review.openstack.org/206163 
> 
> [1] https://review.openstack.org/332177 
> 
> [2]
> 
> https://github.com/openstack/nova/blob/06c537fbe5bb4ac5a3012642c899df815872267c/nova/network/neutronv2/api.py#L270
> 
> 
> [3]
> 
> https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/ironic-ml2-integration.html
> 
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] v2 API - request for feedback on "problem description"

2016-08-02 Thread Devananda van der Veen
Hi all,

Today's ironic-v2-api meeting was pretty empty, so I am posting a summary of our
subteam's activity here.

I have taken the midcycle notes about our API's current pain points / usability
gaps, and written them up into the format we would use for a spec's "Problem
Description", and posted them into an etherpad:

  https://etherpad.openstack.org/p/ironic-v2-api

As context for folks that may not have been in the midcycle discussion, my goal
in this work is to, by Barcelona, have a concrete outline of the problems with
our API -- and a proposal of how we might solve them -- written up as a design
specification.

I would like to invite feedback on this very early draft ahead of our weekly
meeting next week, and then discuss it for a portion of the meeting on Monday.


Thanks for your time,
Devananda

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] network_interface, defaults, and explicitness

2016-08-02 Thread Devananda van der Veen
On 08/01/2016 05:10 AM, Jim Rollenhagen wrote:
> Hey all,
> 
> Our nova patch for networking[0] got stuck for a bit, because Nova needs
> to know which network interface is in use for the node, in order to
> properly set up the port.
> 
> The code landed for network_interface follows the following order for
> what is actually used for the node:
> 1) node.network_interface, if that is None:
> 2) CONF.default_network_interface, if that isNone:
> 3) flat, if using neutron DHCP
> 4) noop, if not using neutron DHCP
> 
> The API will return None for node.network_interface in the API (GET
> /v1/nodes/uuid). This won't work for Nova, because Nova can't know what
> CONF.default_network_interface is.
> 
> I propose that if a network_interface is not sent in the node-create
> call, we write whatever the current default is, so that it is always set
> and not using an implicit value that could change.
> 
> For nodes that exist before the upgrade, we do a database migration to
> set network_interface to CONF.default_network_interface (or if that's
> None, set to flat/noop depending on the DHCP provider).

Sounds quite reasonable to me.

--deva


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] [nova] [neutron] get_all_bw_counters in the Ironic virt driver

2016-07-28 Thread Devananda van der Veen


On 07/28/2016 05:40 PM, Brad Morgan wrote:
> I'd like to solicit some advice about potentially implementing
> get_all_bw_counters() in the Ironic virt driver.
> 
> https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L438
> Example Implementation:
> https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L320
> 
> I'm ignoring the obvious question about how this data will actually be
> collected/fetched as that's probably it's own topic (involving neutron), but I
> have a few questions about the Nova -> Ironic interaction:
> 
> Nova
> * Is get_all_bw_counters() going to stick around for the foreseeable future? 
> If
> not, what (if anything) is the replacement?
> 
> Ironic
> * I assume Ironic would be responsible for knowing how to fetch bandwidth
> counters for a given instance - correct?

The nova.virt.ironic driver would be responsible for implementing that method --
but I don't think that it makes sense to fetch that information from Ironic.

In some cases, it may be possible for the Node's management controller (eg, the
iLO) to collect/measure/expose network traffic counters for each physical
interface on the Node. None of Ironic's in-tree drivers support gathering this
data, afaik; Ironic isn't capturing it, and we don't have an API to expose it
today. If we went this route, it would be a vendor-specific thing, and not
supported by the _*ipmitool class of drivers. In other words, I don't think we
could have a fully open source production-oriented implementation of this 
feature.

On the other hand, with the Neutron integration now underway, if one were using
Neutron and OVS or OVN to manage the physical switches, then I would think that
Neutron could expose the bandwidth counters on the VIFs associated with the
Instance // with the user-defined Ports. I believe OVS supports this, but I
don't see anything in the Neutron API that actually exposes it... (IANANE, so it
may very well be there and I just didn't find it)

I'll defer to Neutron folks here. If the VIF's bandwidth counters can be fetched
from neutron, that would be ideal, as it should work regardless of the server's
management controller.

(I've added [neutron] to the subject line to solicit their input)


--deva

> * If so, what would this look like? (I'm assuming some Ironic API endpoint 
> Nova
> simply calls for counters - but any specific guidance here would be great.)
> 

I

> I appreciate any tips/suggestions, thank you. 
> 
> -- 
> Brad Morgan
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] why do we need setting network driver per node?

2016-07-11 Thread Devananda van der Veen
We spent the majority of today's weekly IRC meeting [1] discussing the finer
points of this question. I agreed to post a summary of those to the list (it's
below the break).

tldr;

* we don't know if network_interface should behave like other hardware
interfaces, but...
* we need to come to an agreement on it this week, so we can proceed with the
network integration work.


Background:

- As proposed in the driver composition reform spec [2], each hardware_type
class (eg, ilo_gen8) shall specify a set of allowable interface implementations
(eg, noop, flat, neutron are all network_interfaces) for each interface type
(eg, deploy_interface, power_interface, network_interface).
- A hardware_type shall also indicate a default for each interface_type. (**
this is what we debated ** )
- There is no CONF option to specify a global default for each interface_type
(eg, there is no CONF.default_power_interface setting, because it varies by
hardware_type)
- There will be a CONF option to enable ("allow") hardware classes and CONF
options to enable ("allow") interface classes.


Points raised in the meeting today:

- in "most" cases, network_interface will depend more on the environment than a
specific Node's hardware or out-of-band management device

- in "exceptional" cases, a Node may only support specific network_interfaces
(eg, certain Cisco hardware, a Blade enclosure)

- maybe hardware_type should specify a default for some interfaces but not
others? Example:
  - the ilo_gen8 hardware class should specify power, deploy, and management
interface defaults to ilo-specific interface classes
  - but the operator should set the network interface as appropriate to their
environment

- if we were to force the operator to specify Node.network_interface (but not
any other interface) when enrolling every node, this will be apoor experience.

- we should add a global CONF setting to allow operators to set a default
network interface for their environment.

- words are hard. we shouldn't call network_interface an "interface" if it
behaves differently than other interfaces.

- we have two "types" of interfaces on drivers: hardware-types and
environment-types. maybe we should treat them differently?

- other things also depend on the environment (rather than the Node's hardware
or BMC) such as deploy_interface and volume_interface.


There was a proposal from sambetts towards the end of the meeting, which I've
edited for clarity (please correct if I misunderstood any of your points). This
seems to capture/address most of the points above and proposes a way forward,
while keeping within the intent of our driver composition reform spec. It was
the only clear suggestion during the meeting.

- in-tree hardware_types declare supported_network_interfaces to be empty [4]
and declare no default_network_interface
- we add a CONF option for default_network_interface, with a sane default value
- this CONF option is validated on conductor load to be supported by all loaded
hardware_types, and the conductor refuses to start if this CONF option is set to
a value not supported by one or more enabled_hardware_types
- if a(n out of tree) hardware_type declares a default_network_interface, this
will take precedence over the CONF option
- a node created without specifying a network interface will check the
hardware_type's supported_network_interfaces, and when that is empty, fall back
to the CONF.default_network_interface, just as other interfaces fall back to the
hardware_type's relevant default
- operators can override a specific node's network_interface, which follows the
usual rules for setting an interface on a Node (ie, the interface must be in
hardware_type.supported_network_interfaces AND in 
CONF.enabled_network_interfaces)


If anyone else has other clear suggestions that address all the issues here,
please reply with them.

I'm going to make myself available tomorrow at 1700 UTC, in both IRC and by
voice [3] (conference line # TBD) to discuss this with anyone. If we need to
discuss it again on Wednesday, we can.


Thanks much,
--devananda



[1] starting at 17:20

http://eavesdrop.openstack.org/meetings/ironic/2016/ironic.2016-07-11-17.00.log.html

[2]
http://specs.openstack.org/openstack/ironic-specs/specs/approved/driver-composition-reform.html

[3] https://wiki.openstack.org/wiki/Infrastructure/Conferencing

[4] I think we may need to have in-tree drivers declare
supported_network_interfaces to be [noop, flat, neutron], but that is not what
Sam suggested during the meeting



On 06/28/2016 08:32 AM, Dmitry Tantsur wrote:
> Hi folks!
> 
> I was reviewing https://review.openstack.org/317391 and realized I don't quite
> understand why we want to have node.network_interface. What's the real life 
> use
> case for it?
> 
> Do we expect some nodes to use Neutron, some - not?
> 
> Do we expect some nodes to benefit from network separation, some - not? There
> may be a use case here, but then we have to expose this field to Nova for
> 

Re: [openstack-dev] [kolla][ironic] My thoughts on Kolla + BiFrost integration

2016-06-27 Thread Devananda van der Veen
At a quick glance, this sequence diagram matches what I envisioned/expected.

I'd like to suggest a few additional steps be called out, however I'm not sure
how to edit this so I'll write them here.


As part of the installation of Ironic, and assuming this is done through
Bifrost, the Actor should configure Bifrost for their particular network
environment. For instance: what eth device is connected to the IPMI network;
what IP ranges can Bifrost assign to physical servers; and so on.

There are a lot of other options during the install that can be changed, but the
network config is the most important. Full defaults for this roles' config
options are here:
 
https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/defaults/main.yml

and documentation is here:
 
https://github.com/openstack/bifrost/tree/master/playbooks/roles/bifrost-ironic-install



Immediately before "Ironic PXE boots..." step, the Actor must perform an action
to "enroll" hardware (the "deployment targets") in Ironic. This could be done in
several ways: passing a YAML file to Bifrost; using the Ironic CLI; or something
else.


"Ironic reports success to the bootstrap operation" is ambiguous. Ironic does
not currently support notifications, so, to learn the status of the deployments,
you will need to poll the Ironic API (eg, "ironic node-list").



Cheers,
--Devananda

On 06/23/2016 06:54 PM, Steven Dake (stdake) wrote:
> Hey folks,
> 
> I created the following sequence diagram to show my thinking on Ironic
> integration.  I recognize some internals of the recently merged bifrost 
> changes
> are not represented in this diagram.  I would like to see a bootstrap action 
> do
> all of the necessary things to bring up BiFrost in a container using Sean's 
> WIP
> Kolla patch followed by bare metal minimal OS load followed by Kolla 
> dependency
> software (docker-engine, docker-py, and ntpd) loading and initialization.
> 
> This diagram expects ssh keys to be installed on the deployment targets via 
> BiFrost.
> 
> https://creately.com/diagram/ipt09l352/ROMDJH4QY1Avy1RYhbMUDraaQ4%3D
> 
> Thoughts welcome, especially from folks in the Ironic community or Sean who is
> leading this work in Kolla.
> 
> Regards,
> -steve
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Proposing two new cores

2016-06-16 Thread Devananda van der Veen


On 06/16/2016 08:12 AM, Jim Rollenhagen wrote:
> Hi all,
> 
> I'd like to propose Jay Faulkner (JayF) and Sam Betts (sambetts) for the
> ironic-core team.
> 
> Jay has been in the community as long as I have, has been IPA and
> ironic-specs core for quite some time. His background is operations, and
> he's getting good with Python. He's given great reviews for quite a
> while now, and the number is steadily increasing. I think it's a
> no-brainer.
> 
> Sam has been in the ironic community for quite some time as well, with
> close ties to the neutron community as well. His background seems to be
> in networking, he's got great python skills as well. His reviews are
> super useful. He doesn't have quite as many as some people, but they are
> always thoughtful, and he often catches things others don't. I do hope
> to see more of his reviews.
> 
> Both Sam and Jay are to the point where I consider their +1 or -1 as
> highly as any other core, so I think it's past time to allow them to +2
> as well.
> 
> Current cores, please reply with your vote.
> 

+2 to both!

--devananda



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic][infra][qa] Ironic grenade work nearly complete

2016-06-10 Thread Devananda van der Veen
On 06/10/2016 05:48 AM, Sean Dague wrote:
> On 06/10/2016 08:41 AM, Jeremy Stanley wrote:
>> On 2016-06-10 11:49:12 +0100 (+0100), Miles Gould wrote:
>>> On 09/06/16 23:21, Jay Faulkner wrote:
 There was some discussion about whether or not the Ironic grenade job
 should be in the check pipeline (even as -nv) for grenade,
>>>
>>> Not having this would mean that changes to grenade could silently break
>>> Ironic's CI, right? That sounds really bad.
>>
>> That's like saying it's really bad that changes to devstack could
>> silently break devstack-based jobs for random projects, and so they
>> should be tested against every one of those jobs. At some point you
>> have to draw a line between running a reasonably representative
>> sample and running so many jobs that you'll never be able to merge
>> another change again (because even very small nondeterministic
>> failure rates compound to make that impossible at a certain scale).
> 
> Nothing should be voting in check in grenade that requires a plugin.
> 
> I'm fine with a few things in check nv if they are doing something out
> of the ordinary that we think needs to be kept on top of. I also expect
> that ironic folks are going to watch for those failures, and say, with
> -1/+1 CR, when they are legit and when it was off the rails. A non
> voting job that doesn't have domain experts validating the content
> regularly with CR means it gets ignored if it fails a bunch.
> 

I'd like to see this job running in the grenade check queue so we can watch it
there, and trace back to anything that affects us in unexpected ways. As Sean
points out, it should not be made voting in grenade's check queue.

It _should_ be made voting in Ironic's queue as soon as we have gathered
stability data for it. I'd love to see that get turned on in a week. With the
current patch volume, I think we'll be able to get plenty of stability data in
that time.

--devananda

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] using ironic as a replacement for existing datacenter baremetal provisioning

2016-06-07 Thread Devananda van der Veen
On 06/07/2016 09:55 AM, Joshua Harlow wrote:
> Joshua Harlow wrote:
>> Clint Byrum wrote:
>>> Excerpts from Joshua Harlow's message of 2016-06-07 08:46:28 -0700:
 Clint Byrum wrote:
> Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +:
>> Hi ironic folks,
>> As I'm trying to explore how GoDaddy can use ironic I've created
>> the following in an attempt to document some of my concerns, and
>> I'm wondering if you folks could help myself identity ongoing work
>> to solve these (or alternatives?)
> Hi Kris. I've been using Ironic in various forms for a while, and I can
> answer a few of these things.
>
>> List of concerns with ironic:
>>
>> 1.)Nova<-> ironic interactions are generally seem terrible?
> I don't know if I'd call it terrible, but there's friction. Things that
> are unchangable on hardware are just software configs in vms (like mac
> addresses, overlays, etc), and things that make no sense in VMs are
> pretty standard on servers (trunked vlans, bonding, etc).
>
> One way we've gotten around it is by using Ironic standalone via
> Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
> and includes playbooks to build config drives and deploy images in a
> fairly rudimentary way without Nova.
>
> I call this the "better than Cobbler" way of getting a toe into the
> Ironic waters.
>
> [1] https://github.com/openstack/bifrost
 Out of curiosity, why ansible vs turning
 https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py
 (or something like it) into a tiny-wsgi-app (pick useful name here) that
 has its own REST api (that looks pretty similar to the public functions
 in that driver file)?
>>>
>>> That's an interesting idea. I think a reason Bifrost doesn't just import
>>> nova virt drivers is that they're likely _not_ a supported public API
>>> (despite not having _'s at the front). Also, a lot of the reason Bifrost
>>> exists is to enable users to get the benefits of all the baremetal
>>> abstraction work done in Ironic without having to fully embrace all of
>>> OpenStack's core. So while you could get a little bit of the stuff from
>>> nova (like config drive building), you'd still need to handle network
>>> address assignment, image management, etc. etc., and pretty soon you
>>> start having to run a tiny glance and a tiny neutron. The Bifrost way
>>> is the opposite: I just want a tiny Ironic, and _nothing_ else.
>>>
>>
>> Ya, I'm just thinking that at a certain point
> 

You've got two statements in here, which I'm going to reply to separately:

> Oops forgot to fill this out, was just thinking that at a certain point it 
> might
> be easier to figure out how to extract that API (meh, if its public or 
> private)

The nova-core team has repeatedly stated that they do not have plans to support
the nova virt driver API as a stable or externally-consumable python API.
Changing that would affect a lot more than Ironic (eg. defcore). A change like
that is not just about what is easier for developers, but also what is better
for the community.

> and just have someone make an executive decision around ironic being a
> stand-alone thing or not (and a capable stand-alone thing, not a
> sorta-standalone-thing).

We already decided to support Ironic as a stand-alone service. So, could you
clarify what you mean when you call it a "sorta-standalone-thing"? In what ways
do you think it's *not* functional? Do you have specific recommendations on what
we can improve, based on experience using either Ironic or Bifrost?


-Devananda

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] using ironic as a replacement for existing datacenter baremetal provisioning

2016-06-06 Thread Devananda van der Veen

On 06/06/2016 01:44 PM, Kris G. Lindgren wrote:
> Hi ironic folks,
> As I'm trying to explore how GoDaddy can use ironic I've created the following
> in an attempt to document some of my concerns, and I'm wondering if you folks
> could help myself identity ongoing work to solve these (or alternatives?)
> List of concerns with ironic:

Hi Kris,

There is a lot of ongoing work in and around the Ironic project. Thanks for
diving in and for sharing your concerns; you're not alone.

I'll respond to each group of concerns, as some of these appear quite similar to
each other and align with stuff we're already doing. Hopefully I can provide
some helpful background to where the project is at today.

> 
> 1.)Nova <-> ironic interactions are generally seem terrible?

These two projects are coming at the task of managing "compute" with
significantly different situations and we've been working, for the last ~2
years, to build a framework that can provide both virtual and physical resources
through one API. It's not a simple task, and we have a lot more to do.


>   -How to accept raid config and partitioning(?) from end users? Seems to not 
> a
> yet agreed upon method between nova/ironic.

Nova expresses partitioning in a very limited way on the flavor. You get root,
swap, and ephemeral partitions -- and that's it. Ironic honors those today, but
they're pinned on the flavor definition, eg. by the cloud admin (or whoever can
define the flavor.

If your users need more complex partitioning, they could create additional
partitions after the instance is created. This limitation within Ironic exists,
in part, because the projects' goal is to provide hardware through the OpenStack
Compute API -- which doesn't express arbitrary partitionability. (If you're
interested, there is a lengthier and more political discussion about whether the
cloud should support "pets" and whether arbitrary partitioning is needed for
"cattle".)


RAID configuration isn't something that Nova allows their users to choose today
- it doesn't fit in the Nova model of "compute", and there is, to my knowledge,
nothing in the Nova API to allow its input. We've discussed this a little bit,
but so far settled on leaving it up to the cloud admin to set this in Ironic.

There has been discussion with the Cinder community over ways to express volume
spanning and mirroring, but apply it to a machines' local disks, but these
discussions didn't result in any traction.

There's also been discussion of ways we could do ad-hoc changes in RAID level,
based on flavor metadata, during the provisioning process (rather than ahead of
time) but no code has been done for this yet, AFAIK.

So, where does that leave us? With the "explosion of flavors" that you
described. It may not be ideal, but that is the common ground we've reached.

>-How to run multiple conductors/nova-computes?   Right now as far as I can
> tell all of ironic front-end by a single nova-compute, which I will have to
> manage via a cluster technology between two or mode nodes.  Because of this 
> and
> the way host-agregates work I am unable to expose fault domains for ironic
> instances (all of ironic can only be under a single AZ (the az that is 
> assigned
> to the nova-compute node)). Unless I create multiple nova-compute servers and
> manage multiple independent ironic setups.  This makes on-boarding/query of
> hardware capacity painful.

Yep. It's not ideal, and the community is very well aware of, and actively
working on, this limitation. It also may not be as bad as you may think. The
nova-compute process doesn't do very much, and tests show it handling some
thousands of ironic nodes fairly well in parallel. Standard active-passive
management of that process should suffice.

A lot of design work has been done to come up with a joint solution by folks on
both the Ironic and Nova teams.
http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/ironic-multiple-compute-hosts.html

As a side note, it's possible (though not tested, recommended, or well
documented) to run more than one nova-compute. See
https://github.com/openstack/ironic/blob/master/ironic/nova/compute/manager.py

>   - Nova appears to be forcing a we are "compute" as long as "compute" is VMs,
> means that we will have a baremetal flavor explosion (ie the mismatch between
> baremetal and VM).
>   - This is a feeling I got from the ironic-nova cross project meeting in
> Austin.  General exmaple goes back to raid config above. I can configure a
> single piece of hardware many different ways, but to fit into nova's world 
> view
> I need to have many different flavors exposed to end-user.  In this way many
> flavors can map back to a single piece of hardware with just a lsightly
> different configuration applied. So how am I suppose to do a single server 
> with
> 6 drives as either: Raid 1 + Raid 5, Raid 5, Raid 10, Raid 6, or JBOD.  Seems
> like I would need to pre-mark out servers that were going to be a specific 

Re: [openstack-dev] [ironic] [oslo] Template to follow for policy support?

2016-06-03 Thread Devananda van der Veen


On 05/31/2016 04:01 PM, Jay Faulkner wrote:
> Hi all,
> 
> 
> During this cycle, on behalf of OSIC, I'll be working on implementing proper
> oslo.policy support for Ironic. The reasons this is needed probably don't need
> to be explained here, so I won't :).
> 
> 
> I have two requests for the list regarding this though:
> 
> 
> 1) Is there a general guideline to follow when designing policy roles? There
> appears to have been some discussion around this already
> here: https://review.openstack.org/#/c/245629/, but it hasn't moved in over a
> month. I want Ironic's implementation of policy to be as 'standard' as 
> possible;
> but I've had trouble finding any kind of standard.
> 
> 
> 2) A general call for contributors to help make this happen in Ironic. I want,
> in the next week, to finish up the research and start on a spec. Anyone 
> willing
> to help with the design or implementation let me know here or in IRC so we can
> work together.
> 
> 
> Thanks in advance,
> 
> Jay Faulkner
> 

Hi Jay,

Morgan and I sat down earlier this week to brainstorm on adding policy checks to
Ironic's API. Turns out, all the glue for enforcing policy is *already* in place
in the project, but we haven't implemented enforcement within specific API
methods yet. It's not going to be that much work -- I already have a POC up
locally with a few new policy settings.

I'll be happy to work on the spec with you, and hope to have the POC in a
shareable form within a couple days.

--Devananda


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Tooling for recovering nodes

2016-05-31 Thread Devananda van der Veen
On 05/31/2016 01:35 AM, Dmitry Tantsur wrote:
> On 05/31/2016 10:25 AM, Tan, Lin wrote:
>> Hi,
>>
>> Recently, I am working on a spec[1] in order to recover nodes which get stuck
>> in deploying state, so I really expect some feedback from you guys.
>>
>> Ironic nodes can be stuck in
>> deploying/deploywait/cleaning/cleanwait/inspecting/deleting if the node is
>> reserved by a dead conductor (the exclusive lock was not released).
>> Any further requests will be denied by ironic because it thinks the node
>> resource is under control of another conductor.
>>
>> To be more clear, let's narrow the scope and focus on the deploying state
>> first. Currently, people do have several choices to clear the reserved lock:
>> 1. restart the dead conductor
>> 2. wait up to 2 or 3 minutes and _check_deploying_states() will clear the 
>> lock.
>> 3. The operator touches the DB to manually recover these nodes.
>>
>> Option two looks very promising but there are some weakness:
>> 2.1 It won't work if the dead conductor was renamed or deleted.
>> 2.2 It won't work if the node's specific driver was not enabled on live
>> conductors.
>> 2.3 It won't work if the node is in maintenance. (only a corner case).
> 
> We can and should fix all three cases.

2.1 and 2.2 appear to be a bug in the behavior of _check_deploying_status().

The method claims to do exactly what you suggest in 2.1 and 2.2 -- it gathers a
list of Nodes reserved by *any* offline conductor and tries to release the lock.
However, it will always fail to update them, because objects.Node.release()
raises a NodeLocked exception when called on a Node locked by a different 
conductor.

Here's the relevant code path:

ironic/conductor/manager.py:
1259 def _check_deploying_status(self, context):
...
1269 offline_conductors = self.dbapi.get_offline_conductors()
...
1273 node_iter = self.iter_nodes(
1274 fields=['id', 'reservation'],
1275 filters={'provision_state': states.DEPLOYING,
1276  'maintenance': False,
1277  'reserved_by_any_of': offline_conductors})
...
1281 for node_uuid, driver, node_id, conductor_hostname in node_iter:
1285 try:
1286 objects.Node.release(context, conductor_hostname, node_id)
...
1292 except exception.NodeLocked:
1293 LOG.warning(...)
1297 continue


As far as 2.3, I think we should change the query string at the start of this
method so that it includes nodes in maintenance mode. I think it's both safe and
reasonable (and, frankly, what an operator will expect) that a node which is in
maintenance mode, and in DEPLOYING state, whose conductor is offline, should
have that reservation cleared and be set to DEPLOYFAILED state.

--devananda

>>
>> Definitely we should improve the option 2, but there are could be more issues
>> I didn't know in a more complicated environment.
>> So my question is do we still need a new command to recover these node easier
>> without accessing DB, like this PoC [2]:
>>   ironic-noderecover --node_uuids=UUID1,UUID2 
>> --config-file=/etc/ironic/ironic.conf
> 
> I'm -1 to anything silently removing the lock until I see a clear use case 
> which
> is impossible to improve within Ironic itself. Such utility may and will be 
> abused.
> 
> I'm fine with anything that does not forcibly remove the lock by default.
> 
>>
>> Best Regards,
>>
>> Tan
>>
>>
>> [1] https://review.openstack.org/#/c/319812
>> [2] https://review.openstack.org/#/c/311273/
>>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] usage of ironic-lib

2016-05-19 Thread Devananda van der Veen
On 05/16/2016 07:14 AM, Lucas Alvares Gomes wrote:
> On Mon, May 16, 2016 at 2:57 PM, Loo, Ruby  wrote:
>> Hi,
>>
>> A patch to ironic-lib made me wonder about what is our supported usage of
>> ironic-lib. Or even the intent/scope of it. This patch changes a method,
>> ‘bootable’ parameter is removed and ‘boot_flag’ parameter is added [1].
>>
>> If this library/method is used by some out-of-tree thing (or even some
>> in-tree but outside of ironic), this will be a breaking change. If this
>> library is meant to be internal to ironic program itself, and to e.g. only
>> be used by ironic and IPA, then that is different. I was under the
>> impression that it was a library and meant to be used by whatever, no
>> restrictions on what that whatever was. It would be WAY easier if we limited
>> this for usage by only a few specified projects.
>>
>> What do people think?
>>
> 
> I still believe that the ironic-lib project was designed to share code
> between the Ironic projects _only_. Otherwise, if it the code was
> supposed to be shared across multiple projects we should have put it
> in oslo instead.

I agree, and don't see a compelling reason, today, for anyone to do the work to
make ironic-lib into a stable library. So...

I think we should keep ironic-lib where it is (in ironic, not oslo) and keep the
scope we intended (only for use within the Ironic project group [1]).

We should more clearly signal that intent within the library (eg, in the README)
and the project description (eg. on PyPI).

[1]
https://github.com/openstack/governance/blob/master/reference/projects.yaml#L1915


my 2c,
Devananda

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [kolla] [bifrost] bifrost container.

2016-05-09 Thread Devananda van der Veen
On Mon, May 9, 2016 at 11:03 AM, Mooney, Sean K 
wrote:

> Hi
>
>
>
> If we choose to use bifrost to deploy ironic standalone I think combining
> kevins previous
>
> suggestion of modifying the bifrost install playbook with Steve Dake’s
> suggestion of creating a series
>
> of supervisord configs for running each of the service is a reasonable
> approch.
>
>
>
> I am currently look to scope how much effort would be required to split
>  the main task in the bifrost-ironic-install role
>
>
> https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/tasks/main.yml
>
> into 3 files which would be included in the main.yml:
>
> Install_componets.yml (executed when skip_install is not defiend)
>
> Bootstrap_components.yml (executed when skip_bootstrap is not defiend)
>
> Start_components.yml  (executed when skip_start is not defiend)
>
> By default all three would be executed maintain the current behavior of
> bifrost today,.
>

At initial glance, this seems reasonable to me, but the details may get
complicated.

For instance,
- in which playbook do you do things like create system users, db schema,
and necessary directories? These will be identical for every build and
every deployment (of the same version and distro)
- in which playbook do you build or download the IPA ramdisk?
- in which playbook do you set up the networking?



>
> During the kolla build of the biforst image the
> https://github.com/openstack/bifrost/blob/master/playbooks/install.yaml
> would be in
>
> run with skip_bootstrap and skip_start defined as true so only
> Install_componets.yml will be executed by the main task.
>
> This would install all software components of bifrost/ironic without
> preforming configuration or starting the services.
>
>
>
> At deployment time during the bootstrap phase we would spawn an instance
> of the biforst-base container and invoke
>
> https://github.com/openstack/bifrost/blob/master/playbooks/install.yaml
> with skip_install and skip_start defined executing Bootstrap_components.yml
>
>
>
> Bootstrap_components.yml would encapsulate all logic related to creating
> the ironic db(running migration scripts) and generating the configuration
>
> Files for the biforst components.
>
>
>
> Finally in the start phase we have 3 options
>
> a)  Spawn an instance of the bifrost-supervisor container and use
> supervisord to run the bifrost/ironic services (fat container)
>
> b)  Spawn an instance of the bifrost-base container and Invoke
> https://github.com/openstack/bifrost/blob/master/playbooks/install.yaml
> with
> skip_install and skip_bootstrap and allow biforst to star the
> services.(fat container)
>
> c)   Spawn a series of containers each running a single service
> sharing the required volumes to allow them  to communicate (app containers)
>
I don't know enough about supervisord to comment on (a), unfortunately.

(b) looks like the least amount of work, but I'm unclear as to when the
bootstrap phase would have been run.

(c) seems like a lot more work in the long run to maintain the code to
create those volumes, separate per-service containers, and so on.



>
>
> I would welcome any input for the bifrost community on this especially
> related to the decomposition of the main.yml into 3 phases.
>
> Im hoping to do a quick poc this week to see how easy it is to do this
> decomposition.
>
>
>
> I would also like to call out upfront that depending on the scope of this
> item I may have to withdraw from contributing to it.
>
> I work in intel’s network platforms group so enabling baremetal
> installation is somewhat outside the standard
>
> Work our division undertakes.  If we can reuse bifrost to do most of the
> heavy lifting of creating the bifrost container and deploying ironic then
>
> The scope of creating the bifrost container is small enough that I can
> justify spending some of my time working on it. if it requires
>
> Significant changes to bifrost or rework of kolla’s ironic support then I
> will have to step back and focus more on feature that are closer aligned to
>
> Our teams core networking and orchestration focus such as enhancing kolla
> to be able to deploy ovs with dpdk  and/or opendaylight which are
>
> Also items I would like to contribute to this cycle. I don’t want to
> commit to delivering this feature unless I know I will have the time to
> work on
>
> It but am happy to help where I can.
>
>
>
@kevin some replies to your questions inline.
>

>
> Regards
>
> Sean.
>
>
>
>
>
> *From:* Fox, Kevin M [mailto:kevin@pnnl.gov]
> *Sent:* Friday, May 6, 2016 9:17 PM
> *To:* OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> *Subject:* Re: [openstack-dev] [kolla] [bifrost] bifrost container.
>
>
>
> I was under the impression bifrost was 2 things, one, an
> installer/configurator of ironic in a stand alone mode, and two, a
> management tool for getting machines deployed without 

Re: [openstack-dev] [kolla] [bifrost] bifrost container.

2016-05-09 Thread Devananda van der Veen
On Fri, May 6, 2016 at 1:16 PM, Fox, Kevin M  wrote:

> I was under the impression bifrost was 2 things, one, an
> installer/configurator of ironic in a stand alone mode, and two, a
> management tool for getting machines deployed without needing nova using
> ironic.
>

"Bifrost is a set of ansible playbooks..."
- https://github.com/openstack/bifrost

It's not "an installer" + "a management tool" -- Bifrost contains a
playbook for installing Ironic (and a whole lot of service dependencies,
configuration files, etc), and it contains a playbook for deploying a
machine image to some hardware, by leveraging Ironic (and all the other
service dependencies) that were prepared earlier. It also contains a lot of
other playbooks as well, many of which are actually sub-components of these
two high-level steps. In describing Bifrost, we have found it useful to
think of these as separate steps, but not separate things.


> The first use case seems like it should just be handled by enhancing
> kolla's ironic container stuff to directly to handle the use case, doing
> things the kolla way. This seems much cleaner to me. Doing it at runtime
> looses most of the benefits of doing it in a container at all.
>

You definitely shouldn't install Ironic and all of its system service
dependencies (nginx, dnsmasq, tftpd, rabbit, mysql) at runtime, but I also
don't think you should completely split things up into
one-service-per-container.


> The second adds a lot of value I think, and thats what the bifrost
> container should be?
>

The "management tool" you refer to would be more accurately described as
"using the bifrost-dynamic-deploy playbook, which leverages the system
produced by the bifrost-ironic-install playbook, to deploy a machine image
to some hardware, which was previously enrolled in Ironic using the
ironic-enroll-dynamic playbook".


Hope that is helpful,
--Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [kolla] [bifrost] bifrost container.

2016-05-09 Thread Devananda van der Veen
On Fri, May 6, 2016 at 10:56 AM, Steven Dake (stdake) 
wrote:

> Sean,
>
> Thanks for taking this on :)  I didn't know you had such an AR :)
>
> From: "Mooney, Sean K" 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Friday, May 6, 2016 at 10:14 AM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: [openstack-dev] [kolla] [bifrost] bifrost container.
>
> Hi everyone.
>
>
>
> Following up on my AR from the kolla host repository session
>
> https://etherpad.openstack.org/p/kolla-newton-summit-kolla-kolla-host-repo
>
> I started working on creating a kolla bifrost container.
>
>
>
> Are some initial success it have hit a roadblock with the current install
> playbook provided by bifrost.
>
> In particular the install playbook both installs the ironic dependencies
> and configure and runs the services.
>
>
>
>
> What I'd do here is ignore the install playbook and duplicate what it
> installs.  We don't want to install at run time, we want to install at
> build time.  You weren't clear if that is what your doing.
>

That's going to be quite a bit of work. The bifrost-install playbook does a
lot more than just install the ironic services and a few system packages;
it also installs rabbit, mysql, nginx, dnsmasq *and* configures all of
these in a very specific way. Re-inventing all of this is basically
re-inventing Bifrost.



> The reason we would ignore the install playbook is because it runs the
> services.  We need to run the services in a different way.
>

Do you really need to run them in a different way? If it's just a matter of
"use a different init system", I wonder how easily that could be
accomodated within the Bifrost project itself If there's another
reason, please elaborate.



>  This will (as we discussed at ODS) be a fat container on the underlord
> cloud – which I guess is ok.  I'd recommend not using systemd, as that will
> break systemd systems badly.  Instead use a different init system, such as
> supervisord.
>
> The installation of ironic and its dependencies would not be a problem but
> the ansible service module is not cable able of starting the
>
> Infrastructure services (mysql,rabbit …) without a running init system
> which is not present during the docker build.
>
>
>
> When I created a biforst container in the past is spawned a Ubuntu upstart
> container then docker exec into the container and ran
>
> Bifrost install script. This works because the init system is running and
> the service module could test and start the relevant services.
>
>
>
>
>
> This leave me with 3 paths forward.
>
>
>
> 1.   I can continue to try and make the bifrost install script work
> with the kolla build system by using sed to modify the install playbook or
> try start systemd during the docker build.
>
> 2.   I can use the kolla build system to build only part of the image
>
> a.the bifrost-base image would be build with the kolla build
> system without running the bifrost playbook. This
> would allow the existing allow the existing features of the build system
> such as adding headers/footers to be used.
>
> b.  After the base image is built by kolla I can spawn an instance of
> bifrost-base with systemd running
>
> c.   I can then connect to this running container and run the bifrost
> install script unmodified.
>
> d.  Once it is finished I can stop the container and export it to an
> image “bifros-postinstall”.
>
> e.  This can either be used directly (fat container) or as the base
> image for other container that run each of the ironic services (thin
> containers)
>
> 3.   I can  skip the kolla build system entirely and create a
> script/playbook that will build the bifrost container similar to 2.
>
>
> 4.
> Make a supervisord set of init scripts and make the docker file do what it
> was intended – install the files.  This is kind of a mashup of your 1-3
> ideas.  Good thinking :)
>
>
>
>
>
> While option 1 would fully use the kolla build system It is my least
> favorite as it is both hacky and complicated to make work.
>
> Docker really was not designed to run systemd as part of docker build.
>
>
>
> For option 2 and 3 I can provide a single playbook/script that will fully
> automate the build but the real question I have
>
> Is should I use the kolla build system to make the base image or not.
>
>
>
> If anyone else has suggestion on how I can progress  please let me know
> but currently I am leaning towards option 2.
>
>
>
>
> If you have questions about my suggestion to use supervisord, hit me up on
> IRC.  Ideally we would also contribute these init scripts back into bifrost
> code base assuming they want them, which I think they would.  Nobody will
> run systemd in a container, and we all have an interest in seeing BiFrost
> as the standard bare metal deployment model inside 

Re: [openstack-dev] [kolla] [bifrost] bifrost container.

2016-05-09 Thread Devananda van der Veen
Hi folks,

It's a shame that etherpad was down during the ODS kolla/bifrost session. My
recollection is that we agreed *not* to re-invent it Bifrost in containers,
but to just stick with running it in a VM (as it is today) because that is
the quickest path, and allows the Kolla community to leverage the work that
the Ironic/Bifrost community is already doing.

I'd like to respond to several of the emails in this thread and, hopefully,
clear up some misunderstandings. Pardon the coming series of responses, but
it's easier than trying to respond to one email.


--Deva


On Mon, May 9, 2016 at 11:06 AM Mooney, Sean K 
wrote:

> Hi
>
>
>
> If we choose to use bifrost to deploy ironic standalone I think combining
> kevins previous
>
> suggestion of modifying the bifrost install playbook with Steve Dake’s
> suggestion of creating a series
>
> of supervisord configs for running each of the service is a reasonable
> approch.
>
>
>
> I am currently look to scope how much effort would be required to split
>  the main task in the bifrost-ironic-install role
>
>
> https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/tasks/main.yml
>
> into 3 files which would be included in the main.yml:
>
> Install_componets.yml (executed when skip_install is not defiend)
>
> Bootstrap_components.yml (executed when skip_bootstrap is not defiend)
>
> Start_components.yml  (executed when skip_start is not defiend)
>
> By default all three would be executed maintain the current behavior of
> bifrost today,.
>
>
>
> During the kolla build of the biforst image the
> https://github.com/openstack/bifrost/blob/master/playbooks/install.yaml
> would be in
>
> run with skip_bootstrap and skip_start defined as true so only
> Install_componets.yml will be executed by the main task.
>
> This would install all software components of bifrost/ironic without
> preforming configuration or starting the services.
>
>
>
> At deployment time during the bootstrap phase we would spawn an instance
> of the biforst-base container and invoke
>
> https://github.com/openstack/bifrost/blob/master/playbooks/install.yaml
> with skip_install and skip_start defined executing Bootstrap_components.yml
>
>
>
> Bootstrap_components.yml would encapsulate all logic related to creating
> the ironic db(running migration scripts) and generating the configuration
>
> Files for the biforst components.
>
>
>
> Finally in the start phase we have 3 options
>
> a)  Spawn an instance of the bifrost-supervisor container and use
> supervisord to run the bifrost/ironic services (fat container)
>
> b)  Spawn an instance of the bifrost-base container and Invoke
> https://github.com/openstack/bifrost/blob/master/playbooks/install.yaml
> with
> skip_install and skip_bootstrap and allow biforst to star the
> services.(fat container)
>
> c)   Spawn a series of containers each running a single service
> sharing the required volumes to allow them  to communicate (app containers)
>
>
>
>
>
> I would welcome any input for the bifrost community on this especially
> related to the decomposition of the main.yml into 3 phases.
>
> Im hoping to do a quick poc this week to see how easy it is to do this
> decomposition.
>
>
>
> I would also like to call out upfront that depending on the scope of this
> item I may have to withdraw from contributing to it.
>
> I work in intel’s network platforms group so enabling baremetal
> installation is somewhat outside the standard
>
> Work our division undertakes.  If we can reuse bifrost to do most of the
> heavy lifting of creating the bifrost container and deploying ironic then
>
> The scope of creating the bifrost container is small enough that I can
> justify spending some of my time working on it. if it requires
>
> Significant changes to bifrost or rework of kolla’s ironic support then I
> will have to step back and focus more on feature that are closer aligned to
>
> Our teams core networking and orchestration focus such as enhancing kolla
> to be able to deploy ovs with dpdk  and/or opendaylight which are
>
> Also items I would like to contribute to this cycle. I don’t want to
> commit to delivering this feature unless I know I will have the time to
> work on
>
> It but am happy to help where I can.
>
>
>
> @kevin some replies to your questions inline.
>
>
>
> Regards
>
> Sean.
>
>
>
>
>
> *From:* Fox, Kevin M [mailto:kevin@pnnl.gov]
> *Sent:* Friday, May 6, 2016 9:17 PM
>
>
> *To:* OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> *Subject:* Re: [openstack-dev] [kolla] [bifrost] bifrost container.
>
>
>
> I was under the impression bifrost was 2 things, one, an
> installer/configurator of ironic in a stand alone mode, and two, a
> management tool for getting machines deployed without needing nova using
> ironic.
>
> *[Mooney, Sean K] yes this is correct, bifrost does provide both install
> playbooks for deploying ironic in 

Re: [openstack-dev] [ironic] API docs now publishing from our tree

2016-05-09 Thread Devananda van der Veen
Works for me. The api-ref stuff is much easier to maintain, review, and the
resulting docs are infinitely more usable for developers.

I'm already working on getting api-ref up to date; the current docs there
are sorely out of date and wrong. I'd like to land these patches quickly
and iterate on improving them, rather than stall any improvements until
this is perfect.

https://review.openstack.org/312795
https://review.openstack.org/313187
https://review.openstack.org/313708

You can view the results from the build in the new api-ref job, eg. here:
http://docs-draft.openstack.org/08/313708/1/check/gate-ironic-api-ref/5c28efa//api-ref/build/html/

--Deva


On Thu, May 5, 2016 at 6:18 AM Jim Rollenhagen 
wrote:

> On Wed, May 04, 2016 at 05:22:05PM -0400, Ruby Loo wrote:
> > Sweet. Thanks Jim (and everyone else that made this happen).
> >
> > I do want to make sure there is one "source of truth" to the API
> > documentation. We are already generating REST API documentation at
> > http://docs.openstack.org/developer/ironic/webapi/v1.html. The info for
> > this comes from docstrings etc in our code files.
> >
> > To make it easier and so that documentation doesn't get out of sync, I
> > don't really want people to have to remember to update the
> code/docstrings
> > as well as the new files that were added to generate this new api-ref
> > documentation.
>
> I agree. I think the first thing we need to do is get the api-ref tree
> up to date. After that, we can refactor the stuff in doc/source/webapi
> to be pointers to the new thing, and drop the API docs from the
> docstrings.
>
> Does that work for everyone?
>
> // jim
>
> > --ruby
> >
> >
> > On Wed, May 4, 2016 at 5:10 PM, Jim Rollenhagen 
> > wrote:
> >
> > > Hey y'all,
> > >
> > > I did the push this week to move the api-ref into our tree, and it's
> now
> > > publishing. \o/
> > >
> > > The docs are here: http://developer.openstack.org/api-ref/baremetal/
> > >
> > > The source is here:
> > > http://git.openstack.org/cgit/openstack/ironic/tree/api-ref/source
> > >
> > > I know devananda is doing a push to update some things there. If you'd
> > > like to help clean up and improve our docs, please sync with him. They
> > > need a lot of love, so all help is welcome. :)
> > >
> > > // jim
> > >
> > >
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
>
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Summit Core Party after Austin

2016-04-21 Thread Devananda van der Veen
On Thu, Apr 21, 2016 at 9:07 AM, Michael Krotscheck 
wrote:

> Hey everyone-
>
> So, HPE is seeking sponsors to continue the core party. The reasons are
> varied - internal sponsors have moved to other projects, the Big Tent has
> drastically increased the # of cores, and the upcoming summit format change
> creates quite a bit of uncertainty on everything surrounding the summit.
>
> Furthermore, the existence of the Core party has been... contentious. Some
> believe it's exclusionary, others think it's inappropriate, yet others
> think it's a good way to thank those of use who agree to be constantly
> pestered for code reviews.
>
> I'm writing this message for two reasons - mostly, to kick off a
> discussion on whether the party is worthwhile.
>

The rationale for the creation of the first "core party" in Hong Kong was
to facilitate a setting for informal discussions that could bring about a
consensus on potentially-contentious cross-project topics, when there was
no other time or location that brought together all the TC members, PTLs,
and project core reviewers -- many of whom did not yet know each other.
Note that Hong Kong was the first summit where the Technical Committee was
composed of elected members, not just PTLs, and we did not have a separate
day at the design summit to discuss cross-project issues.

The first cross-project design summit tracks were held at the following
summit, in Atlanta, though I recall it lacking the necessary participation
to be successful. Today, we have many more avenues to discuss important
topics affecting all (or more than one) projects. The improved transparency
into those discussions is beneficial to everyone; the perceived exclusivity
of the "core party" is helpful to no one.

So, in summary, I believe this party served a good purpose in Hong Kong and
Atlanta. While it provided some developers with a quiet evening for
discussions to happen in Paris, Vancouver, and Tokyo, we now have other
(better) venues for the discussions this party once facilitated, and it has
outlived its purpose.

For what it's worth, I would be happy to see it replaced with smaller
gatherings around cross-project initiatives. I continue to believe that one
of the most important aspects of our face-to-face gatherings, as a
community, is building the camaraderie and social connections between
developers, both within and across corporate and project boundaries.

-Devananda




> Secondly, to signal to other organizations that this promotional
> opportunity is available.
>
> Personally, I appreciate being thanked for my work. I do not necessarily
> need to be thanked in this fashion, however as the past venues have been
> far more subdued than the Tuesday night events (think cocktail party), it's
> a welcome mid-week respite for this overwhelmed little introvert. I don't
> want to see it go, but I will understand if it does.
>
> Some numbers, for those who like them (Thanks to Mark Atwood for providing
> them):
>
> Total repos: 1010
> Total approvers: 1085
> Repos for official teams: 566
> OpenStack repo approvers: 717
> Repos under release management: 90
> Managed release repo approvers: 281
>
> Michael
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] upgrade support between which versions of ironic?

2016-04-20 Thread Devananda van der Veen
On Wed, Apr 20, 2016 at 5:38 AM, Mathieu Mitchell 
wrote:

>
>
> On 2016-04-19 11:29 PM, Tan, Lin wrote:
>
>> I agree this is reasonable to support all these cases in “cold upgrades”
>> but in supports-rolling-upgrade (live upgrade in another word) case it is
>> different and complicated and not necessary,
>>
>> During rolling upgrade, we will have old/new services co-existed, and we
>> need to make services compatible which need some extra code work and this
>> is the main purpose of spec [1]. And as far as I can see, we are  not
>> allowed to skip over releases when rolling upgrading.  So my point is
>> support name release is enough.
>>
>> 1. Because even if we want to support major number release, admins have
>> to upgrade from 5.0 -> 6.0 then 6.0 -> 7.0 in Ruby’s case of 5.0.0, 5.1.0
>> == Mitaka, 5.2.0, 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton. And we might
>> have a higher release frequency in the future. So it’s too much work for
>> upgrade a service every six months.
>>
>> 2. As we usually rolling upgrade the whole cloud, not for ironic only.
>> For example, other projects will upgrade from Mitaka to Netwon, there is
>> not much sense to upgrade Ironic from 5.0 -> 6.0 only.
>>
>>
> As an operator, I disagree with that statement. We follow different
> upgrade paths for Ironic and Glance for example. My area of concern around
> Ironic is compatibility with Nova and Neutron. If we can prove via CI that
> Nova on an older version still works with Ironic on master and vice versa,
> we will successfully avoid having to do them in a lockstep.


I agree that we need to test this scenario and assert via CI that it is
possible to upgrade Nova and Ironic separately, and as we're adding Neutron
integration, we will need to assert the same thing there. Let's call this a
"cloud rolling upgrade". We'll want to run this test on changes in Nova as
well.

We can test that with the grenade partial job. I do not think we need to
test the upgrade sequence in both directions, though -- as we integrate
with more services, that would explode exponentially. Instead, we should
proscribe an order to the upgrades (and document it clearly) and then test
that ordering. For starters, I would upgrade Ironic first (our API needs to
remain backwards compatible) and then upgrade Nova. That said, I'm not sure
how adding Neutron, or eventually Cinder, will affect the upgrade sequence.

Just to be clear, this doesn't replace the need to assert a "service
rolling upgrade", eg. where different releases of the Ironic API and
Conductor services are run at the same time, but we could do that in a test
environment that is only running Ironic, and wouldn't need to trigger that
test on changes in Nova, for example.

--deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] upgrade support between which versions of ironic?

2016-04-19 Thread Devananda van der Veen
Thanks for starting the thread, Ruby.


We need to first establish a grenade job to test "cold upgrades" and assert
the supports-upgrade tag. I believe Ironic meets all the criteria for that
tag except:
- having a job that tests it (so, you know, it might be broken and I might
be wrong)
- having operator documentation describing the process (it should be here:
http://docs.openstack.org/developer/ironic/deploy/upgrade-guide.html ) but
all we have are release-specific upgrade notes.

I think all of the scenario you outline are valid upgrade paths for an
operator, and we should try to allow all of them to work. However, some of
them can be covered by one test case, and you also missed some things I
think need to be covered. Also, I'm interpreting the word "master" in your
scenarios to indicate the proposed change to our master branch, since we do
pre-merge testing

So, here are the test cases I think we need to cover:

=== run on proposed changes to master ===

F. current master to new SHA

We need to ensure that we can upgrade master to the code being proposed.
You listed this last, but I think it's actually the most important one.

D. last named release to new SHA
E. last numbered release to new SHA

Because we cut new releases from master, this is the basis of testing the
upgrade between sequential (named or numbered) releases before we cut a new
(named or numbered) release, and is our most important test to ensure that
we don't break most operators. Based on the user survey, most operators are
using named releases, so if we are resource constrained, I would prefer to
cover (D) before (E)

=== run on proposed changes to a stable branch ===

A. stable/N-1 -> new SHA -> [ stable/N+1 or current master]

We don't need to test upgrades between two named releases (eg. Liberty ->
Mitaka) every time we land a new patch on the master branch, but we do need
to test any time we land a change on a stable branch. Changes to the most
recent stable branch should be upgrade-tested to current master, whereas
changes to any stable branch prior to that should get tested to the
subsequent sequential release.

Eg, a backport to stable/liberty should trigger an upgrade test for both
(stable/kilo -> newpatch) and (newpatch  -> stable/mitaka), whereas a
backport to stable/mitaka should trigger a test for (stable/liberty ->
newpatch) and (newpatch -> master)


Once we've done that, then yes, I agree we should also work towards
asserting supports-rolling-upgrade. That will require a partial upgrade
job, eg. where we run >1 instance of the API and Conductor services and
upgrade some of them.

Regards,
Devananda



On Tue, Apr 19, 2016 at 12:52 PM, Ruby Loo  wrote:

> Hi,
>
> Currently, ironic doesn't support ("live", "online", "rolling", or
> minimal-downtime) upgrades between named versions of ironic. (Where "named
> version" is the final release or stable release that is associated with a
> development cycle). So for example, Liberty -> Mitaka release.
>
> We've been working towards that, and have a spec [1] and a design session
> [2] at the Austin Summit. Upon reading the spec, I started to wonder --
> what do we mean/want, when we talk about supporting upgrades. Do we want to
> support:
> A. sequential named releases, eg. Mitaka -> Newton
> B. sequential major releases, eg 4.x -> 5.0; 5.0 -> 6.1
> C. sequential minor releases, eg 5.1 -> 5.2
> D. last named release to master
> E. last release (major or minor) to master
> F. some-SHA-more-recent-than-last-named to master. This could be some
> numbered (major or minor) release.
>
> Keep in mind that ironic may release any number of numbered releases
> between two named releases, so eg. 4.3.0, 5.0.0, 5.1.0 == Mitaka, 5.2.0,
> 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton.
>
> Note that there are two governance tags: supports-upgrade[3] and
> supports-rolling-upgrade[4], and I believe our goal is for
> supports-rolling-upgrade.
>
> I think and hope that we can all agree that A. is a must :)
>
> What constitutes a major release versus a minor release may have
> implications in this, but that should be in a separate discussion.
>
> What do people think?
>
> --ruby
>
> [1] https://review.openstack.org/299245
> [2]
> https://www.openstack.org/summit/austin-2016/summit-schedule/events/9267
> [3]
> https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-upgrade.rst
> [4]
> https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-rolling-upgrade.rst
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [ironic] Nominating Julia Kreger for core reviewer

2016-03-24 Thread Devananda van der Veen
Very happy and welcoming +1 from me!

On Thu, Mar 24, 2016 at 1:37 PM, Villalovos, John L <
john.l.villalo...@intel.com> wrote:

> +1 for me. Julia has been an awesome resource and person in the Ironic
> community :)
>
> John
>
> -Original Message-
> From: Jim Rollenhagen [mailto:j...@jimrollenhagen.com]
> Sent: Thursday, March 24, 2016 12:09
> To: openstack-dev@lists.openstack.org
> Subject: [openstack-dev] [ironic] Nominating Julia Kreger for core reviewer
>
> Hey all,
>
> I'm nominating Julia Kreger (TheJulia in IRC) for ironic-core. She runs
> the Bifrost project, gives super valuable reviews, is beginning to lead
> the boot from volume efforts, and is clearly an expert in this space.
>
> All in favor say +1 :)
>
> // jim
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] [tripleo] [stable] Phasing out old Ironic ramdisk and its gate jobs

2016-02-23 Thread Devananda van der Veen
Responding to your points out of order, since that makes more sense to me
right now ...

Since currently DIB claims to be backwards compatible, we just need to
> leave master backwards compatible with Kilo and Liberty Ironic, which
> means not deleting the bash ramdisk element. If Ironic wants to remove
> the bash ramdisk support from master, then it ought to be able to do
> so.


Yes, we'd like to remove support (read: code) from Ironic for the bash
ramdisk. It was deprecated in Liberty, and I'd like to remove it soon (no
later than once Newton opens).



> What if you removed the code from Ironic, but left the element in DIB,
> with a note that it only works with stable/liberty and earlier
> versions of Ironic?
>

Sure, except ...


>
> Could we then:
>
> gate master DIB changes on an Ironic stable/liberty job that uses the
> bash ramdisk - this would catch any regressions in DIB that break the
> bash ramdisk
>

Yup. We could do this.


> gate master DIB changes on an Ironic master job - this is what
> gate-tempest-dsvm-ironic-pxe_ssh-dib is already doing (I think).
>

This, we could not do.

Once we remove the support for the bash ramdisk from ironic/master, we will
not be able to test the "deploy-baremetal" element in dib/master against
ironic/master. We will only be able to test DIB with the "ironic-agent"
element against ironic/master. However, since some users of dib still rely
on the bash ramdisk (eg, because they're using older versions of Ironic) we
understand the need to keep that element supported within dib.


>
> Is that a valid option, and would it remove the desire for a stable
> branch of DIB?


> We currently say that DIB is backwards compatible and doesn't use
> stable branches. If there's a desire to change that, I think that's
> certainly open for discussion. But I don't think we're in a situtation
> where it's preventing us from moving forward with removing the bash
> ramdisk code from Ironic aiui, but I might be misunderstanding. I also
> think that having a stable branch sends the message that master isn't
> backwards compatible. If that's not the message, why do we need the
> stable branch?
>
>
We believe we need the stable branch because we believe we should test
master-master for "ironic-agent" and stable-stable for "deploy-baremetal".

On the other hand, we could test master-stable (dib-ironic) for the
"deploy-baremetal" element. If we did that, then we don't need a stable
branch of dib.

Thoughts?
--devananda
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] A proposal to separate the design summit

2016-02-22 Thread Devananda van der Veen
On Mon, Feb 22, 2016 at 11:51 AM, Walter A. Boring IV  wrote:

> On 02/22/2016 09:45 AM, Thierry Carrez wrote:
>
>> Amrith Kumar wrote:
>>
>>> [...]
>>> As a result of this proposal, there will still be four events each year,
>>> two "OpenStack Summit" events and two "MidCycle" events.
>>>
>>
>> Actually, the OpenStack summit becomes the midcycle event. The new
>> separated contributors-oriented event[tm] happens at the beginning of the
>> new cycle.
>>
>> [...]
>>> Given the number of projects, and leaving aside high bandwidth internet
>>> and remote participation, providing dedicated meeting room for the duration
>>> of the MidCycle event for each project is a considerable undertaking. I
>>> believe therefore that the consequence is that the MidCycle event will end
>>> up being of comparable scale to the current Design Summit or larger, and
>>> will likely need a similar venue.
>>>
>>
>> It still is an order of magnitude smaller than the "OpenStack Summit".
>> Think 600 people instead of 6000. The idea behind co-hosting is to
>> facilitate cross-project interactions. You know where to find people, and
>> you can easily arrange a meeting between two teams for an hour.
>>
>> [...]
>>> At the current OpenStack Summit, there is an opportunity for
>>> contributors, customers and operators to interact, not just in technical
>>> meetings, but also in a social setting. I think this is valuable, even
>>> though there seems to be a number of people who believe that this is not
>>> necessarily the case.
>>>
>>
>> I don't think the proposal removes that opportunity. Contributors /can/
>> still go to OpenStack Summits. They just don't /have to/. I just don't
>> think every contributor needs to be present at every OpenStack Summit,
>> while I'd like to see most of them present at every separated
>> contributors-oriented event[tm].
>>
>
> Yes they can, but if contributors go to the design summit, then they also
> have to get travel budget to go to the new Summit.   So, design summits,
> midcycle meetups, and now the split off marketing summit.   This is making
> it overall more expensive for contributors that meet with customers.
>
>
I do not believe this proposal will increase the travel requirements on
contributors. AIUI, there would still be two Conferences per year (but
without the attached design summit) and still be two design/planning events
(with all the projects together, instead of individual midcycles) per year.
That's it.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] A proposal to separate the design summit

2016-02-22 Thread Devananda van der Veen
On Mon, Feb 22, 2016 at 7:14 AM, Thierry Carrez 
wrote:

> Hi everyone,
>
> TL;DR: Let's split the events, starting after Barcelona
>

Thank you for the excellent write-up, Thierry (and everyone else behind
it)! This sounds great to me.


> Long long version:
>
> In a global and virtual community, high-bandwidth face-to-face time is
> essential. This is why we made the OpenStack Design Summits an integral
> part of our processes from day 0. Those were set at the beginning of each
> of our development cycles to help set goals and organize the work for the
> upcoming 6 months. At the same time and in the same location, a more
> traditional conference was happening, ensuring a lot of interaction between
> the upstream (producers) and downstream (consumers) parts of our community.
>
> This setup, however, has a number of issues. For developers first: the
> "conference" part of the common event got bigger and bigger and it is
> difficult to focus on upstream work (and socially bond with your teammates)
> with so much other commitments and distractions. The result is that our
> design summits are a lot less productive than they used to be, and we
> organize other events ("midcycles") to fill our focus and small-group
> socialization needs. The timing of the event (a couple of weeks after the
> previous cycle release) is also suboptimal: it is way too late to gather
> any sort of requirements and priorities for the already-started new cycle,
> and also too late to do any sort of work planning (the cycle work started
> almost 2 months ago).
>
> But it's not just suboptimal for developers. For contributing companies,
> flying all their developers to expensive cities and conference hotels so
> that they can attend the Design Summit is pretty costly, and the goals of
> the summit location (reaching out to users everywhere) do not necessarily
> align with the goals of the Design Summit location (minimize and balance
> travel costs for existing contributors). For the companies that build
> products and distributions on top of the recent release, the timing of the
> common event is not so great either: it is difficult to show off products
> based on the recent release only two weeks after it's out. The summit date
> is also too early to leverage all the users attending the summit to gather
> feedback on the recent release -- not a lot of people would have tried
> upgrades by summit time. Finally a common event is also suboptimal for the
> events organization : finding venues that can accommodate both events is
> becoming increasingly complicated.
>
> Time is ripe for a change. After Tokyo, we at the Foundation have been
> considering options on how to evolve our events to solve those issues. This
> proposal is the result of this work. There is no perfect solution here (and
> this is still work in progress), but we are confident that this strawman
> solution solves a lot more problems than it creates, and balances the needs
> of the various constituents of our community.
>
> The idea would be to split the events. The first event would be for
> upstream technical contributors to OpenStack. It would be held in a
> simpler, scaled-back setting that would let all OpenStack project teams
> meet in separate rooms, but in a co-located event that would make it easy
> to have ad-hoc cross-project discussions. It would happen closer to the
> centers of mass of contributors, in less-expensive locations.
>
> More importantly, it would be set to happen a couple of weeks /before/ the
> previous cycle release. There is a lot of overlap between cycles. Work on a
> cycle starts at the previous cycle feature freeze, while there is still 5
> weeks to go. Most people switch full-time to the next cycle by RC1.
> Organizing the event just after that time lets us organize the work and
> kickstart the new cycle at the best moment. It also allows us to use our
> time together to quickly address last-minute release-critical issues if
> such issues arise.
>
> The second event would be the main downstream business conference, with
> high-end keynotes, marketplace and breakout sessions. It would be organized
> two or three months /after/ the release, to give time for all downstream
> users to deploy and build products on top of the release. It would be the
> best time to gather feedback on the recent release, and also the best time
> to have strategic discussions: start gathering requirements for the next
> cycle, leveraging the very large cross-section of all our community that
> attends the event.


> To that effect, we'd still hold a number of strategic planning sessions at
> the main event to gather feedback, determine requirements and define
> overall cross-project themes, but the session format would not require all
> project contributors to attend. A subset of contributors who would like to
> participate in this sessions can collect and relay feedback to other team
> members for implementation (similar to the Ops 

Re: [openstack-dev] [all][tc] Stabilization cycles: Elaborating on the idea to move it forward

2016-01-21 Thread Devananda van der Veen
On Wed, Jan 20, 2016 at 9:53 AM, Flavio Percoco  wrote:

> Greetings,
>
> At the Tokyo summit, we discussed OpenStack's development themes in a
> cross-project session. In this session a group of folks started discussing
> what
> topics the overall community could focus on as a shared effort. One of the
> things that was raised during this session is the need of having cycles to
> stabilize projects. This was brought up by Robert Collins again in a
> meeting[0]
> the TC had right after the summit and no much has been done ever since.
>
> Now, "stabilization Cycles" are easy to dream about but really hard to do
> and
> enforce. Nonetheless, they are still worth a try or, at the very least, a
> thought. I'll try to go through some of the issues and benefits a
> stabilization
> cycle could bring but bear in mind that the lists below are not
> exhaustive. In
> fact, I'd love for other folks to chime in and help building a case in
> favor or
> against this.
>
> Negative(?) effects
> ===
>
> - Project won't get new features for a period of time Economic impact on
>  developers(?)
> - It was mentioned that some folks receive bonuses for landed features
> - Economic impact on companies/market because no new features were added
> (?)
> - (?)
>
> Positive effects
> 
>
> - Focus on bug fixing
> - Reduce review backlog
> - Refactor *existing* code/features with cleanups
> - Focus on multi-cycle features (if any) and complete those
> - (?)
>
> A stabilization cycle, as it was also discussed in the aforementioned
> meeting[0], doesn't need to be all or nothing. For instance, it should be
> perfectly fine for a project to say that a project would dedicate 50% of
> the
> cycle to stabilization and the rest to complete some pending features.
> Moreover,
> each project is free to choose when/if a stabilization cycle would be good
> for
> it or not.
>
> For example, the Glance team is currently working on refactoring the image
> import workflow. This is a long term effort that will require at least 2
> cycles
> to be completed. Furthermore, it's very likely these changes will
> introduce bugs
> and that will require further work. If the Glance team would decide (this
> is not
> an actual proposal... yet :) to use Newton as a stabilization cycle, the
> team
> would be able to focus all its forces on fixing those bugs, completing the
> feature and tackling other, long-term, pending issues. In the case of
> Glance,
> this would impact *only glance* and not other projects under the Glance
> team
> umbrella like glanceclient and glance_store. In fact, this would be a
> perfect
> time for the glance team to dedicate time to improving glanceclient and
> catch up
> with the server side latest changes.
>
> So, the above sounds quite vague, still but that's the idea. This email is
> not a
> formal proposal but a starting point to move this conversation forward. Is
> this
> something other teams would be interested in? Is this something some teams
> would
> be entirely against? Why?
>
> From a governance perspective, projects are already empowered to do this
> and
> they don't (and won't) need to be granted permission to have stabilization
> cycles. However, the TC could work on formalizing this process so that
> teams
> have a reference to follow when they want to have one. For example, we
> would
> have to formalize how projects announce they want to have a stabilization
> cycle
> (I believe it should be done before the mid-term of the ongoing cycle).
>
> Thoughts? Feedback?
> Flavio
>
>

Thanks for writing this up, Flavio.

The topic's come up in smaller discussion groups several times over the
last few years, mostly with a nod to "that would be great, except the
corporations won't let it happen".

To everyone who's replied with shock to this thread, the reality is that
nearly all of the developer-hours which fuel OpenStack's progress are
funded directly by corporations, whether big or small. Even those folks who
have worked in open source for a long time, and are working on OpenStack by
choice, are being paid by companies deeply invested in the success of this
project. Some developers are adept at separating the demands of their
employer from the best interests of the community. Some are not. I don't
have hard data, but I suspect that most of the nearly-2000 developers who
have contributed to OpenStack during the Mitaka cycle are working on what
ever they're working on BECAUSE IT MATTERS TO THEIR EMPLOYER.

Every project experiences pressure from companies who are trying to land
very specific features. Why? Because they're all chasing first-leader
advantage in the market. Lots of features are in flight right now, and,
even though it sounds unbelievable, many companies announce those features
in their products BEFORE they actually land upstream. Crazy, right?
Except... IT WORKS. Other companies buy their product because they are
buying a PRODUCT from some company. It happens to 

[openstack-dev] [Ironic] [Nova] continuing the "multiple compute host" discussion

2015-12-10 Thread Devananda van der Veen
All,

I'm going to attempt to summarize a discussion that's been going on for
over a year now, and still remains unresolved.

TLDR;


The main touch-point between Nova and Ironic continues to be a pain point,
and despite many discussions between the teams over the last year resulting
in a solid proposal, we have not been able to get consensus on a solution
that meets everyone's needs.

Some folks are asking us to implement a non-virtualization-centric
scheduler / resource tracker in Nova, or advocating that we wait for the
Nova scheduler to be split-out into a separate project. I do not believe
the Nova team is interested in the former, I do not want to wait for the
latter, and I do not believe that either one will be an adequate solution
-- there are other clients (besides Nova) that need to schedule workloads
on Ironic.

We need to decide on a path of least pain and then proceed. I really want
to get this done in Mitaka.


Long version:
-

During Liberty, Jim and I worked with Jay Pipes and others on the Nova team
to come up with a plan. That plan was proposed in a Nova spec [1] and
approved in October, shortly before the Mitaka summit. It got significant
reviews from the Ironic team, since it is predicated on work being done in
Ironic to expose a new "reservations" API endpoint. The details of that
Ironic change were proposed separately [2] but have deadlocked. Discussions
with some operators at and after the Mitaka summit have highlighted a
problem with this plan.

Actually, more than one, so to better understand the divergent viewpoints
that result in the current deadlock, I drew a diagram [3]. If you haven't
read both the Nova and Ironic specs already, this diagram probably won't
make sense to you. I'll attempt to explain it a bit with more words.


[A]
The Nova team wants to remove the (Host, Node) tuple from all the places
that this exists, and return to scheduling only based on Compute Host. They
also don't want to change any existing scheduler filters (especially not
compute_capabilities_filter) or the filter scheduler class or plugin
mechanisms. And, as far as I understand it, they're not interested in
accepting a filter plugin that calls out to external APIs (eg, Ironic) to
identify a Node and pass that Node's UUID to the Compute Host.  [[ nova
team: please correct me on any point here where I'm wrong, or your
collective views have changed over the last year. ]]

[B]
OpenStack deployers who are using Nova + Ironic rely on a few things:
- compute_capabilities_filter to match node.properties['capabilities']
against flavor extra_specs.
- other downstream nova scheduler filters that do other sorts of hardware
matching
These deployers clearly and rightly do not want us to take away either of
these capabilities, so anything we do needs to be backwards compatible with
any current Nova scheduler plugins -- even downstream ones.

[C] To meet the compatibility requirements of [B] without requiring the
nova-scheduler team to do the work, we would need to forklift some parts of
the nova-scheduler code into Ironic. But I think that's terrible, and I
don't think any OpenStack developers will like it. Furthermore, operators
have already expressed their distase for this because they want to use the
same filters for virtual and baremetal instances but do not want to
duplicate the code (because we all know that's a recipe for drift).

[D]
What ever solution we devise for scheduling bare metal resources in Ironic
needs to perform well at the scale Ironic deployments are aiming for (eg,
thousands of Nodes) without the use of Cells. It also must be integrable
with other software (eg, it should be exposed in our REST API). And it must
allow us to run more than one (active-active) nova-compute process, which
we can't today.


OK. That's a lot of words... bear with me, though, as I'm not done yet...

This drawing [3] is a Venn diagram, but not everything overlaps. The Nova
and Ironic specs [0],[1] meet the needs of the Nova team and the Ironic
team, and will provide a more performant, highly-available solution, that
is easier to use with other schedulers or datacenter-management tools.
However, this solution does not meet the needs of some current OpenStack
Operators because it will not support Nova Scheduler filter plugins. Thus,
in the diagram, [A] and [D] overlap but neither one intersects with [B].


Summary
--

We have proposed a solution that fits ironic's HA model into nova-compute's
failure domain model, but that's only half of the picture -- in so doing,
we assumed that scheduling of bare metal resources was simplistic when, in
fact, it needs to be just as rich as the scheduling of virtual resources.

So, at this point, I think we need to accept that the scheduling of
virtualized and bare metal workloads are two different problem domains that
are equally complex.

Either, we:
* build a separate scheduler process in Ironic, forking the Nova scheduler
as a starting point so 

Re: [openstack-dev] [all] [tc] [ironic] Picking an official name for a subproject (ironic-inspector in this case)

2015-12-04 Thread Devananda van der Veen
On Fri, Dec 4, 2015 at 7:30 AM, Thierry Carrez 
wrote:

> Julien Danjou wrote:
> > On Fri, Dec 04 2015, Dmitry Tantsur wrote:
> >
> >> Specifically, I'm talking about ironic-inspector, which is a auxiliary
> service
> >> under the bare metal program. My first assumption is to prefix with
> ironic's
> >> official name, so it should be something like 'baremetal-XXX' or
> 'baremetal
> >> XXX'. Is it correct? Which separator is preferred?
> >
> > FWIW we have 3 different projects under the telemetry umbrella and they
> > do not share any prefix.
>
> My take is to rename ironic-inspector to clouseau, the ironic inspector
> from the Pink Panther series.
>

This will totally get my vote (if there is a vote).

Between the choices you offered, I'd pick "baremetal introspection" as
that's what we call the process which that service performs.

-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [ironic] Hardware composition

2015-12-01 Thread Devananda van der Veen
On Tue, Dec 1, 2015 at 12:07 PM, Andrew Laski  wrote:

> On 12/01/15 at 03:44pm, Vladyslav Drok wrote:
>
>> Hi list!
>>
>> There is an idea of making use of hardware composition (e.g.
>>
>> http://www.intel.com/content/www/us/en/architecture-and-technology/rack-scale-architecture/intel-rack-scale-architecture-resources.html
>> )
>> to create nodes for ironic.
>>
>> The current proposal is:
>>
>> 1. To create hardware-compositor service under ironic umbrella to manage
>> this composition process. Its initial implementation will support Intel
>> RSA, other technologies may be added in future. At the beginning, it will
>> contain the most basic CRUD logic for composed system.
>>
>> 2. Add logic to nova to compose a node using this new project and register
>> it in ironic if the scheduler is not able to find any ironic node matching
>> the flavor. An alternative (as pointed out by Devananda during yesterday's
>> meeting) could be using it in ironic by claims API when it's implemented (
>> https://review.openstack.org/204641).
>>
>
> I don't think the logic should be added to Nova to create these nodes.
> This hardware-compositor looks like a hypervisor that can manage the
> subdivision of a set of resources and exposes them as a "virtual machine"
> in a sense.  So I would expect that work to happen within the virt driver
> as it does for all of the others.
>
> The key thing to work out is how to expose the resources for the Nova
> scheduler to use.  I may be simplifying the problem but it looks like the
> pooled system could be exposed as a compute host that can be scheduled to,
> and as systems are composed from the pool the consumed resources would be
> decremented the same as they're done today.


I believe you've correctly described the ways such resources might be
modeled and scheduled to -- there is, as I understand it, a non-arbitrarily
subdivisible pool that adheres to some rules. This "compositing" moves
guest isolation deeper into the hardware than virtualization does today.

However, this isn't a virt driver any more than nova-baremetal was a virt
driver; these machines, once "composed", still need all the operational
tooling of Ironic to bring a guest OS up. Also, the interfaces to "compose"
these servers are hardware APIs. The service in question will communicate
with a BMC or similar out-of-band hardware management device -- probably
the same BMC that ironic-conductor will communicate with to boot the
servers.

The more I think about this, the more it sounds like a special type of
chassis manager -- a concept we added to Ironic in the beginning but never
used -- and some changes to the nova.virt.ironic driver to call out to
?ironic?new service? and compose the servers on-the-fly, in response to a
boot request, if there is available capacity.

Also, this is pretty far afield from where we're focusing time right now.
We need to get the Ironic claims API and multiple compute host support done
first.

-deva



>
>
>
>> 3. If implemented in nova, there will be no changes to ironic right now
>> (apart from needing the driver to manage these composed nodes, which is
>> redfish I beleive), but there are cases when it may be useful to call this
>> service from ironic directly, e.g. to free the resources when a node is
>> deleted.
>>
>> Thoughts?
>>
>> Thanks,
>> Vlad
>>
>
> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-12-01 Thread Devananda van der Veen
On Tue, Dec 1, 2015 at 3:22 AM, Steven Hardy <sha...@redhat.com> wrote:

> On Mon, Nov 30, 2015 at 03:35:13PM -0800, Devananda van der Veen wrote:
> >On Mon, Nov 30, 2015 at 3:07 PM, Zane Bitter <zbit...@redhat.com>
> wrote:
> >
> >  On 30/11/15 12:51, Ruby Loo wrote:
> >
> >On 30 November 2015 at 10:19, Derek Higgins <der...@redhat.com
> ><mailto:der...@redhat.com>> wrote:
> >
> >Â  Â  Hi All,
> >
> >Â  Â  Â  Â  Â A few months tripleo switch from its devtest based
> CI to
> >one
> >Â  Â  that was based on instack. Before doing this we anticipated
> >Â  Â  disruption in the ci jobs and removed them from non tripleo
> >projects.
> >
> >Â  Â  Â  Â  Â We'd like to investigate adding it back to heat and
> >ironic as
> >Â  Â  these are the two projects where we find our ci provides the
> >most
> >Â  Â  value. But we can only do this if the results from the job
> are
> >Â  Â  treated as voting.
> >
> >What does this mean? That the tripleo job could vote and do a -1
> and
> >block ironic's gate?
> >
> >Â  Â  Â  Â  Â In the past most of the non tripleo projects tended
> to
> >ignore
> >Â  Â  the results from the tripleo job as it wasn't unusual for
> the
> >job to
> >Â  Â  broken for days at a time. The thing is, ignoring the
> results of
> >the
> >Â  Â  job is the reason (the majority of the time) it was broken
> in
> >the
> >Â  Â  first place.
> >Â  Â  Â  Â  Â To decrease the number of breakages we are now no
> longer
> >Â  Â  running master code for everything (for the non tripleo
> projects
> >we
> >Â  Â  bump the versions we use periodically if they are working).
> I
> >Â  Â  believe with this model the CI jobs we run have become a lot
> >more
> >Â  Â  reliable, there are still breakages but far less frequently.
> >
> >Â  Â  What I proposing is we add at least one of our tripleo jobs
> back
> >to
> >Â  Â  both heat and ironic (and other projects associated with
> them
> >e.g.
> >Â  Â  clients, ironicinspector etc..), tripleo will switch to
> running
> >Â  Â  latest master of those repositories and the cores approving
> on
> >those
> >Â  Â  projects should wait for a passing CI jobs before hitting
> >approve.
> >Â  Â  So how do people feel about doing this? can we give it a
> go? A
> >Â  Â  couple of people have already expressed an interest in doing
> >this
> >Â  Â  but I'd like to make sure were all in agreement before
> switching
> >it on.
> >
> >This seems to indicate that the tripleo jobs are non-voting, or at
> >least
> >won't block the gate -- so I'm fine with adding tripleo jobs to
> >ironic.
> >But if you want cores to wait/make sure they pass, then shouldn't
> they
> >be voting? (Guess I'm a bit confused.)
> >
> >  +1
> >
> >  I don't think it hurts to turn it on, but tbh I'm uncomfortable
> with the
> >  mental overhead of a non-voting job that I have to manually treat
> as a
> >  voting job. If it's stable enough to make it a voting job, I'd
> prefer we
> >  just make it voting. And if it's not then I'd like to see it be made
> >  stable enough to be a voting job and then make it voting.
> >
> >This is roughly where I sit as well -- if it's non-voting, experience
> >tells me that it will largely be ignored, and as such, isn't a good
> use of
> >resources.
>
> I'm sure you can appreciate it's something of a chicken/egg problem though
> - if everyone always ignores non-voting jobs, they never become voting.
>
> That effect is magnified with TripleO though, because it consumes so many
> OpenStack projects, any one of which has the capability to break our CI, so
> in an ideal world we'd have voting feedback on all-the-things, but that's
> not where we are right now due in large-part to the steady stream of
> regressions (from Heat, Ironic and other projects).
>
> >I haven't looked at tripleo or tripleoci in a while, so I wont assume
> that
> >my recollection of the CI jobs bears any resemblance to what exists
> today.
> >Could you explain what are

Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Devananda van der Veen
On Mon, Nov 30, 2015 at 3:07 PM, Zane Bitter  wrote:

> On 30/11/15 12:51, Ruby Loo wrote:
>
>>
>>
>> On 30 November 2015 at 10:19, Derek Higgins > > wrote:
>>
>> Hi All,
>>
>>  A few months tripleo switch from its devtest based CI to one
>> that was based on instack. Before doing this we anticipated
>> disruption in the ci jobs and removed them from non tripleo projects.
>>
>>  We'd like to investigate adding it back to heat and ironic as
>> these are the two projects where we find our ci provides the most
>> value. But we can only do this if the results from the job are
>> treated as voting.
>>
>>
>> What does this mean? That the tripleo job could vote and do a -1 and
>> block ironic's gate?
>>
>>
>>  In the past most of the non tripleo projects tended to ignore
>> the results from the tripleo job as it wasn't unusual for the job to
>> broken for days at a time. The thing is, ignoring the results of the
>> job is the reason (the majority of the time) it was broken in the
>> first place.
>>  To decrease the number of breakages we are now no longer
>> running master code for everything (for the non tripleo projects we
>> bump the versions we use periodically if they are working). I
>> believe with this model the CI jobs we run have become a lot more
>> reliable, there are still breakages but far less frequently.
>>
>> What I proposing is we add at least one of our tripleo jobs back to
>> both heat and ironic (and other projects associated with them e.g.
>> clients, ironicinspector etc..), tripleo will switch to running
>> latest master of those repositories and the cores approving on those
>> projects should wait for a passing CI jobs before hitting approve.
>> So how do people feel about doing this? can we give it a go? A
>> couple of people have already expressed an interest in doing this
>> but I'd like to make sure were all in agreement before switching it
>> on.
>>
>> This seems to indicate that the tripleo jobs are non-voting, or at least
>> won't block the gate -- so I'm fine with adding tripleo jobs to ironic.
>> But if you want cores to wait/make sure they pass, then shouldn't they
>> be voting? (Guess I'm a bit confused.)
>>
>
> +1
>
> I don't think it hurts to turn it on, but tbh I'm uncomfortable with the
> mental overhead of a non-voting job that I have to manually treat as a
> voting job. If it's stable enough to make it a voting job, I'd prefer we
> just make it voting. And if it's not then I'd like to see it be made stable
> enough to be a voting job and then make it voting.


This is roughly where I sit as well -- if it's non-voting, experience tells
me that it will largely be ignored, and as such, isn't a good use of
resources.

I haven't looked at tripleo or tripleoci in a while, so I wont assume that
my recollection of the CI jobs bears any resemblance to what exists today.
Could you explain what areas of ironic (or its subprojects) will be covered
by these tests?  If they are already covered by existing tests, then I
don't see the benefit of adding another job; conversely, if this is testing
areas we don't cover today, then there's probably value in running
tripleoci in a voting fashion for now and then moving that coverage into
ironic's project testing.

-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic][security] what is OK to put in DEBUG logs?

2015-11-18 Thread Devananda van der Veen
On Wed, Nov 18, 2015 at 9:48 AM, Ruby Loo  wrote:

> Hi,
>
> I think we all agree that it isn't OK to log credentials (like passwords)
> in DEBUG logs. However, what about other information that might be
> sensitive? A patch was recently submitted to log (in debug) the SWIFT
> temporary URL [1]. I agree that it would be useful for debugging, but since
> that temporary URL could be used (by someone that has access to the logs
> but no admin access to ironic/glance) eg for fetching private images, is it
> OK?
>
> Even though we say that debug shouldn't be used in production, we can't
> enforce what folks choose to do. And we know of at least one company that
> runs their production environment with the debug setting. Which isn't to
> say we shouldn't put things in debug, but I think it would be useful to
> have some guidelines as to what we can safely expose or not.
>
> I took a quick look at the security web page [2] but nothing jumped out at
> me wrt this issue.
>
> Thoughts?
>
> --ruby
>
> [1] https://review.openstack.org/#/c/243141/
> [2] https://security.openstack.org
>
>
In this context, the URL is a time-limited access code being used in place
of a password or keystone auth token to allow an unprivileged client
temporary access to a specific privileged resource, without granting that
client access to any other resources. In some cases, that resource might be
a public Glance image and so one might say, "oh, it's not _that_
sensitive". However, the same module being affected by [1] is also used by
the iLO driver to upload a temporary image containing sensitive
instance-specific data.

I agree that it's not the same risk as exposing a password, but I still
consider this an access token, and therefore don't think it should be
written to log files, even at DEBUG.

-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] [OSC] Quick poll: OpenStackClient command for provision action

2015-11-17 Thread Devananda van der Veen
Thanks for the list, Sam. I've left some comments inline...

On Tue, Nov 10, 2015 at 4:19 AM, Sam Betts (sambetts) 
wrote:

> So you would end up with a set of commands that look like this:
>
> Openstack baremetal [node/driver/chassis] list
>

will this command support filtering, similar to our CLI ?


> Openstack baremetal port list [—node uuid] <— replicate node-port-list
>

> Openstack baremetal [node/port/driver] show UUID
>

>
Openstack baremetal chassis show [—nodes] UUID  <— replicate
> chassis-node-list
>

this one seems off to me. "show" commands should be displaying the details
of the requested object, not listing a summarized set of objects. Instead
of "chassis show --nodes UUID", I think the command should be "node list
--chassis uuid", which also matches the option to "port list --node uuid".


>
> Openstack baremetal [node/chassis/port] create
> Openstack baremetal [node/chassis/port] update UUID
> Openstack baremetal [node/chassis/port] delete UUID
>

all seem good


>
> Openstack baremetal [node/chassis] provide UUID
> Openstack baremetal [node/chassis] activate UUID
> Openstack baremetal [node/chassis] rebuild UUID
> Openstack baremetal [node/chassis] inspect UUID
> Openstack baremetal [node/chassis] manage UUID
> Openstack baremetal [node/chassis] abort UUID
> Openstack baremetal [node/chassis] boot UUID
> Openstack baremetal [node/chassis] shutdown UUID
> Openstack baremetal [node/chassis] reboot UUID
>

Firstly, I would suggest removing the "chassis" noun here, at least until
we sort out how we want to handle grouping within the service. Right now,
"chassis" resources do not support any verbs - they can only be created and
deleted, and used as a loose logical grouping of nodes.

Secondly, I would change some of these verbs as follows:
  s/activate/deploy/ - because the resulting state transitions will show as
"deploying", etc
  s/boot/poweron/ && s/shutdown/poweroff/ - because that is a more accurate
description of what will be done, and it moves away from the "nova boot"
command, which is completely different.


>
> Openstack baremetal node maintain [—done] UUID
>

I would rename this option as well, but I'm not sure to what. The action
represented by this verb is not a lengthy process (as could be implied by
the verb "maintain", like, "go do some maintenance on it"). This is merely
the changing of a flag within the state of the resource, which instructs
ironic-conductor to leave that node alone without changing any other state,
so perhaps a more appropriate verb would be "ignore" or "exclude"... eg,
"Openstack baremetal node [exclude|include] UUID"



> Openstack baremetal node console [—enable, —disable] UUID <— With no
> parameters this acts like node-get-console, otherwise acts
> like node-set-console-mode
> Openstack baremetal node boot-device [—supported, —PXE, —CDROM, etc] UUID
> <— With no parameters this acts like node-get-boot-device, —supported
> makes it act like node-get-supported-boot-devices, and with a type of boot
> device passed in it’ll act like node-set-boot-device
>
> Openstack baremetal [node/driver] passthru
>

> WDYT? I think I’ve covered most of what exists in the Ironic CLI
> currently.
>
> Sam
>
> From: "Haomeng, Wang" 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Tuesday, 10 November 2015 11:41
>
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [Ironic] [OSC] Quick poll: OpenStackClient
> command for provision action
>
> Hi Sam,
>
> Yes, I understand your format is:
>
> #openstack baremetal  
>
> so these can cover all 'node' operations however if we want to cover
> support port/chassis/driver and more ironic resources, so how about below
> proposal?
>
> #openstack baremetal   
>
> The resource/target can be one item in following list:
>
> node
> port
> chassis
> driver
> ...
>
> Make sense?
>
>
>
>
> On Tue, Nov 10, 2015 at 7:25 PM, Sam Betts (sambetts) 
> wrote:
>
>> Openstack baremetal provision provide or —provide Just doesn’t feel right
>> to me, it feels like I am typing more that I need to and it feels like I’m
>> telling it to do the same action twice.
>>
>> I would much rather see:
>>
>> Openstack baremetal provide UUID
>> Openstack baremetal activate UUID
>> Openstack baremetal delete UUID
>> Openstack baremetal rebuild UUID
>> Openstack baremetal inspect UUID
>> Openstack baremetal manage UUID
>> Openstack baremetal abort UUID
>>
>> And for power:
>>
>> Openstack baremetal boot UUID
>> Openstack beremetal shutdown UUID
>> Openstack baremetal reboot UUID
>>
>> WDYT?
>>
>> Sam
>>
>> From: "Haomeng, Wang" 
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> 
>> Date: Tuesday, 10 November 2015 10:49
>> To: "OpenStack Development Mailing List 

Re: [openstack-dev] [ironic] Nominating two new core reviewers

2015-10-09 Thread Devananda van der Veen
++ on both counts!

On Thu, Oct 8, 2015 at 2:47 PM, Jim Rollenhagen 
wrote:

> Hi all,
>
> I've been thinking a lot about Ironic's core reviewer team and how we might
> make it better.
>
> I'd like to grow the team more through trust and mentoring. We should be
> able to promote someone to core based on a good knowledge of *some* of
> the code base, and trust them not to +2 things they don't know about. I'd
> also like to build a culture of mentoring non-cores on how to review, in
> preparation for adding them to the team. Through these pieces, I'm hoping
> we can have a few rounds of core additions this cycle.
>
> With that said...
>
> I'd like to nominate Vladyslav Drok (vdrok) for the core team. His reviews
> have been super high quality, and the quantity is ever-increasing. He's
> also started helping out with some smaller efforts (full tempest, for
> example), and I'd love to see that continue with larger efforts.
>
> I'd also like to nominate John Villalovos (jlvillal). John has been
> reviewing a ton of code and making a real effort to learn everything,
> and keep track of everything going on in the project.
>
> Ironic cores, please reply with your vote; provided feedback is positive,
> I'd like to make this official next week sometime. Thanks!
>
> // jim
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] backwards compat issue with PXEDeply and AgentDeploy drivers

2015-10-07 Thread Devananda van der Veen
Ramesh,

I thought about your points over night, and then looked at our in-tree
driver code from stable/kilo and asked myself, "what if this driver was out
of tree?" They'd all have broken -- for very similar reasons as what I
encountered with my demo driver.

When we split the boot and deploy interfaces, we kept compatibility only at
the boundary between ConductorManager and the Driver class. That's all we
set out to do, because that's all we defined the interface to be. I can
accept that, but I'd like us to think about whether the driver interface:
a) is merely the interfaces defined in ironic/drivers/base.py, as we
previously defined, or
b) also includes one or more of the hardware-agnostic interface
implementations (eg, PXEBoot, AgentDeploy, AgentRAID, inspector.Inspector)

As recent experience has taught me, these classes provide essential
primitives for building new hardware drivers. If we want to support
development of hardware drivers out of tree, but we don't want to include
(b) in our definition of the API, then we are signalling that such drivers
must be implemented entirely out of tree (iow, they're not allowed to
borrow *any* functionality from ironic/drivers/modules/*).

And if we're signalling that, and someone goes and implements such a driver
and then later wants to propose it upstream -- how will we feel about
accepting a completely alternative implementation of, say, the pxe boot
driver?

Curious what others think...
-deva


On Mon, Oct 5, 2015 at 11:35 PM, Ramakrishnan G <
rameshg87.openst...@gmail.com> wrote:

>
> Well it's nice to fix, but I really don't know if we should be fixing it.
> As discussed in one of the Ironic meetings before, we might need to define
> what is our driver API or SDK or DDK or whatever we choose to call it .
> Please see inline for my thoughts.
>
> On Tue, Oct 6, 2015 at 5:54 AM, Devananda van der Veen <
> devananda@gmail.com> wrote:
>
>> tldr; the boot / deploy interface split we did broke an out of tree
>> driver. I've proposed a patch. We should get a fix into stable/liberty too.
>>
>> Longer version...
>>
>> I was rebasing my AMTTool driver [0] on top of master because the in-tree
>> one still does not work for me, only to discover that my driver suddenly
>> failed to deploy. I have filed this bug
>>   https://bugs.launchpad.net/ironic/+bug/1502980
>> because we broke at least one out of tree driver (mine). I highly suspect
>> we've broken many other out of tree drivers that relied on either the
>> PXEDeploy or AgentDeploy interfaces that were present in Kilo release. Both
>> classes in Liberty are making explicit calls to "task.driver.boot" -- and
>> kilo-era driver classes did not define this interface.
>>
>
>
> I would like to think more about what really our driver API is ? We have a
> couple of well defined interfaces in ironic/drivers/base.py which people
> may follow, implement an out-of-tree driver, make it a stevedore entrypoint
> and get it working with Ironic.
>
> But
>
> 1) Do we promise them that in-tree implementations of these interfaces
> will always exist.  For example in boot/deploy work done in Liberty, we
> removed the class PxeDeploy [1].  It actually got broken down to PXEBoot
> and ISCSIDeploy.  In the first place, do we guarantee that they will exist
> for ever in the same place with the same name ? :)
>
> 2) Do we really promise the in-tree implementations of these interfaces
> will behave the same way ? For example, the broken stuff AgentDeploy which
> is an implementation of our DeployInterface.  Do we guarantee that this
> implementation will always keep doing what ever it was every time code is
> rebased ?
>
> [1]
> https://review.openstack.org/#/c/166513/19/ironic/drivers/modules/pxe.py
>
>
>
>>
>> I worked out a patch for the AgentDeploy driver and have proposed it here:
>>   https://review.openstack.org/#/c/231215/1
>>
>> I'd like to ask folks to review it quickly -- we should fix this ASAP and
>> backport it to stable/liberty before the next release, if possible. We
>> should also make a similar fix for the PXEDeploy class. If anyone gets to
>> this before I do, please reply here and let me know so we don't duplicate
>> effort.
>>
>
>
> This isn't going to be as same as above as there is no longer a PXEDeploy
> class any more.  We might need to create a new class PXEDeploy which
> probably inherits from ISCSIDeploy and has task.driver.boot worked around
> in the same way as the above patch.
>
>
>
>>
>> Also, Jim already spotted something in the review that is a bit
>> concerning. It seems like the IloVirtualMediaAgentVendorInterface class
>> expects the driver it is attached to *not* to

[openstack-dev] [Ironic] backwards compat issue with PXEDeply and AgentDeploy drivers

2015-10-05 Thread Devananda van der Veen
tldr; the boot / deploy interface split we did broke an out of tree driver.
I've proposed a patch. We should get a fix into stable/liberty too.

Longer version...

I was rebasing my AMTTool driver [0] on top of master because the in-tree
one still does not work for me, only to discover that my driver suddenly
failed to deploy. I have filed this bug
  https://bugs.launchpad.net/ironic/+bug/1502980
because we broke at least one out of tree driver (mine). I highly suspect
we've broken many other out of tree drivers that relied on either the
PXEDeploy or AgentDeploy interfaces that were present in Kilo release. Both
classes in Liberty are making explicit calls to "task.driver.boot" -- and
kilo-era driver classes did not define this interface.

I worked out a patch for the AgentDeploy driver and have proposed it here:
  https://review.openstack.org/#/c/231215/1

I'd like to ask folks to review it quickly -- we should fix this ASAP and
backport it to stable/liberty before the next release, if possible. We
should also make a similar fix for the PXEDeploy class. If anyone gets to
this before I do, please reply here and let me know so we don't duplicate
effort.

Also, Jim already spotted something in the review that is a bit concerning.
It seems like the IloVirtualMediaAgentVendorInterface class expects the
driver it is attached to *not* to have a boot interface and *not* to call
boot.clean_up_ramdisk. Conversely, other drivers may be expecting
AgentVendorInterface to call boot.clean_up_ramdisk -- since that was its
default behavior in Kilo. I'm not sure what the right way to fix this is,
but I lean towards updating the in-tree driver so we remain
backwards-compatible for out of tree drivers.


-Devananda

[0] https://github.com/devananda/ironic/tree/new-amt-driver
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [ironic] PTL candidacy

2015-09-16 Thread Devananda van der Veen
Hi all,

( repost from https ://
review.openstack.org
/223753
 )

It's that time again, where encumbent PTLs are supposed to write about what
features or changes they accomplished and what goals they have for the next
cycle.

I'm not going to do that this time.

Even you though you may have read similar things from others (either in
this or in previous cycles) I'm going to reiterate something. Contrary to
being the technical lead, OpenStack requires the PTL to do a whole slew of
less glamorous things (or delegate them to other people).

- launchpad monkey
- midcycle coordinator
- release coordinator
- public speaker
- cross project liaison
- vendor buffer
- cat wrangler

Historically, PTL meant "project technical lead", but as OpenStack grew, we
/ the TC realized that it is more^D^D^D^Ddifferent than that, and so now
the acronym is defined as "project team lead" [0]. And that is much more
representative of the responsibilities a PTL has today. In short, being PTL
and the lead architect for a successful/sizable project at the same time ==
two full time jobs.

Even before I was doing anything internal at HP, it seemed like my upstream
work was never done since I was trying to be both the team and tech leads
for Ironic. That said, it was also extremely rewarding to found this
project and exercise my social and organizational skills in building a
community around it. I could not be more satisfied with the results -- all
of you make Ironic much more awesome than I could have done alone. That's
the point, after all :)

Last election cycle, I stepped down from the TC so that I could have more
time for my roles as tech and team lead, and to focus on some internal work
(yup, still three jobs). That other work, for better or for worse, took a
greater tax on me than I had anticipated, and my activity upstream has
suffered (sorry!). This has created room for many of the other core
developers, who've been around the project almost as long as I have, to
come forward and fill in the gaps I left in the project management. And
that's really awesome. Thank you all.

I am thrilled that more of the project responsibilities are being handled
by Jim, Ruby, Chris, Lucas, and everyone else now. They are all leading
different areas in their own ways. As PTLs, each would bring a different
viewpoint to the project's day-to-day operations, and if they were to run,
I would support all of them (even though we disagree some times). Today,
there are multiple people who could run the project in my stead, and that
makes me very happy.

If elected, I promise to continue enabling the core team to do more without
my direct involvement, to continue leading in the technical vision for the
project, and liaising with vendors and operators to ensure the project
matures in such a way that it meets their needs.

If you believe I've done a great job as PTL and want me to continue doing
what I've been doing, then please re-elect me. (*)

If you'd like to see a change of pace, please don't hesitate to elect
another PTL :)

Thank you,
Devananda

(*) If you think I haven't done a great job as PTL, I invite you to tell me
how you think I could do better. For the sake of the election archives,
please don't reply to this email.

[0]
https://github.com/openstack/governance/commit/319fae1ea13775d16f865f886db0388e42cd0d1b
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Suggestion to split install guide

2015-09-11 Thread Devananda van der Veen
I agree that it's far too long right now and should be split up.

I would suggest splitting the section on standalone usage into its own
guide, since many of the advanced topics can apply to using Ironic
with and without other services. So perhaps more like this:

- Basic Install Guide with OpenStack
- Basic Install Gude for stand-alone usage
- Advanced topics
-- working with UEFI
-- using configdrive instead of cloud-init
-- hardware inspection
-- trusted and secure boot
- Driver references
-- one section for each driver's specific guide



On Fri, Sep 11, 2015 at 7:57 AM, Bruno Cornec  wrote:
> Hello,
>
> Dmitry Tantsur said on Fri, Sep 11, 2015 at 10:56:07AM +0200:
>>
>> Our install guide is huge, and I've just approved even more text for it.
>> WDYT about splitting it into "Basic Install Guide", which will contain bare
>> minimum for running ironic and deploying instances, and "Advanced Install
>> Guide", which will the following things:
>> 1. Using Bare Metal service as a standalone service
>> 2. Enabling the configuration drive (configdrive)
>> 3. Inspection
>> 4. Trusted boot
>> 5. UEFI
>
>
> As a recent reader, I'd like to keep the UEFI part in the main doc as
> more and more server will be UEFI by default. The rest seems good to go
> in a separate one.
>
> Bruno.
> --
> Open Source Profession, Linux Community Lead WW  http://opensource.hp.com
> HP EMEA EG Open Source Technology Strategist http://hpintelco.net
> FLOSS projects: http://mondorescue.org http://project-builder.org
> Musique ancienne? http://www.musique-ancienne.org http://www.medieval.org
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack][keystone][ironic] Use only Keystone v3 API in DevStack

2015-09-11 Thread Devananda van der Veen
We (the Ironic team) have talked a couple times about keystone /v3
support and about improving the granularity of policy support within
Ironic. No one stepped up to work on these specifically, and they
weren't prioritized during Liberty ... but I think everyone agreed
that we should get on with the keystone v3 relatively soon.

If Ironic is the only integrated project that doesn't support v3 yet,
then yea, we should get on that as soon as M opens.

-Devananda

On Fri, Sep 11, 2015 at 9:45 AM, Davanum Srinivas  wrote:
> Hi,
>
> Short story/question:
> Is keystone /v3 support important to the ironic team? For Mitaka i guess?
>
> Long story:
> The previous discussion - guidance from keystone team on magnum
> (http://markmail.org/message/jchf2vj752jdzfet) motivated me to dig into the
> experimental job we have in devstack for full keystone v3 api and ended up
> with this review.
>
> https://review.openstack.org/#/c/221300/
>
> So essentially that rips out v2 keystone pipeline *except* for ironic jobs.
> as ironic has some hard-coded dependencies to keystone /v2 api. I've logged
> a bug here:
> https://bugs.launchpad.net/ironic/+bug/1494776
>
> Note that review above depends on Jamie's tempest patch which had some hard
> coded /v2 dependency as well (https://review.openstack.org/#/c/214987/)
>
> follow up question:
> Does anyone know of anything else that does not work with /v3?
>
> Thanks,
> Dims
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack][keystone][ironic] Use only Keystone v3 API in DevStack

2015-09-11 Thread Devananda van der Veen
This has been informal discussions at various times around how differently
privileged users might use Ironic for different things. It would be great
if our API supported policy settings that corresponded to, let's say, a
junior support engineer's read-only access, or a DC technician's need to
perform maintenance on a server without granting them admin access to the
whole cloud. Things like that... but nothing formal has been written yet.

On Fri, Sep 11, 2015 at 1:01 PM, Dolph Mathews <dolph.math...@gmail.com>
wrote:

>
> On Fri, Sep 11, 2015 at 2:55 PM, Yee, Guang <guang@hpe.com> wrote:
>
>> Can you please elaborate on "granularity of policy support within
>> Ironic."? Is there a blueprint/etherpad we can take a look?
>>
>
> See the lack of granularity expressed by Ironic's current policy file:
>
>
> https://github.com/openstack/ironic/blob/5671e7c2df455f97ef996c47c9c4f461a82e1c38/etc/ironic/policy.json
>
>
>>
>>
>> Guang
>>
>>
>> -Original Message-
>> From: Devananda van der Veen [mailto:devananda@gmail.com]
>> Sent: Friday, September 11, 2015 10:25 AM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [devstack][keystone][ironic] Use only
>> Keystone v3 API in DevStack
>>
>> We (the Ironic team) have talked a couple times about keystone /v3
>> support and about improving the granularity of policy support within
>> Ironic. No one stepped up to work on these specifically, and they weren't
>> prioritized during Liberty ... but I think everyone agreed that we should
>> get on with the keystone v3 relatively soon.
>>
>> If Ironic is the only integrated project that doesn't support v3 yet,
>> then yea, we should get on that as soon as M opens.
>>
>> -Devananda
>>
>> On Fri, Sep 11, 2015 at 9:45 AM, Davanum Srinivas <dava...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > Short story/question:
>> > Is keystone /v3 support important to the ironic team? For Mitaka i
>> guess?
>> >
>> > Long story:
>> > The previous discussion - guidance from keystone team on magnum
>> > (http://markmail.org/message/jchf2vj752jdzfet) motivated me to dig
>> > into the experimental job we have in devstack for full keystone v3 api
>> > and ended up with this review.
>> >
>> > https://review.openstack.org/#/c/221300/
>> >
>> > So essentially that rips out v2 keystone pipeline *except* for ironic
>> jobs.
>> > as ironic has some hard-coded dependencies to keystone /v2 api. I've
>> > logged a bug here:
>> > https://bugs.launchpad.net/ironic/+bug/1494776
>> >
>> > Note that review above depends on Jamie's tempest patch which had some
>> > hard coded /v2 dependency as well
>> > (https://review.openstack.org/#/c/214987/)
>> >
>> > follow up question:
>> > Does anyone know of anything else that does not work with /v3?
>> >
>> > Thanks,
>> > Dims
>> >
>> > --
>> > Davanum Srinivas :: https://twitter.com/dims
>> >
>> > __
>> >  OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] reminder: meeting time change

2015-08-17 Thread Devananda van der Veen
Hi folks,

Just a quick reminder - we're switching back to a consistent meeting time,
at 18:00 UTC Mondays.

That means the next meeting is in two hours.

I should have sent this on Friday, but exhaustion and juggling the
midcycle, and well, I forgot :-/

Thanks,
Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

2015-08-03 Thread Devananda van der Veen
On Mon, Aug 3, 2015 at 8:41 AM Joshua Harlow harlo...@outlook.com wrote:

 Clint Byrum wrote:
  Excerpts from Gorka Eguileor's message of 2015-08-02 15:49:46 -0700:
  On Fri, Jul 31, 2015 at 01:47:22AM -0700, Mike Perez wrote:
  On Mon, Jul 27, 2015 at 12:35 PM, Gorka Eguileorgegui...@redhat.com
 wrote:
  I know we've all been looking at the HA Active-Active problem in
 Cinder
  and trying our best to figure out possible solutions to the different
  issues, and since current plan is going to take a while (because it
  requires that we finish first fixing Cinder-Nova interactions), I've
 been
  looking at alternatives that allow Active-Active configurations
 without
  needing to wait for those changes to take effect.
 
  And I think I have found a possible solution, but since the HA A-A
  problem has a lot of moving parts I ended up upgrading my initial
  Etherpad notes to a post [1].
 
  Even if we decide that this is not the way to go, which we'll probably
  do, I still think that the post brings a little clarity on all the
  moving parts of the problem, even some that are not reflected on our
  Etherpad [2], and it can help us not miss anything when deciding on a
  different solution.
  Based on IRC conversations in the Cinder room and hearing people's
  opinions in the spec reviews, I'm not convinced the complexity that a
  distributed lock manager adds to Cinder for both developers and the
  operators who ultimately are going to have to learn to maintain things
  like Zoo Keeper as a result is worth it.
 
  **Key point**: We're not scaling Cinder itself, it's about scaling to
  avoid build up of operations from the storage backend solutions
  themselves.
 
  Whatever people think ZooKeeper scaling level is going to accomplish
  is not even a question. We don't need it, because Cinder isn't as
  complex as people are making it.
 
  I'd like to think the Cinder team is a great in recognizing potential
  cross project initiatives. Look at what Thang Pham has done with
  Nova's version object solution. He made a generic solution into an
  Oslo solution for all, and Cinder is using it. That was awesome, and
  people really appreciated that there was a focus for other projects to
  get better, not just Cinder.
 
  Have people consider Ironic's hash ring solution? The project Akanda
  is now adopting it [1], and I think it might have potential. I'd
  appreciate it if interested parties could have this evaluated before
  the Cinder midcycle sprint next week, to be ready for discussion.
 
  [1] - https://review.openstack.org/#/c/195366/
 
  -- Mike Perez
  Hi all,
 
  Since my original proposal was more complex that it needed be I have a
  new proposal of a simpler solution, and I describe how we can do it with
  or without a DLM since we don't seem to reach an agreement on that.
 
  The solution description was more rushed than previous one so I may have
  missed some things.
 
  http://gorka.eguileor.com/simpler-road-to-cinder-active-active/
 
 
  I like the idea of keeping it simpler Gorka. :)
 
  Note that this is punting back to use the database for coordination,
  which is what most projects have done thus far, and has a number of
  advantages and disadvantages.
 
  Note that the stale-lock problem was solved in an interesting way in
 Heat:
  each engine process gets an instance-of-engine uuid that adds to the
  topic queues it listens to. If it holds a lock, it records this UUID in
  the owner field. When somebody wants to steal the lock (due to timeout)
  they send to this queue, and if there's no response, the lock is stolen.
 
  Anyway, I think what might make more sense than copying that directly,
  is implementing Use the database and oslo.messaging to build a DLM
  as a tooz backend. This way as the negative aspects of that approach
  impact an operator, they can pick a tooz driver that satisfies their
  needs, or even write one to their specific backend needs.

 Oh jeez, using 'the database and oslo.messaging to build a DLM' scares
 me :-/

 There are already N + 1 DLM like-systems out there (and more every day
 if u consider the list at
 https://raftconsensus.github.io/#implementations) so I'd really rather
 use one that is proven to work by academia vs make a frankenstein one.


Joshua,

As has been said on this thread, some projects (eg, Ironic) are already
using a consistent hash ring backed by a database to meet the requirements
they have. Could those requirements also be met with some other tech? Yes.
Would that provide additional functionality or some other benefits? Maybe.
But that's not what this thread was about.

Distributed hash rings are a well understood technique, as are databases.
There's no need to be insulting by calling
not-your-favorite-technology-of-the-day a frankenstein one.

The topic here, which I've been eagerly following, is whether or not Cinder
needs to use a DLM *at all*. Until that is addressed, discussing specific
DLM or distributed KVS is not necessary.

Thanks,

Re: [openstack-dev] [Ironic] Let's talk about API versions

2015-07-31 Thread Devananda van der Veen
It sounds like we all agree -- the client we ship should default to a
fixed, older version. Anyone who wants newer functionality can pass a newer
version to their client.

Here's the current state of things:

server:
- stable/kilo: 1.6
- current: 1.11

client:
- stable/kilo: 1.6
- latest release (0.7.0): 1.6
- current: 1.9

So -- since we haven't released a client that sends a header  1.6, I
propose that we set the client back to sending the 1.6 header right away.
While having the client default to 1.1 would be ideal, this should still
keep the Jackson the Absent of the world as happy as reasonably possible
moving forward without breaking anyone that is packaging Kilo already.

(yes, this may affect Olivia the Contributor, but that's OK because Olivia
will have read this email :) )

-Deva


On Fri, Jul 31, 2015 at 2:50 PM Jim Rollenhagen j...@jimrollenhagen.com
wrote:

 On Fri, Jul 31, 2015 at 02:37:52PM -0700, Clint Byrum wrote:
  Excerpts from Sean Dague's message of 2015-07-31 04:14:54 -0700:
   On 07/30/2015 04:58 PM, Devananda van der Veen wrote:
   snip
Thoughts?
   
* I'm assuming it is possible to make micro version changes to
 the
1.x API
  as 1.10.1, 1.10.2,etc
   
   
Despite most folks calling this microversions, I have been trying
 to
simply call this API version negotiation.
   
To your question, no -- the implementations by Nova and Ironic, and
 the
proposal that the API-WG has drafted [1], do not actually support
MAJOR.MINOR.PATCH semantics.
   
It has been implemented as a combination of an HTTP request to
http(s)://server URL/major/resource URI plus a
header X-OpenStack-service-API-Version: major.minor.
   
The major version number is duplicated in both the URI and the
 header,
though Ironic will error if they do not match. Also, there is no
 patch
or micro version.
   
So, were we to change the major version in the header, I would
 expect
that we also change it in the URL, which means registering a new
endpoint with Keystone, and, well, all of that.
  
   Right, it's important to realize that the microversion mechanism is not
   semver, intentionally. It's inspired by HTTP content negotiation, as
   Deva said. I wrote up a lot of the rational for the model in Nova here,
   which the Ironic model is based off of -
   https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2/
  
 
  Thanks Sean, this post was exactly what I needed to understand the
  inspiration behind the current situation.
 
   Ironic is a little different. It's entirely an admin API. And most
 users
   are going to only talk to an Ironic that they own the deployment
   schedule on. So the multi cloud that you don't own concern might not be
   there. But, it would also be confusing to all users if Ironic goes down
   a different path with microversions, and still calls it the same thing.
  
 
  I think being single-tenant makes the impact of changes different,
  however the solution can be the same. While tools that use Ironic may
  not be out in the wild as much from an operator perspective, there will
  be plenty of tools built to the Ironic API that will want to be
  distributed to users of various versions of Ironic.
 
  It sounds to me like for Ironic, the same assumption should be made as
  in the outlined Jackson the Absent solution. Assume no version is old
  version, and require specifying the new version to get any new behavior.
 
  What is preventing Ironic from embracing that?

 So, this is actually how the Ironic API behaves. However, it was at some
 point decided that the client should have a more recent default version
 (which is the main topic for this thread).

 I agree with you; I think this is the best route.

 // jim

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] baremetal server boot failing at iSCSI/tgtd

2015-07-30 Thread Devananda van der Veen
Hi Vikas,

It looks like the iscsi deploy process wasn't able to attach to the node.
This could be a network issue.

Also, what kernel and ramdisk are you using for the deploy step? Did you
download one, or build it with disk-image-builder? cirros is only suitable
for the instance image, not the deploy.

Regards,
-Deva
On Jul 29, 2015 5:56 PM, Vikas Choudhary choudharyvika...@gmail.com
wrote:

 Hi,
 I am trying to boot an instance using pxe deployment process.I am using
 cirros-0.3.4-x86_64-uec image to boot the baremetal server VM.ramdisk and
 kernel images are being deployed successfully , but at kernel
 initialization time its getting stuck there.In console logs I can see that
 tgtd daemon is failing.

 CONSOLE LOGS:
 Jul 29 23:17:18 (none) daemon.warn tgtd: tgtd logger started, pid:219
 debug:0^M
 waiting for tgtd socket...found^M
 Jul 29 23:17:19 (none) daemon.err tgtd: iser_ib_init(3351) Failed to
 initialize RDMA; load kernel modules?^M
 Jul 29 23:17:19 (none) daemon.err tgtd: work_timer_start(150) use signal
 based scheduler^M
 Jul 29 23:17:19 (none) daemon.err tgtd: bs_init_signalfd(271) could not
 open backing-store module directory /usr/lib/tgt/backing-store

 ir-cond logs:

 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base [-] vendor_passthru
 failed with method pass_deploy_info
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base Traceback (most
 recent call last):
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base   File
 /opt/stack/ironic/ironic/drivers/base.py, line 614, in passthru_handler
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base return
 func(*args, **kwargs)
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base   File
 /opt/stack/ironic/ironic/conductor/task_manager.py, line 129, in wrapper
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base return f(*args,
 **kwargs)
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base   File
 /opt/stack/ironic/ironic/drivers/modules/iscsi_deploy.py, line 815, in
 pass_deploy_info
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base
 uuid_dict_returned = continue_deploy(task, **kwargs)
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base   File
 /opt/stack/ironic/ironic/drivers/modules/iscsi_deploy.py, line 392, in
 continue_deploy
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base
 _fail_deploy(task, msg)
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base   File
 /opt/stack/ironic/ironic/drivers/modules/iscsi_deploy.py, line 365, in
 _fail_deploy
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base raise exception.
 InstanceDeployFailure(msg)
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base
 InstanceDeployFailure: Deploy failed for instance 
 42c3dea0-1f08-44f6-a38a-e6bd16d1212a.
 Error: Unexpected error while running command.
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base Command: sudo
 ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m discovery -t st -p
 50.0.0.10:3260
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base Exit code: 4
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base Stdout: u''
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base Stderr: u'iscsiadm:
 connect to 50.0.0.10 timed out\niscsiadm: connect to 50.0.0.10 timed
 out\niscsiadm: connect to 50.0.0.10 timed out\niscsiadm: connect to
 50.0.0.10 timed out\niscsiadm: connect to 50.0.0.10 timed out\niscsiadm:
 connect to 50.0.0.10 timed out\niscsiadm: connection login retries
 (reopen_max) 5 exceeded\niscsiadm: Could not perform SendTargets discovery:
 encountered connection failure\n'
 2015-07-30 04:55:47.095 355 ERROR ironic.drivers.base
 2015-07-30 04:55:47.217 355 DEBUG ironic.conductor.task_manager [-]
 Successfully released exclusive lock for calling vendor passthru on node
 ac0a1e31-2545-4d26-988a-56134584721e (lock was held 496.37 sec)
 release_resources /opt/stack/ironic/ironic/conductor/task_manager.py:305

 I would really appreciate any help/discussion/pointers.Please suggest.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Lenovo driver submission requests

2015-07-30 Thread Devananda van der Veen
Yes, yes, and yes to what has been said already. (Thanks Dmitry, Lucas, and
Jim)

-Devananda
On Jul 30, 2015 7:05 AM, Jim Rollenhagen j...@jimrollenhagen.com wrote:

 On Thu, Jul 30, 2015 at 09:27:42AM +0200, Dmitry Tantsur wrote:
  On 07/30/2015 08:59 AM, Kai KH Huang wrote:
  Dear Devananda
  
I'm the development leader of Lenovo Cloud Solution. Lenovo is
  planning to contribute its Ironic driver to the OpenStack community. The
  Ironic driver developers already registered as OpenStack members and
  signed agreement with OpenStack org.
  
Sorry for broadcasting the message, but seems we have issue
  sending mails to GMail account from China. Devananda, do you have
  alternative mail for follow-up communication? Thanks.
 
  Hi!
 
  I'm adding [Ironic] tag to your message, so that our mail filters mark
 it as
  belonging to Ironic.
 
  Of course I can't answer for Devananda, but I think you can go ahead with
  proposing a spec [1] for your driver, see some already approved spec
 (e.g.
  [2] and [3]) for example. Drop by our IRC channel #openstack-ironic on
  Freenode if you have urgent questions to the team, or post here with
  [Ironic] tag.

 Just to add to this, there shouldn't be much reason to email the PTL
 (Devananda) directly about your contributions. This discussion should be
 done in the open, on this mailing list, as that is the OpenStack way[0]. :)

 [0] https://wiki.openstack.org/wiki/Open

 // jim

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Let's talk about API versions

2015-07-30 Thread Devananda van der Veen
On Thu, Jul 30, 2015 at 10:21 AM Clint Byrum cl...@fewbar.com wrote:

 Excerpts from Jim Rollenhagen's message of 2015-07-27 13:35:25 -0700:
  Hi friends.
 
  Ironic implemented API micro versions in Kilo. We originally did this
  to allow for breaking changes in the API while allowing users to opt in
  to the breakage.
 
  Since then, we've had a default version for our client that we bump to
  something sensible with each release. Currently it is at 1.8.
  Negotiation is done with the server to figure out what is supported and
  adjust accordingly.
 
  Now we've landed a patch[0] with a new version (1.11) that is not
  backward compatible. It causes newly added Node objects to begin life in
  the ENROLL state, rather than AVAILABLE. This is a good thing, and
  people should want this! However, it is a breaking change. Automation
  that adds nodes to Ironic will need to do different things after the
  node-create call.
 

 Great discussion. I think through this thread I've gained some insight
 into what is going on, and I think the problem is that minor version
 bumps are not for backward incompatible changes. As a user, I want to
 pin everything and move the pins after I've tested and adapted with new
 versions of things. However, I also don't want to have to micro manage
 this process while I move forward on different schedules with my REST
 clients and the Ironic service.

 So, to be clear, I may have missed what you're trying to do with micro
 versions.

 In my world, a 1.10 - 1.11 would be for adding new methods, new
 arguments, or deprecating (but not removing) an existing piece of the
 API. But changing the semantics of an existing thing is a major version
 bump. So I wonder if the right thing to do is to bump to 2.0, make the
 default in the client 1.10* for now, with deprecation warnings that the
 major version will not be assumed for future client libraries, and move
 on with the understanding that micro versions aren't supposed to break
 users of a particular major version.

 Thoughts?

 * I'm assuming it is possible to make micro version changes to the 1.x API
   as 1.10.1, 1.10.2,etc


Despite most folks calling this microversions, I have been trying to
simply call this API version negotiation.

To your question, no -- the implementations by Nova and Ironic, and the
proposal that the API-WG has drafted [1], do not actually support
MAJOR.MINOR.PATCH semantics.

It has been implemented as a combination of an HTTP request to
http(s)://server URL/major/resource URI plus a header
X-OpenStack-service-API-Version:
major.minor.

The major version number is duplicated in both the URI and the header,
though Ironic will error if they do not match. Also, there is no patch or
micro version.

So, were we to change the major version in the header, I would expect
that we also change it in the URL, which means registering a new endpoint
with Keystone, and, well, all of that.

-D

[1] https://review.openstack.org/#/c/187112/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Let's talk about API versions

2015-07-28 Thread Devananda van der Veen
I'm going to reply to several emails in this thread - but separately,
because they're from separate people and with separate POV, and I think it
will be even harder to discern what I'm saying if I merge the contexts
before replying... so bear with me ...


On Mon, Jul 27, 2015 at 1:36 PM Jim Rollenhagen j...@jimrollenhagen.com
wrote:

 Hi friends.

 Ironic implemented API micro versions in Kilo. We originally did this
 to allow for breaking changes in the API while allowing users to opt in
 to the breakage.

 Since then, we've had a default version for our client that we bump to
 something sensible with each release. Currently it is at 1.8.
 Negotiation is done with the server to figure out what is supported and
 adjust accordingly.

 Now we've landed a patch[0] with a new version (1.11) that is not
 backward compatible.


No, that patch isn't backwards compatible -- ** but it could be **.

That's the biggest reason that I disagree with what's proposed here -- it
would be trivially easy for us to make this change be backwards compatible
by allowing an optional parameter in the POST command that changes the
initial state of a node from AVAILABLE to ENROLL.

Allow users to opt-in to the new behavior *and* discover its presence from
the API version header. There is simply no reason for this to be a breaking
change. I simply do not understand how folks think we can version the API
means we should break users even when we don't have to.

Anyway, I've said that before. There are other points I want to respond to
as well, so I'll carry on...



 It causes newly added Node objects to begin life in
 the ENROLL state, rather than AVAILABLE. This is a good thing, and
 people should want this! However, it is a breaking change. Automation
 that adds nodes to Ironic will need to do different things after the
 node-create call.

 Our API versioning scheme makes this opt-in (by specifying the API
 version). However, some folks have a problem with releasing this change
 as-is. The logic is that we might release a client that defaults to 1.11
 or higher, or the user may request 1.12 later to get a new feature, thus
 breaking their application that enrolls nodes.


The current approach is supposed to allow users to make use of new
features as they become available, while also ensuring they don’t break due
to incompatibilities [0] However, you're advocating against providing that
right now -- if a user wants a feature added in 1.12, they will be forced
to accept the breaking change in 1.11.

While I have proposed some changes to our specification for API versioning
[1], removing this statement of intent is not one of the changes I've
proposed.

[0]
http://specs.openstack.org/openstack/ironic-specs/specs/kilo/api-microversions.html#problem-description

[1]
https://review.openstack.org/#/c/196320/2/specs/devref/api-version-negotiation.rst,cm



 This is clearly backwards. Users should read release notes and be aware
 of what changes between versions in the API. Users need to be aware of
 the fact that our API is versioned, and use that to their advantage.


You're conflating user and operator -- which I understand, because you
are both. But not everyone is, and I'm concerned about the impact on users
who are not operators. I'd say maybe we have none of those, except then I
hear from folks like Clint that they care about this sort of change and how
it affects them.



 It seems to me that the goal of the version negotiation in our client
 has been to pretend that our API versions don't exist, from a user
 perspective. We need to stop doing this and force users to think about
 what they are doing when they interact with our API.


The intent is to allow users' tooling to continue working while we evolve
the REST API *and* to give users the ability to work with both old and new
service installations (eg, because the user doesn't control what version of
server they are using).

Again, that goal is based on the assumption that some users are not also
operators.



 It seems to me we have a few options here:

 1) Default the python client and CLI to the earliest supported version.
 This will never break users by default.

 2) Default the python client and CLI to use the special version
 'latest'. This will always use the latest API version, and always
 break people when a new server version (that is not backwards
 compatible) is deployed.


In an environment where a user is interacting with 1 Ironic service of
different versions, this default would lead to what I'll just call odd
behavior. It ends up being your option 4 implicitly, to avoid frustration.



 3) Do what Nova does[1]. Default CLI to latest and python client to
 earliest. This assumes that CLI is typically used for one-time commands
 (and it isn't a big deal if we break a one-off command once), and the
 python client is used for applications.

 4) Require a version to use the client at all. This would be a one-time
 break with how applications initialize the client 

Re: [openstack-dev] [Ironic] Let's talk about API versions

2015-07-28 Thread Devananda van der Veen
On Tue, Jul 28, 2015 at 1:57 PM Jim Rollenhagen j...@jimrollenhagen.com
wrote:

 On Tue, Jul 28, 2015 at 08:23:34PM +, Devananda van der Veen wrote:
  On Mon, Jul 27, 2015 at 4:58 PM Sean Dague s...@dague.net wrote:
 
   On 07/27/2015 07:10 PM, Jim Rollenhagen wrote:
On Mon, Jul 27, 2015 at 03:28:49PM -0700, Clint Byrum wrote:
Excerpts from Sean Dague's message of 2015-07-27 13:41:20 -0700:
So the CLI should actually break less often, and will expose the
 most
functionality you can get out of your cloud.
   
   
What I find odd about this is that I want the CLI to be a faithful
representation of the API, because many times the CLI is the only
 way I
can access the API for integration purposes.
   
Faithful representation of the API is an interesting way to put
 it; it
is a faithful representation either way. This way is a
 representation
of the newest state of the API. It sounds like you're expecting a
representation of the oldest state, or the state that you're used to
(the hardest to predict).
   
   
So lets say I just want a very simple bash script to do something
 with
nodes:
   
id=$(ironic node-create ...|getid)
while true ; do
  state=$(ironic node-get $id | get_state)
  case $state in
  AVAILABLE)
break;;
  ERROR)
# retry or something
;;
  *)
splode(UNKNOWN STATE)
;;
  esac
done
   
Then the script is going to start exploding with a CLI that shows me
ENROLL state.
   
So I'm not sure having the CLI go faster than the python client is a
great way to avoid breakage. It might be the opposite of that, and
 that
might just be a large burden on users.
   
I tend to think that folks using the CLI for automation should be
pinning the version in that automation. A quick
 IRONIC_API_VERSION=1.8
or whatever would solve this problem. When new versions are
 available,
read the notes for that version and adjust your code as necessary. Or
just leave it at the same version until it breaks. :)
  
   Yeh, it's a cli, not a bash API. Having had to maintain devstack
   exercise code for a while, I'll tell you that the above is just a time
   bomb waiting to go off. None of the CLIs have that kind of contract,
 nor
   should they. Inflicting the bad UX of today on future generations
   because someone thinks, incorrectly, that it's a bash SDK is not a good
   trade off.
  
  
  And people script against that CLI all. the. time.
 
  Is it pretty? No. But is it something that people do? Yep.
 
  Does that mean we should try to care? Yea, I think so, and I think that
  means we should try to avoid breaking it when reasonably possible.
 
 
   If you want to pin things, explicitly, like jroll said, do that. And
   microversions lets you until your clouds uplift past your legacy code.
  
 
  if you want to pin the version != all users must explicitly specify
 the
  version
 
  I believe Jim is advocating for the latter. I'm advocating for the
 former.

 The problem that I'm now seeing is you're advocating not just that
 people should be able to pin a version. You're advocating for:

 1. People don't need to pin the version. Fine.
1.1. Side note: perhaps you're actually advocating for people should
 not need to pin the version? It's starting to sound that way.
 2. The client should always default to the most recent compatible
   version. Fine.
 3. We should never break a user. Fine.
 4. 1-3 must all be true always. Not fine.

 We can't reasonably make progress while having the latest API version be
 compatible with the previous API version. The combination above breaks
 down when we also want to continue to produce software (preferably
 quickly). We need to let one of those things go; they can't all be true
 at the same time.

 I think the best thing to let go is 1 or 2, personally. Where we are
 today is letting 3 go, and that's why we're stuck here.


Mmmm, close. I think I expanded on that in my last email (replying to
Dmitry) ... but... tldr;

1. yes
2. no -- the client should default to the minimum supported version. We got
that wrong previously, and that's what is hurting us now.
3. not quite -- we should be very intentional when ever we're going to
break a user AND we should only do it when there is no other option AND we
must provide a lot of warning and deprecation period.

-Deva



 // jim

 
 
  -Deva

 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi

Re: [openstack-dev] [Ironic] Let's talk about API versions

2015-07-28 Thread Devananda van der Veen
On Mon, Jul 27, 2015 at 4:58 PM Sean Dague s...@dague.net wrote:

 On 07/27/2015 07:10 PM, Jim Rollenhagen wrote:
  On Mon, Jul 27, 2015 at 03:28:49PM -0700, Clint Byrum wrote:
  Excerpts from Sean Dague's message of 2015-07-27 13:41:20 -0700:
  So the CLI should actually break less often, and will expose the most
  functionality you can get out of your cloud.
 
 
  What I find odd about this is that I want the CLI to be a faithful
  representation of the API, because many times the CLI is the only way I
  can access the API for integration purposes.
 
  Faithful representation of the API is an interesting way to put it; it
  is a faithful representation either way. This way is a representation
  of the newest state of the API. It sounds like you're expecting a
  representation of the oldest state, or the state that you're used to
  (the hardest to predict).
 
 
  So lets say I just want a very simple bash script to do something with
  nodes:
 
  id=$(ironic node-create ...|getid)
  while true ; do
state=$(ironic node-get $id | get_state)
case $state in
AVAILABLE)
  break;;
ERROR)
  # retry or something
  ;;
*)
  splode(UNKNOWN STATE)
  ;;
esac
  done
 
  Then the script is going to start exploding with a CLI that shows me
  ENROLL state.
 
  So I'm not sure having the CLI go faster than the python client is a
  great way to avoid breakage. It might be the opposite of that, and that
  might just be a large burden on users.
 
  I tend to think that folks using the CLI for automation should be
  pinning the version in that automation. A quick IRONIC_API_VERSION=1.8
  or whatever would solve this problem. When new versions are available,
  read the notes for that version and adjust your code as necessary. Or
  just leave it at the same version until it breaks. :)

 Yeh, it's a cli, not a bash API. Having had to maintain devstack
 exercise code for a while, I'll tell you that the above is just a time
 bomb waiting to go off. None of the CLIs have that kind of contract, nor
 should they. Inflicting the bad UX of today on future generations
 because someone thinks, incorrectly, that it's a bash SDK is not a good
 trade off.


And people script against that CLI all. the. time.

Is it pretty? No. But is it something that people do? Yep.

Does that mean we should try to care? Yea, I think so, and I think that
means we should try to avoid breaking it when reasonably possible.


 If you want to pin things, explicitly, like jroll said, do that. And
 microversions lets you until your clouds uplift past your legacy code.


if you want to pin the version != all users must explicitly specify the
version

I believe Jim is advocating for the latter. I'm advocating for the former.


-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Let's talk about API versions

2015-07-28 Thread Devananda van der Veen
On Mon, Jul 27, 2015 at 1:41 PM Sean Dague s...@dague.net wrote:

 Consider the CLI an auto negotiating microversion application of the
 python API client. And, realistically, should solve some of the issues
 of people running nova foo and getting cryptic errors from the server
 when they are hitting an old version of Nova that doesn't know how to foo.

 So the CLI should actually break less often, and will expose the most
 functionality you can get out of your cloud.


Not a lot to add here -- except to agree and say that the client breaking
in predictable and helpful ways (like giving a useful error message about a
feature not being available in an older service) is far better for users,
and easier for developers to implement with version headers than it was
before. And that's a win for users, IMO.

-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Let's talk about API versions

2015-07-28 Thread Devananda van der Veen
On Tue, Jul 28, 2015 at 1:01 AM Dmitry Tantsur dtant...@redhat.com wrote:

 On 07/27/2015 10:41 PM, Sean Dague wrote:
  On 07/27/2015 04:35 PM, Jim Rollenhagen wrote:
  Hi friends.
 
  Ironic implemented API micro versions in Kilo. We originally did this
  to allow for breaking changes in the API while allowing users to opt in
  to the breakage.
 
  Since then, we've had a default version for our client that we bump to
  something sensible with each release. Currently it is at 1.8.
  Negotiation is done with the server to figure out what is supported and
  adjust accordingly.
 
  Now we've landed a patch[0] with a new version (1.11) that is not
  backward compatible. It causes newly added Node objects to begin life in
  the ENROLL state, rather than AVAILABLE. This is a good thing, and
  people should want this! However, it is a breaking change. Automation
  that adds nodes to Ironic will need to do different things after the
  node-create call.
 
  Our API versioning scheme makes this opt-in (by specifying the API
  version). However, some folks have a problem with releasing this change
  as-is. The logic is that we might release a client that defaults to 1.11
  or higher, or the user may request 1.12 later to get a new feature, thus
  breaking their application that enrolls nodes.
 
  This is clearly backwards. Users should read release notes and be aware
  of what changes between versions in the API. Users need to be aware of
  the fact that our API is versioned, and use that to their advantage.
 
  It seems to me that the goal of the version negotiation in our client
  has been to pretend that our API versions don't exist, from a user
  perspective. We need to stop doing this and force users to think about
  what they are doing when they interact with our API.
 
  It seems to me we have a few options here:
 
  1) Default the python client and CLI to the earliest supported version.
  This will never break users by default.
 
  2) Default the python client and CLI to use the special version
  'latest'. This will always use the latest API version, and always
  break people when a new server version (that is not backwards
  compatible) is deployed.
 
  3) Do what Nova does[1]. Default CLI to latest and python client to
  earliest. This assumes that CLI is typically used for one-time commands
  (and it isn't a big deal if we break a one-off command once), and the
  python client is used for applications.
 
  Actually what Nova is doing is slight different than this, the CLI will
  default to latest on the server. There will be an extra round trip to
  figure out what that is. And the CLI will (long term) just not present
  commands that aren't available at the server level you are talking to.
 
  Consider the CLI an auto negotiating microversion application of the
  python API client. And, realistically, should solve some of the issues
  of people running nova foo and getting cryptic errors from the server
  when they are hitting an old version of Nova that doesn't know how to
 foo.
 
  So the CLI should actually break less often, and will expose the most
  functionality you can get out of your cloud.

 What I find weird about this and similar approaches is that we treat CLI
 and any different from other ways to use API. And I *suspect* this is
 because we, the developers, use it the most and point newcomers to it.
 And I agree it's troublesome to explain that to use manage provision
 verb you have to provide --ironic-api-version 1.6. Or was 1.6 for
 inspect? Hmm, I really don't remember.

 CLI is not different, CLI is not special and CLI is not an auto
 negotiating microversion application of the python API client. By
 saying any of these we just refusing it eat our dog's food. If we
 consciously believe (I personally don't) that versioning is the right
 thing to do for people using our API - lets stop dodging it ourselves.
 Otherwise it looks like we've invented an architecture that people
 presumably dream of, but we don't know if it's even usable (to say
 nothing about useful). I tried writing versioning-aware code for our
 devstack yesterday, and it wasn't that nice and shiny.

 If our versioning requires negotiations, lets have it on API level, so
 that all users get it. Actually, server has the same level of
 information as the client. Let's have X-OpenStack-XXX-Version:
 negotiate figure out a proper version for us - or refuse to process a
 request if it's not possible. And if we think that versioning as it is
 now is unusable at all, lets rework the whole idea.

 tl;dr my vote if for CLI to strictly follow whatever server API does.
 That is, default to the lowest version, require explicit version
 argument to get new features. (it's up to a follow up discussion what to
 do during the deprecation period).


Hi Dmitry,

I appreciate a good rant as much as anyone -- and I post them, too -- but
aside from the last paragraph, it's hard for me to tell what your intent
here is, so I'm going to take a 

Re: [openstack-dev] [nova] Exposing provider networks in network_data.json

2015-07-20 Thread Devananda van der Veen
On Sat, Jul 18, 2015 at 5:42 AM Sam Stoelinga sammiest...@gmail.com wrote:

 +1 on Kevin Benton's comments.
 Ironic should have integration with switches where the switches are SDN
 compatible. The individual bare metal node should not care which vlan,
 vxlan or other translation is programmed at the switch. The individual bare
 metal nodes just knows I have 2 nics and and these are on Neutron network
 x. The SDN controller is responsible for making sure the baremetal node
 only has access to Neutron Network x through changing the switch
 configuration dynamically.

 Making an individual baremetal have access to several vlans and let the
 baremetal node configure a vlan tag at the baremetal node itself is a big
 security risk and should not be supported.


I was previously of this opinion, and have changed my mind.

While I agree that this is true in a multi-tenant case and requires an
SDN-capable TOR, there are users (namely, openstack-infra) asking us to
support a single tenant with a statically-configured TOR, where cloud-init
is used to pass in the (external, unchangeable) VLAN configuration to the
bare metal instance.

-Deva



 Unless an operator specifically configures a baremetal node to be vlan
 trunk.

 Sam Stoelinga

 On Sat, Jul 18, 2015 at 5:10 AM, Kevin Benton blak...@gmail.com wrote:

  which requires VLAN info to be pushed to the host. I keep hearing bare
 metal will never need to know about VLANs so I want to quash that ASAP.

 That's leaking implementation details though if the bare metal host only
 needs to be on one network. It also creates a security risk if the bare
 metal node is untrusted.

 If the tagging is to make it so it can access multiple networks, then
 that makes sense for now but it should ultimately be replaced by the vlan
 trunk ports extension being worked on this cycle that decouples the
 underlying network transport from what gets tagged to the VM/bare metal.
 On Jul 17, 2015 11:47 AM, Jim Rollenhagen j...@jimrollenhagen.com
 wrote:

 On Fri, Jul 17, 2015 at 10:56:36AM -0600, Kevin Benton wrote:
  Check out my comments on the review. Only Neutron knows whether or not
 an
  instance needs to do manual tagging based on the plugin/driver loaded.
 
  For example, Ironic/bare metal ports can be bound by neutron with a
 correct
  driver so they shouldn't get the VLAN information at the instance
 level in
  those cases. Nova has no way to know whether Neutron is configured
 this way
  so Neutron should have an explicit response in the port binding
 information
  indicating that an instance needs to tag.

 Agree. However, I just want to point out that there are neutron drivers
 that exist today[0] that support bonded NICs with trunked VLANs, which
 requires VLAN info to be pushed to the host. I keep hearing bare metal
 will never need to know about VLANs so I want to quash that ASAP.

 As far as Neutron sending the flag to decide whether the instance should
 tag packets, +1, I think that should work.

 // jim
 
  On Fri, Jul 17, 2015 at 9:51 AM, Jim Rollenhagen 
 j...@jimrollenhagen.com
  wrote:
 
   On Fri, Jul 17, 2015 at 01:06:46PM +0100, John Garbutt wrote:
On 17 July 2015 at 11:23, Sean Dague s...@dague.net wrote:
 On 07/16/2015 06:06 PM, Sean M. Collins wrote:
 On Thu, Jul 16, 2015 at 01:23:29PM PDT, Mathieu Gagné wrote:
 So it looks like there is a missing part in this feature. There
   should
 be a way to hide this information if the instance does not
 require
   to
 configure vlan interfaces to make network functional.

 I just commented on the review, but the provider network API
 extension
 is admin only, most likely for the reasons that I think someone
 has
 already mentioned, that it exposes details of the phyiscal
 network
 layout that should not be exposed to tenants.

 So, clearly, under some circumstances the network operator wants
 to
 expose this information, because there was the request for that
   feature.
 The question in my mind is what circumstances are those, and what
 additional information needs to be provided here.

 There is always a balance between the private cloud case which
 wants to
 enable more self service from users (and where the users are
 often also
 the operators), and the public cloud case where the users are
 outsiders
 and we want to hide as much as possible from them.

 For instance, would an additional attribute on a provider
 network that
 says this is cool to tell people about be an acceptable
 approach? Is
 there some other creative way to tell our infrastructure that
 these
 artifacts are meant to be exposed in this installation?

 Just kicking around ideas, because I know a pile of gate
 hardware for
 everyone to use is at the other side of answers to these
 questions. And
 given that we've been running full capacity for days now,
 keeping this
 ball moving forward would be great.
   
Maybe we just 

Re: [openstack-dev] [Ironic] ENROLL node state is introduced - next steps [ACTION RECOMMENDED]

2015-07-20 Thread Devananda van der Veen
So, as you know, I think that a breaking change like this is really bad for
our users, no matter how loudly we declare it or when we do it. Version
negotiation in the REST API doesn't give us license to do this. I also
think most users should not need to know the version number at all, should
be able to expect consistent behavior from their applications when ever
they upgrade either their server or client, and should be able to *opt in
to* changes in behavior.

An alternative to your proposal would be:
- add optional parameter to node creation REST API that accepts initial
state [ENROLL || AVAILABLE]
- default if unspecified remains AVAILABLE for backwards compatibility
- CLI grows an entirely new command node-enroll, to enable the new
behavior
- [optional] deprecate the node-create command in our client over a few
release cycles


Along with the requisite API version bump to indicate the acceptance of the
new parameter, users will get the following combination of behaviors:
a) old client + new server: user app still works the same
b) new client + old server: user app still works the same
c) new client + new server: user app still works the same
d) new client + new server + user opts in to new behavior: user app must
behave differently

By contrast, your current approach will impact users in case (c) as soon as
we release a client which defaults to any API version = 1.11.

-Deva

On Thu, Jul 16, 2015 at 4:30 AM Dmitry Tantsur dtant...@redhat.com wrote:

 Hi all!

 Today we landed a patch [1] that switches (starting with API version
 1.11) node creation API to default to ENROLL node state instead of
 AVAILABLE. Nothing to worry about right now: we don't default to this
 API version (yet?) in our clients. But read on to figure out how not to
 get broken in the future.

 Nodes in ENROLL state a basically just records in the database. They are
 not used for scheduling, the only way to make them enter the play is via
 manage provision actions. This means, when you switch to API 1.11 for
 node creation, your tooling will probably get broken. There are 2 steps
 to get your tooling prepared for it:

 1. Switch to new version right now with fixing whatever breaks.
 If you're targetting liberty I recommend you start explicitly using 1.11
 API, e.g. for CLI:

   $ ironic --ironic-api-version 1.11 node-create 

 2. Even if you're not doing step 1, you can make your code compatible
 with both pre-1.11 and 1.11 API. Just insert 2 more transitions after
 creating a node - manage and provide. E.g. for CLI:

   $ ironic node-set-provision-state UUID manage
   # wait
   $ ironic node-set-provision-state UUID provide
   # wait

 For Kilo it would simply move node to MANAGEABLE and back to AVAILABLE.

 Important side note: some people don't realize that ALL provision state
 transitions are asynchronous. And all of them can fail! Even if manage
 action was seemingly instant and synchronous before 1.11, it was not.
 Now with 1.11 API in place manage action may take substantial time and
 may fail. Make sure your tooling account for it.

 Now it's up to the ironic team to decide [2] whether and when we're
 bumping ironicclient default API version to something above 1.11. Opinions?

 [1] https://review.openstack.org/#/c/194722/
 [2] https://review.openstack.org/#/c/196320/

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] No meeting tonight?

2015-07-20 Thread Devananda van der Veen
Hi all,

My apologies, but I won't be able to attend the IRC meeting tonight.

The agenda is very light (basically non-existent) so I suggest that we just
cancel it and meet next Monday.

Also, I haven't seen any response to my email (*) from one week ago about
the lack of attendance to this (late night for the US) meeting time from
the very folks who this meeting is supposed to serve.

Thanks,
Devananda

(*) http://lists.openstack.org/pipermail/openstack-dev/2015-July/069363.html
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] how about those alternating meeting times?

2015-07-13 Thread Devananda van der Veen
This is a much-overdue follow up to this poll, which got 17 responses.

TL;DR:

The poll indicated that most responders did not personally find the 0500
meeting helpful. Reviewing the meeting logs for the last six months shows
significantly lower core attendance in those meetings. Informal feedback
confirms both of these conclusions.

As much as I want to be inclusive of contributors for whom the 1700GMT
Monday meetings do not work, I feel that the 0500GMT Tuesday meetings have
not been effective and would like to propose that we change back to a
single meeting time.


Longer summary:

In that poll, I asked four questions to gauge both individual attendance
and benefit, as well as perceived benefit to others. I think the results
are quite interesting. (Note that every question had a free form choice as
well, which I'm not tallying here)

Were you able to attend the meetings?
+4 Yes, I attended most or all of the metings
-10 No. Because of the timing, I was only able to attend half of the
meetings.

Were the alternating times helpful to you personally?
+5 Yes. I attended both Monday and Tuesday meetings and found this split
helpful.
-10 No. I was unable to attend half of the meetings and found some or all
of them to be non-productive.

Do you believe that alternating meeting times was helpful to the project
overall, regardless of your own availability?
+11 Yes
- 4 No

Do you think we should continue to split meeting times?
+10 Yes
-3 No

In the freeform responses, I got suggestions to split out subteams (we've
done this for the neutron integration work), as well as a few comments that
DST impact was significant and preventing attendance to half the meetings,
and suggestions to move the meeting a few hours this way or that way.

Lately, I have missed several of the 0500GMT meetings due to tiredness,
travel, or other things. I notice that very few of the core team are there,
too (many thanks to Jim for consistently being there and leading it when
I'm not). While I think it is important to have a meeting time that serves
Haomeng and Ramki and Michael Davies (and everyone else in APAC), these
meetings do not consistently have a majority of cores present and therefor
can't reasonably make any decisions. And on the flip side, these impact the
1700GMT meetings in that those have lost a lot of momentum, seemingly
because the US/EU regulars miss every other week.

So, though it saddens me, I would like to propose that we change back to a
single meeting time effective the week following our midcycle (Aug 10th).
That will give folks two meetings in each timeslot between now and then to
become aware of, and raise any issues with, this change.


Regards,
Devananda


On Mon, May 11, 2015 at 6:14 PM Michael Davies mich...@the-davies.net
wrote:

 On Tue, May 12, 2015 at 9:08 AM, Ruby Loo rlooya...@gmail.com wrote:

 Maybe the question is better posed to those folks -- was it useful or
 not? And if not, why? Because the date/time still didn't work, or because
 not enough (or the right persons) weren't there so their issues of interest
 weren't discussed, or they wouldn't have attended anyway, or ? And if it
 was useful, for how many was it useful? (Devananda's poll will capture some
 of that info.)


 I found it useful - it's nice to be awake at meeting time :)

 There's certainly a subset of the team that I never overlap with now,
 which is a shame, but timezones present challenges for a geographically
 dispersed team.

 Previously the meeting was at 4:30am (or 5:30am depending upon daylight
 savings), which was quite hard, but I did make it most weeks.  The new
 timeslot of 2:30am/pm (3:30am/pm) is certainly only achievable for me every
 other week (no surprises for guessing which one :)

 I think it's great that we try and accommodate contributors from all
 around the globe!

 Michael...
 --
 Michael Davies   mich...@the-davies.net
 Rackspace Cloud Builders Australia
  __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] Reminder - midcycle Sprint in Seattle Aug 12 - 14

2015-07-13 Thread Devananda van der Veen
Hi all!

Just dropping a reminder out here -- our midcycle is one month away! We're
starting to jot down informal notes and plans on an etherpad here:

https://etherpad.openstack.org/p/ironic-liberty-midcycle

Please continue to use that to coordinate all the things. If you plan to
attend, please make sure you're listed there and then go purchase a free
ticket (one per person) on eventbrite so that I can coordinate with site
facilities and send you any last-minute information about the event.

https://www.eventbrite.com/e/openstack-ironic-sprint-august-2015-tickets-17533862254

Looking forward to seeing ya'll soon!

-Devananda
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Mid-Cycle Sprint

2015-06-25 Thread Devananda van der Veen
Event signup page is now live -- please RSVP!

*https://www.eventbrite.com/e/openstack-ironic-sprint-august-2015-tickets-17533862254
https://www.eventbrite.com/e/openstack-ironic-sprint-august-2015-tickets-17533862254*

-Devananda

On Thu, Jun 25, 2015 at 8:46 AM Stafford, John Richard john.staff...@hp.com
wrote:

  Hello Fellow Ironic-ers,



 I am confirming the Mid-Cycle Sprint, hosted in the HP Seattle, WA [USA]
 Office on Aug 12-14, 2015. We look forward to seeing everyone there!



 *Cheers!*

 John Stafford

 Engineering Manager |HP Helion Openstack | Openstack-Ironic

 E: john.staff...@hp.com | V: 360.212.9720 |IRC (Freenode): BadCub



 [image: cid:7AFA25C2-9877-4224-B65B-284C7AC54155@dlink.com]



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api][nova][ironic] Microversion API HTTP header

2015-06-25 Thread Devananda van der Veen
Sean's point and Dmitri's are similar.

There are APIs for projects which do not have official team or program
names. And some teams may produce more than one forward-facing service.
Naming the API based in the team name doesn't make sense.

My previous point is that restricting the API name to the team/program name
will prevent any competition among projects. It'll be impossibly confusing
to users if more than one monitoring project exists, they all have
different API, but each claim to be the one true OpenStack-Monitoring-API

-Deva
On Jun 25, 2015 9:37 AM, Sean Dague s...@dague.net wrote:

 On 06/25/2015 12:04 PM, Anne Gentle wrote:
 
 
  On Thu, Jun 25, 2015 at 10:55 AM, Dmitry Tantsur dtant...@redhat.com
  mailto:dtant...@redhat.com wrote:
 snip
 
 
  I'm not sure where the assumption comes from that people will know
  compute better than nova.
 
 
  I have been supporting developer end users on the Rackspace cloud for
  nearly four years now. I gave a talk in Paris at the Summit about
  supporting developers. Developers using cloud resources expect to use
  computing power or storage capacity to accomplish a broader task. Nova
  and swift have nothing to do with their expectations.

 That's good feedback, and clearly moves the needle in my head.

 It also does open up a question about the big tent nature, because it's
 unclear what projects that do not yet have a generic name allocated to
 them would use.

 -Sean

 --
 Sean Dague
 http://dague.net

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Proposal to add a new repository

2015-06-22 Thread Devananda van der Veen
I'm
On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge tr...@redhat.com wrote:



 On 06/22/2015 10:40 AM, Dmitry Tantsur wrote:
  On 06/22/2015 04:19 PM, Devananda van der Veen wrote:
  Hi John,
 
  Thanks for the excellent summary! I found it very helpful to get caught
  up. I'd like to make sure I understand the direction ahc is going. A
  couple questions...

 Thanks for your interest.

 
  Let me add my $0.5 :)
 
 
  I see that ahc is storing its information in swift. That's clever, but
  if Ironic provided a blob store for each node, would that be better?

 If the blob is large enough, this would be better. Originally we stored
 the data in the extra column of the Ironic db, but that proved disastrous:

 https://bugs.launchpad.net/ironic-inspector/+bug/1461252



 
  We discussed adding a search API to Ironic at the Vancouver summit,
  though no work has been done on that yet, afaik. If ahc is going to grow
  a REST API for searching for nodes based on specific criteria that it
  discovered, could/should we combine these within Ironic's API?
 
  I think John meant having API to replace scripts, so I guess search
  won't help. When we're talking about advanced matching, we're talking
  about the following:
  1. We have a ramdisk tool (based on [8]) to get some insane of facts
  from withing the ramdisk (say, 1000 of them)
  2. We have an Inspector plugin to put them all in Swift (or Ironic blob
  storage as above)
  3. We have config files (aka rules) written in special JSON-alike DSL to
  do matching (one of the weak points is that these are files - I'd like
  API endpoint to accept these rules instead).
  4. We have a script to run this DSL and get some output (match/not match
  + some matched variables - similar to what regexps do).
  As I understood it John want the latter to become an API endpoint,
  accepting rules (and maybe node UUIDs) and outputting some result.
 
  Not sure about benchmarking here, but again, it's probably an API
  endpoint that accepts some minimal expectations, and puts failed nodes
  to maintenance mode, if they fail to comply (again, that's how I
  understood it).
 
  It's not hard to make these API endpoints part of Inspector, but it's
  somewhat undesirable to have them optional...
 
 
   From a service coupling perspective, I like the approach that ahc is
  optional, and also that Ironic-inspector is optional, because this keeps
  the simple use-case for Ironic, well, simple! That said, this seems more
  like a configuration setting (should inspector do extra things?) than an
  entirely separate service, and separating them might be unnecessarily
  complicated.
 
  We keep thinking about it as well. After all, right now it's just a
  couple of utilities. There are 2 more concerns that initially made me
  pull out this code:
  1. ahc-tools currently depends on the library [8], which I wish would be
  developed much more openly


 2. it's cool that inspector is pluggable, but it has its cost: there's a
  poor feedback loop from inspector processing plugins to a user - like
  with all highly asynchronous code
  3. it's also not possible (at least for now) to request a set of
  processing plugins when starting introspection via inspector.
 
  We solved the latter 2 problems by moving code to scripts. So now
  Inspector only puts some data to Swift, and scripts can do everything
 else.
 
  So now we've left with
  1. dependency on hardware library
  2. not very stable interface, much less stable than one of Inspector
 
  We still wonder how to solve these 2 without creating one more
  repository. Any ideas are welcome :)


Oh - good point. There's some neat looking functionality in
enovance/hardware repository, but yea, until this is not a
single-vendor-controlled repository, I think you made the right call.

How would folks feel about moving that into openstack/ ?


 It is a goal of mine to solve issue 1 incrementally over time. Either by
 improving the library (both in function and in openness), or by slowly
 moving the implementation. That does not seem impossible to do within
 the inspector tree.


Or that :) Either way, I agree with the direction -- moving the hardware
inspection functionality into a common repository is good. If it makes
sense to have two repositories (one for inspector, one for a hardware utils
library) that's just fine with me.


However, issue 2 is a fact. We currently have scripts, and we want to
 have a REST API. I do not see a transition between the two that does not
 involve a large amount of churn.


From a quick read of the code, it looks like ahc-tools merely analyzes data
already gathered by inspector  enovance/hardware, providing some more
advanced search/filtering/error-checking capabilities on the client side.
If that read is correct, then I would like to align that work with my
interest in developing a query-API for Ironic. It might take some time to
do that, and so having a repo for client-side scripts is fine for now.

(If that read

Re: [openstack-dev] [Ironic] Proposal to add a new repository

2015-06-22 Thread Devananda van der Veen
Oh - one more thing. If ahc-tools depends on data gathered by
enovance/hardware, then I'm not sure it makes sense to import one to
openstack/ without the other.

-Deva

On Mon, Jun 22, 2015 at 5:08 PM Devananda van der Veen 
devananda@gmail.com wrote:

 I'm
 On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge tr...@redhat.com wrote:



 On 06/22/2015 10:40 AM, Dmitry Tantsur wrote:
  On 06/22/2015 04:19 PM, Devananda van der Veen wrote:
  Hi John,
 
  Thanks for the excellent summary! I found it very helpful to get caught
  up. I'd like to make sure I understand the direction ahc is going. A
  couple questions...

 Thanks for your interest.

 
  Let me add my $0.5 :)
 
 
  I see that ahc is storing its information in swift. That's clever, but
  if Ironic provided a blob store for each node, would that be better?

 If the blob is large enough, this would be better. Originally we stored
 the data in the extra column of the Ironic db, but that proved disastrous:

 https://bugs.launchpad.net/ironic-inspector/+bug/1461252



 
  We discussed adding a search API to Ironic at the Vancouver summit,
  though no work has been done on that yet, afaik. If ahc is going to
 grow
  a REST API for searching for nodes based on specific criteria that it
  discovered, could/should we combine these within Ironic's API?
 
  I think John meant having API to replace scripts, so I guess search
  won't help. When we're talking about advanced matching, we're talking
  about the following:
  1. We have a ramdisk tool (based on [8]) to get some insane of facts
  from withing the ramdisk (say, 1000 of them)
  2. We have an Inspector plugin to put them all in Swift (or Ironic blob
  storage as above)
  3. We have config files (aka rules) written in special JSON-alike DSL to
  do matching (one of the weak points is that these are files - I'd like
  API endpoint to accept these rules instead).
  4. We have a script to run this DSL and get some output (match/not match
  + some matched variables - similar to what regexps do).
  As I understood it John want the latter to become an API endpoint,
  accepting rules (and maybe node UUIDs) and outputting some result.
 
  Not sure about benchmarking here, but again, it's probably an API
  endpoint that accepts some minimal expectations, and puts failed nodes
  to maintenance mode, if they fail to comply (again, that's how I
  understood it).
 
  It's not hard to make these API endpoints part of Inspector, but it's
  somewhat undesirable to have them optional...
 
 
   From a service coupling perspective, I like the approach that ahc is
  optional, and also that Ironic-inspector is optional, because this
 keeps
  the simple use-case for Ironic, well, simple! That said, this seems
 more
  like a configuration setting (should inspector do extra things?) than
 an
  entirely separate service, and separating them might be unnecessarily
  complicated.
 
  We keep thinking about it as well. After all, right now it's just a
  couple of utilities. There are 2 more concerns that initially made me
  pull out this code:
  1. ahc-tools currently depends on the library [8], which I wish would be
  developed much more openly


  2. it's cool that inspector is pluggable, but it has its cost: there's a
  poor feedback loop from inspector processing plugins to a user - like
  with all highly asynchronous code
  3. it's also not possible (at least for now) to request a set of
  processing plugins when starting introspection via inspector.
 
  We solved the latter 2 problems by moving code to scripts. So now
  Inspector only puts some data to Swift, and scripts can do everything
 else.
 
  So now we've left with
  1. dependency on hardware library
  2. not very stable interface, much less stable than one of Inspector
 
  We still wonder how to solve these 2 without creating one more
  repository. Any ideas are welcome :)


 Oh - good point. There's some neat looking functionality in
 enovance/hardware repository, but yea, until this is not a
 single-vendor-controlled repository, I think you made the right call.

 How would folks feel about moving that into openstack/ ?


 It is a goal of mine to solve issue 1 incrementally over time. Either by
 improving the library (both in function and in openness), or by slowly
 moving the implementation. That does not seem impossible to do within
 the inspector tree.


 Or that :) Either way, I agree with the direction -- moving the hardware
 inspection functionality into a common repository is good. If it makes
 sense to have two repositories (one for inspector, one for a hardware utils
 library) that's just fine with me.


 However, issue 2 is a fact. We currently have scripts, and we want to
 have a REST API. I do not see a transition between the two that does not
 involve a large amount of churn.


 From a quick read of the code, it looks like ahc-tools merely analyzes
 data already gathered by inspector  enovance/hardware, providing some more
 advanced search/filtering

Re: [openstack-dev] [Ironic] Proposal to add a new repository

2015-06-22 Thread Devananda van der Veen
Hi John,

Thanks for the excellent summary! I found it very helpful to get caught up.
I'd like to make sure I understand the direction ahc is going. A couple
questions...

I see that ahc is storing its information in swift. That's clever, but if
Ironic provided a blob store for each node, would that be better?

We discussed adding a search API to Ironic at the Vancouver summit, though
no work has been done on that yet, afaik. If ahc is going to grow a REST
API for searching for nodes based on specific criteria that it discovered,
could/should we combine these within Ironic's API?

From a service coupling perspective, I like the approach that ahc is
optional, and also that Ironic-inspector is optional, because this keeps
the simple use-case for Ironic, well, simple! That said, this seems more
like a configuration setting (should inspector do extra things?) than an
entirely separate service, and separating them might be unnecessarily
complicated.

It sounds like this is the direction you'd like to go, and you took the
current approach for expediency. If so, I'd like us to discuss a path to
merge the functionality as it matures, and decide whether a separate
repository is the right way to go long term.

Thanks much,
Devananda

On Mon, Jun 22, 2015, 05:40 John Trowbridge tr...@redhat.com wrote:

 This is a proposal to add a new repository governed by the ironic
 inspector subteam. The current repository is named ahc-tools[1], however
 there is no attachment to this name. ironic-inspector-extra would seem
 to fit if this is moved under the Ironic umbrella.

 What is AHC?
 
 * AHC as a term comes from the enovance edeploy installation method[2].
 * The general concept is that we want to have a very granular picture of
 the physical hardware being used in a deployment in order to be able to
 match specific hardware to specific roles, as well as the ability to
 find poor performing outliers before we attempt to deploy.
 * For example: As a cloud operator, I want to make sure all logical
 disks have random read IOPs within 15% variance of each other.
 * The huge benefit of this tooling over current inspection is the number
 of facts collected (~1000 depending on the hardware), all of which can
 be used for matching.
 * Another example: As an end user, I would like to request a bare metal
 machine with a specific model GPU.

 What is ahc-tools?
 --
 * We first tried to place all of this logic into a plugin in
 inspector[3] (discoverd at the time). [4]
 * This worked fine for just collecting some of the simple facts, however
 we now had a coupling between booting a ramdisk, and matching against
 the collected data.
 * ahc-tools started as a way to uncouple these two steps[5].
 * We also added a wrapper around the enovance report tooling[6], as it
 already had the ability to generate reports based on the collected data,
 but was designed to read in the data from the filesystem.
 * The report tool has two functions.
 * First, it can group the systems by category (NICs, Firmware,
 Processors, etc.).
 * Second, it can use statistical analysis to find performance outliers.

 Why is ahc-tools useful to Ironic?
 --
 * If we run benchmarks on hardware whenever it is turned back in by a
 tenant, we can easily put nodes into maintenance if the hardware is
 performing below some set threshold. This would allow us to have better
 certainty that the end user is getting what we promised them.
 * The advanced matching could also prove very useful. For VMs, I think
 the pets vs cattle analogy holds up very well, however many use cases
 for having cloud based bare metal involve access to specific hardware
 capabilities. I think advanced matching could help bridge this gap.

 Why not just put this code directly into inspector?
 ---
 * Clearly this code is 100% dependent on inspector. However, inspector
 is quite stable, and works great without any of this extra tooling.
 * ahc-tools is very immature, and will need many breaking changes to get
 to the same stability level of inspector.

 Why aren't you following the downstream-stackforge-openstack path?
 
 * This was the initial plan[7], however we were told that under the new
 big tent, that the openstack namespace is no longer meant to signify
 maturity of a project.
 * Instead, we were told we should propose the project directly to
 Ironic, or make a new separate project.

 What is the plan to make ahc-tools better?
 --
 * The first major overhaul we would like to do is to put the reporting
 and matching functionality behind a REST API.
 * Reporting in particular will require significant work, as the current
 wrapper script wraps code that was never designed to be a library (Its
 output is just a series of print statements). One option is to improve
 the library[8] to 

Re: [openstack-dev] [Ironic][Horizon][Tuskar-ui] Making a dashboard for Ironic

2015-06-19 Thread Devananda van der Veen
Hi Jim,

Your characterization of this is incomplete. These are not two equal
projects proposing the same thing in different ways, and while I very much
want to encourage collaboration, I value our community and feel that this
was not done in the spirit of that community.

To be clear: ironic-webclient has been developed with the knowledge and
support of the Ironic developer community, and was not moved into the
openstack/ namespace on my request, because I have been holding projects to
a certain level of maturity before including them in Ironic, roughly
equivalent to the TC's bar for big tent inclusion.

On the other hand, ironic-dashboard was done without the knowledge of any
Ironic cores, nor with even a heads up to the Ironic or Horizon PTLs.
Rather than an open design process, this code was just dropped on github
and Infra was asked to approve the project creation. I have not had the
opportunity to talk with its author *at all* yet.

I'm glad that ya'll didn't just approve the project creation request
without checking with me, and I'm glad we are now having this discussion.

Now that that is cleared up, let's move on.


Hi Zhenguo,

I have some questions about ironic-dashboard that I need answered before
the Ironic Project Team accepts responsibility for it.

The git history appears to be squashed [1], and most files don't have an
attribution header [2], and none of the headers refer to the company who
appears to be behind this (Huawei). What's the rationale for these
inconsistencies, and who is actually behind the code?

Are you going to maintain this project personally, or is there a team at
Huawei (or somewhere else) that is going to do that? Or are you expecting
Ironic's current developer teams to take ownership of this code and
maintain it?

Are you/they going to become part of Ironic's community, attend our weekly
meetings, and follow our design process?

What is the vision / goal that is being working towards? What is the scope
of this dashboard? How does it fit in with our existing plans?


I'm not entirely opposed to having two separate UI projects for Ironic at
the moment, but we should be very clear about the rationale if we go that
route.

-Devananda


[1]
https://github.com/niuzhenguo/ironic-dashboard/commit/4be73d19e54eb75aa31da3d1a38fa65c1287bc7b
[2] https://github.com/niuzhenguo/ironic-dashboard/search?q=copyright


On Fri, Jun 19, 2015 at 12:00 PM James E. Blair cor...@inaugust.com wrote:

 Hi all,

 I'm glad that by asking the ironic-dashboard creators to propose their
 project to ironic initially (rather than stackforge) we have prompted
 this conversation.  We now know that two independent groups of people
 have created standalone ironic dashboards, neither of which is currently
 part of an OpenStack project.

 We have an opportunity for those teams to begin collaborating now.  I
 would encourage them to do so, along with both the Ironic and Horizon
 teams, on a path forward.  Let's end the talk of -1ing someone's every
 patch and instead avail ourselves of one of the many ways in our
 community we can achieve and record consensus.  Michael, you have a plan
 with a number of steps, one of which is already an approved
 cross-project spec.  Perhaps that is a good starting point for this
 discussion.

 -Jim

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api] [Nova] [Ironic] [Magnum] Microversion guideline in API-WG

2015-06-19 Thread Devananda van der Veen
Almost all of our discussions so far on this topic have left something out,
which Monty pointed out to me last week. I'm following up now because
E_TRAVEL...

tldr;
What we're versioning here are API's, not packages. It's not a question of
numbering and dependency ordering, but of communicating support[ed|able]
interfaces between running services. Libtool is more relevant than semver.

http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html

Right now we lack a means to express that the API response is
compatible-with a particular previously-released version of the API. If
we had that, instead of current-version, I believe we would have much
happier users (and a happier Dmitry and jroll).


Long version...
Every HTTP response from Ironic today includes three headers: min, max, and
version. The service can present an older API version, as long as it is
greater-than-or-equal-to the minimum supported version, even if that
version is incompatible with the maximum supported version.  It does this
by rewriting responses to match what was expected in the requested (older)
version.

When the newer version is identical *for all interfaces present* in the
older version, this can be called compatible. Dmitry's point is that we
don't need to hide newer interfaces from users who request an older API
version, because the client won't know or care about things that weren't in
the version it requested.

However, we *do* need to signal their presence, and we don't have a good
means for that right now. We also need to signal to the client that the
response given is compatible with a certain age of API, even if it's
not exactly the same. And we don't have any means for that, either.

Time for an example

Let's say that an incompatible change was made in v1.3. Let's also say that
a change was made in v1.5 that added a new endpoint. Today, this is what
the response headers would look like when calling a server running v1.5.

a) client requests v1.2, receives headers (min: 1.1, max: 1.5, current:
1.2) b) client requests v1.4, receives headers (min: 1.1, max: 1.5, current
1.4) c) client requests v1.5, receives headers (min: 1.1, max: 1.5,
current: 1.5)

What we have implemented today is that in case (b), the service will *hide*
any changes that we introduced in v1.5. But those changes did not affect
any functionality of the v1.4 API, so Dmitry objects to this. So do I.

The issue at hand is the response in case (b) ... though after spending the
last several months working on api versioning, I actually think all of
those are poor responses.

What I believe we should have:
a) client requests v1.2, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.1)
b) client requests v1.4, receives headers  (min: 1.1, max: 1.5,
compatible-with: 1.3)
b) client requests v1.5, receives headers  (min: 1.1, max: 1.5,
compatible-with: 1.3)

Yes -- (b) and (c) are identical responses.

Discuss.

-Devananda


On Tue, Jun 16, 2015 at 7:13 AM Dmitry Tantsur dtant...@redhat.com wrote:

 On 06/16/2015 03:47 PM, Jim Rollenhagen wrote:
  On Tue, Jun 16, 2015 at 08:56:37AM +0200, Dmitry Tantsur wrote:
  On 06/04/2015 08:58 AM, Xu, Hejie wrote:
  Hi, guys,
  I’m working on adding Microversion into the API-WG’s guideline which
  make sure we have consistent Microversion behavior in the API for user.
  The Nova and Ironic already have Microversion implementation, and as I
  know Magnum _https://review.openstack.org/#/c/184975/_ is going to
  implement Microversion also.
  Hope all the projects which support( or plan to) Microversion can join
  the review of guideline.
  The Mircoversion specification(this almost copy from nova-specs):
  _https://review.openstack.org/#/c/187112_
  And another guideline for when we should bump Mircoversion
  _https://review.openstack.org/#/c/187896/_
  As I know, there already have a little different between Nova and
  Ironic’s implementation. Ironic return min/max version when the
 requested
  version doesn’t support in server by http-headers. There isn’t such
  thing in nova. But that is something for version negotiation we need
 for
  nova also.
  Sean have pointed out we should use response body instead of http
  headers, the body can includes error message. Really hope ironic team
  can take a
  look at if you guys have compelling reason for using http headers.
  And if we think return body instead of http headers, we probably need
  think about back-compatible also. Because Microversion itself isn’t
  versioned.
  So I think we should keep those header for a while, does make sense?
  Hope we have good guideline for Microversion, because we only can
 change
  Mircoversion itself by back-compatible way.
  Thanks
  Alex Xu
 
  Hi all!
 
  I'd like to try put in feedback based on living with microversions in
 Kilo
  release of Ironic.
 
  And here's my take, based on my experiences. Keep in mind I'm a core
  reviewer, a developer, and an operator of Ironic.

 Thanks Jim, much appreciated!

 
   

Re: [openstack-dev] [api][nova][ironic] Microversion API HTTP header

2015-06-19 Thread Devananda van der Veen
On Wed, Jun 17, 2015 at 7:31 AM Jay Pipes jaypi...@gmail.com wrote:

 On 06/17/2015 06:30 AM, Lucas Alvares Gomes wrote:
  overlap there rather than competition), how crazy does it sound if we
 say
  that for OpenStack Nova is the compute API and Ironic the Bare Metal
 API and
  so on? Would that be an unacceptable power grab?
 
  It's not that it's unacceptable, but I think that things weren't
  projected that way. Jay started this thread with this sentence:
 
  To be blunt, Nova is the *implementation* of the OpenStack Compute
  API. Ironic is the *implementation* of the OpenStack BareMetal API.
 
  Which I don't think is totally correct, at least for Ironic. The
  Ironic's API have evolved and shaped as we implemented Ironic, I think
  that some decisions we made in the API makes it clear, e.g:
 
  * Resources have JSON attributes. If you look at some attributes of
  the resources you will see that they are just a JSON blob. That's by
  design because we didn't know exactly how the API should look like and
  so by having these JSON fields it allows us to easily extend the
  resource without changing it's structure [1] (see driver_info,
  instance_info, extra)

 OK. Nothing wrong with that.

  * We have a vendor endpoint. This endpoint allows vendor to extend our
  API to expose new hardware capabilities that aren't present in the
  core API. Once multiple vendors starts implementing the same feature
  on this endpoint we then decide whether to promote it to the core API.

 This is a problem. The above means that there is no single OpenStack
 BareMetal API. This means developers that want to write against an
 OpenStack BareMetal API cannot rely on different deployments of Ironic
 exposing the same API. That is a recipe for a lack of interoperability
 and decreased developer ease of use.


Nope - not a problem. Actually it's been really helpful. We've found this
to be a better way to implement driver extensions -- it's clearly *not*
part of Ironic's API, and it's communicated as such in the API itself.

Any standard part of Ironic's functionality is exposed in the standard API,
and hardware-specific extensions, which are not supported by enough vendors
to be standardized yet, are only exposed through the vendor-specific API
endpoint. It's very clear in the REST API what this is -- the end points
are, for example,

  GET /v1/nodes//vendor_passthru/methods
  POST /v1/nodes//vendor_passthru?method=fooparam=bar

  GET /v1/drivers//methods

... and so on. This provides a mechanism to discover what resources and
methods said hardware vendor exposes in their hardware driver. We have
always supported out of tree drivers, and it is possible to upgrade Ironic
without upgrading the driver (or vice versa).

Also to note, our client library doesn't support any of the vendor-specific
methods, and never will. It only supports Ironic's API's ability to
*discover* what vendor-specific methods that driver exposes, and then to
transparently call to them. And none of that is relevant to other OpenStack
projects.

So if an operator wants to write a custom app that uses foo-vendor's
advanced-foo-making capabilities because they only buy Foo hardware --
that's great. They can do that. Presumably, they have a support contract
with Foo vendor. Ironic is merely providing the transport between them.




  * There's a reservation attribute in the Node's resource [1] which
  valueis the hostname of the conductor that is currently holding an
  exclusive lock to act upon this node. This is because internally we
  use a distributed hashing algorithm to be able to route the requests
  from the API service to a conductor service that is able to manage
  that Node. And having this field in the API

 That is just leaking implementation inappropriately out of the public
 REST API, and shouldn't be encouraged, IMHO. Nova has a number of these
 leaky parts of its API, too, of course. Just look at the os-guests API
 extension, which only works when you're using Xen under the hood,
 thereby leaking implementation details about the underlying
 infrastructure out through the public REST API.


yes, this is leaky in the purest sense, but remember that ironic doesn't
expose a *public* API. Only operators and other services should be talking
directly to it -- and this field was requested by operators who find it
helpful to know which service has locked a resource.


  I don't think that any of those decisions were bad by the way, this
  have helped us a lot to understand how a service to manage Bare Metal
  machines should looks like, and we have made wrong decisions too (You
  can get the same information by GET'ing different endpoints in the
  API, the Chassis resources currently have no usage apart from
  logically grouping nodes, etc...)

 Sure, all APIs have warts :) But the warts aren't an excuse to delay
 plugging up those leaks.


  So back to the topic. if we are removing the project name from the
  Header to facilitate another 

Re: [openstack-dev] [nova] How to microversion API code which is not in API layer

2015-06-13 Thread Devananda van der Veen
Yes. A new query parameter is a change in the contract, regardless of where
the code change lies.

-Deva
 On Jun 12, 2015 6:20 PM, Chen CH Ji jiche...@cn.ibm.com wrote:

 Hi
  We have [1] in the db layer and it's directly used by API
 layer , the filters is directly from client's input
  In this case, when doing [2] or similar changes, do we need
 to consider microversion usage when we change options?
  Thanks

 [1]
 https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4440
 [2] https://review.openstack.org/#/c/144883

 Best Regards!

 Kevin (Chen) Ji 纪 晨

 Engineer, zVM Development, CSTL
 Notes: Chen CH Ji/China/IBM@IBMCN   Internet: jiche...@cn.ibm.com
 Phone: +86-10-82454158
 Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
 Beijing 100193, PRC

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api] [Nova] [Ironic] [Magnum] Microversion guideline in API-WG

2015-06-07 Thread Devananda van der Veen
On Jun 5, 2015 4:36 AM, Sean Dague s...@dague.net wrote:

 On 06/05/2015 01:28 AM, Adrian Otto wrote:
 
  On Jun 4, 2015, at 11:03 AM, Devananda van der Veen
  devananda@gmail.com mailto:devananda@gmail.com wrote:
 
 
  On Jun 4, 2015 12:00 AM, Xu, Hejie hejie...@intel.com
  mailto:hejie...@intel.com wrote:
  
   Hi, guys,
  
   I’m working on adding Microversion into the API-WG’s guideline which
  make sure we have consistent Microversion behavior in the API for user.
   The Nova and Ironic already have Microversion implementation, and as
  I know Magnum https://review.openstack.org/#/c/184975/ is going to
  implement Microversion also.
  
   Hope all the projects which support( or plan to) Microversion can
  join the review of guideline.
  
   The Mircoversion specification(this almost copy from nova-specs):
  https://review.openstack.org/#/c/187112
   And another guideline for when we should bump Mircoversion
  https://review.openstack.org/#/c/187896/
  
   As I know, there already have a little different between Nova and
  Ironic’s implementation. Ironic return min/max version when the
requested
   version doesn’t support in server by http-headers. There isn’t such
  thing in nova. But that is something for version negotiation we need
  for nova also.
   Sean have pointed out we should use response body instead of http
  headers, the body can includes error message. Really hope ironic team
  can take a
   look at if you guys have compelling reason for using http headers.
  
   And if we think return body instead of http headers, we probably
  need think about back-compatible also. Because Microversion itself
  isn’t versioned.
   So I think we should keep those header for a while, does make sense?
  
   Hope we have good guideline for Microversion, because we only can
  change Mircoversion itself by back-compatible way.
 
  Ironic returns the min/max/current API version in the http headers for
  every request.
 
  Why would it return this information in a header on success and in the
  body on failure? (How would this inconsistency benefit users?)
 
  To be clear, I'm not opposed to *also* having a useful error message
  in the body, but while writing the client side of api versioning,
  parsing the range consistently from the response header is, IMO,
  better than requiring a conditional.
 
  +1. I fully agree with Devananda on this point. Use the headers
  consistently, and add helpful errors into the body only as an addition
  to that behavior, not a substitute.

 I think the difference between Nova and Ironic here is that Nova doesn't
 send all the headers all the time in the final implementation (that part
 of the spec evolved out I think). Part of that was pressure about Header
 bloat that people were concerned about, as that impacts caching layers.

 I would a agree that if Ironic is sending all the headers all the time,
 that's fine. However, for consistency it would be great to also put a
 real body that explains the issue as well,

Agreed.

 as headers are not the first
 place people look when things go wrong, and are often not logged by
 client side tools on errors (where the body would be).

 -Sean

 --
 Sean Dague
 http://dague.net

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] When to bump the microversion?

2015-06-04 Thread Devananda van der Veen
On Jun 4, 2015 11:11 AM, Chris Friesen chris.frie...@windriver.com
wrote:

 On 06/04/2015 10:14 AM, Devananda van der Veen wrote:


 On Jun 4, 2015 8:57 AM, Monty Taylor mord...@inaugust.com


   So, seriously - let's grow up and start telling people that they do
not
   get to pick and choose user-visible feature sets. If they have an
unholy
   obsession with a particular backend technology that does not allow a
   public feature of the API to work, then they are deploying a broken
   cloud and they need to fix it.
  

 So I just had dinner last night with a very large user of OpenStack
(yes, they
 exist)  whose single biggest request is that we stop differentiating
in the
 API. To them, any difference in the usability / behavior / API between
OpenStack
 deployment X and Y is a serious enough problem that it will have two
effects:
 - vendor lock in
 - they stop using OpenStack
 And since avoiding single vendor lock in is important to them, well,
really it
 has only one result.

 Tl;Dr; Monty is right. We MUST NOT vary the API or behaviour
significantly or
 non-discoverably between clouds. Or we simply won't have users.


 If a vendor wants to differentiate themselves, what about having two
sets of API endpoints?  One that is full vanilla openstack with
bog-standard behaviour, and one that has vendor-specific stuff in it?

 That way the end-users that want interop can just use the standard API
and get common behaviour across clouds, while the end-users that want the
special sauce and are willing to lock in to a vendor to get it can use
the vendor-specific API.


You've just described what ironic has already done with the
/vendor_passthu/ end point.

However, the issue, more broadly, is just discovery of differences in the
API which make one cloud behave differently than another. Sometimes those
aren't related to vendor-specific features at all. Eg, changes which are
the result of config settings, or where a fix or feature gets back ported
(because sometimes someone thinks that's easier than a full upgrade). These
things exist today, but create a terrible experience for users who want to
move a workload between OpenStack clouds, and find that the APIs don't
behave quite the same, even for basic functionality.

-D

 Chris


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api] [Nova] [Ironic] [Magnum] Microversion guideline in API-WG

2015-06-04 Thread Devananda van der Veen
On Jun 4, 2015 12:00 AM, Xu, Hejie hejie...@intel.com wrote:

 Hi, guys,

 I’m working on adding Microversion into the API-WG’s guideline which make
sure we have consistent Microversion behavior in the API for user.
 The Nova and Ironic already have Microversion implementation, and as I
know Magnum https://review.openstack.org/#/c/184975/ is going to implement
Microversion also.

 Hope all the projects which support( or plan to) Microversion can join
the review of guideline.

 The Mircoversion specification(this almost copy from nova-specs):
https://review.openstack.org/#/c/187112
 And another guideline for when we should bump Mircoversion
https://review.openstack.org/#/c/187896/

 As I know, there already have a little different between Nova and
Ironic’s implementation. Ironic return min/max version when the requested
 version doesn’t support in server by http-headers. There isn’t such thing
in nova. But that is something for version negotiation we need for nova
also.
 Sean have pointed out we should use response body instead of http
headers, the body can includes error message. Really hope ironic team can
take a
 look at if you guys have compelling reason for using http headers.

 And if we think return body instead of http headers, we probably need
think about back-compatible also. Because Microversion itself isn’t
versioned.
 So I think we should keep those header for a while, does make sense?

 Hope we have good guideline for Microversion, because we only can change
Mircoversion itself by back-compatible way.

Ironic returns the min/max/current API version in the http headers for
every request.

Why would it return this information in a header on success and in the body
on failure? (How would this inconsistency benefit users?)

To be clear, I'm not opposed to *also* having a useful error message in the
body, but while writing the client side of api versioning, parsing the
range consistently from the response header is, IMO, better than requiring
a conditional.

-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] When to bump the microversion?

2015-06-04 Thread Devananda van der Veen
On Jun 4, 2015 8:57 AM, Monty Taylor mord...@inaugust.com wrote:

 On 06/04/2015 11:27 AM, Dmitry Tantsur wrote:
  On 06/04/2015 05:03 PM, Sean Dague wrote:
  On 06/04/2015 10:50 AM, Dmitry Tantsur wrote:
  On 06/04/2015 04:40 PM, Ruby Loo wrote:
  Hi,
 
  In Kilo, we introduced microversions but it seems to be a
  work-in-progress. There is an effort now to add microversion into the
  API-WG's guidelines, to provide a consistent way of using
microversions
  across OpenStack projects [1]. Specifically, in the context of this
  email, there is a proposed guideline for when to bump the
microversion
  [2].
 
  As I understand this guideline tells to bump microversion on every
  change which I strongly -1 as usual. Reason: it's bump for the sake of
  bump, without any direct benefit for users (no, API discoverability is
  not one, because microversion do not solve it).
 
  I'll post the same comment to the guideline.
 
  Backwards compatible API adds with no user signaling is a fallacy
  because it assumes the arrow of time flows only one way.
 
  If at version 1.5 you have a resource that is
 
  foo {
 bar: ...
  }
 
  And then you decide you want to add another attribute
 
  foo {
 bar: ...
 baz: ...
  }
 
  And you don't bump the version, you'll get a set of users that use a
  cloud with baz, and incorrectly assume that version 1.5 of the API
means
  that baz will always be there. Except, there are lots of clouds out
  there, including ones that might be at the code commit before it was
  added. Because there are lots of deploys in the world, your users can
  effectively go back in time.
 
  So now your API definition for version 1.5 is:
 
  foo, may or may not contain baz, and there is no way of you knowing if
  it will until you try. good luck.
 
  Which is pretty aweful.
 
  Which is not very different from your definition. Version 1.5 contains
  feature xyz, unless it's disabled by the configuration or patched out
  downstream. Well, 1.4 can also contain the feature, if downstream
  backported it. So good luck again.
 
  If you allow to group features under one microversion, that becomes even
  worse - you can have deployment that got microversion only partially.
 
  For example, that's what I would call API discoverability:
 
   $ ironic has-capability foobar
   true
 
  and that's how it would play with versioning:
 
   $ ironic --ironic-api-version 1.2 has-capability foobar
   false
   $ ironic --ironic-api-version 1.6 has-capability foobar
   true
 
  On the contrary, the only thing that microversion tells me is that the
  server installation is based on a particular upstream commit.
 
  To me these are orthogonal problems, and I believe they should be solved
  differently. Our disagreement is due to seeing them as one problem.

 We should stop doing this everywhere in OpenStack. It is the absolute
 worst experience ever.

 Stop allowing people to disable features with config. there is literally
 no user on the face of the planet for whom this is a positive thing.

 1.5 should mean that your server has Set(A) of features. 1.6 should mean
 Set(A+B) - etc. There should be NO VARIATION and any variation on that
 should basically mean that the cloud in question is undeniably broken.

 I understand that vendors and operators keep wanting to wank around with
 their own narccisitic arrogance to differentiate from one another.

 STOP IT

 Seriously, it causes me GIANT amount of pain and quite honestly if I
 wasn't tied to using OpenStack because I work on it, I would have given
 up on it a long time ago because of evil stuff like this.

 So, seriously - let's grow up and start telling people that they do not
 get to pick and choose user-visible feature sets. If they have an unholy
 obsession with a particular backend technology that does not allow a
 public feature of the API to work, then they are deploying a broken
 cloud and they need to fix it.


So I just had dinner last night with a very large user of OpenStack (yes,
they exist)  whose single biggest request is that we stop differentiating
in the API. To them, any difference in the usability / behavior / API
between OpenStack deployment X and Y is a serious enough problem that it
will have two effects:
- vendor lock in
- they stop using OpenStack
And since avoiding single vendor lock in is important to them, well, really
it has only one result.

Tl;Dr; Monty is right. We MUST NOT vary the API or behaviour significantly
or non-discoverably between clouds. Or we simply won't have users.

 
  Looking at your comments in the WG repo you also seem to only be
  considering projects shipped at Major release versions (Kilo, Liberty).
  Which might be true of Red Hat's product policy, but it's not generally
  true that all clouds are at a release boundary. Continous Deployment of
  OpenStack has been a value from Day 1, and many public clouds are not
  using releases, but are using arbitrary points off of master.
 
  I don't know why you decided I 

Re: [openstack-dev] [Ironic] ENROLL state and changing node driver

2015-06-04 Thread Devananda van der Veen
On Jun 4, 2015 5:53 AM, Dmitry Tantsur dtant...@redhat.com wrote:

 Hi!

 While working on the enroll spec [1], I got a thinking: within the new
state machine, when should we allow to change a node driver?

 My initial idea was to only allow driver change in ENROLL. Which sounds
good to me, but then it will be impossible to change a driver after moving
forward: we don't plan on having a way back to ENROLL from MANAGEABLE.

 What do you folks think we should do:
 1. Leave driver field as it was before
 2. Allow changing driver in ENROLL, do not allow later
 3. Allow changing driver in ENROLL only, but create a way back from
MANAGEABLE to ENROLL (unmanage??)


What problem are you trying to solve? Because I don't see a problem with
the current behavior, and you're proposing breaking the API and requiring
users to follow a significantly more complex process should they need to
change what driver is in use for a node, and preventing ever doing that
while a workload is running...

-Deva

 Cheers,
 Dmitry

 [1] https://review.openstack.org/#/c/179151

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] Updates from the Summit

2015-05-28 Thread Devananda van der Veen
Hello!

It's that time - hopefully everyone has recovered from the summit by
now and is getting back in the swing of things.

I'd like to take a little of everyone's time to summarize the main
points we covered while in Vancouver (and record them here for folks
who didn't make it, were in other sessions, etc).


Release Cycle
--

There's a lot of interest in switching from the 6mo release cycle to
an independent release cycle. I've started a discussion on that. [1]


Networking
--

This was probably the biggest and most consistent theme -- neutron
integration to allow for separation of provisioning  tenant networks,
and for tenant network isolation, at the physical switch layer, needs
to be completed. Several companies have done POCs, but we need to move
this into the main code.

Sukhdev has offered to lead this work, and has set up a cross-project
meeting to kick it off. [2] I'd say, this is our highest priority this
cycle.

tldr; we'll need to do some work in both the Nova driver and in Ironic
-- but mostly in Ironic -- to change the sequence of Neutron calls,
and to pass additional information to Neutron. We will also need to
define  document the network metadata which Ironic owns (read: is the
source of truth) and supplies to Neutron.


Driver Matrix
-

We already knew that our driver list was getting out of hand and only
getting worse. So, we came up with a plan to address it.
- name each Driver after the type of hardware it manages (eg, drac,
ilo, amt, ...) rather than the combination of interfaces it implements
- allow dynamic loading/swapping of some interfaces (eg, deploy,
console) where appropriate for that driver, based on the node's
driver_info
- have sane and well tested defaults for each interface which can be exchanged

Why this approach? Because the unique thing about a given server is
the hardware interfaces required to managed it -- not the software
interfaces we might use to copy an image, start a console server, etc.

This will require some careful thought to ensure the config options as
well as the API options (eg, when enrolling a node) remain backwards
compatible.


Driver Interfaces
-

We'll be splitting part of the DeployInterface out into a new
BootInterface this cycle. This will facilitate the creation of drivers
that no-op deployment -- in other words, pure boot from network --
and also facilitate the simplification of the driver matrix mentioned
above.

Additionally, we will be promoting the heartbeat / lookup /
pass_deploy_info methods, which are currently implemented in
VendorPassthru, to become a standard part of the Deploy() interface.
It turns out every deploy driver we've got implements this
functionality, and promoting common interfaces out of VendorPassthru
is one of the reasons that interface exists :)


Cinder integration
-

There are two things people mean when they say cinder integration
with ironic: attach volumes, and boot from volumes.

Attaching iSCSI volumes to hardware is cool *and* most of the work is
already done -- but the cinder CLI commands to get the endpoint
weren't documented, so we didn't know. Also, we need some actor inside
the guest to attach and mount the volume. That could go into
cloud-init-v2, or an optional process that gets added to the machine
images. Without that actor, the user has to initiate the attach and
mount themselves -- which is fine, too, but again, the cinder CLI
command to get that endpoint wasn't documented, and a little work is
needed there.

For out-of-band attach (eg, FCoE) the Cinder API should be similar,
but we'll need to create an interface in Ironic to receive and act
upon that information (ie, pass it down to the hardware driver to
initiate the attachment).

For boot-from-volume, passing the iSCSI information to Ironic and
chain-loading iPXE from it is the most straight-forward way, but
should be delayed until after we refactor the Boot interface. Ditto
for boot-from-FCoE -- if we can delay this until the Boot interface is
refactored, then it merely becomes an option to that interface, rather
than a whole new DeployDriver.


State Machine
--

We designed more states than we were able to implement last cycle.
This cycle, we'll be adding the enrolled state and changing the
process for adding Nodes to Ironic to ensure they are manageable
before making them available to Nova.

We will also implement the zapping state, which is where long-lived
operations should take place (eg, flashing firmware or a slow-build of
a RAID array). If there's time, we'll also implement the rescue
state, though drafts of this appear to be dependent on the network
isolation work, so it may get bumped.


Better Onboarding  Operator docs
---

There were multiple discussions about the need to improve our
operator-focused getting started documentation. Yes. We know :)

I talked with the OpenStack Docs team, and we'll be working on a way
to 

[openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model

2015-05-28 Thread Devananda van der Veen
Hi all,

tl;dr;

At the summit, the Ironic team discussed the challenges we've had with
the current release model and came up with some ideas to address them.
I had a brief follow-up conversation with Doug and Thierry, but I'd
like this to be discussed more openly and for us (the Ironic dev
community) to agree on a clear plan before we take action.

If Ironic moves to a release:independent model, it shouldn't have any
direct effect on other projects we integrate with -- we will continue
to follow release:at-6mo-cycle-end -- but our processes for how we get
there would be different, and that will have an effect on the larger
community.

Longer version...

We captured some notes from our discussion on Thursday afternoon's etherpad:
https://etherpad.openstack.org/p/liberty-ironic-scaling-the-dev-team

Which I've summarized below, and mixed in several themes which didn't
get captured on the 'pad but were discussed somewhere, possibly in a
hallway or on Friday.

Current challenges / observations:
- six weeks' feature freeze is not actually having the desired
stabilizing effect
- the post-release/pre-summit slump only makes that worse
- many folks stop reviewing during this time because they know their
own features won't land, and instead focus their time downstream
- this creates pressure to land features which aren't ready, and
leaves few people to find bugs, write docs, and generally prepare the
release during this window

The alternative we discussed:
- use feature branches for risky / large work, keeping total # of
branches small, and rebasing them regularly on master
- keep trunk moving quickly for smaller / less risky / refactoring changes
- slow down for a week or two before a release, but dont actually
freeze master
- cut releases when new features are available
- OpenStack coordinated releases are taken from latest independent release
- that release will then get backports  stable maintenance, other
independent releases don't

We think this will accomplish a few things:
- make the developer experience better by being more consistent, thus
keeping developers engaged year-round and increase the likelyhood
they'll find and fix bugs
- reduce stress on core reviewers since there's no crunch time at
the end of a cycle
- allow big changes to bake in a feature branch, rather than in a
series of gerrit patches that need to be continually re-reviewed and
cherry-picked to test them.
- allow operators who wish to use Ironic outside of OpenStack to
consume feature releases more rapidly, while still consuming approved
releases instead of being forced to deploy from trunk


For reference, Michael has posted a tracking change to the governance
repo here: https://review.openstack.org/#/c/185203/

Before Ironic actually makes the switch, I would like us to discuss
and document the approach we're going to take more fully, and get
input from other teams on this approach. Often, the devil is in the
details - and, for instance, I don't yet understand how we'll fit this
approach into SemVer, or how this will affect our use of launchpad to
track features (maybe it means we stop doing that?).

Input appreciated.

Thanks,
Devananda

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Using depends-on for patches which require an approved spec

2015-05-27 Thread Devananda van der Veen
I think this will help because it separates the judgement of is this code
good enough to land from the project and release coordination of should
this code land now.

I've been floating the idea of separating +2 and +A powers for the same
purpose: free up many of the technical reviewers from having to *also*
think about release schedules and project priorities, since larger projects
are tending towards having a smaller group of project-drivers who handle
the latter question.

It looks like Depends-On is one way to address that, and it's fairly light
weight; since we can edit the review message, core-reviewers can add that
line if they feel it's needed, without needing to -1 the code in question.
I'm open to trying it in the Ironic project.

-Devananda

On Tue, May 26, 2015 at 8:45 AM Daniel P. Berrange berra...@redhat.com
wrote:

 On Fri, May 22, 2015 at 02:57:23PM -0700, Michael Still wrote:
  Hey,
 
  it would be cool if devs posting changes for nova which depend on us
  approving their spec could use Depends-On to make sure their code
  doesn't land until the spec does.

 Does it actually bring any benefit ?  Any change for which there is
 a spec is already supposed to be tagged with 'Blueprint: foo-bar-wiz'
 and nova core devs are supposed to check the blueprint is approved
 before +A'ing it.  So also adding a Depends-on just feels redundant
 to me, and so is one more hurdle for contributors to remember to
 add. If we're concerned people forget the Blueprint tag, or forget
 to check blueprint approval, then we'll just have same problem with
 depends-on - people will forget to add it, and cores will forget
 to check the dependant change. So this just feels like extra rules
 for no gain and extra pain.

 Regards,
 Daniel
 --
 |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/
 :|
 |: http://libvirt.org  -o- http://virt-manager.org
 :|
 |: http://autobuild.org   -o- http://search.cpan.org/~danberr/
 :|
 |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc
 :|

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] no IRC meeting tonight

2015-05-25 Thread Devananda van der Veen
Some folks are still traveling (or recovering) from the summit, and today
is a holiday in the US, so I'm cancelling tonight's meeting.

There are many action items recorded in the etherpads - we'll have a lot to
talk about next week!

-Devananda
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison

2015-05-25 Thread Devananda van der Veen
Thanks for all your contributions, Ghe! You've done a lot to help keep this
project synced with the common libs -- of course, you're welcome back any
time.

Best
Deva

On Mon, May 25, 2015, 09:49 Ghe Rivero g...@debian.org wrote:

 My focus on the Ironic project has been decreasing in the last cycles, so
 it's about time to relinquish my position as a oslo-ironic liaison so new
 contributors can take over it and help ironic to be the vibrant project it
 is.

 So long, and thanks for all the fish,

 Ghe Rivero
 --
 Pinky: Gee, Brain, what do you want to do tonight?
 The Brain: The same thing we do every night, Pinky—try to take over the
 world!

  .''`.  Pienso, Luego Incordio
 : :' :
 `. `'
   `-www.debian.orgwww.openstack.com

 GPG Key: 26F020F7
 GPG fingerprint: 4986 39DA D152 050B 4699  9A71 66DB 5A36 26F0 20F7
  __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][all] Architecture Diagrams in ascii art?

2015-05-12 Thread Devananda van der Veen
In ironic, we have use asciiflow several times quite successfully in the
spec process.

We also maintain in-tree docs with .PNG graphics, and versioning those has
been a bit of a pain. They were originally taken out of slide decks, and
served their purpose, but really ought to be in a different format or a
different tree now.

-D
On May 12, 2015 2:15 AM, John Garbutt j...@johngarbutt.com wrote:

 On 11 May 2015 at 23:46, Boris Pavlovic bo...@pavlovic.me wrote:
  Couldn't we just use real image files to do this. IIRC, gerrit supports
  displaying image files which are included in a commit. For example, I've
  been
  planning to copy these images:
 
  +1 for real images

 One suggestion I remember around specs was we might want a separate
 repo to contain the images, to stop massively increasing the git clone
 times.

 In the spec template we recommend this to generate diagrams:
 http://asciiflow.com/

 See:
 http://specs.openstack.org/openstack/nova-specs/specs/liberty/template.html

 Thanks,
 John

  On Tue, May 12, 2015 at 1:36 AM, Matthew Treinish mtrein...@kortar.org
  wrote:
 
  On Mon, May 11, 2015 at 02:57:48PM -0700, Joe Gordon wrote:
   When learning about how a project works one of the first things I look
   for
   is a brief architecture description along with a diagram. For most
   OpenStack projects, all I can find is a bunch of random third party
   slides
   and diagrams.
  
   Most Individual OpenStack projects have either no architecture diagram
   or
   ascii art. Searching for 'OpenStack X architecture' where X is any of
   the
   OpenStack projects turns up pretty sad results. For example heat [0]
 an
   Keystone [1] have no diagram. Nova on the other hand does have a
   diagram,
   but its ascii art [2]. I don't think ascii art makes for great user
   facing
   documentation (for any kind of user).
  
   So how can we do better then ascii art architecture diagrams?
  
   [0] http://docs.openstack.org/developer/heat/architecture.html
   [1] http://docs.openstack.org/developer/keystone/architecture.html
   [2] http://docs.openstack.org/developer/nova/devref/architecture.html
 
  Couldn't we just use real image files to do this. IIRC, gerrit supports
  displaying image files which are included in a commit. For example, I've
  been
  planning to copy these images:
 
  https://wiki.openstack.org/wiki/QA/AuthInterface
 
  into the tempest docs for some time. They just need to be updated a bit
 to
  reflect some recent changes.
 
  The only downside I see is that it makes editing more difficult, I guess
  that's
  really the tradeoff.
 
  -Matt Treinish
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][all] Architecture Diagrams in ascii art?

2015-05-12 Thread Devananda van der Veen
Woops. I missed most of this thread in my last reply.

I'm all for using open standard formats and versioning them. However. Not
being a graphical artist myself, I have found the learning curve on some of
those tools daunting, eg. inkscape, which means I'm far less likely to
update a graphic in a format that requires me to go learn that tool first.
Also, it's awkward to require a Python developer to update an SVG because
that is documentation affected by their commit.

If we go with a common tool/format like libre office/ODF. I suggest we
adopt some commonalities, and still keep things simple enough that we can
reasonably expect any developer to update it.

-D
On May 12, 2015 12:33 PM, Sean Dague s...@dague.net wrote:

 On 05/12/2015 01:12 PM, Jeremy Stanley wrote:
  On 2015-05-12 10:04:11 -0700 (-0700), Clint Byrum wrote:
  It's a nice up side. However, as others have pointed out, it's only
  capable of displaying the most basic pieces of the architecture.
 
  For higher level views with more components, I don't think ASCII art
  can provide enough bandwidth to help as much as a vector diagram.
 
  Of course, simply a reminder that just because you have one or two
  complex diagram callouts in a document doesn't mean it's necessary
  to also go back and replace your simpler ASCII art diagrams with
  unintelligible (without rendering) SVG or Postscript or whatever.
  Doing so pointlessly alienates at least some fraction of readers.

 Sure, it's all about trade offs.

 But I believe that statement implicitly assumes that ascii art diagrams
 do not alienate some fraction of readers. And I think that's a bad
 assumption.

 If we all feel alienated every time anyone does anything that's not
 exactly the way we would have done it, it's time to give up and pack it
 in. :) This thread specifically mentioned source based image formats
 that were internationally adopted open standards (w3c SVG, ISO ODG) that
 have free software editors that exist in Windows, Mac, and Linux
 (Inkscape and Open/LibreOffice).

 -Sean

 --
 Sean Dague
 http://dague.net

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] how about those alternating meeting times?

2015-05-11 Thread Devananda van der Veen
Lucas inspired me to take a look at the raw numbers ... so I hacked up a
little python to digest all our meeting logs since we made the switch to
alternating times.

in particular, I'd like to point out the number of meetings with less than
half of our core review team present, ie, where we didn't have quorum to
make any decisions. There were 6 on Tuesdays (two thirds of all Tuesday
meetings), but only 1 on Monday (it was the openstack vacation week).

A few stats below, hackish code here:
http://paste.openstack.org/show/220234/

# of meetings by day:
  Monday: 11
  Tuesday: 9

Total lines in IRC during the meetings by day:
  Monday: 3793
  Tuesday: 2475

Unique attendees per day:
  Monday: total: 54 - cores: 9
  Tuesday: total: 32 - cores: 5



On Mon, May 11, 2015 at 10:21 AM Devananda van der Veen 
devananda@gmail.com wrote:

 At the Paris summit, we discussed and eventually agreed to try out
 alternating meeting times. So, for the last ~6mo, each week our IRC meeting
 has alternated between Monday evening and Tuesday morning UTC (10am PT and
 10pm PT Mondays).

 I'd like us to review how well this has - or has not - worked, so I've
 created a poll to gather input and help decide if we should continue with
 the alternating meetings. Please take a minute to fill this out:

 http://goo.gl/forms/nvbWdOZMFY

 And please reply to this thread if you have any comments or observations
 you'd like to share in addition to the responses on that form.

 Thanks!
 -Deva

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] how about those alternating meeting times?

2015-05-11 Thread Devananda van der Veen
At the Paris summit, we discussed and eventually agreed to try out
alternating meeting times. So, for the last ~6mo, each week our IRC meeting
has alternated between Monday evening and Tuesday morning UTC (10am PT and
10pm PT Mondays).

I'd like us to review how well this has - or has not - worked, so I've
created a poll to gather input and help decide if we should continue with
the alternating meetings. Please take a minute to fill this out:

http://goo.gl/forms/nvbWdOZMFY

And please reply to this thread if you have any comments or observations
you'd like to share in addition to the responses on that form.

Thanks!
-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][python-novaclient] microversion implementation on client side

2015-04-28 Thread Devananda van der Veen
FWIW, we enumerated the use-cases and expected behavior for all
combinations of server [pre versions, older version, newer version]
and client [pre versions, older version, newer version, user-specified
version], in this informational spec:

http://specs.openstack.org/openstack/ironic-specs/specs/kilo/api-microversions.html#proposed-change

Not all of that is implemented yet within our client, but the
auto-negotiation of version is done. While our clients probably don't
share any code, maybe something here can help:

http://git.openstack.org/cgit/openstack/python-ironicclient/tree/ironicclient/common/http.py#n72

-Deva

On Mon, Apr 27, 2015 at 2:49 AM, John Garbutt j...@johngarbutt.com wrote:
 I see these changes as really important.

 We need to establish good patterns other SDKs can copy.

 On 24 April 2015 at 12:05, Alex Xu sou...@gmail.com wrote:
 2015-04-24 18:15 GMT+08:00 Andrey Kurilin akuri...@mirantis.com:
 When user execute cmd without --os-compute-version. The nova client
  should discover the nova server supported version. Then cmd choice the
  latest version supported both by client and server.

 In that case, why X-Compute-API-Version can accept latest value? Also,
 such discovery will require extra request to API side for every client call.


 I think it is convenient for some case. like give user more easy to try nova
 api by some code access the nova api directly. Yes, it need one more extra
 request. But if without discover, we can't ensure the client support server.
 Maybe client too old for server even didn't support the server's min
 version. For better user experience, I think it worth to discover the
 version. And we will call keystone each nova client cli call, so it is
 acceptable.

 We might need to extend the API to make this easier, but I think we
 need to come up with a simple and efficient pattern here.


 Case 1:
 Existing python-novaclient calls, now going to v2.1 API

 We can look for the transitional entry of computev21, as mentioned
 above, but it seems fair to assume v2.1 and v2.0 are accessed from the
 same service catalog entry of compute, by default (eventually).

 Lets be optimistic about what the cloud supports, and request latest
 version from v2.1.

 If its a v2.0 only API endpoint, we will not get back a version header
 with the response, we could error out if the user requested v2.x
 min_version via the CLI parameters.

 In most cases, we get the latest return values, and all is well.


 Case 2:
 User wants some info they know was added to the response in a specific
 microversion

 We can request latest and error out if we don't get a new enough
 version to meet the user's min requirement.


 Case 3:
 Adding support for a new request added in a microversion

 We could just send latest and assume the new functionality, then
 raise an error when you get bad request (or similar), and check the
 version header to see if that was the cause of the problem, so we can
 say why it failed.

 If its supported, everything just works.

 If the user requests a specific version before it was supported, we
 should error out as not supported, I guess?

 In a way it would be cleaner if we had a way for the client to say
 latest but requires 2.3, so you get a bad version request if your
 minimum requirement is not respected, so its much clearer than
 miss-interpreting random errors that you might generate. But I guess
 its not totally required here.


 Would all that work? It should avoid an extra API call to discover the
 specific version we have available.

 '--os-compute-version=None' can be supported, that means will return the
  min version of server supported.

 From my point of view '--os-compute-version=None' is equal to not
 specified value. Maybe, it would be better to accept value min for
 os-compute-version option.

 I think '--os-compute-version=None' means not specified version request
 header when send api request to server. The server behavior is if there
 isn't value specified, the min version will be used.

 --os-compte-version=v2 means no version specified I guess?

 Can we go back to the use cases here please?
 What do the users need here and why?


 3. if the microversion non-supported, but user call cmd with
  --os-compute-version, this should return failed.

 Imo, it should be implemented on API side(return BadRequest when
 X-Compute-API-Version header is presented in V2)

 V2 is already deployed now, and doesn't do that.

 No matter what happens we need to fix that.

 Emm I'm not sure. Because GET '/v2/' already can be used to discover
 microversion support or not. Sounds like add another way to support
 discover? And v2 api didn't return fault with some extra header, that sounds
 like behavior and back-incompatible change.

 -1

 We should not use the URL to detect the version.
 We have other ways to do that for a good reason.

 Thanks,
 John



 On Fri, Apr 24, 2015 at 12:42 PM, Alex Xu sou...@gmail.com wrote:



 2015-04-24 17:24 GMT+08:00 Andrey 

Re: [openstack-dev] [tripleo] [ironic] Where to keep discover images

2015-04-14 Thread Devananda van der Veen
Yea, option #2 is what I had in mind. Like you said, I think most of
the code is already present in Ironic, and if it's not already
factored into a reusable thing (eg, ironic/utils or
ironic/driver/utils or such) then it should be.

Cheers,
-Deva

On Tue, Apr 14, 2015 at 1:40 PM, Dmitry Tantsur divius.ins...@gmail.com wrote:
 Hi,

 actually 2 possibilities here:
 1. discoverd itself handles TFTP
 2. DiscoverdInspect hanldes TFTP

 I vote for the 2nd, as we have all required code in Ironic already. I guess
 initial question was about the 1st case, which I doubt is worth supporting.
 Anyway, nice idea for an improvement!

 Dmitry


 2015-04-14 22:27 GMT+02:00 Devananda van der Veen devananda@gmail.com:

 I'm wondering Rather than have a static config, could the
 DiscoverdInspect interface handle setting up the TFTP config, pulling
 those images from Glance, etc, when a node is moved into the inspect
 state (assuming such integration was desired by the cloud operator)?

 -Deva

 On Fri, Apr 10, 2015 at 12:31 AM, Dmitry Tantsur dtant...@redhat.com
 wrote:
  On 04/10/2015 01:43 AM, Jaromir Coufal wrote:
 
  Hey Dmitry,
 
 
  o/
 
 
  I wanted to ask you about ironic-discoverd.
 
  At the moment, after build, the discovery images are copied into local
  folder:
 
  TFTP_ROOT=${TFTP_ROOT:-/tftpboot}
 
  sudo cp -f $IMAGE_PATH/$DISCOVERY_NAME.kernel
  $TFTP_ROOT/discovery.kernel
  sudo cp -f $IMAGE_PATH/$DISCOVERY_NAME.initramfs
  $TFTP_ROOT/discovery.ramdisk
 
  I am wondering why is that and if discoverd can work with these images
  if they were loaded into glance.
 
 
  Discoverd is not concerned with TFTP configuration (unlike Ironic), so
  you
  can put them everywhere, provided that your TFTP still works. Currently
  we
  use static configuration, as it's the easiest one.
 
 
  I mean it would be definitely more
  convenient than keeping them locally.
 
  Thanks
  -- Jarda
 
 
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 --
 -- Dmitry Tantsur
 --

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo] [ironic] Where to keep discover images

2015-04-14 Thread Devananda van der Veen
I'm wondering Rather than have a static config, could the
DiscoverdInspect interface handle setting up the TFTP config, pulling
those images from Glance, etc, when a node is moved into the inspect
state (assuming such integration was desired by the cloud operator)?

-Deva

On Fri, Apr 10, 2015 at 12:31 AM, Dmitry Tantsur dtant...@redhat.com wrote:
 On 04/10/2015 01:43 AM, Jaromir Coufal wrote:

 Hey Dmitry,


 o/


 I wanted to ask you about ironic-discoverd.

 At the moment, after build, the discovery images are copied into local
 folder:

 TFTP_ROOT=${TFTP_ROOT:-/tftpboot}

 sudo cp -f $IMAGE_PATH/$DISCOVERY_NAME.kernel
 $TFTP_ROOT/discovery.kernel
 sudo cp -f $IMAGE_PATH/$DISCOVERY_NAME.initramfs
 $TFTP_ROOT/discovery.ramdisk

 I am wondering why is that and if discoverd can work with these images
 if they were loaded into glance.


 Discoverd is not concerned with TFTP configuration (unlike Ironic), so you
 can put them everywhere, provided that your TFTP still works. Currently we
 use static configuration, as it's the easiest one.


 I mean it would be definitely more
 convenient than keeping them locally.

 Thanks
 -- Jarda



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] ironic-specs open for Liberty!

2015-04-13 Thread Devananda van der Veen
Jay,

Thanks for staying on top of updating  opening Liberty specs, and for all
your help with specs during Kilo!

-Deva

On Fri, Apr 10, 2015 at 10:40 AM Jay Faulkner j...@jvf.cc wrote:

 Hi,

 Just a note to let you know Liberty specs are open for Ironic.

 Template Changes
 
 There are two minor changes to the spec template for Liberty:
  - State Machine Impact is now a full section, where changes to Ironic’s
 State Machine should be called out.
  - In the REST API Impact section, submitters should indicate if the
 microversion should be incremented as part of the change.


 Kilo Specs
 —
 If you had a spec approved for Kilo that didn’t get implemented, please
 put in a merge request moving the spec from kilo-archive/ to liberty/, and
 ensure your spec complies with the new template by adding a State Machine
 Impact section and indicating any microversion version changes needed in
 the REST API Impact section.


 Backlog Specs
 ———
 As a reminder; Ironic does still employ a spec backlog for desired
 features that aren’t ready for implementation due to time, priority, or
 dependencies. If there are any specs currently in the backlog you’d like to
 propose for Liberty, please move it from backlog/ to liberty/ and add the
 additional required information for a full spec.


 Thanks to everyone for a successful Kilo cycle, and I’m looking forward to
 seeing the new slate of ideas for Liberty.


 -
 Jay Faulkner
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Kilo stable branches for other libraries

2015-04-08 Thread Devananda van der Veen
Thierry,

You left out python-ironicclient, which isn't a surprise as it isn't
actually listed in Nova's requirements.txt file. I don't have a link handy
to cite the previous discussions, but Nova felt that it was not appropriate
to list a driver's dependency in their project's requirements file.

As such, it is installed from pip in devstack/lib/ironic right now.

I've tagged a 0.5.0 version two days ago, and plan a quick fix (0.5.1)
today. I think it's reasonable for this library to be capped just like the
other python-*clients, but I'm not sure how to express that, due to Nova
not allowing this dependency in their requirements.txt file.

-Devananda

On Wed, Apr 8, 2015 at 7:18 AM Matthias Runge mru...@redhat.com wrote:

 On 08/04/15 15:55, Thierry Carrez wrote:

  I'm especially worried with python-cinderclient, python-designateclient
  and django_openstack_auth which are more than 2 months old and may well
  contemplate another kilo release that could be disrupting at this point.
 
 In general: a great idea, and I've been expecting this for a longer time
 now. That would save some work.

 django_openstack_auth: it's quite stable now, although it would make
 sense to cut a newer release for kilo. We will merge that into horizon
 eventually, since it's a helper and we never saw anyone to use it
 outside of horizon.

 Matthias

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] How to deal with microversions in 3rdparty tools

2015-04-08 Thread Devananda van der Veen
On Wed, Apr 8, 2015 at 7:38 AM Chris Friesen chris.frie...@windriver.com
wrote:

 On 04/07/2015 11:35 PM, Michael Davies wrote:
  I agree with Jim R's suggestion - it's really up to the consumer as to
 what they
  want to do.  Having said that...
 
  I think that any consumer wants to use the latest version of the API
 that it can
  support.
 
  And so since we're not planning on making any breaking API changes[1], I
 think
  any consumer wants to:
 
  a) have a concept of the latest API version that it has been coded for
  b) then, in negotiation with the server, choose a version that suffices:
  b1) negotiated_version = min(your code's max version, max Ironic server
 version) and
  b2) negotiated_version  your code's supported version
  b3) negotiated_version  Ironic API's minimum version

 Is that statement about not planning on making any breaking API changes
 an
 intention or a guarantee?

 The reason I ask is that doc/source/devref/api_microversions.rst contains
 an
 explicit mention of making breaking changes:  So breaking changes can be
 added
 to the API without breaking users who don't specifically ask for it.


We will continue to support the same API that we had at the point we
introduced microversioning, which can be represented as v1.1, and is
inferred when there is no version header in the request. This should be
completely compatible with any client built against stable/juno.

There was a change introduced during Kilo which may break some clients,
were they not to upgrade; that change is only exposed when the requested
version is = 1.2. Thus, older clients are not affected (but also do not
see new features).

In versions 1.3 - 1.6, we made several additions that we believed would not
break any client expecting v1.2, but incremented the API microversion to
indicate those changes. In this way, a newer client could discover what is
supported by a server it connects to.

I have no plans to remove support for v1.1 at this time. The language in
the spec lays out the approach we should take, were we ever to find the
need to remove support for v1.1. While I hope we never need to, describing
it was an important part of the process to arrive at how we implemented
microversions.

-Devananda
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] [RFC/FFE] Finishing state machine work for Kilo

2015-04-06 Thread Devananda van der Veen
Hi!

The situation you describe is the same that concerned me with regards to
stable/juno compatibility. As soon as the client library started passing a
version header by default, it exposed Kilo changes to all users. Anyone
testing from trunk would have experienced that when 99ab landed.

I just tagged release 0.5.0 of python-ironicclient which includes that
change. It therefor passes X-OpenStack-Ironic-API-Version == 1.6 (this is
your #2 below). 3rd party apps that pin to  0.5 release of
python-ironicclient will continue to get juno-esque states. I'll announce
this in a separate thread shortly.

As far as whether to default newly created nodes to AVAILABLE, MANAGEABLE,
or ENROLLED - yea, we didn't finish the state machine work this cycle - I'd
rather not introduce a change to the default state now, then change it
again in a few months, leading to further confusion.

As far as your #3, you raise a good point. Passing the version header by
default is a potentially incompatible change with prior versions of the
client. While the CLI and library API didn't change in an incompatible way,
they expose the client to new server behavior... perhaps this should be our
1.0 release of python-ironicclient? Or perhaps, since it's still in the 0.x
range, we should wait until we know the state machine work is done and
*then* tag a 1.0 release of the client?

Re: documenting this, besides release notes, emails, and changelogs - what
would you like to see? I'm happy to put the release notes for the client
into our developer docs, along with a page on upgrading from juno to kilo
that we need to write.

-Deva

On Fri, Apr 3, 2015 at 1:31 AM Dmitry Tantsur dtant...@redhat.com wrote:

 Hi all!

 Today I got an internal email, stating that new ironicclient brakes
 ironic-discoverd. Indeed, after rebase to the latest ironicclient git
 master, discoverd started receiving AVAILABLE state instead of None
 for newly enrolled nodes. It's not a valid state for introspection,
 valid are MANAGEABLE (discoverd stand-alone usage), INSPECTING
 (discoverd via Ironic driver) and None (Juno + discoverd stand-alone).
 Looks like despite introducing microversions we did manage to break
 3rdparty apps relying on states... Also we're in a bit weird situation
 where nodes appear ready for scheduling, despite us having a special
 state for managing nodes _before_ being ready for scheduling.

 I find the situation pretty confusing, and I'd like your comments on the
 following proposal to be implemented before RC1:

 1. add new micro-version 1.7. nodes created by API with this version
 will appear in state MANAGEABLE;
 2. make a client release with current API version set to 1.6 (thus
 excluding change #1 from users of this version);
 3. set current API version for ironicclient to 1.7 and release
 ironicclient version 2.0.0 to designate behavior changes;
 4. document the whole thingy properly

 #1 should be a small change, but it definitely requires FFE.
 Thoughts?

 Dmitry

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic][FFE] secure boot support in iLO drivers

2015-03-19 Thread Devananda van der Veen
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

All,

This feature [0] enables secure boot mode for hardware which supports UEFI.
Ironic added support for UEFI in Juno. This is an incremental improvement,
allowing those users to benefit more from their hardware's security features.

After this morning's final round of reviews to get features in, we agreed to
defer the pxe_ilo driver changes, as those are the highest risk component of
the secure boot blueprint, but grant a short extension for the other two
drivers (iscsi_ilo and agent_ilo) which do not require PXE booting.

The remaining changes are already either approved or very close to, and we're
confident this can be landed in the next couple days without impact outside of
those drivers.

As such, I have blocked the pxe_ilo patch [1], retargeted the blueprint to RC1,
and am granting an exception for those drivers. I'll re-review the status of
this on Monday, March 23rd.

[0] https://blueprints.launchpad.net/ironic/+spec/uefi-secure-boot

[1] https://review.openstack.org/#/c/159322/

- -Devananda
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlUK/woACgkQhFvuBniJg6demgCgxiSlIAPPSqNwUK8gGo2qOpxF
gsEAoM65a5bTlBlaPKOKfpcJsN67INnW
=bdk6
-END PGP SIGNATURE-

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic][FFE] secure boot support in iLO drivers

2015-03-19 Thread Devananda van der Veen
woops! thanks...

-Devananda

On Thu, Mar 19, 2015 at 10:04 AM, Ramakrishnan G
rameshg87.openst...@gmail.com wrote:

 Corrected [1]  is below (link for pxe_ilo patch review):

 [1] https://review.openstack.org/#/c/154808/

 On Thu, Mar 19, 2015 at 10:24 PM, Devananda van der Veen
 devananda@gmail.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 All,

 This feature [0] enables secure boot mode for hardware which supports
 UEFI.
 Ironic added support for UEFI in Juno. This is an incremental improvement,
 allowing those users to benefit more from their hardware's security
 features.

 After this morning's final round of reviews to get features in, we agreed
 to
 defer the pxe_ilo driver changes, as those are the highest risk component
 of
 the secure boot blueprint, but grant a short extension for the other two
 drivers (iscsi_ilo and agent_ilo) which do not require PXE booting.

 The remaining changes are already either approved or very close to, and
 we're
 confident this can be landed in the next couple days without impact
 outside of
 those drivers.

 As such, I have blocked the pxe_ilo patch [1], retargeted the blueprint to
 RC1,
 and am granting an exception for those drivers. I'll re-review the status
 of
 this on Monday, March 23rd.

 [0] https://blueprints.launchpad.net/ironic/+spec/uefi-secure-boot

 [1] https://review.openstack.org/#/c/159322/

 - -Devananda
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1

 iEYEARECAAYFAlUK/woACgkQhFvuBniJg6demgCgxiSlIAPPSqNwUK8gGo2qOpxF
 gsEAoM65a5bTlBlaPKOKfpcJsN67INnW
 =bdk6
 -END PGP SIGNATURE-

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] proposing rameshg87 to ironic-core

2015-03-12 Thread Devananda van der Veen
Without further ado, and since everyone (even though some haven't replied
here) has +1'd this, and since we could really use ramesh's +2's in the run
up to Kilo-3 and feature freeze, even without the customary waiting/voting
period being completely satisfied (after all, when we all agree, why wait a
week?), I'd like to officially welcome him to the core team.

-Devananda

On Tue, Mar 10, 2015 at 10:03 AM David Shrewsbury shrewsbury.d...@gmail.com
wrote:

 +1

 On Mar 9, 2015, at 6:03 PM, Devananda van der Veen 
 devananda@gmail.com wrote:

 Hi all,

 I'd like to propose adding Ramakrishnan (rameshg87) to ironic-core.

 He's been consistently providing good code reviews, and been in the top
 five active reviewers for the last 90 days and top 10 for the last 180
 days. Two cores have recently approached me to let me know that they, too,
 find his reviews valuable.

 Furthermore, Ramakrishnan has made significant code contributions to
 Ironic over the last year. While working primarily on the iLO driver, he
 has also done a lot of refactoring of the core code, touched on several
 other drivers, and maintains the proliantutils library on stackforge. All
 in all, I feel this demonstrates a good and growing knowledge of the
 codebase and architecture of our project, and feel he'd be a valuable
 member of the core team.

 Stats, for those that want them, are below the break.

 Best Regards,
 Devananda



 http://stackalytics.com/?release=allmodule=ironic-groupuser_id=rameshg87

 http://russellbryant.net/openstack-stats/ironic-reviewers-90.txt
 http://russellbryant.net/openstack-stats/ironic-reviewers-180.txt

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Avoiding regression in project governance

2015-03-10 Thread Devananda van der Veen
On Tue, Mar 10, 2015 at 12:12 PM Lauren Sell lau...@openstack.org wrote:

 Dissolving the integrated release without having a solid plan and
 replacement is difficult to communicate to people who depend on OpenStack.
 We’re struggling on that front.

 That said, I’m still optimistic about project structure reform and think
 it could be beneficial to the development community and users if it’s
 executed well. It gives us the opportunity to focus on a tighter core of
 services that are stable, reliable and scalable, while also recognizing
 more innovation that’s already happening in the community, beyond the
 existing integrated release. Coming out of the ops meetup in Philadelphia
 yesterday, a few things were clear:

 - Operators don’t want the wild west. They are nervous about dissolving
 the integrated release, because they want a strong filter and rules -
 dependency mapping, release timing, test coverage - around the most widely
 adopted projects. I’m not sure we’re giving them a lot of confidence.


We're not lowering the testing standards of existing projects... can you be
more clear as to what is creating this concern?


 - They also want some kind of bar or filter for community projects, to
 provide guidance beyond it’s in or out of the community. Tags can help with
 the nuances once they’re in the tent, but I think there’s some support for
 a bit higher bar overall.


What would that bar look like? If what we have in new-project-requirements
isn't enough, what changes are being suggested?


 - That said, several people expressed they did not want duplication to
 prevent a project from making it into the tent. They would like to have
 options beyond the core set of projects


Glad to hear that.


 - The layers concept came back to play. It was clear there was a distinct
 dropoff in operators running projects other than nova, keystone, glance,
 cinder, horizon and neutron


Not surprising


 - The operators community is keen to help define and apply some tags,
 especially those relevant to maturity and stability and general operability

 (I know several of you were at the ops meetup, so please jump in if I’ve
 missed or misrepresented some of the feedback. Notes from the session
 https://etherpad.openstack.org/p/PHL-ops-tags.)

 Based on feedback and conversations yesterday, I think it’s worth evolving
 the overall project criteria to add 1) a requirement for contributor
 diversity,


Joe's got a proposal up for that already.


 2) some criteria for maturity like documentation, test coverage and
 integration/dependency requirements,


I don't think that should be a requirement for entry -- after all, these
are still subjective judgements, subject to invalidation over time -- but I
would like to see metadata (ie, tags) that describe a projects' current
state, and can be updated by the community.


 and 3) make sure there are no trademark issues with the project name,
 since it will be referred to as an OpenStack project. I’m also unclear how
 we’re planning to refer to these projects, as “Foo, an OpenStack community
 project” but not “OpenStack Foo?


That's a good question



 For tags, I think defining a set of projects based on a broad reference
 architecture / use case like base compute” or “compute kernel” and “object
 storage” is critical. Those tags will imply the projects share common
 dependencies and are released together. If we categorize tags that can be
 applied, compute kernel” could be a higher level category and more
 prominent. Defining those initial tags should provide enough direction and
 confidence to start considering new projects.


I've started drafting some tags for layers or use cases -- I'm sure
they'll get expanded on. I'll post a link once I've uploaded to gerrit.

--
Devananda
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Avoiding regression in project governance

2015-03-10 Thread Devananda van der Veen
On Tue, Mar 10, 2015 at 12:00 PM Jeremy Stanley fu...@yuggoth.org wrote:

 On 2015-03-10 14:42:18 -0400 (-0400), Russell Bryant wrote:
 [...]
  As to specific tags, I refer back to this:
 
  http://governance.openstack.org/reference/incubation-
 integration-requirements.html
 
  We worked pretty hard to come up with useful things for projects
  to aim for. In fact, we considered it a minimum. Let's make sure
  we capture the things we still value, which I believe is most of
  it.
 [...]

 Coming from a horizontal resource and facilitation perspective, we
 previously had guidelines like these to help prioritize where effort
 is focused. I was hoping that most of the incubation requirements
 would become tags in some form so that support decisions could still
 be made based on them.


Many of those requirements were subjective (well tested, well documented,
etc) and had to be evaluated by the TC. Are these the sort of tags you're
referring to? If so, and if the TC delegated responsibility to manage the
application of those tags (say, QA team manages the 'well-tested' tag),
would that be sufficient?

If not, which ones do you mean?


 Otherwise I worry we're stuck relying on tags
 which merely declare the set of projects each horizontal team has
 chosen as a priority (in situations where there are ongoing demands
 on team members available time to help those specific projects).


These would be much more objective tags (eg, qa-supported' or
'infra-supported') ... but I see your point that this doesn't help inform
decisions that horizontal teams need to make, it merely reflects the ones
that have already been made.

Nothing says we can't have both sets of tags ... but so far, neither has
been proposed.

Yes, horizontal teams should provide the means for OpenStack
 projects to support themselves where possible, but some activities
 do not scale linearly and do necessitate hard support decisions.
 Guidance from the community as to where it's most effective to spend
 those limited resources is appreciated, and also increases the
 chances that in those situations the prioritized subset overlaps
 substantially between various limited resources (which provides a
 more consistent experience and helps set expectations).



This is in line with my view -- tags should not serve as governance in the
rule-setting sense, but as providing guidance to the community. Guidance as
to what sort of behavior is expected // accepted, and guidance as to the
status of each project's conformity to those expectations.

Wit this, we can, collectively, make more informed decisions, without being
expected to independently gather that information and come to the same
conclusions.

-Devananda
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] proposing rameshg87 to ironic-core

2015-03-09 Thread Devananda van der Veen
Hi all,

I'd like to propose adding Ramakrishnan (rameshg87) to ironic-core.

He's been consistently providing good code reviews, and been in the top
five active reviewers for the last 90 days and top 10 for the last 180
days. Two cores have recently approached me to let me know that they, too,
find his reviews valuable.

Furthermore, Ramakrishnan has made significant code contributions to Ironic
over the last year. While working primarily on the iLO driver, he has also
done a lot of refactoring of the core code, touched on several other
drivers, and maintains the proliantutils library on stackforge. All in all,
I feel this demonstrates a good and growing knowledge of the codebase and
architecture of our project, and feel he'd be a valuable member of the core
team.

Stats, for those that want them, are below the break.

Best Regards,
Devananda



http://stackalytics.com/?release=allmodule=ironic-groupuser_id=rameshg87

http://russellbryant.net/openstack-stats/ironic-reviewers-90.txt
http://russellbryant.net/openstack-stats/ironic-reviewers-180.txt
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Ironic] documenting our core review team policies

2015-03-09 Thread Devananda van der Veen
As the title says, I've gone ahead and written up our core review team
policies, and posted it here:

https://wiki.openstack.org/wiki/Ironic/CoreTeam

Why now, you may ask?

In the interests of full disclosure, we (the current ironic-core team) had
a private discussion recently about adding a new member to the team. It
went great, but someone felt that we should have the initial discussion
publicly, whereas I believe the initial round of hey, do you all trust
this human? discussions should continue to be held in private. This allows
for an actually candid discussion about a person, which is seldom possible
on a mailing list of this size. There's plenty of historical precedent for
this workflow, but it's unreasonable to assume everyone knows the history.
Particularly in light of the recent (and quite valid) everything should be
in the open discussions, writing stuff down and being open about our
policies is the right thing to do.

So.

In addition to what our policies and team structure are, I have also
documented, in said wiki, the primary reasoning that I'm going by in
continuing to have a private initial vetting for any core team changes
(basically what I said above). If anyone disagrees, let's discuss our
policies, and any changes to it, here in the open.

Cheers,
-Devananda
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Adding vendor drivers in Ironic

2015-03-07 Thread Devananda van der Veen
I know I'm arriving late to this thread, but I want to chime in with a
resounding yes! to this approach.

We've held to a fairly strict policy around maintaining compatibility
within the driver API since early on, and we'll continue to do that as
we add new interfaces (see the new ManagementInterface) or enhance
existing ones (like the new @clean_step decorator). I believe that, as
a hardware abstraction layer, we need to enable authors of
vendor-specific plugins/drivers to innovate out of tree -- whether
that is through a driver + library, such as stackforge/proliantutils,
or in a completely out of tree driver (I'm not aware of any open
source examples of this yet, but I have heard of private ones).

As far as the two (dis)advantages you list:

1) vendor drivers reviewed by the community

On the one hand, I believe there is some benefit from the broader
OpenStack community reviewing vendor code. Some... but not much.
Architecture and general principles (is it scalable? are you doing
something that operators will hate?) is valuable, but that's where it
stops. As you pointed out, it is impractical to expect open source
developers (or developers at other hardware companies) to have the
knowledge and expertise necessary (or hardware available to them) to
determine whether or not vendor-specific code will work with specific
vendor hardware.

As an interesting data point, we now have support for 8 different
types of hardware drivers in tree (10 if you count the ssh and vbox
drivers that we use in testing).

2) distros (not) packaging dependencies

Because we can not actually test every driver's dependencies in the
upstream gate (there's no hardware to exercise against), we do not
install those packages in devstack-gate, and so we do not list
driver's python dependencies in requirements.txt. We mock those
libraries and test the in-tree driver code, and rely on the library
and driver authors to ensure their fitness for use against real
hardware. So, yes, third-party-CI is essential to this -- and I wish
more vendors would step up and run CI on their drivers. But I
digress...

After talking with Thomas Goirand about debian packaging of Ironic, we
realized that there wasn't a clear list of our driver's dependencies.
So to provide distros with a single location for this information, we
are now maintaining a driver-requirements.txt file. It's fairly new
-- I added it about a month ago. See
https://github.com/openstack/ironic/blob/master/driver-requirements.txt


Thanks for writing up your experiences with this, Ramakrish. I believe
it is a great example to other driver authors -- and it would be great
to have a wiki or doc page describing the how's and why's of this
approach.

Cheers,
Devananda


On Fri, Feb 27, 2015 at 10:28 PM, Ramakrishnan G
rameshg87.openst...@gmail.com wrote:

 Hello All,

 This is about adding vendor drivers in Ironic.

 In Kilo, we have many vendor drivers getting added in Ironic which is a very
 good thing.  But something I noticed is that, most of these reviews have
 lots of hardware-specific code in them.  This is something most of the other
 Ironic folks cannot understand unless they go and read the hardware manuals
 of the vendor hardware about what is being done.  Otherwise we just need to
 blindly mark the file as reviewed.

 Now let me pitch in with our story about this.  We added a vendor driver for
 HP Proliant hardware (the *ilo drivers in Ironic).  Initially we proposed
 this same thing in Ironic that we will add all the hardware specific code in
 Ironic itself under the directory drivers/modules/ilo.  But few of the
 Ironic folks didn't agree on this (Devananda especially who is from my
 company :)). So we created a new module proliantutils, hosted in our own
 github and recently moved it to stackforge.  We gave a limited set of APIs
 for Ironic to use - like get_host_power_status(), set_host_power(),
 get_one_time_boot(), set_one_time_boot(), etc. (Entire list is here
 https://github.com/stackforge/proliantutils/blob/master/proliantutils/ilo/operations.py).

 We have only seen benefits in doing it.  Let me bring in some examples:

 1) We tried to add support for some lower version of servers.  We could do
 this without making any changes in Ironic (Review in proliantutils
 https://review.openstack.org/#/c/153945/)
 2) We are adding support for newer models of servers (earlier we use to talk
 to servers in protocol called RIBCL, newer servers we will use a protocol
 called RIS) - We could do this with just 14 lines of actual code change in
 Ironic (this was needed mainly because we didn't think we will have to use a
 new protocol itself when we started) -
 https://review.openstack.org/#/c/154403/

 Now talking about the advantages of putting hardware-specific code in
 Ironic:

 1) It's reviewed by Openstack community and tested:
 No. I doubt if I throw in 600 lines of new iLO specific code that is here
 

Re: [openstack-dev] [Ironic] Adding vendor drivers in Ironic

2015-03-07 Thread Devananda van der Veen
On Sun, Mar 1, 2015 at 8:45 AM, Clint Byrum cl...@fewbar.com wrote:
 Excerpts from Gary Kotton's message of 2015-03-01 02:32:37 -0800:
 Hi,
 I am just relaying pain-points that we encountered in neutron. As I have
 said below it makes the development process a lot quicker for people
 working on external drivers. I personally believe that it fragments the
 community and feel that the external drivers loose the community
 contributions and inputs.

 I think you're right that this does change the dynamic in the
 community. One way to lower the barrier is to go ahead and define the
 plugin API very strongly, but then delegate control of drivers in-tree
 to active maintainers, rather than in external repositories. If a driver
 falls below the line in terms of maintenance, then it can be deprecated.
 And if a maintainer feels strongly that they cannot include the driver
 with Ironic for whatever reason, the plugin API being strongly defined
 will allow them to do so.



++ on all counts.

Even with delegation of existing drivers to an active driver
maintainer(s), there is still a cost to the core review team: new
driver submissions have generally not come from existing core
reviewers. That could be mitigated if we were to encourage new drivers
to be developed and proven out of tree while the author becomes active
in the parent project, then when the core team feels ready, allow
the driver in-tree and delegate its maintenance.

-D

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


  1   2   3   4   >