On 25/02/15 15:37, Joe Gordon wrote:


On Sat, Feb 21, 2015 at 5:03 AM, Tim Bell <tim.b...@cern.ch
<mailto:tim.b...@cern.ch>> wrote:


    A few inline comments and a general point

    How do we handle scenarios like volumes when we have a per-component
    janitor rather than a single co-ordinator ?

    To be clean,

    1. nova should shutdown the instance
    2. nova should then ask the volume to be detached
    3. cinder could then perform the 'project deletion' action as
    configured by the operator (such as shelve or backup)
    4. nova could then perform the 'project deletion' action as
    configured by the operator (such as VM delete or shelve)

    If we have both cinder and nova responding to a single message,
    cinder would do 3. Immediately and nova would be doing the shutdown
    which is likely to lead to a volume which could not be shelved cleanly.

    The problem I see with messages is that co-ordination of the actions
    may require ordering between the components.  The disable/enable
    cases would show this in a worse scenario.


You raise two good points.

* How to clean something up may be different for different clouds
* Some cleanup operations have to happen in a specific order

Not sure what the best way to address those two points is.  Perhaps the
best way forward is a openstack-specs spec to hash out these details.

For completeness, if nothing else, it should be noted that another option is for Keystone to refuse to delete the project until all resources within it have been removed by a user.

It's hard to know at this point which would be more painful. Both sound horrific in their own way :D

cheers,
Zane.


    Tim

     > -----Original Message-----
     > From: Ian Cordasco [mailto:ian.corda...@rackspace.com
    <mailto:ian.corda...@rackspace.com>]
     > Sent: 19 February 2015 17:49
     > To: OpenStack Development Mailing List (not for usage questions);
    Joe Gordon
     > Cc: openstack-operat...@lists.openstack.org
    <mailto:openstack-operat...@lists.openstack.org>
     > Subject: Re: [Openstack-operators] [openstack-dev] Resources
    owned by a
     > project/tenant are not cleaned up after that project is deleted
    from keystone
     >
     >
     >
     > On 2/2/15, 15:41, "Morgan Fainberg" <morgan.fainb...@gmail.com
    <mailto:morgan.fainb...@gmail.com>> wrote:
     >
     > >
     > >On February 2, 2015 at 1:31:14 PM, Joe Gordon
    (joe.gord...@gmail.com <mailto:joe.gord...@gmail.com>)
     > >wrote:
     > >
     > >
     > >
     > >On Mon, Feb 2, 2015 at 10:28 AM, Morgan Fainberg
     > ><morgan.fainb...@gmail.com <mailto:morgan.fainb...@gmail.com>>
    wrote:
     > >
     > >I think the simple answer is "yes". We (keystone) should emit
     > >notifications. And yes other projects should listen.
     > >
     > >The only thing really in discussion should be:
     > >
     > >1: soft delete or hard delete? Does the service mark it as
    orphaned, or
     > >just delete (leave this to nova, cinder, etc to discuss)
     > >
     > >2: how to cleanup when an event is missed (e.g rabbit bus goes
    out to
     > >lunch).
     > >
     > >
     > >
     > >
     > >
     > >
     > >I disagree slightly, I don't think projects should directly
    listen to
     > >the Keystone notifications I would rather have the API be something
     > >from a keystone owned library, say keystonemiddleware. So
    something like
     > this:
     > >
     > >
     > >from keystonemiddleware import janitor
     > >
     > >
     > >keystone_janitor = janitor.Janitor()
     > >keystone_janitor.register_callback(nova.tenant_cleanup)
     > >
     > >
     > >keystone_janitor.spawn_greenthread()
     > >
     > >
     > >That way each project doesn't have to include a lot of boilerplate
     > >code, and keystone can easily modify/improve/upgrade the
    notification
     > mechanism.
     > >
     > >


    I assume janitor functions can be used for

    - enable/disable project
    - enable/disable user

    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >Sure. I’d place this into an implementation detail of where that
    > >actually lives. I’d be fine with that being a part of Keystone
    > >Middleware Package (probably something separate from auth_token).
    > >
    > >
    > >—Morgan
    > >
    >
    > I think my only concern is what should other projects do and how much do 
we
    > want to allow operators to configure this? I can imagine it being 
preferable to
    > have safe (without losing much data) policies for this as a default and 
to allow
    > operators to configure more destructive policies as part of deploying 
certain
    > services.
    >

    Depending on the cloud, an operator could want different semantics
    for delete project's impact, between delete or 'shelve' style or
    maybe disable.

     >
     > >
     > >
     > >
     > >
     > >
     > >--Morgan
     > >
     > >Sent via mobile
     > >
     > >> On Feb 2, 2015, at 10:16, Matthew Treinish
    <mtrein...@kortar.org <mailto:mtrein...@kortar.org>> wrote:
     > >>
     > >>> On Mon, Feb 02, 2015 at 11:46:53AM -0600, Matt Riedemann wrote:
     > >>> This came up in the operators mailing list back in June [1] but
     > >>>given the  subject probably didn't get much attention.
     > >>>
     > >>> Basically there is a really old bug [2] from Grizzly that is
    still a
     > >>>problem  and affects multiple projects.  A tenant can be
    deleted in
     > >>>Keystone even  though other resources in other projects are under
     > >>>that project, and those  resources aren't cleaned up.
     > >>
     > >> I agree this probably can be a major pain point for users.
    We've had
     > >>to work around it  in tempest by creating things like:
     > >>
     > >>
     >
     >http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanu
     > >p_s
     > >ervice.py
     >
     ><http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/clean
     > >up_
     > >service.py>
     > >> and
     > >>
     >
     >http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanu
     > >p.p
     > >y
     >
     ><http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/cleanup
     > .
     > >py>
     > >>
     > >> to ensure we aren't dangling resources after a run. But, this
    doesn't
     > >>work in  all cases either. (like with tenant isolation enabled)
     > >>
     > >> I also know there is a stackforge project that is attempting
     > >>something similar
     > >> here:
     > >>
     > >> http://git.openstack.org/cgit/stackforge/ospurge/
     > >>
     > >> It would be much nicer if the burden for doing this was taken off
     > >>users and this  was just handled cleanly under the covers.
     > >>
     > >>>
     > >>> Keystone implemented event notifications back in Havana [3]
    but the
     > >>>other  projects aren't listening on them to know when a
    project has
     > >>>been deleted  and act accordingly.
     > >>>
     > >>> The bug has several people saying "we should talk about this
    at the
     > >>>summit"
     > >>> for several summits, but I can't find any discussion or summit
     > >>>sessions  related back to the bug.
     > >>>
     > >>> Given this is an operations and cross-project issue, I'd like to
     > >>>bring it up  again for the Vancouver summit if there is still
     > >>>interest (which I'm  assuming there is from operators).
     > >>
     > >> I'd definitely support having a cross-project session on this.
     > >>
     > >>>
     > >>> There is a blueprint specifically for the tenant deletion
    case but
     > >>> it's targeted at only Horizon [4].
     > >>>
     > >>> Is anyone still working on this? Is there sufficient interest
    in a
     > >>> cross-project session at the L summit?
     > >>>
     > >>> Thinking out loud, even if nova doesn't listen to events from
     > >>>keystone, we  could at least have a periodic task that looks for
     > >>>instances where the  tenant no longer exists in keystone and then
     > >>>take some action (log a  warning, shutdown/archive/, reap, etc).
     > >>>
     > >>> There is also a spec for L to transfer instance ownership [5]
    which
     > >>>could  maybe come into play, but I wouldn't depend on it.
     > >>>
     > >>> [1]
     >
     >http://lists.openstack.org/pipermail/openstack-operators/2014-June/004559.
     > >html
     >
     ><http://lists.openstack.org/pipermail/openstack-operators/2014-June/004
     > >559
     > >.html>
     > >>> [2] https://bugs.launchpad.net/nova/+bug/967832
     > >>> [3]
     > >https://blueprints.launchpad.net/keystone/+spec/notifications
     > ><https://blueprints.launchpad.net/keystone/+spec/notifications>
     > >>> [4]
     > >https://blueprints.launchpad.net/horizon/+spec/tenant-deletion
     > ><https://blueprints.launchpad.net/horizon/+spec/tenant-deletion>
     > >>> [5] https://review.openstack.org/#/c/105367/
     > >>
     > >> -Matt Treinish
     > >
     > >
     > >> _______________________________________________
     > >> OpenStack-operators mailing list
     > >> openstack-operat...@lists.openstack.org
    <mailto:openstack-operat...@lists.openstack.org>
     > >>
     >
     >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
     >
     ><http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator
     > >s>
     > >
     > >_______________________________________________
     > >OpenStack-operators mailing list
     > >openstack-operat...@lists.openstack.org
    <mailto:openstack-operat...@lists.openstack.org>
     >
     >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
     > >
     > >
     > >
     > >
     > >
     > >
     > >
     > >
     > >
     > >
     >
     > _______________________________________________
     > OpenStack-operators mailing list
     > openstack-operat...@lists.openstack.org
    <mailto:openstack-operat...@lists.openstack.org>
     >
    http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to