Re: [openstack-dev] [nova][searchlight] When do instances get removed from Searchlight?

[McLellan, Steven]

Storing the data isn't a problem, but one issue we've had with situations like 
this where there should be a default filter (soft_deleted==false) that can be 
overridden is that it's quite hard within the context of the query parser to do 
it, because it involves examining an incoming query to see whether that term 
appears at all which can become difficult when looking at, for instance, 
query_string queries.

It probably would not be too difficult, however, to enforce that if a term 
("soft_deleted") appears as the term in a structured query we remove such a 
default filter - so under the 'normal' case, nova would filter out soft_deleted 
instances as well as other instances invisible due to RBAC rules, but if 
{soft_deleted: false} is present as a query term we'd allow them through.

Steve




On 3/6/17, 8:56 AM, "Balazs Gibizer"  wrote:

>
>
>On Mon, Mar 6, 2017 at 3:06 PM, Lei Zhang  
>wrote:
>> 
>> 
>> On Mon, Mar 6, 2017 at 1:21 AM, Matt Riedemann  
>> wrote:
>>> 
>>> So I'm wondering at what point instances stored in searchlight will 
>>> be removed. Maybe there is already an answer to this and the 
>>> searchlight team can just inform me. Otherwise we might need to 
>>> think about data retention policies and how long a deleted instances 
>>> will be stored in searchlight before it's removed. Again, I'm not 
>>> sure if nova would control this or if it's something searchlight 
>>> supports already.
>>> 
>> 
>> Hi,
>> 
>> Currently Searchlight doesn't capture soft delete notifications and 
>> simply remove instance from ES if real delete notification comes. If 
>> these two kind of notifications can be distinguished we could fix 
>> this issue by marking the document in the ES instead of removing it. 
>> And we also need to capture some extra notifications like 
>> restore(suppose it exists) .
>
>Yes, there is instance.restore notification [1] to match 
>instance.soft_delete.
>
>Cheers,
>gibi
>
>[1] https://review.openstack.org/#/c/331972/
>> 
>> About data retention policies, I'm not sure if something like ttl to 
>> define how long a deleted instance should be stored in searchlight is 
>> enough. I know Nova has cli to purge the database about deleted 
>> instances, if there are no notification emitted for these operations, 
>> it's impossible for searchlight to know when these delete instances 
>> are removed.
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][searchlight] When do instances get removed from Searchlight?

[Balazs Gibizer]

On Mon, Mar 6, 2017 at 3:06 PM, Lei Zhang  
wrote:



On Mon, Mar 6, 2017 at 1:21 AM, Matt Riedemann  
wrote:


So I'm wondering at what point instances stored in searchlight will 
be removed. Maybe there is already an answer to this and the 
searchlight team can just inform me. Otherwise we might need to 
think about data retention policies and how long a deleted instances 
will be stored in searchlight before it's removed. Again, I'm not 
sure if nova would control this or if it's something searchlight 
supports already.




Hi,

Currently Searchlight doesn't capture soft delete notifications and 
simply remove instance from ES if real delete notification comes. If 
these two kind of notifications can be distinguished we could fix 
this issue by marking the document in the ES instead of removing it. 
And we also need to capture some extra notifications like 
restore(suppose it exists) .


Yes, there is instance.restore notification [1] to match 
instance.soft_delete.


Cheers,
gibi

[1] https://review.openstack.org/#/c/331972/


About data retention policies, I'm not sure if something like ttl to 
define how long a deleted instance should be stored in searchlight is 
enough. I know Nova has cli to purge the database about deleted 
instances, if there are no notification emitted for these operations, 
it's impossible for searchlight to know when these delete instances 
are removed.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][searchlight] When do instances get removed from Searchlight?

[Balazs Gibizer]

On Mon, Mar 6, 2017 at 3:06 PM, Zhenyu Zheng 
 wrote:

Hi, Gibi

Yes, soft_delete.end notification didn't got handled in SL, and we 
should do it, but what Matt mean here is deferent, even you 'hard' 
delete an instance the record still exists in DB and user with 
certain role can list it using deleted=true, so we should also do it 
in SL


Yes, that is really different. If SL should be able to return hard 
deleted instances as well then the catching the soft_delete 
notification would not help.


Cheers,
gibi




On Monday, March 6, 2017, Balazs Gibizer 
 wrote:



On Mon, Mar 6, 2017 at 3:09 AM, Zhenyu Zheng 
 wrote:

Hi, Matt

AFAIK, searchlight did delete the record, it catch the 
instance.delete notification and perform the action:

http://git.openstack.org/cgit/openstack/searchlight/tree/searchlight/elasticsearch/plugins/nova/notification_handler.py#n100
-> 
http://git.openstack.org/cgit/openstack/searchlight/tree/searchlight/elasticsearch/plugins/nova/notification_handler.py#n307


Hi,

There is instance.soft_delete legacy notification [2] (delete_type 
== 'soft_delete'). This could be transformed to versioned 
notification along with [3]. So I guess there could be a way to 
distinguish between soft delete and real delete on searchlight side 
based on these notifications.


Cheers,
gibi

[2] 
https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1872

[3] https://review.openstack.org/#/c/410297/


I will double check with others from the SL team, and if it is the 
case, we will try to find a way to solve this ASAP.


Thanks,

Kevin Zheng

On Mon, Mar 6, 2017 at 1:21 AM, Matt Riedemann 
 wrote:
I've posted a spec [1] for nova's integration with searchlight for 
listing instance across multiple cells. One of the open questions 
I have on that is when/how do instances get removed from 
searchlight?


When an instance gets deleted via the compute API today, it's not 
really deleted from the database. It's considered "soft" deleted 
and you can still list (soft) deleted instances from the database 
via the compute API if you're an admin.


Nova will be sending instance.destroy notifications to searchlight 
but we don't really want the ES entry removed because we still 
have to support the compute API contract to list deleted 
instances. Granted, this is a pretty limp contract because there 
is no guarantee that you'll be able to list those deleted 
instances forever because once they get archived (moved to shadow 
tables in the nova database) or purged (hard delete), then they 
are gone from that API query path.


So I'm wondering at what point instances stored in searchlight 
will be removed. Maybe there is already an answer to this and the 
searchlight team can just inform me. Otherwise we might need to 
think about data retention policies and how long a deleted 
instances will be stored in searchlight before it's removed. 
Again, I'm not sure if nova would control this or if it's 
something searchlight supports already.


[1] https://review.openstack.org/#/c/441692/

--

Thanks,

Matt Riedemann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][searchlight] When do instances get removed from Searchlight?

[Lei Zhang]

On Mon, Mar 6, 2017 at 1:21 AM, Matt Riedemann  wrote:

>
> So I'm wondering at what point instances stored in searchlight will be
> removed. Maybe there is already an answer to this and the searchlight team
> can just inform me. Otherwise we might need to think about data retention
> policies and how long a deleted instances will be stored in searchlight
> before it's removed. Again, I'm not sure if nova would control this or if
> it's something searchlight supports already.
>
>
Hi,

Currently Searchlight doesn't capture soft delete notifications and simply
remove instance from ES if real delete notification comes. If these two
kind of notifications can be distinguished we could fix this issue by
marking the document in the ES instead of removing it. And we also need to
capture some extra notifications like restore(suppose it exists) .

About data retention policies, I'm not sure if something like ttl to define
how long a deleted instance should be stored in searchlight is enough. I
know Nova has cli to purge the database about deleted instances, if there
are no notification emitted for these operations, it's impossible for
searchlight to know when these delete instances are removed.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][searchlight] When do instances get removed from Searchlight?

[Zhenyu Zheng]

Hi, Gibi

Yes, soft_delete.end notification didn't got handled in SL, and we should
do it, but what Matt mean here is deferent, even you 'hard' delete an
instance the record still exists in DB and user with certain role can list
it using deleted=true, so we should also do it in SL

On Monday, March 6, 2017, Balazs Gibizer 
wrote:

>
>
> On Mon, Mar 6, 2017 at 3:09 AM, Zhenyu Zheng 
> wrote:
>
>> Hi, Matt
>>
>> AFAIK, searchlight did delete the record, it catch the instance.delete
>> notification and perform the action:
>> http://git.openstack.org/cgit/openstack/searchlight/tree/sea
>> rchlight/elasticsearch/plugins/nova/notification_handler.py#n100
>> -> http://git.openstack.org/cgit/openstack/searchlight/tree/sea
>> rchlight/elasticsearch/plugins/nova/notification_handler.py#n307
>>
>
> Hi,
>
> There is instance.soft_delete legacy notification [2] (delete_type ==
> 'soft_delete'). This could be transformed to versioned notification along
> with [3]. So I guess there could be a way to distinguish between soft
> delete and real delete on searchlight side based on these notifications.
>
> Cheers,
> gibi
>
> [2] https://github.com/openstack/nova/blob/master/nova/compute/a
> pi.py#L1872
> [3] https://review.openstack.org/#/c/410297/
>
>
> I will double check with others from the SL team, and if it is the case,
>> we will try to find a way to solve this ASAP.
>>
>> Thanks,
>>
>> Kevin Zheng
>>
>> On Mon, Mar 6, 2017 at 1:21 AM, Matt Riedemann 
>> wrote:
>>
>>> I've posted a spec [1] for nova's integration with searchlight for
>>> listing instance across multiple cells. One of the open questions I have on
>>> that is when/how do instances get removed from searchlight?
>>>
>>> When an instance gets deleted via the compute API today, it's not really
>>> deleted from the database. It's considered "soft" deleted and you can still
>>> list (soft) deleted instances from the database via the compute API if
>>> you're an admin.
>>>
>>> Nova will be sending instance.destroy notifications to searchlight but
>>> we don't really want the ES entry removed because we still have to support
>>> the compute API contract to list deleted instances. Granted, this is a
>>> pretty limp contract because there is no guarantee that you'll be able to
>>> list those deleted instances forever because once they get archived (moved
>>> to shadow tables in the nova database) or purged (hard delete), then they
>>> are gone from that API query path.
>>>
>>> So I'm wondering at what point instances stored in searchlight will be
>>> removed. Maybe there is already an answer to this and the searchlight team
>>> can just inform me. Otherwise we might need to think about data retention
>>> policies and how long a deleted instances will be stored in searchlight
>>> before it's removed. Again, I'm not sure if nova would control this or if
>>> it's something searchlight supports already.
>>>
>>> [1] https://review.openstack.org/#/c/441692/
>>>
>>> --
>>>
>>> Thanks,
>>>
>>> Matt Riedemann
>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][searchlight] When do instances get removed from Searchlight?

[Balazs Gibizer]

On Mon, Mar 6, 2017 at 3:09 AM, Zhenyu Zheng 
 wrote:

Hi, Matt

AFAIK, searchlight did delete the record, it catch the 
instance.delete notification and perform the action:

http://git.openstack.org/cgit/openstack/searchlight/tree/searchlight/elasticsearch/plugins/nova/notification_handler.py#n100
-> 
http://git.openstack.org/cgit/openstack/searchlight/tree/searchlight/elasticsearch/plugins/nova/notification_handler.py#n307


Hi,

There is instance.soft_delete legacy notification [2] (delete_type == 
'soft_delete'). This could be transformed to versioned notification 
along with [3]. So I guess there could be a way to distinguish between 
soft delete and real delete on searchlight side based on these 
notifications.


Cheers,
gibi

[2] 
https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1872

[3] https://review.openstack.org/#/c/410297/


I will double check with others from the SL team, and if it is the 
case, we will try to find a way to solve this ASAP.


Thanks,

Kevin Zheng

On Mon, Mar 6, 2017 at 1:21 AM, Matt Riedemann  
wrote:
I've posted a spec [1] for nova's integration with searchlight for 
listing instance across multiple cells. One of the open questions I 
have on that is when/how do instances get removed from searchlight?


When an instance gets deleted via the compute API today, it's not 
really deleted from the database. It's considered "soft" deleted and 
you can still list (soft) deleted instances from the database via 
the compute API if you're an admin.


Nova will be sending instance.destroy notifications to searchlight 
but we don't really want the ES entry removed because we still have 
to support the compute API contract to list deleted instances. 
Granted, this is a pretty limp contract because there is no 
guarantee that you'll be able to list those deleted instances 
forever because once they get archived (moved to shadow tables in 
the nova database) or purged (hard delete), then they are gone from 
that API query path.


So I'm wondering at what point instances stored in searchlight will 
be removed. Maybe there is already an answer to this and the 
searchlight team can just inform me. Otherwise we might need to 
think about data retention policies and how long a deleted instances 
will be stored in searchlight before it's removed. Again, I'm not 
sure if nova would control this or if it's something searchlight 
supports already.


[1] https://review.openstack.org/#/c/441692/

--

Thanks,

Matt Riedemann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][searchlight] When do instances get removed from Searchlight?

[Zhenyu Zheng]

Hi, Matt

AFAIK, searchlight did delete the record, it catch the instance.delete
notification and perform the action:
http://git.openstack.org/cgit/openstack/searchlight/tree/searchlight/elasticsearch/plugins/nova/notification_handler.py#n100
->
http://git.openstack.org/cgit/openstack/searchlight/tree/searchlight/elasticsearch/plugins/nova/notification_handler.py#n307

I will double check with others from the SL team, and if it is the case, we
will try to find a way to solve this ASAP.

Thanks,

Kevin Zheng

On Mon, Mar 6, 2017 at 1:21 AM, Matt Riedemann  wrote:

> I've posted a spec [1] for nova's integration with searchlight for listing
> instance across multiple cells. One of the open questions I have on that is
> when/how do instances get removed from searchlight?
>
> When an instance gets deleted via the compute API today, it's not really
> deleted from the database. It's considered "soft" deleted and you can still
> list (soft) deleted instances from the database via the compute API if
> you're an admin.
>
> Nova will be sending instance.destroy notifications to searchlight but we
> don't really want the ES entry removed because we still have to support the
> compute API contract to list deleted instances. Granted, this is a pretty
> limp contract because there is no guarantee that you'll be able to list
> those deleted instances forever because once they get archived (moved to
> shadow tables in the nova database) or purged (hard delete), then they are
> gone from that API query path.
>
> So I'm wondering at what point instances stored in searchlight will be
> removed. Maybe there is already an answer to this and the searchlight team
> can just inform me. Otherwise we might need to think about data retention
> policies and how long a deleted instances will be stored in searchlight
> before it's removed. Again, I'm not sure if nova would control this or if
> it's something searchlight supports already.
>
> [1] https://review.openstack.org/#/c/441692/
>
> --
>
> Thanks,
>
> Matt Riedemann
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev