Graham Wallis created ATLAS-2774:
------------------------------------
Summary: Options for hard and soft delete of instances
Key: ATLAS-2774
URL: https://issues.apache.org/jira/browse/ATLAS-2774
Project: Atlas
Issue Type: New Feature
Reporter: Graham Wallis
*Overview*
For Entities and Relationships, the following delete behaviors are desired.
Atlas can be configured to offer hard delete (only). Alternatively it can be
configured to offer soft-delete. Configuration is achieved by setting the
delete handler configuration property (see atlas-application.properties) to
either the SoftDeleteHandlerV1 or the HardDeleteHandlerV1. The default (for
when the property is not set) is to use the SoftDeleteHandlerV1.
The AtlasEntityStoreV2 deleteById() and deleteByIds() and methods and the
AtlasRelationshipStoreV2 deleteById() method are sensitive to the above
configuration. When the configuration is soft these methods will perform a soft
delete; when it is hard they will perform a hard delete.
In addition to the above methods, the AtlasEntityStoreV2 and
AtlasRelationshipStoreV2 offer a (new) purgeById() method that ALWAYS performs
a hard delete. This is true regardless of which delete handler has been
configured. When the configuration is hard, the purgeById() methods and
deleteById() methods are essentially equivalent. When the configuration is
soft, the purgeById() methods continue to provide a hard delete capability.
The AtlasEntityStoreV2 deleteById(), deleteByIds() and purgeById() methods will
delete the specified entities and any relationships to which they are
connected. The AtlasRelationshipStoreV2 deleteById() and purgeById() methods
will delete the specified relationship. Deletion of a relationship may cause an
upate to an entity to which the relationhsip is connected, if it changes the
propagation of classifications, for example.
In a hard delete or purge operation, an affected entity and relationship will
be removed from the graph and will not be returned in response to any future
queries. This is true regardless of whether the affected entity or relationship
has already been soft deleted or not. i.e. it's status could initially be
ACTIVE or DELETED. Following the operation it will not exist.
In a soft delete, the affected entities and relationships (that initially have
status ACTIVE) are updated to set the status to DELETED. These instances can
still be returned from queries and searches if the appropriate control is
selected (to include deleted instances). Without that control selected they
will not be returned.
*Use Case for soft delete*
Soft delete provides a 'safe' means of deleting instances from the repository
whilst preserving an audit trail and (if supported by the repository) enabling
a deleted instance to be restored (to ACTIVE status).
*Use Cases for hard delete*
One use case for hard delete is to permanently remove soft-deleted items some
period of time after they were soft deleted.
Another use case for hard delete is to is to correct (clean up after) a mistake
in which a potentially large batch of incomplete/corrupted/wrong metadata is
loaded and needs to be fully removed.
*Use Scenario for an OMAS user*
An OMAS provides a delete function that does NOT offer the user a choice of
hard/soft. The OMAS's delete function will invoke the relevant soft delete
method provided by the OMRS - deleteEntity or deleteRelationship. These are
both soft delete methods.
The Atlas OMRS Connector's deleteEntity/deleteRelationship methods will check
whether Atlas is configured for hard or soft delete. If Atlas is configured for
hard delete then soft-delete is not possible, and the method will throw a
FunctionNotSupported exception which is caught by the OMAS.
On catching this exception the OMAS author should automatically escalate the
soft delete to a hard delete by calling either of the mandatory OMRS
purgeEntity() or purgeRelationship() methods. In the Atlas OMRS Connector these
methods will invoke AtlasEntityStoreV2 purgeById() or AtlasRelationshipStoreV2
purgeById(). The effect of this is that if soft delete is not possible, a hard
delete is used instead.
An OMAS does NOT provide a purge capability to a non-privileged user, but a
suitably expert OMRS or Atlas repository administrator can explicitly issue a
purge call at the OMRS or Atlas interface, by calling purgeEntity() or
purgeRelationship() (or the relevant Atlas store's purgeById() method).
*Use Scenario for an Administrator*
As highlighted above, a suitably expert repository Administrator can invoke the
purgeEntity(), purgeRelationship() methods directly. They may need to do this
to permanently remove instances following an earlier soft-delete and the
timeout of a period of grace, or following a failed import or batch update.
In addition to the above direct use by a repository administrator of the
'purge' methods, an Administrator could alternatively use the deleteEntity(),
deleteRelationship() methods to perform soft deletes. The invoked method will
behave as described above for an OMAS - performing a soft-delete if possible
and throwing a FunctionNotSupported exception otherwise. In the latter case (no
support for soft-delete), the Administrator COULD then choose to issue a
purgeEntity(), purgeRelationship() call.
*Restore of an entity or relationship*
If an instance has been deleted using a soft-delete, the object still exists
but has a status of 'DELETED'. Whilst in the DELETED state, the object can be:
* optionally included/excluded from search results
* restored using the OMRS method for restoreEntity or restoreRelationship.
* purged from the repository using the purgeEntity() or purgeRelationship()
method (or the underlying Atlas store purgeById() method).
To restore an entity or relationship that has been soft-deleted, an OMAS user
or repository administrator can use the restoreEntity() or
restoreRealtionship() method of the OMRS connector.
In the case of the Atlas OMRS Connector, the restoreEntity() and
restoreRealtionship() methods invoke (new) Atlas store methods to update the
affected entities and relationships, leaving them in ACTIVE state and with
classifications that reflect the propagation options between all the resulting
active entities and relationships.
*Within the Atlas Repository*
The AtlasEntityStoreV2 and AtlasRelationshipStoreV2 provide additional methods
for purgeById() that perform a hard delete, regardless of the choice of
configured delete handler.
The stores also provide additional methods to enable restore of an entity or
relationship that has been soft-deleted.
*Within the Atlas OMRS Connector*
The Atlas OMRS Connector will interrogate the Atlas repository configuration to
determine whether it is configuraed for soft or hard deletes. The Atlas OMRS
Connector implementations of deleteEntity() and deleteRelationship() methods
will therefore have prior knowledge as to whether it is feasible to attempt to
perform a soft-delete. If feasible, the soft-delete will be issued to the
relevant Atlas store. If not feasible, the methods will throw a
FunctionNotSupported exception.
Also within the Atlas OMRS Connector, the mandatory OMRS purgeEntity(),
purgeRelationship() methods will invoke the Atlas stores' purgeById() methods
to perform permanent (hard) removal of an entity or relationship from the
repository. As outline above, these methods may be called either by an OMAS
that has attempted a soft-delete and been 'bounced' or called directly by an
Adminstrator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
