-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73076/
-----------------------------------------------------------
(Updated April 13, 2021, 4:28 p.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon
Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include:
- Modified implementation of EntityGraphRetriever.traverseImpactedVertices to
avoid using recursion. This was a result of testing done that caused
StackOverflow exception when testing with 5K propagations.
Bugs: ATLAS-3919
https://issues.apache.org/jira/browse/ATLAS-3919
Repository: atlas
Description
-------
**Approach**
Model: _TaskDef_
- Definition of task.
- Contains status of the task for querying.
_AbstractTask_
- Takes TaskDef as input and calls run.
- The concrete tasks has logic for doing real work.
_TaskFactory_
- Creates tasks given task type.
- Enumerates all the tasks that are supported.
_TaskManagement_
- Component/Service. Constructed after all existing components are constructed.
- Provides a simple interface for adding tasks.
- Processes tasks that are in pending state.
- Supports HA mode, where the tasks are not executed on passive instance.
_TaskManagement.Registry_
- CRUD operations with TaskDef vertex.
_TaskManagement.Processor_
- Provides mechanism for provides single queue for processing tasks in a way
that is non-blocking.
- Uses task type and the associated TaskFactory to construct concrete task.
Changes to _EntityGraphMapper_
There are 3 new methods added that use portion of existing classification
propagation logic and create a separate method.
New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper.
All calls that operate without tasks get channeled to the existing
implementation. The ones involving tasks are modified to use the new task
framework.
**Concurrent Changes to Classifications**
Via REST APIs and operations from web UI, it is possible to make changes to the
same entity. This situation will be more pronounced in the new setup.
The existing framework has
_GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides
ReentrantLock on an entity that is identified by the supplied GUID. This will
cause a lock on the entity and all the impacted vertices. This has potential
for adversely impacting scalability.
**REST APIs**
+-----------------+--------+----------+----------------------------------------------------------------------+
| api/admin/tasks | GET | guids | (optional) List of guids to be
fetched. |
| | | | If list of guids supplied is empty all
tasks |
| | | | that are in PENDING state are fetched.
If list |
| | | | guids is specified, tasks corresponding
to that guid are fetched. |
+-------------------+------+----------+----------------------------------------------------------------------+
|api/admin/task |DELETE| guids | List of guids to be deleted. No action
is taken if list of guids |
| | | | supplied is empty or a Task guid does
not exist. |
+-------------------+------+----------+----------------------------------------------------------------------+
**New Updates**
- Separate log file for tasks.
- TaskDef now has _startTime_ and _endTime_ to indicating start and end of task.
- Refactored parameter passing during task creation. It now accepts Map instead
of Object array. Each task creates its own set of parameters.
- Reduced use of TASKS_ENABLE flag checks.
- Added _AtlasEntity.getPendingTasks()_ that indicidate presence of tasks for a
given entity.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 771287f75
distro/src/conf/atlas-log4j.xml 7df963eb2
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a
intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 4d8c94894
intg/src/main/java/org/apache/atlas/model/tasks/AtlasTask.java PRE-CREATION
intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 27c7f7391
intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 4c7f8c681
intg/src/main/java/org/apache/atlas/type/Constants.java 7ee852086
intg/src/test/resources/atlas-application.properties 7e74d5107
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java
86b369fc9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
d0ffb853f
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerDelegate.java
1aaea64d7
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
7b2e2d371
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
c241e2371
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
bede9c374
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
b2be41c1b
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
8d744891c
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
2cfcc0bc1
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
8208d11c2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
fe8699dca
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/TestModules.java 71b0a4a68
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
9846d4310
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/tasks/BaseTaskFixture.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java
PRE-CREATION
server-api/src/main/java/org/apache/atlas/RequestContext.java 37d23c23b
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
bb7f8fcc4
test-tools/pom.xml ccad452ea
test-tools/src/main/resources/log4j.properties 4db0598ad
test-tools/src/main/resources/solr/core-template/solrconfig.xml 4d7e444bf
webapp/src/main/java/org/apache/atlas/BeanUtil.java ef5a741ee
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
e8fc111a6
webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java
d6658fe29
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java
77422b2a5
webapp/src/test/resources/atlas-application.properties 3847a3dfb
webapp/src/test/resources/template_metadata.csv 6b6359cd7
Diff: https://reviews.apache.org/r/73076/diff/28/
Changes: https://reviews.apache.org/r/73076/diff/27-28/
Testing (updated)
-------
**Unit tests**
New tests.
Additional unit tests covering end to end scenarios.
**Manual tests**
- Via web ui.
- REST APIs.
- Migration import.
**Volume test**
- Adding classification with propagations for lineage with 2000 elements.
- Verified with 5K linear propagation.
**Pre-commit build**
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/281/
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/450/
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/486/
Thanks,
Ashutosh Mestry
