> On March 11, 2021, 8:52 p.m., Sarath Subramanian wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > > Lines 2817 (patched) > > <https://reviews.apache.org/r/73076/diff/23/?file=2247700#file2247700line2820> > > > > this will end up re-adding existing pending tasks to the list again. > > > > Consider adding only the new taskGuid: > > > > AtlasGraphUtilsV2.addEncodedProperty(entityVertex, > > PENDING_TASKS_PROPERTY_KEY, taskGuid);
Not sure how to do this, current implementation works fine. - Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222668 ----------------------------------------------------------- On March 11, 2021, 6:49 a.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > ----------------------------------------------------------- > > (Updated March 11, 2021, 6:49 a.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > ------- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > **REST APIs** > +-----------------+--------+----------+----------------------------------------------------------------------+ > | api/admin/tasks | GET | guids | (optional) List of guids to be > fetched. | > | | | | If list of guids supplied is empty > all tasks | > | | | | that are in PENDING state are > fetched. If list | > | | | | guids is specified, tasks > corresponding to that guid are fetched. | > +-------------------+------+----------+----------------------------------------------------------------------+ > |api/admin/task |DELETE| guids | List of guids to be deleted. No > action is taken if list of guids | > | | | | supplied is empty or a Task guid does > not exist. | > +-------------------+------+----------+----------------------------------------------------------------------+ > > **New Updates** > - Separate log file for tasks. > - TaskDef now has _startTime_ and _endTime_ to indicating start and end of > task. > - Refactored parameter passing during task creation. It now accepts Map > instead of Object array. Each task creates its own set of parameters. > - Reduced use of TASKS_ENABLE flag checks. > - Added _AtlasEntity.getPendingTasks()_ that indicidate presence of tasks for > a given entity. > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/repository/Constants.java 771287f75 > distro/src/conf/atlas-log4j.xml 7df963eb2 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > 4d8c94894 > intg/src/main/java/org/apache/atlas/model/tasks/AtlasTask.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 27c7f7391 > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 4c7f8c681 > intg/src/main/java/org/apache/atlas/type/Constants.java 7ee852086 > intg/src/test/resources/atlas-application.properties 7e74d5107 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > 86b369fc9 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > d0ffb853f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2cfcc0bc1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 8208d11c2 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java > fe8699dca > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/TestModules.java 71b0a4a68 > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 9846d4310 > > repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTestWithTasks.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/BaseTaskFixture.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java > PRE-CREATION > server-api/src/main/java/org/apache/atlas/RequestContext.java 37d23c23b > > server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java > bb7f8fcc4 > test-tools/pom.xml ccad452ea > test-tools/src/main/resources/log4j.properties 4db0598ad > test-tools/src/main/resources/solr/core-template/solrconfig.xml 4d7e444bf > webapp/src/main/java/org/apache/atlas/BeanUtil.java ef5a741ee > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > e8fc111a6 > webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java > d6658fe29 > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > 77422b2a5 > > > Diff: https://reviews.apache.org/r/73076/diff/23/ > > > Testing > ------- > > **Unit tests** > New tests. > Additional unit tests covering end to end scenarios. > > **Manual tests** > - Via web ui. > - REST APIs. > > **Volume test** > - Adding classification with propagations for lineage with 2000 elements. > > **Pre-commit build** > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/281/ > > > Thanks, > > Ashutosh Mestry > >
