----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222608 -----------------------------------------------------------
Here are comments for partial review; will complete the review in a day. Thanks! common/src/main/java/org/apache/atlas/repository/Constants.java Lines 182 (patched) <https://reviews.apache.org/r/73076/#comment311726> Instead of adding __pendingPropagations to classification vertices, it will be helpful to add __pendingTasks at entity-vertex. Consider adding following: - AtlasEntity.hasPendingTasks flag, so that UI can prompt the user of pending-tasks on the entity - a REST API to retrieve pending task IDs/details on a given entity ID of deferred-action should be added to __pendingTasks of the entity on which the classification was added/updated/deleted. In case of relationship create/update or tag-propagation update it should be added to entity from which the propagation needs to be recomputed. common/src/main/java/org/apache/atlas/repository/Constants.java Lines 211 (patched) <https://reviews.apache.org/r/73076/#comment311727> Consider using a task-specific guid field - like "__taskGuid", to avoid potential collision with entity guids. In fact, consider prefixing all task-vertex properties with "__task_" - like: - __task_guid - __task_type - __task_created_time - __task_updated_time distro/src/conf/atlas-log4j.xml Lines 64 (patched) <https://reviews.apache.org/r/73076/#comment311728> please add maxBackupIndex as well - similar to other appenders. intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java Lines 39 (patched) <https://reviews.apache.org/r/73076/#comment311729> TaskDef => AtlasTask - to be inline with AtlasEntity, AtlasClassification, AtlasGlossary, .. intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java Lines 198 (patched) <https://reviews.apache.org/r/73076/#comment311734> retryCount => attemptCount intg/src/main/java/org/apache/atlas/utils/AtlasJson.java Lines 176 (patched) <https://reviews.apache.org/r/73076/#comment311730> fromLinkedHashMap() - this method doesn't seem to be used. Please review and remove if unused. repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java Lines 241 (patched) <https://reviews.apache.org/r/73076/#comment311735> Is it not possible to get reference to TaksManagement with @Inject, since TaskManagement is a @Component? repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java Lines 337 (patched) <https://reviews.apache.org/r/73076/#comment311737> Only whitespace changes in GraphBackedSearchIndexer? Shouldn't new property-keys for AtlasTask be cread here? repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2786 (patched) <https://reviews.apache.org/r/73076/#comment311739> Couldn't taskManagement be @Injected, similar to bunch of other members in this class? repository/src/main/java/org/apache/atlas/util/BeanUtilRepo.java Lines 26 (patched) <https://reviews.apache.org/r/73076/#comment311732> This implementation exists in webapp/src/main/java/org/apache/atlas/BeanUtil.java as well. Consider moving BeanUtil class to repository, to avoid duplication. - Madhan Neethiraj On Feb. 17, 2021, 5:55 a.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > ----------------------------------------------------------- > > (Updated Feb. 17, 2021, 5:55 a.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > ------- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > **New Updates** > - Separate log file for tasks. > - TaskDef now has _startTime_ and _endTime_ to indicating start and end of > task. > - Refactored parameter passing during task creation. It now accepts Map > instead of Object array. Each task creates its own set of parameters. > - Reduced use of TASKS_ENABLE flag checks. > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca > distro/src/conf/atlas-log4j.xml 7df963eb2 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a > intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > 1edf4eeaf > intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION > intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 > intg/src/test/resources/atlas-application.properties 7e74d5107 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > 86b369fc9 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > d0ffb853f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > ce58e9aa4 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2cfcc0bc1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 8208d11c2 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskLogger.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/util/BeanUtilRepo.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java > 84e9bfa04 > repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java > PRE-CREATION > server-api/src/main/java/org/apache/atlas/RequestContext.java 7de3536f4 > > server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java > bb7f8fcc4 > test-tools/src/main/resources/log4j.properties 4db0598ad > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > e8fc111a6 > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > 77422b2a5 > > > Diff: https://reviews.apache.org/r/73076/diff/16/ > > > Testing > ------- > > **Unit tests** > New tests. > > **Manual tests** > In-progress. > > **Volume test** > Pending. > > **Pre-commit build** > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/281/ > > > Thanks, > > Ashutosh Mestry > >
