----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222619 -----------------------------------------------------------
common/src/main/java/org/apache/atlas/repository/Constants.java Lines 212 (patched) <https://reviews.apache.org/r/73076/#comment311745> - TASK_TYPE_PROPERTY_KEY and TASK_TYPE are duplicates - Is TASK_TYPE_NAME needed, given the type name will be stored in TASK_TYPE repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java Lines 246 (patched) <https://reviews.apache.org/r/73076/#comment311750> Can taskManagement be null, given it is injected? Please review. repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 159 (patched) <https://reviews.apache.org/r/73076/#comment311751> Shouldn't taskManagement be injected, similar to GraphTransactionInterceptor? If yes: - remove getTaskManagement() - mark as final repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 1965 (patched) <https://reviews.apache.org/r/73076/#comment311752> task should be added only if propagateTags is true, right? Consider replacing this 'if' with: if (propagateTags && DEFERRED_ACTION_ENABLED) { repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2035 (patched) <https://reviews.apache.org/r/73076/#comment311753> LOG.warn("propagateClassification(entityGuid={}, classificationVertexId={}): entity vertex not found", entityGuid, classificationVertexId); repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2041 (patched) <https://reviews.apache.org/r/73076/#comment311754> - consider changing the log level to debug, to avoid noise in the log file - LOG.debug("propagateClassification(entityGuid={}, classificationVertexId={}): found no entities to propagate the classification", entityGuid, classificationVertexId); repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2050 (patched) <https://reviews.apache.org/r/73076/#comment311755> - classificationName will be null, if classificationVertex is null; so the log message may not be helpful. Consider updating to: LOG.warn("propagateClassification(entityGuid={}, classificationVertexId={}): classification vertex not found", entityGuid, classificationVertexId); - move #2049 - #2052, immediately after #2045? repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2063 (patched) <https://reviews.apache.org/r/73076/#comment311756> - LOG.error("propagateClassification(entityGuid={}, classificationVertexId={}): error while propagating classification", entityGuid, classificationVertexId, e); repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java Lines 40 (patched) <https://reviews.apache.org/r/73076/#comment311773> Is graph.commit() needed in #40, #56 and #72 - given entityGraphMapper methods called have @GraphTransaction annotation? repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java Lines 41 (patched) <https://reviews.apache.org/r/73076/#comment311774> Is it necessary to explicitly call GraphTransactionInterceptor.releaseLock()? Shouldn't graph commit (via @GraphTransaction) already handle releasing the lock? Also, when is the lock obtained? repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java Lines 52 (patched) <https://reviews.apache.org/r/73076/#comment311772> It looks like tasks are saved in the store only just before execution - in TaskConsumer.run(). That could be after a long time from this point. This increases the chances of loosing the tasks - consider Atlas shutdown/goes down/becomes passive. Consider saving all tasks to store before adding to taskList, to minimize the chances of loosing tasks. repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java Lines 91 (patched) <https://reviews.apache.org/r/73076/#comment311771> getCreateVertex() => getVertex()? - is it necessary to create vertex if not found? repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java Lines 92 (patched) <https://reviews.apache.org/r/73076/#comment311768> Task execution is skipped if task.getAdditionalInfo() is null; purpose of this condition is not clear. Consider adding a comment. repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java Lines 100 (patched) <https://reviews.apache.org/r/73076/#comment311766> When will such tasks (that exceeded retry count) be removed from store? repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java Lines 123 (patched) <https://reviews.apache.org/r/73076/#comment311767> - COMPLETE_WITH_ERRORS => FAILED? - when will failed tasks be removed from the store? repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java Lines 132 (patched) <https://reviews.apache.org/r/73076/#comment311770> Shouldn't commit be handled internally in following calls above? - #91: registry.getCreateVertex(task); - #108: registry.deleteComplete(task); - #113: registry.updateStatus(task); - #124: registry.updateStatus(task); Else, #91 could be committed along with graph commit for the task impl i.e. like tag-propagation. repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java Lines 138 (patched) <https://reviews.apache.org/r/73076/#comment311769> Consider logging following details: - number of successfully executed tasks - number of failed tasks It might be useful to print such stats periodically - say every 100 tasks, or every 5 minutes. repository/src/main/java/org/apache/atlas/tasks/TaskLogger.java Lines 25 (patched) <https://reviews.apache.org/r/73076/#comment311765> TaskLogger is used only with TaskExecutor. Consider moving this class inside TaskExecutor. Also, it adds very thin abstraction over Logger. It might be easier to replace this class with couple of methods (log() and error()) in TaskExecutor. repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java Lines 54 (patched) <https://reviews.apache.org/r/73076/#comment311762> In case of HA deployment, startInternal() should be deferred until instanceIsActive() is called. Please refer to NotificationHookConsumer. repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java Lines 73 (patched) <https://reviews.apache.org/r/73076/#comment311763> In case of HA deployment, instanceIsPassive() should stop processing queuedTasks, as another instance will start processing soon. repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java Lines 122 (patched) <https://reviews.apache.org/r/73076/#comment311764> Why only pending tasks? All tasks in the store are to be executed, right? repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java Lines 49 (patched) <https://reviews.apache.org/r/73076/#comment311761> For "@GraphTransaction" annotation to kick-in, the object should be created by dependency-injection. Consider adding @Component, and add @Inject to constructor. If this method is called from a transaction context, then remove @GraphTransaction annotation from here. Also, please review if getPendingTasks() method need @GraphTransaction annotation. repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java Lines 369 (patched) <https://reviews.apache.org/r/73076/#comment311760> To avoid timing related failures, can the test be made to wait until all pending tasks are executed, instead of a fixed pause time (of 2 seconds)? server-api/src/main/java/org/apache/atlas/RequestContext.java Lines 65 (patched) <https://reviews.apache.org/r/73076/#comment311759> TasksContext indirection doesn't seem to buy much. Consider using a List, with null as initial value: List<AtlasTask> queuedTasks = null; ... public void queueTask(AtlasTask task) { if (queuedTasks == null) { queuedTasks = new ArrayList(); } queuedTasks.add(task); } public List<AtlasTask> getQueueTasks() { return queuedTasks != null ? queuedTasks : Collections.emptyList(); } public void clearCache() { ... if (queuedTasks != null) { queuedTasks.clear(); } webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java Lines 748 (patched) <https://reviews.apache.org/r/73076/#comment311757> - it will be useful to retrieve tasks with specific guids - for example, from UI to show pending tasks on an entity. When the list is empty, return all pending tasks. If not return only the tasks asked for. Consider adding a parameter to this method: public List<AtlasTask> getAsyncTaskStatus(@QueryParam("guid") List<String> guids) - this list can be very large. Consider supporting pagination, in a subsequent patch webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java Lines 755 (patched) <https://reviews.apache.org/r/73076/#comment311758> Consider supporting multiple guids for delete as well: @DELETE @Path("/tasks") @Produces(Servlets.JSON_MEDIA_TYPE) public List<AtlasTask> deleteTask(@QueryParam("guid") List<String> guids) - Madhan Neethiraj On Feb. 19, 2021, 3:49 a.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > ----------------------------------------------------------- > > (Updated Feb. 19, 2021, 3:49 a.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > ------- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > **New Updates** > - Separate log file for tasks. > - TaskDef now has _startTime_ and _endTime_ to indicating start and end of > task. > - Refactored parameter passing during task creation. It now accepts Map > instead of Object array. Each task creates its own set of parameters. > - Reduced use of TASKS_ENABLE flag checks. > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca > distro/src/conf/atlas-log4j.xml 7df963eb2 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a > intg/src/main/java/org/apache/atlas/model/tasks/AtlasTask.java PRE-CREATION > intg/src/test/resources/atlas-application.properties 7e74d5107 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > 86b369fc9 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > d0ffb853f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2cfcc0bc1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 8208d11c2 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskLogger.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/TestModules.java 71b0a4a68 > > repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java > 84e9bfa04 > repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java > PRE-CREATION > server-api/src/main/java/org/apache/atlas/RequestContext.java 7de3536f4 > > server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java > bb7f8fcc4 > test-tools/src/main/resources/log4j.properties 4db0598ad > webapp/src/main/java/org/apache/atlas/BeanUtil.java ef5a741ee > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > e8fc111a6 > webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java > d6658fe29 > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > 77422b2a5 > > > Diff: https://reviews.apache.org/r/73076/diff/17/ > > > Testing > ------- > > **Unit tests** > New tests. > > **Manual tests** > In-progress. > > **Volume test** > Pending. > > **Pre-commit build** > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/281/ > > > Thanks, > > Ashutosh Mestry > >
