-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73076/#review222619
-----------------------------------------------------------




common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 212 (patched)
<https://reviews.apache.org/r/73076/#comment311745>

    - TASK_TYPE_PROPERTY_KEY and TASK_TYPE are duplicates
    - Is TASK_TYPE_NAME needed, given the type name will be stored in TASK_TYPE



repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java
Lines 246 (patched)
<https://reviews.apache.org/r/73076/#comment311750>

    Can taskManagement be null, given it is injected? Please review.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 159 (patched)
<https://reviews.apache.org/r/73076/#comment311751>

    Shouldn't taskManagement be injected, similar to 
GraphTransactionInterceptor? If yes:
     - remove getTaskManagement()
     - mark as final



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 1965 (patched)
<https://reviews.apache.org/r/73076/#comment311752>

    task should be added only if propagateTags is true, right? Consider 
replacing this 'if' with:
      if (propagateTags && DEFERRED_ACTION_ENABLED) {



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 2035 (patched)
<https://reviews.apache.org/r/73076/#comment311753>

    LOG.warn("propagateClassification(entityGuid={}, 
classificationVertexId={}): entity vertex not found", entityGuid, 
classificationVertexId);



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 2041 (patched)
<https://reviews.apache.org/r/73076/#comment311754>

    - consider changing the log level to debug, to avoid noise in the log file
    - LOG.debug("propagateClassification(entityGuid={}, 
classificationVertexId={}): found no entities to propagate the classification", 
entityGuid, classificationVertexId);



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 2050 (patched)
<https://reviews.apache.org/r/73076/#comment311755>

    - classificationName will be null, if classificationVertex is null; so the 
log message may not be helpful. Consider updating to:
      LOG.warn("propagateClassification(entityGuid={}, 
classificationVertexId={}): classification vertex not found", entityGuid, 
classificationVertexId);
    - move #2049 - #2052, immediately after #2045?



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 2063 (patched)
<https://reviews.apache.org/r/73076/#comment311756>

    - LOG.error("propagateClassification(entityGuid={}, 
classificationVertexId={}): error while propagating classification", 
entityGuid, classificationVertexId, e);



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
Lines 40 (patched)
<https://reviews.apache.org/r/73076/#comment311773>

    Is graph.commit() needed in #40, #56 and #72 - given entityGraphMapper 
methods called have @GraphTransaction annotation?



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
Lines 41 (patched)
<https://reviews.apache.org/r/73076/#comment311774>

    Is it necessary to explicitly call 
GraphTransactionInterceptor.releaseLock()? Shouldn't graph commit (via 
@GraphTransaction) already handle releasing the lock?
    
    Also, when is the lock obtained?



repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
Lines 52 (patched)
<https://reviews.apache.org/r/73076/#comment311772>

    It looks like tasks are saved in the store only just before execution - in 
TaskConsumer.run(). That could be after a long time from this point. This 
increases the chances of loosing the tasks - consider Atlas shutdown/goes 
down/becomes passive. Consider saving all tasks to store before adding to 
taskList, to minimize the chances of loosing tasks.



repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
Lines 91 (patched)
<https://reviews.apache.org/r/73076/#comment311771>

    getCreateVertex() => getVertex()?
      - is it necessary to create vertex if not found?



repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
Lines 92 (patched)
<https://reviews.apache.org/r/73076/#comment311768>

    Task execution is skipped if task.getAdditionalInfo() is null; purpose of 
this condition is not clear. Consider adding a comment.



repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
Lines 100 (patched)
<https://reviews.apache.org/r/73076/#comment311766>

    When will such tasks (that exceeded retry count) be removed from store?



repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
Lines 123 (patched)
<https://reviews.apache.org/r/73076/#comment311767>

    - COMPLETE_WITH_ERRORS => FAILED?
    - when will failed tasks be removed from the store?



repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
Lines 132 (patched)
<https://reviews.apache.org/r/73076/#comment311770>

    Shouldn't commit be handled internally in following calls above?
     - #91:  registry.getCreateVertex(task);
     - #108: registry.deleteComplete(task);
     - #113: registry.updateStatus(task);
     - #124: registry.updateStatus(task);
    
    Else, #91 could be committed along with graph commit for the task impl i.e. 
like tag-propagation.



repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
Lines 138 (patched)
<https://reviews.apache.org/r/73076/#comment311769>

    Consider logging following details:
     - number of successfully executed tasks
     - number of failed tasks
    
    It might be useful to print such stats periodically - say every 100 tasks, 
or every 5 minutes.



repository/src/main/java/org/apache/atlas/tasks/TaskLogger.java
Lines 25 (patched)
<https://reviews.apache.org/r/73076/#comment311765>

    TaskLogger is used only with TaskExecutor. Consider moving this class 
inside TaskExecutor. Also, it adds very thin abstraction over Logger. It might 
be easier to replace this class with couple of methods (log() and error()) in 
TaskExecutor.



repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
Lines 54 (patched)
<https://reviews.apache.org/r/73076/#comment311762>

    In case of HA deployment, startInternal() should be deferred until 
instanceIsActive() is called. Please refer to NotificationHookConsumer.



repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
Lines 73 (patched)
<https://reviews.apache.org/r/73076/#comment311763>

    In case of HA deployment, instanceIsPassive() should stop processing 
queuedTasks, as another instance will start processing soon.



repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
Lines 122 (patched)
<https://reviews.apache.org/r/73076/#comment311764>

    Why only pending tasks? All tasks in the store are to be executed, right?



repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
Lines 49 (patched)
<https://reviews.apache.org/r/73076/#comment311761>

    For "@GraphTransaction" annotation to kick-in, the object should be created 
by dependency-injection. Consider adding @Component, and add @Inject to 
constructor.
    
    If this method is called from a transaction context, then remove 
@GraphTransaction annotation from here.
    
    Also, please review if getPendingTasks() method need @GraphTransaction 
annotation.



repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
Lines 369 (patched)
<https://reviews.apache.org/r/73076/#comment311760>

    To avoid timing related failures, can the test be made to wait until all 
pending tasks are executed, instead of a fixed pause time (of 2 seconds)?



server-api/src/main/java/org/apache/atlas/RequestContext.java
Lines 65 (patched)
<https://reviews.apache.org/r/73076/#comment311759>

    TasksContext indirection doesn't seem to buy much. Consider using a List, 
with null as initial value:
    
      List<AtlasTask> queuedTasks = null;
      
      ...
      
      public void queueTask(AtlasTask task) {
        if (queuedTasks == null) {
          queuedTasks = new ArrayList();
        }
        
        queuedTasks.add(task);
      }
      
      public List<AtlasTask> getQueueTasks() {
        return queuedTasks != null ? queuedTasks : Collections.emptyList();
      }
      
      public void clearCache() {
        ...
        
        if (queuedTasks != null) {
          queuedTasks.clear();
        }



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 748 (patched)
<https://reviews.apache.org/r/73076/#comment311757>

    - it will be useful to retrieve tasks with specific guids - for example, 
from UI to show pending tasks on an entity. When the list is empty, return all 
pending tasks. If not return only the tasks asked for. Consider adding a 
parameter to this method:
      public List<AtlasTask> getAsyncTaskStatus(@QueryParam("guid") 
List<String> guids)
    
    - this list can be very large. Consider supporting pagination, in a 
subsequent patch



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 755 (patched)
<https://reviews.apache.org/r/73076/#comment311758>

    Consider supporting multiple guids for delete as well:
      @DELETE
      @Path("/tasks")
      @Produces(Servlets.JSON_MEDIA_TYPE)
      public List<AtlasTask> deleteTask(@QueryParam("guid") List<String> guids)


- Madhan Neethiraj


On Feb. 19, 2021, 3:49 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73076/
> -----------------------------------------------------------
> 
> (Updated Feb. 19, 2021, 3:49 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3919
>     https://issues.apache.org/jira/browse/ATLAS-3919
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> Model: _TaskDef_
> - Definition of task.
> - Contains status of the task for querying.
> 
> _AbstractTask_
> - Takes TaskDef as input and calls run.
> - The concrete tasks has logic for doing real work.
> 
> _TaskFactory_
> 
> - Creates tasks given task type.
> - Enumerates all the tasks that are supported.
> 
> _TaskManagement_
> 
> - Component/Service. Constructed after all existing components are 
> constructed.
> - Provides a simple interface for adding tasks.
> - Processes tasks that are in pending state.
> - Supports HA mode, where the tasks are not executed on passive instance.
> 
> _TaskManagement.Registry_
> 
> - CRUD operations with TaskDef vertex.
> 
> _TaskManagement.Processor_
> - Provides mechanism for provides single queue for processing tasks in a way 
> that is non-blocking.
> - Uses task type and the associated TaskFactory to construct concrete task.
> 
> Changes to _EntityGraphMapper_
> There are 3 new methods added that use portion of existing classification 
> propagation logic and create a separate method.
> 
> New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. 
> All calls that operate without tasks get channeled to the existing 
> implementation. The ones involving tasks are modified to use the new task 
> framework.
> 
> **Concurrent Changes to Classifications**
> Via REST APIs and operations from web UI, it is possible to make changes to 
> the same entity. This situation will be more pronounced in the new setup.
> 
> The existing framework has 
> _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides 
> ReentrantLock on an entity that is identified by the supplied GUID. This will 
> cause a lock on the entity and all the impacted vertices. This has potential 
> for adversely impacting scalability.
> 
> **New Updates**
> - Separate log file for tasks.
> - TaskDef now has _startTime_ and _endTime_ to indicating start and end of 
> task.
> - Refactored parameter passing during task creation. It now accepts Map 
> instead of Object array. Each task creates its own set of parameters.
> - Reduced use of TASKS_ENABLE flag checks.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca 
>   distro/src/conf/atlas-log4j.xml 7df963eb2 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a 
>   intg/src/main/java/org/apache/atlas/model/tasks/AtlasTask.java PRE-CREATION 
>   intg/src/test/resources/atlas-application.properties 7e74d5107 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 86b369fc9 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  d0ffb853f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  2cfcc0bc1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8208d11c2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskLogger.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/TestModules.java 71b0a4a68 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  84e9bfa04 
>   repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java 
> PRE-CREATION 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 7de3536f4 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  bb7f8fcc4 
>   test-tools/src/main/resources/log4j.properties 4db0598ad 
>   webapp/src/main/java/org/apache/atlas/BeanUtil.java ef5a741ee 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> e8fc111a6 
>   webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 
> d6658fe29 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 
> 77422b2a5 
> 
> 
> Diff: https://reviews.apache.org/r/73076/diff/17/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests.
> 
> **Manual tests**
> In-progress.
> 
> **Volume test**
> Pending.
> 
> **Pre-commit build**
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/281/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Reply via email to