-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73076/#review222608
-----------------------------------------------------------



Here are comments for partial review; will complete the review in a day. Thanks!


common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 182 (patched)
<https://reviews.apache.org/r/73076/#comment311726>

    Instead of adding __pendingPropagations to classification vertices, it will 
be helpful to add __pendingTasks at entity-vertex.
    
    Consider adding following:
      - AtlasEntity.hasPendingTasks flag, so that UI can prompt the user of 
pending-tasks on the entity
      - a REST API to retrieve pending task IDs/details on a given entity
    
    ID of deferred-action should be added to __pendingTasks of the entity on 
which the classification was added/updated/deleted. In case of relationship 
create/update or tag-propagation update it should be added to entity from which 
the propagation needs to be recomputed.



common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 211 (patched)
<https://reviews.apache.org/r/73076/#comment311727>

    Consider using a task-specific guid field - like "__taskGuid", to avoid 
potential collision with entity guids.
    
    In fact, consider prefixing all task-vertex properties with "__task_" - 
like:
      - __task_guid
      - __task_type
      - __task_created_time
      - __task_updated_time



distro/src/conf/atlas-log4j.xml
Lines 64 (patched)
<https://reviews.apache.org/r/73076/#comment311728>

    please add maxBackupIndex as well - similar to other appenders.



intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java
Lines 39 (patched)
<https://reviews.apache.org/r/73076/#comment311729>

    TaskDef => AtlasTask
     - to be inline with AtlasEntity, AtlasClassification, AtlasGlossary, ..



intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java
Lines 198 (patched)
<https://reviews.apache.org/r/73076/#comment311734>

    retryCount => attemptCount



intg/src/main/java/org/apache/atlas/utils/AtlasJson.java
Lines 176 (patched)
<https://reviews.apache.org/r/73076/#comment311730>

    fromLinkedHashMap() - this method doesn't seem to be used. Please review 
and remove if unused.



repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java
Lines 241 (patched)
<https://reviews.apache.org/r/73076/#comment311735>

    Is it not possible to get reference to TaksManagement with @Inject, since 
TaskManagement is a @Component?



repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
Lines 337 (patched)
<https://reviews.apache.org/r/73076/#comment311737>

    Only whitespace changes in GraphBackedSearchIndexer? Shouldn't new 
property-keys for AtlasTask be cread here?



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 2786 (patched)
<https://reviews.apache.org/r/73076/#comment311739>

    Couldn't taskManagement be @Injected, similar to bunch of other members in 
this class?



repository/src/main/java/org/apache/atlas/util/BeanUtilRepo.java
Lines 26 (patched)
<https://reviews.apache.org/r/73076/#comment311732>

    This implementation exists in 
webapp/src/main/java/org/apache/atlas/BeanUtil.java as well. Consider moving 
BeanUtil class to repository, to avoid duplication.


- Madhan Neethiraj


On Feb. 17, 2021, 5:55 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73076/
> -----------------------------------------------------------
> 
> (Updated Feb. 17, 2021, 5:55 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3919
>     https://issues.apache.org/jira/browse/ATLAS-3919
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> Model: _TaskDef_
> - Definition of task.
> - Contains status of the task for querying.
> 
> _AbstractTask_
> - Takes TaskDef as input and calls run.
> - The concrete tasks has logic for doing real work.
> 
> _TaskFactory_
> 
> - Creates tasks given task type.
> - Enumerates all the tasks that are supported.
> 
> _TaskManagement_
> 
> - Component/Service. Constructed after all existing components are 
> constructed.
> - Provides a simple interface for adding tasks.
> - Processes tasks that are in pending state.
> - Supports HA mode, where the tasks are not executed on passive instance.
> 
> _TaskManagement.Registry_
> 
> - CRUD operations with TaskDef vertex.
> 
> _TaskManagement.Processor_
> - Provides mechanism for provides single queue for processing tasks in a way 
> that is non-blocking.
> - Uses task type and the associated TaskFactory to construct concrete task.
> 
> Changes to _EntityGraphMapper_
> There are 3 new methods added that use portion of existing classification 
> propagation logic and create a separate method.
> 
> New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. 
> All calls that operate without tasks get channeled to the existing 
> implementation. The ones involving tasks are modified to use the new task 
> framework.
> 
> **Concurrent Changes to Classifications**
> Via REST APIs and operations from web UI, it is possible to make changes to 
> the same entity. This situation will be more pronounced in the new setup.
> 
> The existing framework has 
> _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides 
> ReentrantLock on an entity that is identified by the supplied GUID. This will 
> cause a lock on the entity and all the impacted vertices. This has potential 
> for adversely impacting scalability.
> 
> **New Updates**
> - Separate log file for tasks.
> - TaskDef now has _startTime_ and _endTime_ to indicating start and end of 
> task.
> - Refactored parameter passing during task creation. It now accepts Map 
> instead of Object array. Each task creates its own set of parameters.
> - Reduced use of TASKS_ENABLE flag checks.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca 
>   distro/src/conf/atlas-log4j.xml 7df963eb2 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> 1edf4eeaf 
>   intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 
>   intg/src/test/resources/atlas-application.properties 7e74d5107 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 86b369fc9 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  d0ffb853f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  ce58e9aa4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  2cfcc0bc1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8208d11c2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskLogger.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/util/BeanUtilRepo.java 
> PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  84e9bfa04 
>   repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java 
> PRE-CREATION 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 7de3536f4 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  bb7f8fcc4 
>   test-tools/src/main/resources/log4j.properties 4db0598ad 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> e8fc111a6 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 
> 77422b2a5 
> 
> 
> Diff: https://reviews.apache.org/r/73076/diff/16/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests.
> 
> **Manual tests**
> In-progress.
> 
> **Volume test**
> Pending.
> 
> **Pre-commit build**
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/281/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Reply via email to