> On Jan. 10, 2021, 9:43 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> > Lines 2037 (patched)
> > <https://reviews.apache.org/r/73076/diff/8/?file=2243641#file2243641line2040>
> >
> >     consider adding tags to impacted vertices in batches (25 or 50) and 
> > committing the transaction, batch commits will save time rather than 
> > committing a single transaction with large changes.

I am planning to change this as part of a follow-up patch.


> On Jan. 10, 2021, 9:43 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapperWithTasks.java
> > Lines 191 (patched)
> > <https://reviews.apache.org/r/73076/diff/8/?file=2243642#file2243642line191>
> >
> >     In line 191 and 203, we pass classificationNames/classifications in 
> > addition to classifications vertexIds.
> >     
> >     Since classification vertexIds is available, we may not need to pass 
> > classificationNames or Classifications. These can be mapped from 
> > classificationVertex using entityGraphRetriever.

I had the implementaiton this way. I realized that this becomes problematic if 
classificaitons are added and removed. It better to fetch the IDs when we get 
to schedule the task.


> On Jan. 10, 2021, 9:43 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
> > Lines 83 (patched)
> > <https://reviews.apache.org/r/73076/diff/8/?file=2243644#file2243644line83>
> >
> >     this line always passes 0th classificationVertexId instead of i'th:
> >     
> >     classificationVertexIds.get(0) => classificationVertexIds.get(i)

This is simply a mechanism for providing arguments. Right now, only 1 argument 
is being passed, hence the index is always set to 0.


> On Jan. 10, 2021, 9:43 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java
> > Lines 103 (patched)
> > <https://reviews.apache.org/r/73076/diff/8/?file=2243647#file2243647line103>
> >
> >     taskDef status here is always set to 'COMPLETE' irrespective of the 
> > outcome of task.run()
> >     
> >     task.run() should return the status of the execution and it should be 
> > set to taskDef. Please review.

task.run can potentially throw an exception, in that case the status will be 
updated by the exception handlers. Status will be set to complete only on 
successful completion of the task. Hope this makes sense.


> On Jan. 10, 2021, 9:43 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
> > Lines 263 (patched)
> > <https://reviews.apache.org/r/73076/diff/8/?file=2243649#file2243649line263>
> >
> >     consider soft deleting the task vertex. It will be useful to check the 
> > status of the completed or failed tasks. We can purge the task vertices 
> > periodically (every hour, every day) or if some threshold is reached to 
> > avoid proliferations of these vertices.

In my testing I realized a lot of task data gets generated. This will cause a 
huge prolifiaration of task vertexs that are complete. Hence I am vertices of 
tasks that succeed. There is entry in log files to track the start and done 
status of tasks.


> On Jan. 10, 2021, 9:43 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
> > Lines 267 (patched)
> > <https://reviews.apache.org/r/73076/diff/8/?file=2243649#file2243649line267>
> >
> >     why update status of taskVertex and hard delete it?

The vertex holding the status will be available (not deleted) only if it fails.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73076/#review222428
-----------------------------------------------------------


On Jan. 12, 2021, 6:34 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73076/
> -----------------------------------------------------------
> 
> (Updated Jan. 12, 2021, 6:34 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3919
>     https://issues.apache.org/jira/browse/ATLAS-3919
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> Model: _TaskDef_
> - Definition of task.
> - Contains status of the task for querying.
> 
> _AbstractTask_
> - Takes TaskDef as input and calls run.
> - The concrete tasks has logic for doing real work.
> 
> _TaskFactory_
> 
> - Creates tasks given task type.
> - Enumerates all the tasks that are supported.
> 
> _TaskManagement_
> 
> - Component/Service. Constructed after all existing components are 
> constructed.
> - Provides a simple interface for adding tasks.
> - Processes tasks that are in pending state.
> - Supports HA mode, where the tasks are not executed on passive instance.
> 
> _TaskManagement.Registry_
> 
> - CRUD operations with TaskDef vertex.
> 
> _TaskManagement.Processor_
> - Provides mechanism for provides single queue for processing tasks in a way 
> that is non-blocking.
> - Uses task type and the associated TaskFactory to construct concrete task.
> 
> Changes to _EntityGraphMapper_
> There are 3 new methods added that use portion of existing classification 
> propagation logic and create a separate method.
> 
> New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. 
> All calls that operate without tasks get channeled to the existing 
> implementation. The ones involving tasks are modified to use the new task 
> framework.
> 
> **Concurrent Changes to Classifications**
> Via REST APIs and operations from web UI, it is possible to make changes to 
> the same entity. This situation will be more pronounced in the new setup.
> 
> The existing framework has 
> _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides 
> ReentrantLock on an entity that is identified by the supplied GUID. This will 
> cause a lock on the entity and all the impacted vertices. This has potential 
> for adversely impacting scalability.
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 
>   intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 
>   intg/src/test/resources/atlas-application.properties 7e74d5107 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 86b369fc9 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  244072289 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  2cfcc0bc1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapperWithTasks.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/util/BeanUtilRepo.java 
> PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  84e9bfa04 
>   repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java 
> PRE-CREATION 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 216ba08d3 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  bb7f8fcc4 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> b20b40474 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 
> 77422b2a5 
> 
> 
> Diff: https://reviews.apache.org/r/73076/diff/9/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests.
> 
> **Manual tests**
> In-progress.
> 
> **Volume test**
> Pending.
> 
> **Pre-commit build**
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/281/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Reply via email to