Review Request 73182: ATLAS-4139 [Atlas: Glossary Term Bulk Import] When AdditionalAttributes is not provided in key:value format, the error message does not convey that

2021-02-10 Thread Mandar Ambawane 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73182/
---

Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-4139
https://issues.apache.org/jira/browse/ATLAS-4139


Repository: atlas


Description
---

This patch changes the error message


Diffs
-

  repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java 
2c84ec7 


Diff: https://reviews.apache.org/r/73182/diff/1/


Testing
---

Basic Testing is done


Thanks,

Mandar Ambawane

Re: Review Request 73182: ATLAS-4139 [Atlas: Glossary Term Bulk Import] When AdditionalAttributes is not provided in key:value format, the error message does not convey that

2021-02-10 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73182/#review222582
---



Can you please combine this with your other review. This change is really small 
to warrant a separate review or a commit.

- Ashutosh Mestry


On Feb. 10, 2021, 3:40 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73182/
> ---
> 
> (Updated Feb. 10, 2021, 3:40 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4139
> https://issues.apache.org/jira/browse/ATLAS-4139
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch changes the error message
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java 
> 2c84ec7 
> 
> 
> Diff: https://reviews.apache.org/r/73182/diff/1/
> 
> 
> Testing
> ---
> 
> Basic Testing is done
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Review Request 73181: ATLAS-4146 [Atlas: Glossary] On updating the related terms, UI allows the user to add the already added term

2021-02-10 Thread Mandar Ambawane 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73181/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-4146
https://issues.apache.org/jira/browse/ATLAS-4146


Repository: atlas


Description
---

This patch checks whether any duplicate Related Glossary Terms present in the 
payload sent by UI for Update operation.

This check is done for all the GlossaryTerm Relations.

If duplicate entry is found then Exception is thrown.


Diffs
-

  repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
82243e2 


Diff: https://reviews.apache.org/r/73181/diff/1/


Testing
---

Manual Testing is done


Thanks,

Mandar Ambawane

[jira] [Assigned] (ATLAS-4146) [Atlas: Glossary] On updating the related terms, UI allows the user to add the already added term

2021-02-10 Thread Mandar Ambawane (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandar Ambawane reassigned ATLAS-4146:
--

Assignee: Mandar Ambawane

> [Atlas: Glossary] On updating the related terms, UI allows the user to add 
> the already added term
> -
>
> Key: ATLAS-4146
> URL: https://issues.apache.org/jira/browse/ATLAS-4146
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Mandar Ambawane
>Priority: Major
> Attachments: Screen Recording 2021-02-05 at 6.50.31 PM.mov
>
>
> Steps to repro:
>  # Create a glossary(glossary_1) and a few terms (term_1, term_2, term_3, 
> term_4, term_5)
>  # Click on term_1 and add related terms "alsoSee"
>  # Choose term_5 of glossary_1
>  # Provide inputs and assign
>  # term_5 will be added as a related term
>  # Now, add one more related term and choose the same term_5 of glossary_1 
> [UI allows this]
>  # Provide new input and assign
>  # The assignment will be successful
> This makes the user think, that term_5 with new data is saved. But that is 
> not the case, only old data is present
> Expectation:
> Do not allow the user to assign the same term again.
> If allowed, update the data with the new data
> [^Screen Recording 2021-02-05 at 6.50.31 PM.mov]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4146) [Atlas: Glossary] On updating the related terms, UI allows the user to add the already added term

2021-02-10 Thread Mandar Ambawane (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandar Ambawane updated ATLAS-4146:
---
Component/s:  atlas-core

> [Atlas: Glossary] On updating the related terms, UI allows the user to add 
> the already added term
> -
>
> Key: ATLAS-4146
> URL: https://issues.apache.org/jira/browse/ATLAS-4146
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Mandar Ambawane
>Priority: Major
> Attachments: Screen Recording 2021-02-05 at 6.50.31 PM.mov
>
>
> Steps to repro:
>  # Create a glossary(glossary_1) and a few terms (term_1, term_2, term_3, 
> term_4, term_5)
>  # Click on term_1 and add related terms "alsoSee"
>  # Choose term_5 of glossary_1
>  # Provide inputs and assign
>  # term_5 will be added as a related term
>  # Now, add one more related term and choose the same term_5 of glossary_1 
> [UI allows this]
>  # Provide new input and assign
>  # The assignment will be successful
> This makes the user think, that term_5 with new data is saved. But that is 
> not the case, only old data is present
> Expectation:
> Do not allow the user to assign the same term again.
> If allowed, update the data with the new data
> [^Screen Recording 2021-02-05 at 6.50.31 PM.mov]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-4139) [Atlas: Glossary Term Bulk Import] When AdditionalAttributes is not provided in key:value format, the error message does not convey that

2021-02-10 Thread Mandar Ambawane (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandar Ambawane reassigned ATLAS-4139:
--

Assignee: Mandar Ambawane

> [Atlas: Glossary Term Bulk Import] When AdditionalAttributes is not provided 
> in key:value format, the error message does not convey that
> 
>
> Key: ATLAS-4139
> URL: https://issues.apache.org/jira/browse/ATLAS-4139
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Mandar Ambawane
>Priority: Major
> Attachments: Screenshot 2021-02-05 at 4.07.46 PM.png
>
>
> Consider the following input for bulk import term_glossary
> {code:java}
> GlossaryName, TermName, ShortDescription, LongDescription, Examples, 
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor, 
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso, 
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> glossary_1,term_1, "glossary_1 term_1 short description", "glossary_1 term_1 
> long description", "Examples", "AB", "glossary_1 term_1 Usage", "random_key"
>  {code}
> Here *AdditionalAttributes* is *random_key*
> When this is imported we get
> {code:java}
> errorCode: "ATLAS-500-00-001"errorMessage: "The uploaded file has not been 
> processed due to the following errors : ↵[↵The Data in the uploaded file is 
> incorrectly specified  : random_key]" {code}
> It mentions The Data in the uploaded file is incorrectly specified : 
> random_key
> It would be good to convey as "AdditionalAttributes has to a key:value pair" 
> or something similar



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73181: ATLAS-4146 [Atlas: Glossary] On updating the related terms, UI allows the user to add the already added term

2021-02-10 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73181/#review222583
---



Mandar, this check should be handled from UI side - not to allow saving an 
already assigned 'Term'.

- Sarath Subramanian


On Feb. 10, 2021, 5 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73181/
> ---
> 
> (Updated Feb. 10, 2021, 5 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4146
> https://issues.apache.org/jira/browse/ATLAS-4146
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch checks whether any duplicate Related Glossary Terms present in the 
> payload sent by UI for Update operation.
> 
> This check is done for all the GlossaryTerm Relations.
> 
> If duplicate entry is found then Exception is thrown.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
> 82243e2 
> 
> 
> Diff: https://reviews.apache.org/r/73181/diff/1/
> 
> 
> Testing
> ---
> 
> Manual Testing is done
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73181: ATLAS-4146 [Atlas: Glossary] On updating the related terms, UI allows the user to add the already added term

2021-02-10 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73181/#review222584
---




repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 1055 (patched)


why qualifiedName needs to be sent back? if 'duplicateTermHeaders' has any 
value then this confirms the terms are duplicated? Please review.


- Sarath Subramanian


On Feb. 10, 2021, 5 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73181/
> ---
> 
> (Updated Feb. 10, 2021, 5 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4146
> https://issues.apache.org/jira/browse/ATLAS-4146
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch checks whether any duplicate Related Glossary Terms present in the 
> payload sent by UI for Update operation.
> 
> This check is done for all the GlossaryTerm Relations.
> 
> If duplicate entry is found then Exception is thrown.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
> 82243e2 
> 
> 
> Diff: https://reviews.apache.org/r/73181/diff/1/
> 
> 
> Testing
> ---
> 
> Manual Testing is done
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

[jira] [Resolved] (ATLAS-2409) Disabled Integration Test: EntityV2JerseyResourceIT.java:325

2021-02-10 Thread Ashutosh Mestry (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry resolved ATLAS-2409.

Resolution: Fixed

> Disabled Integration Test: EntityV2JerseyResourceIT.java:325
> 
>
> Key: ATLAS-2409
> URL: https://issues.apache.org/jira/browse/ATLAS-2409
> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Assignee: Ashutosh Mestry
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The above test needs to be reviewed and should probably be re-enabled



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (ATLAS-2410) Disabled Integration Test: EntityV2JerseyResourceIT.java:330

2021-02-10 Thread Ashutosh Mestry (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry resolved ATLAS-2410.

Resolution: Fixed

> Disabled Integration Test: EntityV2JerseyResourceIT.java:330
> 
>
> Key: ATLAS-2410
> URL: https://issues.apache.org/jira/browse/ATLAS-2410
> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Assignee: Ashutosh Mestry
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The above test needs to be reviewed and should probably be re-enabled



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (ATLAS-2411) Disabled Integration Test: EntityV2JerseyResourceIT.java:335

2021-02-10 Thread Ashutosh Mestry (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry resolved ATLAS-2411.

Resolution: Fixed

> Disabled Integration Test: EntityV2JerseyResourceIT.java:335
> 
>
> Key: ATLAS-2411
> URL: https://issues.apache.org/jira/browse/ATLAS-2411
> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Assignee: Ashutosh Mestry
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The above test needs to be reviewed and should probably be re-enabled



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (ATLAS-4152) [Atlas: Spooling] Multiple entries are created for same table when the table is dropped while kafka is down

2021-02-10 Thread Dharshana M Krishnamoorthy (Jira) 
Dharshana M Krishnamoorthy created ATLAS-4152:
-

 Summary: [Atlas: Spooling] Multiple entries are created for same 
table when the table is dropped while kafka is down
 Key: ATLAS-4152
 URL: https://issues.apache.org/jira/browse/ATLAS-4152
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Reporter: Dharshana M Krishnamoorthy


A single table has multiple duplicate entries when the table is dropped while 
kafka is down (Spooling scenario)

Steps to re-produce:
 * Stop kafka broker
 * Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ... 
abc_table_n)
 * Make sure the data is spooled
 * Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ... 
xyz_table_n)
 * Wait for 5 mins for the tables to reflect in atlas
 * Fire a basic search with abc_* prefix and verify the tables are created
 * Collect the createdTime of all the table and verify the order

Here tables were created with prefix btwxb_table.

*btwxb_table_5, btwxb_table_10, btwxb_table_15 and btwxb_table_20* are dropped 
when kafka is down
 Each of those tables are having a total of 3 entries per table_name. All the 
tetails are same except the guid
{code:java}
{
"searchParameters":{
"includeSubTypes":true,
"excludeDeletedEntities":false,
"includeSubClassifications":true,
"typeName":"hive_table",
"limit":40,
"offset":0,
"includeClassificationAttributes":false,
"query":"btwxb*"
},
"queryText":"btwxb*",
"approximateCount":28,
"queryType":"BASIC",
"entities":[
{
"status":"DELETED",
"isIncomplete":false,
"guid":"9c348843-ebf0-4a0f-a909-4cdbf75ea39d",
"meanings":[

],
"labels":[

],
"typeName":"hive_table",
"meaningNames":[

],
"displayText":"btwxb_table_15",
"attributes":{
"owner":"hrt_qa",
"qualifiedName":"default.btwxb_table_15@cm",
"createTime":1612959528000,
"name":"btwxb_table_15"
},
"classificationNames":[

]
},
{
"status":"DELETED",
"isIncomplete":false,
"guid":"a25fbfc3-f6fb-4f9f-bce4-1da3f2de5921",
"meanings":[

],
"labels":[

],
"typeName":"hive_table",
"meaningNames":[

],
"displayText":"btwxb_table_5",
"attributes":{
"owner":"hrt_qa",
"qualifiedName":"default.btwxb_table_5@cm",
"createTime":1612959525000,
"name":"btwxb_table_5"
},
"classificationNames":[

]
},
{
"status":"ACTIVE",
"isIncomplete":false,
"guid":"bf6de3f6-48f2-4941-b5f2-a1ef3756ffa2",
"meanings":[

],
"labels":[

],
"typeName":"hive_table",
"meaningNames":[

],
"displayText":"btwxb_table_8",
"attributes":{
"owner":"hrt_qa",
"qualifiedName":"default.btwxb_table_8@cm",
"createTime":1612959527000,
"name":"btwxb_table_8"
},
"classificationNames":[

]
},
{
"status":"ACTIVE",
"isIncomplete":false,
"guid":"977adf75-9336-484f-91ed-7976817fb729",
"meanings":[

],
"labels":[

],
"typeName":"hive_table",
"meaningNames":[

],
"displayText":"btwxb_table_7",
"attributes":{
"owner":"hrt_qa",
"qualifiedName":"default.btwxb_table_7@cm",
"createTime":1612959527000,
"name":"btwxb_table_7"
},
"classificationNames":[

]
},
{
"status":"ACTIVE",
"isIncomplete":false,
"guid":"d43fb189-2dd5-4682-8cc1-d2b8566be7fb",
"meanings":[

],
"labels":[

],
"typeName":"hive_table",
"meaningNames":[

],
"displayText":"btwxb_table_9",
"attributes":{
"owner":"hrt_qa",
"qualifiedName":"default.btwxb_table_9@cm",
"createTime":1612959527000,

Re: Review Request 73076: Deferred Actions Implementation

2021-02-10 Thread Ashutosh Mestry via Review Board 


> On Feb. 5, 2021, 7:16 a.m., Madhan Neethiraj wrote:
> > intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java
> > Lines 65 (patched)
> > 
> >
> > Since TaskDef will be serialized/deserialized, consider using 
> > Map as type for additionalInfo, instead of Object.

This has been decored with @JsonIgnore. Hence will not get serialized.


> On Feb. 5, 2021, 7:16 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> > Lines 1926 (patched)
> > 
> >
> > List of entities to propagate must be computed while processing the 
> > entity creation/update; the result of this computation must be stored in 
> > the deferred-action and executed later. Computation of entities to 
> > propagate the classification must not be deferred. Please review and update.

Computing is time consuming in case of large datasets.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73076/#review222558
---


On Feb. 10, 2021, 6:40 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73076/
> ---
> 
> (Updated Feb. 10, 2021, 6:40 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3919
> https://issues.apache.org/jira/browse/ATLAS-3919
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Model: _TaskDef_
> - Definition of task.
> - Contains status of the task for querying.
> 
> _AbstractTask_
> - Takes TaskDef as input and calls run.
> - The concrete tasks has logic for doing real work.
> 
> _TaskFactory_
> 
> - Creates tasks given task type.
> - Enumerates all the tasks that are supported.
> 
> _TaskManagement_
> 
> - Component/Service. Constructed after all existing components are 
> constructed.
> - Provides a simple interface for adding tasks.
> - Processes tasks that are in pending state.
> - Supports HA mode, where the tasks are not executed on passive instance.
> 
> _TaskManagement.Registry_
> 
> - CRUD operations with TaskDef vertex.
> 
> _TaskManagement.Processor_
> - Provides mechanism for provides single queue for processing tasks in a way 
> that is non-blocking.
> - Uses task type and the associated TaskFactory to construct concrete task.
> 
> Changes to _EntityGraphMapper_
> There are 3 new methods added that use portion of existing classification 
> propagation logic and create a separate method.
> 
> New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. 
> All calls that operate without tasks get channeled to the existing 
> implementation. The ones involving tasks are modified to use the new task 
> framework.
> 
> **Concurrent Changes to Classifications**
> Via REST APIs and operations from web UI, it is possible to make changes to 
> the same entity. This situation will be more pronounced in the new setup.
> 
> The existing framework has 
> _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides 
> ReentrantLock on an entity that is identified by the supplied GUID. This will 
> cause a lock on the entity and all the impacted vertices. This has potential 
> for adversely impacting scalability.
> 
> **New Updates**
> - Separate log file for tasks.
> - TaskDef now has _startTime_ and _endTime_ to indicating start and end of 
> task.
> - Refactored parameter passing during task creation. It now accepts Map 
> instead of Object array. Each task creates its own set of parameters.
> - Reduced use of TASKS_ENABLE flag checks.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca 
>   distro/src/conf/atlas-log4j.xml 7df963eb2 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a 
>   intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 
>   intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 
>   intg/src/test/resources/atlas-application.properties 7e74d5107 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 86b369fc9 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  ce58e9aa4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  2cfcc0bc1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapperWithTasks.java
>  PRE-CREATION 
>

Re: Review Request 73076: Deferred Actions Implementation

2021-02-10 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73076/
---

(Updated Feb. 11, 2021, 6:37 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon 
Rodrigues, and Sarath Subramanian.


Changes
---

Updates include: 
- Refactoring. Removed class _EntityGraphMapperWithTasks_.
- Simplified individual task methods.
- Addressed review comments.


Bugs: ATLAS-3919
https://issues.apache.org/jira/browse/ATLAS-3919


Repository: atlas


Description
---

**Approach**
Model: _TaskDef_
- Definition of task.
- Contains status of the task for querying.

_AbstractTask_
- Takes TaskDef as input and calls run.
- The concrete tasks has logic for doing real work.

_TaskFactory_

- Creates tasks given task type.
- Enumerates all the tasks that are supported.

_TaskManagement_

- Component/Service. Constructed after all existing components are constructed.
- Provides a simple interface for adding tasks.
- Processes tasks that are in pending state.
- Supports HA mode, where the tasks are not executed on passive instance.

_TaskManagement.Registry_

- CRUD operations with TaskDef vertex.

_TaskManagement.Processor_
- Provides mechanism for provides single queue for processing tasks in a way 
that is non-blocking.
- Uses task type and the associated TaskFactory to construct concrete task.

Changes to _EntityGraphMapper_
There are 3 new methods added that use portion of existing classification 
propagation logic and create a separate method.

New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. 
All calls that operate without tasks get channeled to the existing 
implementation. The ones involving tasks are modified to use the new task 
framework.

**Concurrent Changes to Classifications**
Via REST APIs and operations from web UI, it is possible to make changes to the 
same entity. This situation will be more pronounced in the new setup.

The existing framework has 
_GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides 
ReentrantLock on an entity that is identified by the supplied GUID. This will 
cause a lock on the entity and all the impacted vertices. This has potential 
for adversely impacting scalability.

**New Updates**
- Separate log file for tasks.
- TaskDef now has _startTime_ and _endTime_ to indicating start and end of task.
- Refactored parameter passing during task creation. It now accepts Map instead 
of Object array. Each task creates its own set of parameters.
- Reduced use of TASKS_ENABLE flag checks.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca 
  distro/src/conf/atlas-log4j.xml 7df963eb2 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a 
  intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
1edf4eeaf 
  intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 
  intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 
  intg/src/test/resources/atlas-application.properties 7e74d5107 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
86b369fc9 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 ce58e9aa4 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 2cfcc0bc1 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
 PRE-CREATION 
  repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/tasks/TaskLogger.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/util/BeanUtilRepo.java PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
 84e9bfa04 
  repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java 
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java 
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java 
PRE-CREATION 
  server-api/src/main/java/org/apache/atlas/RequestContext.java 7de3536f4