Review Request 69816: ATLAS-3035: updated entity-get/delete to retrieve/delete soft-ref entities

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69816/
---

Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, 
Kapildeo Nayak, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3035
https://issues.apache.org/jira/browse/ATLAS-3035


Repository: atlas


Description
---

Updated entity-get/delete to retrieve/delete soft-ref entities for owne-ref 
attributes


Diffs
-

  intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 27409c742 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
45bf8dca4 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 c57e30ab5 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 98734cc1d 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 912bc3ecd 


Diff: https://reviews.apache.org/r/69816/diff/1/


Testing
---

- verified that entity-get populates extInfo with details of owned-ref soft-ref 
entities
- verified that entity-delete marks owned-ref soft-ref entities as deleted


Thanks,

Madhan Neethiraj

[jira] [Updated] (ATLAS-3035) Entity retrieval to include soft-referenced entities in extInfo

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3035:

Attachment: ATLAS-3035-branch-1.0.patch

> Entity retrieval to include soft-referenced entities in extInfo
> ---
>
> Key: ATLAS-3035
> URL: https://issues.apache.org/jira/browse/ATLAS-3035
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: ATLAS-3035-branch-1.0.patch, ATLAS-3035.patch
>
>
> ATLAS-2813 introduced the notion of soft-reference for attributes that refer 
> to other entities. This should only change the storage details of the 
> relationship, but shouldn't introduce any change in the data exchanged via 
> REST APIs. However, the return from entity-get operations do not include 
> soft-referenced entities in extInfo. This should be updated such that the 
> return value would populate extInfo - just as in case of regular 
> object-references.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69822: ATLAS-3033 - ATLAS Hive hook fails for CTAS query involving UDFs

[Nixon Rodrigues]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69822/
---

Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
Subramanian.


Bugs: ATLAS-3033
https://issues.apache.org/jira/browse/ATLAS-3033


Repository: atlas


Description
---

ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
with the exception as below

org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
_dummy_table.


FIX :-  Added fix to skip hive _dummy_table temp table while getting Table 
object details from hiveContext during lineage input creation.


Diffs
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 249f48b5f 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
4a6b41786 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 5c52cf442 


Diff: https://reviews.apache.org/r/69822/diff/1/


Testing
---

Tested by creating hive table with CTAS which creates temp table.


Thanks,

Nixon Rodrigues

[jira] [Updated] (ATLAS-3015) Tool: Associate Classifications to Entities based on QualifiedName

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-3015:
---
Attachment: ATLAS-3015-Classification-Updater-tool-master.patch

> Tool: Associate Classifications to Entities based on QualifiedName
> --
>
> Key: ATLAS-3015
> URL: https://issues.apache.org/jira/browse/ATLAS-3015
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: 0001-ATLAS-3015.patch, 
> ATLAS-3015-Classification-Updater-tool-master.patch
>
>
> *Background*
> Situation: A setup where Atlas classifications associated with entities the 
> data is subsequently erased and all data is lost. Entities are then 
> re-created using _import-hive.sh_. With this approach, the entities are 
> created within Atlas but there is no easy way to associate classifications to 
> these entities.
> *Solution and Approach Guidance*
>  # Before erasing Atlas database, create file with classifications and 
> entity-classification mapping.
>  # Erase the database.
>  # Create entities using appropriate means, say _import-hive.sh_.
>  # Read the file from step 1 and update Atlas.
> The tool should:
>  * Use _AtlasClient_ REST APIs.
>  * External and can be executed against a known Atlas instance.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69816: ATLAS-3035: updated entity-get/delete to retrieve/delete soft-ref entities

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69816/#review212248
---


Fix it, then Ship it!




Ship It!


repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
Lines 759 (patched)


consider reusing existing method -> AtlasGraphUtilsV2.findByGuid(guid) to 
retrieve vertex using guid.


- Sarath Subramanian


On Jan. 23, 2019, 1 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69816/
> ---
> 
> (Updated Jan. 23, 2019, 1 a.m.)
> 
> 
> Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, 
> Kapildeo Nayak, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3035
> https://issues.apache.org/jira/browse/ATLAS-3035
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Updated entity-get/delete to retrieve/delete soft-ref entities for owne-ref 
> attributes
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 27409c742 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 45bf8dca4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  c57e30ab5 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  98734cc1d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  912bc3ecd 
> 
> 
> Diff: https://reviews.apache.org/r/69816/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that entity-get populates extInfo with details of owned-ref 
> soft-ref entities
> - verified that entity-delete marks owned-ref soft-ref entities as deleted
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

[jira] [Updated] (ATLAS-3015) Tool: Associate Classifications to Entities based on QualifiedName

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-3015:
---
Attachment: (was: ATLAS-3015-Classification-Updater-tool-master.patch)

> Tool: Associate Classifications to Entities based on QualifiedName
> --
>
> Key: ATLAS-3015
> URL: https://issues.apache.org/jira/browse/ATLAS-3015
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: 0001-ATLAS-3015.patch, 
> ATLAS-3015-Classification-Updater-tool-master.patch
>
>
> *Background*
> Situation: A setup where Atlas classifications associated with entities the 
> data is subsequently erased and all data is lost. Entities are then 
> re-created using _import-hive.sh_. With this approach, the entities are 
> created within Atlas but there is no easy way to associate classifications to 
> these entities.
> *Solution and Approach Guidance*
>  # Before erasing Atlas database, create file with classifications and 
> entity-classification mapping.
>  # Erase the database.
>  # Create entities using appropriate means, say _import-hive.sh_.
>  # Read the file from step 1 and update Atlas.
> The tool should:
>  * Use _AtlasClient_ REST APIs.
>  * External and can be executed against a known Atlas instance.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69789: Tool: Classification Updater

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69789/
---

(Updated Jan. 23, 2019, 7:14 p.m.)


Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Bugs: ATLAS-3015
https://issues.apache.org/jira/browse/ATLAS-3015


Repository: atlas


Description
---

**Approach**
- Uses the new _EntityREST_ APIs to retrieve entity-classification association.
- File containing classification definitions is generated.
- During the prepare stage, output of new REST API (_bulk/header_) is written 
to another file.
- During the update stage, the output from _prepare_ stage is processed and 
associated Atlas server is updated.

**Usage**
Usage: classification-updater.sh [-s ] [-f ] [-d ]
step: Specify which step to execute:
   prepare: prepare classifications and associated entities.
   update: update classifications and entities.
dir: [optional] Directory where read/write will happen.
   If not specified, current directory will be used.
from: [mandatory for 'prepare' step, optional for 'update' step] Date, in 
-MM-DD format, from where audits need to be read.
   If not specified, current directory will be used.


Diffs (updated)
-

  tools/classification-updater/pom.xml PRE-CREATION 
  
tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
 PRE-CREATION 
  tools/classification-updater/src/main/resources/atlas-log4j.xml PRE-CREATION 
  tools/classification-updater/src/main/resources/update-classifications.sh 
PRE-CREATION 


Diff: https://reviews.apache.org/r/69789/diff/3/

Changes: https://reviews.apache.org/r/69789/diff/2-3/


Testing
---

**Functional**
- Tool executed in environments with various authentication mechanisms.


**Volume**
- Tool was executed in environment with large data.


Thanks,

Ashutosh Mestry

[jira] [Commented] (ATLAS-3034) Perf enhancement to avoid unnecessary lookup when creating new relationships

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750309#comment-16750309
 ] 

ASF subversion and git services commented on ATLAS-3034:


Commit 7c471dc313daf7c9e30e77024ac3e2c94a9a5403 in atlas's branch 
refs/heads/branch-1.0 from Sarath Subramanian
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=7c471dc ]

ATLAS-3034: Perf enhancement to avoid unnecessary lookup when creating new 
relationships

(cherry picked from commit a9aa5b0e1b0a7c2867aef16358e3c3c1977bb140)


> Perf enhancement to avoid unnecessary lookup when creating new relationships
> 
>
> Key: ATLAS-3034
> URL: https://issues.apache.org/jira/browse/ATLAS-3034
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
>
> Perf enhancement to avoid unnecessary lookup when creating new relationships



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69816: ATLAS-3035: updated entity-get/delete to retrieve/delete soft-ref entities

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69816/#review212243
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
Lines 244 (patched)


Consider moving this to a separate method.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Line 501 (original)


Perfer being defensive.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Line 740 (original), 737 (patched)


(Optional) Prefer being defensive.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Line 762 (original), 756 (patched)


Prefer being defensive.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Lines 780 (patched)


Duplicate code, mapVertexToObjectId.


- Ashutosh Mestry


On Jan. 23, 2019, 9 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69816/
> ---
> 
> (Updated Jan. 23, 2019, 9 a.m.)
> 
> 
> Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, 
> Kapildeo Nayak, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3035
> https://issues.apache.org/jira/browse/ATLAS-3035
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Updated entity-get/delete to retrieve/delete soft-ref entities for owne-ref 
> attributes
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 27409c742 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 45bf8dca4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  c57e30ab5 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  98734cc1d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  912bc3ecd 
> 
> 
> Diff: https://reviews.apache.org/r/69816/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that entity-get populates extInfo with details of owned-ref 
> soft-ref entities
> - verified that entity-delete marks owned-ref soft-ref entities as deleted
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 69816: ATLAS-3035: updated entity-get/delete to retrieve/delete soft-ref entities

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69816/#review212244
---



Please update _SoftReferenceTest_ to reflect the updates.

- Ashutosh Mestry


On Jan. 23, 2019, 9 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69816/
> ---
> 
> (Updated Jan. 23, 2019, 9 a.m.)
> 
> 
> Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, 
> Kapildeo Nayak, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3035
> https://issues.apache.org/jira/browse/ATLAS-3035
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Updated entity-get/delete to retrieve/delete soft-ref entities for owne-ref 
> attributes
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 27409c742 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 45bf8dca4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  c57e30ab5 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  98734cc1d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  912bc3ecd 
> 
> 
> Diff: https://reviews.apache.org/r/69816/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that entity-get populates extInfo with details of owned-ref 
> soft-ref entities
> - verified that entity-delete marks owned-ref soft-ref entities as deleted
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

[jira] [Commented] (ATLAS-3034) Perf enhancement to avoid unnecessary lookup when creating new relationships

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750307#comment-16750307
 ] 

ASF subversion and git services commented on ATLAS-3034:


Commit a9aa5b0e1b0a7c2867aef16358e3c3c1977bb140 in atlas's branch 
refs/heads/master from Sarath Subramanian
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=a9aa5b0 ]

ATLAS-3034: Perf enhancement to avoid unnecessary lookup when creating new 
relationships


> Perf enhancement to avoid unnecessary lookup when creating new relationships
> 
>
> Key: ATLAS-3034
> URL: https://issues.apache.org/jira/browse/ATLAS-3034
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
>
> Perf enhancement to avoid unnecessary lookup when creating new relationships



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69789: Tool: Classification Updater

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69789/#review212253
---


Ship it!




Ship It!

- Sarath Subramanian


On Jan. 23, 2019, 11:14 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69789/
> ---
> 
> (Updated Jan. 23, 2019, 11:14 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3015
> https://issues.apache.org/jira/browse/ATLAS-3015
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> - Uses the new _EntityREST_ APIs to retrieve entity-classification 
> association.
> - File containing classification definitions is generated.
> - During the prepare stage, output of new REST API (_bulk/header_) is written 
> to another file.
> - During the update stage, the output from _prepare_ stage is processed and 
> associated Atlas server is updated.
> 
> **Usage**
> Usage: classification-updater.sh [-s ] [-f ] [-d ]
> step: Specify which step to execute:
>prepare: prepare classifications and associated entities.
>update: update classifications and entities.
> dir: [optional] Directory where read/write will happen.
>If not specified, current directory will be used.
> from: [mandatory for 'prepare' step, optional for 'update' step] Date, in 
> -MM-DD format, from where audits need to be read.
>If not specified, current directory will be used.
> 
> 
> Diffs
> -
> 
>   tools/classification-updater/pom.xml PRE-CREATION 
>   
> tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
>  PRE-CREATION 
>   tools/classification-updater/src/main/resources/atlas-log4j.xml 
> PRE-CREATION 
>   tools/classification-updater/src/main/resources/update-classifications.sh 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69789/diff/3/
> 
> 
> Testing
> ---
> 
> **Functional**
> - Tool executed in environments with various authentication mechanisms.
> 
> 
> **Volume**
> - Tool was executed in environment with large data.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 69789: Tool: Classification Updater

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69789/#review212251
---




tools/classification-updater/src/main/resources/update-classifications.sh
Lines 80 (patched)


check if IMPORT_ARGS is null/empty then printUsage()


- Sarath Subramanian


On Jan. 23, 2019, 11:14 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69789/
> ---
> 
> (Updated Jan. 23, 2019, 11:14 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3015
> https://issues.apache.org/jira/browse/ATLAS-3015
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> - Uses the new _EntityREST_ APIs to retrieve entity-classification 
> association.
> - File containing classification definitions is generated.
> - During the prepare stage, output of new REST API (_bulk/header_) is written 
> to another file.
> - During the update stage, the output from _prepare_ stage is processed and 
> associated Atlas server is updated.
> 
> **Usage**
> Usage: classification-updater.sh [-s ] [-f ] [-d ]
> step: Specify which step to execute:
>prepare: prepare classifications and associated entities.
>update: update classifications and entities.
> dir: [optional] Directory where read/write will happen.
>If not specified, current directory will be used.
> from: [mandatory for 'prepare' step, optional for 'update' step] Date, in 
> -MM-DD format, from where audits need to be read.
>If not specified, current directory will be used.
> 
> 
> Diffs
> -
> 
>   tools/classification-updater/pom.xml PRE-CREATION 
>   
> tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
>  PRE-CREATION 
>   tools/classification-updater/src/main/resources/atlas-log4j.xml 
> PRE-CREATION 
>   tools/classification-updater/src/main/resources/update-classifications.sh 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69789/diff/3/
> 
> 
> Testing
> ---
> 
> **Functional**
> - Tool executed in environments with various authentication mechanisms.
> 
> 
> **Volume**
> - Tool was executed in environment with large data.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 69789: Tool: Classification Updater

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69789/#review212254
---




tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
Lines 325 (patched)


consider serializing classificationDefs list using 
AtlasType.toJson(classificationDefs) instead of individual conversion.


- Sarath Subramanian


On Jan. 23, 2019, 11:14 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69789/
> ---
> 
> (Updated Jan. 23, 2019, 11:14 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3015
> https://issues.apache.org/jira/browse/ATLAS-3015
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> - Uses the new _EntityREST_ APIs to retrieve entity-classification 
> association.
> - File containing classification definitions is generated.
> - During the prepare stage, output of new REST API (_bulk/header_) is written 
> to another file.
> - During the update stage, the output from _prepare_ stage is processed and 
> associated Atlas server is updated.
> 
> **Usage**
> Usage: classification-updater.sh [-s ] [-f ] [-d ]
> step: Specify which step to execute:
>prepare: prepare classifications and associated entities.
>update: update classifications and entities.
> dir: [optional] Directory where read/write will happen.
>If not specified, current directory will be used.
> from: [mandatory for 'prepare' step, optional for 'update' step] Date, in 
> -MM-DD format, from where audits need to be read.
>If not specified, current directory will be used.
> 
> 
> Diffs
> -
> 
>   tools/classification-updater/pom.xml PRE-CREATION 
>   
> tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
>  PRE-CREATION 
>   tools/classification-updater/src/main/resources/atlas-log4j.xml 
> PRE-CREATION 
>   tools/classification-updater/src/main/resources/update-classifications.sh 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69789/diff/3/
> 
> 
> Testing
> ---
> 
> **Functional**
> - Tool executed in environments with various authentication mechanisms.
> 
> 
> **Volume**
> - Tool was executed in environment with large data.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 69789: Tool: Classification Updater

[Ashutosh Mestry]

> On Jan. 23, 2019, 8:30 p.m., Sarath Subramanian wrote:
> > tools/classification-updater/src/main/resources/update-classifications.sh
> > Lines 80 (patched)
> > 
> >
> > check if IMPORT_ARGS is null/empty then printUsage()

I moved the check to the Java class. Hence removed the null check.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69789/#review212251
---


On Jan. 23, 2019, 7:14 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69789/
> ---
> 
> (Updated Jan. 23, 2019, 7:14 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3015
> https://issues.apache.org/jira/browse/ATLAS-3015
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> - Uses the new _EntityREST_ APIs to retrieve entity-classification 
> association.
> - File containing classification definitions is generated.
> - During the prepare stage, output of new REST API (_bulk/header_) is written 
> to another file.
> - During the update stage, the output from _prepare_ stage is processed and 
> associated Atlas server is updated.
> 
> **Usage**
> Usage: classification-updater.sh [-s ] [-f ] [-d ]
> step: Specify which step to execute:
>prepare: prepare classifications and associated entities.
>update: update classifications and entities.
> dir: [optional] Directory where read/write will happen.
>If not specified, current directory will be used.
> from: [mandatory for 'prepare' step, optional for 'update' step] Date, in 
> -MM-DD format, from where audits need to be read.
>If not specified, current directory will be used.
> 
> 
> Diffs
> -
> 
>   tools/classification-updater/pom.xml PRE-CREATION 
>   
> tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
>  PRE-CREATION 
>   tools/classification-updater/src/main/resources/atlas-log4j.xml 
> PRE-CREATION 
>   tools/classification-updater/src/main/resources/update-classifications.sh 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69789/diff/3/
> 
> 
> Testing
> ---
> 
> **Functional**
> - Tool executed in environments with various authentication mechanisms.
> 
> 
> **Volume**
> - Tool was executed in environment with large data.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

[jira] [Commented] (ATLAS-3015) Tool: Associate Classifications to Entities based on QualifiedName

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750454#comment-16750454
 ] 

ASF subversion and git services commented on ATLAS-3015:


Commit 4985b9f2a64a0f16f323bcbac600457215636852 in atlas's branch 
refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=4985b9f ]

ATLAS-3015: Classification Updater tool.


> Tool: Associate Classifications to Entities based on QualifiedName
> --
>
> Key: ATLAS-3015
> URL: https://issues.apache.org/jira/browse/ATLAS-3015
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: 0001-ATLAS-3015.patch, 
> ATLAS-3015-Classification-Updater-tool-master.patch
>
>
> *Background*
> Situation: A setup where Atlas classifications associated with entities the 
> data is subsequently erased and all data is lost. Entities are then 
> re-created using _import-hive.sh_. With this approach, the entities are 
> created within Atlas but there is no easy way to associate classifications to 
> these entities.
> *Solution and Approach Guidance*
>  # Before erasing Atlas database, create file with classifications and 
> entity-classification mapping.
>  # Erase the database.
>  # Create entities using appropriate means, say _import-hive.sh_.
>  # Read the file from step 1 and update Atlas.
> The tool should:
>  * Use _AtlasClient_ REST APIs.
>  * External and can be executed against a known Atlas instance.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69816: ATLAS-3035: updated entity-get/delete to retrieve/delete soft-ref entities

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69816/#review212256
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Lines 780 (patched)


mapVertexToObjectId() needs edgeLabel and edge to find the vertex to map. 
This one is specifically for soft-ref entities.


- Madhan Neethiraj


On Jan. 23, 2019, 9 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69816/
> ---
> 
> (Updated Jan. 23, 2019, 9 a.m.)
> 
> 
> Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, 
> Kapildeo Nayak, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3035
> https://issues.apache.org/jira/browse/ATLAS-3035
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Updated entity-get/delete to retrieve/delete soft-ref entities for owne-ref 
> attributes
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 27409c742 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 45bf8dca4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  c57e30ab5 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  98734cc1d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  912bc3ecd 
> 
> 
> Diff: https://reviews.apache.org/r/69816/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that entity-get populates extInfo with details of owned-ref 
> soft-ref entities
> - verified that entity-delete marks owned-ref soft-ref entities as deleted
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

[jira] [Updated] (ATLAS-3015) Tool: Associate Classifications to Entities based on QualifiedName

[Nikhil Bonte (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3015:

Attachment: 0001-ATLAS-3015.patch

> Tool: Associate Classifications to Entities based on QualifiedName
> --
>
> Key: ATLAS-3015
> URL: https://issues.apache.org/jira/browse/ATLAS-3015
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: 0001-ATLAS-3015.patch, 
> ATLAS-3015-Classification-Updater-tool-master.patch
>
>
> *Background*
> Situation: A setup where Atlas classifications associated with entities the 
> data is subsequently erased and all data is lost. Entities are then 
> re-created using _import-hive.sh_. With this approach, the entities are 
> created within Atlas but there is no easy way to associate classifications to 
> these entities.
> *Solution and Approach Guidance*
>  # Before erasing Atlas database, create file with classifications and 
> entity-classification mapping.
>  # Erase the database.
>  # Create entities using appropriate means, say _import-hive.sh_.
>  # Read the file from step 1 and update Atlas.
> The tool should:
>  * Use _AtlasClient_ REST APIs.
>  * External and can be executed against a known Atlas instance.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3035) Entity retrieval to include soft-referenced entities in extInfo

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750519#comment-16750519
 ] 

ASF subversion and git services commented on ATLAS-3035:


Commit 2341d58a6ef14de410b75abdbc1378c6f81db6fa in atlas's branch 
refs/heads/master from Madhan Neethiraj
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=2341d58 ]

ATLAS-3035: updated entity-get/delete to retrieve/delete soft-ref entities


> Entity retrieval to include soft-referenced entities in extInfo
> ---
>
> Key: ATLAS-3035
> URL: https://issues.apache.org/jira/browse/ATLAS-3035
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: ATLAS-3035-branch-1.0.patch, ATLAS-3035.patch
>
>
> ATLAS-2813 introduced the notion of soft-reference for attributes that refer 
> to other entities. This should only change the storage details of the 
> relationship, but shouldn't introduce any change in the data exchanged via 
> REST APIs. However, the return from entity-get operations do not include 
> soft-referenced entities in extInfo. This should be updated such that the 
> return value would populate extInfo - just as in case of regular 
> object-references.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3035) Entity retrieval to include soft-referenced entities in extInfo

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750520#comment-16750520
 ] 

ASF subversion and git services commented on ATLAS-3035:


Commit c40c906da96750a977ebeae39b75123e02b09242 in atlas's branch 
refs/heads/branch-1.0 from Madhan Neethiraj
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=c40c906 ]

ATLAS-3035: updated entity-get/delete to retrieve/delete soft-ref entities


> Entity retrieval to include soft-referenced entities in extInfo
> ---
>
> Key: ATLAS-3035
> URL: https://issues.apache.org/jira/browse/ATLAS-3035
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: ATLAS-3035-branch-1.0.patch, ATLAS-3035.patch
>
>
> ATLAS-2813 introduced the notion of soft-reference for attributes that refer 
> to other entities. This should only change the storage details of the 
> relationship, but shouldn't introduce any change in the data exchanged via 
> REST APIs. However, the return from entity-get operations do not include 
> soft-referenced entities in extInfo. This should be updated such that the 
> return value would populate extInfo - just as in case of regular 
> object-references.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)