[GitHub] [atlas] mneethiraj commented on issue #78: The method hivedb. Getcatalogname() does not exist in version 2.0

2019-12-04 Thread GitBox 
mneethiraj commented on issue #78: The method hivedb. Getcatalogname() does not 
exist in version 2.0
URL: https://github.com/apache/atlas/pull/78#issuecomment-561990704
 
 
   @xinda77 - can you please file a JIRA in https://issues.apache.org/jira/ 
with details of the issue?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [atlas] xinda77 opened a new pull request #78: The method hivedb. Getcatalogname() does not exist in version 2.0

2019-12-04 Thread GitBox 
xinda77 opened a new pull request #78: The method hivedb. Getcatalogname() does 
not exist in version 2.0
URL: https://github.com/apache/atlas/pull/78
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (ATLAS-3548) Support multiple ServiceTypes for one entity type

2019-12-04 Thread charles shen (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

charles shen updated ATLAS-3548:

Description: 
There are cases where one entity type can belong to more than one ServiceType, 
eg, [AWS 
EFS|[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEFS.htm|https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEFS.html]]
 can belong to ServiceType AWS and FILE_SYSTEM.

The possible change is to change the ServiceType from String to Array.

  was:
There are cases where one entity type can belong to more than one ServiceType, 
eg, [AWS 
EFS|[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEFS.html]] can 
belong to ServiceType AWS and FILE_SYSTEM.

The possible change is to change the ServiceType from String to Array.


> Support multiple ServiceTypes for one entity type
> -
>
> Key: ATLAS-3548
> URL: https://issues.apache.org/jira/browse/ATLAS-3548
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: charles shen
>Priority: Major
>
> There are cases where one entity type can belong to more than one 
> ServiceType, eg, [AWS 
> EFS|[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEFS.htm|https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEFS.html]]
>  can belong to ServiceType AWS and FILE_SYSTEM.
> The possible change is to change the ServiceType from String to Array.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-04 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988468#comment-16988468
 ] 

ASF subversion and git services commented on ATLAS-3545:


Commit 6b96a40d1787ebeae247140c49af5de2fd59b1d4 in atlas's branch 
refs/heads/branch-2.0 from Sarath Subramanian
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=6b96a40 ]

ATLAS-3545: NullPointerException while trying to delete classification

(cherry picked from commit c95aba218b9318ce44abf574faff62908f24916d)


> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: ATLAS-3545.002.patch, Screenshot 2019-11-30 at 
> 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 
> 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-04 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988467#comment-16988467
 ] 

ASF subversion and git services commented on ATLAS-3545:


Commit c95aba218b9318ce44abf574faff62908f24916d in atlas's branch 
refs/heads/master from Sarath Subramanian
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=c95aba2 ]

ATLAS-3545: NullPointerException while trying to delete classification


> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: ATLAS-3545.002.patch, Screenshot 2019-11-30 at 
> 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 
> 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71870: ATLAS-3545: NullPointerException while trying to delete classification

2019-12-04 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71870/#review218935
---


Ship it!




Ship It!

- Madhan Neethiraj


On Dec. 5, 2019, 4:17 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71870/
> ---
> 
> (Updated Dec. 5, 2019, 4:17 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer 
> Shaikh, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3545
> https://issues.apache.org/jira/browse/ATLAS-3545
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Corrupt/Null classification vertices is causing NPE during entity search and 
> retrieval. Null checks have been added to avoid NPE and logging added to 
> identify potential corrupt classification vertex.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  a114d252e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  6b111732a 
> 
> 
> Diff: https://reviews.apache.org/r/71870/diff/2/
> 
> 
> Testing
> ---
> 
> Precommit: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1576
> 
> Manually verified that null/corrupt classification vertices are ignored 
> during search and entity retrieval.
> 
> Approproate WARN messages are printed in the log file:
> 
> *2019-12-03 23:26:45,243 WARN  - [pool-2-thread-10 - 
> 305383de-19f2-4f93-89db-3b54db870c3f:] ~ Ignoring invalid classification 
> vertex: vertex[id=81932528] (EntityGraphRetriever:284)*
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-04 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3545:
--
Attachment: ATLAS-3545.002.patch

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: ATLAS-3545.002.patch, Screenshot 2019-11-30 at 
> 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 
> 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification

2019-12-04 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3545:
--
Attachment: (was: ATLAS-3545.001.patch)

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71870: ATLAS-3545: NullPointerException while trying to delete classification

2019-12-04 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71870/
---

(Updated Dec. 4, 2019, 8:17 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer 
Shaikh, and Sidharth Mishra.


Bugs: ATLAS-3545
https://issues.apache.org/jira/browse/ATLAS-3545


Repository: atlas


Description
---

Corrupt/Null classification vertices is causing NPE during entity search and 
retrieval. Null checks have been added to avoid NPE and logging added to 
identify potential corrupt classification vertex.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 a114d252e 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 6b111732a 


Diff: https://reviews.apache.org/r/71870/diff/2/

Changes: https://reviews.apache.org/r/71870/diff/1-2/


Testing
---

Precommit: 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1575/

Manually verified that null/corrupt classification vertices are ignored during 
search and entity retrieval.

Approproate WARN messages are printed in the log file:

*2019-12-03 23:26:45,243 WARN  - [pool-2-thread-10 - 
305383de-19f2-4f93-89db-3b54db870c3f:] ~ Ignoring invalid classification 
vertex: vertex[id=81932528] (EntityGraphRetriever:284)*


Thanks,

Sarath Subramanian

[jira] [Commented] (ATLAS-3546) isOptional for composition relationship category?

2019-12-04 Thread Madhan Neethiraj (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988422#comment-16988422
 ] 

Madhan Neethiraj commented on ATLAS-3546:
-

[~api123] - about the first scenario: to create a single rdbms_column, 
providing object-id of an existing rdbms_table entity should be enough; there 
should be no need to provide rdbms_instance/rdbms_db/... For example:
{noformat}
{
  "guid": "-1",
  "typeName": "rdbms_column",
  "attributes": {
"name":  "id",
"qualifiedName": "sales.order_items.id@prod",
"data_type": "long",
"isPrimaryKey":  false
  },
  "relationshipAttributes": {
"table": {
  "relationshipType": "rdbms_table_columns",
  "typeName": "rdbms_table",
  "uniqueAttributes": {
"qualifiedName": "sales.order_items@prod"
  }
}
  }
}
{noformat}

For the second scenario, perhaps you can consider modeling this as an 
aggregation. Composition, by definition needs a container entity to be present 
for a contained entity to exist. If this constraint can't be met, then perhaps 
the relationship shouldn't be modeled as composition.

You might also want to consider the enhancement added in ATLAS-3405, which will 
create a shell/empty entity for non-existing object-references. In your 
example, if rdbms_table with {{qualifiedName=sales.order_items@prod doesn't 
exist}}, Atlas will create an shell entity for the rdbms_table (with only 
qualifiedName attribute populated), and create rdbms_column entity with link to 
the created table entity. The table entity can be populated later in a 
subsequent call. Note that configuration 
{{atlas.rest.create.shell.entity.for.non-existing.ref=true}} should be set for 
the above handling; else Atlas will fail creation of the column entity.

> isOptional for composition relationship category?
> -
>
> Key: ATLAS-3546
> URL: https://issues.apache.org/jira/browse/ATLAS-3546
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: charles shen
>Priority: Major
>
> I noticed since 
> [ATLAS-3051|https://issues.apache.org/jira/browse/ATLAS-3051], the 
> relationship attribute must be specified in the end def which is not 
> container and relationship category is composition. 
> I understand it's to prohibit orphan children but is it too strong? Reason 
> below:
>  # I have to provide all the entities along the relationship path in the 
> payload when creating a child, eg, for RDBMS, I have to provide 
> rdbms_instance, rdbms_db, rdbms_table, rdbms_column where I just want to 
> create a single rdbms_column, it brings performance overhead to check 
> existence of rdbms_instance, rdbms_db, etc..., 
>  # I have defined a composition relationship type where each end is the same 
> entity type, it couldn't be created successfully anymore since it always 
> requires the mandatory attribute where it's the same type and then falls into 
> infinite loop.
> Three possible fixes:
>  # Remove the isOptional constraint, since ownedRef/inverseRef doesn't have 
> such constraint.
>  # Add isOptional to relationship type end def.
>  # Add option in Rest to ignore the isOptional constraint for relationship 
> type.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

2019-12-04 Thread Sidharth Mishra 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/
---

(Updated Dec. 5, 2019, 2:49 a.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
Subramanian.


Bugs: ATLAS-3518
https://issues.apache.org/jira/browse/ATLAS-3518


Repository: atlas


Description
---

Right now for purge, delete entity, etc. at Atlas we create audit entry at 
HBase. User can go to entity and click on audit to see the audit information 
for that particular entity. But if user purge one entity there will not be any 
vertex at Janus graph and user will not able to get the purge audit which is 
important for governance purpose.

As part of fix, implemented Atlas Audit Framework which is being used for Purge 
operation and can be extended for other operations. In Future once we change 
the AtlasAudit type as super type for Import export as well we can use this 
framework for all Atlas operations. 

admin/audit rest has been added which will return all the Atlas audit 
operations including import export at Atlas.


Diffs (updated)
-

  addons/models/-Area0/0010-base_model.json 2f5fdaf14 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c 
  dashboardv2/public/js/utils/Enums.js 58542ec59 
  dashboardv3/public/js/utils/Enums.js 58542ec59 
  intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
PRE-CREATION 
  
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java 
b448d510b 
  
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 c9ed79750 
  repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
  repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
be1698fb4 
  repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369 
  
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java 
PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
 78865c569 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
 ba7a8a04b 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
 925b2a007 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
 4d438525a 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
 db8c6c83c 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
 25e0a5362 
  
repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java
 1863b8d74 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java
 d0c06a107 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 c14850f43 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java
 78fdaca8d 
  
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
 42b63535b 
  
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java
 a1b512fb9 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
 7a1ed18cb 
  
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java
 48b958271 
  
repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java
 c9bb11c56 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
 0ffc3d595 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 18e950e8c 
  
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java
 050bda3ae 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
 6f9c05e7a 
  
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java
 e56e48308 
  repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
baeafd4b7 
  repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java 
PRE-CREATION 
  server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
40fd6efdc 
  webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 
afc4c50d7 
  
webapp/src/test/java/org/apache

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

2019-12-04 Thread Sidharth Mishra 


> On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote:
> > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
> > Lines 124 (patched)
> > 
> >
> > define equals() and hashCode() method as well.

AtlasAuditEntry is derived from AtlasBaseModelObject and the base class 
contains both of these method with guid check. Please let me know do we still 
need these two to be overridden?


> On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote:
> > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
> > Lines 125 (patched)
> > 
> >
> > incorrect toString() implementation. please format in alignment with 
> > other model classes.

AtlasAuditEntry is derived from AtlasBaseModelObject and the base class 
contains the toString() which calls this toString(sb). At the end it returns 
the correct alignment.


> On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote:
> > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
> > Lines 128 (patched)
> > 
> >
> > => StringUtils.join(entityIds, ","))

Fixed. Could you please let me know the advantage of StringUtils.join over 
String.join


> On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
> > Lines 76 (patched)
> > 
> >
> > move debug log to beginning of method.

This debug msg is to put debug message when we successfully added the audit 
entry


> On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Lines 437 (patched)
> > 
> >
> > shouldn't purge events be audited only on successful purge on entities? 
> > consider checking EntityMutationResponse for any purge events present and 
> > then audit them.

I think its better to audit admin action purge even if we don't purge any 
entities. Please let me know if you still think we should not audit the 
unsuccessful purge action by admin.


- Sidharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218927
---


On Dec. 4, 2019, 9:57 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
> 
> (Updated Dec. 4, 2019, 9:57 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Right now for purge, delete entity, etc. at Atlas we create audit entry at 
> HBase. User can go to entity and click on audit to see the audit information 
> for that particular entity. But if user purge one entity there will not be 
> any vertex at Janus graph and user will not able to get the purge audit which 
> is important for governance purpose.
> 
> As part of fix, implemented Atlas Audit Framework which is being used for 
> Purge operation and can be extended for other operations. In Future once we 
> change the AtlasAudit type as super type for Import export as well we can use 
> this framework for all Atlas operations. 
> 
> admin/audit rest has been added which will return all the Atlas audit 
> operations including import export at Atlas.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf14 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 8c0a6407c 
>   dashboardv2/public/js/utils/Enums.js 58542ec59 
>   dashboardv3/public/js/utils/Enums.js 58542ec59 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  b448d510b 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java 
> eb8634533 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  c9ed79750 
>   repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> be1698fb4 
>   repository/s

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

2019-12-04 Thread Sidharth Mishra 


> On Dec. 5, 2019, 12:39 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> > Lines 1057 (patched)
> > 
> >
> > This will cause 'DELETED' entities to be skipped even though purge is 
> > in progress. This isn't correct. Please review and update.
> > 
> >   final String guid  = AtlasGraphUtilsV2.getIdFromVertex(vertex);
> >   
> >   ret = guid == null;
> > 
> >   if (!ret) {
> > if (reqContext.isPurgeRequested()) {
> >   ret = requestContext.isPurgedEntity(guid) || getState(vertex) == 
> > ACTIVE;
> > } else {
> >   ret = requestContext.isDeletedEntity(guid) || getState(vertex) == 
> > DELETED;
> > }
> >   }

As the code for purge, hard delete is same, the requestContext still stores the 
purged entities as part of deleteEntity List.


> On Dec. 5, 2019, 12:39 a.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Lines 436 (patched)
> > 
> >
> > Audit generation should be done inside entityStore.purgeByIds(), in the 
> > same graph transaction. Avoid spilling such details into REST layer.

As I earlier replied to your previous comment, Moving auditing to inside 
AtlasEntityStore.purgeByIds() is causing circular Injection issue. 
AtlasAuditService depends on DataAccess and DataAccess depends on 
AtlasEntityStore. So AtlasEntityStore can't depends on AtlasAuditService again. 
KIndly suggest is there any other option to avoid this circular injection issue.


- Sidharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218929
---


On Dec. 4, 2019, 9:57 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
> 
> (Updated Dec. 4, 2019, 9:57 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Right now for purge, delete entity, etc. at Atlas we create audit entry at 
> HBase. User can go to entity and click on audit to see the audit information 
> for that particular entity. But if user purge one entity there will not be 
> any vertex at Janus graph and user will not able to get the purge audit which 
> is important for governance purpose.
> 
> As part of fix, implemented Atlas Audit Framework which is being used for 
> Purge operation and can be extended for other operations. In Future once we 
> change the AtlasAudit type as super type for Import export as well we can use 
> this framework for all Atlas operations. 
> 
> admin/audit rest has been added which will return all the Atlas audit 
> operations including import export at Atlas.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf14 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 8c0a6407c 
>   dashboardv2/public/js/utils/Enums.js 58542ec59 
>   dashboardv3/public/js/utils/Enums.js 58542ec59 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  b448d510b 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java 
> eb8634533 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  c9ed79750 
>   repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> be1698fb4 
>   repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 
> 02f78b369 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
>  78865c569 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
>  ba7a8a04b 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
>  925b2a007 
>   
> repository/src/test/ja

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

2019-12-04 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218929
---




addons/models/-Area0/0010-base_model.json
Lines 458 (patched)


operationResult => result



repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
Lines 155 (patched)


For startTime and endTime attributes, I think use of operators GTE and LTE 
(respectively) would be useful. Consider adding operator as a parameter to this 
method.



repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
Lines 1057 (patched)


This will cause 'DELETED' entities to be skipped even though purge is in 
progress. This isn't correct. Please review and update.

  final String guid  = AtlasGraphUtilsV2.getIdFromVertex(vertex);
  
  ret = guid == null;

  if (!ret) {
if (reqContext.isPurgeRequested()) {
  ret = requestContext.isPurgedEntity(guid) || getState(vertex) == 
ACTIVE;
} else {
  ret = requestContext.isDeletedEntity(guid) || getState(vertex) == 
DELETED;
}
  }



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 31 (original), 32 (patched)


Consider avoiding unnecessary changes to existing import statements. If 
this is done automatically by IDE, please configure IDE to not perform this 
change.



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 436 (patched)


Audit generation should be done inside entityStore.purgeByIds(), in the 
same graph transaction. Avoid spilling such details into REST layer.


- Madhan Neethiraj


On Dec. 4, 2019, 9:57 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
> 
> (Updated Dec. 4, 2019, 9:57 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Right now for purge, delete entity, etc. at Atlas we create audit entry at 
> HBase. User can go to entity and click on audit to see the audit information 
> for that particular entity. But if user purge one entity there will not be 
> any vertex at Janus graph and user will not able to get the purge audit which 
> is important for governance purpose.
> 
> As part of fix, implemented Atlas Audit Framework which is being used for 
> Purge operation and can be extended for other operations. In Future once we 
> change the AtlasAudit type as super type for Import export as well we can use 
> this framework for all Atlas operations. 
> 
> admin/audit rest has been added which will return all the Atlas audit 
> operations including import export at Atlas.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf14 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 8c0a6407c 
>   dashboardv2/public/js/utils/Enums.js 58542ec59 
>   dashboardv3/public/js/utils/Enums.js 58542ec59 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  b448d510b 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java 
> eb8634533 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  c9ed79750 
>   repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> be1698fb4 
>   repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 
> 02f78b369 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
>  78865c569 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
>  ba7a8a04b 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

2019-12-04 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218927
---




client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 598 (patched)


'GET_ATLAS_AUDIT' => 'GET_ATLAS_AUDITS'



intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
Lines 56 (patched)


unused constructor. consider removing it.



intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
Lines 124 (patched)


define equals() and hashCode() method as well.



intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
Lines 125 (patched)


incorrect toString() implementation. please format in alignment with other 
model classes.



intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
Lines 122 (patched)


assign ret to null and handle null in callers.



intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
Lines 128 (patched)


=> StringUtils.join(entityIds, ","))



intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java
Line 143 (original), 143 (patched)


revert if change not intended.



repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
Lines 76 (patched)


move debug log to beginning of method.



repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
Lines 92 (patched)


avoid too many ternary operations in a single line. add null checks/default 
values inside getSearchCriteria() method.



repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
Lines 272 (patched)


nit: revert new line change



repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
Lines 1050 (patched)


consider refactoring this method



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 437 (patched)


shouldn't purge events be audited only on successful purge on entities? 
consider checking EntityMutationResponse for any purge events present and then 
audit them.



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 438 (patched)


'addAtlasAudit()' => 'add()'

'startTimestamp' can be got from RequestContext.get().getRequestTime()


- Sarath Subramanian


On Dec. 4, 2019, 1:57 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
> 
> (Updated Dec. 4, 2019, 1:57 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Right now for purge, delete entity, etc. at Atlas we create audit entry at 
> HBase. User can go to entity and click on audit to see the audit information 
> for that particular entity. But if user purge one entity there will not be 
> any vertex at Janus graph and user will not able to get the purge audit which 
> is important for governance purpose.
> 
> As part of fix, implemented Atlas Audit Framework which is being used for 
> Purge operation and can be extended for other operations. In Future once we 
> change the AtlasAudit type as super type for Import export as well we can use 
> this framework for all Atlas operations. 
> 
> admin/audit rest has been added which will return all the Atlas audit 
> operations including import export at Atlas.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf14 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 8c0a6407c 
>   dashboardv2/public/js/utils/Enums.js 58542ec59 
>   dashboardv3/public/js/utils/Enums.js 58542ec59 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  b448d510b 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java 
> eb8634533 
>

Re: Review Request 71870: ATLAS-3545: NullPointerException while trying to delete classification

2019-12-04 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71870/#review218928
---


Fix it, then Ship it!





repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 1866 (patched)


Consider throwing following exception here, instead of 'return':
  throw new 
AtlasBaseException(AtlasErrorCode.CLASSIFICATION_NOT_FOUND_FOR_ENTITY, 
classificationName, entityGuid);


- Madhan Neethiraj


On Dec. 4, 2019, 7:32 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71870/
> ---
> 
> (Updated Dec. 4, 2019, 7:32 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer 
> Shaikh, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3545
> https://issues.apache.org/jira/browse/ATLAS-3545
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Corrupt/Null classification vertices is causing NPE during entity search and 
> retrieval. Null checks have been added to avoid NPE and logging added to 
> identify potential corrupt classification vertex.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  a114d252e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  6b111732a 
> 
> 
> Diff: https://reviews.apache.org/r/71870/diff/1/
> 
> 
> Testing
> ---
> 
> Precommit: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1575/
> 
> Manually verified that null/corrupt classification vertices are ignored 
> during search and entity retrieval.
> 
> Approproate WARN messages are printed in the log file:
> 
> *2019-12-03 23:26:45,243 WARN  - [pool-2-thread-10 - 
> 305383de-19f2-4f93-89db-3b54db870c3f:] ~ Ignoring invalid classification 
> vertex: vertex[id=81932528] (EntityGraphRetriever:284)*
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

2019-12-04 Thread Sidharth Mishra 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/
---

(Updated Dec. 4, 2019, 9:57 p.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
Subramanian.


Bugs: ATLAS-3518
https://issues.apache.org/jira/browse/ATLAS-3518


Repository: atlas


Description
---

Right now for purge, delete entity, etc. at Atlas we create audit entry at 
HBase. User can go to entity and click on audit to see the audit information 
for that particular entity. But if user purge one entity there will not be any 
vertex at Janus graph and user will not able to get the purge audit which is 
important for governance purpose.

As part of fix, implemented Atlas Audit Framework which is being used for Purge 
operation and can be extended for other operations. In Future once we change 
the AtlasAudit type as super type for Import export as well we can use this 
framework for all Atlas operations. 

admin/audit rest has been added which will return all the Atlas audit 
operations including import export at Atlas.


Diffs (updated)
-

  addons/models/-Area0/0010-base_model.json 2f5fdaf14 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c 
  dashboardv2/public/js/utils/Enums.js 58542ec59 
  dashboardv3/public/js/utils/Enums.js 58542ec59 
  intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
PRE-CREATION 
  
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java 
b448d510b 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java 
eb8634533 
  
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 c9ed79750 
  repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
  repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
be1698fb4 
  repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369 
  
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java 
PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
 78865c569 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
 ba7a8a04b 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
 925b2a007 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
 4d438525a 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
 db8c6c83c 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
 25e0a5362 
  
repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java
 1863b8d74 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java
 d0c06a107 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 c14850f43 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java
 78fdaca8d 
  
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
 42b63535b 
  
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java
 a1b512fb9 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
 7a1ed18cb 
  
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java
 48b958271 
  
repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java
 c9bb11c56 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
 0ffc3d595 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 18e950e8c 
  
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java
 050bda3ae 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
 6f9c05e7a 
  
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java
 e56e48308 
  repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
baeafd4b7 
  repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java 
PRE-CREATION 
  server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
40fd6efdc 
  webapp/src/test/java/org/apache/a

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

2019-12-04 Thread Sidharth Mishra 


> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote:
> > addons/models/-Area0/0010-base_model.json
> > Lines 5 (patched)
> > 
> >
> > "Audit Operation defines the operations we support for admin audit" => 
> > "Defines admin operations in Apache Atlas"

Removed the admin and changed it to "Defines operations in Apache Atlas which 
can be audited"


> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Lines 436 (patched)
> > 
> >
> > Instead of generating audit in a seperate graph transaction here, 
> > consider generating in the transaction used by 
> > AtlasEntityStore.purgeByIds() i.e. move auditing to inside 
> > AtlasEntityStore.purgeByIds().

Moving auditing to inside AtlasEntityStore.purgeByIds() is causing circular 
Injection issue. AtlasAuditService depends on DataAccess and DataAccess depends 
on AtlasEntityStore. So AtlasEntityStore can't depends on AtlasAuditService 
again.


- Sidharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218865
---


On Dec. 4, 2019, 9:47 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
> 
> (Updated Dec. 4, 2019, 9:47 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Right now for purge, delete entity, etc. at Atlas we create audit entry at 
> HBase. User can go to entity and click on audit to see the audit information 
> for that particular entity. But if user purge one entity there will not be 
> any vertex at Janus graph and user will not able to get the purge audit which 
> is important for governance purpose.
> 
> As part of fix, implemented Atlas Audit Framework which is being used for 
> Purge operation and can be extended for other operations. In Future once we 
> change the AtlasAudit type as super type for Import export as well we can use 
> this framework for all Atlas operations. 
> 
> admin/audit rest has been added which will return all the Atlas audit 
> operations including import export at Atlas.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf14 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 8c0a6407c 
>   dashboardv2/public/js/utils/Enums.js 58542ec59 
>   dashboardv3/public/js/utils/Enums.js 58542ec59 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  b448d510b 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java 
> eb8634533 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  c9ed79750 
>   repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> be1698fb4 
>   repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 
> 02f78b369 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
>  78865c569 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
>  ba7a8a04b 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
>  925b2a007 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
>  4d438525a 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  db8c6c83c 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
>  25e0a5362 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java
>  1863b8d74 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java
>  d0c06a107 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServ

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

2019-12-04 Thread Sidharth Mishra 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/
---

(Updated Dec. 4, 2019, 9:47 p.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
Subramanian.


Bugs: ATLAS-3518
https://issues.apache.org/jira/browse/ATLAS-3518


Repository: atlas


Description
---

Right now for purge, delete entity, etc. at Atlas we create audit entry at 
HBase. User can go to entity and click on audit to see the audit information 
for that particular entity. But if user purge one entity there will not be any 
vertex at Janus graph and user will not able to get the purge audit which is 
important for governance purpose.

As part of fix, implemented Atlas Audit Framework which is being used for Purge 
operation and can be extended for other operations. In Future once we change 
the AtlasAudit type as super type for Import export as well we can use this 
framework for all Atlas operations. 

admin/audit rest has been added which will return all the Atlas audit 
operations including import export at Atlas.


Diffs (updated)
-

  addons/models/-Area0/0010-base_model.json 2f5fdaf14 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c 
  dashboardv2/public/js/utils/Enums.js 58542ec59 
  dashboardv3/public/js/utils/Enums.js 58542ec59 
  intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
PRE-CREATION 
  
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java 
b448d510b 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java 
eb8634533 
  
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 c9ed79750 
  repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
  repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
be1698fb4 
  repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369 
  
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java 
PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
 78865c569 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
 ba7a8a04b 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
 925b2a007 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
 4d438525a 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
 db8c6c83c 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
 25e0a5362 
  
repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java
 1863b8d74 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java
 d0c06a107 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 c14850f43 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java
 78fdaca8d 
  
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
 42b63535b 
  
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java
 a1b512fb9 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
 7a1ed18cb 
  
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java
 48b958271 
  
repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java
 c9bb11c56 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
 0ffc3d595 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 18e950e8c 
  
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java
 050bda3ae 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
 6f9c05e7a 
  
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java
 e56e48308 
  repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
baeafd4b7 
  repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java 
PRE-CREATION 
  server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
40fd6efdc 
  webapp/src/test/java/org/apache/a

[jira] [Assigned] (ATLAS-3549) Add a new REST endpoint to get EntityHeader using unique attributes

2019-12-04 Thread Mandar Ambawane (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandar Ambawane reassigned ATLAS-3549:
--

Assignee: Mandar Ambawane

> Add a new REST endpoint to get EntityHeader using unique attributes
> ---
>
> Key: ATLAS-3549
> URL: https://issues.apache.org/jira/browse/ATLAS-3549
> Project: Atlas
>  Issue Type: Bug
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (ATLAS-3549) Add a new REST endpoint to get EntityHeader using unique attributes

2019-12-04 Thread Mandar Ambawane (Jira) 
Mandar Ambawane created ATLAS-3549:
--

 Summary: Add a new REST endpoint to get EntityHeader using unique 
attributes
 Key: ATLAS-3549
 URL: https://issues.apache.org/jira/browse/ATLAS-3549
 Project: Atlas
  Issue Type: Bug
Reporter: Mandar Ambawane






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (ATLAS-3548) Support multiple ServiceTypes for one entity type

2019-12-04 Thread charles shen (Jira) 
charles shen created ATLAS-3548:
---

 Summary: Support multiple ServiceTypes for one entity type
 Key: ATLAS-3548
 URL: https://issues.apache.org/jira/browse/ATLAS-3548
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: charles shen


There are cases where one entity type can belong to more than one ServiceType, 
eg, [AWS 
EFS|[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEFS.html]] can 
belong to ServiceType AWS and FILE_SYSTEM.

The possible change is to change the ServiceType from String to Array.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)