Re: Review Request 72895: ATLAS-3950 : Read Type Auth : Classification, Business metadata , Entity types are able to have attributes of type which are not permissible to read

2020-09-29 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72895/#review221973
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 29, 2020, 10:53 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72895/
> ---
> 
> (Updated Sept. 29, 2020, 10:53 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3950
> https://issues.apache.org/jira/browse/ATLAS-3950
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> User is able to create an attribute for 
> businessmetadata/classification/entitydef of typename which doesn't have read 
> access in the ranger policy. UI works fine but but through REST , User is 
> able to add any type as attribute typename.
> 
> Steps to reproduce:
> 
> User has following rights in ranger policy :
> 
> CRUD permissions on hive_table type only
> 
> CRUD permissions on all business_metadata type.
> 
> hrt_qa creates a business metadata bm1
> 
> hrt_qa is able to create an attribute for bm1 - say attrib1 which allows 
> Applicable types to be anything. UI displays only hive_table but through REST 
> , hrt_qa is able to add any type as Applicable type.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  2cb2b4789 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
>  6b4fa65ae 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
>  93e7012c2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityDefStoreV2.java
>  e5153de0e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipDefStoreV2.java
>  64ef18bf7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasStructDefStoreV2.java
>  9a45f00c7 
> 
> 
> Diff: https://reviews.apache.org/r/72895/diff/15/
> 
> 
> Testing
> ---
> 
> Tested with following JSON : 
> 
> Entity typedef creation with an attribute of typename which admin doesn't 
> have type-read rights for:
> {
> "enumDefs": [],
> "structDefs": [],
> "classificationDefs": [],
> "entityDefs": [
>   {
> "category": "ENTITY",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1537261952180,
> "updateTime": 1537262097732,
> "version": 3,
> "name": "testtab",
> "description": "sport",
> "typeVersion": "1.4",
> "attributeDefs": [
>   {
> "name": "test_name",
> "typeName": "string",
> "isOptional": false,
> "cardinality": "SINGLE",
> "valuesMinCount": 1,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> },
> {
> "name": "year",
> "typeName": "int",
> "isOptional": false,
> "cardinality": "SINGLE",
> "valuesMinCount": 1,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> },
> {
> "name": "place",
> "typeName": "array",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> }
> ],
> "superTypes": [
>   "DataSet"
> ],
> "subTypes": []
> }
> ],
> "relationshipDefs": []
> }
> 
> Classification typedef creation with entitytypes for which doesn't have 
> type-read rights for:
> 
> {
>   "classificationDefs": [
> {
>   "name": "testclass",
>   "description": "oif",
>   "superTypes": [],
>   "attributeDefs": [],
&

Re: Review Request 72912: ATLAS-3962 : Include business metadata def header in typdefs headers API

2020-09-29 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72912/#review221972
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 29, 2020, 5:03 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72912/
> ---
> 
> (Updated Sept. 29, 2020, 5:03 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3962
> https://issues.apache.org/jira/browse/ATLAS-3962
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Bug details:- While trying to fetch headers of all types, data about business 
> metadata is not fetched via "atlas/v2/types/typedefs/headers" .We could 
> verify is there is business metadata in 
> "api/atlas/v2/types/typedefs?type=business_metadata" though
> 
> The patch provided fix to add business metadata def header in typdefs headers.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 9abcd3449 
> 
> 
> Diff: https://reviews.apache.org/r/72912/diff/1/
> 
> 
> Testing
> ---
> 
> Tested output of atlas/v2/types/typedefs/headers and 
> api/atlas/v2/types/typedefs?type=business_metadata API
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72895: ATLAS-3950 : Read Type Auth : Classification, Business metadata , Entity types are able to have attributes of type which are not permissible to read

2020-09-28 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72895/#review221970
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityDefStoreV2.java
Lines 95 (patched)
<https://reviews.apache.org/r/72895/#comment311035>

- Attribute of 
struct-def/classification-def/relationship-def/business-metadata-def also need 
to be checked similarly. It will be useful to add another 
verifyTypeReadAccess() method, with Collection(AttributeDef attributeDefs), and 
call from relevant places
- in addition, please review handling the same in update() call as well.


- Madhan Neethiraj


On Sept. 28, 2020, 6:04 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72895/
> ---
> 
> (Updated Sept. 28, 2020, 6:04 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3950
> https://issues.apache.org/jira/browse/ATLAS-3950
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> User is able to create an attribute for 
> businessmetadata/classification/entitydef of typename which doesn't have read 
> access in the ranger policy. UI works fine but but through REST , User is 
> able to add any type as attribute typename.
> 
> Steps to reproduce:
> 
> User has following rights in ranger policy :
> 
> CRUD permissions on hive_table type only
> 
> CRUD permissions on all business_metadata type.
> 
> hrt_qa creates a business metadata bm1
> 
> hrt_qa is able to create an attribute for bm1 - say attrib1 which allows 
> Applicable types to be anything. UI displays only hive_table but through REST 
> , hrt_qa is able to add any type as Applicable type.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  2cb2b4789 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
>  6b4fa65ae 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
>  9ffede4e3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityDefStoreV2.java
>  e5153de0e 
> 
> 
> Diff: https://reviews.apache.org/r/72895/diff/10/
> 
> 
> Testing
> ---
> 
> Tested with following JSON : 
> 
> Entity typedef creation with an attribute of typename which admin doesn't 
> have type-read rights for:
> {
> "enumDefs": [],
> "structDefs": [],
> "classificationDefs": [],
> "entityDefs": [
>   {
> "category": "ENTITY",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1537261952180,
> "updateTime": 1537262097732,
> "version": 3,
> "name": "testtab",
> "description": "sport",
> "typeVersion": "1.4",
> "attributeDefs": [
>   {
> "name": "test_name",
> "typeName": "string",
> "isOptional": false,
> "cardinality": "SINGLE",
> "valuesMinCount": 1,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> },
> {
> "name": "year",
> "typeName": "int",
> "isOptional": false,
> "cardinality": "SINGLE",
> "valuesMinCount": 1,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> },
> {
> "name": "place",
> "typeName": "array",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> }
> ],
> "superTypes": [
>   "DataSet"
> ],
> "subTypes": []
> }
> ],
> "relationshipDefs": []
> }
> 
> Classification typedef creation with entitytypes for which doesn't have 
> type-read rights for:
> 
> {
>

Re: Review Request 72895: ATLAS-3950 : Read Type Auth : Classification, Business metadata , Entity types are able to have attributes of type which are not permissible to read

2020-09-28 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72895/#review221966
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
Lines 63 (patched)
<https://reviews.apache.org/r/72895/#comment311033>

#63: consider adding null check for 'types'
#65: consider adding null check for 'def'



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
Lines 102 (patched)
<https://reviews.apache.org/r/72895/#comment311034>

Instead of creating "Collection entityTypes" here, 
consider adding method:
  class AtlasAbstractDefStoreV2 {
public void verifyTypeReadAccess(Collection typeNames) {
  if (typeNames != null) {
for (String typeName : typeNames) {
  AtlasBaseTypeDef def = typeRegistry.getTypeDefByName(typeName);
  
  if (def != null) {
AtlasAuthorizationUtils.verifyAccess(new 
AtlasTypeAccessRequest(AtlasPrivilege.TYPE_READ, def), "read type-def of 
category ", def.getCategory(), " ", def.getName());
  }
}
  }
}

and replace #96, #97, #102 - #110 with a call to 
verifyTypeReadAccess(classificationDef.getEntityTypes());

Same applies for AtlasEntityDefStoreV2.create() as well.


- Madhan Neethiraj


On Sept. 28, 2020, 8:11 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72895/
> ---
> 
> (Updated Sept. 28, 2020, 8:11 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3950
> https://issues.apache.org/jira/browse/ATLAS-3950
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> User is able to create an attribute for 
> businessmetadata/classification/entitydef of typename which doesn't have read 
> access in the ranger policy. UI works fine but but through REST , User is 
> able to add any type as attribute typename.
> 
> Steps to reproduce:
> 
> User has following rights in ranger policy :
> 
> CRUD permissions on hive_table type only
> 
> CRUD permissions on all business_metadata type.
> 
> hrt_qa creates a business metadata bm1
> 
> hrt_qa is able to create an attribute for bm1 - say attrib1 which allows 
> Applicable types to be anything. UI displays only hive_table but through REST 
> , hrt_qa is able to add any type as Applicable type.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  2cb2b4789 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
>  6b4fa65ae 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
>  9ffede4e3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityDefStoreV2.java
>  e5153de0e 
> 
> 
> Diff: https://reviews.apache.org/r/72895/diff/8/
> 
> 
> Testing
> ---
> 
> Tested with following JSON : 
> 
> Entity typedef creation with an attribute of typename which admin doesn't 
> have type-read rights for:
> {
> "enumDefs": [],
> "structDefs": [],
> "classificationDefs": [],
> "entityDefs": [
>   {
> "category": "ENTITY",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1537261952180,
> "updateTime": 1537262097732,
> "version": 3,
> "name": "testtab",
> "description": "sport",
> "typeVersion": "1.4",
> "attributeDefs": [
>   {
> "name": "test_name",
> "typeName": "string",
> "isOptional": false,
> "cardinality": "SINGLE",
> "valuesMinCount": 1,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> },
> {
> "name": "year",
> "typeName": "int",
> "isOptional": false,
> "cardinality": "SINGLE",
> "valuesMinCount": 1,
> "valuesMaxCount": 1,
> "isUnique": fals

Re: Review Request 72892: ATLAS-3951 : Authorize entity-type for type-read permission before creating entity of that type.

2020-09-25 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72892/#review221962
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
Lines 1140 (patched)
<https://reviews.apache.org/r/72892/#comment311027>

#1141 below validates that the user has ENTITY_CREATE permission. I think 
additional validation for type-read permission is unnecessary; to restrict an 
user from creating entities, there should be ENTITY_CREATE policies in place.


- Madhan Neethiraj


On Sept. 22, 2020, 12:17 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72892/
> ---
> 
> (Updated Sept. 22, 2020, 12:17 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3951
> https://issues.apache.org/jira/browse/ATLAS-3951
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Bug :- User is able to create entities of type though he doesn't have read 
> permissions on the type.
> 
> This patch updates the createOrUpdate entities flow to add type-read 
> authorization for entity-type of the entity tobe created.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  7cf77ea04 
> 
> 
> Diff: https://reviews.apache.org/r/72892/diff/1/
> 
> 
> Testing
> ---
> 
> Tried Creating entities with non authorized and authorized entity types.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72894: ATLAS-3952 :- Authorize Super And SubTypes and depend entityType for type-read access while creating Classificationdef

2020-09-25 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72894/#review221961
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 25, 2020, 8:53 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72894/
> ---
> 
> (Updated Sept. 25, 2020, 8:53 a.m.)
> 
> 
> Review request for atlas, chaitali, Jayendra Parab, Madhan Neethiraj, Nikhil 
> Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3952
> https://issues.apache.org/jira/browse/ATLAS-3952
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch consist two fixes.
> 1) Authorize Super And SubTypes and entityType for type-read access before 
> creating Classificationdef.
> 2) Move create-type access check before duplicate "Given type _ already 
> exists" validation.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 4a79b6fd5 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
>  9ffede4e3 
> 
> 
> Diff: https://reviews.apache.org/r/72894/diff/4/
> 
> 
> Testing
> ---
> 
> Tested creating classification while superTypes/ subTypes and entityType with 
> valid/invalid read-Type access.
> Testing duplicate classification while valid/invalid create-type access.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72895: ATLAS-3950 : Read Type Auth : Classification, Business metadata , Entity types are able to have attributes of type which are not permissible to read

2020-09-25 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72895/#review221960
---




intg/src/main/java/org/apache/atlas/type/AtlasBusinessMetadataType.java
Lines 168 (patched)
<https://reviews.apache.org/r/72895/#comment311026>

I suggest to avoid this method; instead add following version of 
verifyTypeReadAccess() in AtlasAbstractDefStoreV2; and call this new method 
from AtlasBusinessMetadataDefStoreV2,  AtlasClassificationDefStoreV2 and 
AtlasEntityDefStoreV2:

  class AtlasAbstractDefStoreV2 {
public void verifyTypeReadAccess(Collection types) 
throws AtlasBaseException {
  for (AtlasType type : types) {
AtlasBaseTypeDef def = 
typeRegistry.getTypeDefByName(type.getTypeName());
AtlasAuthorizationUtils.verifyAccess(new 
AtlasTypeAccessRequest(AtlasPrivilege.TYPE_READ, def), "read type-def of 
category ", def.getCategory(), " ", def.getName());
  }
  }


- Madhan Neethiraj


On Sept. 23, 2020, 4:34 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72895/
> ---
> 
> (Updated Sept. 23, 2020, 4:34 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3950
> https://issues.apache.org/jira/browse/ATLAS-3950
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> User is able to create an attribute for 
> businessmetadata/classification/entitydef of typename which doesn't have read 
> access in the ranger policy. UI works fine but but through REST , User is 
> able to add any type as attribute typename.
> 
> Steps to reproduce:
> 
> User has following rights in ranger policy :
> 
> CRUD permissions on hive_table type only
> 
> CRUD permissions on all business_metadata type.
> 
> hrt_qa creates a business metadata bm1
> 
> hrt_qa is able to create an attribute for bm1 - say attrib1 which allows 
> Applicable types to be anything. UI displays only hive_table but through REST 
> , hrt_qa is able to add any type as Applicable type.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasBusinessMetadataType.java 
> f17300140 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  2cb2b4789 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
>  6b4fa65ae 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
>  9ffede4e3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityDefStoreV2.java
>  e5153de0e 
> 
> 
> Diff: https://reviews.apache.org/r/72895/diff/7/
> 
> 
> Testing
> ---
> 
> Tested with following JSON : 
> 
> Entity typedef creation with an attribute of typename which admin doesn't 
> have type-read rights for:
> {
> "enumDefs": [],
> "structDefs": [],
> "classificationDefs": [],
> "entityDefs": [
>   {
> "category": "ENTITY",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1537261952180,
> "updateTime": 1537262097732,
> "version": 3,
> "name": "testtab",
> "description": "sport",
> "typeVersion": "1.4",
> "attributeDefs": [
>   {
> "name": "test_name",
> "typeName": "string",
> "isOptional": false,
> "cardinality": "SINGLE",
> "valuesMinCount": 1,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> },
> {
> "name": "year",
> "typeName": "int",
> "isOptional": false,
> "cardinality": "SINGLE",
> "valuesMinCount": 1,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false,
> "includeInNotification": false
> },
> {
> "name": "place",
> "typeName": "array",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
&g

Review Request 72906: ATLAS-3934: Dockerfile: updated default to build from local repo (instead of from github)

2020-09-23 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72906/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, Sarath Subramanian, 
and Sidharth Mishra.


Bugs: ATLAS-3934
https://issues.apache.org/jira/browse/ATLAS-3934


Repository: atlas


Description
---

- added 'src' volume in docker-compose-atlas-build.yml, so that build uses 
local repo
- to address slow access to mounted volumes in MacOS, added "delegated" to .m2 
and src volumes. Ref: https://docs.docker.com/docker-for-mac/osxfs-caching/


Diffs
-

  dev-support/atlas-docker/Dockerfile.atlas-build 58d127a1e 
  dev-support/atlas-docker/README.md 21f9b7548 
  dev-support/atlas-docker/docker-compose.atlas-build.yml a1bc2f7b7 
  dev-support/atlas-docker/scripts/atlas-build.sh b920034dc 


Diff: https://reviews.apache.org/r/72906/diff/1/


Testing
---

- verified build succeeds with local repo (in host)
- verified build succeeds with repo downloaded in container (from github)


Thanks,

Madhan Neethiraj



Re: Review Request 72894: ATLAS-3952 :- Authorize Super And SubTypes and depend entityType for type-read access while creating Classificationdef

2020-09-23 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72894/#review221940
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
Lines 75 (patched)
<https://reviews.apache.org/r/72894/#comment311008>

This will add entityTypes to superTypes of the classification. I suggest to 
replace #74 - #80 with the following:
  verifyTypeReadAccess(classificationDef.getSuperTypes());
  verifyTypeReadAccess(classificationDef.getEntityTypes());

And make sure to handle null as parameter value in verifyTypeReadAccess().


- Madhan Neethiraj


On Sept. 23, 2020, 6:42 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72894/
> ---
> 
> (Updated Sept. 23, 2020, 6:42 p.m.)
> 
> 
> Review request for atlas, chaitali, Jayendra Parab, Madhan Neethiraj, Nikhil 
> Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3952
> https://issues.apache.org/jira/browse/ATLAS-3952
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch consist two fixes.
> 1) Authorize Super And SubTypes and entityType for type-read access before 
> creating Classificationdef.
> 2) Move create-type access check before duplicate "Given type _ already 
> exists" validation.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 4a79b6fd5 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
>  9ffede4e3 
> 
> 
> Diff: https://reviews.apache.org/r/72894/diff/3/
> 
> 
> Testing
> ---
> 
> Tested creating classification while superTypes/ subTypes and entityType with 
> valid/invalid read-Type access.
> Testing duplicate classification while valid/invalid create-type access.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72894: ATLAS-3952 :- Authorize Super And SubTypes and depend entityType for type-read access while creating Classificationdef

2020-09-22 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72894/#review221921
---




intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java
Line 753 (original)
<https://reviews.apache.org/r/72894/#comment310974>

I suggest to not remove this validation from here. BTW, has this validation 
been moved to else where?



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
Lines 77 (patched)
<https://reviews.apache.org/r/72894/#comment310971>

'subTypes' is a read-only/computed field. It is ignored during 
create/update operations. Please remove subTypes from 'typeNames'.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
Lines 80 (patched)
<https://reviews.apache.org/r/72894/#comment310972>

verifyReadAccessForSuperAndSubTypes() => verifyTypeReadAccess()



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
Lines 379 (patched)
<https://reviews.apache.org/r/72894/#comment310973>

When multiple classifications are created in one REST call, 
typeRegistry.getTypeDefByName(typeName) will return null when typeName is one 
of the newly created type. For example:

  {
"classificationDefs": [
  {
"name": "tagBase"
  },
  {
"name": "tagDerived",
"superTypes": [ "typeBase" ]
  }
]
  }

Please validate above.


- Madhan Neethiraj


On Sept. 22, 2020, 10:59 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72894/
> -------
> 
> (Updated Sept. 22, 2020, 10:59 a.m.)
> 
> 
> Review request for atlas, chaitali, Jayendra Parab, Madhan Neethiraj, Nikhil 
> Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3952
> https://issues.apache.org/jira/browse/ATLAS-3952
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch consist two fixes.
> 1) Authorize Super And SubTypes and entityType for type-read access before 
> creating Classificationdef.
> 2) Move create-type access check before duplicate "Given type _ already 
> exists" validation.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 4a79b6fd5 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2.java
>  9ffede4e3 
> 
> 
> Diff: https://reviews.apache.org/r/72894/diff/1/
> 
> 
> Testing
> ---
> 
> Tested creating classification while superTypes/ subTypes and entityType with 
> valid/invalid read-Type access.
> Testing duplicate classification while valid/invalid create-type access.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72890: ATLAS-3949 : Relationship search API, add parameter to get classification attributes in search results

2020-09-21 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72890/#review221918
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 21, 2020, 1:26 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72890/
> ---
> 
> (Updated Sept. 21, 2020, 1:26 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3949
> https://issues.apache.org/jira/browse/ATLAS-3949
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Improvement:** Adding boolean parameter 'includeClassificationAttributes' 
> in Relationship search Api to get all attributes of the classification 
> associated to the entity.
> 
> **Example of api:** 
> /api/atlas/v2/search/relationship?guid={guid}=tables=10=true
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  4d97a0859 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  f8143453a 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java dd9980a82 
> 
> 
> Diff: https://reviews.apache.org/r/72890/diff/1/
> 
> 
> Testing
> ---
> 
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/27
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Review Request 72879: ATLAS-3946 : Filter TypeDefs in Metrics API and show data accordingly

2020-09-19 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72879/#review221903
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 19, 2020, 5:21 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72879/
> ---
> 
> (Updated Sept. 19, 2020, 5:21 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3946
> https://issues.apache.org/jira/browse/ATLAS-3946
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Filter TypeDefs in Metrics API and show data accordingly, currently data is 
> shown for all types even if user do not have access to those types.
> 
> This patch filterTypeDefs after authorization and updates the data for 
> required types.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> 7917f32e5 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java 
> e74a9e93a 
> 
> 
> Diff: https://reviews.apache.org/r/72879/diff/4/
> 
> 
> Testing
> ---
> 
> Tested Statistics UI from different users with different policies for 
> entityType and classfication.
> 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/25/console
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72879: ATLAS-3946 : Filter TypeDefs in Metrics API and show data accordingly

2020-09-18 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72879/#review221892
---




repository/src/main/java/org/apache/atlas/services/MetricsService.java
Lines 147 (patched)
<https://reviews.apache.org/r/72879/#comment310951>

entityDef.subTypes includes only direct sub-types. To get sub-types in all 
levels, consider the following:

  AtlasEntityType entityType = 
typeRegistry.getEntityType(entityDef.getName();
  
  for (String type : entityType.getTypeAndAllSubTypes()) {
... earlier implementation
  }


- Madhan Neethiraj


On Sept. 18, 2020, 8:59 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72879/
> ---
> 
> (Updated Sept. 18, 2020, 8:59 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3946
> https://issues.apache.org/jira/browse/ATLAS-3946
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Filter TypeDefs in Metrics API and show data accordingly, currently data is 
> shown for all types even if user do not have access to those types.
> 
> This patch filterTypeDefs after authorization and updates the data for 
> required types.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> 7917f32e5 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java 
> e74a9e93a 
> 
> 
> Diff: https://reviews.apache.org/r/72879/diff/3/
> 
> 
> Testing
> ---
> 
> Tested Statistics UI from different users with different policies for 
> entityType and classfication.
> 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/25/console
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72879: ATLAS-3946 : Filter TypeDefs in Metrics API and show data accordingly

2020-09-17 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72879/#review221888
---




repository/src/main/java/org/apache/atlas/services/MetricsService.java
Line 129 (original), 139 (patched)
<https://reviews.apache.org/r/72879/#comment310950>

Shouldn't entityDefs be used here, instead of 
typeRegistry.getAllEntityTypes()? Please review.


- Madhan Neethiraj


On Sept. 17, 2020, 5:24 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72879/
> ---
> 
> (Updated Sept. 17, 2020, 5:24 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3946
> https://issues.apache.org/jira/browse/ATLAS-3946
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Filter TypeDefs in Metrics API and show data accordingly, currently data is 
> shown for all types even if user do not have access to those types.
> 
> This patch filterTypeDefs after authorization and updates the data for 
> required types.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> 7917f32e5 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java 
> e74a9e93a 
> 
> 
> Diff: https://reviews.apache.org/r/72879/diff/2/
> 
> 
> Testing
> ---
> 
> Tested Statistics UI from different users with different policies for 
> entityType and classfication.
> 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/25/console
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72879: ATLAS-3946 : Filter TypeDefs in Metrics API and show data accordingly

2020-09-17 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72879/#review221885
---


Fix it, then Ship it!





repository/src/main/java/org/apache/atlas/services/MetricsService.java
Lines 241 (patched)
<https://reviews.apache.org/r/72879/#comment310949>

Instead of parameter 'typesDef', consider returning from this method:
  AtlasTypesDef getTypesDef() {
AtlasTypesDef ret = new AtlasTypesDef();

...

return ret;
  }


- Madhan Neethiraj


On Sept. 17, 2020, 9:26 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72879/
> ---
> 
> (Updated Sept. 17, 2020, 9:26 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3946
> https://issues.apache.org/jira/browse/ATLAS-3946
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Filter TypeDefs in Metrics API and show data accordingly, currently data is 
> shown for all types even if user do not have access to those types.
> 
> This patch filterTypeDefs after authorization and updates the data for 
> required types.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> 7917f32e5 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java 
> e74a9e93a 
> 
> 
> Diff: https://reviews.apache.org/r/72879/diff/1/
> 
> 
> Testing
> ---
> 
> Tested Statistics UI from different users with different policies for 
> entityType and classfication.
> 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/25/console
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72880: ATLAS-3947 : Skip authorization for read of _ALL_ENTITY_TYPES and _ALL_CLASSIFICATION_TYPES types

2020-09-17 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72880/#review221884
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 17, 2020, 10:28 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72880/
> ---
> 
> (Updated Sept. 17, 2020, 10:28 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3947
> https://issues.apache.org/jira/browse/ATLAS-3947
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Skip authorization for read of _ALL_ENTITY_TYPES type of entity type and 
> _ALL_CLASSIFICATION_TYPES type of classification type.
> This are internally required by for system attributes search.
> 
> This patch Skip authorization for read of _ALL_ENTITY_TYPES entityType and 
> and _ALL_CLASSIFICATION_TYPES classificationType.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  458ec49ea 
> 
> 
> Diff: https://reviews.apache.org/r/72880/diff/1/
> 
> 
> Testing
> ---
> 
> Tested API's with no policies on types for a user. _ALL_ENTITY_TYPES and 
> _ALL_CLASSIFICATION_TYPES are loaded correctly after it.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



[ANNOUNCE] New Committer: Nikhil Bonte

2020-09-16 Thread Madhan Neethiraj
Atlas community,

 

Apache Atlas PMC has asked Nikhil Bonte to become a committer and we are 
pleased to announce that he has accepted.

 

Welcome Nikhil, great to have you on board.

 

On behalf of Apache Atlas PMC,

 

Madhan



Re: Review Request 72873: ATLAS-3938 : Deletion of non existing hive entities

2020-09-16 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72873/#review221880
---


Fix it, then Ship it!





client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Line 666 (original), 668 (patched)
<https://reviews.apache.org/r/72873/#comment310948>

Adding a paramter to existing method will break current users of this API. 
I suggest to retain the current method, and introduce a new one with additional 
parameter:

   public AtlasSearchResult basicSearch(String typeName, String 
classification, String query, boolean excludeDeletedEntities, int limit, int 
offset) throws AtlasServiceException {
 return this.basicSearch(typeName, null, classification, query, 
excludeDeletedEntities, limit, offset);
   }


- Madhan Neethiraj


On Sept. 16, 2020, 4:26 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72873/
> ---
> 
> (Updated Sept. 16, 2020, 4:26 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3938
> https://issues.apache.org/jira/browse/ATLAS-3938
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is 
> not enabled, we dont have anyway to get the database and table sync with hive.
> 
> **Workaround:** Added support to delete hive entities in Atlas which are 
> dropped in hive.
> 
> **Usage:** ./import-hive.sh -deleteNonExisting
> 
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
> pageSize/limit while fetching entities from Atlas
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/bin/import-hive.sh b1660ea65 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  24e06dc2b 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 18811f8c7 
> 
> 
> Diff: https://reviews.apache.org/r/72873/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Request to add to Contributor list.

2020-09-16 Thread Madhan Neethiraj
Deep S,

Thanks for your interest in contributing to Apache Atlas. You have been added 
as a contributor. Welcome to Apache Atlas community!

Regards,
Madhan

On 9/16/20, 11:38 AM, "Deep Singh"  wrote:

Hi Atlas Team,

I am interested in contributing to the Apache Atlas project, kindly add me
as a contributor.
My Apache Jira id is deepam1982

Thanks and Regards,
DeepS

On Tue, Sep 15, 2020 at 1:50 PM Deep Singh  wrote:

> Hi Atlas Team,
>
> I am interested in contributing to the Apache Atlas project, kindly add me
> as a contributor.
> My email id is deepam1...@gmail.com
>
> .
>
> Thanks and Regards,
> DeepS
>




Re: Review Request 72869: ATLAS-3942 : Refactoring AtlasTypeDefGraphStore code to remove authz check reading typedef while delete-type

2020-09-16 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72869/#review221867
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 16, 2020, 6:32 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72869/
> ---
> 
> (Updated Sept. 16, 2020, 6:32 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3942
> https://issues.apache.org/jira/browse/ATLAS-3942
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Problem : delete-type is triggering read-type authz access internally when 
> called by method getByName. due to this delete-type access is depend on 
> read-type access.
> This patch propose refactoring of AtlasTypeDefGraphStore code to remove authz 
> check reading typedef internally.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  0b1317ad6 
> 
> 
> Diff: https://reviews.apache.org/r/72869/diff/2/
> 
> 
> Testing
> ---
> 
> Tested delete classfication when user do not have read access has deny access 
> to update type.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



[jira] [Commented] (ATLAS-3906) “NoSuchMethodErrors” due to multiple versions of org.apache.curator:curator-client:jar

2020-09-14 Thread Madhan Neethiraj (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195501#comment-17195501
 ] 

Madhan Neethiraj commented on ATLAS-3906:
-

[~Bing-ok] - thanks for the detailed description and the suggestions to address 
the issue. Upgrading version of org.apache.hadoop:hadoop-common from 3.1.1 to 
3.3.0 seems like a good option. Can you please submit a review with this fix?

> “NoSuchMethodErrors” due to multiple versions of 
> org.apache.curator:curator-client:jar
> --
>
> Key: ATLAS-3906
> URL: https://issues.apache.org/jira/browse/ATLAS-3906
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Bing-ok
>Priority: Major
>
> h1. Issue description
> Hi, there are multiple versions of _*org.apache.curator:curator-client*_ in 
> _*atlas/server-api*_. As shown in the following dependency tree, library  
> _*org.apache.curator:curator-client:2.12.0*_  is transitively introduced by 
> *_org.apache.hadoop:hadoop-auth:3.1.1_*, but has been managed to be version 
> *_4.0.1_*. 
> However, one method defined in shadowed version 
> _*org.apache.curator:curator-client:2.12.0*_ is referenced by client project 
> via *_org.apache.hadoop:hadoop-auth:3.1.1_*, but missing in the actually 
> loaded version _*org.apache.curator:curator-client:4.0.1*_.
> For instance, the following missing method(defined in 
> _*org.apache.curator:curator-client:2.12.0*_) is actually referenced by 
> _*atlas/server-api*_, which will introduce a runtime error(i.e., 
> "NoSuchMethodError") into _*atlas/server-api*_.
>  _*Missing method: org.codelibs.core.lang.StringUtil: java.lang.String 
> newStringUnsafe(char[])*_ is invoked by _*atlas/server-api*_ via the 
> following path:
> {noformat}
> paths--
>  getEntity(java.lang.String)> atlas\server-api\target\classes
>  java.lang.Object get(java.lang.Object)> 
> Repositories\org\apache\curator\curator-client\4.0.1\curator-client-4.0.1.jar
>  apply(java.lang.Object)> 
> Repositories\org\apache\curator\curator-framework\4.0.1\curator-framework-4.0.1.jar
>  apply(org.apache.curator.framework.api.CuratorListener)> 
> Repositories\org\apache\curator\curator-framework\4.0.1\curator-framework-4.0.1.jar
>  checkInterrupted(java.lang.Throwable)>{noformat}
> h1. Suggested fixing solutions
>  1. Upgrade dependency *_org.apache.hadoop:hadoop-common_* from _*3.1.1*_ to 
> *_3.3.0_*. Because the newer version 
> *_org.apache.hadoop:hadoop-common:3.3.0_* does not invoke the above missing 
> method, such upgrading can solve the problem.
>  2. Declare a direct dependency _*org.apache.curator:curator-client:2.12.0*_ 
> in the pom file of _*atlas/server-api*_, to override 
> _*org.apache.curator:curator-client*_'s managed version.
> 3. Use configuration attribute  to unify the version of 
> library _*org.apache.curator:curator-client*_ to be _*2.12.0*_ in 
> _*atlas/server-api*_'s pom file.
> Please let me know which solution do you prefer? I can submit a PR to fix it.
> Thank you very much for your attention.
>  Best regards,
> h1. Dependency tree
> {noformat}
> [INFO] org.apache.atlas:atlas-server-api:jar:3.0.0-SNAPSHOT
> [INFO] - org.apache.hadoop:hadoop-common:jar:3.1.1:compile
> [INFO] +- org.apache.hadoop:hadoop-auth:jar:3.1.1:compile
> [INFO] | - org.apache.curator:curator-framework:jar:4.0.1:compile (version 
> managed from 2.12.0)
> [INFO] | - _*(org.apache.curator:curator-client:jar:4.0.1:compile - version 
> managed from 2.12.0; omitted for duplicate)*_
> [INFO] - _*org.apache.curator:curator-client:jar:4.0.1:compile*_{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Review Request 72865: ATLAS-3934: docker scripts to build and run Apache Atlas in containers

2020-09-11 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72865/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, keval bhatt, Mehul 
Parikh, Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, 
and Sidharth Mishra.


Bugs: ATLAS-3934
https://issues.apache.org/jira/browse/ATLAS-3934


Repository: atlas


Description
---

- added docker files, docker-compose files and scripts to build Atlas in a 
container. The image to build Atlas supports following volumns:
-- scripts: build scripts in the local directory will be made available in the 
container
-- dist:archives from the build will be copied into this volume
-- .m2: if provided, build uses the give .m2 directory, which can help 
avoid large downloads
-- patches: patche files, if present, will be applied on top of the specified 
github branch
-- src: if provided, builds from the sources in this directory (instead of 
from sources from github)

- added docker file, docker-compose file and scripts to install and run Atlas 
in a container.


Diffs
-

  dev-support/atlas-docker/.dockerignore PRE-CREATION 
  dev-support/atlas-docker/Dockerfile.atlas PRE-CREATION 
  dev-support/atlas-docker/Dockerfile.atlas-base PRE-CREATION 
  dev-support/atlas-docker/Dockerfile.atlas-build PRE-CREATION 
  dev-support/atlas-docker/README.md 5296f05ba 
  dev-support/atlas-docker/dist/.gitignore PRE-CREATION 
  dev-support/atlas-docker/docker-compose.atlas-base.yml PRE-CREATION 
  dev-support/atlas-docker/docker-compose.atlas-build.yml PRE-CREATION 
  dev-support/atlas-docker/docker-compose.atlas.yml PRE-CREATION 
  dev-support/atlas-docker/patches/.gitignore PRE-CREATION 
  dev-support/atlas-docker/scripts/atlas-build.sh PRE-CREATION 
  dev-support/atlas-docker/scripts/atlas.sh PRE-CREATION 


Diff: https://reviews.apache.org/r/72865/diff/1/


Testing
---

- verified build with the following command:
   docker-compose -f docker-compose.atlas-base.yml -f 
docker-compose.atlas-build.yml up

- verified install and running of Atlas with the following command:
   docker-compose -f docker-compose.atlas-base.yml -f docker-compose.atlas.yml 
up -d

- Ran quick start successfully


Thanks,

Madhan Neethiraj



Re: Review Request 72863: ATLAS-3939: added build profile berkeley-solr

2020-09-11 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72863/
---

(Updated Sept. 12, 2020, 12:15 a.m.)


Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Sarath Subramanian, 
and Sidharth Mishra.


Summary (updated)
-

ATLAS-3939: added build profile berkeley-solr


Bugs: ATLAS-3939
https://issues.apache.org/jira/browse/ATLAS-3939


Repository: atlas


Description
---

added build profile berkeley-solr


Diffs (updated)
-

  distro/.gitignore PRE-CREATION 
  distro/pom.xml b92bd3548 
  distro/src/bin/atlas_config.py 2d6143331 
  distro/src/bin/atlas_start.py 963faf402 
  distro/src/bin/atlas_stop.py 2e15c0f3f 
  docs/src/documents/Setup/BuildInstruction.md 05f98fb9b 
  docs/src/documents/Setup/InstallationInstruction.md fdff3a57e 
  webapp/pom.xml 792b16445 


Diff: https://reviews.apache.org/r/72863/diff/2/

Changes: https://reviews.apache.org/r/72863/diff/1-2/


Testing
---

- built Atlas with profile berkeley-solr
- deployed Atlas built above, started and successfully ran quick start


Thanks,

Madhan Neethiraj



[jira] [Created] (ATLAS-3939) Add build profile for berkeleydb-solr

2020-09-11 Thread Madhan Neethiraj (Jira)
Madhan Neethiraj created ATLAS-3939:
---

 Summary: Add build profile for berkeleydb-solr
 Key: ATLAS-3939
 URL: https://issues.apache.org/jira/browse/ATLAS-3939
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj


Atlas supports a number of build profiles, like 
external-hbase-sol/embedded-hbase-solr/embedded-cassandra-solr/berkeley-elasticsearch,
 for various combinations of backend stores an index stores. This should be 
extended to support berkeley-solr as well; this profile can be useful for 
dev/testing of Atlas.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (ATLAS-3807) Release Atlas 2.1.0

2020-09-10 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj resolved ATLAS-3807.
-
Fix Version/s: 2.1.0
   Resolution: Fixed

> Release Atlas 2.1.0
> ---
>
> Key: ATLAS-3807
> URL: https://issues.apache.org/jira/browse/ATLAS-3807
> Project: Atlas
>  Issue Type: Task
>Affects Versions: 2.1.0
>    Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 2.1.0
>
>
> parent task to capture release tasks for Apache Atlas 2.1.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72821: ATLAS-3935 Use Audit framework to capture audit entries for Import/Export operations

2020-09-09 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72821/#review221828
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 9, 2020, 10:04 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72821/
> ---
> 
> (Updated Sept. 9, 2020, 10:04 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3935
> https://issues.apache.org/jira/browse/ATLAS-3935
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> By using Audit Framework, capture audit entries for Import/Export operations.
> 
> These audit entries can be seen under the "Audits" tab of Admin section.
> 
> Add Import and Export specific information in Audit entry
> 
> Total no. of entities imported/ exported
> Total no. of entities imported/ exported of specific TypeDef
> Export operation Fetch Type
> 
> 
> Diffs
> -
> 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 3a5ae5c 
> 
> 
> Diff: https://reviews.apache.org/r/72821/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72821: ATLAS-3935 Use Audit framework to capture audit entries for Import/Export operations

2020-09-08 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72821/#review221815
---




webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 417 (patched)
<https://reviews.apache.org/r/72821/#comment310902>

why only log FetchTypeOptionValue in audit? Consider including entire 
AtlasExportRequest.options.



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 418 (patched)
<https://reviews.apache.org/r/72821/#comment310901>

atlasObjectIds => objectIds



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 470 (patched)
<https://reviews.apache.org/r/72821/#comment310903>

atlasObjectIds => objectIds



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 755 (patched)
<https://reviews.apache.org/r/72821/#comment310898>

atlasObjectIds => objectIds



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 760 (patched)
<https://reviews.apache.org/r/72821/#comment310899>

typeDefCountMap => entityCountByType
  this is about entity-count by their typename, right?



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 761 (patched)
<https://reviews.apache.org/r/72821/#comment310900>

resultCount - wouldn't this be simply the length of the list argument 
atlasObjectIds?


- Madhan Neethiraj


On Sept. 8, 2020, 2:18 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72821/
> ---
> 
> (Updated Sept. 8, 2020, 2:18 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3935
> https://issues.apache.org/jira/browse/ATLAS-3935
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> By using Audit Framework, capture audit entries for Import/Export operations.
> 
> These audit entries can be seen under the "Audits" tab of Admin section.
> 
> Add Import and Export specific information in Audit entry
> 
> Total no. of entities imported/ exported
> Total no. of entities imported/ exported of specific TypeDef
> Export operation Fetch Type
> 
> 
> Diffs
> -
> 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 3a5ae5c 
> 
> 
> Diff: https://reviews.apache.org/r/72821/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



[DRAFT] Board report for Apache Atlas: September 2020

2020-09-07 Thread Madhan Neethiraj
Atlas team,

Please review the draft board report below and send your feedback/comments.

Thanks,
Madhan

## Description:
Apache Atlas is a scalable and extensible set of core foundational governance
services that enables enterprises to effectively and efficiently meet their
compliance requirements within Hadoop and allows integration with the complete
enterprise data ecosystem.

## Issues:
There are no issues requiring board attention at this time.

## Membership Data:
Apache Atlas was founded 2017-06-20 (3 years ago)
There are currently 42 committers and 32 PMC members in this project.
The Committer-to-PMC ratio is roughly 4:3.

Community changes, past quarter:
 - No new PMC members. Last addition was Ashutosh Mestry on 2019-04-16.
 - Nikhil P Bonte was added as committer on 2020-09-03

## Project Activity:
 - Atlas community released 2.1.0, more than a year in making, with a number
   of new features, enhancements and fixes, like:
  - quick search, with type-ahead suggestions
  - business Metadata to enable augmenting entity-types with additional
attributes, search entities using these attributes
  - ability to add labels on entities, and search entities using labels
  - ability to add entity instance specific custom attributes i.e. attributes
not defined in entity-def or business metadata
  - REST APIs to purge deleted entities
  - ability to find entities by more than one classification
  - support to process notifications from multiple Kafka topics
  - model updates to capture Spark, AWS S3, ADLS-Gen2 entities
  - performance, UI improvements
 - the community is working on enhancements and fixes for next release
  - audit enhancements to track entity-type create/update/data
  - authorization enhancements to support type-read permission
  - support for date-range in search filters
  - working on adding Java and Python APIs

## Community Health:
 - dev@atlas.apache.org had a 28% decrease in traffic in the past quarter
   (1254 emails compared to 1734)
 - u...@atlas.apache.org had a 50% increase in traffic in the past quarter (30
   emails compared to 20)
 - 96 issues opened in JIRA, past quarter (-46% decrease)
 - 60 issues closed in JIRA, past quarter (-81% decrease)
 - 123 commits in the past quarter (-58% decrease)
 - 18 code contributors in the past quarter (-21% decrease)
 - 7 PRs opened on GitHub, past quarter (-22% decrease)
 - 4 PRs closed on GitHub, past quarter (-55% decrease)

## Releases:
  2.1.0was released on 07/16/2020
  0.8.4was released on 06/21/2019
  1.2.0was released on 06/12/2019
  2.0.0was released on 05/14/2019
  0.8.3was released on 10/31/2018
  1.1.0was released on 09/17/2018
  1.0.0was released on 06/02/2018
  0.8.2was released on 02/05/2018
  1.0.0-alpha  was released on 01/25/2018
  0.8.1was released on 08/29/2017
  0.8-incubating   was released on 03/16/2017
  0.7.1-incubating was released on 01/26/2017
  0.7-incubating   was released on 07/09/2016
  0.6-incubating   was released on 12/31/2015
  0.5-incubating   was released on 07/11/2015





Re: Review Request 72815: Deletion of non existing hive entities

2020-09-04 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/#review221808
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 5, 2020, 5:02 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> ---
> 
> (Updated Sept. 5, 2020, 5:02 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is 
> not enabled, we dont have anyway to get the database and table sync with hive.
> 
> **Workaround:** Added support to delete hive entities in Atlas which are 
> dropped in hive.
> 
> **Usage:** ./import-hive.sh -deleteNonExisting
> 
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
> pageSize/limit while fetching entities from Atlas
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  e659ca041 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 6968e8358 
> 
> 
> Diff: https://reviews.apache.org/r/72815/diff/3/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Review Request 72815: Deletion of non existing hive entities

2020-09-04 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/#review221806
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 862 (patched)
<https://reviews.apache.org/r/72815/#comment310888>

Since there is only one criteria in the filter, there is no need for 
innerFc. Consider the following:

  SearchParameters.FilterCriteria fc = new 
SearchParameters.FilterCriteria();
 
  fc.setAttributeName(ATTRIBUTE_CLUSTER_NAME);
  fc.setAttributeValue(clusterName);
  fc.setOperator(SearchParameters.Operator.EQ);

Also, the filter criteira is the same across all iteration of the for loop 
at #858. Consider setting up the criteira before once before entering the for 
loop.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 959 (patched)
<https://reviews.apache.org/r/72815/#comment310890>

Returning after deleting one entity? Please review.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 968 (patched)
<https://reviews.apache.org/r/72815/#comment310892>

deleteHiveMetadata() => deleteEntitiesForNonExistingHiveMetadata()



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 999 (patched)
<https://reviews.apache.org/r/72815/#comment310891>

There is only one parameter to replace, as LOG.error() will print execption 
details for 'e'. Please review and remove ', {}' at the end of the message.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 1006 (patched)
<https://reviews.apache.org/r/72815/#comment310889>

Consider replacing:
 Queue  => List
     LinkedList => ArrayList


- Madhan Neethiraj


On Sept. 3, 2020, 12:21 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> ---
> 
> (Updated Sept. 3, 2020, 12:21 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is 
> not enabled, we dont have anyway to get the database and table sync with hive.
> 
> **Workaround:** Added support to delete hive entities in Atlas which are 
> dropped in hive.
> 
> **Usage:** ./import-hive.sh -deleteNonExisting
> 
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
> pageSize/limit while fetching entities from Atlas
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  e659ca041 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 6968e8358 
> 
> 
> Diff: https://reviews.apache.org/r/72815/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



[jira] [Assigned] (ATLAS-3934) Dockerfile should build from local repository rather than pulling code from git

2020-08-31 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj reassigned ATLAS-3934:
---

Assignee: Madhan Neethiraj

> Dockerfile should build from local repository rather than pulling code from 
> git
> ---
>
> Key: ATLAS-3934
> URL: https://issues.apache.org/jira/browse/ATLAS-3934
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Syed Atif Akhtar
>    Assignee: Madhan Neethiraj
>Priority: Trivial
>  Labels: atlas, build, development_environment, docker
>
> *Expected Behaviour:*
>  When a developer wants to test out their changes using the dockerfile in dev 
> support, they should be able to create the dockerfile with atlas built and 
> mount from a local directory with the changes they have done.
> *Current Behaviour:*
> The dockerfile currently defined in dev-support pulls the source code from 
> git instead of building the code that the user has defined
>  
> {code:java}
> RUN git clone https://github.com/apache/atlas.git -b master{code}
>  
> *Suggestion:*
>  Maintain separate dockerfiles for publishing an image of Atlas vs development



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72815: ENGESC-3520: Deletion of non existing hive entities

2020-08-28 Thread Madhan Neethiraj
ionUtils.isNotEmpty(guidsToDelete)) {
try {
  deleteByGuid(guidsToDelete);
} catch (AtlasServiceException e) {
  LOG.error("Failed to delete Atlas entities for database {} , {}", 
hiveDbName, e);

  if(failOnError) {
throw e;
  }
}
  }


- Madhan Neethiraj


On Aug. 28, 2020, 1:33 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> ---
> 
> (Updated Aug. 28, 2020, 1:33 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is 
> not enabled, we dont have anyway to get the database and table sync with hive.
> 
> **Workaround:** Added support to delete hive entities in Atlas which are 
> dropped in hive.
> 
> **Usage:** ./import-hive.sh -deleteNonExisting
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  e659ca041 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 6968e8358 
> 
> 
> Diff: https://reviews.apache.org/r/72815/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Atlas Contributor Request

2020-08-26 Thread Madhan Neethiraj
Dharshana,

Thanks for your interest in contributing to Apache Atlas. You have been added 
as a contributor. Welcome to Apache Atlas community!

Regards,
Madhan


On 8/26/20, 8:13 AM, "Dharshana M KrishnaMoorthy"  
wrote:

Hi Atlas Team,

I am interested in contributing to the Apache Atlas project, kindly add me
as a
contributor.

My Jira id is dharshmk.

Thanks and Regards,
Dharshana MK




Re: Review Request 72477: ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, UPDATE and DELETE

2020-08-14 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72477/#review221585
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
Lines 413 (patched)
<https://reviews.apache.org/r/72477/#comment310654>

Given this method returns true only when all entries are of type 
ENTITY_TYPE_AUDIT_ENTRY, it will be efficient traverse the list only until 
another type is found. Consider the following:
  private static final Predicate 
PRED_IS_NOT_TYPE_AUDIT_ENTITY = obj -> 
!obj.getTypeName().equals(AtlasAuditService.ENTITY_TYPE_AUDIT_ENTRY);

  private boolean skipAuditEntries(List entityHeaders) {
return CollectionUtils.isEmpty(entityHeaders) || 
!entityHeaders.stream().anyMatch(PRED_IS_NOT_TYPE_AUDIT_ENTITY);
      }


- Madhan Neethiraj


On Aug. 14, 2020, 2:06 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72477/
> ---
> 
> (Updated Aug. 14, 2020, 2:06 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3583
> https://issues.apache.org/jira/browse/ATLAS-3583
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, 
> UPDATE and DELETE
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 9777b7a 
>   
> addons/models/-Area0/patches/006-base_model_add_atlas_operation_attributes.json
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> a95cf4e 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  a0dc816 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
>  8e7c1b3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  0dc3193 
>   test-tools/src/main/resources/solr/core-template/solrconfig.xml 8ebbeff 
> 
> 
> Diff: https://reviews.apache.org/r/72477/diff/8/
> 
> 
> Testing
> ---
> 
> Basic testing is done.
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1871/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1885/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1888/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2071/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2082/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2100/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72714: ATLAS-3898 :- Authorize read of typedefs.

2020-08-14 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72714/#review221584
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 14, 2020, 1:35 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72714/
> ---
> 
> (Updated Aug. 14, 2020, 1:35 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3898
> https://issues.apache.org/jira/browse/ATLAS-3898
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently in the Atlas-Ranger plugin for types resource READ permission is 
> not available and read access is available by default to all types of any 
> category.
> 
> This patch authorize read for all typedefs. It implicitly add read permission 
> if create,update or delete permission is assinged to user.
> 
> a)Instrumented authorization code to read classification typedef endpoints in 
> Atlas code with create permission.
> b)Added logAudit property in AtlasRequest to skip auditing for filtering 
> calls.
> 
> 
> Diffs
> -
> 
>   
> authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationUtils.java
>  460b454b9 
>   authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizer.java 
> 95d69e685 
>   authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java 
> 7287b3dd7 
>   
> authorization/src/main/java/org/apache/atlas/authorize/AtlasTypesDefFilterRequest.java
>  PRE-CREATION 
>   
> authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
>  734991691 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  79f52700b 
> 
> 
> Diff: https://reviews.apache.org/r/72714/diff/2/
> 
> 
> Testing
> ---
> 
> Done end to end testing with Atlas with SimpleAuthorizer for getAll typedefs 
> API, and type def API with name & guid.
> 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2110
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient

2020-08-12 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72698/#review221556
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 4, 2020, 4:27 p.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
> 
> (Updated Aug. 4, 2020, 4:27 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Using this project users can get an idea as how to integrate with Atlas using 
> AtlasCleint. This helps the user to understand the basic rest functionality 
> of Atlas such as
> 
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
> 
> 
> Diffs
> -
> 
>   atlas-examples/pom.xml PRE-CREATION 
>   atlas-examples/sample-app/README.md PRE-CREATION 
>   atlas-examples/sample-app/pom.xml PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/DiscoveryExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/EntityExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/GlossaryExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/LineageExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleApp.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleAppConstants.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
>  PRE-CREATION 
>   atlas-examples/sample-app/src/main/resources/atlas-application.properties 
> PRE-CREATION 
>   pom.xml 5e0442ae5 
> 
> 
> Diff: https://reviews.apache.org/r/72698/diff/7/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>



Re: Review Request 72756: ATLAS-3918 : Regression : Renaming table has exceptions in Atlas application logs

2020-08-11 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72756/#review221551
---



For some DDLs like 'create view', HMS doesn't have all details. So, it is 
critical that we process DDLs in HiveServer2.

Consider adding a new message-type ENTITY_RENAME, and for such messages Atlas 
should ignore if source entity is not found.

- Madhan Neethiraj


On Aug. 11, 2020, 3:21 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72756/
> ---
> 
> (Updated Aug. 11, 2020, 3:21 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3918
> https://issues.apache.org/jira/browse/ATLAS-3918
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Issue : Messages are being sent for DDL operations one from HMS and 
> HiveServer2 which causes duplicate messages hence it throws exception:
> 
>  message 
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":1594802485264,"message":{"type":"ENTITY_PARTIAL_UPDATE_V2","user":"hrt_qa","entityId":{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.t1@cm"}},"entity":{"entity":{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"MANAGED_TABLE","temporary":false,"lastAccessTime":1594802469000,"aliases":["t1"],"createTime":1594802469000,"qualifiedName":"default.t2@cm","name":"t2","comment":null,"parameters":{"last_modified_time":"1594802481","rawDataSize":"0","numRows":"0","transient_lastDdlTime":"1594802481","bucketing_version":"2","last_modified_by":"hrt_qa","numFilesErasureCoded":"0","totalSize":"0","transactional_properties":"default","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}}","numFiles":"0","transactional":"true"},"retention":0},"guid":"-21782793831367970","isIncomplete":fal
 se,"provenanceType":0,"version":0,"proxy":false 
>  
> This patch fix it by providing a property "DDL_MESSAGES_ENABLE" in 
> configuration when enabled the HiveServer2 will send ddl messages and if 
> disabled then it will only send non ddl messages.
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 651323490 
> 
> 
> Diff: https://reviews.apache.org/r/72756/diff/3/
> 
> 
> Testing
> ---
> 
> Tested by creating and renaming the hive_table hence ddl messages are not 
> sent from hiveserver2 after the property is set to false and so no exception 
> is thrown: 
> > create table t1(id int);
> 
> > alter table t1 rename to t2;
> 
> 
> Thanks,
> 
> chaitali
> 
>



Re: Review Request 72477: ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, UPDATE and DELETE

2020-08-11 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72477/#review221545
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 11, 2020, 5:25 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72477/
> ---
> 
> (Updated Aug. 11, 2020, 5:25 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3583
> https://issues.apache.org/jira/browse/ATLAS-3583
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, 
> UPDATE and DELETE
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 9777b7a 
>   
> addons/models/-Area0/patches/006-base_model_add_atlas_operation_attributes.json
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> a95cf4e 
>   intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java 
> 9120062 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 2b9cf6e 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  a0dc816 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
>  8e7c1b3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  0dc3193 
>   repository/src/test/java/org/apache/atlas/TestModules.java a298934 
>   test-tools/src/main/resources/solr/core-template/solrconfig.xml 8ebbeff 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestTypeDefsREST.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72477/diff/7/
> 
> 
> Testing
> ---
> 
> Basic testing is done.
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1871/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1885/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1888/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2071/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2082/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2100/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72477: ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, UPDATE and DELETE

2020-08-08 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72477/#review221511
---




repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
Lines 47 (patched)
<https://reviews.apache.org/r/72477/#comment310558>

atlasAuditService => auditService
  - avoid 'atlas' prefix in variable/member names



repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
Lines 76 (patched)
<https://reviews.apache.org/r/72477/#comment310554>

add following condition as well:
  CollectionUtils.isNotEmpty(updatedTypes)



repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
Lines 77 (patched)
<https://reviews.apache.org/r/72477/#comment310555>

typeNames => createdTypeNames



repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
Lines 81 (patched)
<https://reviews.apache.org/r/72477/#comment310556>

Which use case results in duplicate entries in updatedTypes? If no 
duplicates are expected, simply return updatedTypes here. In fact, given 
'updatedTypes' is modified above (#79), consider not returning any value from 
this method.



repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
Lines 90 (patched)
<https://reviews.apache.org/r/72477/#comment310557>

atlasBaseTypeDefList => typeDefs
  - avoid 'atlas' prefix in variable names



repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
Lines 94 (patched)
<https://reviews.apache.org/r/72477/#comment310560>

clientId => clientIp



repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
Lines 106 (patched)
<https://reviews.apache.org/r/72477/#comment310561>

Can you add details of the audit log generated? Computing 
groupByCategoryMap may not be useful; it should be enough to use following 
string as audit entry result:
  AtlasJson.toJson(typeDefs);
  
Audit entry params can be set to the list of typeNames.



repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
Lines 108 (patched)
<https://reviews.apache.org/r/72477/#comment310559>

AtlasAuthorizationUtils.getCurrentUserName() => 
RequestContext.get().getUser()



repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Lines 725 (patched)
<https://reviews.apache.org/r/72477/#comment310552>

I suggest to create a PatchHandler for enum-def updates, instead of 
updating AddAttributePatchHandler:

  class UpdateEnumDefPatchHandler extends PatchHandler {
public UpdateEnumDefPatchHandler(AtlasTypeDefStore typeDefStore, 
AtlasTypeRegistry typeRegistry) {
  super(typeDefStore, typeRegistry, new String[] { "UPDATE_ENUMDEF" });
}

@Override
public PatchStatus applyPatch(TypeDefPatch patch) throws 
AtlasBaseException {
  ...
}
  }



repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
Line 354 (original), 354 (patched)
<https://reviews.apache.org/r/72477/#comment310553>

CollectionUtils.size(obj) handles null value. Please review if following 
changes are necessary:
 - #354 - #359
 - #376 - #381


- Madhan Neethiraj


On Aug. 8, 2020, 2:42 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72477/
> -------
> 
> (Updated Aug. 8, 2020, 2:42 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3583
> https://issues.apache.org/jira/browse/ATLAS-3583
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, 
> UPDATE and DELETE
> 
> 
> Diffs
> -
> 
>   
> addons/models/-Area0/patches/006-base_model_add_atlas_operation_attributes.json
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> a95cf4e 
>   intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java 
> 9120062 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 2b9cf6e 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  a0dc816 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/a

Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

2020-08-07 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72713/#review221503
---


Fix it, then Ship it!





repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 654 (patched)
<https://reviews.apache.org/r/72713/#comment310542>

I suggest to have endTime for THIS_MONTH as:
  endTime   = startTime.plusMonths(1).minusNanos(1);



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 667 (patched)
<https://reviews.apache.org/r/72713/#comment310546>

I suggest to support following time-range as well:

•   This quarter
startTime = now.minusMonths((now.getMonth() - 1) % 
3).withHour(0).withMinute(0).withSecond(0).withNano(0);
endTime   = startTime.plusMonths(3).minusNanos(1);
•   This year
startTime = 
now.withMonth(1).withHour(0).withMinute(0).withSecond(0).withNano(0);
endTime   = startTime.plusYears(1).minusNanos(1);
•   Last quarter
startTime = now.minusMonths((now.getMonth() - 1) % 
3).minusMonths(3).withHour(0).withMinute(0).withSecond(0).withNano(0);
endTime   = startTime.plusMonths(3).minusNanos(1);
•   Last year
startTime = 
now.minusYears(1).withMonth(1).withHour(0).withMinute(0).withSecond(0).withNano(0);
endTime   = startTime.plusYears(1).minusNanos(1);
•   Last 3 months
startTime = 
now.minusMonths(3).withDayOfMonth(1).withHour(0).withMinute(0).withSecond(0).withNano(0);
endTime   = startTime.plusMonths(3).minusNanos(1);
•   Last 6 months
startTime = 
now.minusMonths(6).withDayOfMonth(1).withHour(0).withMinute(0).withSecond(0).withNano(0);
endTime   = startTime.plusMonths(6).minusNanos(1);
•   Last 12 months
startTime = 
now.minusMonths(12).withDayOfMonth(1).withHour(0).withMinute(0).withSecond(0).withNano(0);
endTime   = startTime.plusMonths(12).minusNanos(1);



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 210 (patched)
<https://reviews.apache.org/r/72713/#comment310544>

"->" -> "<=="


- Madhan Neethiraj


On Aug. 7, 2020, 9:32 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72713/
> ---
> 
> (Updated Aug. 7, 2020, 9:32 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3892
> https://issues.apache.org/jira/browse/ATLAS-3892
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1.This patch implements the search history feature where from UI we will 
> receive below filters.
> 2.In backend we have introduced a new operator "TIME_RANGE" where all the 
> below 7 attribute values will have TIME_RANGE operator.
> 3.There will be a slight change in the JSON where previously we were passing 
> the Attribute value in Search parameters from the UI but for below filters   
> we will not pass any attributeValue except for "custom range" filter we 
> provide attribute value 
>   for eg ;"attributeValue":"159354180,159544260" .
> 
> New 7 filters :
> 
> 1.last 7 days.
> 2.Last month.
> 3.Last 30 days.
> 4.Today.
> 5.Yesterday.
> 6.Custom Range.
> 7.This Month.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java a71787bc0 
>   intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> fcc4494f2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> c5e3d6f0b 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> cf158cd99 
>   
> repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
>  7b40c2184 
> 
> 
> Diff: https://reviews.apache.org/r/72713/diff/7/
> 
> 
> Testing
> ---
> 
> Tested with below json: 
> eg : 
> {
>   "excludeDeletedEntities": true,
>   "includeSubClassifications": true,
>   "includeSubTypes": true,
>   "includeClassificationAttributes": true,
>   "entityFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "__timestamp",
>   "operator": "TIME_RANGE",
>   "attributeValue": "YESTERDAY"
>   }
> ]
>   },
>   "tagFilters": nul

Review Request 72746: ATLAS-3912: moved classes from intg to repository or common module

2020-08-06 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72746/
---

Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, Sarath Subramanian, and Sidharth Mishra.


Summary (updated)
-

ATLAS-3912: moved classes from intg to repository or common module


Bugs: ATLAS-3912
https://issues.apache.org/jira/browse/ATLAS-3912


Repository: atlas


Description (updated)
---

moved classes from intg to repository or common module


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/entitytransform/Action.java  
  
intg/src/main/java/org/apache/atlas/entitytransform/AtlasEntityTransformer.java 
 
  intg/src/main/java/org/apache/atlas/entitytransform/BaseEntityHandler.java  
  intg/src/main/java/org/apache/atlas/entitytransform/Condition.java  
  intg/src/main/java/org/apache/atlas/entitytransform/EntityAttribute.java  
  
intg/src/main/java/org/apache/atlas/entitytransform/HdfsPathEntityHandler.java  
  
intg/src/main/java/org/apache/atlas/entitytransform/HiveColumnEntityHandler.java
  
  
intg/src/main/java/org/apache/atlas/entitytransform/HiveDatabaseEntityHandler.java
  
  
intg/src/main/java/org/apache/atlas/entitytransform/HiveStorageDescriptorEntityHandler.java
  
  
intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java 
 
  
intg/src/main/java/org/apache/atlas/entitytransform/TransformationConstants.java
  
  intg/src/main/java/org/apache/atlas/entitytransform/TransformerContext.java  
  intg/src/main/java/org/apache/atlas/listener/ChangedTypeDefs.java  
  intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java  
  intg/src/main/java/org/apache/atlas/listener/TypeDefChangeListener.java  
  intg/src/main/java/org/apache/atlas/pc/StatusReporter.java  
  intg/src/main/java/org/apache/atlas/pc/WorkItemBuilder.java  
  intg/src/main/java/org/apache/atlas/pc/WorkItemConsumer.java  
  intg/src/main/java/org/apache/atlas/pc/WorkItemManager.java  
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java  
  intg/src/main/java/org/apache/atlas/store/DeleteType.java de5e5e435 
  
intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java
  
  intg/src/test/java/org/apache/atlas/pc/StatusReporterTest.java  
  intg/src/test/java/org/apache/atlas/pc/WorkItemConsumerTest.java  
  intg/src/test/java/org/apache/atlas/pc/WorkItemConsumerWithResultsTest.java  
  intg/src/test/java/org/apache/atlas/pc/WorkItemManagerTest.java  
  intg/src/test/java/org/apache/atlas/pc/WorkItemManagerWithResultsTest.java  
  intg/src/test/resources/json/entity1.json  
  intg/src/test/resources/json/transform1.json  
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerDelegate.java
 e414a8c39 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 06341a6d4 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 9f8c4c120 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/SoftReferenceTest.java
 2ffc972ab 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreHardDeleteV2Test.java
 623235f4b 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreSoftDeleteV2Test.java
 b169988d0 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2Test.java
 24683a504 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/InverseReferenceUpdateHardDeleteV2Test.java
 ed72d5f0b 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/InverseReferenceUpdateSoftDeleteV2Test.java
 92d1ddcc2 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/InverseReferenceUpdateV2Test.java
 6364fd44b 
  server-api/src/main/java/org/apache/atlas/RequestContext.java c5cbf0546 
  webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 7e6698158 
  webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityRESTDelete.java 
6e604f536 


Diff: https://reviews.apache.org/r/72746/diff/1/


Testing (updated)
---

Pre-commit tests run: 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2090


Thanks,

Madhan Neethiraj



[jira] [Created] (ATLAS-3912) move classes to appropriate modules

2020-08-06 Thread Madhan Neethiraj (Jira)
Madhan Neethiraj created ATLAS-3912:
---

 Summary: move classes to appropriate modules
 Key: ATLAS-3912
 URL: https://issues.apache.org/jira/browse/ATLAS-3912
 Project: Atlas
  Issue Type: Improvement
  Components: atlas-intg
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj


Classes that are unrelated to integration of Atlas clients should be moved out 
of intg module, for example:
 - org.apache.atlas.entitytransform
 - org.apache.atlas.pc
 - org.apache.atlas.listener
 - org.apache.atlas.store

Classes in these packages should be move to modules like repository or common.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72644: ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name

2020-08-05 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72644/#review221479
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 5, 2020, 5:37 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72644/
> ---
> 
> (Updated Aug. 5, 2020, 5:37 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3872
> https://issues.apache.org/jira/browse/ATLAS-3872
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Cause of the problem:
> Type-def Column is created by quick-start script.
> 
> Here in this case, Type-def Column has the attribute "name". And it has 
> property isOptional = true
> 
> Type-def Column is a child of Type-def Asset which also has the attribute 
> "name". And it has property isOptional = false
> 
> Now while creating Entity of Type-def Column from UI, it does not show 
> attribute "name" in the mandatory fields because for Type-def Column its 
> property isOptional = true
> 
> But while validating the attributes from server side we iterate through all 
> the attributes of Entity type (Including Parent's attributes)
> In this case, while validating Type-def Asset's attribute "name" (which has 
> property isOptional = false) we get an error because its value is sent as 
> "null" from UI side.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java a14e43a 
>   intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 
> e0843cb 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java b0d88fc 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 2b9cf6e 
>   intg/src/test/java/org/apache/atlas/model/ModelTestUtil.java 5df9525 
>   repository/src/test/resources/logging-v1-full.zip 
> 69c54eec000964a1f6f3fef96b26cca188a28ba1 
>   repository/src/test/resources/sales-v1-full.zip 
> 07afbf6ecae312a840b54cae5ab01335c513d180 
>   repository/src/test/resources/salesNewTypeAttrs-next.zip 
> e4c8505be1587fb7cc8b6354bdda701e9557da99 
>   repository/src/test/resources/salesNewTypeAttrs.zip 
> a4467e641fffb10189288fb2f26dc4fbc06bbd42 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 82a9f5c 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java d5cf9b7 
>   webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
> 6db6b58 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java
>  0cd707f 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
>  7c820e7 
> 
> 
> Diff: https://reviews.apache.org/r/72644/diff/6/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2007/console
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2014/console
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2083/console
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2088/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72644: ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name

2020-08-05 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72644/#review221474
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 4, 2020, 6:14 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72644/
> ---
> 
> (Updated Aug. 4, 2020, 6:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3872
> https://issues.apache.org/jira/browse/ATLAS-3872
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Cause of the problem:
> Type-def Column is created by quick-start script.
> 
> Here in this case, Type-def Column has the attribute "name". And it has 
> property isOptional = true
> 
> Type-def Column is a child of Type-def Asset which also has the attribute 
> "name". And it has property isOptional = false
> 
> Now while creating Entity of Type-def Column from UI, it does not show 
> attribute "name" in the mandatory fields because for Type-def Column its 
> property isOptional = true
> 
> But while validating the attributes from server side we iterate through all 
> the attributes of Entity type (Including Parent's attributes)
> In this case, while validating Type-def Asset's attribute "name" (which has 
> property isOptional = false) we get an error because its value is sent as 
> "null" from UI side.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java a14e43a 
>   intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 
> e0843cb 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee458 
>   repository/src/test/resources/sales-v1-full.zip 
> 07afbf6ecae312a840b54cae5ab01335c513d180 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 82a9f5c 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java d5cf9b7 
>   webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
> 6db6b58 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
>  7c820e7 
> 
> 
> Diff: https://reviews.apache.org/r/72644/diff/5/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2007/console
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2014/console
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2083/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-04 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221471
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 5, 2020, 4:27 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated Aug. 5, 2020, 4:27 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 044179d44 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/6/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2084
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-04 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221470
---


Fix it, then Ship it!





intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Line 19 (original), 19 (patched)
<https://reviews.apache.org/r/72646/#comment310487>

Only import related changes are in AtlasEntityType.java. Please review and 
revert.


- Madhan Neethiraj


On Aug. 4, 2020, 11:01 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated Aug. 4, 2020, 11:01 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 044179d44 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/5/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: to be a contributor

2020-08-02 Thread Madhan Neethiraj
Dan,

Thanks for your interest in contributing to Apache Atlas. You have been added 
as a contributor. Welcome to Apache Atlas community!

Regards,
Madhan




On 8/1/20, 8:41 PM, "朱丹"  wrote:

Hi Guys,



I want to contribute to Apache Atlas.

Would you please give me the permission as a contributor?

My JIRA ID is wstcilct.



Thanks,

Dan





-- 
Avast 防毒软件已对此电子邮件执行病毒检查。
https://www.avast.com/antivirus




Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-01 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221445
---


Fix it, then Ship it!





repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1523 (patched)
<https://reviews.apache.org/r/72646/#comment310458>

This handles edge attributes of string type. Consider handling other 
datatypes as well in a subsequent patch.

An implementaiton similar to vertex-attribute predicate #226 - #271, which 
uses VertexAttributePredicateGenerator; perhaps, introducing 
EdgeAttributePredicateGenerator will be useful.


- Madhan Neethiraj


On July 31, 2020, 12:43 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 31, 2020, 12:43 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/4/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

2020-07-31 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72713/#review221444
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 31, 2020, 12:34 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72713/
> ---
> 
> (Updated July 31, 2020, 12:34 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3892
> https://issues.apache.org/jira/browse/ATLAS-3892
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1.This patch implements the search history feature where from UI we will 
> receive below filters.
> 2.In backend we have introduced a new operator "TIME_RANGE" where all the 
> below 7 attribute values will have TIME_RANGE operator.
> 3.There will be a slight change in the JSON where previously we were passing 
> the Attribute value in Search parameters from the UI but for below filters   
> we will not pass any attributeValue except for "custom range" filter we 
> provide attribute value 
>   for eg ;"attributeValue":"159354180,159544260" .
> 
> New 7 filters :
> 
> 1.last 7 days.
> 2.Last month.
> 3.Last 30 days.
> 4.Today.
> 5.Yesterday.
> 6.Custom Range.
> 7.This Month.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java a71787bc0 
>   intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> fcc4494f2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> c9a605355 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   
> repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
>  8e42d1700 
> 
> 
> Diff: https://reviews.apache.org/r/72713/diff/6/
> 
> 
> Testing
> ---
> 
> Tested with below json: 
> eg : 
> {
>   "excludeDeletedEntities": true,
>   "includeSubClassifications": true,
>   "includeSubTypes": true,
>   "includeClassificationAttributes": true,
>   "entityFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "__timestamp",
>   "operator": "TIME_RANGE",
>   "attributeValue": "YESTERDAY"
>   }
> ]
>   },
>   "tagFilters": null,
>   "attributes": [
> "__timestamp"
>   ],
>   "limit": 25,
>   "offset": 0,
>   "typeName": "hdfs_path",
>   "classification": null,
>   "termName": null
> }
> eg: Only For custom range filter we provide attribute value.
> {
>   "excludeDeletedEntities": true,
>   "includeSubClassifications": true,
>   "includeSubTypes": true,
>   "includeClassificationAttributes": true,
>   "entityFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "__timestamp",
>   "operator": "TIME_RANGE",
>   "attributeValue": "1595956315473,159596100"
>   }
> ]
>   },
>   "tagFilters": null,
>   "attributes": [
> "__timestamp"
>   ],
>   "limit": 25,
>   "offset": 0,
>   "typeName": "Table",
>   "classification": null,
>   "termName": null
> }
> 
> * Added Test cases to check new processDateRange method functionality
> 
> 
> Thanks,
> 
> chaitali
> 
>



Re: Review Request 72644: ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name

2020-07-31 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72644/#review221443
---




intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
Lines 723 (patched)
<https://reviews.apache.org/r/72644/#comment310457>

Consider replacing #723 - #780 with the following; and call this method 
from resolveReferencesPhase2() in AtlasEntityType and AtlasClassificationType.

  protected void ensureNoAttributeOverride(AtlasTypeRegistry typeRegistry, 
List superTypes) throws AtlasBaseException {
for (AtlasStructType superType : superTypes) {
  for (AtlasAttributeDef attributeDef : 
this.structDef.getAttributeDefs()) {
if 
(superType.getAllAttributes().containsKey(attributeDef.getName())) {
  throw new 
AtlasBaseException(AtlasErrorCode.ATTRIBUTE_NAME_ALREADY_EXISTS_IN_PARENT_TYPE, 
this.getName(), attributeDef.getName(), superType.getName());
}
  }
}
  }


- Madhan Neethiraj


On July 31, 2020, 11:56 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72644/
> ---
> 
> (Updated July 31, 2020, 11:56 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3872
> https://issues.apache.org/jira/browse/ATLAS-3872
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Cause of the problem:
> Type-def Column is created by quick-start script.
> 
> Here in this case, Type-def Column has the attribute "name". And it has 
> property isOptional = true
> 
> Type-def Column is a child of Type-def Asset which also has the attribute 
> "name". And it has property isOptional = false
> 
> Now while creating Entity of Type-def Column from UI, it does not show 
> attribute "name" in the mandatory fields because for Type-def Column its 
> property isOptional = true
> 
> But while validating the attributes from server side we iterate through all 
> the attributes of Entity type (Including Parent's attributes)
> In this case, while validating Type-def Asset's attribute "name" (which has 
> property isOptional = false) we get an error because its value is sent as 
> "null" from UI side.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java a14e43a 
>   intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 
> e0843cb 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee458 
>   repository/src/test/resources/sales-v1-full.zip 
> 07afbf6ecae312a840b54cae5ab01335c513d180 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 82a9f5c 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java d5cf9b7 
>   webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
> 6db6b58 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
>  7c820e7 
> 
> 
> Diff: https://reviews.apache.org/r/72644/diff/4/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2007/console
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2014/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-31 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221439
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 31, 2020, 3:57 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 31, 2020, 3:57 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java a942b9f93 
>   intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 63116d467 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/12/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2076/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-30 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221430
---


Fix it, then Ship it!





intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 53 (patched)
<https://reviews.apache.org/r/72666/#comment310433>

#53 is not needed, as this is handled in reset().


- Madhan Neethiraj


On July 30, 2020, 10:58 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 30, 2020, 10:58 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 63116d467 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/11/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72722: Index Consistency: Java Patch Handler Implementation

2020-07-30 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72722/#review221429
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 30, 2020, 10:49 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72722/
> ---
> 
> (Updated July 30, 2020, 10:49 p.m.)
> 
> 
> Review request for atlas, Damian Warszawski, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3907
> https://issues.apache.org/jira/browse/ATLAS-3907
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Please see JIRA for details.
> 
> Changes:
> - New: *IndexConsistencyPatch*: New java patch handler.
> - Modified: Added method: 
> *AtlasJanusGraphManagement.setConsistencyForAllIndexes* that enumerates all 
> the existing indexes and calls the *setConsistency* method.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
>  093edf978 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72722/diff/3/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> None.
> 
> **Functional tests**
> Used the script below to concurrently create same entity. See attached 
> (t.json) that contains entity payload.
> 
> Steps to test:
> 1. Deploy Atlas without this patch (or any other patch) with *setConsistency*.
> 2. Execute the script below with the payload *t1.json*.
> 3. Notice that multiple entities of *hdfs_path* type get created. Few of the 
> REST calls may fail with *AtlasSchemaViolation* exception.
> 4. Deploy this change and restart Atlas. Notice that the new patch gets 
> applied.
> 5. Modify *t1.json* by replacing e1 with e2.
> 6. Execute the script below.
> 7. Notice that only 1 entity *hdfs_path* with name e2 gets created.
> 
> *multiple-curl.sh*
> ```bash
> #!/bin/bash
> URL=
> echo ${URL}
> for i in `seq 10`; do
> curl -u admin:admin123 -X POST http://${URL}:31000/api/atlas/v2/entity/bulk 
> --header 'Content-Type: application/json' -d @./t.json &
> done
> ```
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2074/:
>  Succeeded.
> 
> 
> File Attachments
> 
> 
> entity payload
>   
> https://reviews.apache.org/media/uploaded/files/2020/07/30/62c25f8c-87cb-458d-a49e-ce41a96855de__t.json
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72722: Index Consistency: Java Patch Handler Implementation

2020-07-30 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72722/#review221426
---


Fix it, then Ship it!





graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 296 (patched)
<https://reviews.apache.org/r/72722/#comment310425>

Perhaps following condition can be added as well, to skip updates if lock 
is already set (this can happen in fresh installs):
  index.getConsistency() == ConsistencyModifier.LOCK


- Madhan Neethiraj


On July 30, 2020, 8:45 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72722/
> ---
> 
> (Updated July 30, 2020, 8:45 p.m.)
> 
> 
> Review request for atlas, Damian Warszawski, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3907
> https://issues.apache.org/jira/browse/ATLAS-3907
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Please see JIRA for details.
> 
> Changes:
> - New: *IndexConsistencyPatch*: New java patch handler.
> - Modified: Added method: 
> *AtlasJanusGraphManagement.setConsistencyForAllIndexes* that enumerates all 
> the existing indexes and calls the *setConsistency* method.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
>  093edf978 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72722/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> None.
> 
> **Functional tests**
> Used the script below to concurrently create same entity. See attached 
> (t.json) that contains entity payload.
> 
> Steps to test:
> 1. Deploy Atlas without this patch (or any other patch) with *setConsistency*.
> 2. Execute the script below with the payload *t1.json*.
> 3. Notice that multiple entities of *hdfs_path* type get created. Few of the 
> REST calls may fail with *AtlasSchemaViolation* exception.
> 4. Deploy this change and restart Atlas. Notice that the new patch gets 
> applied.
> 5. Modify *t1.json* by replacing e1 with e2.
> 6. Execute the script below.
> 7. Notice that only 1 entity *hdfs_path* with name e2 gets created.
> 
> *multiple-curl.sh*
> ```bash
> #!/bin/bash
> URL=
> echo ${URL}
> for i in `seq 10`; do
> curl -u admin:admin123 -X POST http://${URL}:31000/api/atlas/v2/entity/bulk 
> --header 'Content-Type: application/json' -d @./t.json &
> done
> ```
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2074/:
>  Succeeded.
> 
> 
> File Attachments
> 
> 
> entity payload
>   
> https://reviews.apache.org/media/uploaded/files/2020/07/30/62c25f8c-87cb-458d-a49e-ce41a96855de__t.json
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-30 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221422
---




intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 59 (patched)
<https://reviews.apache.org/r/72666/#comment310410>

set length to 0 here?
  this.length = 0;



repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
Lines 81 (patched)
<https://reviews.apache.org/r/72666/#comment310411>

withInitial() be called only on the first access from a thread. Calling 
reset() from here is not useful.



repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
Line 93 (original), 100 (patched)
<https://reviews.apache.org/r/72666/#comment310412>

AUDIT_EVENTS_BUFFER.get() will not call ThreadLocal.withInitial(), hence 
will not have the list reset - #81. As suggested earlier, consider using a 
method like:
  FixedBufferList getAuditEventsList() {
FixedBufferList ret = AUDIT_EVENTS_BUFFER.get();

ret.reset();

return ret;
  }



webapp/src/main/java/org/apache/atlas/notification/EntityNotificationSender.java
Line 127 (original), 127 (patched)
<https://reviews.apache.org/r/72666/#comment310413>

This file has only whitespace changes. Please revert.


- Madhan Neethiraj


On July 30, 2020, 5:55 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 30, 2020, 5:55 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 63116d467 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationSender.java
>  c2ae5122b 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/10/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72722: Index Consistency: Java Patch Handler Implementation

2020-07-30 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72722/#review221421
---




graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
Lines 174 (patched)
<https://reviews.apache.org/r/72722/#comment310406>

consistency is needed only for unique-indexes, right? If yes, I suggest to 
rename setConsistencyForAllIndexes() => updateUniqueIndexesForConsistencyLock()



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 281 (patched)
<https://reviews.apache.org/r/72722/#comment310407>

why not use 'this.management'?



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 286 (patched)
<https://reviews.apache.org/r/72722/#comment310408>

consider replacing #285, #286 with this.commit().



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 298 (patched)
<https://reviews.apache.org/r/72722/#comment310409>

Add "!index.isUnique()" as well.


- Madhan Neethiraj


On July 30, 2020, 5:42 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72722/
> ---
> 
> (Updated July 30, 2020, 5:42 p.m.)
> 
> 
> Review request for atlas, Damian Warszawski, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3907
> https://issues.apache.org/jira/browse/ATLAS-3907
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Please see JIRA for details.
> 
> Changes:
> - New: *IndexConsistencyPatch*: New java patch handler.
> - Modified: Added method: 
> *AtlasJanusGraphManagement.setConsistencyForAllIndexes* that enumerates all 
> the existing indexes and calls the *setConsistency* method.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
>  093edf978 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72722/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> None.
> 
> **Functional tests**
> Used the script below to concurrently create same entity. See attached 
> (t.json) that contains entity payload.
> 
> Steps to test:
> 1. Deploy Atlas without this patch (or any other patch) with *setConsistency*.
> 2. Execute the script below with the payload *t1.json*.
> 3. Notice that multiple entities of *hdfs_path* type get created. Few of the 
> REST calls may fail with *AtlasSchemaViolation* exception.
> 4. Deploy this change and restart Atlas. Notice that the new patch gets 
> applied.
> 5. Modify *t1.json* by replacing e1 with e2.
> 6. Execute the script below.
> 7. Notice that only 1 entity *hdfs_path* with name e2 gets created.
> 
> *multiple-curl.sh*
> ```bash
> #!/bin/bash
> URL=
> echo ${URL}
> for i in `seq 10`; do
> curl -u admin:admin123 -X POST http://${URL}:31000/api/atlas/v2/entity/bulk 
> --header 'Content-Type: application/json' -d @./t.json &
> done
> ```
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2074/
> 
> 
> File Attachments
> 
> 
> entity payload
>   
> https://reviews.apache.org/media/uploaded/files/2020/07/30/62c25f8c-87cb-458d-a49e-ce41a96855de__t.json
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

2020-07-30 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72713/#review221419
---




repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 634 (patched)
<https://reviews.apache.org/r/72713/#comment310402>

localDate => now



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 635 (patched)
<https://reviews.apache.org/r/72713/#comment310403>

Is 'ts' instance needed? only use is to call method valueOf(), which is a 
static method. I suggest to remove 'ts', and replace the calls with 
"Timestamp.valueOf()"



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 642 (patched)
<https://reviews.apache.org/r/72713/#comment310405>

The end time should be 'end of today'; not the current time. I suggest the 
following:

 final LocalDateTime now = LocalDateTime.now();
 final LocalDateTime startTime;
 final LocalDateTime endTime;

 switch (attrVal) {
   case "LAST_7_DAYS":
 startTime = 
now.minusDays(6).withHour(0).withMinute(0).withSecond(0).withNano(0);
 endTime   = startTime.plusDays(7).minusNanos(1);
   break;

   case "LAST_30_DAYS":
 startTime = 
now.minusDays(29).withHour(0).withMinute(0).withSecond(0).withNano(0);
 endTime   = startTime.plusDays(30).minusNanos(1);
   break;

   case "LAST_MONTH":
 startTime = 
now.minusMonths(1).withDayOfMonth(0).withHour(0).withMinute(0).withSecond(0).withNano(0);
 endTime   = startTime.plusMonths(1).minusNanos(1);
   break;

   case "THIS_MONTH":
 startTime = 
now.withDayOfMonth(1).withHour(0).withMinute(0).withSecond(0).withNano(0);
 endTime   = startTime.plusMonths(1).minusNanos(1);
   break;

   case "TODAY":
 startTime = now.withHour(0).withMinute(0).withSecond(0).withNano(0);
 endTime   = startTime.plusDays(1).minusNanos(1);
   break;

   case "YESTERDAY":
 startTime = 
now.minusDays(1).withHour(0).withMinute(0).withSecond(0).withNano(0);
 endTime   = startTime.plusDays(1).minusNanos(1);
   break;

   default:
 startTime = null;
 endTime   = null;
   break;
 }
 
 if (startTime == null || endTime == null) {
   // parse attrVal and update
 } else {
   attrVal = Timestamp.valueOf(startTime) + ATTRIBUTE_VALUE_DELIMITER + 
Timestamp.valueOf(endTime)
 }



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 739 (patched)
<https://reviews.apache.org/r/72713/#comment310393>

rangeStart and rangeEnd are referenced only inside the block at #747. 
Please move #739, #740 inside that block.



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 754 (patched)
<https://reviews.apache.org/r/72713/#comment310394>

Throw AtlasBaseException here when attrVal doesn't have expected value. 
This will be handled at #767 (prints a warn log)



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 848 (patched)
<https://reviews.apache.org/r/72713/#comment310395>

Even for single-line blocks, use { } - for consistency.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 192 (patched)
<https://reviews.apache.org/r/72713/#comment310396>

add @Override annotation here as well



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 207 (patched)
<https://reviews.apache.org/r/72713/#comment310399>

"<-" -> "<=="



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 804 (patched)
<https://reviews.apache.org/r/72713/#comment310397>

Wouldn't this cause infinite recursion, as this calls itself? Shouldn't 
this call the existing method at #801 i.e. method that takes a single attrVal?



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Line 977 (original), 1019 (patched)
<https://reviews.apache.org/r/72713/#comment310398>

This change to constructor signature doesn't seem necessary. Please revert. 
Also, note that the new constructor at #1025 has same arguments, but in 
different order.



repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
Lines 319 (patched)
<https://reviews.apache.org/r/72713/#comment310400>

mark filtercriteriaDateRange() as private and static; move to end of this 
class.



repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
Lines 329 (patched)
<https://reviews.apache.org/r/72713/#comment310401>

attributeName is not set in filterCriteria? For completene

Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

2020-07-30 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72713/#review221417
---




repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Line 526 (original), 534 (patched)
<https://reviews.apache.org/r/72713/#comment310382>

Block #534 - #557 is executed only for TIME_RANGE operator. This doesn't 
look right, given this block was executed for all operators before thisi patch. 
Please review and update.



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 797 (patched)
<https://reviews.apache.org/r/72713/#comment310383>

Consider moving #797, #798, #807 - #815 under a  "case 
AtlasBaseTypeDef.ATLAS_TYPE_DATE:", like:

   case AtlasBaseTypeDef.ATLAS_TYPE_DATE:
 if (op == SearchParameters.Operator.TIME_RANGE && 
StringUtils.isNotEmpty(attrVal)) {
   String[] parts  = attrVal.split(ATTRIBUTE_VALUE_DELIMITER);
   String   rangeStart = parts.length > 0 ? parts[0];
   String   rangeEnd   = parts.length > 1 ? parts[1];
   
   attrValue  = StringUtils.isEmpty(rangeStart) ? null : 
Long.parseLong(rangeStart);
   attrValue2 = StringUtils.isEmpty(rangeEnd)   ? null : 
Long.parseLong(rangeEnd);
 } else {
   attrValue  = StringUtils.isEmpty(attrVal) ? null : 
Long.parseLong(attrVal);
 }



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 180 (patched)
<https://reviews.apache.org/r/72713/#comment310388>

getBETPredicate() => getInRangePredicate()



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 188 (patched)
<https://reviews.apache.org/r/72713/#comment310390>

'final' for argument is not useful, and is a distraction in reading the 
code. Please remove.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 189 (patched)
<https://reviews.apache.org/r/72713/#comment310389>

Instead of returning null, I suggest:
  return generatePredicate(attrName, attrVal, attrVal, attrClass);



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 195 (patched)
<https://reviews.apache.org/r/72713/#comment310391>

add "attrVal2 == null" as well.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Line 767 (original), 800 (patched)
<https://reviews.apache.org/r/72713/#comment310392>

Interfaces can have 'default' methods, so there is no need to change 
VertexAttributePredicateGenerator to an abstract class.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 803 (patched)
<https://reviews.apache.org/r/72713/#comment310387>

Consider renaming the new method as generateInRangePredicate()



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1008 (patched)
<https://reviews.apache.org/r/72713/#comment310385>

Mark 'value2' as final, and initialize to null in constructors at #1010 and 
#1016.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1021 (patched)
<https://reviews.apache.org/r/72713/#comment310384>

value  => rangeStart
value2 => rangeEnd



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1075 (patched)
<https://reviews.apache.org/r/72713/#comment310386>

getBETPredicate() => getInRangePredicate()


- Madhan Neethiraj


On July 30, 2020, 5:57 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72713/
> -------
> 
> (Updated July 30, 2020, 5:57 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3892
> https://issues.apache.org/jira/browse/ATLAS-3892
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1.This patch implements the search history feature where from UI we will 
> receive below filters.
> 2.In backend we have introduced a new operator "BETWEEN" where all the below 
> 7 operators will be converted to BETWEEN operator.
> 3.There will be a slight change in the JSON where previously we were passing 
> the Attribute value in Search parameters from the UI but for below filters   
> we will not pass any attributeValue except for "custom range" filter we 
> provide attribute value 
>   for eg ;"attributeValue":"159354180,159544260" .
> 
> New 7 filters :
> 
> 1.last 7 days.
> 2.Last month.
> 3.La

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-29 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221416
---




intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 539 (patched)
<https://reviews.apache.org/r/72646/#comment310381>

I suggest to move this method to AtlasAttribute:

  class AtlasAttribute {
...

public AtlasEntityType getReferencedEntityType(AtlasTypeRegistry 
typeRegistry) {
  return 
getReferencedEntityType(typeRegistry.getType(attributeDef.getTypeName()))
}
  }


- Madhan Neethiraj


On July 29, 2020, 1:41 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 29, 2020, 1:41 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/3/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Deleted Relationships definition

2020-07-29 Thread Madhan Neethiraj
Where in UI do you see the relationship in active state? Note that if the state 
of the related-entity itself is active, UI would show related entity's name as 
active. Rendering entity name as 'deleted' would be misleading, as the entity 
is still active.

Madhan



On 7/29/20, 3:22 AM, "Verdan Mahmood"  wrote:

I have a question related to the relationships.

Let's say I have two entities User and Table. and I developed a
relationship between them.
If I delete the relationship using the Relationship API, in the UI the
relationship remains the same with no difference in the color or state,
even though the realtionshipStatus is DELETED. Is this expected behaviour
or should that relation be gone from the frontend?

Can someone please confirm if this is expected behaviour or should that
relation be gone from the frontend? or am I missing something?

P.S: I do not want to delete any of the entities.

Best,
*Verdan Mahmood*
M: (+31) 655 576 560




Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-29 Thread Madhan Neethiraj
reeTextSearchProcessorTest.searchByNameSortBy:91 expected [3] but found [0]
  FreeTextSearchProcessorTest.searchTablesByName:71 expected [3] but found [0]

Tests run: 756, Failures: 15, Errors: 0, Skipped: 14


On 7/29/20, 6:11 AM, "Damian Warszawski"  wrote:



> On July 28, 2020, 10:55 p.m., Madhan Neethiraj wrote:
> > Ship It!

Can I ask you to merge it? I don't think I have permissions to merge it to 
master.


- Damian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221401
---


On July 25, 2020, 8:53 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 25, 2020, 8:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and 
Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but 
results in duplicates when race condition occurs.
> 
> 
> Reference to jira 
https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
 6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/3/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on 
our dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>





Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-29 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221413
---



What is the impact on the memory usage improvement, after FixedBufferList use 
is removed from EntityNotificationListenerV2?


intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 41 (patched)
<https://reviews.apache.org/r/72666/#comment310376>

"incrementCapacityBy == 0" -> "incrementCapacityBy <= 0"



repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
Line 278 (original), 269 (patched)
<https://reviews.apache.org/r/72666/#comment310377>

In case of any exception in instanceConverter.getAndCacheEntity(), 'events' 
list might be left with some entries - since events.toList() may not be called. 
And these entries will be used in subsequent event dispatch from this thread.

I suggest the following:
 - add methods:
   class EntityAuditListenerV2 {
 ..
 FixedBufferList getEventsList() {
   FixedBufferList ret = AUDIT_EVENTS_BUFFER.get();
 
   ret.reset();
 
   return ret;
 }
   }

   class FixedBufferList {
  ..
  public void reset() {
for (int i = 0; i < length; i++) {
  buffer.get(i).clear();
}

length = 0;
  }
  
  public List toList() {
return this.buffer.subList(0, this.length); // no need to reset 
length here
  }
   }


- Madhan Neethiraj


On July 29, 2020, 6:23 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 29, 2020, 6:23 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 63116d467 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationSender.java
>  c2ae5122b 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/9/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-28 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221401
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 25, 2020, 8:53 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 25, 2020, 8:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/3/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

2020-07-28 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72713/#review221398
---




intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
Lines 475 (patched)
<https://reviews.apache.org/r/72713/#comment310343>

- Consider renaming operator BETWEEN as TIME_RANGE
- I suggest to not model the time periods (LAST_7_DAYS, LAST_30_DAYS, .., 
CUSTOM_RANGE) as operators. Instead, these should be valid values for  operator 
TIME_RANGE.


- Madhan Neethiraj


On July 28, 2020, 5:30 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72713/
> ---
> 
> (Updated July 28, 2020, 5:30 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3892
> https://issues.apache.org/jira/browse/ATLAS-3892
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1.This patch implements the search history feature where from UI we will 
> receive below filters.
> 2.In backend we have introduced a new operator "BETWEEN" where all the below 
> 7 operators will be converted to BETWEEN operator.
> 3.There will be a slight change in the JSON where previously we were passing 
> the Attribute value in Search parameters from the UI but for below filters   
> we will not pass any attributeValue except for "custom range" filter we 
> provide attribute value 
>   for eg ;"attributeValue":"159354180,159544260" .
> 
> New 7 filters :
> 
> 1.last 7 days.
> 2.Last month.
> 3.Last 30 days.
> 4.Today.
> 5.Yesterday.
> 6.Custom Range.
> 7.This Month.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java a71787bc0 
>   intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> fcc4494f2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> c9a605355 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   
> repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
>  8e42d1700 
> 
> 
> Diff: https://reviews.apache.org/r/72713/diff/3/
> 
> 
> Testing
> ---
> 
> Tested with below json: 
> eg : 
> {
>   "excludeDeletedEntities": true,
>   "includeSubClassifications": true,
>   "includeSubTypes": true,
>   "includeClassificationAttributes": true,
>   "entityFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "__timestamp",
> "operator": "THISMONTH",
> "attributeValue": ""
>   }
> ]
>   },
>   "tagFilters": null,
>   "attributes": [
> "__timestamp"
>   ],
>   "limit": 25,
>   "offset": 0,
>   "typeName": "Table",
>   "classification": null,
>   "termName": null
> }
> eg: Only For custom range filter we provide attribute value.
> {
>   "excludeDeletedEntities": true,
>   "includeSubClassifications": true,
>   "includeSubTypes": true,
>   "includeClassificationAttributes": true,
>   "entityFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "createTime",
> "operator": "CUSTOM_RANGE",
> "attributeValue": "159354180,159544260"
>   }
> ]
>   },
>   "tagFilters": null,
>   "attributes": [
> "createTime"
>   ],
>   "limit": 25,
>   "offset": 0,
>   "typeName": "Table",
>   "classification": null,
>   "termName": null
> }
> 
> * Added Test cases to check new processDateRange method functionality
> 
> 
> Thanks,
> 
> chaitali
> 
>



Re: Asking for JIRA permissions

2020-07-28 Thread Madhan Neethiraj
Viktor,

Thanks for your interest in contributing to Apache Atlas. I added you as a 
contributor to Apache Atlas. You can now assign JIRAs to yourself, and add 
patches.

Review board (https://reviews.apache.org/) is preferred by most Atlas folks to 
review and provide feedback on the patches.

Regards,
Madhan




On 6/25/20, 2:56 AM, "Viktor Somogyi-Vass" 
 wrote:

Hi devs,

I created a JIRA [1] that I'd like to resolve (and have a PR already) but
I'd like to assign it to myself. Could you please grant me the
required permissions?
Also, do you accept PR's against the master branch or does it have to be
provided via JIRA in git patch format?

[1]
https://issues.apache.org/jira/browse/ATLAS-3864

Thanks,
Viktor




Re: Review Request 72644: ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name

2020-07-28 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72644/#review221390
---




intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
Lines 173 (patched)
<https://reviews.apache.org/r/72644/#comment310325>

Consider including parent type name in the message, like:
  Invalid attribute name: {0}.{1}. Attribute already exists in parent type 
{}"



intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java
Lines 509 (patched)
<https://reviews.apache.org/r/72644/#comment310326>

Instead of checkForExistingAttributeInParentTypeDefs() method to verify 
this, I suggest to perform the validation in following methods:
  - AtlasEntityType.resolveReferences()
  - AtlasClassificationType.resolveReferences()


- Madhan Neethiraj


On July 7, 2020, 9:30 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72644/
> ---
> 
> (Updated July 7, 2020, 9:30 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3872
> https://issues.apache.org/jira/browse/ATLAS-3872
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Cause of the problem:
> Type-def Column is created by quick-start script.
> 
> Here in this case, Type-def Column has the attribute "name". And it has 
> property isOptional = true
> 
> Type-def Column is a child of Type-def Asset which also has the attribute 
> "name". And it has property isOptional = false
> 
> Now while creating Entity of Type-def Column from UI, it does not show 
> attribute "name" in the mandatory fields because for Type-def Column its 
> property isOptional = true
> 
> But while validating the attributes from server side we iterate through all 
> the attributes of Entity type (Including Parent's attributes)
> In this case, while validating Type-def Asset's attribute "name" (which has 
> property isOptional = false) we get an error because its value is sent as 
> "null" from UI side.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java a14e43a 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 4a79b6f 
>   repository/src/test/resources/sales-v1-full.zip 
> 07afbf6ecae312a840b54cae5ab01335c513d180 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 82a9f5c 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java d5cf9b7 
>   webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
> 6800288 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
>  7c820e7 
> 
> 
> Diff: https://reviews.apache.org/r/72644/diff/3/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2007/console
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2014/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient

2020-07-26 Thread Madhan Neethiraj
es/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
Lines 159 (patched)
<https://reviews.apache.org/r/72698/#comment310295>

name is an attribute of Asset, which is super-type of DatatSet. Please 
remove this attribute from SampleAppConstants.COLUMN_TYPE.



atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
Lines 177 (patched)
<https://reviews.apache.org/r/72698/#comment310296>

createStructDef() => createSerDeStructDef()



atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
Lines 179 (patched)
<https://reviews.apache.org/r/72698/#comment310299>

Replace hardcoded string, serdeType, with a constant:
  SampleAppConstants.STRUCT_TYPE_SERDE



atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
Lines 184 (patched)
<https://reviews.apache.org/r/72698/#comment310297>

createEnumDef() => createTableTypeEnumDef()



atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
Lines 185 (patched)
<https://reviews.apache.org/r/72698/#comment310300>

Replace hardcoded string, tableType, with a constant:
  SampleAppConstants.ENUM_TABLE_TYPE



atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
Lines 209 (patched)
<https://reviews.apache.org/r/72698/#comment310298>

Replace hardcoded string, bmWithAllTypes, with a constant:
  SampleAppConstants.BUSINESSMETADATA_TYPE

Same for #217 as well.



atlas-examples/sample-app/src/main/resources/atlas-application.properties
Lines 1 (patched)
<https://reviews.apache.org/r/72698/#comment310281>

Lincese text missing



atlas-examples/sample-app/src/main/resources/atlas-application.properties
Lines 10 (patched)
<https://reviews.apache.org/r/72698/#comment310284>

Does the sample use Kafka notifications? If not, I suggest to remove these 
configurations.


- Madhan Neethiraj


On July 23, 2020, 6:10 p.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
> 
> (Updated July 23, 2020, 6:10 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Using this project users can get an idea as how to integrate with Atlas using 
> AtlasCleint. This helps the user to understand the basic rest functionality 
> of Atlas such as
> 
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
> 
> 
> Diffs
> -
> 
>   atlas-examples/pom.xml PRE-CREATION 
>   atlas-examples/sample-app/README.md PRE-CREATION 
>   atlas-examples/sample-app/pom.xml PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/DiscoveryExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/EntityExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/GlossaryExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/LineageExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleApp.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleAppConstants.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
>  PRE-CREATION 
>   atlas-examples/sample-app/src/main/resources/atlas-application.properties 
> PRE-CREATION 
>   pom.xml 5e0442ae5 
> 
> 
> Diff: https://reviews.apache.org/r/72698/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>



Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-24 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221368
---




graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 50 (patched)
<https://reviews.apache.org/r/72695/#comment310257>

lockEnabled => CONSISTENCY_LOCK_ENABLED



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Line 271 (original), 267 (patched)
<https://reviews.apache.org/r/72695/#comment310256>

call to indexBuilder.unique() shouldn't require lockEnabled=true. Please 
retain current if condition:
  "(lockEnabled && isUnique)" => "(isUnique)"



intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 75 (patched)
<https://reviews.apache.org/r/72695/#comment310255>

STORAGE_LOCK_ENABLED("atlas.graph.storage.lock.enabled") => 
STORAGE_CONSISTENCY_LOCK_ENABLED("atlas.graph.storage.consistency-lock.enabled")


- Madhan Neethiraj


On July 24, 2020, 8:23 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 24, 2020, 8:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/2/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-24 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221351
---




intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 67 (patched)
<https://reviews.apache.org/r/72666/#comment310241>

incrementCapacityBy is already an instance member, line #29 above. It is 
unnecessary to pass it as  parameter to methods at #67, #79. Please review and 
update.



intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 73 (patched)
<https://reviews.apache.org/r/72666/#comment310242>

This assumes that access is done in strict ascending order sequence, 
without any gaps. This is not a good assumption for a generic/reusable class 
implementation. This will be broken with the following access pattern:
  FixedBufferList list = new FixedBufferList<>(10, 1);
  
  list.getForUpdate(0);
  list.getForUpdate(15);

Please review my earlier comment and update to remove this assumption. 
Also, I suggest to collapse following methods into one, named ensureCapacity():
 - request()
 - ensureCapacity(): it not obvious what the return value of this method 
is, and it is critical for the caller at #74. All this noise, and confusion can 
be avoided by simply collapsing this method into previous method at #67.
 - instantiateItems(): this method is not callable from any other context, 
as it simply "adds()" to the buffer



webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
Line 168 (original), 175 (patched)
<https://reviews.apache.org/r/72666/#comment310243>

sendNotifications() can be called from the same thread multiple times - for 
example when a transaction involves create/update/delete of entities i.e. a 
graph transaction can call more than one of the following methods:
- onEntitiesAdded()
- onEntitiesUpdated()
- onEntitiesDeleted()
- onEntitiesPurged()
- onClassificationsAdded()
- onClassificationsUpdated()
- onClassificationsDeleted()

Each call would end up calling EntityNotificationSender.send() with the 
notification objects, which in turn stores the objects in a list. The second 
call in the same transaction will end up overwriting EntityNotificationV2 
instances in list buffer - which will cause the notification objects stored in 
EntityNotificationSender.send() be updated as well. This will result in 
incorrect notification to be sent. Please review this carefully and update.


- Madhan Neethiraj


On July 24, 2020, 5:18 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 24, 2020, 5:18 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> reposito

Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-23 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221333
---




intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 51 (patched)
<https://reviews.apache.org/r/72666/#comment310221>

this assumes that the elements are accessed in ascending oeder - since 
getAndIncrementLength() simply increments the length by 1. I suggest to get rid 
of getAndIncrementLength() and update getForUpdate() as below:

  public T getForUpdate(int index) {
ensureCapacity(index + 1);

T ret = buffer.get(index);

// update length only if the accessed index is beyond the current length
if (this.length <= index) {
  this.length = index + 1;
}

return ret;
  }



intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 55 (patched)
<https://reviews.apache.org/r/72666/#comment310223>

To make it easier to read, I suggest to get rid of method 
resetCurrentLength() and update toList() as below:
  public List toList(boolean resetList) {
List ret = this.buffer.subList(0, this.length);

if (resetList) {
  this.length = 0;
}
  }



intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 60 (patched)
<https://reviews.apache.org/r/72666/#comment310226>

- is 'incrementCapacityBy' argument needed, given this is available in 
instance memeber this.incrementCapacityBy?
- consider renaming request() => ensureCapacity()
- consider folding adjustCapacity(), instantiateItems() into this method, 
as below:

  private void ensureCapacity(int capacity) {
if (capacity > this.buffer.size()) {
  int currCapacity = this.buffer.size();
  int newCapacity  = currCapacity + incrementCapacityBy;

  while (newCapacity < capacity) {
newCapacity += incrementCapacityBy;
  }

  this.buffer.ensureCapacity(newCapacity);

  // initialize new entries
  for (int i = currCapacity; i < newCapacity; i++) {
this.buffer.add(itemClass.newInstance());
  }
}
  }



intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 71 (patched)
<https://reviews.apache.org/r/72666/#comment310225>

adjustCapacity() => ensureCapacity()



intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 87 (patched)
<https://reviews.apache.org/r/72666/#comment310227>

Consider moving instantiateItems() into adjustCapacity() implementation; 
this will avoid callers of adjustCapacity() to call instantiateItems() as well .



intg/src/main/java/org/apache/atlas/utils/FixedBufferListAccessor.java
Lines 44 (patched)
<https://reviews.apache.org/r/72666/#comment310228>

Why does this block need to be under synchronized? No state is updated in 
this block.



intg/src/test/java/org/apache/atlas/utils/FixedBufferListAccessorTest.java
Lines 33 (patched)
<https://reviews.apache.org/r/72666/#comment310229>

verifyPeriodicPurge() - the method name doesn't seem to related the 
implementation. Please review and update.



repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
Lines 90 (patched)
<https://reviews.apache.org/r/72666/#comment310230>

FixedBufferListAccessor class doesn't seem to add much value. Consider the 
following simper usage:

  private static final ThreadLocal> 
AUDIT_EVENTS_BUFFER =
  new ThreadLocal.withInitial(() -> new 
FixedBufferList(FIXED_BUFFER_INITIAL_SIZE_DEFAULT, 
FIXED_BUFFER_INCREMENT_DEFAULT));

Note the use of 'static' above; fixedBufferListAccessor can also be 
eliminated. You might consider a simple accesser method in 
EntityAuditListenerV2:

  private FixedBufferList getAuditEventsBuffer() {
return AUDIT_EVENTS_BUFFER.get();
  }
    
Same applies for EntityNotificationListenerV2 as well.


- Madhan Neethiraj


On July 23, 2020, 6:48 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> -------
> 
> (Updated July 23, 2020, 6:48 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects

Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient

2020-07-21 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72698/#review221300
---




atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 18 (patched)
<https://reviews.apache.org/r/72698/#comment310177>

- move all Java files to to org.apache.atlas.examples package
- also, consider moving this to a sub-project named SampleApp, similar to 
Ranger examples here: 
https://github.com/apache/ranger/tree/master/ranger-examples



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 31 (patched)
<https://reviews.apache.org/r/72698/#comment310178>

AtlasClientBaseExample => SampleApp



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 32 (patched)
<https://reviews.apache.org/r/72698/#comment310176>

CLUSTER_SUFFIX => METADATA_NAMESPACE_SUFFIX



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 55 (patched)
<https://reviews.apache.org/r/72698/#comment310184>

empSalaryGlossary is used only in GlossaryExample. Consider moving this 
filed to GlossaryExample. Review other fields for the same.



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 62 (patched)
<https://reviews.apache.org/r/72698/#comment310179>

urls => atlasServerUrls
 - also in #66, #73



atlas-examples/src/main/java/org/apache/atlas/DiscoveryExample.java
Lines 28 (patched)
<https://reviews.apache.org/r/72698/#comment310183>

Classes DiscoveryExample, EntityRestBaseExample, GlossaryExample, 
LineageExample and TypeDefExample derive from only to share few constants. I 
suggest to avoid this hierachy. To share constants, consider having all 
constants in a class like 'SampleAppConstants'.


- Madhan Neethiraj


On July 21, 2020, 4:24 p.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
> 
> (Updated July 21, 2020, 4:24 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Using this project users can get an idea as how to integrate with Atlas using 
> AtlasCleint. This helps the user to understand the basic rest functionality 
> of Atlas such as
> 
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
> 
> 
> Diffs
> -
> 
>   atlas-examples/pom.xml PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/DiscoveryExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/EntityRestBaseExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/GlossaryExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/LineageExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java 
> PRE-CREATION 
>   pom.xml 5e0442ae5 
> 
> 
> Diff: https://reviews.apache.org/r/72698/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>



Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-21 Thread Madhan Neethiraj


> On July 21, 2020, 8:38 p.m., Ashutosh Mestry wrote:
> > Here's what I have tried so far:
> > - Concurrent entity creation using my own test rig. This creates entities 
> > concurrently but it does not allow for entity with same qualifiedName to be 
> > created by different workers.
> > - Ingest via Kafka queue. This is to verify that there is not performance 
> > degradation. 
> > 
> > So far results are positive. I did not obseve any performance or impact on 
> > accuracy.
> > 
> > Yet to try:
> > - Migration from 0.8 to latest.
> > - Migration import using ZipDirect format.

Ashutosh - what is the behavior seen in case of simultaneous attempts to entity 
an entity? Do all but 1 attempt fail?


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221296
---


On July 20, 2020, 9:48 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 20, 2020, 9:48 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AbstractGraphDatabaseTest.java
>  35004157f 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusDatabaseTest.java
>  5cd55093e 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e35f3594f 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
>  5a9ac2abe 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
>  d3111f110 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/1/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-21 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221297
---




graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 260 (patched)
<https://reviews.apache.org/r/72695/#comment310175>

Consistency lock might be relevant/needed only for unique-index i.e. 
isUnique=true. If this true, consider calling setConsistency(LOCK) when 
isUnique is true, without requiring additional argument. Same applies for #278 
as well.

It will be useful support following configuration, to optionally disable 
consistentcy-lock:
  atlas.graph.storage.unique-key.consitency-lock.enabled

This configuration can be sent to AtlasJanusGraphManagement during 
construction - from AtlasJanusGraph.getManagementSystem().

Above will avoid updates to many methods for the addition of lockEnabled 
argument.


- Madhan Neethiraj


On July 20, 2020, 9:48 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 20, 2020, 9:48 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AbstractGraphDatabaseTest.java
>  35004157f 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusDatabaseTest.java
>  5cd55093e 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e35f3594f 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
>  5a9ac2abe 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
>  d3111f110 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/1/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



[jira] [Resolved] (ATLAS-3875) Adding missing APIs in AtlasClient with test cases

2020-07-20 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj resolved ATLAS-3875.
-
Fix Version/s: 2.2.0
   3.0.0
   Resolution: Fixed

> Adding missing APIs in AtlasClient with test cases
> --
>
> Key: ATLAS-3875
> URL: https://issues.apache.org/jira/browse/ATLAS-3875
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jyoti Singh
>Assignee: Jyoti Singh
>Priority: Major
>  Labels: api, client
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3875-1.patch
>
>
> There are many new APIs added to Atlas Project but the corresponding  APIs 
> are missing from AtlasClientv2. The aim of this task is to complete the gap 
> amongst existing APIs and their endpoints in Atls client. This will also 
> include adding test cases via integration testing.
> There are functions from AtlasClient for the following REST endpoints
>  * TypeRest
>  * EntityRest
>  * LineageRest
>  * DiscoveryRest
>  * GlossaryRest
>  * RelationshipRest



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints

2020-07-19 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72638/#review221263
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 19, 2020, 9:12 p.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72638/
> ---
> 
> (Updated July 19, 2020, 9:12 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> There are many missing API endpoints in AtlasClientV2. This solution is 
> adding functions corresponding to those missing APIs for the following Rest 
> endpoints.
> 
> 1. TypeRest
> 2. EntityRest
> 3. LineageRest
> 4. DiscoveryRest
> 5. GlossaryRest
> 6. RelationshipRest
> 
> This will enable users to test and integrate with APIs more effectively via 
> AtlasCient.
> 
> 
> Diffs
> -
> 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 7b6d1d0f3 
>   client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java e3d2ebc34 
>   webapp/pom.xml 3c55b4dd4 
>   webapp/src/test/java/org/apache/atlas/web/TestUtils.java e22a1c10d 
>   webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
> 680028892 
>   webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
> 808f62354 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
>  cc883d615 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java 
> PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java 
> PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java
>  331ea2c47 
>   webapp/src/test/resources/json/search-parameters/attribute-filters.json 
> PRE-CREATION 
>   webapp/src/test/resources/template.csv PRE-CREATION 
>   webapp/src/test/resources/template_metadata.csv PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72638/diff/8/
> 
> 
> Testing
> ---
> 
> IT has been added for newly created functions in AtlasClientV2.
> Validated by running IT on local system.
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>



Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints

2020-07-17 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72638/#review221258
---




client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 472 (patched)
<https://reviews.apache.org/r/72638/#comment310114>

- please remove 'final' for method parameters
- partialUpdateEntityAttrByGuid() => partialUpdateEntityByGuid()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 490 (patched)
<https://reviews.apache.org/r/72638/#comment310115>

removeEntityClassifications() => removeEntityClassification()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 495 (patched)
<https://reviews.apache.org/r/72638/#comment310116>

deleteClassificationByGuid() => removeEntityClassification()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 539 (patched)
<https://reviews.apache.org/r/72638/#comment310117>

importBMAttributes() => bulkUpdateBusinessAttributes()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 547 (patched)
<https://reviews.apache.org/r/72638/#comment310118>

entityGuid => typeName? Please review and update.



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 551 (patched)
<https://reviews.apache.org/r/72638/#comment310119>

entityGuid => typeName? Please review and update.



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 556 (patched)
<https://reviews.apache.org/r/72638/#comment310120>

exportDataForEntity() => getTemplateForBulkUpdateBusinessAttributes()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 562 (patched)
<https://reviews.apache.org/r/72638/#comment310121>

Please remove 'final' from method parameters - line #562, #570.



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 629 (patched)
<https://reviews.apache.org/r/72638/#comment310122>

For consistency with other methods, I suggest to reorder the parameters:
 - typeName, attrName, attrValuePrefix, limit, offset



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 775 (patched)
<https://reviews.apache.org/r/72638/#comment310123>

- please remove 'atlas' prefix from variable names
   atlasGlossary => glossary
- please make sure to leave a space before '{'; here and rest of the patch



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 831 (patched)
<https://reviews.apache.org/r/72638/#comment310124>

sort => sortByAttribute
 - lines #831, #842, #853, #863, #873, #884, #894



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 936 (patched)
<https://reviews.apache.org/r/72638/#comment310125>

importFileData() => importGlossary()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 953 (patched)
<https://reviews.apache.org/r/72638/#comment310126>

importData() => importGlossary()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 958 (patched)
<https://reviews.apache.org/r/72638/#comment310127>

performImportData() => importGlossary()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 972 (patched)
<https://reviews.apache.org/r/72638/#comment310128>

getTemplateByStream() => readStreamContents()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 1175 (patched)
<https://reviews.apache.org/r/72638/#comment310130>

GET_TEMPLATE => GET_IMPORT_GLOSSARY_TEMPLATE



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 1176 (patched)
<https://reviews.apache.org/r/72638/#comment310129>

IMPORT_DATA => IMPORT_GLOSSARY



client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
Lines 685 (patched)
<https://reviews.apache.org/r/72638/#comment310131>

- hardcoded path "/src/test/resources/" doesn't look right. is this a temp 
location? or is the user expected have contents under this directory?
- if this is needed only for testing, consider moving this to test code.


- Madhan Neethiraj


On July 16, 2020, 3:17 a.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72638/
> ---
> 
> (Updated July 16, 2020, 3:17 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Reposit

[ANNOUNCE] Apache Atlas 2.1.0 released

2020-07-16 Thread Madhan Neethiraj
All,

Apache Atlas team is happy to announce the release of Apache Atlas 2.1.0.

Atlas is a scalable and extensible set of core foundational governance services 
– enabling enterprises to effectively and efficiently meet their compliance 
requirements within Hadoop and allows integration with the whole enterprise 
data ecosystem.

The release artifacts are available at: https://atlas.apache.org/#/Downloads

The binary artifacts are available from Maven central and its mirrors.

To use these artifacts, please use the following documentation: 
https://atlas.apache.org/2.1.0/index.html

More details on Apache Atlas can be found at: https://atlas.apache.org

We thank everyone who made this release possible.

Thanks,
Apache Atlas team




[jira] [Updated] (ATLAS-3359) Relationships : Unique attribute in a relationship is not honored

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3359:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Relationships : Unique attribute in a relationship is not honored
> -
>
> Key: ATLAS-3359
> URL: https://issues.apache.org/jira/browse/ATLAS-3359
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nixon Rodrigues
>Assignee: Mayank Jain
>Priority: Major
> Fix For: 2.2.0
>
>
> 1.Created a relationship typedef between type1 & type2 with attribute "attr1" 
> which is unique (set "IsUnique" to true).
> 2. Created entity1 (type1) and entity2(type2) and created a relationship 
> instance between them with "attr1" value , say "value1".
> 3. Created entity3 (type1) and entity4(type2) and created a relationship 
> instance between them with "attr1" value with the same value "value1".
> Expected the relationship creation between entity3 and entity4 would fail 
> because a relationship with "attr1 " with "value1" is already created between 
> entity1 and entity2. But request passed and the relationship is created 
> successfully.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3839) import-hive.sh build target directory error

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3839:

Fix Version/s: (was: 2.1.0)
   2.2.0

> import-hive.sh build target directory error
> ---
>
> Key: ATLAS-3839
> URL: https://issues.apache.org/jira/browse/ATLAS-3839
> Project: Atlas
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.1.0, 2.0.0
>Reporter: limbo
>Priority: Minor
>  Labels: patch-available
> Fix For: 2.2.0
>
> Attachments: ATLAS-3839.patch
>
>
> import-hive.sh build to bin directory rather than in hook-bin



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3468) Exclude JTS topology suite library from being packaged

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3468:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Exclude JTS topology suite library from being packaged
> --
>
> Key: ATLAS-3468
> URL: https://issues.apache.org/jira/browse/ATLAS-3468
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: ATLAS-3468.002.patch
>
>
> Exclude JTS topology suite library from being packaged



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3646) Create new 'spark_ml_model_dataset' and 'spark_ml_pipeline_dataset' relationship definitions

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3646:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Create new 'spark_ml_model_dataset' and 'spark_ml_pipeline_dataset' 
> relationship definitions
> 
>
> Key: ATLAS-3646
> URL: https://issues.apache.org/jira/browse/ATLAS-3646
> Project: Atlas
>  Issue Type: Task
>Reporter: Vladislav Glinskiy
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Create new 'spark_ml_model_dataset' and 'spark_ml_pipeline_dataset' 
> relationship definitions. This is required in order to integrate Spark Atlas 
> Connector's ML event processor.
> Previously, Spark Atlas Connector used the 'spark_ml_directory' model for the 
> ML model directory and 'spark_ml_model_ml_directory', 
> 'spark_ml_pipeline_ml_directory' relationship definitions. Usage of the 
> 'spark_ml_directory'  was reverted in the scope of 
> [https://github.com/hortonworks-spark/spark-atlas-connector/issues/61], 
> [https://github.com/hortonworks-spark/spark-atlas-connector/pull/62] so ML 
> model directory is 'DataSet' entity(i.e. 'hdfs_path', 'fs_path').
> Thus, new relationship definitions must be created, since there is no 
> straightforward way to update existing ones to use 'DataSet' type instead of 
> it's child type 'spark_ml_directory'.
> See:
>  * ATLAS-3640
>  * [https://github.com/apache/atlas/pull/88#issuecomment-592699723]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3665) Add 'queryText' attribute to the 'spark_process' type

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3665:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Add 'queryText' attribute to the 'spark_process' type
> -
>
> Key: ATLAS-3665
> URL: https://issues.apache.org/jira/browse/ATLAS-3665
> Project: Atlas
>  Issue Type: Task
>Reporter: Vladislav Glinskiy
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Add 'queryText' attribute to the 'spark_process' type in order to make 
> `spark_process` more readable by the user. The `queryText` attribute stores 
> exact SQL query that are executed within the Spark session as a Spark process.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3353) HMS Hook does not capture the deleted column with alter table replace columns command.

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3353:

Fix Version/s: (was: 2.1.0)
   2.2.0

> HMS Hook does not capture the deleted column with alter table replace columns 
> command.
> --
>
> Key: ATLAS-3353
> URL: https://issues.apache.org/jira/browse/ATLAS-3353
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.2.0
>
>
> For the below commands, HMS Hook does not capture the replaced column as 
> deleted entity.
> {noformat}
> create table table_with_dropped_cols_1 (id int,name string,dob date)
> alter table table_with_dropped_cols_1 replace columns (id int,name 
> string,address string)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3501) Update guava version to 28.0-jre

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3501:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Update guava version to 28.0-jre
> 
>
> Key: ATLAS-3501
> URL: https://issues.apache.org/jira/browse/ATLAS-3501
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Aadarsh Jajodia
>Assignee: Aadarsh Jajodia
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: 0001-ATLAS-3501-Update-guava-version-to-28.0-jre.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3724) Qualified name pattern for Column cause ambiguity in Quickstart data.

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3724:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Qualified name pattern for Column cause ambiguity in Quickstart data.
> -
>
> Key: ATLAS-3724
> URL: https://issues.apache.org/jira/browse/ATLAS-3724
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0, 3.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
> Fix For: 2.0.0, 2.2.0
>
> Attachments: ATLAS-3724.patch
>
>
> 1) Create Entity of type Column name = app_id, qualified name = app_id@cl1
> assign Table - logging_fact_monthly_mv
> 2) Again Create same entity with same qualified name
> assign different Table - log_fact_daily_mv
> 3) Check Column Relationship, Only one Table relationship is seen
> 4) Check Table (logging_fact_monthly_mv,log_fact_daily_mv) In Both 
> Relationship, Column app_id is seen



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3341) Atlas knox SSO doesnt work when using external load balancer for Atlas UI.

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3341:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Atlas knox SSO doesnt work when using external load balancer for Atlas UI.
> --
>
> Key: ATLAS-3341
> URL: https://issues.apache.org/jira/browse/ATLAS-3341
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> If Atlas ports are not open to access, so leveraging SSL offloading and using 
> External loabalancer enabled with SSL to access Atlas UI.
> However with the above setup Knox SSO redirection is not working as expected. 
> As the redirectUrl/Forwarded url created is not set for proper redirection.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3410) Term assigned to historical entity is not visible in Details page.

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3410:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Term assigned to historical entity is not visible in Details page.
> --
>
> Key: ATLAS-3410
> URL: https://issues.apache.org/jira/browse/ATLAS-3410
> Project: Atlas
>  Issue Type: Bug
>Reporter: Rahul Kurup
>Assignee: chaitali borole
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3410-Server.patch, ATLAS-3410-dashboardv3.patch, 
> ATLAS-3410.patch
>
>
> This issue is related to an earlier one (ATLAS-3403). When a entity with a 
> term associated to it is deleted, and if you navigate to the Details page for 
> that entity, the classification is displayed but not the term.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3837) execute import-hive.sh raise exception: Failed to load application properties

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3837:

Fix Version/s: (was: 2.1.0)
   2.2.0

> execute import-hive.sh raise exception: Failed to load application properties
> -
>
> Key: ATLAS-3837
> URL: https://issues.apache.org/jira/browse/ATLAS-3837
> Project: Atlas
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.2.0, 2.0.0
>Reporter: limbo
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: ATLAS-3837.patch
>
>
> when run import-hive.sh, the script raise as follow:
> {code:java}
> 2020-06-11 10:49:01,177 INFO - [main:] ~ Looking for 
> atlas-application.properties in classpath (ApplicationProperties:97)
> 2020-06-11 10:49:01,181 INFO - [main:] ~ Looking for 
> /atlas-application.properties in classpath (ApplicationProperties:102)
> 2020-06-11 10:49:01,182 INFO - [main:] ~ Loading atlas-application.properties 
> from null (ApplicationProperties:110)
> 2020-06-11 10:49:01,199 ERROR - [main:] ~ Import failed 
> (HiveMetaStoreBridge:176)
> org.apache.atlas.AtlasException: Failed to load application properties
>  at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:121)
>  at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:73)
>  at 
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:120)
> Caused by: org.apache.commons.configuration.ConfigurationException: Cannot 
> locate configuration source null
>  at 
> org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:259)
>  at 
> org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238)
>  at 
> org.apache.commons.configuration.AbstractFileConfiguration.(AbstractFileConfiguration.java:197)
>  at 
> org.apache.commons.configuration.PropertiesConfiguration.(PropertiesConfiguration.java:284)
>  at 
> org.apache.atlas.ApplicationProperties.(ApplicationProperties.java:56)
>  at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:112)
>  ... 2 more
> {code}
> the script can not find the config file of atlas-application.properties 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3498) Update basic-search to enable search using labels

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3498:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Update basic-search to enable search using labels
> -
>
> Key: ATLAS-3498
> URL: https://issues.apache.org/jira/browse/ATLAS-3498
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.2.0
>
>
> Update basic-search to enable search using labels



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3255) UI: IE 10 - Page Limit button gets displaced after clicking it.

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3255:

Fix Version/s: (was: 2.1.0)
   2.2.0

> UI: IE 10 - Page Limit button gets displaced after clicking it.
> ---
>
> Key: ATLAS-3255
> URL: https://issues.apache.org/jira/browse/ATLAS-3255
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Minor
> Fix For: 3.0.0, 1.3.0, 2.2.0
>
>
> In IE 10, the Page Limit button position gets displaced and moves to the 
> middle of the search results page after clicking it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3852) Entity Bulk Create with unique reference

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3852:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Entity Bulk Create with unique reference
> 
>
> Key: ATLAS-3852
> URL: https://issues.apache.org/jira/browse/ATLAS-3852
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Vasanth kumar RJ
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: ATLAS-3852.patch, BulkPostCreate.json
>
>
> Entities created in bulk and unique referenced in same request. Bulk create 
> DB, Table and Column referenced in a request itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3683) Entity details > Business Metadata, multi-valued attribute spacing issue

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3683:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Entity details > Business Metadata, multi-valued attribute spacing issue
> 
>
> Key: ATLAS-3683
> URL: https://issues.apache.org/jira/browse/ATLAS-3683
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Reporter: Umesh Padashetty
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: Screenshot 2020-03-25 at 1.14.15 AM.png
>
>
> While applying Business metadata attribute to an entity, once the 
> multi-valued attribute is applied, the multi values are displayed without any 
> spacing. This is a bit unreadable. It would be good if we can add a space 
> between comma-separated values.
> CC: [~kevalbhatt]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3824) update README.txt to replace earlier version references to 2.1.0

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3824:

Fix Version/s: (was: 2.1.0)
   2.2.0

> update README.txt to replace earlier version references to 2.1.0
> 
>
> Key: ATLAS-3824
> URL: https://issues.apache.org/jira/browse/ATLAS-3824
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>    Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: ATLAS-3824.patch
>
>
> Update references to earlier version in README.txt with 2.1.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3407) Regression, Impala and Spark Hook : Rename table doesn't rename existing table , but creates new table with updated name.

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3407:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Regression, Impala and Spark Hook : Rename table doesn't rename existing 
> table , but creates new table with updated name.
> -
>
> Key: ATLAS-3407
> URL: https://issues.apache.org/jira/browse/ATLAS-3407
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.2.0
>
>
> with hive hook , rename table works as expected.
> With spark and impala hook (hms hook), rename table creates new table with 
> updated name and doesn't update the existing table entity.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3610:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Enable logging to multiple targets (logdir, stdout)
> ---
>
> Key: ATLAS-3610
> URL: https://issues.apache.org/jira/browse/ATLAS-3610
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 3.0.0
>Reporter: Mariusz Górski
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, 
> ATLAS-3610-3.patch, ATLAS-3610.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> *Problem description*
> When starting Atlas, Python scripts are invoked to spawn all necessary 
> processes. This is achieved with the use of _subprocess.Popen()_ in 
> _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout 
> is being redirected to log files, which (for java processes) can override 
> log4j configuration (if configured with stdout handler). This creates 
> situation when log4j configuration cannot be relied upon and prevents logging 
> to stdout.
> *Proposed solution*
>  * Add possibility to log to stdout by using multiple stdout/stderr handlers
>  * Adjust _runProcess()_ function from _atlas_config.py_ script (and all 
> functions relying on it) to provide _logconsole_ kwarg which will add 
> additional handler to Popen stdout/stderr redirection
>  * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for 
> backwards compatibility.
>  * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_  
> and provide it in functions invoking atlas/solr/elasticsearch/zookeeper
> *Predicted Benefits*
> This improvement would bring the benefit of enabling logging to stdout (which 
> is supressed for now even if configured in spawned components), which could 
> be very useful when running Atlas in docker container or in Kubernetes/OC, 
> where some logging scraping is present.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3890) Enhancement for all BulkAPIs.

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3890:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Enhancement for all BulkAPIs.
> -
>
> Key: ATLAS-3890
> URL: https://issues.apache.org/jira/browse/ATLAS-3890
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
>  Labels: api-change
> Fix For: 2.2.0
>
>
> According to below linked Atlas Jira two Bulk Apis are enhanced to return 
> authorised entities or modify them when some of then entity types are blocked 
> by policies in Ranger for access to user.Hence we are improving the other 
> bulk Apis too.
> Previously user only used to get 403 error if he tries to access or modify 
> entities which comprised of both authorised as well as unauthorised typenames.
> DELETE /bulk - EntityREST.deleteByGuids()
> GET /bulk/uniqueAttribute/type/\{typeName} - 
> EntityREST.getEntitiesByUniqueAttributes()



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3855) Bulk entity tag association and bulk api enhancement

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3855:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Bulk entity tag association and bulk api enhancement
> 
>
> Key: ATLAS-3855
> URL: https://issues.apache.org/jira/browse/ATLAS-3855
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
>  Labels: api-change
> Fix For: 2.2.0
>
>
> API : /api/atlas/v2/entity/bulk/classification
> Body :
> {code:java}
> {"classification":{"typeName":"PII","attributes":{},"propagate":true,"removePropagationsOnEntityDelete":false,"validityPeriods":[]},"entityGuids":["guid1","guid2"]}
>  {code}
> when user doesn't have authorisation on either of guid1 or guid2 , the bulk 
> entity call fails with 403. 
> bulk api _v2/entity/bulk_ to retrieve a list of entities identified by its 
> guids.
> This bulk api fails with 403 if some guids belong to entities on which user 
> is *unauthorized* and other guids belong to entities on which user is 
> *authorized*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (ATLAS-3809) Publish release artifacts

2020-07-15 Thread Madhan Neethiraj (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj resolved ATLAS-3809.
-
Fix Version/s: 2.1.0
 Assignee: Madhan Neethiraj
   Resolution: Done

> Publish release artifacts
> -
>
> Key: ATLAS-3809
> URL: https://issues.apache.org/jira/browse/ATLAS-3809
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 2.1.0
>    Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[RESULT] [VOTE] Release Apache Atlas version 2.1.0

2020-07-14 Thread Madhan Neethiraj
Atlas team,

Thanks for taking time to verify, give feedback and vote on Apache Atlas 2.1.0 
release candidate #3.

More than 72 hours have passed since the email to vote for the release; the 
vote is now closed. The vote passes with 16 +1s and no 0s and -1s.

Here are the vote details:

Four +1 votes from following PMC members:
  - Keval Bhatt
  - Madhan Neethiraj
  - Nixon Rodrigues
  - Sarath Subramanian

Twelve +1 votes from following committers/contributors:
  - Chaitali Borole
  - Jayendra Parab
  - Mandar Ambawane
  - Mayank Jain
  - Nikhil Bonte
  - Pinal Shah
  - Ramesh Mani (Committer)
  - Sameer Shaikh
  - Sharmadha Sainath (Committer)
  - Sidharth Mishra
  - Umesh Padashetty
  - Vishal Suvagia

Voting thread: 
https://lists.apache.org/thread.html/rd718a560b01c49017ea521e12769754d9382b35707f53b918353de5e%40%3Cdev.atlas.apache.org%3E

I will begin the process of creating the release artifacts for Apache Atlas 
2.1.0.

Thanks,
Madhan




Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints

2020-07-14 Thread Madhan Neethiraj
I suggest to reorder the arguments:
typeName, uniqAttributes, labels



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 723 (patched)
<https://reviews.apache.org/r/72638/#comment310009>

- addLabelsByTypeName() => addLabels()
- I suggest to reorder the arguments:
typeName, uniqAttributes, labels



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 727 (patched)
<https://reviews.apache.org/r/72638/#comment310011>

- deleteLabelsByTypeName() => removeLabels()
- I suggest to reorder the arguments:
typeName, uniqAttributes, labels



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 751 (patched)
<https://reviews.apache.org/r/72638/#comment310012>

if any of these parameters are optional, consider moving them towards end 
of the method; and add them to 'queryParams' only if they are not null.



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 780 (patched)
<https://reviews.apache.org/r/72638/#comment310013>

- What value should be provided to fieldName? Consider adding a comment.
- if this value can be null, update #783 only if the value is not null



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 792 (patched)
<https://reviews.apache.org/r/72638/#comment310014>

- executeSavedSearchByName() => executeSavedSearch()
- To be consistent with #774, please reorder the parameters:
  - userName, searchName



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 797 (patched)
<https://reviews.apache.org/r/72638/#comment310015>

Make sure to leave a blank like before start of a method. Please review and 
update other occurances as well.



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 798 (patched)
<https://reviews.apache.org/r/72638/#comment310016>

executeSavedSearchByGuid() => executeSavedSearch()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 801 (patched)
<https://reviews.apache.org/r/72638/#comment310017>

getQuickSearch() => quickSearch()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 811 (patched)
<https://reviews.apache.org/r/72638/#comment310018>

postQuickSearch() => quickSearch()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 828 (patched)
<https://reviews.apache.org/r/72638/#comment310019>

getDetailGlossary() => getGlossaryExtInfo()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 864 (patched)
<https://reviews.apache.org/r/72638/#comment310020>

partialUpdates => attributes



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 872 (patched)
<https://reviews.apache.org/r/72638/#comment310021>

partialUpdates => attributes



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 880 (patched)
<https://reviews.apache.org/r/72638/#comment310022>

partialUpdates => attributes



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 907 (patched)
<https://reviews.apache.org/r/72638/#comment310023>

getGlossaryTermHeadersByGuid() => getGlossaryTermHeaders()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 918 (patched)
<https://reviews.apache.org/r/72638/#comment310024>

getGlossaryCategoriesByGuid() => getGlossaryCategories()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 928 (patched)
<https://reviews.apache.org/r/72638/#comment310025>

getGlossaryCategoriesHeaders() => getGlossaryCategoryHeaders()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 979 (patched)
<https://reviews.apache.org/r/72638/#comment310026>

disassociateTermAssignmentFromEntities() is a duplicate of 
removeTermAssignmentFromEntities() - looking at GlossaryREST.java. I suggest to 
remove method removeTermAssignmentFromEntities()



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 992 (patched)
<https://reviews.apache.org/r/72638/#comment310027>

- produceTemplate() => getGlossaryTemplate()
- consider returning a string



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 996 (patched)
<https://reviews.apache.org/r/72638/#comment310028>

- inputStream is not used?
- what would callers send for fileDetail?
- consider returning List - similar to 
GlossaryREST.importGlossaryData()


- Madhan Neethiraj


On July 10, 2020, 12:05 a.m., Jyoti Singh wrote:
> 
> -------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72638/
> ---
>

  1   2   3   4   5   6   7   8   9   10   >