Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71737/#review218585
---




intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
Line 49 (original), 49 (patched)


Move the new enum element at the end, to retain the ordinal value of 
existing elements.



repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
Line 315 (original), 315 (patched)


This file has only whitespace changes. Please review and revert.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
Lines 492 (patched)


This should be logged at warn level.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
Lines 500 (patched)


Ideally, we should add a new privilege - like admin-purge. That  will 
require updates to Ranger model as well. For now, its okay to require 
admin-import privilege. However, move #500 outside this 'for' loop - to #484.



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 442 (patched)


Consider adding another method in a subsequent patch, that take 
List as parameter. Such method can be useful in purging entities 
using typeName/uniqueAttributes.


- Madhan Neethiraj


On Nov. 7, 2019, 11:54 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71737/
> ---
> 
> (Updated Nov. 7, 2019, 11:54 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3477
> https://issues.apache.org/jira/browse/ATLAS-3477
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> As part of this solution users can purge the entities which are already soft 
> deleted. This will be useful as someone may be interested to purge some list 
> of soft deleted entities from Atlas. Due to governance compliances, only 
> admin can perform purge of entities.
> 
> Purge will also create separate Audit entry at HBase similar to delete entity 
> operation.
> 
> 
> Diffs
> -
> 
>   authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java 
> 59c596d66 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 199d6bf58 
>   intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 
> 444167eae 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> e9cc7cd3d 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  7ace00dff 
>   intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java 
> daf4ca88c 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  43a9b842c 
>   
> repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
>  01e339d23 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  b94590bf1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  d2544df4f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  3389d2465 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  17a96493f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
>  4d57d8bc8 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
>  48f0cd3c2 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 464d46f1e 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
>  5d506bbc2 
> 
> 
> Diff: https://reviews.apache.org/r/71737/diff/1/
> 
> 
> Testing
> ---
> 
> IT has been added. 
> Validated manually by running below mentioned queries and REST:
> 
> 
> a) create table thursday_07_11_1 (id int);
>create table thursday_07_11_1_ctas as select * from thursday_07_11_1;
> 
>create table thursday_07_11_2 (id int);
>create table thursday_07_11_2_ctas as select * from thursday_07_11_2;
> b) The above will create two lineages. Go to Atlas UI and get the vertices 
> for atleast any two.
> 
> 
> c) Using rest delete those two entities or use below query:
> drop table thursday_07_11_1;
> drop table thursday_07_11_1_ctas;
> 

Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71736/#review218584
---




intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
Lines 65 (patched)


ALL_ENTITY_TYPE_QUERY - this doesn't have to be in 'models' package; 
consider moving this out of 'intg' library, to 'repository' module.



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 82 (patched)


- for easier read, please have each attribute in a separate line
- only indexed attributes are relevant for search. Consider excluding 
non-indexed attributes - like IS_PROXY_KEY, HOME_ID_KEY



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 129 (patched)


Consider following alternate approach, to avoid instantiating 
system-attributes in each AtlasEntityType/AtlasClassificationType instance:
  class AtlasTypeRegistry {
private final AtlasEntityType baseEntityType = 
initBaseEntityType();
private final AtlasClassificationType baseClassificationType = 
initBaseClassificationType();

AtlasEntityType getBaseEntityType() { return baseEntityType; }

AtlasClassificationType getBaseClassificationType() { return 
baseClassificationType; }

private static AtlasEntityType initBaseEntityType() {
  List attributeDefs = new ArrayList<>() {{
add(new AtlasAttributeDef(ENTITY_TYPE_PROPERTY_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, false, true)));
add(new AtlasAttributeDef(GUID_PROPERTY_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, true, true)));
add(new AtlasAttributeDef(TRAIT_NAMES_PROPERTY_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, AtlasAttributeDef.Cardinality.SET, false, 
true)));
add(new AtlasAttributeDef(PROPAGATED_TRAIT_NAMES_PROPERTY_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, AtlasAttributeDef.Cardinality.LIST, false, 
true)));
add(new AtlasAttributeDef(HISTORICAL_GUID_PROPERTY_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, false, true)));
add(new AtlasAttributeDef(CLASSIFICATION_TEXT_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, false, true)));
add(new AtlasAttributeDef(CLASSIFICATION_NAMES_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, false, true)));
add(new AtlasAttributeDef(PROPAGATED_CLASSIFICATION_NAMES_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, false, true)));
add(new AtlasAttributeDef(IS_INCOMPLETE_PROPERTY_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, false, true)));
add(new AtlasAttributeDef(LABELS_PROPERTY_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, false, true)));
add(new AtlasAttributeDef(CUSTOM_ATTRIBUTES_PROPERTY_KEY, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING, false, true)));
  }}
  
  AtlasEntityDef entityDef = new AtlasEntityDef("", "", "", 
attributeDefs);

  return new AtlasEntityType(entityDef);
}
  }

  class AtlasEntityType {
private AtlasEntityType baseType = null;

void resolveReferences(AtlasTypeRegistry typeRegistry) {
  baseType = typeRegistry.getBaseEntityType();

  ...
}

public boolean isSystemAttribute(String attributeName) {
  return baseType != null && baseType.hasAttribute(attributeName);
}

public AtlasAttribute getSystemAttribute(String attributeName) {
  return baseType != null ? baseType.getAttribute(attributeName) : null;
}
  }

  class AtlasClassificationType {
private AtlasClassificationType baseType = null;

void resolveReferences(AtlasTypeRegistry typeRegistry) {
  baseType = typeRegistry.getBaseClassificationType();

  ...
}

public boolean isSystemAttribute(String attributeName) {
  return baseType != null && baseType.hasAttribute(attributeName);
}

public AtlasAttribute getSystemAttribute(String attributeName) {
  return baseType != null ? baseType.getAttribute(attributeName) : null;
}
  }



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 131 (patched)


AtlasAttribute instantiation here will set qualifiedName and 
vertextPropertyName to:
  this.typeName + "." + sysAttributeName" - like hive_table.__guid
  
For system attributes, this isn't correct. The vertex property name is 
simply the value of sysAttributeName. Please review AtlasAttribute constructor 
and update it to handle system attributes for base-types (i.e. empty typename)


- Madhan Neethiraj


On Nov. 9, 2019, 1:12 a.m., Le Ma wrote:
> 
> 

Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes

[Le Ma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71736/
---

(Updated Nov. 9, 2019, 1:12 a.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan 
Neethiraj, and Sarath Subramanian.


Changes
---

Update Patch


Bugs: ATLAS-3482
https://issues.apache.org/jira/browse/ATLAS-3482


Repository: atlas


Description
---

Introduce ALL_ENTITY_TYPE to support search on system attributes across all 
entity types. System attributes will be passed in as normal entity attributes.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
8f0e5912d 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
e10965b87 
  intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java ed1e5ded2 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 
  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb 
  intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 672f38132 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
03eb92bcc 
  
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
 152ade8d4 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
7ad32bdb9 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 7c551304b 
  webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 
cd8f8981c 


Diff: https://reviews.apache.org/r/71736/diff/2/

Changes: https://reviews.apache.org/r/71736/diff/1-2/


Testing
---

- Added new unit tests, passed.
- will run pre-commit job.


Thanks,

Le Ma

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

[Aadarsh Jajodia]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/
---

(Updated Nov. 8, 2019, 9:42 p.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, 
and Sarath Subramanian.


Summary (updated)
-

ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas 
and also the corresponding type registry changes for the same


Bugs: ATLAS-3486
https://issues.apache.org/jira/browse/ATLAS-3486


Repository: atlas


Description (updated)
---

This change is the first part of the bigger task defined as part of ATLAS-3485.
This adds the data model needed for supporting namespaces and namespace 
attributes and also updates the type registry to include the applicable 
namespace attributes for every entity type


Diffs
-

  intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
f06f64f450f407e3f9a0e742726ff4dd12ccc695 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
e10965b870c4a300b41e93ee046b5f6d6b722728 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
3634fdfd313639eb97b3c4698e091487b0e44a80 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
4ee68a936f99bb4c819b5335da2cc8bf7d539397 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
884447f81c57faf917a8d0565fc0a0c7ebbd99f0 
  intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
8b4fd1c3b9005d0a8852f2828475b4ad6a806822 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
0883d54f490e22c6510e6fc0cb804b87713a7ecb 
  intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
dba2d88146eff314191ae6bb24ad7337b0ea10ae 
  intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
02613b5f7250b14324ed294c22de079b74d55b08 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
530d5cda44b7d3746f5cb8dd5a23b3c68b254cac 
  intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
 a5ccfb5b2055c88f596312f4033bc0034d3d165c 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
 51dd16b8518c9a16088547f3e95c0ef401695895 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
 6f9c05e7a7a30c678a0a376d25402c6026d0b391 
  
tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
 91ff89a00d779af9437ba5f6d129bd4595e5f036 
  webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
6cd0ee331b7ae24757b58e76ec47bf556106846a 


Diff: https://reviews.apache.org/r/71649/diff/3/


Testing
---

Added unit tests


Thanks,

Aadarsh Jajodia

[jira] [Commented] (ATLAS-3517) Enhance suggestions REST API to have optional 'fieldName' parameter to get suggestions from

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970504#comment-16970504
 ] 

ASF subversion and git services commented on ATLAS-3517:


Commit 7e3990bf3816ab7f0b79b33b2a72186ae5b63d63 in atlas's branch 
refs/heads/branch-2.0 from Sarath Subramanian
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=7e3990b ]

ATLAS-3517: Enhance suggestions REST API to have optional 'fieldName' parameter 
to get suggestions from

(cherry picked from commit 1b5ca4c5346c1b2fad0d276b509a56f78489755d)


> Enhance suggestions REST API to have optional 'fieldName' parameter to get 
> suggestions from
> ---
>
> Key: ATLAS-3517
> URL: https://issues.apache.org/jira/browse/ATLAS-3517
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
>
> Suggestions REST API searches against all indexed fields (terms) to retrieve 
> the top frequently used terms.
> we don't have a way to retrieve suggestions only from a specific indexed 
> field.
> This enhancement introduces a new optional parameter in suggestions API - 
> 'fieldName', which when specified will provide suggestions against that 
> indexed field name.
>  
> This will be useful to get suggestions on only a specific field - say labels 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3517) Enhance suggestions REST API to have optional 'fieldName' parameter to get suggestions from

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970503#comment-16970503
 ] 

ASF subversion and git services commented on ATLAS-3517:


Commit 1b5ca4c5346c1b2fad0d276b509a56f78489755d in atlas's branch 
refs/heads/master from Sarath Subramanian
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=1b5ca4c ]

ATLAS-3517: Enhance suggestions REST API to have optional 'fieldName' parameter 
to get suggestions from


> Enhance suggestions REST API to have optional 'fieldName' parameter to get 
> suggestions from
> ---
>
> Key: ATLAS-3517
> URL: https://issues.apache.org/jira/browse/ATLAS-3517
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
>
> Suggestions REST API searches against all indexed fields (terms) to retrieve 
> the top frequently used terms.
> we don't have a way to retrieve suggestions only from a specific indexed 
> field.
> This enhancement introduces a new optional parameter in suggestions API - 
> 'fieldName', which when specified will provide suggestions against that 
> indexed field name.
>  
> This will be useful to get suggestions on only a specific field - say labels 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71738: ATLAS-3517: Enhance suggestions REST API to have optional 'fieldName' parameter to get suggestions from

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71738/#review218581
---


Ship it!




Ship It!

- Madhan Neethiraj


On Nov. 8, 2019, 6:01 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71738/
> ---
> 
> (Updated Nov. 8, 2019, 6:01 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer 
> Shaikh, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3517
> https://issues.apache.org/jira/browse/ATLAS-3517
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Suggestions REST API searches against all indexed fields (terms) to retrieve 
> the top frequently used terms.
> 
> we don't have a way to retrieve suggestions only from a specific indexed 
> field.
> 
> This enhancement introduces a new optional parameter in suggestions API - 
> 'fieldName', which when specified will provide suggestions against that 
> indexed field name.
> 
>  
> 
> This will be useful to get suggestions on only a specific field - say labels 
> 
> 
> *Solution:*
> 
> Solr already provides a way to search on etrms against a specific field.
> 
> https://lucene.apache.org/solr/guide/6_6/the-terms-component.html
> 
> "terms.fl=" need to added as part of the request to solr 
> client on terms search
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
>  09a88d930 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java
>  278ec5db1 
>   
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSuggestionsResult.java
>  054ea2e87 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  3feae27b5 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  23fd7e9c5 
>   
> repository/src/main/java/org/apache/atlas/discovery/SuggestionsProvider.java 
> 4ee1b13a9 
>   
> repository/src/main/java/org/apache/atlas/discovery/SuggestionsProviderImpl.java
>  c00a65066 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 908ae7033 
> 
> 
> Diff: https://reviews.apache.org/r/71738/diff/2/
> 
> 
> Testing
> ---
> 
> Precommit: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1535/console
> 
> Manually validated that suggestions api filters based on provided field name.
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 71738: ATLAS-3517: Enhance suggestions REST API to have optional 'fieldName' parameter to get suggestions from

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71738/
---

(Updated Nov. 8, 2019, 10:01 a.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer 
Shaikh, and Sidharth Mishra.


Changes
---

addresssed review comments.


Bugs: ATLAS-3517
https://issues.apache.org/jira/browse/ATLAS-3517


Repository: atlas


Description
---

Suggestions REST API searches against all indexed fields (terms) to retrieve 
the top frequently used terms.

we don't have a way to retrieve suggestions only from a specific indexed field.

This enhancement introduces a new optional parameter in suggestions API - 
'fieldName', which when specified will provide suggestions against that indexed 
field name.

 

This will be useful to get suggestions on only a specific field - say labels 


*Solution:*

Solr already provides a way to search on etrms against a specific field.

https://lucene.apache.org/solr/guide/6_6/the-terms-component.html

"terms.fl=" need to added as part of the request to solr 
client on terms search


Diffs (updated)
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
 09a88d930 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java
 278ec5db1 
  
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSuggestionsResult.java 
054ea2e87 
  
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 
3feae27b5 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
23fd7e9c5 
  repository/src/main/java/org/apache/atlas/discovery/SuggestionsProvider.java 
4ee1b13a9 
  
repository/src/main/java/org/apache/atlas/discovery/SuggestionsProviderImpl.java
 c00a65066 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 908ae7033 


Diff: https://reviews.apache.org/r/71738/diff/2/

Changes: https://reviews.apache.org/r/71738/diff/1-2/


Testing
---

Precommit: 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1535/console

Manually validated that suggestions api filters based on provided field name.


Thanks,

Sarath Subramanian

[jira] [Commented] (ATLAS-3515) Migration: New Types Referenced within Existing Types

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970426#comment-16970426
 ] 

ASF subversion and git services commented on ATLAS-3515:


Commit 77dbea47bf42daceefea6bd03b81116aefa2f755 in atlas's branch 
refs/heads/branch-2.0 from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=77dbea4 ]

ATLAS-3515: Migration import now creates new types and then updates existing 
types.


> Migration: New Types Referenced within Existing Types
> -
>
> Key: ATLAS-3515
> URL: https://issues.apache.org/jira/browse/ATLAS-3515
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0, trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk, 2.1.0
>
> Attachments: ATLAS-3515-Migration-import-type-definition.patch
>
>
> *Background*
> During migration import (and regular Import) types coming in are processed in 
> the following way:
>  * New types are determined.
>  * Modification to existing types are determined.
> *Steps to Duplicate*
> _Pre-requisites:_
>  * Create type definition with new _structDef_.
>  * Modify _hive_table_ (existing) such a new attribute references the 
> _structDef_.
> _Steps_
>  * Initiate migration import.
> _Expected outcome_
>  * New types should be part of the types definition (using REST APIs and 
> property pages).
>  * Data should get migrated.
> _Actual outcome_
>  * Type definition import fails.
> _Additional information_
> Stack trace:
> {code:java}
> org.apache.atlas.exception.AtlasBaseException: Given typename 
> hive_table_profile_data was invalid
> at 
> org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.tryUpdateByName(AtlasTypeDefGraphStore.java:1025)
> 
> at 
> org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateEntityDefByName(AtlasTypeDefGraphStore.java:285)
> 
> at 
> org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore$$FastClassBySprin
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [atlas] mneethiraj commented on a change in pull request #75: ATLAS-3519: pool performance in hive lineage query

[GitBox]

mneethiraj commented on a change in pull request #75: ATLAS-3519: pool 
performance in hive lineage query
URL: https://github.com/apache/atlas/pull/75#discussion_r344252885
 
 

 ##
 File path: 
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 ##
 @@ -330,13 +332,16 @@ public void exportToGson(OutputStream os) throws 
IOException {
 
 @Override
 public GremlinGroovyScriptEngine getGremlinScriptEngine() {
+return scriptEngine;
 
 Review comment:
   GremlinGroovyScriptEngine returned here may not safe for use in multiple 
threads simultaneously i.e. side-effects of an execution in a thread might be 
visible in other threads. I suggest to use a 
ThreadLocal, to instantiate and use one script 
engine per thread.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (ATLAS-3519) pool performance in hive lineage query

[Madhan Neethiraj (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970362#comment-16970362
 ] 

Madhan Neethiraj commented on ATLAS-3519:
-

[~leslizhang] - thanks for details of the issue and the patch that improves the 
lineage query performance from 3/sec to 200/sec!

GremlinGroovyScriptEngine returned by AtlasJanusGraph.getGremlinScriptEngine() 
may not safe for use in multiple threads simultaneously i.e. side-effects of an 
execution in a thread might be visible in other threads. I suggest to use a 
ThreadLocal in AtlasJanusGraph, to instantiate and 
use one script engine per thread.

> pool performance in hive lineage query 
> ---
>
> Key: ATLAS-3519
> URL: https://issues.apache.org/jira/browse/ATLAS-3519
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.1.0, 2.0.0
>Reporter: zhangxiong
>Priority: Minor
>  Labels: lineage, performance
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> when query lineage info for hive, the throughput reachs only 3 query per 
> second.
> we use jstack and jfr+flame graph tools and found that the  
> GremlinGroovyScriptEngine object loading class  logic takes much of the time. 
> so we make the GremlinGroovyScriptEngine a global singleton object, which 
> would be shared accross all query threads. finally the query performance 
> improved much and got 200 qps.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [atlas] bolkedebruin commented on issue #75: ATLAS-3519: pool performance in hive lineage query

[GitBox]

bolkedebruin commented on issue #75: ATLAS-3519: pool performance in hive 
lineage query
URL: https://github.com/apache/atlas/pull/75#issuecomment-551745658
 
 
   @zhangxiong0301 Usually the Atlas folks use the Apache review board to 
review patches. Although we should maybe request them to start using github PRs 
as well cc @mneethiraj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [atlas] zhangxiong0301 opened a new pull request #75: ATLAS-3519: pool performance in hive lineage query

[GitBox]

zhangxiong0301 opened a new pull request #75: ATLAS-3519: pool performance in 
hive lineage query
URL: https://github.com/apache/atlas/pull/75
 
 
   when query lineage info for hive, the throughput reachs only 3 query per 
second.
   
   we use jstack and jfr+flame graph tools and found that the  
GremlinGroovyScriptEngine object loading class  logic takes much of the time. 
so we make the GremlinGroovyScriptEngine a global singleton object, which would 
be shared accross all query threads. finally the query performance improved 
much and got 200 qps.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (ATLAS-3519) pool performance in hive lineage query

[zhangxiong (Jira)]

zhangxiong created ATLAS-3519:
-

 Summary: pool performance in hive lineage query 
 Key: ATLAS-3519
 URL: https://issues.apache.org/jira/browse/ATLAS-3519
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangxiong


when query lineage info for hive, the throughput reachs only 3 query per second.



we use jstack and jfr+flame graph tools and found that the  
GremlinGroovyScriptEngine object loading class  logic takes much of the time. 
so we make the GremlinGroovyScriptEngine a global singleton object, which would 
be shared accross all query threads. finally the query performance improved 
much and got 200 qps.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3511) Relationship attributes not forming while creating entities using a bulk entity API request

[chaitali borole (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970044#comment-16970044
 ] 

chaitali borole commented on ATLAS-3511:


Hi,

I have attached the corrected entity instance JSON for bulk api call where 
multiple relationshipattributes  can be created.

Can you please verify from your side.

 

> Relationship attributes not forming while creating entities using a bulk 
> entity API request
> ---
>
> Key: ATLAS-3511
> URL: https://issues.apache.org/jira/browse/ATLAS-3511
> Project: Atlas
>  Issue Type: Bug
>Reporter: Rahul Kurup
>Assignee: Mayank Jain
>Priority: Major
> Attachments: entity_instance, entity_instancecorrected, 
> entity_typedef_1, entity_typedef_2, relationship_typedef
>
>
> Steps:
> 1.Create 2 entity types based on the JSON structure (see attached file 
> [^entity_typedef_1] and [^entity_typedef_2])  with relationship attributes as 
> described in attached file [^relationship_typedef]  through a POST request on 
> the typedef API ({color:#505050}api/atlas/v2/types/typedefs{color}).
> 2. Create entities with relationship attributes for each of the entity types 
> (as described in step 1) through a POST request on the bulk API (see attached 
> file [^entity_instance] for reference) ([^relationship_typedef])
> You will observe that the definition of the attributes and the number of 
> attributes varies for each of the entities. Hence we are not able to create 
> relationship instances using bulk REST API call.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-3511) Relationship attributes not forming while creating entities using a bulk entity API request

[chaitali borole (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chaitali borole reassigned ATLAS-3511:
--

Assignee: chaitali borole  (was: Mayank Jain)

> Relationship attributes not forming while creating entities using a bulk 
> entity API request
> ---
>
> Key: ATLAS-3511
> URL: https://issues.apache.org/jira/browse/ATLAS-3511
> Project: Atlas
>  Issue Type: Bug
>Reporter: Rahul Kurup
>Assignee: chaitali borole
>Priority: Major
> Attachments: entity_instance, entity_instancecorrected, 
> entity_typedef_1, entity_typedef_2, relationship_typedef
>
>
> Steps:
> 1.Create 2 entity types based on the JSON structure (see attached file 
> [^entity_typedef_1] and [^entity_typedef_2])  with relationship attributes as 
> described in attached file [^relationship_typedef]  through a POST request on 
> the typedef API ({color:#505050}api/atlas/v2/types/typedefs{color}).
> 2. Create entities with relationship attributes for each of the entity types 
> (as described in step 1) through a POST request on the bulk API (see attached 
> file [^entity_instance] for reference) ([^relationship_typedef])
> You will observe that the definition of the attributes and the number of 
> attributes varies for each of the entities. Hence we are not able to create 
> relationship instances using bulk REST API call.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3511) Relationship attributes not forming while creating entities using a bulk entity API request

[chaitali borole (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chaitali borole updated ATLAS-3511:
---
Attachment: entity_instancecorrected

> Relationship attributes not forming while creating entities using a bulk 
> entity API request
> ---
>
> Key: ATLAS-3511
> URL: https://issues.apache.org/jira/browse/ATLAS-3511
> Project: Atlas
>  Issue Type: Bug
>Reporter: Rahul Kurup
>Assignee: Mayank Jain
>Priority: Major
> Attachments: entity_instance, entity_instancecorrected, 
> entity_typedef_1, entity_typedef_2, relationship_typedef
>
>
> Steps:
> 1.Create 2 entity types based on the JSON structure (see attached file 
> [^entity_typedef_1] and [^entity_typedef_2])  with relationship attributes as 
> described in attached file [^relationship_typedef]  through a POST request on 
> the typedef API ({color:#505050}api/atlas/v2/types/typedefs{color}).
> 2. Create entities with relationship attributes for each of the entity types 
> (as described in step 1) through a POST request on the bulk API (see attached 
> file [^entity_instance] for reference) ([^relationship_typedef])
> You will observe that the definition of the attributes and the number of 
> attributes varies for each of the entities. Hence we are not able to create 
> relationship instances using bulk REST API call.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [atlas] zhangxiong0301 opened a new pull request #74: [ATLAS-3424][addons/hbase-brige] hbase 2.2.0 does not execute the Atlas implemented coprocessor

[GitBox]

zhangxiong0301 opened a new pull request #74: [ATLAS-3424][addons/hbase-brige] 
hbase 2.2.0 does not execute the Atlas implemented coprocessor
URL: https://github.com/apache/atlas/pull/74
 
 
   in atlas 2.0.0, we use hbase-hook to capture the changes of name 
space,table,column faimily schema in hbase 2.2.0. According to steps of the  
official document, we found that the hbase master loaded HBaseAtlasCoprocessor 
, but it will not call the  methods like postCreateTable/postDeleteTable.
   from remote debug,we found that HBaseAtlasCoprocessor must override the 
default implementation of MasterCoprocessor.getMasterObserver method to make 
Atlas coprocessor runs correctly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (ATLAS-3424) hbase 2.2.0 doesn't execute the Atlas implemented coprocessor

[zhangxiong (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangxiong updated ATLAS-3424:
--
Attachment: (was: HBaseAtlasCoprocessor.java.patch)

> hbase 2.2.0 doesn't execute the Atlas implemented  coprocessor
> --
>
> Key: ATLAS-3424
> URL: https://issues.apache.org/jira/browse/ATLAS-3424
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Affects Versions: 2.0.0
> Environment: atlas2.0 , hbase2.2.0
>Reporter: zhangxiong
>Assignee: zhangxiong
>Priority: Major
>  Labels: hbase-hhok, patch
>
> in atlas 2.0.0, we use hbase-hook to capture the changes of name 
> space,table,column faimily schema in hbase 2.2.0. According to steps of the  
> official document, we found that the hbase master loaded 
> HBaseAtlasCoprocessor , but it will not call the  methods like 
> postCreateTable/postDeleteTable.
> from remote debug,we found that HBaseAtlasCoprocessor must override the 
> default implementation of MasterCoprocessor.getMasterObserver method to make 
> Atlas coprocessor runs correctly.
> ,



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-3424) hbase 2.2.0 doesn't execute the Atlas implemented coprocessor

[zhangxiong (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangxiong reassigned ATLAS-3424:
-

Assignee: zhangxiong

> hbase 2.2.0 doesn't execute the Atlas implemented  coprocessor
> --
>
> Key: ATLAS-3424
> URL: https://issues.apache.org/jira/browse/ATLAS-3424
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Affects Versions: 2.0.0
> Environment: atlas2.0 , hbase2.2.0
>Reporter: zhangxiong
>Assignee: zhangxiong
>Priority: Major
>  Labels: hbase-hhok, patch
>
> in atlas 2.0.0, we use hbase-hook to capture the changes of name 
> space,table,column faimily schema in hbase 2.2.0. According to steps of the  
> official document, we found that the hbase master loaded 
> HBaseAtlasCoprocessor , but it will not call the  methods like 
> postCreateTable/postDeleteTable.
> from remote debug,we found that HBaseAtlasCoprocessor must override the 
> default implementation of MasterCoprocessor.getMasterObserver method to make 
> Atlas coprocessor runs correctly.
> ,



--
This message was sent by Atlassian Jira
(v8.3.4#803005)