Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-04 Thread Nixon Rodrigues 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221472
---


Ship it!




Ship It!

- Nixon Rodrigues


On Aug. 5, 2020, 4:27 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated Aug. 5, 2020, 4:27 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 044179d44 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/6/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2084
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-04 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221471
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 5, 2020, 4:27 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated Aug. 5, 2020, 4:27 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 044179d44 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/6/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2084
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-04 Thread Pinal Shah 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated Aug. 5, 2020, 4:27 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
  
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 
86572598a 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
044179d44 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/6/

Changes: https://reviews.apache.org/r/72646/diff/5-6/


Testing (updated)
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2084


Thanks,

Pinal Shah

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-04 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221470
---


Fix it, then Ship it!





intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Line 19 (original), 19 (patched)


Only import related changes are in AtlasEntityType.java. Please review and 
revert.


- Madhan Neethiraj


On Aug. 4, 2020, 11:01 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated Aug. 4, 2020, 11:01 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 044179d44 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/5/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-04 Thread Pinal Shah 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated Aug. 4, 2020, 11:01 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Changes
---

added support for Edge attribute predicate


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
  
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 
86572598a 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
044179d44 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/5/

Changes: https://reviews.apache.org/r/72646/diff/4-5/


Testing
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
(Failed in Impala build)


Thanks,

Pinal Shah

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-08-01 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221445
---


Fix it, then Ship it!





repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1523 (patched)


This handles edge attributes of string type. Consider handling other 
datatypes as well in a subsequent patch.

An implementaiton similar to vertex-attribute predicate #226 - #271, which 
uses VertexAttributePredicateGenerator; perhaps, introducing 
EdgeAttributePredicateGenerator will be useful.


- Madhan Neethiraj


On July 31, 2020, 12:43 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 31, 2020, 12:43 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/4/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-31 Thread Pinal Shah 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated July 31, 2020, 12:43 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Changes
---

Addressed review comment


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
  
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 
86572598a 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/4/

Changes: https://reviews.apache.org/r/72646/diff/3-4/


Testing
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
(Failed in Impala build)


Thanks,

Pinal Shah

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-29 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221416
---




intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 539 (patched)


I suggest to move this method to AtlasAttribute:

  class AtlasAttribute {
...

public AtlasEntityType getReferencedEntityType(AtlasTypeRegistry 
typeRegistry) {
  return 
getReferencedEntityType(typeRegistry.getType(attributeDef.getTypeName()))
}
  }


- Madhan Neethiraj


On July 29, 2020, 1:41 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 29, 2020, 1:41 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/3/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-29 Thread Pinal Shah 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated July 29, 2020, 1:41 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description (updated)
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs
-

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 
86572598a 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/3/


Testing
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
(Failed in Impala build)


Thanks,

Pinal Shah

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-29 Thread Pinal Shah 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated July 29, 2020, 1:38 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Changes
---

Addressed Review comments


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 
86572598a 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/3/

Changes: https://reviews.apache.org/r/72646/diff/2-3/


Testing
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
(Failed in Impala build)


Thanks,

Pinal Shah

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-23 Thread Pinal Shah 


> On July 23, 2020, 10:50 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Line 607 (original), 616 (patched)
> > 
> >
> > what if out/in edges size is a lot, do you bring everything into memory 
> > just to get size? 
> > 
> > Do you need entire edges in the list or maybe we can maintain a counter 
> > and increment?

Yes Sarath you are right but,
approximate count is count without limit/offset
I brought it to inmemory to filter out 'ACTIVE' edges incase of excludeDeleted 
flag
Other way is to fire another query with 'ACTIVE'filter


> On July 23, 2020, 10:50 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Lines 682 (patched)
> > 
> >
> > Constants.GUID_PROPERTY_KEY).value() can be null (mostly not, but some 
> > bad vertices). Consider checking for null value of guid.
> > 
> > Also, since you already have vertex, do we need additional 
> > 'endVerticesGuid' list? Can we extract 'guid' from Vertex 'v' and add it to 
> > 'resultList' is same loop. 
> > 
> > resultList.add(entityRetriever.toAtlasEntityHeader(endVertexGuid, 
> > attributes));
> > 
> > (or)
> > 
> > Maybe map Vertex to AtlasVertex and directly call:
> > 
> > resultList.add(entityRetriever.toAtlasEntityHeader(atlasVertex, 
> > attributes));

I have added  (v != null && 
v.property(Constants.GUID_PROPERTY_KEY).isPresent()) for the null check
Correct, we dont need extra list, will address it


- Pinal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221343
---


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-23 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221343
---




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Line 607 (original), 616 (patched)


what if out/in edges size is a lot, do you bring everything into memory 
just to get size? 

Do you need entire edges in the list or maybe we can maintain a counter and 
increment?



repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 682 (patched)


Constants.GUID_PROPERTY_KEY).value() can be null (mostly not, but some bad 
vertices). Consider checking for null value of guid.

Also, since you already have vertex, do we need additional 
'endVerticesGuid' list? Can we extract 'guid' from Vertex 'v' and add it to 
'resultList' is same loop. 

resultList.add(entityRetriever.toAtlasEntityHeader(endVertexGuid, 
attributes));

(or)

Maybe map Vertex to AtlasVertex and directly call:

resultList.add(entityRetriever.toAtlasEntityHeader(atlasVertex, 
attributes));


- Sarath Subramanian


On July 6, 2020, 2:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 2:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-22 Thread Ashutosh Mestry via Review Board 


> On July 20, 2020, 10:54 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Line 649 (original), 662 (patched)
> > 
> >
> > Consider replacing this with AtlasGraphTraversal. This should be easy 
> > change but will give good performance mileage.
> 
> Pinal Shah wrote:
> Thanks Ashutosh for reviewing.
> 
> You mean to add these methods, (bothE, otherV) to AtlasGraphTraversal and 
> make use from it?
> How will it make difference in performance?

You are right, moving to AtlasGraphTraversal will not affect performance. It is 
just that it will be in 1 place.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
---


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-21 Thread Pinal Shah 


> On July 20, 2020, 10:54 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Line 649 (original), 662 (patched)
> > 
> >
> > Consider replacing this with AtlasGraphTraversal. This should be easy 
> > change but will give good performance mileage.

Thanks Ashutosh for reviewing.

You mean to add these methods, (bothE, otherV) to AtlasGraphTraversal and make 
use from it?
How will it make difference in performance?


- Pinal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
---


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-21 Thread Pinal Shah 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221282
---




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 705 (patched)


I tested it with 5000 edge vertices.
Will vertexTotal() method be efficient than this?


- Pinal Shah


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-21 Thread Pinal Shah 


> On July 20, 2020, 10:54 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Lines 705 (patched)
> > 
> >
> > What is the volume of data you have tested this with?

I tested it with 5000 edge vertices.
Will vertexTotal() method be efficient than this?


- Pinal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
---


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-20 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
---




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 622 (patched)


getId should be sufficient here.



repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Line 649 (original), 662 (patched)


Consider replacing this with AtlasGraphTraversal. This should be easy 
change but will give good performance mileage.



repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 705 (patched)


What is the volume of data you have tested this with?


- Ashutosh Mestry


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-06 Thread Pinal Shah 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated July 6, 2020, 9:14 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs
-

  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 


Diff: https://reviews.apache.org/r/72646/diff/1/


Testing (updated)
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
(Failed in Impala build)


Thanks,

Pinal Shah

Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-06 Thread Pinal Shah 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs
-

  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 


Diff: https://reviews.apache.org/r/72646/diff/1/


Testing
---

Manually tested
Precommit :


Thanks,

Pinal Shah