Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221323
---




repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
Lines 44 (patched)


I suggest use AtlasConfiguration for this. Also, I think this should be 
true by default.


- Ashutosh Mestry


On July 20, 2020, 9:48 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 20, 2020, 9:48 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AbstractGraphDatabaseTest.java
>  35004157f 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusDatabaseTest.java
>  5cd55093e 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e35f3594f 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
>  5a9ac2abe 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
>  d3111f110 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/1/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient

[Sidharth Mishra]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72698/#review221321
---




atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 63 (patched)


char[] is always preferred over String - check 
https://stackoverflow.com/questions/8881291/why-is-char-preferred-over-string-for-passwords#:~:text=Since%20Strings%20are%20immutable%20there,2.

Please check if we can add a new contructor at client v2 and do this ow 
lets file a jira and track this for client v2



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 75 (patched)


Please check if we should change the code snippet from 72:84 to something 
like:

public static void main(String[] args) throws Exception {
String[] basicAuthUsernamePassword = null;
String[] urls = null;

urls = getServerUrl();
if (!AuthenticationUtil.isKerberosAuthenticationEnabled()) {
basicAuthUsernamePassword = getUserInput();
new AtlasClientBaseExample(urls, basicAuthUsernamePassword);
} else {
new AtlasClientBaseExample(urls);
}

try {
// TypeDef



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 83 (patched)


In case of kerberos urls will be null as we populate that if kerberos not 
enabled.



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 86 (patched)


Instead of commenting like typedef then entity examples etc. it would be 
good to move these to separate private functions. Please refer - 

FUNCTIONS SHOULD DO ONE THING. THEY SHOULD DO IT WELL. THEY SHOULD DO IT 
ONLY (more details - 
https://learning.oreilly.com/library/view/clean-code/9780136083238/chapter03.html#ch3)


https://softwareengineering.stackexchange.com/questions/137941/should-a-method-do-one-thing-and-be-good-at-it



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 140 (patched)


Please use Console.readPassword instead


- Sidharth Mishra


On July 21, 2020, 4:24 p.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
> 
> (Updated July 21, 2020, 4:24 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Using this project users can get an idea as how to integrate with Atlas using 
> AtlasCleint. This helps the user to understand the basic rest functionality 
> of Atlas such as
> 
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
> 
> 
> Diffs
> -
> 
>   atlas-examples/pom.xml PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/DiscoveryExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/EntityRestBaseExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/GlossaryExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/LineageExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java 
> PRE-CREATION 
>   pom.xml 5e0442ae5 
> 
> 
> Diff: https://reviews.apache.org/r/72698/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>

Re: Review Request 72666: Notification: Solution to Memory Build-up

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221317
---


Ship it!




Ship It!

- Nikhil Bonte


On July 22, 2020, 4:16 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 22, 2020, 4:16 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferListAccessor.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListAccessorTest.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
>  a677b315c 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/6/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Ashutosh Mestry via Review Board]

> On July 21, 2020, 8:38 p.m., Ashutosh Mestry wrote:
> > Here's what I have tried so far:
> > - Concurrent entity creation using my own test rig. This creates entities 
> > concurrently but it does not allow for entity with same qualifiedName to be 
> > created by different workers.
> > - Ingest via Kafka queue. This is to verify that there is not performance 
> > degradation. 
> > 
> > So far results are positive. I did not obseve any performance or impact on 
> > accuracy.
> > 
> > Yet to try:
> > - Migration from 0.8 to latest.
> > - Migration import using ZipDirect format.
> 
> Madhan Neethiraj wrote:
> Ashutosh - what is the behavior seen in case of simultaneous attempts to 
> entity an entity? Do all but 1 attempt fail?

Please see my comment below.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221296
---


On July 20, 2020, 9:48 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 20, 2020, 9:48 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AbstractGraphDatabaseTest.java
>  35004157f 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusDatabaseTest.java
>  5cd55093e 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e35f3594f 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
>  5a9ac2abe 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
>  d3111f110 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/1/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Ashutosh Mestry via Review Board]

> On July 22, 2020, 4:30 p.m., Ashutosh Mestry wrote:
> > I attempted to create same entity (entity with same qualifiedName) from 
> > multiple requests.
> > 
> > I did this experiment:
> > - Created an entity payload.
> > - Used bulk entity CURL calls from 5 different requests. 
> > 
> > Repeated this exercise on database with:
> > - Existing data: I was able to create duplicate entities.
> > - Blank database: Only 1 entity with the same qualifiedName was created.
> > 
> > My thinking is that we will see real value of this if we have a Java patch 
> > handler that fixes existing data.

Additionally: The zip-file-based migration import proceeded without any 
performance deterioration.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221314
---


On July 20, 2020, 9:48 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 20, 2020, 9:48 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AbstractGraphDatabaseTest.java
>  35004157f 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusDatabaseTest.java
>  5cd55093e 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e35f3594f 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
>  5a9ac2abe 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
>  d3111f110 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/1/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221314
---



I attempted to create same entity (entity with same qualifiedName) from 
multiple requests.

I did this experiment:
- Created an entity payload.
- Used bulk entity CURL calls from 5 different requests. 

Repeated this exercise on database with:
- Existing data: I was able to create duplicate entities.
- Blank database: Only 1 entity with the same qualifiedName was created.

My thinking is that we will see real value of this if we have a Java patch 
handler that fixes existing data.

- Ashutosh Mestry


On July 20, 2020, 9:48 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 20, 2020, 9:48 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  fca789027 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AbstractGraphDatabaseTest.java
>  35004157f 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusDatabaseTest.java
>  5cd55093e 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e35f3594f 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
>  5a9ac2abe 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
>  d3111f110 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/1/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72666: Notification: Solution to Memory Build-up

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/
---

(Updated July 22, 2020, 4:16 p.m.)


Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Changes
---

Updates include:
- Added configuration property 
atlas.notification.fixed.buffer.items.increment.count with default set to 10.


Bugs: ATLAS-3878
https://issues.apache.org/jira/browse/ATLAS-3878


Repository: atlas


Description
---

**Background**
See JIRA for details.

*Analysis* Using memory profiling tools, it was observed that large number of 
notification objects were created. These stayed in memory and later were 
promoted to higher generation, thereby taking even longer to be collected.

**Approach**
Using the fixed-buffer approach to address the problem of creating large number 
of small objects.

New *FixedBufferList* This is an encapsulation over *ArrayList*. During initial 
allocation, list is populated with default values. Features:
- Setting of values to these pre-allocated objects is achieved by first doing a 
*get* on the element and then assigning values to it.
- *toList* fetches the sub-list from the encapsulating list. This uses the 
state within the class to fetch the right length for the returning array.

New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
necessary since the list is now part class's state.
Modified *EntityAuditListenerV2* Uses the new classes.
Modifed *EntityNotificationListener* Uses the new classes.

**Verification**
- Using the test setup, the memory usage was observed over a period of 24 hrs. 
- Memory usage and object allocation was obvserved using memory profiler.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
  intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/utils/FixedBufferListAccessor.java 
PRE-CREATION 
  intg/src/test/java/org/apache/atlas/utils/FixedBufferListAccessorTest.java 
PRE-CREATION 
  intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
 79527acfa 
  
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
 a677b315c 


Diff: https://reviews.apache.org/r/72666/diff/6/

Changes: https://reviews.apache.org/r/72666/diff/5-6/


Testing
---

**Unit testing**
Unit tests added for the new classes.

**Volume testing**
Setup:
- Node: Threads 40, Core: 40, Allocated Memory: 12 GB
- Multiple Kafka queues ingesting data.
- Bulk entity creation using custom script ingesting 100M entities.

Memory usage stayed between 0 and 5% during the 24 hr period.

With:
- Workers: 64
- Batch size: 50 (fewer elements in batch improve commit time and audit write 
time).
- Throughput: ~1.2 M entities per hour. Without out of memory error.

**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/


Thanks,

Ashutosh Mestry

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

[Ashutosh Mestry via Review Board]

> On July 20, 2020, 10:54 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Line 649 (original), 662 (patched)
> > 
> >
> > Consider replacing this with AtlasGraphTraversal. This should be easy 
> > change but will give good performance mileage.
> 
> Pinal Shah wrote:
> Thanks Ashutosh for reviewing.
> 
> You mean to add these methods, (bothE, otherV) to AtlasGraphTraversal and 
> make use from it?
> How will it make difference in performance?

You are right, moving to AtlasGraphTraversal will not affect performance. It is 
just that it will be in 1 place.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
---


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

[jira] [Commented] (ATLAS-3901) AD user default role

[Nixon Rodrigues (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162689#comment-17162689
 ] 

Nixon Rodrigues commented on ATLAS-3901:


[~theo11],

Atlas lookup for user group from local os linux  user group or from 
hadoopUserGroup mapping.
Please sync/map your AD users to os users / hadoop usergroup, so that groups 
are read correctly.

> AD user default role
> 
>
> Key: ATLAS-3901
> URL: https://issues.apache.org/jira/browse/ATLAS-3901
> Project: Atlas
>  Issue Type: Bug
>Reporter: theo11
>Priority: Major
>
> Hello,
> I'm having trouble to set up AD users to be correctly binded to DATA_STEWARD 
> role. Login works correctly, but user has no permissions like relationships 
> etc.
> Needed property in atlas-application.properties is set as follows:
> atlas.authentication.method.ldap.ad.default.role=DATA_STEWARD
> All roles are defaults from atlas-simple-authz-policy.json. There are no 
> related error entries in Atlas log.
> Could you advice?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3897) UI: Normalize list of propagated classifications

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162669#comment-17162669
 ] 

ASF subversion and git services commented on ATLAS-3897:


Commit ec314fdeb871a479194af5b543a72df76472cb4c in atlas's branch 
refs/heads/master from Keval Bhatt
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=ec314fd ]

ATLAS-3897: UI: Normalize list of propagated classifications


> UI: Normalize list of propagated classifications
> 
>
> Key: ATLAS-3897
> URL: https://issues.apache.org/jira/browse/ATLAS-3897
> Project: Atlas
>  Issue Type: Bug
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3897-1.patch, ATLAS-3897-2.patch, 
> ATLAS-3897-3.patch, ATLAS-3897.patch, Screen Shot 2020-07-17 at 6.13.04 PM.png
>
>
> When an entity has multiple instances of the same propagated classification 
> (e.g. via join of Hive tables), the lineage view will show each of them in 
> the list. The result may be a long list of classifications with identical 
> entries They don't seem to offer any additional information.
> I propose we normalize the list (think _uniq_ command in unix)
>  
> !Screen Shot 2020-07-17 at 6.13.04 PM.png|width=593,height=359!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3897) UI: Normalize list of propagated classifications

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162670#comment-17162670
 ] 

ASF subversion and git services commented on ATLAS-3897:


Commit 6f3fc5ac3c9b14b6ecfea00c71effe2564504119 in atlas's branch 
refs/heads/branch-2.0 from Keval Bhatt
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=6f3fc5a ]

ATLAS-3897: UI: Normalize list of propagated classifications

(cherry picked from commit ec314fdeb871a479194af5b543a72df76472cb4c)


> UI: Normalize list of propagated classifications
> 
>
> Key: ATLAS-3897
> URL: https://issues.apache.org/jira/browse/ATLAS-3897
> Project: Atlas
>  Issue Type: Bug
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3897-1.patch, ATLAS-3897-2.patch, 
> ATLAS-3897-3.patch, ATLAS-3897.patch, Screen Shot 2020-07-17 at 6.13.04 PM.png
>
>
> When an entity has multiple instances of the same propagated classification 
> (e.g. via join of Hive tables), the lineage view will show each of them in 
> the list. The result may be a long list of classifications with identical 
> entries They don't seem to offer any additional information.
> I propose we normalize the list (think _uniq_ command in unix)
>  
> !Screen Shot 2020-07-17 at 6.13.04 PM.png|width=593,height=359!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (ATLAS-3901) AD user default role

[theo11 (Jira)]

theo11 created ATLAS-3901:
-

 Summary: AD user default role
 Key: ATLAS-3901
 URL: https://issues.apache.org/jira/browse/ATLAS-3901
 Project: Atlas
  Issue Type: Bug
Reporter: theo11


Hello,

I'm having trouble to set up AD users to be correctly binded to DATA_STEWARD 
role. Login works correctly, but user has no permissions like relationships etc.

Needed property in atlas-application.properties is set as follows:

atlas.authentication.method.ldap.ad.default.role=DATA_STEWARD

All roles are defaults from atlas-simple-authz-policy.json. There are no 
related error entries in Atlas log.

Could you advice?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3897) UI: Normalize list of propagated classifications

[Durga Kadam (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162658#comment-17162658
 ] 

Durga Kadam commented on ATLAS-3897:


+1 for the patch [~kevalbhatt]

> UI: Normalize list of propagated classifications
> 
>
> Key: ATLAS-3897
> URL: https://issues.apache.org/jira/browse/ATLAS-3897
> Project: Atlas
>  Issue Type: Bug
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3897-1.patch, ATLAS-3897-2.patch, 
> ATLAS-3897-3.patch, ATLAS-3897.patch, Screen Shot 2020-07-17 at 6.13.04 PM.png
>
>
> When an entity has multiple instances of the same propagated classification 
> (e.g. via join of Hive tables), the lineage view will show each of them in 
> the list. The result may be a long list of classifications with identical 
> entries They don't seem to offer any additional information.
> I propose we normalize the list (think _uniq_ command in unix)
>  
> !Screen Shot 2020-07-17 at 6.13.04 PM.png|width=593,height=359!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3879) Ozone: ozone_key entity is directly created under ozone_bucket

[Nikhil Bonte (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3879:

Description: 
The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
entity is getting created under ozone_bucket, but there is no support to create 
a hierarchy of parent ozone_key and child ozone_key.

 

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_volume)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support creating the hierarchy of 
ozone_keys.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_volume)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.

  was:
The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
entity is getting created under ozone_bucket, but there is no support to create 
a hierarchy of parent ozone_key and child ozone_key.

 

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support creating the hierarchy of 
ozone_keys.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.


> Ozone: ozone_key entity is directly created under ozone_bucket
> --
>
> Key: ATLAS-3879
> URL: https://issues.apache.org/jira/browse/ATLAS-3879
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
>
> The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
> entity is getting created under ozone_bucket, but there is no support to 
> create a hierarchy of parent ozone_key and child ozone_key.
>  
> e.g.
> {code:java}
> -> CREATE EXTERNAL TABLE sales (id int) row format delimited fields 
> terminated by ' ' stored as textfile location 
> 'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
> This query will create only 1 ozone_key directly under bucket "bucket1".
> Created entities :
>  * o3fs://volume1@cm (ozone_volume)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)
>  
> This Jira will improve implementation to support creating the hierarchy of 
> ozone_keys.
> With this, above query will result into following entites
>  * o3fs://volume1@cm (ozone_volume)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
> ozone_key*
>  
> This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
> models.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72698/#review221306
---




atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 20 (patched)


nit: removed unused imports line 20,21



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 134 (patched)


'url' is never used. consider removing.



atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java
Lines 58 (patched)


isCompositeSourceConstraint => isOwnedRefConstraint



atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java
Lines 59 (patched)


isCompositeTargetConstraint => isInverseRefConstraint


- Sarath Subramanian


On July 21, 2020, 9:24 a.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
> 
> (Updated July 21, 2020, 9:24 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Using this project users can get an idea as how to integrate with Atlas using 
> AtlasCleint. This helps the user to understand the basic rest functionality 
> of Atlas such as
> 
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
> 
> 
> Diffs
> -
> 
>   atlas-examples/pom.xml PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/DiscoveryExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/EntityRestBaseExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/GlossaryExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/LineageExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java 
> PRE-CREATION 
>   pom.xml 5e0442ae5 
> 
> 
> Diff: https://reviews.apache.org/r/72698/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>