[jira] [Commented] (ATLAS-3782) Support NOT_CONTAINS operator in basic search with search parameters

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164400#comment-17164400
 ] 

ASF subversion and git services commented on ATLAS-3782:


Commit 9a067d38e27fc2255647133373b952a6a64ed1a4 in atlas's branch 
refs/heads/master from Pinal Shah
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=9a067d3 ]

ATLAS-3782 : Support NOT_CONTAINS operator in basic search

Signed-off-by: nixonrodrigues 


> Support NOT_CONTAINS operator in basic search with search parameters
> 
>
> Key: ATLAS-3782
> URL: https://issues.apache.org/jira/browse/ATLAS-3782
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Andreas Danckert
>Assignee: Pinal
>Priority: Minor
>  Labels: BasicSearch
>
> The operator 'SearchParameters.Operator.NOT_CONTAINS' is defined and 
> implemented in SearchProcessors. It would allow a search of entities that do 
> not contain in given string in a specified attribute (eg exclude entities 
> from search that contain 'temp' in the qualified name). 
> However, this operator is currently ignored when used in REST calls and thus 
> not effective. Presumably the reason for this behaviour is that the operator 
> is not listed as case value in any of the switch statements on the search 
> operator in SearchProcessor.java.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72567: ATLAS-3782 : Support NOT_CONTAINS operator in basic search

[Nixon Rodrigues]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72567/#review221358
---


Ship it!




Ship It!

- Nixon Rodrigues


On June 30, 2020, 1:09 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72567/
> ---
> 
> (Updated June 30, 2020, 1:09 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3782
> https://issues.apache.org/jira/browse/ATLAS-3782
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> The operator 'SearchParameters.Operator.NOT_CONTAINS' is defined and 
> implemented in SearchProcessors. It would allow a search of entities that do 
> not contain in given string in a specified attribute (eg exclude entities 
> from search that contain 'temp' in the qualified name). 
> 
> **WorkAround:**
> JanusGraph doesn't allow NOT_CONTAINS operator. So we will handle this in 
> inmemory
> BasicSearch generates query via three modes
> 1. Index query -> NOT_CONTAINS will not be supported
> 2. InMemeory Predicates -> NOT_CONTAINS will be supported, Already handled 
> #123 SearchProcessor
> 3. Graph query -> NOT_CONTAINS will not be supported
> 
> As in index and graph query wiil not support not_Contains operator, We need 
> to apply filter(inMemoryPredicate) after either index/graph query.
> 
> To support above, I have modified ClassificationSearchProcessor
> + For both cases index as well as graph, added typeNamePredicate and 
> attributePredicate
> + Added these predicate after query
> - Removed gremlinQuery block
> 
> **Operator Value:**
> It can be either "not_contains" or "NOT_CONTAINS"
> 
> **Note:**
> As part of this jira, not_contains is also added in quick search.
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java
>  6c06a3cbe 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  9c72cd4a2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> c9a605355 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 8b98b3990 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java
>  9b16e919d 
>   
> repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
>  b7ce97845 
> 
> 
> Diff: https://reviews.apache.org/r/72567/diff/4/
> 
> 
> Testing
> ---
> 
> Added testcases
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1999
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221368
---




graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Lines 50 (patched)


lockEnabled => CONSISTENCY_LOCK_ENABLED



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
Line 271 (original), 267 (patched)


call to indexBuilder.unique() shouldn't require lockEnabled=true. Please 
retain current if condition:
  "(lockEnabled && isUnique)" => "(isUnique)"



intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 75 (patched)


STORAGE_LOCK_ENABLED("atlas.graph.storage.lock.enabled") => 
STORAGE_CONSISTENCY_LOCK_ENABLED("atlas.graph.storage.consistency-lock.enabled")


- Madhan Neethiraj


On July 24, 2020, 8:23 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 24, 2020, 8:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/2/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

[jira] [Commented] (ATLAS-3902) Import Service: Importing Data With Differing GUIDs for Same Unique Attributes Causes Errors in Certain Cases

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164500#comment-17164500
 ] 

ASF subversion and git services commented on ATLAS-3902:


Commit 500ea95c3e2fbaa231026959304fa6f7ade055be in atlas's branch 
refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=500ea95 ]

ATLAS-3902: Import Service: UpdateVertexGuid Now Makes Updates to 
AtlasEntityWithExtInfo


> Import Service: Importing Data With Differing GUIDs for Same Unique 
> Attributes Causes Errors in Certain Cases
> -
>
> Key: ATLAS-3902
> URL: https://issues.apache.org/jira/browse/ATLAS-3902
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0, trunk, 2.1.0
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk, 2.1.0
>
>
> *Background*
> Consider the scenario where 2 clusters containing Atlas are setup to be 
> synchronized Atlas' export and import APIs. If the source Atlas has changes 
> where table is dropped and re-created with same name. The table's entity 
> within Atlas  will get a new GUID but will continue to have the same 
> _qualifiedName_.
> This case is handled within the Import API.
> However, the case that is not handled is to perform similar update on to the 
> table's storage descriptor.
> *Steps to Duplicate*
>  # Create a schema within Hive containing database, tables, columns and 
> views. Atlas will reflect the changes. Perform export. Generate _s1.zip_.
>  # Drop schema.
>  # Re-create the same schema within Hive. Perform export. Generate _s2.zip_.
>  # Clear Atlas database.
>  # Import _s1.zip_. Observe _application.log_.
>  # Import s2.zip. Observe _application.log_. During import log will generate 
> messages like '_GUID Updated: Entity..._'
> _Expected result:_ Import should succeed with messages indicating changes 
> entity's GUID.
> _Actual result_: Import fails with errors indicating schema violation 
> (_AtlasSchemaViolation_)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Ashutosh Mestry via Review Board]

> On July 23, 2020, 9:28 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
> > Line 39 (original), 39 (patched)
> > 
> >
> > We will need to add another *JavaPatch* to update existing data.
> 
> Damian Warszawski wrote:
> Could you elaborate on it? Does the locking itself change the structure 
> of the index?
> We usually do the full-import when index is changed.

I need to experiment with this little more. Not sure what the final 
implementation will be, but the JavaPatch will perhaps run a reindex on the 
entire dataset. This may take a while depending on size of the data.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221338
---


On July 24, 2020, 8:23 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 24, 2020, 8:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/2/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Ashutosh Mestry via Review Board]

> On July 23, 2020, 4:52 a.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
> > Lines 44 (patched)
> > 
> >
> > I suggest use AtlasConfiguration for this. Also, I think this should be 
> > true by default.
> 
> Damian Warszawski wrote:
> Great simplification. Thanks Ashutosh.
> 
> Damian Warszawski wrote:
> Not sure if that should be true by default as long as not having a proper 
> benchmarks how the performance is degradated.

I ran few tests that I have for making sure that performance does not get 
impacted. My test results did not show that to be a problem.

I test involved creating large number of entities via REST calls. During that 
time, I had the metrics enabled. I did not see any deterioration.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221323
---


On July 24, 2020, 8:23 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 24, 2020, 8:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/2/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient

[Sidharth Mishra]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72698/#review221369
---


Ship it!




Ship It!

- Sidharth Mishra


On July 23, 2020, 6:10 p.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
> 
> (Updated July 23, 2020, 6:10 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Using this project users can get an idea as how to integrate with Atlas using 
> AtlasCleint. This helps the user to understand the basic rest functionality 
> of Atlas such as
> 
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
> 
> 
> Diffs
> -
> 
>   atlas-examples/pom.xml PRE-CREATION 
>   atlas-examples/sample-app/README.md PRE-CREATION 
>   atlas-examples/sample-app/pom.xml PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/DiscoveryExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/EntityExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/GlossaryExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/LineageExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleApp.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleAppConstants.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
>  PRE-CREATION 
>   atlas-examples/sample-app/src/main/resources/atlas-application.properties 
> PRE-CREATION 
>   pom.xml 5e0442ae5 
> 
> 
> Diff: https://reviews.apache.org/r/72698/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>

[jira] [Commented] (ATLAS-3902) Import Service: Importing Data With Differing GUIDs for Same Unique Attributes Causes Errors in Certain Cases

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164533#comment-17164533
 ] 

ASF subversion and git services commented on ATLAS-3902:


Commit addb04e31e912a70829ed59730edd28ee6d80fae in atlas's branch 
refs/heads/branch-2.0 from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=addb04e ]

ATLAS-3902: Import Service: UpdateVertexGuid Now Makes Updates to 
AtlasEntityWithExtInfo


> Import Service: Importing Data With Differing GUIDs for Same Unique 
> Attributes Causes Errors in Certain Cases
> -
>
> Key: ATLAS-3902
> URL: https://issues.apache.org/jira/browse/ATLAS-3902
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0, trunk, 2.1.0
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk, 2.1.0
>
>
> *Background*
> Consider the scenario where 2 clusters containing Atlas are setup to be 
> synchronized Atlas' export and import APIs. If the source Atlas has changes 
> where table is dropped and re-created with same name. The table's entity 
> within Atlas  will get a new GUID but will continue to have the same 
> _qualifiedName_.
> This case is handled within the Import API.
> However, the case that is not handled is to perform similar update on to the 
> table's storage descriptor.
> *Steps to Duplicate*
>  # Create a schema within Hive containing database, tables, columns and 
> views. Atlas will reflect the changes. Perform export. Generate _s1.zip_.
>  # Drop schema.
>  # Re-create the same schema within Hive. Perform export. Generate _s2.zip_.
>  # Clear Atlas database.
>  # Import _s1.zip_. Observe _application.log_.
>  # Import s2.zip. Observe _application.log_. During import log will generate 
> messages like '_GUID Updated: Entity..._'
> _Expected result:_ Import should succeed with messages indicating changes 
> entity's GUID.
> _Actual result_: Import fails with errors indicating schema violation 
> (_AtlasSchemaViolation_)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Damian Warszawski]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/
---

(Updated July 24, 2020, 8:23 a.m.)


Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
Subramanian.


Changes
---

move locking conf to AtlasConfiguration


Repository: atlas


Description
---

Optional configuration to support locks on JanusGraph to ensure data consitency.

JanusGraph is eventually consistent by default which is efficient but results 
in duplicates when race condition occurs.


Reference to jira 
https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398


Diffs (updated)
-

  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
 6ef9cb76c 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 


Diff: https://reviews.apache.org/r/72695/diff/2/

Changes: https://reviews.apache.org/r/72695/diff/1-2/


Testing
---

Not possible to reproduce the error on local machine. Enable locking on our dev 
env and have not introduce any regression.


Thanks,

Damian Warszawski

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Damian Warszawski]

> On July 21, 2020, 9:06 p.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
> > Lines 260 (patched)
> > 
> >
> > Consistency lock might be relevant/needed only for unique-index i.e. 
> > isUnique=true. If this true, consider calling setConsistency(LOCK) when 
> > isUnique is true, without requiring additional argument. Same applies for 
> > #278 as well.
> > 
> > It will be useful support following configuration, to optionally 
> > disable consistentcy-lock:
> >   atlas.graph.storage.unique-key.consitency-lock.enabled
> > 
> > This configuration can be sent to AtlasJanusGraphManagement during 
> > construction - from AtlasJanusGraph.getManagementSystem().
> > 
> > Above will avoid updates to many methods for the addition of 
> > lockEnabled argument.

Great suggestion. Changed accordingly. Thanks.


- Damian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221297
---


On July 24, 2020, 8:23 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 24, 2020, 8:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/2/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Damian Warszawski]

> On July 23, 2020, 4:52 a.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
> > Lines 44 (patched)
> > 
> >
> > I suggest use AtlasConfiguration for this. Also, I think this should be 
> > true by default.

Great simplification. Thanks Ashutosh.


- Damian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221323
---


On July 24, 2020, 8:23 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 24, 2020, 8:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/2/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Damian Warszawski]

> On July 23, 2020, 9:28 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
> > Line 39 (original), 39 (patched)
> > 
> >
> > We will need to add another *JavaPatch* to update existing data.

Could you elaborate on it? Does the locking itself change the structure of the 
index?
We usually do the full-import when index is changed.


- Damian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221338
---


On July 24, 2020, 8:23 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 24, 2020, 8:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/2/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

[Damian Warszawski]

> On July 23, 2020, 4:52 a.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
> > Lines 44 (patched)
> > 
> >
> > I suggest use AtlasConfiguration for this. Also, I think this should be 
> > true by default.
> 
> Damian Warszawski wrote:
> Great simplification. Thanks Ashutosh.

Not sure if that should be true by default as long as not having a proper 
benchmarks how the performance is degradated.


- Damian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221323
---


On July 24, 2020, 8:23 a.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 24, 2020, 8:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/2/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Re: Review Request 72666: Notification: Solution to Memory Build-up

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221351
---




intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 67 (patched)


incrementCapacityBy is already an instance member, line #29 above. It is 
unnecessary to pass it as  parameter to methods at #67, #79. Please review and 
update.



intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 73 (patched)


This assumes that access is done in strict ascending order sequence, 
without any gaps. This is not a good assumption for a generic/reusable class 
implementation. This will be broken with the following access pattern:
  FixedBufferList list = new FixedBufferList<>(10, 1);
  
  list.getForUpdate(0);
  list.getForUpdate(15);

Please review my earlier comment and update to remove this assumption. 
Also, I suggest to collapse following methods into one, named ensureCapacity():
 - request()
 - ensureCapacity(): it not obvious what the return value of this method 
is, and it is critical for the caller at #74. All this noise, and confusion can 
be avoided by simply collapsing this method into previous method at #67.
 - instantiateItems(): this method is not callable from any other context, 
as it simply "adds()" to the buffer



webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
Line 168 (original), 175 (patched)


sendNotifications() can be called from the same thread multiple times - for 
example when a transaction involves create/update/delete of entities i.e. a 
graph transaction can call more than one of the following methods:
- onEntitiesAdded()
- onEntitiesUpdated()
- onEntitiesDeleted()
- onEntitiesPurged()
- onClassificationsAdded()
- onClassificationsUpdated()
- onClassificationsDeleted()

Each call would end up calling EntityNotificationSender.send() with the 
notification objects, which in turn stores the objects in a list. The second 
call in the same transaction will end up overwriting EntityNotificationV2 
instances in list buffer - which will cause the notification objects stored in 
EntityNotificationSender.send() be updated as well. This will result in 
incorrect notification to be sent. Please review this carefully and update.


- Madhan Neethiraj


On July 24, 2020, 5:18 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 24, 2020, 5:18 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
>  a677b315c 
> 
> 
> Diff: