[jira] [Comment Edited] (ATLAS-2816) Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2

2018-08-09 Thread Apoorv Naik (JIRA) 


[ 
https://issues.apache.org/jira/browse/ATLAS-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575740#comment-16575740
 ] 

Apoorv Naik edited comment on ATLAS-2816 at 8/10/18 4:12 AM:
-

One suggestion, use the followReferences flag instead of hardcoding the 
ignoreRelationship param. This would make is easier to toggle if certain 
deployment scenario wants to use the relationship details to be captured in the 
entityText. Also follow this guideline for patch creation,

 
 # Work on a local branch
 # Commit the patch on local branch
 # Generate patch using "git format-patch origin/master" (this way you get 
credit by including author info in the patch)
 # Attach the patch to JIRA

 

HTH


was (Author: apoorvnaik):
One suggestion, use the followReferences flag instead of hardcoding the 
ignoreRelationship param. This would make is easier to toggle if certain 
deployment scenario wants to use the relationship details to be captured in the 
entityText.

 

HTH

> Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2
> 
>
> Key: ATLAS-2816
> URL: https://issues.apache.org/jira/browse/ATLAS-2816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Chengbing Liu
>Assignee: Apoorv Naik
>Priority: Major
> Attachments: ATLAS-2816.01.patch
>
>
> We encountered a problem when using Hive bridge in production. One database 
> has 5000+ tables. Importing the first table costs only tens of milliseconds, 
> and then it becomes slower with more tables. In the end, it costs 1~2 seconds 
> to import one table.
> After investigation, we realized that it is not necessary for the 
> {{FullTextMapperV2}} to retrieve all the relationship of the database each 
> time a table is imported. The time complexity of importing a whole database 
> actually goes to O(n^2) (n is number of tables).
> We propose to add a parameter to the constructor of {{EntityGraphRetriever}}: 
> {{ignoreRelationship}}. When set to true, {{mapVertexToAtlasEntity}} will 
> skip the {{mapRelationshipAttributes}} call. Since {{FullTextMapperV2}} will 
> not use relationship attributes of the entity, this can save plenty of time 
> when importing entities with a large number of relations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2816) Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2

2018-08-09 Thread Apoorv Naik (JIRA) 


[ 
https://issues.apache.org/jira/browse/ATLAS-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575740#comment-16575740
 ] 

Apoorv Naik commented on ATLAS-2816:


One suggestion, use the followReferences flag instead of hardcoding the 
ignoreRelationship param. This would make is easier to toggle if certain 
deployment scenario wants to use the relationship details to be captured in the 
entityText.

 

HTH

> Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2
> 
>
> Key: ATLAS-2816
> URL: https://issues.apache.org/jira/browse/ATLAS-2816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Chengbing Liu
>Priority: Major
> Attachments: ATLAS-2816.01.patch
>
>
> We encountered a problem when using Hive bridge in production. One database 
> has 5000+ tables. Importing the first table costs only tens of milliseconds, 
> and then it becomes slower with more tables. In the end, it costs 1~2 seconds 
> to import one table.
> After investigation, we realized that it is not necessary for the 
> {{FullTextMapperV2}} to retrieve all the relationship of the database each 
> time a table is imported. The time complexity of importing a whole database 
> actually goes to O(n^2) (n is number of tables).
> We propose to add a parameter to the constructor of {{EntityGraphRetriever}}: 
> {{ignoreRelationship}}. When set to true, {{mapVertexToAtlasEntity}} will 
> skip the {{mapRelationshipAttributes}} call. Since {{FullTextMapperV2}} will 
> not use relationship attributes of the entity, this can save plenty of time 
> when importing entities with a large number of relations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (ATLAS-2816) Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2

2018-08-09 Thread Apoorv Naik (JIRA) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Naik reassigned ATLAS-2816:
--

Assignee: Apoorv Naik

> Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2
> 
>
> Key: ATLAS-2816
> URL: https://issues.apache.org/jira/browse/ATLAS-2816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Chengbing Liu
>Assignee: Apoorv Naik
>Priority: Major
> Attachments: ATLAS-2816.01.patch
>
>
> We encountered a problem when using Hive bridge in production. One database 
> has 5000+ tables. Importing the first table costs only tens of milliseconds, 
> and then it becomes slower with more tables. In the end, it costs 1~2 seconds 
> to import one table.
> After investigation, we realized that it is not necessary for the 
> {{FullTextMapperV2}} to retrieve all the relationship of the database each 
> time a table is imported. The time complexity of importing a whole database 
> actually goes to O(n^2) (n is number of tables).
> We propose to add a parameter to the constructor of {{EntityGraphRetriever}}: 
> {{ignoreRelationship}}. When set to true, {{mapVertexToAtlasEntity}} will 
> skip the {{mapRelationshipAttributes}} call. Since {{FullTextMapperV2}} will 
> not use relationship attributes of the entity, this can save plenty of time 
> when importing entities with a large number of relations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2816) Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2

2018-08-09 Thread Chengbing Liu (JIRA) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chengbing Liu updated ATLAS-2816:
-
Attachment: ATLAS-2816.01.patch

> Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2
> 
>
> Key: ATLAS-2816
> URL: https://issues.apache.org/jira/browse/ATLAS-2816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Chengbing Liu
>Priority: Major
> Attachments: ATLAS-2816.01.patch
>
>
> We encountered a problem when using Hive bridge in production. One database 
> has 5000+ tables. Importing the first table costs only tens of milliseconds, 
> and then it becomes slower with more tables. In the end, it costs 1~2 seconds 
> to import one table.
> After investigation, we realized that it is not necessary for the 
> {{FullTextMapperV2}} to retrieve all the relationship of the database each 
> time a table is imported. The time complexity of importing a whole database 
> actually goes to O(n^2) (n is number of tables).
> We propose to add a parameter to the constructor of {{EntityGraphRetriever}}: 
> {{ignoreRelationship}}. When set to true, {{mapVertexToAtlasEntity}} will 
> skip the {{mapRelationshipAttributes}} call. Since {{FullTextMapperV2}} will 
> not use relationship attributes of the entity, this can save plenty of time 
> when importing entities with a large number of relations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (ATLAS-2807) Re-evaluate classification propagation during entity delete

2018-08-09 Thread Sarath Subramanian (JIRA) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-2807:
--
Comment: was deleted

(was: Code Review: https://reviews.apache.org/r/68183/)

> Re-evaluate classification propagation during entity delete
> ---
>
> Key: ATLAS-2807
> URL: https://issues.apache.org/jira/browse/ATLAS-2807
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha, 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: 
> ATLAS-2807-Re-evaluate-classification-propagation-du-MASTER.patch, 
> Classification Propagation during entity delete1.pdf
>
>
> The current behavior - when we delete an entity *f1*, all tags associate to 
> *f1* which got propagated
> to downstream entities are removed – *PII* tag propagated to *process1* and 
> *t1* is removed
> *Proposed Change*
> _*Soft Delete:*_
> When entity *f1* is deleted, retain the propagated classification edges, so 
> the downstream entities
> – *process1* and *t1* continue to have *PII* classification associated to them
> _*Hard Delete:*_
> In cases of hard delete, the source entity (*f1*) is deleted from atlas, but 
> its classification vertex
> (*PII*) continue to exist and will continue propagating to *process1* and 
> *t1*.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2807) Re-evaluate classification propagation during entity delete

2018-08-09 Thread Sarath Subramanian (JIRA) 


[ 
https://issues.apache.org/jira/browse/ATLAS-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575689#comment-16575689
 ] 

Sarath Subramanian commented on ATLAS-2807:
---

Code Review: https://reviews.apache.org/r/68287/

> Re-evaluate classification propagation during entity delete
> ---
>
> Key: ATLAS-2807
> URL: https://issues.apache.org/jira/browse/ATLAS-2807
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha, 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: 
> ATLAS-2807-Re-evaluate-classification-propagation-du-MASTER.patch, 
> Classification Propagation during entity delete1.pdf
>
>
> The current behavior - when we delete an entity *f1*, all tags associate to 
> *f1* which got propagated
> to downstream entities are removed – *PII* tag propagated to *process1* and 
> *t1* is removed
> *Proposed Change*
> _*Soft Delete:*_
> When entity *f1* is deleted, retain the propagated classification edges, so 
> the downstream entities
> – *process1* and *t1* continue to have *PII* classification associated to them
> _*Hard Delete:*_
> In cases of hard delete, the source entity (*f1*) is deleted from atlas, but 
> its classification vertex
> (*PII*) continue to exist and will continue propagating to *process1* and 
> *t1*.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68287: ATLAS-2807: Re-evaluate classification propagation during entity delete

2018-08-09 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68287/
---

Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and 
Nixon Rodrigues.


Bugs: ATLAS-2807
https://issues.apache.org/jira/browse/ATLAS-2807


Repository: atlas


Description
---

The current behavior - when we delete an entity f1, all tags associate to f1 
which got propagated


to downstream entities are removed – PII tag propagated to process1 and t1 is 
removed


Proposed Change


Soft Delete:


When entity f1 is deleted, retain the propagated classification edges, so the 
downstream entities


– process1 and t1 continue to have PII classification associated to them


Hard Delete:


In cases of hard delete, the source entity (f1) is deleted from atlas, but its 
classification vertex


(PII) continue to exist and will continue propagating to process1 and t1.


Diffs
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 7988d1d81 
  intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 328b76734 
  intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
3132379b9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
d328873f1 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
 c6be4061b 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 4a0924bbe 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
 a95e6898e 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
 41e65d4a3 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 28a26ab4f 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 f57ce99f7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 8f4faaff8 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 58005c458 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
 8ceeedb01 
  webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
e01cba92d 
  webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fd331faa0 
  webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java 
78bd53ce7 


Diff: https://reviews.apache.org/r/68287/diff/1/


Testing
---

Validated basic cases using REST clirnt.


PreCommit: In progress


Thanks,

Sarath Subramanian

[jira] [Updated] (ATLAS-2807) Re-evaluate classification propagation during entity delete

2018-08-09 Thread Sarath Subramanian (JIRA) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-2807:
--
Attachment: 
ATLAS-2807-Re-evaluate-classification-propagation-du-MASTER.patch

> Re-evaluate classification propagation during entity delete
> ---
>
> Key: ATLAS-2807
> URL: https://issues.apache.org/jira/browse/ATLAS-2807
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha, 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: 
> ATLAS-2807-Re-evaluate-classification-propagation-du-MASTER.patch, 
> Classification Propagation during entity delete1.pdf
>
>
> The current behavior - when we delete an entity *f1*, all tags associate to 
> *f1* which got propagated
> to downstream entities are removed – *PII* tag propagated to *process1* and 
> *t1* is removed
> *Proposed Change*
> _*Soft Delete:*_
> When entity *f1* is deleted, retain the propagated classification edges, so 
> the downstream entities
> – *process1* and *t1* continue to have *PII* classification associated to them
> _*Hard Delete:*_
> In cases of hard delete, the source entity (*f1*) is deleted from atlas, but 
> its classification vertex
> (*PII*) continue to exist and will continue propagating to *process1* and 
> *t1*.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2816) Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2

2018-08-09 Thread Chengbing Liu (JIRA) 


[ 
https://issues.apache.org/jira/browse/ATLAS-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575686#comment-16575686
 ] 

Chengbing Liu commented on ATLAS-2816:
--

[~apoorvnaik], I just found ATLAS-2815 removes 
{{mapRelationshipAttributes(entityVertex, entity)}} and then adds it back, 
looks like it's an accidental change?
 I will provide a patch based on the latest code today.

> Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2
> 
>
> Key: ATLAS-2816
> URL: https://issues.apache.org/jira/browse/ATLAS-2816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Chengbing Liu
>Priority: Major
>
> We encountered a problem when using Hive bridge in production. One database 
> has 5000+ tables. Importing the first table costs only tens of milliseconds, 
> and then it becomes slower with more tables. In the end, it costs 1~2 seconds 
> to import one table.
> After investigation, we realized that it is not necessary for the 
> {{FullTextMapperV2}} to retrieve all the relationship of the database each 
> time a table is imported. The time complexity of importing a whole database 
> actually goes to O(n^2) (n is number of tables).
> We propose to add a parameter to the constructor of {{EntityGraphRetriever}}: 
> {{ignoreRelationship}}. When set to true, {{mapVertexToAtlasEntity}} will 
> skip the {{mapRelationshipAttributes}} call. Since {{FullTextMapperV2}} will 
> not use relationship attributes of the entity, this can save plenty of time 
> when importing entities with a large number of relations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : Atlas-master-UnitTests #2137

2018-08-09 Thread Apache Jenkins Server 
See

[jira] [Updated] (ATLAS-2807) Re-evaluate classification propagation during entity delete

2018-08-09 Thread Sarath Subramanian (JIRA) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-2807:
--
Attachment: (was: 
0001-ATLAS-2807-Re-evaluate-classification-propagation-du.patch)

> Re-evaluate classification propagation during entity delete
> ---
>
> Key: ATLAS-2807
> URL: https://issues.apache.org/jira/browse/ATLAS-2807
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha, 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: Classification Propagation during entity delete1.pdf
>
>
> The current behavior - when we delete an entity *f1*, all tags associate to 
> *f1* which got propagated
> to downstream entities are removed – *PII* tag propagated to *process1* and 
> *t1* is removed
> *Proposed Change*
> _*Soft Delete:*_
> When entity *f1* is deleted, retain the propagated classification edges, so 
> the downstream entities
> – *process1* and *t1* continue to have *PII* classification associated to them
> _*Hard Delete:*_
> In cases of hard delete, the source entity (*f1*) is deleted from atlas, but 
> its classification vertex
> (*PII*) continue to exist and will continue propagating to *process1* and 
> *t1*.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: Atlas-master-UnitTests #2136

2018-08-09 Thread Apache Jenkins Server 
See 

--
[...truncated 204.50 KB...]
[INFO] Exclude: .svn/**
[INFO] Exclude: .git/**
[INFO] Exclude: .gitignore
[INFO] Exclude: **/.idea/**
[INFO] Exclude: **/*.twiki
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/*.json
[INFO] Exclude: **/*.log
[INFO] Exclude: **/target/**
[INFO] Exclude: **/target*/**
[INFO] Exclude: **/build/**
[INFO] Exclude: **/*.patch
[INFO] Exclude: derby.log
[INFO] Exclude: **/logs/**
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.project
[INFO] Exclude: **/.settings/**
[INFO] Exclude: **/test-output/**
[INFO] Exclude: **/mock/**
[INFO] Exclude: **/data/**
[INFO] Exclude: **/maven-eclipse.xml
[INFO] Exclude: **/.externalToolBuilders/**
[INFO] Exclude: **/build.log
[INFO] Exclude: **/.bowerrc
[INFO] Exclude: *.json
[INFO] Exclude: **/overlays/**
[INFO] Exclude: dev-support/**
[INFO] Exclude: **/users-credentials.properties
[INFO] Exclude: **/public/css/animate.min.css
[INFO] Exclude: **/public/css/bootstrap-sidebar.css
[INFO] Exclude: **/public/js/external_lib/**
[INFO] Exclude: **/node_modules/**
[INFO] Exclude: **/public/js/libs/**
[INFO] Exclude: **/atlas.data/**
[INFO] Exclude: **/${sys:atlas.data}/**
[INFO] Exclude: **/policy-store.txt
[INFO] Exclude: **/*rebel*.xml
[INFO] Exclude: **/*rebel*.xml.bak
[INFO] Exclude: **/test/resources/**
[INFO] 125 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 
0, approved: 125 licenses.
[INFO] 
[INFO] --- maven-resources-plugin:2.7:copy-resources (copy-resources) @ 
atlas-webapp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 15 resources
[INFO] 
[INFO] --- maven-resources-plugin:2.7:copy-resources (copy-solr-resources) @ 
atlas-webapp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 9 resources
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ atlas-webapp 
---
[INFO] 
[INFO] --- keytool-maven-plugin:1.5:clean (clean) @ atlas-webapp ---
[WARNING] Keystore file 
'
 doesn't exist.
[INFO] 
[INFO] --- keytool-maven-plugin:1.5:generateKeyPair (generateKeyPair) @ 
atlas-webapp ---
[WARNING] 
[WARNING] Warning:
[WARNING] The JKS keystore uses a proprietary format. It is recommended to 
migrate to PKCS12 which is an industry standard format using "keytool 
-importkeystore -srckeystore 

 -destkeystore 

 -deststoretype pkcs12".
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
atlas-webapp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 2 resources to META-INF
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ atlas-webapp 
---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 69 source files to 

[INFO] 
:
 Some input files use or override a deprecated API.
[INFO] 
:
 Recompile with -Xlint:deprecation for details.
[INFO] 
:
 Some input files use unchecked or unsafe operations.
[INFO] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
atlas-webapp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 9 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ 
atlas-webapp ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 48 source files to 

[INFO] 
:
 Some input files use or override a deprecated API.
[INFO] 
:
 Recompile with -Xlint:deprecation for details.
[INFO]

Re: Review Request 68270: Cluster Entity Updated after Successful Import if Replication Options are Available in Import Request

2018-08-09 Thread Ashutosh Mestry 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68270/
---

(Updated Aug. 9, 2018, 8:19 p.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Sarath Subramanian.


Changes
---

Updates include:
- Fixed the case where _lastModifiedTimestamp_ was not getting updated 
correctly.


Bugs: ATLAS-2814
https://issues.apache.org/jira/browse/ATLAS-2814


Repository: atlas


Description
---

**Approach**
- New model _ReplicationDetails_ store replication timestamp.
- _AuditWriter_ updates appropriate _AtlasCluster_ entity with 
_ReplicationDetails_.


Diffs (updated)
-

  client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java 
f73ba2c6e6c9d05e86eb6c17f923a6dda1df5dd9 
  intg/src/main/java/org/apache/atlas/model/clusterinfo/AtlasCluster.java 
3ce50e38b3af45521c2c83d0e6c05398747bcf86 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 
85a606c7d3b2158238b2a6defbb9185883434d78 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java 
bfb7637711e08e17fd5f5b16bf121ac541b631a4 
  intg/src/main/java/org/apache/atlas/model/impexp/ReplicationDetails.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java 
6a3fbecda84dc48e0df1359c1b5a248958db0a4c 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ClusterService.java 
fd8e2bfe6bbdb7ae5e66ab67ecdf94e9527d9b09 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ExportImportAuditService.java
 e90b6b942d359f78ccf59ee955c4667f2033e6c4 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 
b15f828e470eaee972a4ee3f8447f404f813c699 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
98ef389bd1bf8f87d23a06dc231d4d93f06b2231 
  repository/src/main/java/org/apache/atlas/repository/ogm/AtlasClusterDTO.java 
424fb88a5ad2d84ac2245848d061892d40a3f50c 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
 9e7a119a3abfb50401f9747dad1ffa55facc51ed 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ClusterServiceTest.java
 cfd272fb00a970ab0ba28f96b237de6bd5c694e1 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
 f3803e5ae15a772eb4ff04192a83a5b412631791 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
 fcf90d364950d3afa2c18689d8e22616cfcd3c19 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
 86ab22294918701270db3ba849f5b0532c1a21a4 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 dc25e92d7115db31cbd7f99bd1a2cb5f75e766a5 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
 881368ce2c6803c8cf887a1fc91737c045c5e663 
  server-api/src/main/java/org/apache/atlas/RequestContextV1.java 
8506d186978f579c6b18c4347fc7cb2636848f16 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
d4e72620b15f69d258e5a2a96521c65a7ef71114 
  
webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
 fc804d2e5a1debd7e728dca1773f91402d41fc26 
  webapp/src/test/resources/json/export-incremental.json 
97108411f9e0c177ac13eb0a8ec3bd4475499f89 
  webapp/src/test/resources/stocks-base.zip 
40c7f37eefb46a944921f6a74a916191704cb9a3 


Diff: https://reviews.apache.org/r/68270/diff/3/

Changes: https://reviews.apache.org/r/68270/diff/2-3/


Testing
---

**Unit tests**
New tests added.


Thanks,

Ashutosh Mestry

[jira] [Commented] (ATLAS-2816) Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2

2018-08-09 Thread Apoorv Naik (JIRA) 


[ 
https://issues.apache.org/jira/browse/ATLAS-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575327#comment-16575327
 ] 

Apoorv Naik commented on ATLAS-2816:


Please submit a patch or create a review on reviewboard with suggested changes.

> Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2
> 
>
> Key: ATLAS-2816
> URL: https://issues.apache.org/jira/browse/ATLAS-2816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Chengbing Liu
>Priority: Major
>
> We encountered a problem when using Hive bridge in production. One database 
> has 5000+ tables. Importing the first table costs only tens of milliseconds, 
> and then it becomes slower with more tables. In the end, it costs 1~2 seconds 
> to import one table.
> After investigation, we realized that it is not necessary for the 
> {{FullTextMapperV2}} to retrieve all the relationship of the database each 
> time a table is imported. The time complexity of importing a whole database 
> actually goes to O(n^2) (n is number of tables).
> We propose to add a parameter to the constructor of {{EntityGraphRetriever}}: 
> {{ignoreRelationship}}. When set to true, {{mapVertexToAtlasEntity}} will 
> skip the {{mapRelationshipAttributes}} call. Since {{FullTextMapperV2}} will 
> not use relationship attributes of the entity, this can save plenty of time 
> when importing entities with a large number of relations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68188: ATLAS-2808: Enable Atlas to support entity proxies

2018-08-09 Thread David Radley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68188/#review207015
---




common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 96 (patched)


I suggest a comment to document what the flag is. In the style of the 
homeId comment.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 174 (patched)


I am wondering what your thoughts are on validation in Atlas. Specifically 
for cases like:
- the entity is a proxy but has no homeid. 
- the entity is a proxy but has none manditory attributes set which should 
not be returned
- the entity is a proxy so should not be in search results

As coded the caller is responsible for this sort of validation.


- David Radley


On Aug. 3, 2018, 1:33 p.m., Graham Wallis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68188/
> ---
> 
> (Updated Aug. 3, 2018, 1:33 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS_2808: Support for entity proxies
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 
> 7988d1d81595eb3d65ecca2480c0e67d17f53573 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> dc2be412b9ec1b07d7c7befc722810273464cd3f 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> d328873f1f2695954d79e476a40166f48ffda482 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f57ce99f764a5c779f1cfa4325849c15ac804b2d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  448d03bc069869db85cd86e14021a3fa59b57e33 
> 
> 
> Diff: https://reviews.apache.org/r/68188/diff/1/
> 
> 
> Testing
> ---
> 
> Tested with saveRelationshipReferenceCopy between entities that do not exist 
> in the Atlas repository.
> 
> 
> Thanks,
> 
> Graham Wallis
> 
>