date:20210112

Re: Review Request 73076: Deferred Actions Implementation

2021-01-12 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73076/#review222458
---




repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
Lines 371 (patched)


line 371-377 can be removed? since this is replaced by pause()



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 753 (patched)


API duplicated? line 746/753


- Sarath Subramanian


On Jan. 11, 2021, 10:34 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73076/
> ---
> 
> (Updated Jan. 11, 2021, 10:34 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3919
> https://issues.apache.org/jira/browse/ATLAS-3919
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Model: _TaskDef_
> - Definition of task.
> - Contains status of the task for querying.
> 
> _AbstractTask_
> - Takes TaskDef as input and calls run.
> - The concrete tasks has logic for doing real work.
> 
> _TaskFactory_
> 
> - Creates tasks given task type.
> - Enumerates all the tasks that are supported.
> 
> _TaskManagement_
> 
> - Component/Service. Constructed after all existing components are 
> constructed.
> - Provides a simple interface for adding tasks.
> - Processes tasks that are in pending state.
> - Supports HA mode, where the tasks are not executed on passive instance.
> 
> _TaskManagement.Registry_
> 
> - CRUD operations with TaskDef vertex.
> 
> _TaskManagement.Processor_
> - Provides mechanism for provides single queue for processing tasks in a way 
> that is non-blocking.
> - Uses task type and the associated TaskFactory to construct concrete task.
> 
> Changes to _EntityGraphMapper_
> There are 3 new methods added that use portion of existing classification 
> propagation logic and create a separate method.
> 
> New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. 
> All calls that operate without tasks get channeled to the existing 
> implementation. The ones involving tasks are modified to use the new task 
> framework.
> 
> **Concurrent Changes to Classifications**
> Via REST APIs and operations from web UI, it is possible to make changes to 
> the same entity. This situation will be more pronounced in the new setup.
> 
> The existing framework has 
> _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides 
> ReentrantLock on an entity that is identified by the supplied GUID. This will 
> cause a lock on the entity and all the impacted vertices. This has potential 
> for adversely impacting scalability.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 
>   intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 
>   intg/src/test/resources/atlas-application.properties 7e74d5107 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 86b369fc9 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  244072289 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  2cfcc0bc1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapperWithTasks.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/util/BeanUtilRepo.java 
> PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  84e9bfa04 
>   repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java 
> PRE-CREATION 
>

[jira] [Commented] (ATLAS-4092) Remove unused attributes and add right attributes to the atlas admin audit api

2021-01-12 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/ATLAS-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263919#comment-17263919
 ] 

ASF subversion and git services commented on ATLAS-4092:


Commit 9a407456019b54957ac5e3269a104bebcf18c0c0 in atlas's branch 
refs/heads/branch-2.0 from Mandar Ambawane
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=9a40745 ]

ATLAS-4092 Remove unused attributes and add right attributes to the atlas admin 
audit api

Signed-off-by: Nikhil Bonte 
(cherry picked from commit 8045c7841216974cafa8b04aa14fbd5aa893ba55)


> Remove unused attributes and add right attributes to the atlas admin audit api
> --
>
> Key: ATLAS-4092
> URL: https://issues.apache.org/jira/browse/ATLAS-4092
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Attachments: ATLAS-4092.patch
>
>
> {code:java|title=Sample payload:}
> {
>   "currentPage": "page",
>   "pageSize": "per_page",
>   "totalPages": "total_pages",
>   "totalRecords": "total_entries",
>   "sortKey": "sort_by",
>   "order": "order",
>   "directions": {
> "1": "desc",
> "-1": "asc"
>   },
>   "auditFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "operation",
> "operator": "eq",
> "attributeValue": "TYPE_DEF_CREATE"
>   }
> ]
>   },
>   "limit": 25,
>   "offset": 0
> }
> {code}
> Here the following attributes do not work:
> {code}
> "sortKey": "sort_by",
> "order": "order",
> {code}
> Instead the following works
> {code}
> "sortBy" : "startTime",
> "sortOrder" : "DESCENDING",
> {code}
> sortKey and order has to be removed and sortBy and sortOrder has to be added



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4092) Remove unused attributes and add right attributes to the atlas admin audit api

2021-01-12 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/ATLAS-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263914#comment-17263914
 ] 

ASF subversion and git services commented on ATLAS-4092:


Commit 8045c7841216974cafa8b04aa14fbd5aa893ba55 in atlas's branch 
refs/heads/master from Mandar Ambawane
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=8045c78 ]

ATLAS-4092 Remove unused attributes and add right attributes to the atlas admin 
audit api

Signed-off-by: Nikhil Bonte 


> Remove unused attributes and add right attributes to the atlas admin audit api
> --
>
> Key: ATLAS-4092
> URL: https://issues.apache.org/jira/browse/ATLAS-4092
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Attachments: ATLAS-4092.patch
>
>
> {code:java|title=Sample payload:}
> {
>   "currentPage": "page",
>   "pageSize": "per_page",
>   "totalPages": "total_pages",
>   "totalRecords": "total_entries",
>   "sortKey": "sort_by",
>   "order": "order",
>   "directions": {
> "1": "desc",
> "-1": "asc"
>   },
>   "auditFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "operation",
> "operator": "eq",
> "attributeValue": "TYPE_DEF_CREATE"
>   }
> ]
>   },
>   "limit": 25,
>   "offset": 0
> }
> {code}
> Here the following attributes do not work:
> {code}
> "sortKey": "sort_by",
> "order": "order",
> {code}
> Instead the following works
> {code}
> "sortBy" : "startTime",
> "sortOrder" : "DESCENDING",
> {code}
> sortKey and order has to be removed and sortBy and sortOrder has to be added



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-12 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222457
---




graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/GraphQueryTest.java
Line 50 (original), 50 (patched)


revert this?



repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 139 (patched)


refactor line 139-142 to a method getDslQueryExecutor()



repository/src/main/java/org/apache/atlas/query/GremlinQuery.java
Lines 20 (patched)


nit: review/remove unused imports



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 49 (patched)


SelectClauseInfo => selectClauseInfo



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 116 (patched)


'unique' argument is never used. should the result return only unique 
entries if this is set?



repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 68 (patched)


'end' is never used; consider removing.



repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 80 (patched)


review and remove TODO?



repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 103 (patched)


can there be a case where query contains 'offset' and no 'limit' set?


- Sarath Subramanian


On Jan. 12, 2021, 3:49 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 12, 2021, 3:49 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/GraphQueryTest.java
>  4b0176311 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

2021-01-12 Thread Pinal Shah



> On Jan. 12, 2021, 8:20 p.m., Sarath Subramanian wrote:
> > docs/src/documents/Search/SearchAdvanced.md
> > Lines 228 (patched)
> > 
> >
> > can we add a new section and examples for relationship search?

okay, i will add new section specifically for relationship


- Pinal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/#review222445
---


On Jan. 12, 2021, 11:54 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73058/
> ---
> 
> (Updated Jan. 12, 2021, 11:54 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4057
> https://issues.apache.org/jira/browse/ATLAS-4057
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Supporting glossary term in advanced search, will allow user to search the 
> entities which are assigned to particular glossary term.
> 
>  
> **WorkAround:**
> 
> 'hasTerm' keyword is added to support glossary term.
> 
> User needs to add fully qualifiedName of glossary term.
> 
>  
> 
> **Example queries to search with term:**
> 
> hive_table hasTerm "modernTrade@salesGlossary",
> 
> hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension
> 
> **Example queries to search with relationship:**
> 
> hive_table where db.name = "sales"
> 
> 
> Diffs
> -
> 
>   docs/src/documents/Search/SearchAdvanced.md 46be14278 
>   
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
> 6c48a82f4 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
> 15d1bb4cb 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 142b9cac9 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
> f1c10600c 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 6b33edb53 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
>  3139d4374 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
>  a1a727c82 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  1839e76c7 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 487cf2778 
> 
> 
> Diff: https://reviews.apache.org/r/73058/diff/8/
> 
> 
> Testing
> ---
> 
> Added testcases in DSLQueriesTest.java
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

2021-01-12 Thread Sidharth Kumar Mishra (Jira)



 [ 
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sidharth Kumar Mishra updated ATLAS-4099:
-
Attachment: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch

> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory 
> created by Hook
> 
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location 
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName : 
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because 
> of this , duplicate entities are created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

2021-01-12 Thread Sidharth Kumar Mishra (Jira)



 [ 
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sidharth Kumar Mishra updated ATLAS-4099:
-
Attachment: (was: 
ATLAS-4099-Removed-slash-from-adls_gen2_directory.patch)

> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory 
> created by Hook
> 
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location 
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName : 
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because 
> of this , duplicate entities are created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory

2021-01-12 Thread Sidharth Mishra


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73140/
---

(Updated Jan. 13, 2021, 12:44 a.m.)


Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, 
Radhika Kundam, and Sarath Subramanian.


Bugs: ATLAS-4099
https://issues.apache.org/jira/browse/ATLAS-4099


Repository: atlas


Description
---

ATLAS-4099: Removed slash from adls_gen2_directory


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
0e4b507ff 
  common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
2d2ff2c47 


Diff: https://reviews.apache.org/r/73140/diff/2/

Changes: https://reviews.apache.org/r/73140/diff/1-2/


Testing
---

Already tested manually and precommit will be run


Thanks,

Sidharth Mishra

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-12 Thread Ashutosh Mestry via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/
---

(Updated Jan. 12, 2021, 11:49 p.m.)


Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sarath Subramanian.


Changes
---

Updates include: 
- Added unit tests for _NOT_CLASSIFIED, _CLASSIFIED keywords.


Bugs: ATLAS-2932
https://issues.apache.org/jira/browse/ATLAS-2932


Repository: atlas


Description (updated)
---

**Approach**
General Approach:
- Ensure that Gremlin-based implementation continues to function. This will 
ensure backward compatibility and will guarad against regressions or behavior 
deviations, if any, are introduced with the new implementation.
- Continue using existing _GremlinQueryComposer_. 
- Translate the clauses to _AtlasGraphTraversal_.
- Projects that were earlier handled in Gremlin query are now handled in a 
separate Java class.

Details:
- Modified: _GremlinQueryComposer_: Continue composing queries using existing 
claues. Additions alone.
- New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
- Modifed: _EntityDiscoveryService_ uses the new exectors.
- New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
_AtlasGraphTraversal_.
- Modified: _AtlasGraphTraversal_: Additional methods.
- New: _SelectClauseProjections_: All queries with select operations.

**Configuration**
_atlas.dsl.executor.traversal_ 
- true: Uses the traversal-based implementation for query execution.
- false: Uses the script-engine based implementation for query execution.

**Credits**
- Apoorv Naik (apoorvnaik): Original implementation.
- Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
short-comings.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
c016f6340 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
 881bb1e0f 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 0dd573b89 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
 c33c4f4d0 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/GraphQueryTest.java
 4b0176311 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
01a6c303a 
  repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 9704b0f9e 
  repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
9f30e4dc0 
  repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
969fcd2b1 
  
repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 


Diff: https://reviews.apache.org/r/73128/diff/5/

Changes: https://reviews.apache.org/r/73128/diff/4-5/


Testing
---

**Unit Tests**
- Additional tests.
- _GremlinCompoerTest_ continues to validate the clause creation.
- Improved validation for _minMaxCount_ tests.
- Refactored: _TableValidator_ Validates output from a query with select claues.

**Volume Tests**
- Old implementation: 528 queries take over 10 mins to execute.
- New implemeentations: 528 queries execute in 7 secs.


Thanks,

Ashutosh Mestry

Re: Review Request 73139: ATLAS-4086: Fixes the search and other major issues in python client

2021-01-12 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73139/#review222454
---




intg/src/main/python/apache_atlas/model/glossary.py
Line 19 (original), 19 (patched)


Can you please look this following error?

Traceback (most recent call last):
  File "test.py", line 3, in 
from apache_atlas.client.base_client import AtlasClient
  File 
"<...>/atlas/intg/src/main/python/apache_atlas/client/base_client.py", line 26, 
in 
from apache_atlas.client.discovery import DiscoveryClient
  File "<...>/atlas/intg/src/main/python/apache_atlas/client/discovery.py", 
line 18, in 
from apache_atlas.model.discovery import (AtlasQuickSearchResult, 
AtlasSearchResult,
  File "<...>/atlas/intg/src/main/python/apache_atlas/model/discovery.py", 
line 21, in 
from apache_atlas.model.instance import AtlasEntityHeader
  File "<...>/atlas/intg/src/main/python/apache_atlas/model/instance.py", 
line 19, in 
from apache_atlas.model.glossary import AtlasTermAssignmentHeader
  File "<...>/atlas/intg/src/main/python/apache_atlas/model/glossary.py", 
line 19, in 
import apache_atlas.model.instance as instance
AttributeError: module 'apache_atlas.model' has no attribute 'instance'


- Madhan Neethiraj


On Jan. 12, 2021, 9:54 p.m., Verdan Mahmood wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73139/
> ---
> 
> (Updated Jan. 12, 2021, 9:54 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Pinal Shah, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4086
> https://issues.apache.org/jira/browse/ATLAS-4086
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Signed-off-by: verdan 
> 
> This review request fixes a number of major issues in the python atlas client.
> 
> 
> Diffs
> -
> 
>   intg/src/main/python/apache_atlas/client/admin.py 
> ebe72dcc79b18b5955f5403889895bc7bfe4923b 
>   intg/src/main/python/apache_atlas/client/discovery.py 
> 04f5fc9cf68edbd5973d48b402d764030a7068ca 
>   intg/src/main/python/apache_atlas/client/entity.py 
> 7edacf9e72a3a5d3cc55008b39b60fb88dc5c607 
>   intg/src/main/python/apache_atlas/client/glossary.py 
> efa0612fa0c78b850b7df76cc7e89d183ffb8e41 
>   intg/src/main/python/apache_atlas/client/relationship.py 
> f5790a4c369b6f2a517b81425d8fa237ca2b7cdc 
>   intg/src/main/python/apache_atlas/client/typedef.py 
> aa1581d86df26979f3a6c6bb798ea7b0f6ab3bd0 
>   intg/src/main/python/apache_atlas/model/glossary.py 
> 7a0faac37f3c74cdd0eb49012b88c376323e4b3f 
>   intg/src/main/python/apache_atlas/model/instance.py 
> 8cfb254c49328b495454158eec26bd1cc250b929 
>   intg/src/main/python/setup.py 7c563c9c0610eeba169b1c8bec73a246c2a95d4b 
> 
> 
> Diff: https://reviews.apache.org/r/73139/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Verdan Mahmood
> 
>

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

2021-01-12 Thread Deep Singh

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73119/
---

(Updated Jan. 12, 2021, 11:35 p.m.)

Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath
Subramanian, and Sidharth Mishra.

Bugs: ATLAS-4094
https://issues.apache.org/jira/browse/ATLAS-4094

Repository: atlas

Description
---

Created a new entity audit API with Sorting functionality.

/api/atlas/v2/entity/{entity-guid}/audit/sortby/{sort-column}?offset=2=5

sort-column may have 3 values "user", "action", or "timestamp".

HBase does not support query with sorted results. To support this API inmemory
sort has to be performed.
Audit entry can potentially have entire entity dumped into it. Loading entire
audit entries for an entity can be memory intensive. Therefore we load audit
entries with limited columns first, perform sort on this light weight list,
then get the relevant section by removing offsets and reducing to limits. With
this reduced list we create MultiRowRangeFilter and then again scan the table
to get all the columns from the table this time.

Diffs
-

intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
083acac73

repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
8a453fd43

repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
07784d1c4

repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
9fca74470

repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
900df0205

repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
ef9e259ea
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 0d6d0c845

Diff: https://reviews.apache.org/r/73119/diff/3/

Testing (updated)
---

Manual testing was done.

Did testing on a setup with 1 million audit entries spread across 1000
entities.
Existing Rest API took 6-12 milliseconds for preparing the result.
In-memory sort and double scan approach took 55-75 milliseconds.
Single Full scan and in-memory approach took 250-300 milliseconds.

As it was expected, the new API is 4X slower than the existing API therefore
the existing API still should be the primary API for querying audit events. And
the new API should be used only if sorting is required. Overall the server
response time of the new API is less than 80 millisecond, compared to < 25
milliseconds for the existing API.

Thanks,

Deep Singh

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

2021-01-12 Thread Deep Singh

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73119/
---

(Updated Jan. 12, 2021, 11:32 p.m.)

Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath
Subramanian, and Sidharth Mishra.

Bugs: ATLAS-4094
https://issues.apache.org/jira/browse/ATLAS-4094

Repository: atlas

Description
---

Created a new entity audit API with Sorting functionality.

/api/atlas/v2/entity/{entity-guid}/audit/sortby/{sort-column}?offset=2=5

sort-column may have 3 values "user", "action", or "timestamp".

Diffs (updated)
-

intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
083acac73

repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
8a453fd43

repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
07784d1c4

repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
9fca74470

repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
900df0205

repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
ef9e259ea
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 0d6d0c845

Diff: https://reviews.apache.org/r/73119/diff/3/

Changes: https://reviews.apache.org/r/73119/diff/2-3/

Testing (updated)
---

Manual testing was done.

As it was expected, the new API is 10X slower than the existing API therefore
the existing API still should be the primary API for querying audit events. And
the new API should be used only if sorting is required. Overall the server
response time of the new API is less than 80 millisecond, compared to < 25
milliseconds for the existing API.

Thanks,

Deep Singh

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

2021-01-12 Thread Deep Singh



> On Jan. 7, 2021, 6:17 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
> > Lines 302 (patched)
> > 
> >
> > Does use of second scan result in significant performance improvements 
> > i.e. avoiding population of event.details and event.entity during the first 
> > scan? If not, it will be simper to move #320 - #326 to #293 and avoid this 
> > second scan.

Yes, there is significant performance improvement with the two scan approach. 
The first scan is a full scan and it is done with limited columns. The second 
scan is a limited row scan with all the columns. If we want to avoid the second 
scan then the first scan has to be a full scan with all the columns which would 
be very expensive. On a setup with 1 million audit entries, spread across a 
thousand entities, the two scan approach was 5X faster than the single scan 
approach.


> On Jan. 7, 2021, 6:17 a.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
> > Lines 802 (patched)
> > 
> >
> > Why not add 'sortBy' query-param to existing REST API method, instead 
> > of adding a new method? Also, consider adding 'sortOrder' query-param - 
> > with default value as 'asc'.

Added the 'sortorder' as a query-param in the new API. 
There are fundamental differences between the two APIs
 - Offset in the existing rest API is achieved by startkey parameter where 
hbase row key is expected. New API takes offset in the form of an integer.
 - Performance and mechanism are different. New API is done with in-memory sort 
and therefore is 10X slower than the old one. If we mix the APIs it would be 
very difficult and confusing to present the benchmarking.
 - Old API support filter by column, but the new one does not.

Also, to differentiate the behavior based on parameters would be very confusing 
from docs perspective.


- Deep


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73119/#review222416
---


On Jan. 6, 2021, 7:06 p.m., Deep Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73119/
> ---
> 
> (Updated Jan. 6, 2021, 7:06 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4094
> https://issues.apache.org/jira/browse/ATLAS-4094
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Created a new entity audit API with Sorting functionality.
> 
> /api/atlas/v2/entity/{entity-guid}/audit/sortby/{sort-column}?offset=2=5
> 
> sort-column may have 3 values "user", "action", or "timestamp".
> 
> HBase does not support query with sorted results. To support this API 
> inmemory sort has to be performed.
> Audit entry can potentially have entire entity dumped into it. Loading entire 
> audit entries for an entity can be memory intensive. Therefore we load audit 
> entries with limited columns first, perform sort on this light weight list, 
> then get the relevant section by removing offsets and reducing to limits. 
> With this reduced list we create MultiRowRangeFilter and then again scan the 
> table to get all the columns from the table this time.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  8a453fd43 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
>  07784d1c4 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  9fca74470 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
>  900df0205 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
>  ef9e259ea 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 0d6d0c845 
> 
> 
> Diff: https://reviews.apache.org/r/73119/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing was done.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-12 Thread Ashutosh Mestry via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/
---

(Updated Jan. 12, 2021, 10:41 p.m.)


Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sarath Subramanian.


Changes
---

Updates include: Additional unit tests for isIncomplete clause.


Bugs: ATLAS-2932
https://issues.apache.org/jira/browse/ATLAS-2932


Repository: atlas


Description
---

**Approach**
General Approach:
- Ensure that Gremlin-based implementation continues to function. This will 
ensure backward compatibility and will guarad against regressions or behavior 
deviations, if any, are introduced with the new implementation.
- Continue using existing _GremlinQueryComposer_. 
- Translate the clauses to _AtlasGraphTraversal_.
- Projects that were earlier handled in Gremlin query are now handled in a 
separate Java class.

Details:
- Modified: _GremlinQueryComposer_: Continue composing queries using existing 
claues. Additions alone.
- New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
- Modifed: _EntityDiscoveryService_ uses the new exectors.
- New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
_AtlasGraphTraversal_.
- Modified: _AtlasGraphTraversal_: Additional methods.
- New: _SelectClauseProjections_: All queries with select operations.

**Credits**
- Apoorv Naik (apoorvnaik): Original implementation.
- Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
short-comings.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
c016f6340 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
 881bb1e0f 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 0dd573b89 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
 c33c4f4d0 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/GraphQueryTest.java
 4b0176311 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
01a6c303a 
  repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 9704b0f9e 
  repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
9f30e4dc0 
  repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
969fcd2b1 
  
repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 


Diff: https://reviews.apache.org/r/73128/diff/4/

Changes: https://reviews.apache.org/r/73128/diff/3-4/


Testing
---

**Unit Tests**
- Additional tests.
- _GremlinCompoerTest_ continues to validate the clause creation.
- Improved validation for _minMaxCount_ tests.
- Refactored: _TableValidator_ Validates output from a query with select claues.

**Volume Tests**
- Old implementation: 528 queries take over 10 mins to execute.
- New implemeentations: 528 queries execute in 7 secs.


Thanks,

Ashutosh Mestry

Re: Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory

2021-01-12 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73140/#review222450
---


Ship it!




Ship It!

- Sarath Subramanian


On Jan. 12, 2021, 1:47 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73140/
> ---
> 
> (Updated Jan. 12, 2021, 1:47 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, 
> Radhika Kundam, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4099
> https://issues.apache.org/jira/browse/ATLAS-4099
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4099: Removed slash from adls_gen2_directory
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
> 0e4b507ff 
> 
> 
> Diff: https://reviews.apache.org/r/73140/diff/1/
> 
> 
> Testing
> ---
> 
> Already tested manually and precommit will be run
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73139: ATLAS-4086: Fixes the search and other major issues in python client

2021-01-12 Thread Verdan Mahmood


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73139/
---

(Updated Jan. 12, 2021, 9:54 p.m.)


Review request for atlas, Madhan Neethiraj, Pinal Shah, and Sarath Subramanian.


Bugs: ATLAS-4086
https://issues.apache.org/jira/browse/ATLAS-4086


Repository: atlas


Description
---

Signed-off-by: verdan 

This review request fixes a number of major issues in the python atlas client.


Diffs (updated)
-

  intg/src/main/python/apache_atlas/client/admin.py 
ebe72dcc79b18b5955f5403889895bc7bfe4923b 
  intg/src/main/python/apache_atlas/client/discovery.py 
04f5fc9cf68edbd5973d48b402d764030a7068ca 
  intg/src/main/python/apache_atlas/client/entity.py 
7edacf9e72a3a5d3cc55008b39b60fb88dc5c607 
  intg/src/main/python/apache_atlas/client/glossary.py 
efa0612fa0c78b850b7df76cc7e89d183ffb8e41 
  intg/src/main/python/apache_atlas/client/relationship.py 
f5790a4c369b6f2a517b81425d8fa237ca2b7cdc 
  intg/src/main/python/apache_atlas/client/typedef.py 
aa1581d86df26979f3a6c6bb798ea7b0f6ab3bd0 
  intg/src/main/python/apache_atlas/model/glossary.py 
7a0faac37f3c74cdd0eb49012b88c376323e4b3f 
  intg/src/main/python/apache_atlas/model/instance.py 
8cfb254c49328b495454158eec26bd1cc250b929 
  intg/src/main/python/setup.py 7c563c9c0610eeba169b1c8bec73a246c2a95d4b 


Diff: https://reviews.apache.org/r/73139/diff/2/

Changes: https://reviews.apache.org/r/73139/diff/1-2/


Testing
---


Thanks,

Verdan Mahmood

Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory

2021-01-12 Thread Sidharth Mishra


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73140/
---

Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, 
Radhika Kundam, and Sarath Subramanian.


Bugs: ATLAS-4099
https://issues.apache.org/jira/browse/ATLAS-4099


Repository: atlas


Description
---

ATLAS-4099: Removed slash from adls_gen2_directory


Diffs
-

  common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
0e4b507ff 


Diff: https://reviews.apache.org/r/73140/diff/1/


Testing
---

Already tested manually and precommit will be run


Thanks,

Sidharth Mishra

[jira] [Created] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

2021-01-12 Thread Sidharth Kumar Mishra (Jira)

Sidharth Kumar Mishra created ATLAS-4099:


 Summary: adls_gen2_directory, Hive Hook : An extra "/" is seen in 
adls_gen2_directory created by Hook
 Key: ATLAS-4099
 URL: https://issues.apache.org/jira/browse/ATLAS-4099
 Project: Atlas
  Issue Type: Bug
Reporter: Sidharth Kumar Mishra
Assignee: Sidharth Kumar Mishra


For the query :
{code:java}
create external table ext_table(id int) location 
'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
adls_gen2_directory created by Hook has QualifiedName : 

*abfs://container@storageaccount/test51/@cm*

whereas , adls_gen2_directory created by Extractor has QualifiedName:

*abfs://**container@storageaccount/test51**@cm*

There is an extra "/" after storage account name which causes issue. Because of 
this , duplicate entities are created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-12 Thread Ashutosh Mestry via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/
---

(Updated Jan. 12, 2021, 9:01 p.m.)


Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sarath Subramanian.


Changes
---

Updates include: 
- Rebased.
- Addressed review comments.


Bugs: ATLAS-2932
https://issues.apache.org/jira/browse/ATLAS-2932


Repository: atlas


Description
---

**Approach**
General Approach:
- Ensure that Gremlin-based implementation continues to function. This will 
ensure backward compatibility and will guarad against regressions or behavior 
deviations, if any, are introduced with the new implementation.
- Continue using existing _GremlinQueryComposer_. 
- Translate the clauses to _AtlasGraphTraversal_.
- Projects that were earlier handled in Gremlin query are now handled in a 
separate Java class.

Details:
- Modified: _GremlinQueryComposer_: Continue composing queries using existing 
claues. Additions alone.
- New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
- Modifed: _EntityDiscoveryService_ uses the new exectors.
- New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
_AtlasGraphTraversal_.
- Modified: _AtlasGraphTraversal_: Additional methods.
- New: _SelectClauseProjections_: All queries with select operations.

**Credits**
- Apoorv Naik (apoorvnaik): Original implementation.
- Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
short-comings.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
c016f6340 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
 881bb1e0f 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 0dd573b89 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
 c33c4f4d0 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/GraphQueryTest.java
 4b0176311 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
01a6c303a 
  repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 9704b0f9e 
  repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
9f30e4dc0 
  repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
969fcd2b1 
  
repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 


Diff: https://reviews.apache.org/r/73128/diff/3/

Changes: https://reviews.apache.org/r/73128/diff/2-3/


Testing
---

**Unit Tests**
- Additional tests.
- _GremlinCompoerTest_ continues to validate the clause creation.
- Improved validation for _minMaxCount_ tests.
- Refactored: _TableValidator_ Validates output from a query with select claues.

**Volume Tests**
- Old implementation: 528 queries take over 10 mins to execute.
- New implemeentations: 528 queries execute in 7 secs.


Thanks,

Ashutosh Mestry

Review Request 73139: ATLAS-4086: Fixes the search and other major issues in python client

2021-01-12 Thread Verdan Mahmood


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73139/
---

Review request for atlas, Madhan Neethiraj, Pinal Shah, and Sarath Subramanian.


Bugs: ATLAS-4086
https://issues.apache.org/jira/browse/ATLAS-4086


Repository: atlas


Description
---

Signed-off-by: verdan 

This review request fixes a number of major issues in the python atlas client.


Diffs
-

  intg/src/main/python/apache_atlas/client/admin.py 
ebe72dcc79b18b5955f5403889895bc7bfe4923b 
  intg/src/main/python/apache_atlas/client/discovery.py 
04f5fc9cf68edbd5973d48b402d764030a7068ca 
  intg/src/main/python/apache_atlas/client/entity.py 
7edacf9e72a3a5d3cc55008b39b60fb88dc5c607 
  intg/src/main/python/apache_atlas/client/glossary.py 
efa0612fa0c78b850b7df76cc7e89d183ffb8e41 
  intg/src/main/python/apache_atlas/client/relationship.py 
f5790a4c369b6f2a517b81425d8fa237ca2b7cdc 
  intg/src/main/python/apache_atlas/client/typedef.py 
aa1581d86df26979f3a6c6bb798ea7b0f6ab3bd0 
  intg/src/main/python/apache_atlas/model/glossary.py 
7a0faac37f3c74cdd0eb49012b88c376323e4b3f 
  intg/src/main/python/apache_atlas/model/instance.py 
8cfb254c49328b495454158eec26bd1cc250b929 
  intg/src/main/python/setup.py 7c563c9c0610eeba169b1c8bec73a246c2a95d4b 


Diff: https://reviews.apache.org/r/73139/diff/1/


Testing
---


Thanks,

Verdan Mahmood

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

2021-01-12 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/#review222445
---




docs/src/documents/Search/SearchAdvanced.md
Lines 228 (patched)


can we add a new section and examples for relationship search?


- Sarath Subramanian


On Jan. 12, 2021, 3:54 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73058/
> ---
> 
> (Updated Jan. 12, 2021, 3:54 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4057
> https://issues.apache.org/jira/browse/ATLAS-4057
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Supporting glossary term in advanced search, will allow user to search the 
> entities which are assigned to particular glossary term.
> 
>  
> **WorkAround:**
> 
> 'hasTerm' keyword is added to support glossary term.
> 
> User needs to add fully qualifiedName of glossary term.
> 
>  
> 
> **Example queries to search with term:**
> 
> hive_table hasTerm "modernTrade@salesGlossary",
> 
> hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension
> 
> **Example queries to search with relationship:**
> 
> hive_table where db.name = "sales"
> 
> 
> Diffs
> -
> 
>   docs/src/documents/Search/SearchAdvanced.md 46be14278 
>   
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
> 6c48a82f4 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
> 15d1bb4cb 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 142b9cac9 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
> f1c10600c 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 6b33edb53 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
>  3139d4374 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
>  a1a727c82 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  1839e76c7 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 487cf2778 
> 
> 
> Diff: https://reviews.apache.org/r/73058/diff/8/
> 
> 
> Testing
> ---
> 
> Added testcases in DSLQueriesTest.java
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

2021-01-12 Thread Ashutosh Mestry via Review Board



> On Jan. 12, 2021, 5:44 p.m., Ashutosh Mestry wrote:
> > Can you please update documentation with these new keywords?

Nevermind, i see it. Thanks!


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/#review222443
---


On Jan. 12, 2021, 11:54 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73058/
> ---
> 
> (Updated Jan. 12, 2021, 11:54 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4057
> https://issues.apache.org/jira/browse/ATLAS-4057
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Supporting glossary term in advanced search, will allow user to search the 
> entities which are assigned to particular glossary term.
> 
>  
> **WorkAround:**
> 
> 'hasTerm' keyword is added to support glossary term.
> 
> User needs to add fully qualifiedName of glossary term.
> 
>  
> 
> **Example queries to search with term:**
> 
> hive_table hasTerm "modernTrade@salesGlossary",
> 
> hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension
> 
> **Example queries to search with relationship:**
> 
> hive_table where db.name = "sales"
> 
> 
> Diffs
> -
> 
>   docs/src/documents/Search/SearchAdvanced.md 46be14278 
>   
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
> 6c48a82f4 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
> 15d1bb4cb 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 142b9cac9 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
> f1c10600c 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 6b33edb53 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
>  3139d4374 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
>  a1a727c82 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  1839e76c7 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 487cf2778 
> 
> 
> Diff: https://reviews.apache.org/r/73058/diff/8/
> 
> 
> Testing
> ---
> 
> Added testcases in DSLQueriesTest.java
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

2021-01-12 Thread Ashutosh Mestry via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/#review222443
---



Can you please update documentation with these new keywords?

- Ashutosh Mestry


On Jan. 12, 2021, 11:54 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73058/
> ---
> 
> (Updated Jan. 12, 2021, 11:54 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4057
> https://issues.apache.org/jira/browse/ATLAS-4057
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Supporting glossary term in advanced search, will allow user to search the 
> entities which are assigned to particular glossary term.
> 
>  
> **WorkAround:**
> 
> 'hasTerm' keyword is added to support glossary term.
> 
> User needs to add fully qualifiedName of glossary term.
> 
>  
> 
> **Example queries to search with term:**
> 
> hive_table hasTerm "modernTrade@salesGlossary",
> 
> hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension
> 
> **Example queries to search with relationship:**
> 
> hive_table where db.name = "sales"
> 
> 
> Diffs
> -
> 
>   docs/src/documents/Search/SearchAdvanced.md 46be14278 
>   
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
> 6c48a82f4 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
> 15d1bb4cb 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 142b9cac9 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
> f1c10600c 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 6b33edb53 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
>  3139d4374 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
>  a1a727c82 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  1839e76c7 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 487cf2778 
> 
> 
> Diff: https://reviews.apache.org/r/73058/diff/8/
> 
> 
> Testing
> ---
> 
> Added testcases in DSLQueriesTest.java
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-12 Thread Nikhil Bonte


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222441
---




repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 83 (patched)


Need to handle limit & offset here.
Assume you have 30 columns
query: "Column groupby (name)" will return all the 30 columns
Same behaviour if limit is passed in query.


- Nikhil Bonte


On Jan. 11, 2021, 12:27 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 11, 2021, 12:27 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 249381059 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries execute in 7 secs.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-12 Thread Nikhil Bonte


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222442
---




repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
Lines 589-598 (original), 613-622 (patched)


enable & validate these queries


- Nikhil Bonte


On Jan. 11, 2021, 12:27 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 11, 2021, 12:27 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 249381059 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries execute in 7 secs.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-12 Thread Nikhil Bonte


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222440
---




repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 80 (patched)


Remove TODO, this block gets executed. e.g. query :
hive_column groupby (name)


- Nikhil Bonte


On Jan. 11, 2021, 12:27 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 11, 2021, 12:27 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 249381059 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries execute in 7 secs.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

2021-01-12 Thread Pinal Shah


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/
---

(Updated Jan. 12, 2021, 11:54 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Changes
---

added doc


Bugs: ATLAS-4057
https://issues.apache.org/jira/browse/ATLAS-4057


Repository: atlas


Description
---

Supporting glossary term in advanced search, will allow user to search the 
entities which are assigned to particular glossary term.

 
**WorkAround:**

'hasTerm' keyword is added to support glossary term.

User needs to add fully qualifiedName of glossary term.

 

**Example queries to search with term:**

hive_table hasTerm "modernTrade@salesGlossary",

hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension

**Example queries to search with relationship:**

hive_table where db.name = "sales"


Diffs (updated)
-

  docs/src/documents/Search/SearchAdvanced.md 46be14278 
  repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
6c48a82f4 
  repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
15d1bb4cb 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
142b9cac9 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
f1c10600c 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
6b33edb53 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
 3139d4374 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
 a1a727c82 
  repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
  
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
 1839e76c7 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 
  repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
487cf2778 


Diff: https://reviews.apache.org/r/73058/diff/8/

Changes: https://reviews.apache.org/r/73058/diff/7-8/


Testing
---

Added testcases in DSLQueriesTest.java
Precommit : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/


Thanks,

Pinal Shah

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

2021-01-12 Thread Pinal Shah


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/
---

(Updated Jan. 12, 2021, 10:53 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Changes
---

addressed - nit: imports


Bugs: ATLAS-4057
https://issues.apache.org/jira/browse/ATLAS-4057


Repository: atlas


Description
---

Supporting glossary term in advanced search, will allow user to search the 
entities which are assigned to particular glossary term.

 
**WorkAround:**

'hasTerm' keyword is added to support glossary term.

User needs to add fully qualifiedName of glossary term.

 

**Example queries to search with term:**

hive_table hasTerm "modernTrade@salesGlossary",

hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension

**Example queries to search with relationship:**

hive_table where db.name = "sales"


Diffs (updated)
-

  repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
6c48a82f4 
  repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
15d1bb4cb 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
142b9cac9 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
f1c10600c 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
6b33edb53 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
 3139d4374 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
 a1a727c82 
  repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
  
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
 1839e76c7 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 
  repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
487cf2778 


Diff: https://reviews.apache.org/r/73058/diff/7/

Changes: https://reviews.apache.org/r/73058/diff/6-7/


Testing
---

Added testcases in DSLQueriesTest.java
Precommit : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/


Thanks,

Pinal Shah

Review Request 73132: ATLAS-4079 : Atlas-Upgrade jackson to 2.11.0+

2021-01-12 Thread chaitali


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73132/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-4079
https://issues.apache.org/jira/browse/ATLAS-4079


Repository: atlas


Description
---

This patch updates jackson databind version to 2.11.3


Diffs
-

  pom.xml 7bfd68b94 


Diff: https://reviews.apache.org/r/73132/diff/1/


Testing
---

https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/324/console
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
Atlas server running


Thanks,

chaitali

Re: Review Request 73076: Deferred Actions Implementation

[jira] [Commented] (ATLAS-4092) Remove unused attributes and add right attributes to the atlas admin audit api

[jira] [Commented] (ATLAS-4092) Remove unused attributes and add right attributes to the atlas admin audit api

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

Re: Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73139: ATLAS-4086: Fixes the search and other major issues in python client

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory

Re: Review Request 73139: ATLAS-4086: Fixes the search and other major issues in python client

Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory

[jira] [Created] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Review Request 73139: ATLAS-4086: Fixes the search and other major issues in python client

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

Review Request 73132: ATLAS-4079 : Atlas-Upgrade jackson to 2.11.0+

29 matches

Site Navigation

Mail list logo

Footer information