[jira] [Updated] (ATLAS-4102) UI - DSL Search: UI hangs When Executing Certain Queries

[Prasad P. Pawar (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad P. Pawar updated ATLAS-4102:
---
Attachment: Atlas-4102.png

> UI - DSL Search:  UI hangs When Executing Certain Queries
> -
>
> Key: ATLAS-4102
> URL: https://issues.apache.org/jira/browse/ATLAS-4102
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Prasad P. Pawar
>Priority: Major
> Attachments: 
> 0001-Atlas-4102-UI-DSL-Search-UI-hangs-When-Executing-Cer.patch, 
> Atlas-4102.png
>
>
> *Background*
> Certain types of Advanced Search (DSL) queries within Atlas return 
> _AtlasSearchResult_ in a slightly different format.
> The current UI is not able to render the search results.
> The REST APIs however, return the output successfully.
> *Steps to Duplicate*
>  # Within Atlas web UI, Advanced Search, execute: _hive_table groupby (owner) 
> select min(name)_ _URL: 
> http://:31000/index.html#!/search/searchResult?query=hive_table%20groupby%20(owner)%20select%20min(name)&searchType=dsl&dslChecked=true_
> _Expected result:_ Search results should be displayed.
> Actual result: Web UI freezes. Upon inspecting using Developer tools, a JS 
> exception is displayed.
> Additional info:
> Query: _hive_table groupby(owner) select owner, max(name)_
> Output json:
> {code:java}
> {
> "attributes": {
> "name": [
> "owner",
> "max(name)"
> ],
> "values": [
> [
> "hive",
> "tuarp_uarmt_pmtreq_h"
> ],
> [
> "iadetldev",
> "video_analytics_log_id"
> ],
> [
> "atsacoli",
> "aj_psync_test_h"
> ]
> ]
> },
> "approximateCount": -1
> }{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4099:
--
Fix Version/s: 2.2.0
   3.0.0

> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory 
> created by Hook
> 
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location 
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName : 
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because 
> of this , duplicate entities are created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4099:
--
Component/s:  atlas-core

> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory 
> created by Hook
> 
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location 
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName : 
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because 
> of this , duplicate entities are created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264644#comment-17264644
 ] 

ASF subversion and git services commented on ATLAS-4099:


Commit 8a27effd7b27bb5f880ff01d57135be258a57607 in atlas's branch 
refs/heads/branch-2.0 from sidmishra
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=8a27eff ]

ATLAS-4099: Removed trailing path separator from adls_gen2_directory 
qualifiedName

Signed-off-by: Sarath Subramanian 
(cherry picked from commit d27790dfdde09a58db15064bbdaf77d224f61ecc)


> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory 
> created by Hook
> 
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location 
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName : 
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because 
> of this , duplicate entities are created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4099:
--
Affects Version/s: 2.1.0

> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory 
> created by Hook
> 
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location 
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName : 
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because 
> of this , duplicate entities are created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (ATLAS-4102) UI - DSL Search: UI hangs When Executing Certain Queries

[Prasad P. Pawar (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264641#comment-17264641
 ] 

Prasad P. Pawar edited comment on ATLAS-4102 at 1/14/21, 6:51 AM:
--

HI,
I have provided the fix in this available patch.


was (Author: prasadpp13):
HI,
I have provided the fix in theis available patch.

> UI - DSL Search:  UI hangs When Executing Certain Queries
> -
>
> Key: ATLAS-4102
> URL: https://issues.apache.org/jira/browse/ATLAS-4102
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Prasad P. Pawar
>Priority: Major
> Attachments: 
> 0001-Atlas-4102-UI-DSL-Search-UI-hangs-When-Executing-Cer.patch
>
>
> *Background*
> Certain types of Advanced Search (DSL) queries within Atlas return 
> _AtlasSearchResult_ in a slightly different format.
> The current UI is not able to render the search results.
> The REST APIs however, return the output successfully.
> *Steps to Duplicate*
>  # Within Atlas web UI, Advanced Search, execute: _hive_table groupby (owner) 
> select min(name)_ _URL: 
> http://:31000/index.html#!/search/searchResult?query=hive_table%20groupby%20(owner)%20select%20min(name)&searchType=dsl&dslChecked=true_
> _Expected result:_ Search results should be displayed.
> Actual result: Web UI freezes. Upon inspecting using Developer tools, a JS 
> exception is displayed.
> Additional info:
> Query: _hive_table groupby(owner) select owner, max(name)_
> Output json:
> {code:java}
> {
> "attributes": {
> "name": [
> "owner",
> "max(name)"
> ],
> "values": [
> [
> "hive",
> "tuarp_uarmt_pmtreq_h"
> ],
> [
> "iadetldev",
> "video_analytics_log_id"
> ],
> [
> "atsacoli",
> "aj_psync_test_h"
> ]
> ]
> },
> "approximateCount": -1
> }{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264643#comment-17264643
 ] 

ASF subversion and git services commented on ATLAS-4099:


Commit d27790dfdde09a58db15064bbdaf77d224f61ecc in atlas's branch 
refs/heads/master from sidmishra
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=d27790d ]

ATLAS-4099: Removed trailing path separator from adls_gen2_directory 
qualifiedName

Signed-off-by: Sarath Subramanian 


> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory 
> created by Hook
> 
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location 
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName : 
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because 
> of this , duplicate entities are created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Ashutosh Mestry via Review Board]

> On Jan. 14, 2021, 5:12 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
> > Lines 145 (patched)
> > 
> >
> > if vertex doesn't have the property, would value be null? Please review 
> > and update to handle this case.

Result will be NULL if property does not exist. I have handled NULL condition 
here.


> On Jan. 14, 2021, 5:12 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
> > Lines 160 (patched)
> > 
> >
> > AtlasNumericTypeComparator handles string values as well. Is this else 
> > block better than using AtlasNumericTypeComparator - as in if block?

I have renamed this class.


> On Jan. 14, 2021, 5:12 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
> > Lines 185 (patched)
> > 
> >
> > AtlasNumericTypeComparator handles String values as well. What type of 
> > values this class doesn't handle?
> > Perhaps this should be renamed as AtlasAttributeValueComparator?

I have renamed this.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222470
---


On Jan. 14, 2021, 1:04 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 14, 2021, 1:04 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/
---

(Updated Jan. 14, 2021, 6:46 a.m.)


Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Additional refactoring.


Bugs: ATLAS-2932
https://issues.apache.org/jira/browse/ATLAS-2932


Repository: atlas


Description
---

**Approach**
General Approach:
- Ensure that Gremlin-based implementation continues to function. This will 
ensure backward compatibility and will guarad against regressions or behavior 
deviations, if any, are introduced with the new implementation.
- Continue using existing _GremlinQueryComposer_. 
- Translate the clauses to _AtlasGraphTraversal_.
- Projects that were earlier handled in Gremlin query are now handled in a 
separate Java class.

Details:
- Modified: _GremlinQueryComposer_: Continue composing queries using existing 
claues. Additions alone.
- New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
- Modifed: _EntityDiscoveryService_ uses the new exectors.
- New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
_AtlasGraphTraversal_.
- Modified: _AtlasGraphTraversal_: Additional methods.
- New: _SelectClauseProjections_: All queries with select operations.

**Configuration**
_atlas.dsl.executor.traversal_ 
- true: Uses the traversal-based implementation for query execution.
- false: Uses the script-engine based implementation for query execution.

**Credits**
- Apoorv Naik (apoorvnaik): Original implementation.
- Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
short-comings.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
c016f6340 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
 881bb1e0f 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 0dd573b89 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
 c33c4f4d0 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
01a6c303a 
  repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
  repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 9704b0f9e 
  repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
9f30e4dc0 
  repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
969fcd2b1 
  
repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 


Diff: https://reviews.apache.org/r/73128/diff/9/

Changes: https://reviews.apache.org/r/73128/diff/8-9/


Testing
---

**Unit Tests**
- Additional tests.
- _GremlinCompoerTest_ continues to validate the clause creation.
- Improved validation for _minMaxCount_ tests.
- Refactored: _TableValidator_ Validates output from a query with select claues.

**Volume Tests**
- Old implementation: 528 queries take over 10 mins to execute.
- New implemeentations: 528 queries execute in 7 secs.


Thanks,

Ashutosh Mestry

[jira] [Commented] (ATLAS-4057) DSL Search : Support glossary terms and relationships

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264638#comment-17264638
 ] 

ASF subversion and git services commented on ATLAS-4057:


Commit fc5b0deb288706e6277cde8df09e1482fb3fc614 in atlas's branch 
refs/heads/branch-2.0 from Pinal
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=fc5b0de ]

ATLAS-4057 : DSL Search : Support glossary terms and relationship

Signed-off-by: Pinal 


> DSL Search : Support glossary terms and relationships
> -
>
> Key: ATLAS-4057
> URL: https://issues.apache.org/jira/browse/ATLAS-4057
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
>  Labels: DSL
>
> Supporting glossary term in advanced search, will allow user to search the 
> entities which are assigned to particular term.
>  
> 'hasTerm' keyword is added to support glossary term.
> User needs to add fully qualifiedName of glossary term.
>  
> Example queries to search with glossary terms:
> hive_table hasTerm "modernTrade@salesGlossary",
> hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension
>  
> Example queries to search with relationship:
> hive_table where db.name = "sales"



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4057) DSL Search : Support glossary terms and relationships

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264637#comment-17264637
 ] 

ASF subversion and git services commented on ATLAS-4057:


Commit 2742ba4b776377d7ebdc166214531788e8294776 in atlas's branch 
refs/heads/master from Pinal
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=2742ba4 ]

ATLAS-4057 : DSL Search : Support glossary terms and relationship

Signed-off-by: Pinal 


> DSL Search : Support glossary terms and relationships
> -
>
> Key: ATLAS-4057
> URL: https://issues.apache.org/jira/browse/ATLAS-4057
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
>  Labels: DSL
>
> Supporting glossary term in advanced search, will allow user to search the 
> entities which are assigned to particular term.
>  
> 'hasTerm' keyword is added to support glossary term.
> User needs to add fully qualifiedName of glossary term.
>  
> Example queries to search with glossary terms:
> hive_table hasTerm "modernTrade@salesGlossary",
> hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension
>  
> Example queries to search with relationship:
> hive_table where db.name = "sales"



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/
---

(Updated Jan. 14, 2021, 6:40 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Changes
---

rebase trailing whitespaces


Bugs: ATLAS-4057
https://issues.apache.org/jira/browse/ATLAS-4057


Repository: atlas


Description
---

Supporting glossary term in advanced search, will allow user to search the 
entities which are assigned to particular glossary term.

 
**WorkAround:**

'hasTerm' keyword is added to support glossary term.

User needs to add fully qualifiedName of glossary term.

 

**Example queries to search with term:**

hive_table hasTerm "modernTrade@salesGlossary",

hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension

**Example queries to search with relationship:**

hive_table where db.name = "sales"


Diffs (updated)
-

  docs/src/documents/Search/SearchAdvanced.md 46be14278 
  repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
6c48a82f4 
  repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
15d1bb4cb 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
142b9cac9 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
f1c10600c 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
6b33edb53 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
 3139d4374 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
 a1a727c82 
  repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
  
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
 1839e76c7 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 
  repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
487cf2778 


Diff: https://reviews.apache.org/r/73058/diff/10/

Changes: https://reviews.apache.org/r/73058/diff/9-10/


Testing
---

Added testcases in DSLQueriesTest.java
Precommit : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/


Thanks,

Pinal Shah

[jira] [Created] (ATLAS-4102) UI - DSL Search: Web UI hangs When Executing Certain Queries

[Nixon Rodrigues (Jira)]

Nixon Rodrigues created ATLAS-4102:
--

 Summary: UI - DSL Search: Web UI hangs When Executing Certain 
Queries
 Key: ATLAS-4102
 URL: https://issues.apache.org/jira/browse/ATLAS-4102
 Project: Atlas
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Nixon Rodrigues
Assignee: Prasad P. Pawar


*Background*

Certain types of Advanced Search (DSL) queries within Atlas return 
_AtlasSearchResult_ in a slightly different format.

The current UI is not able to render the search results.

The REST APIs however, return the output successfully.

*Steps to Duplicate*
 # Within Atlas web UI, Advanced Search, execute: _hive_table groupby (owner) 
select min(name)_ _URL: 
http://:31000/index.html#!/search/searchResult?query=hive_table%20groupby%20(owner)%20select%20min(name)&searchType=dsl&dslChecked=true_

_Expected result:_ Search results should be displayed.

Actual result: Web UI freezes. Upon inspecting using Developer tools, a JS 
exception is displayed.

Additional info:

Query: _hive_table groupby(owner) select owner, max(name)_

Output json:
{code:java}
{
"attributes": {
"name": [
"owner",
"max(name)"
],
"values": [
[
"hive",
"tuarp_uarmt_pmtreq_h"
],
[
"iadetldev",
"video_analytics_log_id"
],
[
"atsacoli",
"aj_psync_test_h"
]
]
},
"approximateCount": -1
}{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4102) UI - DSL Search: UI hangs When Executing Certain Queries

[Nixon Rodrigues (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-4102:
---
Summary: UI - DSL Search:  UI hangs When Executing Certain Queries  (was: 
UI - DSL Search: Web UI hangs When Executing Certain Queries)

> UI - DSL Search:  UI hangs When Executing Certain Queries
> -
>
> Key: ATLAS-4102
> URL: https://issues.apache.org/jira/browse/ATLAS-4102
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Prasad P. Pawar
>Priority: Major
>
> *Background*
> Certain types of Advanced Search (DSL) queries within Atlas return 
> _AtlasSearchResult_ in a slightly different format.
> The current UI is not able to render the search results.
> The REST APIs however, return the output successfully.
> *Steps to Duplicate*
>  # Within Atlas web UI, Advanced Search, execute: _hive_table groupby (owner) 
> select min(name)_ _URL: 
> http://:31000/index.html#!/search/searchResult?query=hive_table%20groupby%20(owner)%20select%20min(name)&searchType=dsl&dslChecked=true_
> _Expected result:_ Search results should be displayed.
> Actual result: Web UI freezes. Upon inspecting using Developer tools, a JS 
> exception is displayed.
> Additional info:
> Query: _hive_table groupby(owner) select owner, max(name)_
> Output json:
> {code:java}
> {
> "attributes": {
> "name": [
> "owner",
> "max(name)"
> ],
> "values": [
> [
> "hive",
> "tuarp_uarmt_pmtreq_h"
> ],
> [
> "iadetldev",
> "video_analytics_log_id"
> ],
> [
> "atsacoli",
> "aj_psync_test_h"
> ]
> ]
> },
> "approximateCount": -1
> }{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4100) Recent datamodel changes (Jan 2021) broke our installation

[Nikhil Bonte (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264633#comment-17264633
 ] 

Nikhil Bonte commented on ATLAS-4100:
-

[~nevi_me]
{quote}I can see the entity count on the sidebar
{quote}
Based on this I assume that typeDefs API returns type definitions (GET 
/v2/types/typedefs)
{quote}and inspecting the API shows that it retrieves information, but that 
information can't be shown.
{quote}
Not clear which API, but I guess you are referring to search API & it returns 
search results but results are not being rendered on UI. 

If that is the case, there might some issue with UI, please check the console 
tab from the browser's developer tools if it has any clue.
{quote}The export API isn't working,...
{quote}
Could you please tell us more about what is the issue with export API?

 

> Recent datamodel changes (Jan 2021) broke our installation
> --
>
> Key: ATLAS-4100
> URL: https://issues.apache.org/jira/browse/ATLAS-4100
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Neville Dipale
>Priority: Critical
>
> Hi Atlas developers
> We run an Atlas installation (with external Cassandra & ElasticSearch) where 
> we build Atlas via a Docker script, pointing to specific git commits.
> The last update we have was from early December 2020. Yesterday I updated to 
> the latest git tree, but after the update, the Atlas UI doesn't show 
> entities. I can see the entity count on the sidebar, and inspecting the API 
> shows that it retrieves information, but that information can't be shown.
> I suspect that the changes to the cloud datamodel could be the cause (or at 
> least one of the causes). ATLAS-4082 created patches to the Azure entity 
> definitions, but ATLAS-4083 directly modified the entity definitions. I can't 
> definitively pinpoint the issue, but I noticed the issue with no longer being 
> able to view data, after trying to apply the changes to ATLAS-4083 to our 
> installation.
> The export API isn't working, so we have no viable alternative but to try 
> extract the data manually from Cassandra, or worse, to destroy our data & 
> start from scratch.
> Further, I unfortunately don't have any logs to attach, because in order for 
> us to run Atlas with LDAP integration + a keystore file, the way we run Atlas 
> (going in the container to manually start each time) doesn't output logs, 
> even though we have configured the log4j.xml file propertly.
> I suppose I'm opening this in case other people come across this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222470
---




repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java
Lines 231 (patched)


Shouldn't the argument to setIsPrimitiveAttr() be index into items (and not 
primitiveTypeCount)?



repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
Lines 52 (patched)


subTraversals is used only within process(). Consider moving this field as 
a local variable within process().



repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
Lines 40 (patched)


Please consider marking all fields as final.



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 145 (patched)


if vertex doesn't have the property, would value be null? Please review and 
update to handle this case.



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 152 (patched)


maxV is used only within 'if' block at #156. Consider moving #152 to inside 
'if' block.



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 154 (patched)


final?



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 160 (patched)


AtlasNumericTypeComparator handles string values as well. Is this else 
block better than using AtlasNumericTypeComparator - as in if block?



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 168 (patched)


maxV is used only within 'if' block at #172. Consider moving #168 to inside 
'if' block.



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 170 (patched)


final?



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 176 (patched)


AtlasNumericTypeComparator handles string values as well. Is this else 
block better than using AtlasNumericTypeComparator - as in if block?



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 185 (patched)


AtlasNumericTypeComparator handles String values as well. What type of 
values this class doesn't handle?
Perhaps this should be renamed as AtlasAttributeValueComparator?



repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
Lines 198 (patched)


Consider handling null values:
  if (p1 == null && p2 == null) {
return 0;
  } else if (p1 == null) {
return -1;
  } else if (p2 == null) {
return 1;
  }  else if (p1 instanceof String && p2 instanceof String) {
...



repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 72 (patched)


getAtlasSearchResult() =>  getSearchResult()



repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 73 (patched)


resultList will not be empty, ref #67; Please review and remove #73  - #76.


- Madhan Neethiraj


On Jan. 14, 2021, 1:04 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 14, 2021, 1:04 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
>

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73119/#review222469
---




intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
Lines 318 (patched)


Please move field declarations at the start of the class, before methods.



intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
Lines 325 (patched)


for better readability, have statements in separate lines, like:
  case SORT_COLUMN_USER:
comparator = new EntityAuditEventV2.UserComparator();
break;



repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
Lines 193 (patched)


Given existing listEventsV2() method support filter by auditAction, 
consider retaining the same for the new method as well.



repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
Lines 287 (patched)


scanner is reassigned before closing it. Please review and close.



repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
Lines 297 (patched)


For better readability, avoid assignment + test statements like #297. 
Consider replacing with:
  String colDef = getResultString(result, COLUMN_DEFINITION);
  
  if (colDef != null) {
event.setEntityDefinition(colDef);
  }



webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 801 (patched)


Instead of adding a new REST endpoint, I strongly suggest to update 
existing method by adding following arguments:
  @QueryParam("offset") @DefaultValue("-1") int offset,
  @QueryParam("sortBy") String sortBy,
  @QueryParam("sortOrder") String sortOrder

When a value is provided for offset or sortBy, call the new listEventsV2() 
method; else call existing method.


- Madhan Neethiraj


On Jan. 12, 2021, 11:35 p.m., Deep Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73119/
> ---
> 
> (Updated Jan. 12, 2021, 11:35 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4094
> https://issues.apache.org/jira/browse/ATLAS-4094
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Created a new entity audit API with Sorting functionality.
> 
> /api/atlas/v2/entity/{entity-guid}/audit/sortby/{sort-column}?offset=2&count=5
> 
> sort-column may have 3 values "user", "action", or "timestamp".
> 
> HBase does not support query with sorted results. To support this API 
> inmemory sort has to be performed.
> Audit entry can potentially have entire entity dumped into it. Loading entire 
> audit entries for an entity can be memory intensive. Therefore we load audit 
> entries with limited columns first, perform sort on this light weight list, 
> then get the relevant section by removing offsets and reducing to limits. 
> With this reduced list we create MultiRowRangeFilter and then again scan the 
> table to get all the columns from the table this time.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 083acac73 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  8a453fd43 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
>  07784d1c4 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  9fca74470 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
>  900df0205 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
>  ef9e259ea 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 0d6d0c845 
> 
> 
> Diff: https://reviews.apache.org/r/73119/diff/3/
> 
> 
> Testing
> ---
> 
> Manual testing was done.
> 
> Did testing on a setup with 1 million audit entries spread across 1000 
> entities. 
> Existing Rest API took 6-12 milliseconds for preparing the result.
> In-memory sort and double scan approach took 55-75 milliseconds.
> Single Full scan and in-memory approach took 250-300 milliseconds.
> 
> As it was expected, the new API is 4X slower than the existing API therefore 
> the existing API still should be the primary API for querying audit events. 
> And the new API should be used only if s

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Ashutosh Mestry via Review Board]

> On Jan. 13, 2021, 8:46 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Line 120 (original), 142 (patched)
> > 
> >
> > - for readability, and to be consistent with rest of Atlas sources, 
> > please move private methods after all public methods
> > - given this is a oneliner, and called only once from the constructor, 
> > is there a need for a separate method?

This was done to address another comment. I will revert it.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222466
---


On Jan. 14, 2021, 1:04 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 14, 2021, 1:04 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/7/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take o

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/
---

(Updated Jan. 14, 2021, 1:04 a.m.)


Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Additional refactoring for simplicity.


Bugs: ATLAS-2932
https://issues.apache.org/jira/browse/ATLAS-2932


Repository: atlas


Description
---

**Approach**
General Approach:
- Ensure that Gremlin-based implementation continues to function. This will 
ensure backward compatibility and will guarad against regressions or behavior 
deviations, if any, are introduced with the new implementation.
- Continue using existing _GremlinQueryComposer_. 
- Translate the clauses to _AtlasGraphTraversal_.
- Projects that were earlier handled in Gremlin query are now handled in a 
separate Java class.

Details:
- Modified: _GremlinQueryComposer_: Continue composing queries using existing 
claues. Additions alone.
- New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
- Modifed: _EntityDiscoveryService_ uses the new exectors.
- New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
_AtlasGraphTraversal_.
- Modified: _AtlasGraphTraversal_: Additional methods.
- New: _SelectClauseProjections_: All queries with select operations.

**Configuration**
_atlas.dsl.executor.traversal_ 
- true: Uses the traversal-based implementation for query execution.
- false: Uses the script-engine based implementation for query execution.

**Credits**
- Apoorv Naik (apoorvnaik): Original implementation.
- Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
short-comings.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
c016f6340 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
 881bb1e0f 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 0dd573b89 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
 c33c4f4d0 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
01a6c303a 
  repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
  repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 9704b0f9e 
  repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
9f30e4dc0 
  repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
969fcd2b1 
  
repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 


Diff: https://reviews.apache.org/r/73128/diff/7/

Changes: https://reviews.apache.org/r/73128/diff/6-7/


Testing
---

**Unit Tests**
- Additional tests.
- _GremlinCompoerTest_ continues to validate the clause creation.
- Improved validation for _minMaxCount_ tests.
- Refactored: _TableValidator_ Validates output from a query with select claues.

**Volume Tests**
- Old implementation: 528 queries take over 10 mins to execute.
- New implemeentations: 528 queries execute in 7 secs.


Thanks,

Ashutosh Mestry

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/#review222468
---


Ship it!




Ship It!

- Sarath Subramanian


On Jan. 13, 2021, 4:31 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73058/
> ---
> 
> (Updated Jan. 13, 2021, 4:31 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4057
> https://issues.apache.org/jira/browse/ATLAS-4057
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Supporting glossary term in advanced search, will allow user to search the 
> entities which are assigned to particular glossary term.
> 
>  
> **WorkAround:**
> 
> 'hasTerm' keyword is added to support glossary term.
> 
> User needs to add fully qualifiedName of glossary term.
> 
>  
> 
> **Example queries to search with term:**
> 
> hive_table hasTerm "modernTrade@salesGlossary",
> 
> hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension
> 
> **Example queries to search with relationship:**
> 
> hive_table where db.name = "sales"
> 
> 
> Diffs
> -
> 
>   docs/src/documents/Search/SearchAdvanced.md 46be14278 
>   
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
> 6c48a82f4 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
> 15d1bb4cb 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 142b9cac9 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
> f1c10600c 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 6b33edb53 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
>  3139d4374 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
>  a1a727c82 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  1839e76c7 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 487cf2778 
> 
> 
> Diff: https://reviews.apache.org/r/73058/diff/9/
> 
> 
> Testing
> ---
> 
> Added testcases in DSLQueriesTest.java
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222466
---




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Line 120 (original), 142 (patched)


- for readability, and to be consistent with rest of Atlas sources, please 
move private methods after all public methods
- given this is a oneliner, and called only once from the constructor, is 
there a need for a separate method?



repository/src/main/java/org/apache/atlas/query/AtlasDSL.java
Lines 153 (patched)


- per earlier comment, why not final for resolvedLimit and resolvedOffset 
as well?
- suggest to rename as:
  resolvedLimit  => limit
  resolvedOffset => offset



repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
Line 88 (original), 92 (patched)


- context is not initialized here. It should be, perhaps with:
   this.context = new Context(lookup);
- also, mark this.context as final


- Madhan Neethiraj


On Jan. 13, 2021, 8:26 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 13, 2021, 8:26 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Ashutosh Mestry via Review Board]

- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222455
---


On Jan. 13, 2021, 8:26 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 13, 2021, 8:26 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/6/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries execute in 7 secs.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Ashutosh Mestry via Review Board]

> On Jan. 13, 2021, 5:12 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
> > Lines 51 (patched)
> > 
> >
> > subTraversals is used only within process() method. Consider removing 
> > it as a member.

The subTraversals are collected when subclaueses are processed, these are then 
used in OR or AND clasuse. Hence this needs to be part of the state variable.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222455
---


On Jan. 13, 2021, 8:26 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 13, 2021, 8:26 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/6/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries execute in 7 secs.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/
---

(Updated Jan. 13, 2021, 8:26 p.m.)


Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Refactored for simplicity.
- Additional unit tests.


Bugs: ATLAS-2932
https://issues.apache.org/jira/browse/ATLAS-2932


Repository: atlas


Description
---

**Approach**
General Approach:
- Ensure that Gremlin-based implementation continues to function. This will 
ensure backward compatibility and will guarad against regressions or behavior 
deviations, if any, are introduced with the new implementation.
- Continue using existing _GremlinQueryComposer_. 
- Translate the clauses to _AtlasGraphTraversal_.
- Projects that were earlier handled in Gremlin query are now handled in a 
separate Java class.

Details:
- Modified: _GremlinQueryComposer_: Continue composing queries using existing 
claues. Additions alone.
- New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
- Modifed: _EntityDiscoveryService_ uses the new exectors.
- New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
_AtlasGraphTraversal_.
- Modified: _AtlasGraphTraversal_: Additional methods.
- New: _SelectClauseProjections_: All queries with select operations.

**Configuration**
_atlas.dsl.executor.traversal_ 
- true: Uses the traversal-based implementation for query execution.
- false: Uses the script-engine based implementation for query execution.

**Credits**
- Apoorv Naik (apoorvnaik): Original implementation.
- Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
short-comings.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
c016f6340 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
 881bb1e0f 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 0dd573b89 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
 c33c4f4d0 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
01a6c303a 
  repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 9704b0f9e 
  repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
9f30e4dc0 
  repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
969fcd2b1 
  
repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 


Diff: https://reviews.apache.org/r/73128/diff/6/

Changes: https://reviews.apache.org/r/73128/diff/5-6/


Testing
---

**Unit Tests**
- Additional tests.
- _GremlinCompoerTest_ continues to validate the clause creation.
- Improved validation for _minMaxCount_ tests.
- Refactored: _TableValidator_ Validates output from a query with select claues.

**Volume Tests**
- Old implementation: 528 queries take over 10 mins to execute.
- New implemeentations: 528 queries execute in 7 secs.


Thanks,

Ashutosh Mestry

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Ashutosh Mestry via Review Board]

> On Jan. 13, 2021, 6:15 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
> > Lines 103 (patched)
> > 
> >
> > can there be a case where query contains 'offset' and no 'limit' set?

Yes. In that case, we set it to default limit of 25 and offset is 0. There are 
test cases to verify this. This gets set in GremlinQueryComposer.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222457
---


On Jan. 12, 2021, 11:49 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 12, 2021, 11:49 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/GraphQueryTest.java
>  4b0176311 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 36b514e84 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/5/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries exec

Re: Review Request 73139: ATLAS-4086: Fixes the search and other major issues in python client

[Madhan Neethiraj]

> On Jan. 12, 2021, 11:38 p.m., Madhan Neethiraj wrote:
> > intg/src/main/python/apache_atlas/model/glossary.py
> > Line 19 (original), 19 (patched)
> > 
> >
> > Can you please look this following error?
> > 
> > Traceback (most recent call last):
> >   File "test.py", line 3, in 
> > from apache_atlas.client.base_client import AtlasClient
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/client/base_client.py", line 
> > 26, in 
> > from apache_atlas.client.discovery import DiscoveryClient
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/client/discovery.py", line 
> > 18, in 
> > from apache_atlas.model.discovery import (AtlasQuickSearchResult, 
> > AtlasSearchResult,
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/model/discovery.py", line 
> > 21, in 
> > from apache_atlas.model.instance import AtlasEntityHeader
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/model/instance.py", line 19, 
> > in 
> > from apache_atlas.model.glossary import AtlasTermAssignmentHeader
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/model/glossary.py", line 19, 
> > in 
> > import apache_atlas.model.instance as instance
> > AttributeError: module 'apache_atlas.model' has no attribute 'instance'
> 
> Verdan Mahmood wrote:
> Can you please share your test.py file? Also, how do you run/install this 
> for testing? 
> Tried to run with the clean install and things are working properly on my 
> end.

- I ran the sample code in apache-atlas homepage 
https://pypi.org/project/apache-atlas/
- I set env variable PYTHONPATH=/intg/src/main/python


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73139/#review222454
---


On Jan. 12, 2021, 9:54 p.m., Verdan Mahmood wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73139/
> ---
> 
> (Updated Jan. 12, 2021, 9:54 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Pinal Shah, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4086
> https://issues.apache.org/jira/browse/ATLAS-4086
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Signed-off-by: verdan 
> 
> This review request fixes a number of major issues in the python atlas client.
> 
> 
> Diffs
> -
> 
>   intg/src/main/python/apache_atlas/client/admin.py 
> ebe72dcc79b18b5955f5403889895bc7bfe4923b 
>   intg/src/main/python/apache_atlas/client/discovery.py 
> 04f5fc9cf68edbd5973d48b402d764030a7068ca 
>   intg/src/main/python/apache_atlas/client/entity.py 
> 7edacf9e72a3a5d3cc55008b39b60fb88dc5c607 
>   intg/src/main/python/apache_atlas/client/glossary.py 
> efa0612fa0c78b850b7df76cc7e89d183ffb8e41 
>   intg/src/main/python/apache_atlas/client/relationship.py 
> f5790a4c369b6f2a517b81425d8fa237ca2b7cdc 
>   intg/src/main/python/apache_atlas/client/typedef.py 
> aa1581d86df26979f3a6c6bb798ea7b0f6ab3bd0 
>   intg/src/main/python/apache_atlas/model/glossary.py 
> 7a0faac37f3c74cdd0eb49012b88c376323e4b3f 
>   intg/src/main/python/apache_atlas/model/instance.py 
> 8cfb254c49328b495454158eec26bd1cc250b929 
>   intg/src/main/python/setup.py 7c563c9c0610eeba169b1c8bec73a246c2a95d4b 
> 
> 
> Diff: https://reviews.apache.org/r/73139/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Verdan Mahmood
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222455
---




repository/src/main/java/org/apache/atlas/query/AtlasDSL.java
Lines 140 (patched)


Can't this.queryMetadata be used here as well, similar to #127?



repository/src/main/java/org/apache/atlas/query/AtlasDSL.java
Lines 146 (patched)


Lines #140 - #146 is repeat of lines in translate() above. And the caller 
TraversalBaseExecutor.toTraversal() calls both methods, on the same Translator 
object. Consider creating gremlinQueryComposer in the constructor.



repository/src/main/java/org/apache/atlas/query/AtlasDSL.java
Lines 177 (patched)


Consider marking all data members as final, as they are unchanged after 
initialization in constructor.



repository/src/main/java/org/apache/atlas/query/GremlinQuery.java
Lines 26 (patched)


Consider marking result as final.



repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
Lines 85 (patched)


Consider marking all members final, as none of them need to be updated 
after construction.



repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
Lines 713 (patched)


- childClauses doesn't seem to be used. Please review and remove if unused
- if childClauses is necessary (i.e. will be used later), consider marking 
it as final



repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
Line 693 (original), 766 (patched)


Consider marking validator as final, as it is unchanged after construction.



repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
Lines 51 (patched)


subTraversals is used only within process() method. Consider removing it as 
a member.



repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
Lines 72 (patched)


Review if subTraversals should be reset/initialized for each iteration of 
the for-loop at #63.



repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
Lines 93 (patched)


subTraversals could be null, if #64 is false. Consider avoiding such 
implicit assumption (that subTraversals will not be referenced when #64 is 
false) - by initializing subTraversals to an ArrayList.


- Madhan Neethiraj


On Jan. 12, 2021, 11:49 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 12, 2021, 11:49 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>

Re: Review Request 73142: ATLAS-4101 : 500 internal server error is thrown instead of 400 malformed request from atlas when the payload is malformed.

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73142/#review222460
---


Ship it!




Ship It!

- Sarath Subramanian


On Jan. 13, 2021, 3 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73142/
> ---
> 
> (Updated Jan. 13, 2021, 3 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nikhil Bonte, Pinal Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4101
> https://issues.apache.org/jira/browse/ATLAS-4101
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Problem: If request json is malformed Atlas server return 500 internal server 
> exception instead bad request.
> 
> Fix : Fixed http error code for JsonParseException by overriding readFrom  
> method of JacksonJsonProvider.
> 
> 
> Diffs
> -
> 
>   webapp/src/main/java/org/apache/atlas/web/util/AtlasJsonProvider.java 
> efe0486c8 
> 
> 
> Diff: https://reviews.apache.org/r/73142/diff/1/
> 
> 
> Testing
> ---
> 
> Response code is 400 after applying fix.
> 
>  $$$ curl --header "Accept:application/json" -H "Content-Type: 
> application/json"  -kv -u admin:admin  -X POST 
> http://localhost:21000/api/atlas/v2/types/typedefs -d 
> '{"classificationDefs":[{"name":"tag_1"},]}'
> Note: Unnecessary use of -X or --request, POST is already inferred.
> *   Trying ::1...
> * TCP_NODELAY set
> * Connection failed
> * connect to ::1 port 21000 failed: Connection refused
> *   Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 21000 (#0)
> * Server auth using Basic with user 'admin'
> > POST /api/atlas/v2/types/typedefs HTTP/1.1
> > Host: localhost:21000
> > Authorization: Basic YWRtaW46YWRtaW4=
> > User-Agent: curl/7.54.0
> > Accept:application/json
> > Content-Type: application/json
> > Content-Length: 42
> > 
> * upload completely sent off: 42 out of 42 bytes
> < HTTP/1.1 400 Bad Request
> < Date: Wed, 13 Jan 2021 10:55:01 GMT
> < Set-Cookie: ATLASSESSIONID=node01doyctnupiijs15wvr6mvkbxw611.node0; Path=/; 
> HttpOnly
> < Expires: Thu, 01 Jan 1970 00:00:00 GMT
> < X-Frame-Options: DENY
> < X-Content-Type-Options: nosniff
> < X-XSS-Protection: 1; mode=block
> < Strict-Transport-Security: max-age=31536000; includeSubDomains
> < Content-Type: application/json;charset=utf-8
> < X-Content-Type-Options: nosniff
> < Content-Security-Policy: default-src 'self'; script-src 'self' 
> 'unsafe-inline' 'unsafe-eval' blob: data:; connect-src 'self'; img-src 'self' 
> blob: data:; style-src 'self' 'unsafe-inline';font-src 'self' data:
> < Server: Apache Atlas
> < Transfer-Encoding: chunked
> < 
> * Connection #0 to host localhost left intact
> {"error":"Unexpected character (']' (code 93)): expected a value\n at 
> [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 42]\n 
> at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 
> 41] (through reference chain: 
> org.apache.atlas.model.typedef.AtlasTypesDef[\\"classificationDefs\\"])"}
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/
---

(Updated Jan. 13, 2021, 12:31 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, and Sarath Subramanian.


Changes
---

added documentation


Bugs: ATLAS-4057
https://issues.apache.org/jira/browse/ATLAS-4057


Repository: atlas


Description
---

Supporting glossary term in advanced search, will allow user to search the 
entities which are assigned to particular glossary term.

 
**WorkAround:**

'hasTerm' keyword is added to support glossary term.

User needs to add fully qualifiedName of glossary term.

 

**Example queries to search with term:**

hive_table hasTerm "modernTrade@salesGlossary",

hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension

**Example queries to search with relationship:**

hive_table where db.name = "sales"


Diffs (updated)
-

  docs/src/documents/Search/SearchAdvanced.md 46be14278 
  repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
6c48a82f4 
  repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
36b514e84 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 
15d1bb4cb 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
142b9cac9 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
f1c10600c 
  repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
6b33edb53 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
 3139d4374 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
 a1a727c82 
  repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 
  
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
 1839e76c7 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf 
  repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
487cf2778 


Diff: https://reviews.apache.org/r/73058/diff/9/

Changes: https://reviews.apache.org/r/73058/diff/8-9/


Testing
---

Added testcases in DSLQueriesTest.java
Precommit : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/


Thanks,

Pinal Shah

Re: Review Request 73139: ATLAS-4086: Fixes the search and other major issues in python client

[Verdan Mahmood]

> On Jan. 12, 2021, 11:38 p.m., Madhan Neethiraj wrote:
> > intg/src/main/python/apache_atlas/model/glossary.py
> > Line 19 (original), 19 (patched)
> > 
> >
> > Can you please look this following error?
> > 
> > Traceback (most recent call last):
> >   File "test.py", line 3, in 
> > from apache_atlas.client.base_client import AtlasClient
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/client/base_client.py", line 
> > 26, in 
> > from apache_atlas.client.discovery import DiscoveryClient
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/client/discovery.py", line 
> > 18, in 
> > from apache_atlas.model.discovery import (AtlasQuickSearchResult, 
> > AtlasSearchResult,
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/model/discovery.py", line 
> > 21, in 
> > from apache_atlas.model.instance import AtlasEntityHeader
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/model/instance.py", line 19, 
> > in 
> > from apache_atlas.model.glossary import AtlasTermAssignmentHeader
> >   File 
> > "<...>/atlas/intg/src/main/python/apache_atlas/model/glossary.py", line 19, 
> > in 
> > import apache_atlas.model.instance as instance
> > AttributeError: module 'apache_atlas.model' has no attribute 'instance'

Can you please share your test.py file? Also, how do you run/install this for 
testing? 
Tried to run with the clean install and things are working properly on my end.


- Verdan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73139/#review222454
---


On Jan. 12, 2021, 9:54 p.m., Verdan Mahmood wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73139/
> ---
> 
> (Updated Jan. 12, 2021, 9:54 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Pinal Shah, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4086
> https://issues.apache.org/jira/browse/ATLAS-4086
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Signed-off-by: verdan 
> 
> This review request fixes a number of major issues in the python atlas client.
> 
> 
> Diffs
> -
> 
>   intg/src/main/python/apache_atlas/client/admin.py 
> ebe72dcc79b18b5955f5403889895bc7bfe4923b 
>   intg/src/main/python/apache_atlas/client/discovery.py 
> 04f5fc9cf68edbd5973d48b402d764030a7068ca 
>   intg/src/main/python/apache_atlas/client/entity.py 
> 7edacf9e72a3a5d3cc55008b39b60fb88dc5c607 
>   intg/src/main/python/apache_atlas/client/glossary.py 
> efa0612fa0c78b850b7df76cc7e89d183ffb8e41 
>   intg/src/main/python/apache_atlas/client/relationship.py 
> f5790a4c369b6f2a517b81425d8fa237ca2b7cdc 
>   intg/src/main/python/apache_atlas/client/typedef.py 
> aa1581d86df26979f3a6c6bb798ea7b0f6ab3bd0 
>   intg/src/main/python/apache_atlas/model/glossary.py 
> 7a0faac37f3c74cdd0eb49012b88c376323e4b3f 
>   intg/src/main/python/apache_atlas/model/instance.py 
> 8cfb254c49328b495454158eec26bd1cc250b929 
>   intg/src/main/python/setup.py 7c563c9c0610eeba169b1c8bec73a246c2a95d4b 
> 
> 
> Diff: https://reviews.apache.org/r/73139/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Verdan Mahmood
> 
>

Review Request 73142: ATLAS-4101 : 500 internal server error is thrown instead of 400 malformed request from atlas when the payload is malformed.

[Nixon Rodrigues]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73142/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nikhil Bonte, Pinal Shah, and Sarath Subramanian.


Bugs: ATLAS-4101
https://issues.apache.org/jira/browse/ATLAS-4101


Repository: atlas


Description
---

Problem: If request json is malformed Atlas server return 500 internal server 
exception instead bad request.

Fix : Fixed http error code for JsonParseException by overriding readFrom  
method of JacksonJsonProvider.


Diffs
-

  webapp/src/main/java/org/apache/atlas/web/util/AtlasJsonProvider.java 
efe0486c8 


Diff: https://reviews.apache.org/r/73142/diff/1/


Testing
---

Response code is 400 after applying fix.

 $$$ curl --header "Accept:application/json" -H "Content-Type: 
application/json"  -kv -u admin:admin  -X POST 
http://localhost:21000/api/atlas/v2/types/typedefs -d 
'{"classificationDefs":[{"name":"tag_1"},]}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 21000 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 21000 (#0)
* Server auth using Basic with user 'admin'
> POST /api/atlas/v2/types/typedefs HTTP/1.1
> Host: localhost:21000
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.54.0
> Accept:application/json
> Content-Type: application/json
> Content-Length: 42
> 
* upload completely sent off: 42 out of 42 bytes
< HTTP/1.1 400 Bad Request
< Date: Wed, 13 Jan 2021 10:55:01 GMT
< Set-Cookie: ATLASSESSIONID=node01doyctnupiijs15wvr6mvkbxw611.node0; Path=/; 
HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Type: application/json;charset=utf-8
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src 'self'; script-src 'self' 
'unsafe-inline' 'unsafe-eval' blob: data:; connect-src 'self'; img-src 'self' 
blob: data:; style-src 'self' 'unsafe-inline';font-src 'self' data:
< Server: Apache Atlas
< Transfer-Encoding: chunked
< 
* Connection #0 to host localhost left intact
{"error":"Unexpected character (']' (code 93)): expected a value\n at [Source: 
(org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 42]\n at 
[Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 41] 
(through reference chain: 
org.apache.atlas.model.typedef.AtlasTypesDef[\\"classificationDefs\\"])"}


Thanks,

Nixon Rodrigues

[jira] [Created] (ATLAS-4101) 500 internal server error is thrown instead of 400 malformed request from atlas when the payload is malformed

[Nixon Rodrigues (Jira)]

Nixon Rodrigues created ATLAS-4101:
--

 Summary: 500 internal server error is thrown instead of 400 
malformed request from atlas when the payload is malformed
 Key: ATLAS-4101
 URL: https://issues.apache.org/jira/browse/ATLAS-4101
 Project: Atlas
  Issue Type: Bug
  Components: atlas-webui
Reporter: Nixon Rodrigues
Assignee: Nixon Rodrigues


{code:java|title=Malformed payload for creating a classification}
{
"classificationDefs": [
{
"name": "tag_1",
}
]
}
{code}
{code:java|title=Malformed payload for creating a classification}
{
"classificationDefs": [
{
"name": "tag_1"
},
]
}
{code}
There is an additional "," in the above 2 payloads, which make it a client side 
payload error.

But when firing such a request 500 internal server error is thrown
 *These are client errors that are being thrown as server errors*
{code:java}
2020-12-11 07:47:28,482 ERROR - [etp402695541-552 - 
6420db24-b42f-45eb-a6bb-25ffb3c8ce0e:] ~ Error handling a request: 
31167d3d0dc60f70 (ExceptionMapperUtil:32)2020-12-11 07:47:28,482 ERROR - 
[etp402695541-552 - 6420db24-b42f-45eb-a6bb-25ffb3c8ce0e:] ~ Error handling a 
request: 31167d3d0dc60f70 
(ExceptionMapperUtil:32)com.fasterxml.jackson.databind.JsonMappingException: 
Unexpected character (']' (code 93)): expected a value at [Source: 
(org.eclipse.jetty.server.HttpInputOverHTTP); line: 6, column: 6] (through 
reference chain: 
org.apache.atlas.model.typedef.AtlasTypesDef["classificationDefs"]) at 
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:397)
 at 
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:356)
 at 
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1719)
 at 
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:290)
 at 
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
 at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1682) 
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:977) 
at 
com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:814) 
at 
com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:490)
 at 
com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
 at 
com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:86)
 at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
 at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:166)
 at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
 at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
 at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
 at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
 at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
 at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
 at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
 at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
 at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
 at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
 at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
 at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:106) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
 at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
 at 
org.springframework.security.web.access.intercept.Fil

[jira] [Created] (ATLAS-4100) Recent datamodel changes broke our installation

[Neville Dipale (Jira)]

Neville Dipale created ATLAS-4100:
-

 Summary: Recent datamodel changes broke our installation
 Key: ATLAS-4100
 URL: https://issues.apache.org/jira/browse/ATLAS-4100
 Project: Atlas
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Neville Dipale


Hi Atlas developers

We run an Atlas installation (with external Cassandra & ElasticSearch) where we 
build Atlas via a Docker script, pointing to specific git commits.

The last update we have was from early December 2020. Yesterday I updated to 
the latest git tree, but after the update, the Atlas UI doesn't show entities. 
I can see the entity count on the sidebar, and inspecting the API shows that it 
retrieves information, but that information can't be shown.

I suspect that the changes to the cloud datamodel could be the cause (or at 
least one of the causes). ATLAS-4082 created patches to the Azure entity 
definitions, but ATLAS-4083 directly modified the entity definitions. I can't 
definitively pinpoint the issue, but I noticed the issue with no longer being 
able to view data, after trying to apply the changes to ATLAS-4083 to our 
installation.

The export API isn't working, so we have no viable alternative but to try 
extract the data manually from Cassandra, or worse, to destroy our data & start 
from scratch.

Further, I unfortunately don't have any logs to attach, because in order for us 
to run Atlas with LDAP integration + a keystore file, the way we run Atlas 
(going in the container to manually start each time) doesn't output logs, even 
though we have configured the log4j.xml file propertly.

I suppose I'm opening this in case other people come across this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4100) Recent datamodel changes (Jan 2021) broke our installation

[Neville Dipale (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neville Dipale updated ATLAS-4100:
--
Summary: Recent datamodel changes (Jan 2021) broke our installation  (was: 
Recent datamodel changes broke our installation)

> Recent datamodel changes (Jan 2021) broke our installation
> --
>
> Key: ATLAS-4100
> URL: https://issues.apache.org/jira/browse/ATLAS-4100
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Neville Dipale
>Priority: Critical
>
> Hi Atlas developers
> We run an Atlas installation (with external Cassandra & ElasticSearch) where 
> we build Atlas via a Docker script, pointing to specific git commits.
> The last update we have was from early December 2020. Yesterday I updated to 
> the latest git tree, but after the update, the Atlas UI doesn't show 
> entities. I can see the entity count on the sidebar, and inspecting the API 
> shows that it retrieves information, but that information can't be shown.
> I suspect that the changes to the cloud datamodel could be the cause (or at 
> least one of the causes). ATLAS-4082 created patches to the Azure entity 
> definitions, but ATLAS-4083 directly modified the entity definitions. I can't 
> definitively pinpoint the issue, but I noticed the issue with no longer being 
> able to view data, after trying to apply the changes to ATLAS-4083 to our 
> installation.
> The export API isn't working, so we have no viable alternative but to try 
> extract the data manually from Cassandra, or worse, to destroy our data & 
> start from scratch.
> Further, I unfortunately don't have any logs to attach, because in order for 
> us to run Atlas with LDAP integration + a keystore file, the way we run Atlas 
> (going in the container to manually start each time) doesn't output logs, 
> even though we have configured the log4j.xml file propertly.
> I suppose I'm opening this in case other people come across this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)