date:20210115

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-15 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222483
---




repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
Lines 284 (patched)


If 'attributeType.getNormalizedValue(attrVal)' is null, #283 will cause 
NPE. i.e. when attrValue is not of expected format.

If attrValue is not validated earlier, I suggest to handle NULL condition 
here.



repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
Lines 108 (patched)


Can you review how this handles nested and/or clauses? Like:
  hive_table where (name = 'test_tbl' or ((owner = 'hive' and name = 
'my_tbl) or (owner = 'sys' or createTime > "2021-01-01'T'00:00:00.000'Z'")))


- Madhan Neethiraj


On Jan. 16, 2021, 7:06 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 16, 2021, 7:06 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Nikhil Bonte, Nixon 
> Rodrigues, Pinal Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java ca0b4be91 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 87c8bd2ee 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 958c4281c 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 6c69855b7 
>   repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/11/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> -

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-15 Thread Ashutosh Mestry via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/
---

(Updated Jan. 16, 2021, 7:06 a.m.)


Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Reverted _Glossary_ support to a much simpler implementation.
- Reverted _DSLVisitor_ to earlier implementation incorporating Glossary term 
keyword.
- Reverted _DSLComposerTest_ to earlier clauses while incorporating Gossary 
term.
- Added _TERM_ to _GremlinClause_ to support Glossary.
- Simplified _GremlinClauseToTraversalTranslator_ when passing collected 
_AtlasGraphTraversals_ collected from sub-clauses.


Bugs: ATLAS-2932
https://issues.apache.org/jira/browse/ATLAS-2932


Repository: atlas


Description
---

**Approach**
General Approach:
- Ensure that Gremlin-based implementation continues to function. This will 
ensure backward compatibility and will guarad against regressions or behavior 
deviations, if any, are introduced with the new implementation.
- Continue using existing _GremlinQueryComposer_. 
- Translate the clauses to _AtlasGraphTraversal_.
- Projects that were earlier handled in Gremlin query are now handled in a 
separate Java class.

Details:
- Modified: _GremlinQueryComposer_: Continue composing queries using existing 
claues. Additions alone.
- New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
- Modifed: _EntityDiscoveryService_ uses the new exectors.
- New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
_AtlasGraphTraversal_.
- Modified: _AtlasGraphTraversal_: Additional methods.
- New: _SelectClauseProjections_: All queries with select operations.

**Configuration**
_atlas.dsl.executor.traversal_ 
- true: Uses the traversal-based implementation for query execution.
- false: Uses the script-engine based implementation for query execution.

**Credits**
- Apoorv Naik (apoorvnaik): Original implementation.
- Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
short-comings.


Diffs (updated)
-

  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
 881bb1e0f 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
 c33c4f4d0 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
01a6c303a 
  repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
  repository/src/main/java/org/apache/atlas/query/DSLVisitor.java ca0b4be91 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 9704b0f9e 
  repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
9f30e4dc0 
  repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
87c8bd2ee 
  repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
969fcd2b1 
  
repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java 
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 958c4281c 
  repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
6c69855b7 
  repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/73128/diff/11/

Changes: https://reviews.apache.org/r/73128/diff/10-11/


Testing
---

**Unit Tests**
- Additional tests.
- _GremlinCompoerTest_ continues to validate the clause creation.
- Improved validation for _minMaxCount_ tests.
- Refactored: _TableValidator_ Validates output from a query with select claues.

**Volume Tests**
- Old implementation: 528 queries take over 10 mins to execute.
- New implemeentations: 528 queries execute in 7 secs.


Thanks,

Ashutosh Mestry

Review Request 73146: ATLAS-4077: replaced print statements in Python files with call to print(), for Python3

2021-01-15 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73146/
---

Review request for atlas, Ashutosh Mestry, Deep Singh, Nikhil Bonte, Nixon 
Rodrigues, Sarath Subramanian, and Siddharth Singh.


Bugs: ATLAS-4077
https://issues.apache.org/jira/browse/ATLAS-4077


Repository: atlas


Description
---

Python3 doesn't support print statements; hence replaced it with calls to 
print() function


Diffs
-

  distro/src/test/python/scripts/TestMetadata.py 251de937c 
  tools/atlas-index-repair/src/main/resources/repair_index.py bbdc85442 
  tools/atlas-migration-exporter/atlas_migration_export.py 591043fcd 


Diff: https://reviews.apache.org/r/73146/diff/1/


Testing
---

- pre-commit tests run: 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/337/


Thanks,

Madhan Neethiraj

[jira] [Updated] (ATLAS-4077) update Python client to work with Kerberos and 2.7

2021-01-15 Thread Madhan Neethiraj (Jira)



 [ 
https://issues.apache.org/jira/browse/ATLAS-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-4077:

Attachment: ATLAS-4077-3.patch

> update Python client to work with Kerberos and 2.7
> --
>
> Key: ATLAS-4077
> URL: https://issues.apache.org/jira/browse/ATLAS-4077
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4077-3.patch, ATLAS-4077.patch
>
>
> Python client for Atlas introduced in ATLAS-3875 requires Python 3.6 or 
> above. It will help make the client available for Python 2.7 as well. Also, 
> the client should support Kerberos authentication.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-15 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222481
---




repository/src/main/java/org/apache/atlas/query/AtlasDSL.java
Line 138 (original), 135 (patched)


nit: consider avoiding double negation. Can be rewritten as:
  if (CollectionUtils.isEmpty(gremlinQueryComposer.getErrorList())) {
return;
  }



repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
Lines 65 (patched)


trLists and collected point to the same list object, and neither are 
reassigned later. Why is trLists needed?


- Madhan Neethiraj


On Jan. 15, 2021, 9:22 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 15, 2021, 9:22 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java ca0b4be91 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 87c8bd2ee 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 958c4281c 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 6c69855b7 
>   repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/10/
> 
> 
> Testing
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-15 Thread Ashutosh Mestry via Review Board



> On Jan. 14, 2021, 3:58 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
> > Lines 88 (patched)
> > 
> >
> > subTraversals could be an empty list - in iterations after #90, #99. 
> > Consider one of the following to address this:
> >  - move #58 inside for-loop at #60
> >  - update 'if' at #88 and #97 to check for empty list
> >if (subTraversals != null && !subTraversals.isEmpty())
> >  - set subTraversals to null in #90 and #99

I have addressed in a different way.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222472
---


On Jan. 15, 2021, 9:22 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 15, 2021, 9:22 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Configuration**
> _atlas.dsl.executor.traversal_ 
> - true: Uses the traversal-based implementation for query execution.
> - false: Uses the script-engine based implementation for query execution.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java ca0b4be91 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 87c8bd2ee 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java 
> PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 958c4281c 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 6c69855b7 
>   repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java 
> PRE-CREATION 
> 
> 
> Diff:

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

2021-01-15 Thread Ashutosh Mestry via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/
---

(Updated Jan. 15, 2021, 9:22 p.m.)


Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Additional refactoring.
- Additional unit tests to verify generated traversal.
- Refactored _GremlinQueryComposerTest_.


Bugs: ATLAS-2932
https://issues.apache.org/jira/browse/ATLAS-2932


Repository: atlas


Description
---

**Approach**
General Approach:
- Ensure that Gremlin-based implementation continues to function. This will 
ensure backward compatibility and will guarad against regressions or behavior 
deviations, if any, are introduced with the new implementation.
- Continue using existing _GremlinQueryComposer_. 
- Translate the clauses to _AtlasGraphTraversal_.
- Projects that were earlier handled in Gremlin query are now handled in a 
separate Java class.

Details:
- Modified: _GremlinQueryComposer_: Continue composing queries using existing 
claues. Additions alone.
- New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
- Modifed: _EntityDiscoveryService_ uses the new exectors.
- New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
_AtlasGraphTraversal_.
- Modified: _AtlasGraphTraversal_: Additional methods.
- New: _SelectClauseProjections_: All queries with select operations.

**Configuration**
_atlas.dsl.executor.traversal_ 
- true: Uses the traversal-based implementation for query execution.
- false: Uses the script-engine based implementation for query execution.

**Credits**
- Apoorv Naik (apoorvnaik): Original implementation.
- Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
short-comings.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
c016f6340 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
 881bb1e0f 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 0dd573b89 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
 c33c4f4d0 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
01a6c303a 
  repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
  repository/src/main/java/org/apache/atlas/query/DSLVisitor.java ca0b4be91 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 9704b0f9e 
  repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
9f30e4dc0 
  repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
87c8bd2ee 
  repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
969fcd2b1 
  
repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java 
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 958c4281c 
  repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
6c69855b7 
  repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/73128/diff/10/

Changes: https://reviews.apache.org/r/73128/diff/9-10/


Testing
---

**Unit Tests**
- Additional tests.
- _GremlinCompoerTest_ continues to validate the clause creation.
- Improved validation for _minMaxCount_ tests.
- Refactored: _TableValidator_ Validates output from a query with select claues.

**Volume Tests**
- Old implementation: 528 queries take over 10 mins to execute.
- New implemeentations: 528 queries execute in 7 secs.


Thanks,

Ashutosh Mestry

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

2021-01-15 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73119/#review222480
---



can you include some test coverage for this sorting functionality

- Sarath Subramanian


On Jan. 15, 2021, 9:33 a.m., Deep Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73119/
> ---
> 
> (Updated Jan. 15, 2021, 9:33 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4094
> https://issues.apache.org/jira/browse/ATLAS-4094
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Modified the existing API to take few new parameters. If new parameters are 
> provided, the result is genrated by in-memory sort.
> new parameters are offset, sortBy, and sortOrder. 
> 
> /api/atlas/v2/entity/{entity-guid}/audit?sortBy={user|actor|timestamp}={asc|desc}=10=20
> 
> HBase does not support query with sorted results. To support this API 
> inmemory sort has to be performed.
> Audit entry can potentially have entire entity dumped into it. Loading entire 
> audit entries for an entity can be memory intensive. Therefore we load audit 
> entries with limited columns first, perform sort on this light weight list, 
> then get the relevant section by removing offsets and reducing to limits. 
> With this reduced list we create MultiRowRangeFilter and then again scan the 
> table to get all the columns from the table this time.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 083acac73 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  8a453fd43 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
>  07784d1c4 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  9fca74470 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
>  900df0205 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
>  ef9e259ea 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 0d6d0c845 
> 
> 
> Diff: https://reviews.apache.org/r/73119/diff/4/
> 
> 
> Testing
> ---
> 
> Manual testing was done.
> 
> Did testing on a setup with 1 million audit entries spread across 1000 
> entities. 
> Existing Rest API took 6-12 milliseconds for preparing the result.
> In-memory sort and double scan approach took 55-75 milliseconds.
> Single Full scan and in-memory approach took 250-300 milliseconds.
> 
> As it was expected, the new API is 4X slower than the existing API therefore 
> the existing API still should be the primary API for querying audit events. 
> And the new API should be used only if sorting is required. Overall the 
> server response time of the new API is less than 80 millisecond, compared to 
> < 25 milliseconds for the existing API.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

2021-01-15 Thread Deep Singh

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73119/
---

(Updated Jan. 15, 2021, 5:33 p.m.)

Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath
Subramanian, and Sidharth Mishra.

Bugs: ATLAS-4094
https://issues.apache.org/jira/browse/ATLAS-4094

Repository: atlas

Description (updated)
---

Modified the existing API to take few new parameters. If new parameters are
provided, the result is genrated by in-memory sort.
new parameters are offset, sortBy, and sortOrder.

/api/atlas/v2/entity/{entity-guid}/audit?sortBy={user|actor|timestamp}={asc|desc}=10=20

HBase does not support query with sorted results. To support this API inmemory
sort has to be performed.
Audit entry can potentially have entire entity dumped into it. Loading entire
audit entries for an entity can be memory intensive. Therefore we load audit
entries with limited columns first, perform sort on this light weight list,
then get the relevant section by removing offsets and reducing to limits. With
this reduced list we create MultiRowRangeFilter and then again scan the table
to get all the columns from the table this time.

Diffs
-

intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
083acac73

repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
8a453fd43

repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
07784d1c4

repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
9fca74470

repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
900df0205

repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
ef9e259ea
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 0d6d0c845

Diff: https://reviews.apache.org/r/73119/diff/4/

Testing
---

Manual testing was done.

Did testing on a setup with 1 million audit entries spread across 1000
entities.
Existing Rest API took 6-12 milliseconds for preparing the result.
In-memory sort and double scan approach took 55-75 milliseconds.
Single Full scan and in-memory approach took 250-300 milliseconds.

As it was expected, the new API is 4X slower than the existing API therefore
the existing API still should be the primary API for querying audit events. And
the new API should be used only if sorting is required. Overall the server
response time of the new API is less than 80 millisecond, compared to < 25
milliseconds for the existing API.

Thanks,

Deep Singh

[jira] [Commented] (ATLAS-4101) 500 internal server error is thrown instead of 400 malformed request from atlas when the payload is malformed

2021-01-15 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/ATLAS-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266073#comment-17266073
 ] 

ASF subversion and git services commented on ATLAS-4101:


Commit 2559aa510883a0194a971d9b884835d419f8282c in atlas's branch 
refs/heads/branch-2.0 from Nixon Rodrigues
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=2559aa5 ]

ATLAS-4101 : Fix http error code for JsonParseException by overriding readFrom 
method of JacksonJsonProvider.

(cherry picked from commit 6edd4bc110a3fdde42659f797c588fa75432cc79)


> 500 internal server error is thrown instead of 400 malformed request from 
> atlas when the payload is malformed
> -
>
> Key: ATLAS-4101
> URL: https://issues.apache.org/jira/browse/ATLAS-4101
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
>
> {code:java|title=Malformed payload for creating a classification}
> {
> "classificationDefs": [
> {
> "name": "tag_1",
> }
> ]
> }
> {code}
> {code:java|title=Malformed payload for creating a classification}
> {
> "classificationDefs": [
> {
> "name": "tag_1"
> },
> ]
> }
> {code}
> There is an additional "," in the above 2 payloads, which make it a client 
> side payload error.
> But when firing such a request 500 internal server error is thrown
>  *These are client errors that are being thrown as server errors*
> {code:java}
> 2020-12-11 07:47:28,482 ERROR - [etp402695541-552 - 
> 6420db24-b42f-45eb-a6bb-25ffb3c8ce0e:] ~ Error handling a request: 
> 31167d3d0dc60f70 (ExceptionMapperUtil:32)2020-12-11 07:47:28,482 ERROR - 
> [etp402695541-552 - 6420db24-b42f-45eb-a6bb-25ffb3c8ce0e:] ~ Error handling a 
> request: 31167d3d0dc60f70 
> (ExceptionMapperUtil:32)com.fasterxml.jackson.databind.JsonMappingException: 
> Unexpected character (']' (code 93)): expected a value at [Source: 
> (org.eclipse.jetty.server.HttpInputOverHTTP); line: 6, column: 6] (through 
> reference chain: 
> org.apache.atlas.model.typedef.AtlasTypesDef["classificationDefs"]) at 
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:397)
>  at 
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:356)
>  at 
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1719)
>  at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:290)
>  at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
>  at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1682) 
> at 
> com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:977) 
> at 
> com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:814) 
> at 
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:490)
>  at 
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
>  at 
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:86)
>  at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
>  at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:166)
>  at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>  at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
>  at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>  at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>  at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>  at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>  at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
>  at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
>  at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
>  at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
>  at 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Review Request 73146: ATLAS-4077: replaced print statements in Python files with call to print(), for Python3

[jira] [Updated] (ATLAS-4077) update Python client to work with Kerberos and 2.7

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api

[jira] [Commented] (ATLAS-4101) 500 internal server error is thrown instead of 400 malformed request from atlas when the payload is malformed

10 matches

Site Navigation

Mail list logo

Footer information