Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'with' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search:** hive_table with "modernTrade@salesGlossary", hive_table with "modernTrade@salesGlossary" and hive_table isA Dimension Diffs - repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 249381059 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf Diff: https://reviews.apache.org/r/73058/diff/1/ Testing --- Added testcases in DSLQueriesTest.java Thanks, Pinal Shah
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- (Updated Dec. 3, 2020, 4:58 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'with' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search:** hive_table with "modernTrade@salesGlossary", hive_table with "modernTrade@salesGlossary" and hive_table isA Dimension Diffs - repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 249381059 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf Diff: https://reviews.apache.org/r/73058/diff/1/ Testing (updated) --- Added testcases in DSLQueriesTest.java Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/240/ Thanks, Pinal Shah
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- (Updated Dec. 3, 2020, 6:05 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- fixed test Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'with' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search:** hive_table with "modernTrade@salesGlossary", hive_table with "modernTrade@salesGlossary" and hive_table isA Dimension Diffs (updated) - repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 249381059 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 487cf2778 Diff: https://reviews.apache.org/r/73058/diff/2/ Changes: https://reviews.apache.org/r/73058/diff/1-2/ Testing --- Added testcases in DSLQueriesTest.java Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/240/ Thanks, Pinal Shah
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms
> On Dec. 3, 2020, 5:58 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
> > Lines 208 (patched)
> > <https://reviews.apache.org/r/73058/diff/2/?file=2242677#file2242677line217>
> >
> > This doesn't make senese.
this has been used in DSLVisitor#241 to modify query accordingly.
if it is non primitive (relationship or glossary), add attribute inside and()
condition, to avoid traversing back to the original vertex.
Example:
hive_table where db.name = "sales";
original query:
g.V().has("__typeName",
'hive_Table').out('__hive_table.db').has('Asset.__s_name',
eq('product_id')).dedup().in('__Table.db').dedup().limit(25).toList()
Current query:
g.V().has('__typeName',
'hive_table').and(__.out('__hive_table.db').has('Asset.__s_name',
eq('product_id')).dedup()).dedup().limit(25).toList()
> On Dec. 3, 2020, 5:58 p.m., Ashutosh Mestry wrote:
> > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
> > Line 102 (original), 102 (patched)
> > <https://reviews.apache.org/r/73058/diff/2/?file=2242687#file2242687line102>
> >
> > Can you please include a test that verifies the genreated gremlin query?
yes sure
> On Dec. 3, 2020, 5:58 p.m., Ashutosh Mestry wrote:
> > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
> > Line 190 (original), 190 (patched)
> > <https://reviews.apache.org/r/73058/diff/2/?file=2242687#file2242687line190>
> >
> > What is the reason for these queries changing?
Refer above comment
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73058/#review81
---
On Dec. 3, 2020, 6:05 a.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73058/
> ---
>
> (Updated Dec. 3, 2020, 6:05 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj,
> Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4057
> https://issues.apache.org/jira/browse/ATLAS-4057
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Supporting glossary term in advanced search, will allow user to search the
> entities which are assigned to particular glossary term.
>
>
> **WorkAround:**
>
> 'with' keyword is added to support glossary term.
>
> User needs to add fully qualifiedName of glossary term.
>
>
>
> **Example queries to search:**
>
> hive_table with "modernTrade@salesGlossary",
>
> hive_table with "modernTrade@salesGlossary" and hive_table isA Dimension
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java
> 6c48a82f4
> repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f
> repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
> 249381059
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4
> 15d1bb4cb
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java
> 142b9cac9
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4
> f1c10600c
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java
> 6b33edb53
>
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java
> 3139d4374
>
> repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java
> a1a727c82
> repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642
>
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> 1839e76c7
> repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
> 3bb3b07bf
>
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
> 487cf2778
>
>
> Diff: https://reviews.apache.org/r/73058/diff/4/
>
>
> Testing
> ---
>
> Added testcases in DSLQueriesTest.java
> Precommit :
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/245/
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- (Updated Dec. 4, 2020, 1:37 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- Addressed comments Replaced 'with' keyword with 'hasTerm' Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description (updated) --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'hasTerm' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search:** hive_table hasTerm "modernTrade@salesGlossary", hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension Diffs (updated) - repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 249381059 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 487cf2778 Diff: https://reviews.apache.org/r/73058/diff/5/ Changes: https://reviews.apache.org/r/73058/diff/4-5/ Testing --- Added testcases in DSLQueriesTest.java Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/245/ Thanks, Pinal Shah
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- (Updated Dec. 14, 2020, 9:56 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Summary (updated) - ATLAS-4057 : DSL Search : Support glossary terms and relationship Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description (updated) --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'hasTerm' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search with term:** hive_table hasTerm "modernTrade@salesGlossary", hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension **Example queries to search with relationship:** hive_table where db.name = "sales" Diffs (updated) - repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 249381059 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 487cf2778 Diff: https://reviews.apache.org/r/73058/diff/6/ Changes: https://reviews.apache.org/r/73058/diff/5-6/ Testing --- Added testcases in DSLQueriesTest.java Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/245/ Thanks, Pinal Shah
Review Request 73102: ATLAS-4088 : Quick Search: Unable to get Aggregation Metric when query text is empty
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73102/ --- Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4088 https://issues.apache.org/jira/browse/ATLAS-4088 Repository: atlas Description --- In case the query text is passed as empty string, Eg: **/api/atlas/v2/search/quick?typeName=hive_table&query=** The aggregation metrics in the response does not contain any information like owners, state of the asset, etc. Diffs - graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java 43114e9ca Diff: https://reviews.apache.org/r/73102/diff/1/ Testing --- manually tested Thanks, Pinal Shah
Review Request 73122: ATLAS-4095 : DSL Search : __isIncomplete System Attribute returns incorrect results or throws error
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73122/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4095 https://issues.apache.org/jira/browse/ATLAS-4095 Repository: atlas Description --- **Issue:** With DSL search having 'isIncomplete' System attribute in where clause, it returns empty results. **Example:** "Table where __isIncomplete = false" **Workaround:** Need to map boolean value to integer value before querying. Also need to handle check for the presence of attribute. Diffs - repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 249381059 repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 7b03d118b Diff: https://reviews.apache.org/r/73122/diff/1/ Testing --- Thanks, Pinal Shah
Re: Review Request 73122: ATLAS-4095 : DSL Search : __isIncomplete System Attribute returns incorrect results or throws error
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73122/ --- (Updated Jan. 7, 2021, 10:25 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4095 https://issues.apache.org/jira/browse/ATLAS-4095 Repository: atlas Description --- **Issue:** With DSL search having 'isIncomplete' System attribute in where clause, it returns empty results. **Example:** "Table where __isIncomplete = false" **Workaround:** Need to map boolean value to integer value before querying. Also need to handle check for the presence of attribute. Diffs - repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 249381059 repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 7b03d118b Diff: https://reviews.apache.org/r/73122/diff/2/ Testing (updated) --- https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/306/ Thanks, Pinal Shah
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- (Updated Jan. 12, 2021, 10:53 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- addressed - nit: imports Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'hasTerm' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search with term:** hive_table hasTerm "modernTrade@salesGlossary", hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension **Example queries to search with relationship:** hive_table where db.name = "sales" Diffs (updated) - repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 36b514e84 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 487cf2778 Diff: https://reviews.apache.org/r/73058/diff/7/ Changes: https://reviews.apache.org/r/73058/diff/6-7/ Testing --- Added testcases in DSLQueriesTest.java Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/ Thanks, Pinal Shah
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- (Updated Jan. 12, 2021, 11:54 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- added doc Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'hasTerm' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search with term:** hive_table hasTerm "modernTrade@salesGlossary", hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension **Example queries to search with relationship:** hive_table where db.name = "sales" Diffs (updated) - docs/src/documents/Search/SearchAdvanced.md 46be14278 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 36b514e84 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 487cf2778 Diff: https://reviews.apache.org/r/73058/diff/8/ Changes: https://reviews.apache.org/r/73058/diff/7-8/ Testing --- Added testcases in DSLQueriesTest.java Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/ Thanks, Pinal Shah
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
> On Jan. 12, 2021, 8:20 p.m., Sarath Subramanian wrote: > > docs/src/documents/Search/SearchAdvanced.md > > Lines 228 (patched) > > <https://reviews.apache.org/r/73058/diff/8/?file=2244142#file2244142line228> > > > > can we add a new section and examples for relationship search? okay, i will add new section specifically for relationship - Pinal --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/#review222445 --- On Jan. 12, 2021, 11:54 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73058/ > --- > > (Updated Jan. 12, 2021, 11:54 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4057 > https://issues.apache.org/jira/browse/ATLAS-4057 > > > Repository: atlas > > > Description > --- > > Supporting glossary term in advanced search, will allow user to search the > entities which are assigned to particular glossary term. > > > **WorkAround:** > > 'hasTerm' keyword is added to support glossary term. > > User needs to add fully qualifiedName of glossary term. > > > > **Example queries to search with term:** > > hive_table hasTerm "modernTrade@salesGlossary", > > hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension > > **Example queries to search with relationship:** > > hive_table where db.name = "sales" > > > Diffs > - > > docs/src/documents/Search/SearchAdvanced.md 46be14278 > > repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java > 6c48a82f4 > repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f > repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java > 36b514e84 > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 > 15d1bb4cb > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java > 142b9cac9 > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens > PRE-CREATION > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 > f1c10600c > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java > 6b33edb53 > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java > 3139d4374 > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java > a1a727c82 > repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 1839e76c7 > repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java > 3bb3b07bf > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > 487cf2778 > > > Diff: https://reviews.apache.org/r/73058/diff/8/ > > > Testing > --- > > Added testcases in DSLQueriesTest.java > Precommit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/ > > > Thanks, > > Pinal Shah > >
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- (Updated Jan. 13, 2021, 12:31 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- added documentation Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'hasTerm' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search with term:** hive_table hasTerm "modernTrade@salesGlossary", hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension **Example queries to search with relationship:** hive_table where db.name = "sales" Diffs (updated) - docs/src/documents/Search/SearchAdvanced.md 46be14278 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 36b514e84 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 487cf2778 Diff: https://reviews.apache.org/r/73058/diff/9/ Changes: https://reviews.apache.org/r/73058/diff/8-9/ Testing --- Added testcases in DSLQueriesTest.java Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/ Thanks, Pinal Shah
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/ --- (Updated Jan. 14, 2021, 6:40 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- rebase trailing whitespaces Bugs: ATLAS-4057 https://issues.apache.org/jira/browse/ATLAS-4057 Repository: atlas Description --- Supporting glossary term in advanced search, will allow user to search the entities which are assigned to particular glossary term. **WorkAround:** 'hasTerm' keyword is added to support glossary term. User needs to add fully qualifiedName of glossary term. **Example queries to search with term:** hive_table hasTerm "modernTrade@salesGlossary", hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension **Example queries to search with relationship:** hive_table where db.name = "sales" Diffs (updated) - docs/src/documents/Search/SearchAdvanced.md 46be14278 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 6c48a82f4 repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 36b514e84 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 15d1bb4cb repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 142b9cac9 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 f1c10600c repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 6b33edb53 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java 3139d4374 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java PRE-CREATION repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java a1a727c82 repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 1839e76c7 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3bb3b07bf repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 487cf2778 Diff: https://reviews.apache.org/r/73058/diff/10/ Changes: https://reviews.apache.org/r/73058/diff/9-10/ Testing --- Added testcases in DSLQueriesTest.java Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/ Thanks, Pinal Shah
Review Request 73160: ATLAS-4112 : Basic Search : Attribute search of QualifiedName beginswith operator not returning results when the value ends with a digit+dot
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73160/ --- Review request for atlas and Sarath Subramanian. Bugs: ATLAS-4112 https://issues.apache.org/jira/browse/ATLAS-4112 Repository: atlas Description --- Problem: While searching with wildcard search i.e contains, beginsWith, endsWith operator, which has escape characters in search attribute value. Diffs - intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 050980994 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 01daf53d2 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 0da60d37f Diff: https://reviews.apache.org/r/73160/diff/1/ Testing --- Added unit test cases Thanks, Pinal Shah
Re: Review Request 73160: ATLAS-4112 : Basic Search : Attribute search of QualifiedName beginswith operator not returning results when the value ends with a digit+dot
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73160/ --- (Updated Jan. 27, 2021, 11:27 a.m.) Review request for atlas and Sarath Subramanian. Changes --- removed freetext search processor test cases Bugs: ATLAS-4112 https://issues.apache.org/jira/browse/ATLAS-4112 Repository: atlas Description (updated) --- **Problem:** While searching with wildcard search i.e contains, beginsWith, endsWith operator, which has escape characters in search attribute value gives incorrext results **Example:** While searching qualifiedName beginsWith "default.datagen_table_sensitive_7251." **Solution:** Remove quotes and wildcard '*' from query Diffs (updated) - intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 050980994 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 01daf53d2 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 0da60d37f Diff: https://reviews.apache.org/r/73160/diff/2/ Changes: https://reviews.apache.org/r/73160/diff/1-2/ Testing --- Added unit test cases Thanks, Pinal Shah
Re: Review Request 73160: ATLAS-4112 : Basic Search : Incorrect results when searching with value having special char and contains/beginsWith/endsWith operator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73160/ --- (Updated Jan. 28, 2021, 4:56 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- Precommit link Summary (updated) - ATLAS-4112 : Basic Search : Incorrect results when searching with value having special char and contains/beginsWith/endsWith operator Bugs: ATLAS-4112 https://issues.apache.org/jira/browse/ATLAS-4112 Repository: atlas Description (updated) --- **Problem:** While searching with wildcard search i.e contains, beginsWith, endsWith operator, which has escape characters in search attribute value gives incorrect results **Example:** While searching qualifiedName beginsWith "default.datagen_table_sensitive_725_1." **Solution:** Remove quotes and wildcard '*' from query **Steps to reproduce:** 1. Create a hive_table with name ends with a digit: datagen_table_sensitive_725_1 2. Search By Type: hive_columns and select attribute filter: qualifiedName as "default.datagen_table_sensitive_725_1." shows no Records found but if qualifiedName is changed to "default.datagen_table_sensitive_725_1" it shows results with qualifiedName of column "default.datagen_table_sensitive_725_1.col1@cm" Diffs - intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 050980994 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 01daf53d2 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 0da60d37f Diff: https://reviews.apache.org/r/73160/diff/2/ Testing (updated) --- Added unit test cases Precommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/353/console Thanks, Pinal Shah
Re: Review Request 73166: Clause Change for Glossary
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73166/#review222545 --- repository/src/main/java/org/apache/atlas/query/GremlinClause.java Line 72 (original), 72 (patched) <https://reviews.apache.org/r/73166/#comment311646> Purpose of adding 'and(.in)' was to avoid traversing to the edge and then end vertex - Pinal Shah On Feb. 1, 2021, 5:20 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73166/ > --- > > (Updated Feb. 1, 2021, 5:20 p.m.) > > > Review request for atlas, Nikhil Bonte, Pinal Shah, and Sarath Subramanian. > > > Bugs: ATLAS-4109 > https://issues.apache.org/jira/browse/ATLAS-4109 > > > Repository: atlas > > > Description > --- > > Replaced _or_ with _where_ for simplicity. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/query/GremlinClause.java > b7e5330ce > > repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java > a7b05fd88 > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > 0e8af186d > > > Diff: https://reviews.apache.org/r/73166/diff/1/ > > > Testing > --- > > **Unit tests** > - Existing tests run. > - Updated _GremlinQueryComposer_ to include the new clause. > > **PC-build** > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/363/ > > > Thanks, > > Ashutosh Mestry > >
Re: Review Request 73160: ATLAS-4112 : Basic Search : Incorrect results when searching with value having special char and contains/beginsWith/endsWith operator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73160/ --- (Updated Feb. 3, 2021, 4:16 p.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- refactored Bugs: ATLAS-4112 https://issues.apache.org/jira/browse/ATLAS-4112 Repository: atlas Description --- **Problem:** While searching with wildcard search i.e contains, beginsWith, endsWith operator, which has escape characters in search attribute value gives incorrect results **Example:** While searching qualifiedName beginsWith "default.datagen_table_sensitive_725_1." **Solution:** Remove quotes and wildcard '*' from query **Steps to reproduce:** 1. Create a hive_table with name ends with a digit: datagen_table_sensitive_725_1 2. Search By Type: hive_columns and select attribute filter: qualifiedName as "default.datagen_table_sensitive_725_1." shows no Records found but if qualifiedName is changed to "default.datagen_table_sensitive_725_1" it shows results with qualifiedName of column "default.datagen_table_sensitive_725_1.col1@cm" Diffs (updated) - intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 050980994 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 01daf53d2 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 0da60d37f Diff: https://reviews.apache.org/r/73160/diff/3/ Changes: https://reviews.apache.org/r/73160/diff/2-3/ Testing --- Added unit test cases Precommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/353/console Thanks, Pinal Shah
Re: Review Request 73166: Clause Change for Glossary
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73166/#review222578 --- Ship it! Ship It! - Pinal Shah On Feb. 3, 2021, 9:46 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73166/ > --- > > (Updated Feb. 3, 2021, 9:46 p.m.) > > > Review request for atlas, Nikhil Bonte, Pinal Shah, and Sarath Subramanian. > > > Bugs: ATLAS-4109 > https://issues.apache.org/jira/browse/ATLAS-4109 > > > Repository: atlas > > > Description > --- > > Replaced _or_ with _where_ for simplicity. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/query/GremlinClause.java > b7e5330ce > > repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java > 156a65ccb > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > 3594e2f34 > > > Diff: https://reviews.apache.org/r/73166/diff/2/ > > > Testing > --- > > **Unit tests** > - Existing tests run. > - Updated _GremlinQueryComposer_ to include the new clause. > > **PC-build** > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/363/ > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/366/ > > > Thanks, > > Ashutosh Mestry > >
Review Request 73211: ATLAS-4182 : Quick Search: Incorrect aggregation Metric when less than operator is selected to filter
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73211/
---
Review request for atlas, Jayendra Parab, Nixon Rodrigues, and Sarath
Subramanian.
Bugs: ATLAS-4182
https://issues.apache.org/jira/browse/ATLAS-4182
Repository: atlas
Description
---
**Issue:** The aggregation metrics for POST /api/atlas/v2/search/quick API
returned the count of all entities (of a selected type) in case the createTime
< '{value}' filter is applied which results in no entities for the given search
range.
**Example payload:**
{"assetProvider":"ATLAS","excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"tagFilters":null,"attributes":["qualifiedName","createTime","db","parameters"],"query":"","typeName":"hive_table","entityFilters":{"condition":"AND","criterion":[{"condition":"AND","criterion":[
{"attributeName":"createTime","operator":"lt","attributeValue":"1613973469738"}
]}]},"classification":null,"childClassification":null,"limit":100,"offset":0}
Diffs
-
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java
5712fd3d0
Diff: https://reviews.apache.org/r/73211/diff/1/
Testing
---
Manually tested
Thanks,
Pinal Shah
Re: Review Request 73233: ATLAS-4211 : DSL Search : Regression: When fetching string attribute with null value returns null , earlier versions returned empty
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73233/ --- (Updated March 17, 2021, 5:18 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4211 https://issues.apache.org/jira/browse/ATLAS-4211 Repository: atlas Description --- For TraversalBasedExecutor which is default executor, values:[[""]] is replaced to values: [[null]] whenever there is null value for any of attributes in Select Clause Diffs - repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java 7018c1578 Diff: https://reviews.apache.org/r/73233/diff/1/ Testing (updated) --- Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/456/console Thanks, Pinal Shah
Re: Review Request 73233: ATLAS-4211 : DSL Search : Regression: When fetching string attribute with null value returns null , earlier versions returned empty
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73233/ --- (Updated March 18, 2021, 3:52 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Changes --- addressed review comment Bugs: ATLAS-4211 https://issues.apache.org/jira/browse/ATLAS-4211 Repository: atlas Description --- For TraversalBasedExecutor which is default executor, values:[[""]] is replaced to values: [[null]] whenever there is null value for any of attributes in Select Clause Diffs (updated) - repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java 7018c1578 Diff: https://reviews.apache.org/r/73233/diff/2/ Changes: https://reviews.apache.org/r/73233/diff/1-2/ Testing --- Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/456/console Thanks, Pinal Shah
Review Request 73248: ATLAS-4219 : Basic Search : Search with Glossary term returning approximate count as -1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73248/ --- Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4219 https://issues.apache.org/jira/browse/ATLAS-4219 Repository: atlas Description --- Searching with Glossary term returns approximate count as -1 in response. Without Term it returns proper approximate count. Diffs - repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java d9db9bf8b Diff: https://reviews.apache.org/r/73248/diff/1/ Testing --- Approximate count will be same regardless of any filters or limit/offset Precommit : Thanks, Pinal Shah
Re: Review Request 73248: ATLAS-4219 : Basic Search : Search with Glossary term returning approximate count as -1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73248/ --- (Updated March 22, 2021, 6:04 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4219 https://issues.apache.org/jira/browse/ATLAS-4219 Repository: atlas Description --- Searching with Glossary term returns approximate count as -1 in response. Without Term it returns proper approximate count. Diffs - repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java d9db9bf8b Diff: https://reviews.apache.org/r/73248/diff/1/ Testing (updated) --- Approximate count will be same regardless of any filters or limit/offset Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/462/console Thanks, Pinal Shah
Review Request 73271: ATLAS-4238 : Quick Search: handle special characters to get correct results
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73271/
---
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues,
and Sarath Subramanian.
Bugs: ATLAS-4238
https://issues.apache.org/jira/browse/ATLAS-4238
Repository: atlas
Description
---
**Current working:** During freetext/quick search, wildcard '*' is appended
after the search string from UI.
Whenever attribute search value has "**+-&|!(){}[]^"~?:**" special characters,
then '*' will not be appended to get the correct result.
('*' is removed because for TEXT type attribute, it tokenize the value on
special characters)
**Issue:** '@' or 'digit+.' etc there are many other special characters which
gets tokenize for 'TEXT' type attribute. hence if the search value has these
special characters which are not added in above list, it doesn't give results.
**Solution:** We need to handle all the special characters which tokenize the
string
Diffs
-
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
4ae2d1279
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
9846d4310
Diff: https://reviews.apache.org/r/73271/diff/1/
Testing
---
added test cases
Precommit:
Thanks,
Pinal Shah
Re: Review Request 73271: ATLAS-4238 : Quick Search: handle special characters to get correct results
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73271/
---
(Updated April 8, 2021, 4:13 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues,
and Sarath Subramanian.
Changes
---
added Precommit
Bugs: ATLAS-4238
https://issues.apache.org/jira/browse/ATLAS-4238
Repository: atlas
Description
---
**Current working:** During freetext/quick search, wildcard '*' is appended
after the search string from UI.
Whenever attribute search value has "**+-&|!(){}[]^"~?:**" special characters,
then '*' will not be appended to get the correct result.
('*' is removed because for TEXT type attribute, it tokenize the value on
special characters)
**Issue:** '@' or 'digit+.' etc there are many other special characters which
gets tokenize for 'TEXT' type attribute. hence if the search value has these
special characters which are not added in above list, it doesn't give results.
**Solution:** We need to handle all the special characters which tokenize the
string
Diffs
-
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
4ae2d1279
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
9846d4310
Diff: https://reviews.apache.org/r/73271/diff/1/
Testing (updated)
---
added test cases
Precommit:
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/487/console
Thanks,
Pinal Shah
Re: Review Request 73172: ATLAS-4124 Restrict user from deleting Business Metadata, if Business Metadata is already associated with an entity
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73172/#review222795
---
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
Lines 398 (patched)
<https://reviews.apache.org/r/73172/#comment311972>
I think there is no need to maintain Map
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
Lines 410 (patched)
<https://reviews.apache.org/r/73172/#comment311971>
Can you move this inside for loop of attributeDefs? #406
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
Lines 420 (patched)
<https://reviews.apache.org/r/73172/#comment311969>
Replace ',' with SearchContext.TYPENAME_DELIMITER
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
Lines 432 (patched)
<https://reviews.apache.org/r/73172/#comment311970>
can you add blank line above
- Pinal Shah
On April 8, 2021, 12:13 p.m., Mandar Ambawane wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73172/
> ---
>
> (Updated April 8, 2021, 12:13 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues,
> Pinal Shah, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4124
> https://issues.apache.org/jira/browse/ATLAS-4124
>
>
> Repository: atlas
>
>
> Description
> ---
>
> We can delete the entire BusinessMetadata TypeDef using the curl command.
>
> But that will affect the entities which already have Business Attributes from
> this Deleted Business Metadata.
> Those Business Attributes will be removed from the Entity.
>
> This patch will restrict user from deleting Business Metadata, if Business
> Metadata is already associated with an entity.
>
> This patch adds support for Multi-Valued Attribute NOT NULL search in Basic
> Serch.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
> 367c5b8
> intg/src/test/java/org/apache/atlas/TestUtilsV2.java 80ab6b4
> repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
> 275fc78
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
> d0d54af
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
> 96fb3b4
> webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java
> PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/73172/diff/5/
>
>
> Testing
> ---
>
> Test cases added
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 73172: ATLAS-4124 Restrict user from deleting Business Metadata, if Business Metadata is already associated with an entity
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73172/#review222804 --- repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java Lines 398 (patched) <https://reviews.apache.org/r/73172/#comment311986> replace this.typeRegistry.getBusinessMetadataDefByName(typeName); to typeRegistry.getBusinessMetadataDefByName(typeName); Can you please review of 'this' is needed repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java Lines 403 (patched) <https://reviews.apache.org/r/73172/#comment311985> String qualifiedName= AtlasStructType.AtlasAttribute.getQualifiedAttributeName(businessMetadataDef, attributeDef.getName()); String vertexPropertyName = AtlasStructType.AtlasAttribute.generateVertexPropertyName(businessMetadataDef, attributeDef, qualifiedName); Set applicableTypes = AtlasJson.fromJson(attributeDef.getOption(AtlasBusinessMetadataDef.ATTR_OPTION_APPLICABLE_ENTITY_TYPES), Set.class); Can you align like above - Pinal Shah On April 10, 2021, 8:29 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73172/ > --- > > (Updated April 10, 2021, 8:29 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > Pinal Shah, and Sarath Subramanian. > > > Bugs: ATLAS-4124 > https://issues.apache.org/jira/browse/ATLAS-4124 > > > Repository: atlas > > > Description > --- > > We can delete the entire BusinessMetadata TypeDef using the curl command. > > But that will affect the entities which already have Business Attributes from > this Deleted Business Metadata. > Those Business Attributes will be removed from the Entity. > > This patch will restrict user from deleting Business Metadata, if Business > Metadata is already associated with an entity. > > This patch adds support for Multi-Valued Attribute NOT NULL search in Basic > Serch. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > 367c5b8 > intg/src/test/java/org/apache/atlas/TestUtilsV2.java 80ab6b4 > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > 275fc78 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java > d0d54af > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java > 96fb3b4 > webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/73172/diff/7/ > > > Testing > --- > > Test cases added > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 73172: ATLAS-4124 Restrict user from deleting Business Metadata, if Business Metadata is already associated with an entity
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73172/#review222805 --- Ship it! Ship It! - Pinal Shah On April 10, 2021, 8:29 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73172/ > --- > > (Updated April 10, 2021, 8:29 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > Pinal Shah, and Sarath Subramanian. > > > Bugs: ATLAS-4124 > https://issues.apache.org/jira/browse/ATLAS-4124 > > > Repository: atlas > > > Description > --- > > We can delete the entire BusinessMetadata TypeDef using the curl command. > > But that will affect the entities which already have Business Attributes from > this Deleted Business Metadata. > Those Business Attributes will be removed from the Entity. > > This patch will restrict user from deleting Business Metadata, if Business > Metadata is already associated with an entity. > > This patch adds support for Multi-Valued Attribute NOT NULL search in Basic > Serch. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > 367c5b8 > intg/src/test/java/org/apache/atlas/TestUtilsV2.java 80ab6b4 > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > 275fc78 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java > d0d54af > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java > 96fb3b4 > webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/73172/diff/7/ > > > Testing > --- > > Test cases added > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 73271: ATLAS-4238 : Quick Search: handle special characters to get correct results
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73271/
---
(Updated April 14, 2021, 4:10 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues,
and Sarath Subramanian.
Changes
---
added formatting
Bugs: ATLAS-4238
https://issues.apache.org/jira/browse/ATLAS-4238
Repository: atlas
Description (updated)
---
**Current working:** During freetext/quick search, wildcard '*' is appended
after the search string from UI.
Whenever attribute search value has "**+-&|!(){}[]^"~?:**" special characters,
then '*' will not be appended to get the correct result.
('*' is removed because for TEXT type attribute, it tokenize the value on
special characters)
**Issue:** '@' or 'digit+.' etc there are many other special characters which
gets tokenize for 'TEXT' type attribute. hence if the search value has these
special characters which are not added in above list, it doesn't give results.
**Solution:** We need to handle all the special characters which tokenize the
string
**Note:** This patch doesn't handle escaping special characters when search as
part of freetext search
- Whenever user tends to search for **'STRING'** type attribute example: 'name'
i. **Client has to escape 'space'**
ii.**For the partial search, client has to append '*' **
Example: to search string with name = sales department@hdfc, user has to
search like sales\ department*
- Whenever user tends to search for **'TEXT'** type attribute example:
'qualifiedName'
i. **No Need of escape 'space'**. If it is escaped partial search will not
work
ii.**For partial search, no need of appending '*' **
Example: to search string with qualifiedName = sales department@hdfc,
user has to search like sales department or sales\ department@hdfc
Diffs (updated)
-
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
4ae2d1279
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
9846d4310
Diff: https://reviews.apache.org/r/73271/diff/2/
Changes: https://reviews.apache.org/r/73271/diff/1-2/
Testing
---
added test cases
Precommit:
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/487/console
Thanks,
Pinal Shah
Re: Review Request 73271: ATLAS-4238 : Quick Search: handle special characters to get correct results
> On April 13, 2021, 11:22 p.m., Sidharth Mishra wrote:
> > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> > Lines 614 (patched)
> > <https://reviews.apache.org/r/73271/diff/1/?file=2248494#file2248494line614>
> >
> > Please let me know in that case how the "search_name_with\ nameblank*"
> > retunring one result where as "search_qf_with\ qfblank*" returning 0 result.
Can you refer 'note' in the description
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73271/#review222816
---
On April 14, 2021, 4:10 a.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73271/
> ---
>
> (Updated April 14, 2021, 4:10 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues,
> and Sarath Subramanian.
>
>
> Bugs: ATLAS-4238
> https://issues.apache.org/jira/browse/ATLAS-4238
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Current working:** During freetext/quick search, wildcard '*' is appended
> after the search string from UI.
> Whenever attribute search value has "**+-&|!(){}[]^"~?:**" special
> characters, then '*' will not be appended to get the correct result.
>
> ('*' is removed because for TEXT type attribute, it tokenize the value on
> special characters)
>
> **Issue:** '@' or 'digit+.' etc there are many other special characters which
> gets tokenize for 'TEXT' type attribute. hence if the search value has these
> special characters which are not added in above list, it doesn't give results.
>
> **Solution:** We need to handle all the special characters which tokenize the
> string
>
> **Note:** This patch doesn't handle escaping special characters when search
> as part of freetext search
>
> - Whenever user tends to search for **'STRING'** type attribute example:
> 'name'
>i. **Client has to escape 'space'**
>ii.**For the partial search, client has to append '*' **
>
>Example: to search string with name = sales department@hdfc, user has
> to search like sales\ department*
>
> - Whenever user tends to search for **'TEXT'** type attribute example:
> 'qualifiedName'
>i. **No Need of escape 'space'**. If it is escaped partial search will
> not work
>ii.**For partial search, no need of appending '*' **
>
>Example: to search string with qualifiedName = sales department@hdfc,
> user has to search like sales department or sales\ department@hdfc
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 4ae2d1279
>
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> 9846d4310
>
>
> Diff: https://reviews.apache.org/r/73271/diff/2/
>
>
> Testing
> ---
>
> added test cases
> Precommit:
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/487/console
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 73271: ATLAS-4238 : Quick Search: handle special characters to get correct results
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73271/
---
(Updated April 14, 2021, 6:40 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues,
and Sarath Subramanian.
Changes
---
addressed comments
Bugs: ATLAS-4238
https://issues.apache.org/jira/browse/ATLAS-4238
Repository: atlas
Description
---
**Current working:** During freetext/quick search, wildcard '*' is appended
after the search string from UI.
Whenever attribute search value has "**+-&|!(){}[]^"~?:**" special characters,
then '*' will not be appended to get the correct result.
('*' is removed because for TEXT type attribute, it tokenize the value on
special characters)
**Issue:** '@' or 'digit+.' etc there are many other special characters which
gets tokenize for 'TEXT' type attribute. hence if the search value has these
special characters which are not added in above list, it doesn't give results.
**Solution:** We need to handle all the special characters which tokenize the
string
**Note:** This patch doesn't handle escaping special characters when search as
part of freetext search
- Whenever user tends to search for **'STRING'** type attribute example: 'name'
i. **Client has to escape 'space'**
ii.**For the partial search, client has to append '*' **
Example: to search string with name = sales department@hdfc, user has to
search like sales\ department*
- Whenever user tends to search for **'TEXT'** type attribute example:
'qualifiedName'
i. **No Need of escape 'space'**. If it is escaped partial search will not
work
ii.**For partial search, no need of appending '*' **
Example: to search string with qualifiedName = sales department@hdfc,
user has to search like sales department or sales\ department@hdfc
Diffs (updated)
-
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
4ae2d1279
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
9846d4310
Diff: https://reviews.apache.org/r/73271/diff/3/
Changes: https://reviews.apache.org/r/73271/diff/2-3/
Testing
---
added test cases
Precommit:
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/487/console
Thanks,
Pinal Shah
Re: Review Request 73271: ATLAS-4238 : Quick Search: handle special characters to get correct results
> On April 14, 2021, 5:18 a.m., Sidharth Mishra wrote:
> > The precommit has failed. Please chck before merge.
it failed at Hive-bridge module, which is not related to this patch
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73271/#review222818
---
On April 14, 2021, 6:40 a.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73271/
> ---
>
> (Updated April 14, 2021, 6:40 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues,
> and Sarath Subramanian.
>
>
> Bugs: ATLAS-4238
> https://issues.apache.org/jira/browse/ATLAS-4238
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Current working:** During freetext/quick search, wildcard '*' is appended
> after the search string from UI.
> Whenever attribute search value has "**+-&|!(){}[]^"~?:**" special
> characters, then '*' will not be appended to get the correct result.
>
> ('*' is removed because for TEXT type attribute, it tokenize the value on
> special characters)
>
> **Issue:** '@' or 'digit+.' etc there are many other special characters which
> gets tokenize for 'TEXT' type attribute. hence if the search value has these
> special characters which are not added in above list, it doesn't give results.
>
> **Solution:** We need to handle all the special characters which tokenize the
> string
>
> **Note:** This patch doesn't handle escaping special characters when search
> as part of freetext search
>
> - Whenever user tends to search for **'STRING'** type attribute example:
> 'name'
>i. **Client has to escape 'space'**
>ii.**For the partial search, client has to append '*' **
>
>Example: to search string with name = sales department@hdfc, user has
> to search like sales\ department*
>
> - Whenever user tends to search for **'TEXT'** type attribute example:
> 'qualifiedName'
>i. **No Need of escape 'space'**. If it is escaped partial search will
> not work
>ii.**For partial search, no need of appending '*' **
>
>Example: to search string with qualifiedName = sales department@hdfc,
> user has to search like sales department or sales\ department@hdfc
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 4ae2d1279
>
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> 9846d4310
>
>
> Diff: https://reviews.apache.org/r/73271/diff/3/
>
>
> Testing
> ---
>
> added test cases
> Precommit:
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/487/console
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 72644: ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72644/#review222823 --- Ship it! Ship It! - Pinal Shah On April 8, 2021, 1:01 p.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72644/ > --- > > (Updated April 8, 2021, 1:01 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3872 > https://issues.apache.org/jira/browse/ATLAS-3872 > > > Repository: atlas > > > Description > --- > > Cause of the problem: > Type-def Column is created by quick-start script. > > Here in this case, Type-def Column has the attribute "name". And it has > property isOptional = true > > Type-def Column is a child of Type-def Asset which also has the attribute > "name". And it has property isOptional = false > > Now while creating Entity of Type-def Column from UI, it does not show > attribute "name" in the mandatory fields because for Type-def Column its > property isOptional = true > > But while validating the attributes from server side we iterate through all > the attributes of Entity type (Including Parent's attributes) > In this case, while validating Type-def Asset's attribute "name" (which has > property isOptional = false) we get an error because its value is sent as > "null" from UI side. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 773fae2 > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 22259bc > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 27c7f73 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java d89aca2 > intg/src/test/java/org/apache/atlas/TestUtilsV2.java 80ab6b4 > intg/src/test/java/org/apache/atlas/model/ModelTestUtil.java 5df9525 > repository/src/test/resources/logging-v1-full.zip > 69c54eec000964a1f6f3fef96b26cca188a28ba1 > repository/src/test/resources/sales-v1-full.zip > 07afbf6ecae312a840b54cae5ab01335c513d180 > repository/src/test/resources/salesNewTypeAttrs-next.zip > e4c8505be1587fb7cc8b6354bdda701e9557da99 > repository/src/test/resources/salesNewTypeAttrs.zip > a4467e641fffb10189288fb2f26dc4fbc06bbd42 > webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 82a9f5c > webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java d5cf9b7 > webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java > 6db6b58 > > webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java > 0cd707f > > webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java > 7c820e7 > > > Diff: https://reviews.apache.org/r/72644/diff/7/ > > > Testing > --- > > PreCommit: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2007/console > > PreCommit: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2014/console > > PreCommit: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2083/console > > PreCommit: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2088/console > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 72644: ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name
> On April 14, 2021, 11:28 a.m., Pinal Shah wrote: > > Ship It! Need to trigger Precommit - Pinal --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72644/#review222823 --- On April 8, 2021, 1:01 p.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72644/ > --- > > (Updated April 8, 2021, 1:01 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3872 > https://issues.apache.org/jira/browse/ATLAS-3872 > > > Repository: atlas > > > Description > --- > > Cause of the problem: > Type-def Column is created by quick-start script. > > Here in this case, Type-def Column has the attribute "name". And it has > property isOptional = true > > Type-def Column is a child of Type-def Asset which also has the attribute > "name". And it has property isOptional = false > > Now while creating Entity of Type-def Column from UI, it does not show > attribute "name" in the mandatory fields because for Type-def Column its > property isOptional = true > > But while validating the attributes from server side we iterate through all > the attributes of Entity type (Including Parent's attributes) > In this case, while validating Type-def Asset's attribute "name" (which has > property isOptional = false) we get an error because its value is sent as > "null" from UI side. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 773fae2 > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 22259bc > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 27c7f73 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java d89aca2 > intg/src/test/java/org/apache/atlas/TestUtilsV2.java 80ab6b4 > intg/src/test/java/org/apache/atlas/model/ModelTestUtil.java 5df9525 > repository/src/test/resources/logging-v1-full.zip > 69c54eec000964a1f6f3fef96b26cca188a28ba1 > repository/src/test/resources/sales-v1-full.zip > 07afbf6ecae312a840b54cae5ab01335c513d180 > repository/src/test/resources/salesNewTypeAttrs-next.zip > e4c8505be1587fb7cc8b6354bdda701e9557da99 > repository/src/test/resources/salesNewTypeAttrs.zip > a4467e641fffb10189288fb2f26dc4fbc06bbd42 > webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 82a9f5c > webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java d5cf9b7 > webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java > 6db6b58 > > webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java > 0cd707f > > webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java > 7c820e7 > > > Diff: https://reviews.apache.org/r/72644/diff/7/ > > > Testing > --- > > PreCommit: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2007/console > > PreCommit: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2014/console > > PreCommit: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2083/console > > PreCommit: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2088/console > > > Thanks, > > Mandar Ambawane > >
Review Request 73292: ATLAS-4254 : Basic Search : Optimize pagination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Bugs: ATLAS-4254 https://issues.apache.org/jira/browse/ATLAS-4254 Repository: atlas Description --- Details in JIRA description Diffs - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7 intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4ae2d1279 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 3750799f7 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java c2778063c webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 8aec3a6d5 Diff: https://reviews.apache.org/r/73292/diff/1/ Testing --- MAnual testing WIP unit testcases Thanks, Pinal Shah
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize pagination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/ --- (Updated April 22, 2021, 3:01 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Changes --- added unit testcases Bugs: ATLAS-4254 https://issues.apache.org/jira/browse/ATLAS-4254 Repository: atlas Description --- Details in JIRA description Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7 intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4ae2d1279 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 3750799f7 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java c2778063c repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 8aec3a6d5 Diff: https://reviews.apache.org/r/73292/diff/2/ Changes: https://reviews.apache.org/r/73292/diff/1-2/ Testing (updated) --- MAnual testing unit testcases Thanks, Pinal Shah
Re: Review Request 73284: ATLAS-4247: "Advanced Search" is spelt as "Advance Search" in Atlas Beta UI
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73284/#review222852 --- Ship it! Ship It! - Pinal Shah On April 16, 2021, 1:41 p.m., Umesh Padashetty wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73284/ > --- > > (Updated April 16, 2021, 1:41 p.m.) > > > Review request for atlas, Madhan Neethiraj, Prasad Pawar, Sameer Shaikh, and > Sarath Subramanian. > > > Bugs: ATLAS-4247 > https://issues.apache.org/jira/browse/ATLAS-4247 > > > Repository: atlas > > > Description > --- > > In classic UI, we have/always had Advanced Search > > > Diffs > - > > dashboardv3/public/js/views/search/tree/CustomFilterTreeLayoutView.js > ac7002458 > docs/docz-lib/config/menu.js 7c2fcdb21 > docs/src/documents/Search/SearchAdvanced.md 872c2be81 > > > Diff: https://reviews.apache.org/r/73284/diff/1/ > > > Testing > --- > > > Thanks, > > Umesh Padashetty > >
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize pagination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/ --- (Updated April 26, 2021, 3:55 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Bugs: ATLAS-4254 https://issues.apache.org/jira/browse/ATLAS-4254 Repository: atlas Description (updated) --- **Current Working of Pagination** - When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0 - Query is then executed from 0 offset and limit - Filter out the results obtained from query - Generate result set by skipping the results from 0 to the given offset **New Working Structure** - New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value. - The marker should be set as '*' for the start of the page - The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request - if marker is set then offset should not be greater than 0 **Perfomance details** Took usecase of ranger tagsync: - On searching tagged entities starting with offset: 0 and limit: 1000 returned 1 entities in ~ 21s - On searching tagged entities starting with marker: * and limit: 1000 returned 1 entities in ~ 10s (both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers) - Around 51% improvement - Improvement is seen as the page size increases. WIP, need to solve code redundancy Can we create V2 class for SearchProcessors? Diffs - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7 intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4ae2d1279 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 3750799f7 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java c2778063c repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 8aec3a6d5 Diff: https://reviews.apache.org/r/73292/diff/2/ Testing --- MAnual testing unit testcases Thanks, Pinal Shah
Re: Review Request 73249: ATLAS-4221 : Solr collection names should be configurable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73249/#review222873 --- distro/src/bin/atlas_start.py Line 135 (original), 138 (patched) <https://reviews.apache.org/r/73249/#comment312044> What happens if the collection name, updated in properties already exists in solr Can we restrict this from not allowing to point old collection? repository/src/test/java/org/apache/atlas/TestSolrConfigurableNames.java Lines 43 (patched) <https://reviews.apache.org/r/73249/#comment312046> Can you set collection names here? repository/src/test/resources/atlas-application.properties Lines 1 (patched) <https://reviews.apache.org/r/73249/#comment312047> Can you please review if this is required? - Pinal Shah On April 26, 2021, 5:41 a.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73249/ > --- > > (Updated April 26, 2021, 5:41 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > Pinal Shah, and Sarath Subramanian. > > > Bugs: ATLAS-4221 > https://issues.apache.org/jira/browse/ATLAS-4221 > > > Repository: atlas > > > Description > --- > > Currently in Atlas, the collection names are not configurable > > We need to provide functionality where in names of the solr collections can > be customized using property configuration > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 4df38a5e6 > distro/src/bin/atlas_config.py 80b3bfb78 > distro/src/bin/atlas_start.py 7cf35a92a > distro/src/conf/atlas-application.properties e06e74a26 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 05b078416 > repository/src/test/java/org/apache/atlas/TestSolrConfigurableNames.java > PRE-CREATION > repository/src/test/resources/atlas-application.properties PRE-CREATION > > tools/atlas-index-repair/src/main/java/org/apache/atlas/tools/RepairIndex.java > 37565188e > > > Diff: https://reviews.apache.org/r/73249/diff/6/ > > > Testing > --- > > mvn clean install > mvn clean package -Pdist,embedded-hbase-solr > atlas server up and running > Collections created with entities linked > > > Thanks, > > chaitali > >
Re: Review Request 73290: ATLAS-4251: Upgrade luceneMatchVersion in Solr configs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73290/#review222893 --- distro/src/conf/solr/solrconfig.xml Line 38 (original), 38 (patched) <https://reviews.apache.org/r/73290/#comment312074> luceneMatchVersion 8.4.1 is corresponding to solr.version 8.6.3? - Pinal Shah On April 21, 2021, 3:05 p.m., Nikhil Bonte wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73290/ > --- > > (Updated April 21, 2021, 3:05 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon > Rodrigues, Pinal Shah, and Sarath Subramanian. > > > Bugs: ATLAS-4251 > https://issues.apache.org/jira/browse/ATLAS-4251 > > > Repository: atlas > > > Description > --- > > Altas' Solr config currently has luceneMatchVersion as 5.0.0 > > This improvement will upgrade it to 8.4.1 > > > Diffs > - > > dev-support/atlas-docker/config/solr/solrconfig.xml 21d19ef19 > distro/src/conf/solr/solrconfig.xml 21d19ef19 > repository/src/test/resources/solr/core-template/solrconfig.xml 9264f99d4 > test-tools/src/main/resources/solr/core-template/solrconfig.xml 977ebc49c > > > Diff: https://reviews.apache.org/r/73290/diff/1/ > > > Testing > --- > > Manually Tested > > Pre-commit build - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/511/consoleFull > (IN PROGRESS) > > > Thanks, > > Nikhil Bonte > >
Review Request 73310: ATLAS-4267 : Quick Search : AggregationMetrics is incorrect with some special characters
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73310/ --- Review request for atlas, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-4267 https://issues.apache.org/jira/browse/ATLAS-4267 Repository: atlas Description --- Quick search with attribute filter : qualifiedName - beginsWith - "default.datagen_table_sensitive_725_1." has empty AggregationMetrics whereas with "default.datagen_table_sensitive_725_1" has right AggregationMetrics. __ Diffs - graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java 1dd8be7e9 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 3750799f7 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java f55577c8c Diff: https://reviews.apache.org/r/73310/diff/1/ Testing --- Unit test case added Thanks, Pinal Shah
Re: Review Request 73310: ATLAS-4267 : Quick Search : AggregationMetrics is incorrect with some special characters
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73310/ --- (Updated April 29, 2021, 10:46 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-4267 https://issues.apache.org/jira/browse/ATLAS-4267 Repository: atlas Description --- Quick search with attribute filter : qualifiedName - beginsWith - "default.datagen_table_sensitive_725_1." has empty AggregationMetrics whereas with "default.datagen_table_sensitive_725_1" has right AggregationMetrics. __ Diffs (updated) - graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java 1dd8be7e9 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilderTest.java c2acc5b34 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 3750799f7 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java f55577c8c Diff: https://reviews.apache.org/r/73310/diff/2/ Changes: https://reviews.apache.org/r/73310/diff/1-2/ Testing (updated) --- Unit test case added Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/529/console Thanks, Pinal Shah
Re: Review Request 73310: ATLAS-4267 : Quick Search : AggregationMetrics is incorrect with some special characters
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73310/ --- (Updated April 29, 2021, 5:09 p.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian. Changes --- addressed review comments Bugs: ATLAS-4267 https://issues.apache.org/jira/browse/ATLAS-4267 Repository: atlas Description --- Quick search with attribute filter : qualifiedName - beginsWith - "default.datagen_table_sensitive_725_1." has empty AggregationMetrics whereas with "default.datagen_table_sensitive_725_1" has right AggregationMetrics. __ Diffs (updated) - graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java 1dd8be7e9 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilderTest.java c2acc5b34 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 3750799f7 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java f55577c8c Diff: https://reviews.apache.org/r/73310/diff/3/ Changes: https://reviews.apache.org/r/73310/diff/2-3/ Testing --- Unit test case added Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/529/console Thanks, Pinal Shah
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize pagination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/ --- (Updated May 7, 2021, 12:40 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian. Changes --- refactored Bugs: ATLAS-4254 https://issues.apache.org/jira/browse/ATLAS-4254 Repository: atlas Description (updated) --- **Current Working of Pagination** - When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0 - Query is then executed from 0 offset and limit - Filter out the results obtained from query - Generate result set by skipping the results from 0 to the given offset **New Working Structure** - New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value. - The marker should be set as '*' for the start of the page - The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request - if marker is set then offset should not be greater than 0 **Perfomance details** Took usecase of ranger tagsync: - On searching tagged entities starting with offset: 0 and limit: 1000 returned 1 entities in ~ 21s - On searching tagged entities starting with marker: * and limit: 1000 returned 1 entities in ~ 10s (both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers) - Around 51% improvement - Improvement is seen as the page size increases. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7 intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2 repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d Diff: https://reviews.apache.org/r/73292/diff/3/ Changes: https://reviews.apache.org/r/73292/diff/2-3/ Testing --- MAnual testing unit testcases Thanks, Pinal Shah
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize pagination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/ --- (Updated May 10, 2021, 4:43 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian. Changes --- addressed review comments Bugs: ATLAS-4254 https://issues.apache.org/jira/browse/ATLAS-4254 Repository: atlas Description --- **Current Working of Pagination** - When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0 - Query is then executed from 0 offset and limit - Filter out the results obtained from query - Generate result set by skipping the results from 0 to the given offset **New Working Structure** - New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value. - The marker should be set as '*' for the start of the page - The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request - if marker is set then offset should not be greater than 0 **Perfomance details** Took usecase of ranger tagsync: - On searching tagged entities starting with offset: 0 and limit: 1000 returned 1 entities in ~ 21s - On searching tagged entities starting with marker: * and limit: 1000 returned 1 entities in ~ 10s (both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers) - Around 51% improvement - Improvement is seen as the page size increases. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7 intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2 repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d Diff: https://reviews.apache.org/r/73292/diff/4/ Changes: https://reviews.apache.org/r/73292/diff/3-4/ Testing --- MAnual testing unit testcases Thanks, Pinal Shah
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize pagination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/ --- (Updated May 10, 2021, 4:56 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-4254 https://issues.apache.org/jira/browse/ATLAS-4254 Repository: atlas Description (updated) --- **Current Working of Pagination** - When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0 - Query is then executed from 0 offset and limit - Filter out the results obtained from query - Generate result set by skipping the results from 0 to the given offset **New Working Structure** - New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value. - The marker should be set as '*' for the start of the page - The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request - if marker is set then offset should not be greater than 0 **Perfomance details** Took usecase of ranger tagsync: - On searching tagged entities starting with offset: 0 and limit: 1000 returned 1 entities in ~ 21s - On searching tagged entities starting with marker: * and limit: 1000 returned 1 entities in ~ 10s (both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers) - Removed 'atlas.graph.index.search.max-result-set-size=150' property from atlas-application.properties - Around 51% improvement - Improvement is seen as the page size increases. Diffs - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7 intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2 repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d Diff: https://reviews.apache.org/r/73292/diff/4/ Testing --- MAnual testing unit testcases Thanks, Pinal Shah
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize pagination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/ --- (Updated May 13, 2021, 3:43 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-4254 https://issues.apache.org/jira/browse/ATLAS-4254 Repository: atlas Description (updated) --- **Current Working of Pagination** - When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0 - Query is then executed from 0 offset and limit - Filter out the results obtained from query - Generate result set by skipping the results from 0 to the given offset **New Working Structure** - New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value. - The marker should be set as '*' for the start of the page - The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request - if marker is set then offset should not be greater than 0 **Perfomance details** Took usecase of ranger tagsync: - On searching tagged entities starting with offset: 0 and limit: 1000 returned 1 entities in ~ 21s - On searching tagged entities starting with marker: * and limit: 1000 returned 1 entities in ~ 10s (both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers) - Removed 'atlas.graph.index.search.max-result-set-size=150' property from atlas-application.properties - Around 25% improvement - Improvement is seen as the page size increases. Diffs - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7 intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2 repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d Diff: https://reviews.apache.org/r/73292/diff/5/ Testing --- MAnual testing unit testcases Thanks, Pinal Shah
Review Request 73412: ATLAS-4332 : DSL Query : query with like operator and "/" in search text throws 500
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73412/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4332 https://issues.apache.org/jira/browse/ATLAS-4332 Repository: atlas Description --- **Problem:** In DSL query , searching with like operator in combination of special characters like '/' in search text throws 500 **Example:** hdfs_path name="/warehouse/tablespace/external/hive/db_fdchj.db" returns right entity. hdfs_path name like "/warehouse/*" throws following exception: Diffs - repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 320acbe22 repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java c53a324a1 repository/src/test/java/org/apache/atlas/BasicTestSetup.java 99e075ac5 repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java b10400157 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 0c1dd59d2 Diff: https://reviews.apache.org/r/73412/diff/1/ Testing --- unit testcase added Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/640/console (In Progress) Thanks, Pinal Shah
Re: Review Request 73412: ATLAS-4332 : DSL Query : query with like operator and "/" in search text throws 500
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73412/ --- (Updated June 10, 2021, 11:32 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Changes --- addressed review comment Bugs: ATLAS-4332 https://issues.apache.org/jira/browse/ATLAS-4332 Repository: atlas Description --- **Problem:** In DSL query , searching with like operator in combination of special characters like '/' in search text throws 500 **Example:** hdfs_path name="/warehouse/tablespace/external/hive/db_fdchj.db" returns right entity. hdfs_path name like "/warehouse/*" throws following exception: Diffs (updated) - repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 320acbe22 repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java c53a324a1 repository/src/test/java/org/apache/atlas/BasicTestSetup.java 99e075ac5 repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java b10400157 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 0c1dd59d2 repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 3a2d66d8c Diff: https://reviews.apache.org/r/73412/diff/2/ Changes: https://reviews.apache.org/r/73412/diff/1-2/ Testing --- unit testcase added Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/640/console (In Progress) Thanks, Pinal Shah
Re: Review Request 73412: ATLAS-4332 : DSL Query : query with like operator and "/" in search text throws 500
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73412/ --- (Updated June 10, 2021, 4:48 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Changes --- addressed comments Bugs: ATLAS-4332 https://issues.apache.org/jira/browse/ATLAS-4332 Repository: atlas Description --- **Problem:** In DSL query , searching with like operator in combination of special characters like '/' in search text throws 500 **Example:** hdfs_path name="/warehouse/tablespace/external/hive/db_fdchj.db" returns right entity. hdfs_path name like "/warehouse/*" throws following exception: Diffs (updated) - repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 320acbe22 repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java c53a324a1 repository/src/test/java/org/apache/atlas/BasicTestSetup.java 99e075ac5 repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java b10400157 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 0c1dd59d2 repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 3a2d66d8c Diff: https://reviews.apache.org/r/73412/diff/3/ Changes: https://reviews.apache.org/r/73412/diff/2-3/ Testing --- unit testcase added Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/640/console (In Progress) Thanks, Pinal Shah
Re: Review Request 73370: ATLAS-4300 : DSL Search : Support search by classification and its attribute
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73370/ --- (Updated June 14, 2021, 2:21 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Changes --- added doc, addressed review comment Bugs: ATLAS-4300 https://issues.apache.org/jira/browse/ATLAS-4300 Repository: atlas Description --- **Problem:** Advanced Search support Searching entities by classification Ex. Table isA Dimension But user cannot search entities by classification attributes Ex. To search entities which has classification Dimension and Dimension.attr = value **Improvement:** User will be able to search entities through classification attributes too. **Example query:** "From Table where name = 'sales_fact' and Dimension.attr = 'value' " Diffs (updated) - docs/src/documents/Search/SearchAdvanced.md 45e63675a repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 80250fb96 repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java cff7afffc repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java d2906ea40 repository/src/main/java/org/apache/atlas/query/RegistryBasedLookup.java eb3c349f6 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 4091fe982 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 4bdf479f1 repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 04f602c0f repository/src/test/java/org/apache/atlas/BasicTestSetup.java a821b2549 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3404dc6e2 repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 6220c23b9 Diff: https://reviews.apache.org/r/73370/diff/3/ Changes: https://reviews.apache.org/r/73370/diff/2-3/ Testing --- Unit test cases added Thanks, Pinal Shah
Review Request 73431: ATLAS-4341 : Regression: DSL query with like operator and regex expression does not work
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73431/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4341 https://issues.apache.org/jira/browse/ATLAS-4341 Repository: atlas Description --- **Issue:** hdfs_path having 5 entities with suffix 0 to 5 /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name%20like%20%22hdfs_path_vzxtp_*%22 **Finds the result** But /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name like "hdfs_path_vzxtp_[0-9]" **Returns empty** Diffs - repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java d2906ea40 Diff: https://reviews.apache.org/r/73431/diff/1/ Testing --- Manually verified Precommit : Thanks, Pinal Shah
Re: Review Request 73431: ATLAS-4341 : Regression: DSL query with like operator and regex expression does not work
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73431/ --- (Updated June 22, 2021, 5:21 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4341 https://issues.apache.org/jira/browse/ATLAS-4341 Repository: atlas Description --- **Issue:** hdfs_path having 5 entities with suffix 0 to 5 /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name%20like%20%22hdfs_path_vzxtp_*%22 **Finds the result** But /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name like "hdfs_path_vzxtp_[0-9]" **Returns empty** Diffs (updated) - repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java d2906ea40 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3404dc6e2 Diff: https://reviews.apache.org/r/73431/diff/2/ Changes: https://reviews.apache.org/r/73431/diff/1-2/ Testing (updated) --- Manually verified Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/692/ (In Progress) Thanks, Pinal Shah
Re: Review Request 73431: ATLAS-4341 : Regression: DSL query with like operator and regex expression does not work
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73431/ --- (Updated June 22, 2021, 5:30 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4341 https://issues.apache.org/jira/browse/ATLAS-4341 Repository: atlas Description --- **Issue:** hdfs_path having 5 entities with suffix 0 to 5 /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name%20like%20%22hdfs_path_vzxtp_*%22 **Finds the result** But /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name like "hdfs_path_vzxtp_[0-9]" **Returns empty** Diffs (updated) - repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java d2906ea40 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3404dc6e2 repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 6220c23b9 Diff: https://reviews.apache.org/r/73431/diff/3/ Changes: https://reviews.apache.org/r/73431/diff/2-3/ Testing --- Manually verified Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/692/ (In Progress) Thanks, Pinal Shah
Re: Review Request 73431: ATLAS-4341 : Regression: DSL query with like operator and regex expression does not work
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73431/ --- (Updated June 22, 2021, 5:32 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4341 https://issues.apache.org/jira/browse/ATLAS-4341 Repository: atlas Description --- **Issue:** hdfs_path having 5 entities with suffix 0 to 5 /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name%20like%20%22hdfs_path_vzxtp_*%22 **Finds the result** But /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name like "hdfs_path_vzxtp_[0-9]" **Returns empty** Diffs - repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java d2906ea40 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 3404dc6e2 repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 6220c23b9 Diff: https://reviews.apache.org/r/73431/diff/3/ Testing (updated) --- Manually verified Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/693/ (In Progress) Thanks, Pinal Shah
Review Request 73439: ATLAS-4347 : DSL Search : Caching Mechanism for translated dsl queries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73439/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4347 https://issues.apache.org/jira/browse/ATLAS-4347 Repository: atlas Description --- **Approach** - DSL Query is converted to Tinkerpop Traversals. - Caching traversal with respective to query saves time of translation. - Whenever same query with same limit and offset is fired again, it will pick the translated query from cache. **Config** - atlas.dsl.cached.translator when 'true' uses caching mechanism (Default: true) Diffs - intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 9ef848752 repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java 724cb8244 Diff: https://reviews.apache.org/r/73439/diff/1/ Testing --- Thanks, Pinal Shah
Re: Review Request 73439: ATLAS-4347 : DSL Search : Caching Mechanism for translated dsl queries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73439/ --- (Updated July 7, 2021, 4:28 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4347 https://issues.apache.org/jira/browse/ATLAS-4347 Repository: atlas Description --- **Approach** - DSL Query is converted to Tinkerpop Traversals. - Caching traversal with respective to query saves time of translation. - Whenever same query with same limit and offset is fired again, it will pick the translated query from cache. **Config** - atlas.dsl.cached.translator when 'true' uses caching mechanism (Default: true) Diffs - intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 9ef848752 repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java 724cb8244 Diff: https://reviews.apache.org/r/73439/diff/1/ Testing (updated) --- Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/718/ Thanks, Pinal Shah
Re: Review Request 73461: ATLAS-4362 : Basic Search : Support filtering with multiple entity/classification type with its corresponding attributes
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73461/
---
(Updated July 26, 2021, 4:26 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj,
Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-4362
https://issues.apache.org/jira/browse/ATLAS-4362
Repository: atlas
Description
---
**Problem**
- Searching with multiple entity/classification type like hive_table,
hive_column is possible in basic search.
- Searching with multiple entity/classification type like hive_table,
hive_column with common attributes like system Attributes or attribute to be
search present in both the type is possible.
- But searching with multiple entity/classification type like hive_table,
hive_column with its respective attributes like hive_table.tableType is not
possible
**Changes**
- Field typeName (String) is added in SearchParameters.FilterCriteria
- Example:
"entityFilters": {
"condition": "AND",
"criterion": [{
"attributeName": "tableType",
"operator": "eq",
"attributeValue": "Managed",
"typeName": "hive_table"
}]
}
**Appraoch**
- If 'typeName' is added under filterCriteria, extra filter of *type* is added
with *AND* condition to *attribute* filter
- Example:
("typeName": hive_table AND "hive_table.tableType": (Managed))
- To get the correct results, it is preferable to set typeName as
'ALL_ENTITY_TYPES'
- Classification is also handled same way
Diffs
-
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
78fb4a48f
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
01954d07e
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
f69dc4215
repository/src/test/java/org/apache/atlas/BasicTestSetup.java a1d7b62a5
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
a9fbd435a
repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java
121dca9f5
repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
764568915
Diff: https://reviews.apache.org/r/73461/diff/2/
Testing
---
added test cases
Thanks,
Pinal Shah
Re: Planning for Apache Atlas 2.2.0 release
+1 for Apache Atlas 2.2.0 release Thanks Sidharth for volunteering. Regards, Pinal Shah On Wed, Jul 28, 2021 at 6:41 PM Mandar Ambawane < mandar.ambaw...@freestoneinfotech.com> wrote: > +1 for the Atlas 2.2.0 release. > > Thanks Siddharth for volunteering the release. > > On Wed, Jul 28, 2021 at 6:27 PM Jayendra Parab > wrote: > > > +1 for Apache Atlas 2.2.0 release > > > > Thanks Siddharth for volunteering! > > > > Regards, > > Jayendra. > > > > On Wed, Jul 28, 2021 at 2:28 PM Nixon Rodrigues < > > nixon.rodrig...@freestoneinfotech.com> wrote: > > > >> +1 for the Atlas 2.2.0 release. > >> > >> Thanks Siddharth for volunteering the release. > >> > >> Regards, > >> Nixon Rodrigue > >> > >> > >> On Wed, Jul 28, 2021 at 1:23 PM Madhan Neethiraj > >> wrote: > >> > >>> +1 for Apache Atlas 2.2.0 release. It has been a while now! > >>> > >>> > >>> > >>> Thanks Sid! > >>> > >>> > >>> > >>> Madhan > >>> > >>> > >>> > >>> > >>> > >>> *From: *Nikhil Bonte > >>> *Reply-To: *"u...@atlas.apache.org" > >>> *Date: *Tuesday, July 27, 2021 at 10:20 PM > >>> *To: *"u...@atlas.apache.org" > >>> *Cc: *"dev@atlas.apache.org" > >>> *Subject: *Re: Planning for Apache Atlas 2.2.0 release > >>> > >>> > >>> > >>> Thanks for volunteering Sidharth! > >>> > >>> > >>> > >>> +1 for the release Apache Atlas 2.2. > >>> > >>> > >>> > >>> > >>> > >>> Regards > >>> > >>> Nikhil P. Bonte > >>> > >>> > >>> > >>> > >>> > >>> On Wed, Jul 28, 2021 at 10:46 AM Ashutosh Mestry > > >>> wrote: > >>> > >>> Sidharth: Thanks for volunteering! > >>> > >>> > >>> > >>> +1 for the release. > >>> > >>> > >>> > >>> *~ ashutosh* > >>> > >>> *Ashutosh Mestry* * . Staff Software Engineer . > >>> Cloudera, Inc. . * > >>> > >>> ... > >>> > >>> *No hurry, no pause. – Tim Ferriss, Life Hacker, Author* > >>> > >>> > >>> > >>> > >>> > >>> *From: *Sarath Subramanian > >>> *Date: *Tuesday, July 27, 2021 at 10:06 PM > >>> *To: *dev@atlas.apache.org > >>> *Cc: *u...@atlas.apache.org > >>> *Subject: *Re: Planning for Apache Atlas 2.2.0 release > >>> > >>> Thanks for volunteering to release Apache Atlas 2.2 > >>> > >>> +1 > >>> > >>> > >>> Thanks, > >>> Sarath > >>> > >>> > >>> On Tue, Jul 27, 2021 at 12:11 PM Sidharth Mishra > > >>> wrote: > >>> > >>> > Dear Atlas community, > >>> > > >>> > Since Apache Atlas 2.1 was released, there are many new features, > >>> > improvements and fixes. Here are few features/enhancements since last > >>> > release, Apache Atlas 2.1: > >>> > > >>> > - Introduced Deferred Action - Tag propagation will be handled as > >>> > a background process (AtlasTask) > >>> > > >>> > - Added Re-indexing as JAVA_PATCH > >>> > > >>> > - Created JavaPatch to add new super types to existing entities > >>> > > >>> > - Supported for Business Metadata in Atlas Export API > >>> > > >>> > - Supported HA for admin/task API > >>> > > >>> > - Enhanced export service to export Terms > >>> > > >>> > - Added new models and addition attributes to Atlas for Apache > >>> > Flink, GCP and ADLS Gen 2 > >>> > > >>> > - Added support to Configure HiveServer2 Hook to send > Lineage-only > >>> > Messages > >>> > > >>> > - Improved and enhanced Bulk Glossary Import supporting import > >>> > with relations > >>> > > >>> > - Dependency between Atlas and Kafka Zookeeper removed > >>>
Review Request 73503: ATLAS-4386 : Relationship search: Sorting does not work when passed relationship label
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73503/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4386 https://issues.apache.org/jira/browse/ATLAS-4386 Repository: atlas Description --- **Problem:** Relationship Search is not able to sort when relationship label is passed as a queryParam instead of attribute. **Example:** api/atlas/v2/search/relationship?limit=25&offset=0&guid=cca5c29d-307f-4a9b-ac6c-79f41a08a90b&relation=__hive_table.columns&sortBy=name&sortOrder=DESCENDING Diffs - repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java f2290c672 repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ff7fd150e repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java a9fbd435a Diff: https://reviews.apache.org/r/73503/diff/1/ Testing --- added test cases Thanks, Pinal Shah
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
+1 for Apache Atlas 2.2.0 rc1
Validation:
- Downloaded source
- mvn build with embedded-hbase-solr profile.
- Verified no binaries are present
- Ran quick_start
- Validated quick_search, aggregation metrics
- Validated basic search
- Validated advanced search with terms, relationships, classification
- Validated Relationship search
*Can we update README.txt third point?*
Thanks,
Pinal Shah
On Wed, Aug 11, 2021 at 1:54 AM Sarath Subramanian
wrote:
> +1 for Apache Atlas 2.2.0 rc1
>
> Validated the following:
> * Downloaded source and validated MD5 and SHA512 hash
> * Validated signature from release manager ("Sidharth Mishra <
> sidmis...@apache.org>")
> * Verified source doesn't contain any binaries
> * No errors in application log startup.
> * Build the source successfully using embedded-hbase-solr profile
> * Started Atlas and ran quick start to create sample types and entities
> * Validated metrics UI works with all entity types and count
> * Validated Quick search, Basic search and DSL search, Glossary
> * Admin Page - TypeSystem UI, Admin audits
> * Tag propagation use-cases
>
>
> Thanks,
> Sarath
>
> On Mon, Aug 9, 2021 at 3:55 PM Sidharth Mishra
> wrote:
>
> > Atlas team,
> >
> > Apache Atlas 2.2.0 rc1, with the following fixes since rc0, is now
> > available for vote within the dev community.
> >
> > ATLAS-4390 - Changed Atlas branch to release-2.2.0-rc1
> > ATLAS-4391 - NoSuchFileException for
> {atlas.log.dir}/atlas-metrics.out
> > ATLAS-4343 - [Entity Audits] 'Propagated Classification Added'
> > Timestamp is < 'Entity
> > Created' Timestamp
> > ATLAS-4387 - UI: Propagated Classifications shows wrong count for
> > entity type on Entity detail
> > page
> > ATLAS-4386 - Relationship search: Sorting does not work when
> > relationship label is passed
> >
> > Links to the release artifacts are given below. Please review and vote.
> >
> > The vote will be open for at least 72 hours or until necessary votes
> > are reached.
> >
> > [ ] +1 Approve
> >
> > [ ] +0 No opinion
> >
> > [ ] -1 Disapprove (and reason why)
> >
> > Thanks,
> > Sid
> >
> > List of all issues addressed in this release:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
> >
> > Git tag for the release:
> > https://github.com/apache/atlas/tree/release-2.2.0-rc1
> >
> > Sources for the release:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz
> >
> > Source release verification:
> >
> > PGP Signature:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.asc
> >
> > SHA512 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.sha512
> >
> > MD5 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.md5
> >
> >
> > Keys to verify the signature of the release artifacts are available
> > at: https://dist.apache.org/repos/dist/release/atlas/KEYS
> >
> >
> > New features/enhancements:
> >
> > - Introduced Deferred Action - Tag propagation will be handled as
> > a background process (AtlasTask)
> >
> > - Added Re-indexing as JAVA_PATCH
> >
> > - Created JavaPatch to add new super types to existing entities
> >
> > - Supported for Business Metadata in Atlas Export API
> >
> > - Supported HA for admin/task API
> >
> > - Enhanced export service to export Terms
> >
> > - Added new models and addition attributes to Atlas for Apache
> > Flink, GCP and ADLS Gen 2
> >
> > - Added support to Configure HiveServer2 Hook to send Lineage-only
> > Messages
> >
> > - Improved and enhanced Bulk Glossary Import supporting import
> > with relations
> >
> > - Dependency between Atlas and Kafka Zookeeper removed
> >
> > - Improved performance of GraphHelper's guid and status getter
> methods
> >
> > - Added new privilege - "admin-audits" for Atlas Admin Audits
> > authorization
> >
> > - Basic search Improvements - search text with special characters,
> > optimize pagination
> >
> > - Updated free-text search processor to support Ela
Review Request 73584: ATLAS-4422 : Quick Search: Provide sorting
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73584/
---
Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath
Subramanian.
Bugs: ATLAS-4422
https://issues.apache.org/jira/browse/ATLAS-4422
Repository: atlas
Description
---
**Improvement:**
This will add a sorting functionality in quick search API and get the result
list in Sorted manner.
**Note:**
- Result will be sorted only when 'typeName' has a valid entity type name as a
value.
- Only string attribute type (not TEXT) guarantees sorted results
**Example:**
{ "query":"sales",
"typeName":"Asset",
"sortBy":"__typeName",
"sortOrder":"ASCENDING" }
Diffs
-
intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
9e5e9778a
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
5b3b9486b
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e4c74a9d1
Diff: https://reviews.apache.org/r/73584/diff/1/
Testing
---
Tested manually
Thanks,
Pinal Shah
Re: Review Request 73584: ATLAS-4422 : Quick Search: Provide sorting
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73584/
---
(Updated Sept. 15, 2021, 4:03 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath
Subramanian.
Changes
---
added UTs
Bugs: ATLAS-4422
https://issues.apache.org/jira/browse/ATLAS-4422
Repository: atlas
Description
---
**Improvement:**
This will add a sorting functionality in quick search API and get the result
list in Sorted manner.
**Note:**
- Result will be sorted only when 'typeName' has a valid entity type name as a
value.
- Only string attribute type (not TEXT) guarantees sorted results
**Example:**
{ "query":"sales",
"typeName":"Asset",
"sortBy":"__typeName",
"sortOrder":"ASCENDING" }
Diffs (updated)
-
intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
9e5e9778a
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
5b3b9486b
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
5e47295cb
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e4c74a9d1
Diff: https://reviews.apache.org/r/73584/diff/2/
Changes: https://reviews.apache.org/r/73584/diff/1-2/
Testing (updated)
---
added Unit tests
Thanks,
Pinal Shah
Review Request 73641: Repair Composite indexes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73641/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Repository: atlas Description --- If by any chance, commposite index are not enabled, this patch fixes the composite index from REGISTERED state to ENABLED state Diffs - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java 7e3b2f440 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java 1cc7f8b55 repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java d30971afc repository/src/main/java/org/apache/atlas/repository/patches/RepairCompositeIndexPatch.java PRE-CREATION Diff: https://reviews.apache.org/r/73641/diff/1/ Testing --- manually tested Thanks, Pinal Shah
Re: Review Request 73603: ATLAS-4436 Basic Search : Search By Query '*' gets internal entities also
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73603/#review223592 --- Ship it! Ship It! - Pinal Shah On Sept. 28, 2021, 6:02 a.m., Shraddha Pardeshi wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73603/ > --- > > (Updated Sept. 28, 2021, 6:02 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah. > > > Bugs: ATLAS-4436 > https://issues.apache.org/jira/browse/ATLAS-4436 > > > Repository: atlas > > > Description > --- > > when we add "*" in search by query in response it brings all the internal > entities like '__AtlasAuditEntry'. But expected behaviour is, it should not > show internal types. > So here filtering those internal types in basic search > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > 86f2cea23 > > repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java > 2d8a448b1 > > repository/src/test/java/org/apache/atlas/discovery/FreeTextSearchProcessorTest.java > e497bb786 > > > Diff: https://reviews.apache.org/r/73603/diff/3/ > > > Testing > --- > > - Local testing done > - Added unit testcase > - PC - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/883/consoleFull > > > Thanks, > > Shraddha Pardeshi > >
Re: Review Request 73641: Repair Composite indexes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73641/ --- (Updated Oct. 8, 2021, 4:05 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Bugs: ATLAS-4447 https://issues.apache.org/jira/browse/ATLAS-4447 Repository: atlas Description --- If by any chance, commposite index are not enabled, this patch fixes the composite index from REGISTERED state to ENABLED state Diffs - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java 7e3b2f440 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java 1cc7f8b55 repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java d30971afc repository/src/main/java/org/apache/atlas/repository/patches/RepairCompositeIndexPatch.java PRE-CREATION Diff: https://reviews.apache.org/r/73641/diff/1/ Testing --- manually tested Thanks, Pinal Shah
Re: Review Request 73641: Repair Composite indexes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73641/ --- (Updated Oct. 11, 2021, 8 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Changes --- Addressed review comments Bugs: ATLAS-4447 https://issues.apache.org/jira/browse/ATLAS-4447 Repository: atlas Description --- If by any chance, commposite index are not enabled, this patch fixes the composite index from REGISTERED state to ENABLED state Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java 7e3b2f440 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java 1cc7f8b55 repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java d30971afc repository/src/main/java/org/apache/atlas/repository/patches/UpdateCompositeIndexStatusPatch.java PRE-CREATION Diff: https://reviews.apache.org/r/73641/diff/2/ Changes: https://reviews.apache.org/r/73641/diff/1-2/ Testing --- manually tested Thanks, Pinal Shah
Re: Review Request 73641: ATLAS-4447: Update Composite indexes status
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73641/ --- (Updated Oct. 11, 2021, 7:20 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Changes --- added configuration for the patch added commit per index Bugs: ATLAS-4447 https://issues.apache.org/jira/browse/ATLAS-4447 Repository: atlas Description --- If by any chance, commposite index are not enabled, this patch fixes the composite index from REGISTERED state to ENABLED state Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java 7e3b2f440 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java 1cc7f8b55 intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 20f8f7343 repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java d30971afc repository/src/main/java/org/apache/atlas/repository/patches/UpdateCompositeIndexStatusPatch.java PRE-CREATION Diff: https://reviews.apache.org/r/73641/diff/3/ Changes: https://reviews.apache.org/r/73641/diff/2-3/ Testing --- manually tested Thanks, Pinal Shah
Re: Review Request 73641: ATLAS-4447: Update Composite indexes status
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73641/ --- (Updated Oct. 12, 2021, 5:30 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Changes --- changed the configuration to set default true Bugs: ATLAS-4447 https://issues.apache.org/jira/browse/ATLAS-4447 Repository: atlas Description --- If by any chance, commposite index are not enabled, this patch fixes the composite index from REGISTERED state to ENABLED state Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java 7e3b2f440 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java 1cc7f8b55 intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 20f8f7343 repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java d30971afc repository/src/main/java/org/apache/atlas/repository/patches/UpdateCompositeIndexStatusPatch.java PRE-CREATION Diff: https://reviews.apache.org/r/73641/diff/4/ Changes: https://reviews.apache.org/r/73641/diff/3-4/ Testing --- manually tested Thanks, Pinal Shah
Re: Review Request 73641: ATLAS-4447: Update Composite indexes status
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73641/ --- (Updated Oct. 12, 2021, 8:52 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Changes --- rebase Bugs: ATLAS-4447 https://issues.apache.org/jira/browse/ATLAS-4447 Repository: atlas Description --- If by any chance, commposite index are not enabled, this patch fixes the composite index from REGISTERED state to ENABLED state Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java 50d17a223 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java b3eb07142 intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 20f8f7343 repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java d30971afc repository/src/main/java/org/apache/atlas/repository/patches/UpdateCompositeIndexStatusPatch.java PRE-CREATION Diff: https://reviews.apache.org/r/73641/diff/5/ Changes: https://reviews.apache.org/r/73641/diff/4-5/ Testing --- manually tested Thanks, Pinal Shah
Re: Review Request 73654: ATLAS-4454 : Upgrade JanusGraph version to 0.6.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73654/ --- (Updated Oct. 19, 2021, 7:45 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Changes --- addressed review comments Bugs: ATLAS-4454 https://issues.apache.org/jira/browse/ATLAS-4454 Repository: atlas Description --- Upgrading janusgraph version - 0.6.0 Upgrading tinkerpop version - 3.5.1 Diffs (updated) - graphdb/janus-hbase2/pom.xml 9ad64062f graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseKeyColumnValueStore.java 9aa552d4a graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java 40bcd1ad3 graphdb/janus/pom.xml f38c4e9a3 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 0d47e38fd graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java 2d5bd8ae7 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java 16fc4451b graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 81ae202f1 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java 82ae67514 pom.xml 84630e76b repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java dd56ed141 test-tools/src/main/resources/solr/core-template/solrconfig.xml e7956cb36 tools/atlas-index-repair/pom.xml a723bfa0c Diff: https://reviews.apache.org/r/73654/diff/3/ Changes: https://reviews.apache.org/r/73654/diff/2-3/ Testing --- mvn clean package -Pdist,embedded-hbase-solr- passes Quick start - passes Testing by removing janusgraph-cassandra dependency - left Testing more combination of NEQ, NULL operators - left Precommit running - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/900 Temporary changes for solrconfig.xml to pass PC Thanks, Pinal Shah
Re: Review Request 73654: ATLAS-4454 : Upgrade JanusGraph version to 0.6.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73654/ --- (Updated Oct. 19, 2021, 11:27 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Bugs: ATLAS-4454 https://issues.apache.org/jira/browse/ATLAS-4454 Repository: atlas Description --- Upgrading janusgraph version - 0.6.0 Upgrading tinkerpop version - 3.5.1 Diffs - graphdb/janus-hbase2/pom.xml 9ad64062f graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseKeyColumnValueStore.java 9aa552d4a graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java 40bcd1ad3 graphdb/janus/pom.xml f38c4e9a3 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 0d47e38fd graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java 2d5bd8ae7 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java 16fc4451b graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 81ae202f1 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java 82ae67514 pom.xml 84630e76b repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java dd56ed141 test-tools/src/main/resources/solr/core-template/solrconfig.xml e7956cb36 tools/atlas-index-repair/pom.xml a723bfa0c Diff: https://reviews.apache.org/r/73654/diff/3/ Testing (updated) --- mvn clean package -Pdist,embedded-hbase-solr- passes Quick start - passes Precommit running - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/900 Temporary changes for solrconfig.xml to pass PC Thanks, Pinal Shah
Re: Review Request 73654: ATLAS-4454 : Upgrade JanusGraph version to 0.6.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73654/ --- (Updated Oct. 20, 2021, 5:45 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Changes --- addressed review comments Bugs: ATLAS-4454 https://issues.apache.org/jira/browse/ATLAS-4454 Repository: atlas Description --- Upgrading janusgraph version - 0.6.0 Upgrading tinkerpop version - 3.5.1 Diffs (updated) - graphdb/janus-hbase2/pom.xml 9ad64062f graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseKeyColumnValueStore.java 9aa552d4a graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java 40bcd1ad3 graphdb/janus/pom.xml f38c4e9a3 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 0d47e38fd graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java 2d5bd8ae7 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java 16fc4451b graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 81ae202f1 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java 82ae67514 pom.xml 84630e76b repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java dd56ed141 test-tools/src/main/resources/solr/core-template/solrconfig.xml e7956cb36 tools/atlas-index-repair/pom.xml a723bfa0c Diff: https://reviews.apache.org/r/73654/diff/4/ Changes: https://reviews.apache.org/r/73654/diff/3-4/ Testing --- mvn clean package -Pdist,embedded-hbase-solr- passes Quick start - passes Precommit running - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/900 Temporary changes for solrconfig.xml to pass PC Thanks, Pinal Shah
Re: Review Request 73654: ATLAS-4454 : Upgrade JanusGraph version to 0.6.0
> On Oct. 19, 2021, 5:57 p.m., Madhan Neethiraj wrote: > > graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseKeyColumnValueStore.java > > Lines 157 (patched) > > <https://reviews.apache.org/r/73654/diff/3/?file=2254164#file2254164line157> > > > > Is there no functional impact in not implemeting this method? Thanks Madhan for reviewing. As this method in introduced in interface KeyColumnValueStore, there is a need to override this method. The implementation (throw new UnsupportedOperationException();) is exactly same as of actual HBaseKeyColumnValueStore class of janusgraph - Pinal --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73654/#review223634 --- On Oct. 20, 2021, 5:45 p.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73654/ > --- > > (Updated Oct. 20, 2021, 5:45 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath > Subramanian. > > > Bugs: ATLAS-4454 > https://issues.apache.org/jira/browse/ATLAS-4454 > > > Repository: atlas > > > Description > --- > > Upgrading janusgraph version - 0.6.0 > Upgrading tinkerpop version - 3.5.1 > > > Diffs > - > > graphdb/janus-hbase2/pom.xml 9ad64062f > > graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseKeyColumnValueStore.java > 9aa552d4a > > graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java > 40bcd1ad3 > graphdb/janus/pom.xml f38c4e9a3 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java > 0d47e38fd > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java > 2d5bd8ae7 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java > 16fc4451b > graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java > 81ae202f1 > > graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java > 82ae67514 > pom.xml 84630e76b > repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java > dd56ed141 > test-tools/src/main/resources/solr/core-template/solrconfig.xml e7956cb36 > tools/atlas-index-repair/pom.xml a723bfa0c > > > Diff: https://reviews.apache.org/r/73654/diff/4/ > > > Testing > --- > > mvn clean package -Pdist,embedded-hbase-solr- passes > Quick start - passes > > Precommit running - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/900 > > > Temporary changes for solrconfig.xml to pass PC > > > Thanks, > > Pinal Shah > >
Review Request 73798: ATLAS-4529 : DSL Search : OrderBy does not work for ScriptEngineBasedExecutor (gremlin groovy)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73798/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Bugs: ATLAS-4529 https://issues.apache.org/jira/browse/ATLAS-4529 Repository: atlas Description --- After upgrading janusgraph version to 0.6.0 When below property is changed to 'false', order by doesn't work atlas.dsl.executor.traversal=false Diffs - repository/src/main/java/org/apache/atlas/query/GremlinClause.java 7014b4a4f repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 547a43dd7 repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 52cb68caa Diff: https://reviews.apache.org/r/73798/diff/1/ Testing --- Thanks, Pinal Shah
Review Request 73836: ATLAS-4551: Add maven jacoco plugin for code coverage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73836/ --- Review request for atlas, Jayendra Parab and Sarath Subramanian. Bugs: ATLAS-4551 https://issues.apache.org/jira/browse/ATLAS-4551 Repository: atlas Description --- Jacoco maven plugin generates test coverage report by measuring lines of code exercised during the tests *Step* mvn clean install *Report* report will be generated in /target/site/jacoco directory of each module. For more Graphical interface, import the genrated reports of jacoco to SonarQube mvn sonar:sonar -Dsonar.host.url=http://localhost:9000 -DskipTests -Dsonar.coverage.jacoco.xmlReportPaths=/target/site/jacoco.xml -Dsonar.projectKey=Atlas -Dsonar.projectName=Atlas -Drat.numUnapprovedLicenses=1 -Dsonar.login=bc6e210c3f4c1873eee82e31059215a9deeff526 Diffs - pom.xml 2f7b00af3 Diff: https://reviews.apache.org/r/73836/diff/1/ Testing --- Thanks, Pinal Shah
Re: Review Request 73956: ATLAS-4569: Kafka importer should use only kafka-clients from Kafka libs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73956/#review224378 --- Ship it! Ship It! - Pinal Shah On April 27, 2022, 9:49 a.m., Andras Katona wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73956/ > --- > > (Updated April 27, 2022, 9:49 a.m.) > > > Review request for atlas, Dharshana MK, Disha Talreja, Jayendra Parab, > madhan, and Madhan Neethiraj. > > > Bugs: ATLAS-4569 > https://issues.apache.org/jira/browse/ATLAS-4569 > > > Repository: atlas > > > Description > --- > > Putting kafka-clients and slf4j-api to classpath from kafka/libs. Other > dependencies of kafka-clients are compression libs, which are not used, hence > not required. > > > Diffs > - > > addons/kafka-bridge/src/bin/import-kafka.sh fbc16f0d0 > > > Diff: https://reviews.apache.org/r/73956/diff/1/ > > > Testing > --- > > manually on a cluster > > > Thanks, > > Andras Katona > >
Re: Review Request 73925: ATLAS-4575:Version Compatibility - Handle cross version messages in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73925/#review224506 --- Ship it! Ship It! - Pinal Shah On June 14, 2022, 9:05 a.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73925/ > --- > > (Updated June 14, 2022, 9:05 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Bugs: ATLAS-4575 > https://issues.apache.org/jira/browse/ATLAS-4575 > > > Repository: atlas > > > Description > --- > > Atlas server can receive notification messages from atlas hooks running on > different version (than Atlas server version). When messages are not > compatible, appropriate actions should be taken: > > Log Error/Warning in Atlas logs about cross-compat version messages > > > Diffs > - > > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java > b43bc7c66 > notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java > 1af1f3efa > > > Diff: https://reviews.apache.org/r/73925/diff/6/ > > > Testing > --- > > Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1070/console > mvn clean -DskipTests package -Pdist,embedded-hbase-solr > > > Thanks, > > Vinayak Marraiya > >
Review Request 74047: ATLAS:4558: Fix AtlasRepairIndex tool when tls is enabled
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74047/ --- Review request for atlas, Jayendra Parab, Mandar Ambawane, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4558 https://issues.apache.org/jira/browse/ATLAS-4558 Repository: atlas Description --- **Issue :** When tls is enabled, while running atlasrepair tool, it couldn't connect to solr because it is unable to find valid certification **Fix :** By creating SSLContext **Steps :** If Kerberos is enabled, - kinit -kt /etc/security/keytabs/atlas.service.keytab atlas/fqdn@DOMAIN - Add below to DEFAULT_JVM_OPTS in repair_index.py -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=atlas_jaas.conf If SSL is enabled, we need to make sure Solr cert or RootCA certificate is added hence use below atlas-application properties - keystore.file - truststore.file - cert.stores.credential.provider.path Diffs - intg/src/main/java/org/apache/atlas/utils/SSLUtil.java PRE-CREATION tools/atlas-index-repair/src/main/java/org/apache/atlas/tools/RepairIndex.java 37565188e Diff: https://reviews.apache.org/r/74047/diff/1/ Testing --- Manually tested on SSL enabled cluster Thanks, Pinal Shah
Re: Review Request 74047: ATLAS:4558: Fix AtlasRepairIndex tool when tls is enabled
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74047/ --- (Updated July 7, 2022, 7:21 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Radhika Kundam, and Sidharth Mishra. Changes --- added document Bugs: ATLAS-4558 https://issues.apache.org/jira/browse/ATLAS-4558 Repository: atlas Description --- **Issue :** When tls is enabled, while running atlasrepair tool, it couldn't connect to solr because it is unable to find valid certification **Fix :** By creating SSLContext **Steps :** If Kerberos is enabled, - kinit -kt /etc/security/keytabs/atlas.service.keytab atlas/fqdn@DOMAIN - Add below to DEFAULT_JVM_OPTS in repair_index.py -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=atlas_jaas.conf If SSL is enabled, we need to make sure Solr cert or RootCA certificate is added hence use below atlas-application properties - keystore.file - truststore.file - cert.stores.credential.provider.path Diffs (updated) - docs/src/documents/Tools/AtlasRepairIndex.md 416d58a89 intg/src/main/java/org/apache/atlas/utils/SSLUtil.java PRE-CREATION tools/atlas-index-repair/src/main/java/org/apache/atlas/tools/RepairIndex.java 37565188e Diff: https://reviews.apache.org/r/74047/diff/2/ Changes: https://reviews.apache.org/r/74047/diff/1-2/ Testing --- Manually tested on SSL enabled cluster Thanks, Pinal Shah
Re: Review Request 74017: ATLAS-4531 _ALL_ENTITY_TYPES showing same basic search results even when offset is changed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74017/#review224550 --- Ship it! Ship It! - Pinal Shah On June 14, 2022, 10:14 a.m., Sanket Shelar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74017/ > --- > > (Updated June 14, 2022, 10:14 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah. > > > Bugs: ATLAS-4531 > https://issues.apache.org/jira/browse/ATLAS-4531 > > > Repository: atlas > > > Description > --- > > Whenever we fire a basic search for _ALL_ENTITY_TYPES, the same set of > results are always fetched even if you change the offset. > > For ex: > If these are the results when you select page limit as 2 and offset as 0 > (there are total 59 results): > x1 > x2 > > Then on increasing the offset to 3 you still get the above results x1 and x2. > > This only occurs for _ALL_ENTITY_TYPES. If you search for specific data types > like hive_table and others the behaviour is normal. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > f45ccaf24 > > repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java > 764568915 > > > Diff: https://reviews.apache.org/r/74017/diff/2/ > > > Testing > --- > > Manual testing is done. > > Pre-commit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1103/consoleFull > > > 1) Tried all Combinations of Basic search fields. > > 2) Ran an all_entity_search search and saved it and re ran it again from > Favourite search. > > 3) Checked with different Limit and Offset Combinations. > > 4) Navigation between different pages of search in UI. > > 5) Tested wherever basic search was invoked like in Classification,Metrics > etc. > > 6) Tested using different search filters using filters and filter groups. > > > Thanks, > > Sanket Shelar > >
Re: [DRAFT] [REPORT] Apache Atlas - July 2022
Thanks Madhan, +1 for the report Regards Pinal Shah On Mon, Jul 11, 2022 at 11:50 PM Nikhil Bonte wrote: > +1 > Thanks Madhan for putting this together. > > Regards > Nikhil P. Bonte > > On Mon, Jul 11, 2022, 10:56 PM Ashutosh Mestry wrote: > > > +1 > > > > Thanks for taking care of this! > > > > ~ ashutosh > > > > On 2022/07/11 05:45:50 Madhan Neethiraj wrote: > > > Atlas community, > > > > > > Please review the following draft report for this quarter. > > > > > > Thanks, > > > Madhan > > > > > > ## Description: > > > The mission of Apache Atlas is the creation and maintenance of software > > related > > > to a scalable and extensible set of core foundational governance > > services - > > > enabling enterprises to effectively and efficiently meet their > > compliance > > > requirements > > > > > > ## Issues: > > > There are not issues requiring board attention. > > > > > > ## Membership Data: > > > Apache Atlas was founded 2017-06-20 (5 years ago) > > > There are currently 45 committers and 32 PMC members in this project. > > > The Committer-to-PMC ratio is roughly 3:2. > > > > > > Community changes, past quarter: > > > - No new PMC members. Last addition was Ashutosh Mestry on 2019-04-16. > > > - No new committers. Last addition was Radhika Kundam on 2022-02-04. > > > > > > ## Project Activity: > > > - resolved failures with Atlas HA with index-recovery enabled > > > - notification message processing improvements to enable server and > > hooks to > > > be of different versions > > > - Kafka importer utility improvements > > > - Impala hook fix to handle large query string > > > - Glossary module performance improvements to handle glossaries with > > large > > > number of categories and terms > > > - import zip functionality improvements > > > - several UI enhancements/fixes > > > - dependent component version updates: Spring 5.3.20, jQuery-UI 1.13.0, > > netty > > > 4.1.77, Spring security 5.7.1, gson 2.9.0 > > > > > > ## Community Health: > > > - dev@atlas.apache.org had a 22% decrease in traffic in the past > > quarter (680 > > > emails compared to 868) > > > - u...@atlas.apache.org had a 37% decrease in traffic in the past > > quarter (12 > > > emails compared to 19) > > > - 48 issues opened in JIRA, past quarter (-12% change) > > > - 24 issues closed in JIRA, past quarter (-20% change) > > > - 48 commits in the past quarter (-27% change) > > > - 11 code contributors in the past quarter (-15% change) > > > - 24 PRs opened on GitHub, past quarter (20% increase) > > > - 11 PRs closed on GitHub, past quarter (450% increase) > > > > > > ## Most Recent releases: > > > 2.2.0 was released on 2021-08-17 > > > 2.1.0 was released on 2020-07-15 > > > 0.8.4 was released on 2019-06-21 > > > > > > > > > > > > > > >
Re: Review Request 74055: ATLAS-4633: Multiple typos in official Apache Atlas Docs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74055/#review224563 --- Ship it! Ship It! - Pinal Shah On July 14, 2022, 5:36 a.m., Umesh Padashetty wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74055/ > --- > > (Updated July 14, 2022, 5:36 a.m.) > > > Review request for atlas, Jayendra Parab, keval bhatt, Madhan Neethiraj, > Mandar Ambawane, Nixon Rodrigues, Pinal Shah, Prasad Pawar, and Sarath > Subramanian. > > > Bugs: ATLAS-4633 > https://issues.apache.org/jira/browse/ATLAS-4633 > > > Repository: atlas > > > Description > --- > > Multiple typos in official Apache Atlas Docs > > > Diffs > - > > docs/src/documents/Asf/asfinfo.md 99f7db675 > docs/src/documents/BusinessMetadata.md 65755fb35 > docs/src/documents/Downloads/Downloads.md b8bf09250 > docs/src/documents/Glossary.md e6e1439d9 > docs/src/documents/HighAvailability.md ed0fe8b9f > docs/src/documents/Hook/HookHBase.md cbb8d81f2 > docs/src/documents/Hook/HookHive.md 0fba4f5fa > docs/src/documents/Hook/HookSqoop.md 3f600bb05 > docs/src/documents/Hook/HookStorm.md 37b23e542 > docs/src/documents/Import-Export/ExportAPI.md 849d452ab > docs/src/documents/Import-Export/ImportAPIOptions.md 1f6c8e39d > docs/src/documents/Import-Export/ImportEntityTransforms.md f1676b7c0 > docs/src/documents/Import-Export/ImportExportAPI.md c903c17a3 > docs/src/documents/Migration/Migration-0.8-to-1.0.md 7efd5752b > docs/src/documents/Misc/AtlasServer.md ee33c3cce > docs/src/documents/Misc/SoftReference.md 31ad20321 > docs/src/documents/Notifications.md 34d5c1a4b > docs/src/documents/Overview.md 3a82656ad > docs/src/documents/Project-Info/MailingLists.md 884330363 > docs/src/documents/Project-Info/TeamList.md d260fa353 > docs/src/documents/Search/SearchAdvanced.md c22981e1b > docs/src/documents/Security/AtlasRangerAuthorizer.md 9b019ec12 > docs/src/documents/Security/AtlasSimpleAuthorizer.md fc470d742 > docs/src/documents/Security/Authentication.md 3bdb5f7e8 > docs/src/documents/Security/Security.md af7d80e51 > docs/src/documents/Setup/BuildInstruction.md b067676cd > docs/src/documents/Setup/Configuration.md b72a0cc44 > docs/src/documents/Setup/EclipseSetup.md 1092e8319 > docs/src/documents/Setup/InstallationInstruction.md 100eaf51e > docs/src/documents/Tools/AtlasRepairIndex.md 0161cb1cb > docs/src/documents/TypeSystem.md 7d6125591 > docs/src/documents/Whats-New/WhatsNew-2.0.md 724e8ab31 > docs/src/documents/Whats-New/WhatsNew-2.1.md 82079f9f5 > > > Diff: https://reviews.apache.org/r/74055/diff/1/ > > > Testing > --- > > > Thanks, > > Umesh Padashetty > >
Re: Review Request 73623: ATLAS-4442 : Support Relationship search
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73623/
---
(Updated Sept. 6, 2022, 7:48 a.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath
Subramanian.
Bugs: ATLAS-4442
https://issues.apache.org/jira/browse/ATLAS-4442
Repository: atlas
Description
---
**Current Working :**
Atlas is entirely entity-based – it is not possible to search for relationships.
**Improvement :**
Get all relationships (edges) for particular relationship type, having some
criteria on its properties
**Example :**
Get all the relations matching
type = rel_type and edge_property1 = edge_value1
**Request Payload :**
{"relationshipName" : "rel_type",
"relationshipFilters": { "condition": "OR", "criterion": [
{"attributeName": "edge_property1","operator": "eq",
"attributeValue": "edge_value1" }
] },
"limit":25,
"offset":0 }
Diffs
-
common/src/main/java/org/apache/atlas/repository/Constants.java 2669c8a7e
docs/src/documents/Search/SearchRelationship.md PRE-CREATION
docs/src/resources/images/twiki/relationship_search_model.png PRE-CREATION
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
2edca64d7
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
ff3fd4a78
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java
ce0f84b6e
intg/src/main/java/org/apache/atlas/model/discovery/RelationshipSearchParameters.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
78fb4a48f
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java
ad3b98ed4
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
0f564c200
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
8fbc22fa0
repository/src/main/java/org/apache/atlas/discovery/GraphIndexQueryBuilder.java
1cd8786be
repository/src/main/java/org/apache/atlas/discovery/RelationshipSearchProcessor.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
01954d07e
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
f69dc4215
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
d65bb1a2a
repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
44cd8efff
repository/src/main/java/org/apache/atlas/repository/patches/EdgePatchProcessor.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/patches/RelationshipTypeNamePatch.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
89e9422cd
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
2fce123f8
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
a0fd71f9a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
794847555
repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
9c2975645
repository/src/test/java/org/apache/atlas/BasicTestSetup.java a1d7b62a5
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
8c257d866
repository/src/test/java/org/apache/atlas/discovery/RelationshipSearchProcessorTest.java
PRE-CREATION
repository/src/test/resources/relationship_search_def.json PRE-CREATION
test-tools/src/main/resources/solr/core-template/solrconfig.xml 3341e7199
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 0b044d4cf
Diff: https://reviews.apache.org/r/73623/diff/5/
Testing
---
added unit test cases
Thanks,
Pinal Shah
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74036/#review224646 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java Line 1170 (original), 1176 (patched) <https://reviews.apache.org/r/74036/#comment313451> Put this log inside else condition addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java Lines 1281 (patched) <https://reviews.apache.org/r/74036/#comment313452> Can we log, db qualifiedName here? addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java Lines 1284 (patched) <https://reviews.apache.org/r/74036/#comment313449> is this count needed? addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java Lines 1285 (patched) <https://reviews.apache.org/r/74036/#comment313450> As you are finding a particular database, will there be more than one db? Is this log needed? - Pinal Shah On Sept. 6, 2022, 7:21 a.m., Paresh Devalia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74036/ > --- > > (Updated Sept. 6, 2022, 7:21 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah. > > > Bugs: ATLAS-4627 > https://issues.apache.org/jira/browse/ATLAS-4627 > > > Repository: atlas > > > Description > --- > > PROBLEM STATEMENT > > Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and > hive_tez. > Created tables with 10 record in DB_1 and DB_2 . > In Atlas UI we checked it has no entity created, so we ran import-hive.sh > bash import-hive.sh -d DB_1 -t table_1, so this command will only create one > entity in atlas. > bash import-hive.sh -d DB_1, so this command will create all entity in atlas. > > Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as > hooks are disable, so we will ran import-hive command to delete from atlas . > > bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will > delete all the table from all DB which are present in Atlas and dropped from > Hive. > As we have mention -d and -t argument in command for DB and table wise > deleting, but still it ran for all DB and Tables. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 28365bc5c > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > ae7ab1a22 > > > Diff: https://reviews.apache.org/r/74036/diff/4/ > > > Testing > --- > > Manual testing is done, also testcase running successfully. > > 1) mvn clean package -Pdist,embedded-hbase-solr > Atlas server runnning > 2) Quick started was passed successfully > > Pre-commit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console > > Hive DB Testing > > 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and > hive_tez. > 2) Created tables with 10 record in DB_1 and DB_2 . > 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh > 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create > one entity in atlas. > 5) bash import-hive.sh -d DB_1, so this command will create all entity in > atlas. > 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as > hooks are disable, so we will ran import-hive command to delete from atlas > 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command > will delete only table_1 from DB_1 which are present in Atlas and dropped > from Hive. > 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command > will delete only table_1 table from DB_2 which are present in Atlas and > dropped from Hive. > 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete > all table from DB_1 which are present in Atlas and dropped from Hive. > 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete > all table from DB_2 which are present in Atlas and dropped from Hive. > 11) Checked the deleted table on statistics page in Atlas UI. > > > Thanks, > > Paresh Devalia > >
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74036/#review224647 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java Lines 1298 (patched) <https://reviews.apache.org/r/74036/#comment313453> Need to address all the above comments in this method - Pinal Shah On Sept. 6, 2022, 7:21 a.m., Paresh Devalia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74036/ > --- > > (Updated Sept. 6, 2022, 7:21 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah. > > > Bugs: ATLAS-4627 > https://issues.apache.org/jira/browse/ATLAS-4627 > > > Repository: atlas > > > Description > --- > > PROBLEM STATEMENT > > Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and > hive_tez. > Created tables with 10 record in DB_1 and DB_2 . > In Atlas UI we checked it has no entity created, so we ran import-hive.sh > bash import-hive.sh -d DB_1 -t table_1, so this command will only create one > entity in atlas. > bash import-hive.sh -d DB_1, so this command will create all entity in atlas. > > Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as > hooks are disable, so we will ran import-hive command to delete from atlas . > > bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will > delete all the table from all DB which are present in Atlas and dropped from > Hive. > As we have mention -d and -t argument in command for DB and table wise > deleting, but still it ran for all DB and Tables. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 28365bc5c > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > ae7ab1a22 > > > Diff: https://reviews.apache.org/r/74036/diff/4/ > > > Testing > --- > > Manual testing is done, also testcase running successfully. > > 1) mvn clean package -Pdist,embedded-hbase-solr > Atlas server runnning > 2) Quick started was passed successfully > > Pre-commit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console > > Hive DB Testing > > 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and > hive_tez. > 2) Created tables with 10 record in DB_1 and DB_2 . > 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh > 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create > one entity in atlas. > 5) bash import-hive.sh -d DB_1, so this command will create all entity in > atlas. > 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as > hooks are disable, so we will ran import-hive command to delete from atlas > 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command > will delete only table_1 from DB_1 which are present in Atlas and dropped > from Hive. > 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command > will delete only table_1 table from DB_2 which are present in Atlas and > dropped from Hive. > 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete > all table from DB_1 which are present in Atlas and dropped from Hive. > 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete > all table from DB_2 which are present in Atlas and dropped from Hive. > 11) Checked the deleted table on statistics page in Atlas UI. > > > Thanks, > > Paresh Devalia > >
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74036/#review224649 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java Line 1193 (original), 1204 (patched) <https://reviews.apache.org/r/74036/#comment313455> Can you put this inside else condition? - Pinal Shah On Sept. 7, 2022, 12:45 p.m., Paresh Devalia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74036/ > --- > > (Updated Sept. 7, 2022, 12:45 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah. > > > Bugs: ATLAS-4627 > https://issues.apache.org/jira/browse/ATLAS-4627 > > > Repository: atlas > > > Description > --- > > PROBLEM STATEMENT > > Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and > hive_tez. > Created tables with 10 record in DB_1 and DB_2 . > In Atlas UI we checked it has no entity created, so we ran import-hive.sh > bash import-hive.sh -d DB_1 -t table_1, so this command will only create one > entity in atlas. > bash import-hive.sh -d DB_1, so this command will create all entity in atlas. > > Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as > hooks are disable, so we will ran import-hive command to delete from atlas . > > bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will > delete all the table from all DB which are present in Atlas and dropped from > Hive. > As we have mention -d and -t argument in command for DB and table wise > deleting, but still it ran for all DB and Tables. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 28365bc5c > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > ae7ab1a22 > > > Diff: https://reviews.apache.org/r/74036/diff/6/ > > > Testing > --- > > Manual testing is done, also testcase running successfully. > > 1) mvn clean package -Pdist,embedded-hbase-solr > Atlas server runnning > 2) Quick started was passed successfully > > Pre-commit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console > > Hive DB Testing > > 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and > hive_tez. > 2) Created tables with 10 record in DB_1 and DB_2 . > 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh > 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create > one entity in atlas. > 5) bash import-hive.sh -d DB_1, so this command will create all entity in > atlas. > 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as > hooks are disable, so we will ran import-hive command to delete from atlas > 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command > will delete only table_1 from DB_1 which are present in Atlas and dropped > from Hive. > 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command > will delete only table_1 table from DB_2 which are present in Atlas and > dropped from Hive. > 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete > all table from DB_1 which are present in Atlas and dropped from Hive. > 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete > all table from DB_2 which are present in Atlas and dropped from Hive. > 11) Checked the deleted table on statistics page in Atlas UI. > > > Thanks, > > Paresh Devalia > >
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74036/#review224650 --- Ship it! Ship It! - Pinal Shah On Sept. 7, 2022, 1:02 p.m., Paresh Devalia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74036/ > --- > > (Updated Sept. 7, 2022, 1:02 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah. > > > Bugs: ATLAS-4627 > https://issues.apache.org/jira/browse/ATLAS-4627 > > > Repository: atlas > > > Description > --- > > PROBLEM STATEMENT > > Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and > hive_tez. > Created tables with 10 record in DB_1 and DB_2 . > In Atlas UI we checked it has no entity created, so we ran import-hive.sh > bash import-hive.sh -d DB_1 -t table_1, so this command will only create one > entity in atlas. > bash import-hive.sh -d DB_1, so this command will create all entity in atlas. > > Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as > hooks are disable, so we will ran import-hive command to delete from atlas . > > bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will > delete all the table from all DB which are present in Atlas and dropped from > Hive. > As we have mention -d and -t argument in command for DB and table wise > deleting, but still it ran for all DB and Tables. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java > 28365bc5c > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java > ae7ab1a22 > > > Diff: https://reviews.apache.org/r/74036/diff/7/ > > > Testing > --- > > Manual testing is done, also testcase running successfully. > > 1) mvn clean package -Pdist,embedded-hbase-solr > Atlas server runnning > 2) Quick started was passed successfully > > Pre-commit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console > > Hive DB Testing > > 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and > hive_tez. > 2) Created tables with 10 record in DB_1 and DB_2 . > 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh > 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create > one entity in atlas. > 5) bash import-hive.sh -d DB_1, so this command will create all entity in > atlas. > 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as > hooks are disable, so we will ran import-hive command to delete from atlas > 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command > will delete only table_1 from DB_1 which are present in Atlas and dropped > from Hive. > 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command > will delete only table_1 table from DB_2 which are present in Atlas and > dropped from Hive. > 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete > all table from DB_1 which are present in Atlas and dropped from Hive. > 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete > all table from DB_2 which are present in Atlas and dropped from Hive. > 11) Checked the deleted table on statistics page in Atlas UI. > > > Thanks, > > Paresh Devalia > >
Re: Review Request 74103: ATLAS-4669 : Atlas - Upgrade Junit to 4.13.2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74103/#review224654 --- addons/hbase-bridge/pom.xml Line 168 (original), 168 (patched) <https://reviews.apache.org/r/74103/#comment313456> Can we add placeholder of junit.version instead of 4.13.2? - Pinal Shah On Sept. 8, 2022, 10:01 a.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74103/ > --- > > (Updated Sept. 8, 2022, 10:01 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Bugs: ATLAS-4669 > https://issues.apache.org/jira/browse/ATLAS-4669 > > > Repository: atlas > > > Description > --- > > currently Junit version is 4.13.1 > > > Diffs > - > > addons/hbase-bridge/pom.xml a6ed51421 > pom.xml a5d4b1f9e > > > Diff: https://reviews.apache.org/r/74103/diff/2/ > > > Testing > --- > > mvn clean -DskipTests package -Pdist,embedded-hbase-solr > mvn clean install -DskipTests -Drat.skip > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1210/consoleFull > > > Thanks, > > chaitali > >
Re: Review Request 74103: ATLAS-4669 : Atlas - Upgrade Junit to 4.13.2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74103/#review224655 --- Ship it! Ship It! - Pinal Shah On Sept. 8, 2022, 11:27 a.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74103/ > --- > > (Updated Sept. 8, 2022, 11:27 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Bugs: ATLAS-4669 > https://issues.apache.org/jira/browse/ATLAS-4669 > > > Repository: atlas > > > Description > --- > > currently Junit version is 4.13.1 > > > Diffs > - > > addons/hbase-bridge/pom.xml a6ed51421 > pom.xml a5d4b1f9e > > > Diff: https://reviews.apache.org/r/74103/diff/3/ > > > Testing > --- > > mvn clean -DskipTests package -Pdist,embedded-hbase-solr > mvn clean install -DskipTests -Drat.skip > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1210/consoleFull > > > Thanks, > > chaitali > >
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74109/ --- (Updated Sept. 12, 2022, 1:43 p.m.) Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4671 https://issues.apache.org/jira/browse/ATLAS-4671 Repository: atlas Description --- **Issue** : Basic search has AtlasEntityHeader of each entity in the response. AtlasEntityHeader has many attributes including classification and terms. hence for each attribute, it will request janusgraph to get all the header attributes. **Approach** : To overcome, we can add a flag to exclude other attributes and add only selected attributes from 'attributes' field in the response Diffs - intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433 intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java 79f5aae0d intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 78fb4a48f repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 8fbc22fa0 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 01954d07e repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java fbc739652 Diff: https://reviews.apache.org/r/74109/diff/2/ Testing (updated) --- Unit testcases added Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/ Thanks, Pinal Shah
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74109/ --- (Updated Sept. 13, 2022, 1:46 p.m.) Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4671 https://issues.apache.org/jira/browse/ATLAS-4671 Repository: atlas Description --- **Issue** : Basic search has AtlasEntityHeader of each entity in the response. AtlasEntityHeader has many attributes including classification and terms. hence for each attribute, it will request janusgraph to get all the header attributes. **Approach** : To overcome, we can add a flag to exclude other attributes and add only selected attributes from 'attributes' field in the response Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433 intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java 79f5aae0d intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 78fb4a48f repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 8fbc22fa0 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 01954d07e repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java fbc739652 Diff: https://reviews.apache.org/r/74109/diff/3/ Changes: https://reviews.apache.org/r/74109/diff/2-3/ Testing --- Unit testcases added Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/ Thanks, Pinal Shah
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
> On Sept. 13, 2022, 4:44 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> > Lines 373 (patched)
> > <https://reviews.apache.org/r/74109/diff/3/?file=2269964#file2269964line373>
> >
> > It is not clear why entity-ref attributes are not allowed when
> > excludeHeaderAttributes is true. Is it for faster query response i.e., to
> > avoid look up? Given the attribute is explicitly asked for, the caller
> > should know the performance impact. Why not allow this case?
Whenever "excludeHeaderAttributes":"true", response is bit different
Example:
request: "attributes":["qualifiedName"], "excludeHeaderAttributes":"true"
response: {
"entities": [],
"attributes": {
"name": ["qualifiedName"],
"values": [
["sales@sys"],
["customer@sys"]
]
}
}
List entities, will be blank
AttributeSearchResult attributes, populate attribute values here.
Now case when
request: "attributes":["qualifiedName","db"], "excludeHeaderAttributes":"true"
We need to fit below "db" value json, to AttributeSearchResult
"db": {
"guid": "6e44a899-320c-4026-a66a-d67a76ef7c41",
"typeName": "hive_db",
"uniqueAttributes": {
"qualifiedName": "sys@cl1"
}
}
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/#review224662
---
On Sept. 13, 2022, 1:46 p.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74109/
> ---
>
> (Updated Sept. 13, 2022, 1:46 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4671
> https://issues.apache.org/jira/browse/ATLAS-4671
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Issue** : Basic search has AtlasEntityHeader of each entity in the response.
> AtlasEntityHeader has many attributes including classification and terms.
> hence for each attribute, it will request janusgraph to get all the header
> attributes.
>
> **Approach** : To overcome, we can add a flag to exclude other attributes and
> add only selected attributes from 'attributes' field in the response
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
>
> intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
> 79f5aae0d
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
> 78fb4a48f
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 8fbc22fa0
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> 01954d07e
>
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> fbc739652
>
>
> Diff: https://reviews.apache.org/r/74109/diff/3/
>
>
> Testing
> ---
>
> Unit testcases added
> Precommit :
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/
---
(Updated Sept. 14, 2022, 5:38 a.m.)
Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
Bugs: ATLAS-4671
https://issues.apache.org/jira/browse/ATLAS-4671
Repository: atlas
Description (updated)
---
**Issue** : Basic search has AtlasEntityHeader of each entity in the response.
AtlasEntityHeader has many attributes including classification and terms. hence
for each attribute, it will request janusgraph
**Approach** : To overcome, we can add a flag to exclude other attributes and
add only selected attributes from 'attributes' field in the response
**Request**: {
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"limit": 25,
"offset": 0,
"typeName": "hdfs_path",
"attributes": ["path", "name"],
"excludeHeaderAttributes": "true"
}
**Response**: {
"queryType": "BASIC",
"searchParameters": {
"typeName": "hdfs_path",
"excludeDeletedEntities": true,
"includeClassificationAttributes": true,
"includeSubTypes": true,
"includeSubClassifications": true,
"limit": 25,
"offset": 0,
"attributes": ["path", "name"]
},
"attributes": {
"name": ["path", "name"],
"values": [
["/data/warehouse/customer", "customer"],
["/data/warehouse/sales", "sales"]
]
},
"approximateCount": 2
}
Diffs
-
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
79f5aae0d
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
78fb4a48f
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
8fbc22fa0
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
01954d07e
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
fbc739652
Diff: https://reviews.apache.org/r/74109/diff/3/
Testing (updated)
---
Unit testcases added
Precommit :
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
Performance Readings:
Basic Search on 20K entities, limit 1000
Thanks,
Pinal Shah
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
> On Sept. 13, 2022, 4:44 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> > Lines 373 (patched)
> > <https://reviews.apache.org/r/74109/diff/3/?file=2269964#file2269964line373>
> >
> > It is not clear why entity-ref attributes are not allowed when
> > excludeHeaderAttributes is true. Is it for faster query response i.e., to
> > avoid look up? Given the attribute is explicitly asked for, the caller
> > should know the performance impact. Why not allow this case?
>
> Pinal Shah wrote:
> Whenever "excludeHeaderAttributes":"true", response is bit different
>
> Example:
> request: "attributes":["qualifiedName"], "excludeHeaderAttributes":"true"
> response: {
> "entities": [],
> "attributes": {
> "name": ["qualifiedName"],
> "values": [
> ["sales@sys"],
> ["customer@sys"]
> ]
> }
> }
>
> List entities, will be blank
> AttributeSearchResult attributes, populate attribute values here.
>
>
> Now case when
> request: "attributes":["qualifiedName","db"],
> "excludeHeaderAttributes":"true"
> We need to fit below "db" value json, to AttributeSearchResult
> "db": {
> "guid": "6e44a899-320c-4026-a66a-d67a76ef7c41",
> "typeName": "hive_db",
> "uniqueAttributes": {
> "qualifiedName": "sys@cl1"
> }
>}
>
> Madhan Neethiraj wrote:
> AtlasSearchResult.values is of type List>, hence should be
> able to handle obj-ref values as well - right?
yes Madhan
Thanks for reviewing.i will add this support of referred attributes
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/#review224662
---
On Sept. 14, 2022, 5:38 a.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74109/
> ---
>
> (Updated Sept. 14, 2022, 5:38 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4671
> https://issues.apache.org/jira/browse/ATLAS-4671
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Issue** : Basic search has AtlasEntityHeader of each entity in the response.
> AtlasEntityHeader has many attributes including classification and terms.
> hence for each attribute, it will request janusgraph
>
> **Approach** : To overcome, we can add a flag to exclude other attributes and
> add only selected attributes from 'attributes' field in the response
>
> **Request**: {
> "excludeDeletedEntities": true,
> "includeSubClassifications": true,
> "includeSubTypes": true,
> "includeClassificationAttributes": true,
> "limit": 25,
> "offset": 0,
> "typeName": "hdfs_path",
> "attributes": ["path", "name"],
> "excludeHeaderAttributes": "true"
>}
>
> **Response**: {
> "queryType": "BASIC",
> "searchParameters": {
> "typeName": "hdfs_path",
> "excludeDeletedEntities": true,
> "includeClassificationAttributes": true,
> "includeSubTypes": true,
> "includeSubClassifications": true,
> "limit": 25,
> "offset": 0,
> "attributes": ["path", "name"]
> },
> "attributes": {
> "name": ["path", "name"],
> "values": [
> ["/data/warehouse/customer", "customer"],
> ["/data/warehouse/sales", "sales"]
> ]
> },
> "approximateCount": 2
> }
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
>
> intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
> 79f5aae0d
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
> 78fb4a48f
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 8fbc22fa0
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> 01954d07e
>
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> fbc739652
>
>
> Diff: https://reviews.apache.org/r/74109/diff/3/
>
>
> Testing
> ---
>
> Unit testcases added
> Precommit :
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
>
> Performance Readings:
> Basic Search on 20K entities, limit 1000
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/
---
(Updated Sept. 14, 2022, 12:11 p.m.)
Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
Changes
---
addressed review comments (supported referred attributes)
Bugs: ATLAS-4671
https://issues.apache.org/jira/browse/ATLAS-4671
Repository: atlas
Description
---
**Issue** : Basic search has AtlasEntityHeader of each entity in the response.
AtlasEntityHeader has many attributes including classification and terms. hence
for each attribute, it will request janusgraph
**Approach** : To overcome, we can add a flag to exclude other attributes and
add only selected attributes from 'attributes' field in the response
**Request**: {
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"limit": 25,
"offset": 0,
"typeName": "hdfs_path",
"attributes": ["path", "name"],
"excludeHeaderAttributes": "true"
}
**Response**: {
"queryType": "BASIC",
"searchParameters": {
"typeName": "hdfs_path",
"excludeDeletedEntities": true,
"includeClassificationAttributes": true,
"includeSubTypes": true,
"includeSubClassifications": true,
"limit": 25,
"offset": 0,
"attributes": ["path", "name"]
},
"attributes": {
"name": ["path", "name"],
"values": [
["/data/warehouse/customer", "customer"],
["/data/warehouse/sales", "sales"]
]
},
"approximateCount": 2
}
Diffs (updated)
-
intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
79f5aae0d
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
78fb4a48f
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
8fbc22fa0
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
01954d07e
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
a8fe5a762
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
fbc739652
Diff: https://reviews.apache.org/r/74109/diff/4/
Changes: https://reviews.apache.org/r/74109/diff/3-4/
Testing
---
Unit testcases added
Precommit :
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
Performance Readings:
Basic Search on 20K entities, limit 1000
Thanks,
Pinal Shah
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
> On Sept. 14, 2022, 11:11 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Lines 478 (patched)
> > <https://reviews.apache.org/r/74109/diff/4/?file=2269974#file2269974line478>
> >
> > Use of HashSet here will eliminate duplicate rows. Is this by choice?
> > Wouldn't this result in incorrect handling of pagination (offset)?
These elimination happens after handling pagination and getting the resultSet.
but yes, you are correct, from user end it will be confusing, that they have
requested for 10, and have got less.
> On Sept. 14, 2022, 11:11 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Lines 507 (patched)
> > <https://reviews.apache.org/r/74109/diff/4/?file=2269974#file2269974line507>
> >
> > return here would result in skipping of scrubSearchResults() - which is
> > incorrect. Please review and update.
> >
> > Instead of introducing #471 - #509, consider updating existing for loop
> > below at #549, like:
> >
> > for (AtlasVertex atlasVertex : resultList) {
> > if (searchContext.excludeHeaderAttributes()) {
> > // code from #475 - #506
> > } else {
> > // existing code
> > ...
> > }
> > }
Correct, but we are skipping many attributes, i.e classifications and also we
don't have 'entities' (we just have the 'attributes') in AtlasSearchResult
I thought to align this with DSL search, Select Clause where AtlasSearchResult
has only 'attributes' field.
If we add support to scrubSearchResult
- we need to create 'entities' from what information
- we need to get 'guid' from the vertex
- we need to get 'classifications'
- we need to add 'referredEntities'
getting classifications and referredEntities will reduce the performance which
we are expecting with this improvement.
I will try implementing this once.
Any suugestions?
> On Sept. 14, 2022, 11:11 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
> > Line 1356 (original), 1356 (patched)
> > <https://reviews.apache.org/r/74109/diff/4/?file=2269976#file2269976line1356>
> >
> > Consider retaining getVertexAttribute() as private, based on comment in
> > EntityDiscoveryService.java #494.
comment on #494,
By entityRetriever.getVertexAttribute();
do you mean entityRetriever.getVertexAttribute(vertex, attribute); ?
We dont have any existing method, other than this which returns the value of
attribute.
And to access this in EntityDiscoveryService, we need to make this as public
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/#review224671
---
On Sept. 14, 2022, 12:11 p.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74109/
> ---
>
> (Updated Sept. 14, 2022, 12:11 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4671
> https://issues.apache.org/jira/browse/ATLAS-4671
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Issue** : Basic search has AtlasEntityHeader of each entity in the response.
> AtlasEntityHeader has many attributes including classification and terms.
> hence for each attribute, it will request janusgraph
>
> **Approach** : To overcome, we can add a flag to exclude other attributes and
> add only selected attributes from 'attributes' field in the response
>
> **Request**: {
> "excludeDeletedEntities": true,
> "includeSubClassifications": true,
> "includeSubTypes": true,
> "includeClassificationAttributes": true,
> "limit": 25,
> "offset": 0,
> "typeName": "hdfs_path",
> "attributes": ["path", "name"],
> "excludeHeaderAttributes": "true"
>}
>
> **Response**: {
> "queryType": "BASIC",
> "searchParameters": {
> "typeName": "hdfs_path",
> "excludeDeletedEntities": true,
>
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/
---
(Updated Sept. 21, 2022, 9:48 a.m.)
Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
Changes
---
addressed review comments
Bugs: ATLAS-4671
https://issues.apache.org/jira/browse/ATLAS-4671
Repository: atlas
Description (updated)
---
**Issue** : Basic search has AtlasEntityHeader of each entity in the response.
AtlasEntityHeader has many attributes including classification and terms. hence
for each attribute, it will request janusgraph
**Approach** : To overcome, we can add a flag to exclude other attributes and
add only selected attributes from 'attributes' field in the response
**Working** : This improvement can be seen, if user has **entityTypes**,
**excludeHeaderAttributes=true** and **valid entity attributes (not
relationship) in 'attributes' field** , in the request payload.
**Request**: {
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"limit": 25,
"offset": 0,
"typeName": "hdfs_path",
"attributes": ["path", "name"],
"excludeHeaderAttributes": "true"
}
**Response**: {
"queryType": "BASIC",
"searchParameters": {
"typeName": "hdfs_path",
"excludeDeletedEntities": true,
"includeClassificationAttributes": true,
"includeSubTypes": true,
"includeSubClassifications": true,
"limit": 25,
"offset": 0,
"attributes": ["path", "name"]
},
"attributes": {
"name": ["path", "name"],
"values": [
["/data/warehouse/customer", "customer"],
["/data/warehouse/sales", "sales"]
]
},
"approximateCount": 2
}
Diffs (updated)
-
intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
79f5aae0d
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
78fb4a48f
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
8fbc22fa0
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
01954d07e
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
fbc739652
Diff: https://reviews.apache.org/r/74109/diff/6/
Changes: https://reviews.apache.org/r/74109/diff/5-6/
Testing
---
Unit testcases added
Precommit :
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
Performance Readings:
Basic Search on 20K entities, limit 1000
Thanks,
Pinal Shah
