Re: Review Request 72474: ATLAS-3783 : DSL query search should return results for both the relationship edge directions

2020-05-13 Thread chaitali

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72474/
---

(Updated May 14, 2020, 5:46 a.m.)


Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Bugs: ATLAS-3783
https://issues.apache.org/jira/browse/ATLAS-3783


Repository: atlas


Description
---

Unable to get results for Dsl query eg;hive_column where table.name="employee".
Due to regression only one edge direction was giving results.
This patch gives a fix for results for both relationship edge direction queries.


Diffs (updated)
-

  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
04b177533 
  repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
6ab61e16b 
  repository/src/main/java/org/apache/atlas/query/Lookup.java 3c192d824 
  repository/src/main/java/org/apache/atlas/query/RegistryBasedLookup.java 
2b34abff8 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 0bbff2f46 
  repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
6f4df88f0 


Diff: https://reviews.apache.org/r/72474/diff/11/

Changes: https://reviews.apache.org/r/72474/diff/10-11/


Testing
---

TestCases added.


Thanks,

chaitali



Re: Review Request 72489: ATLAS-3788 : BasicSearch: Classification with System attribute(indexed) filters has pagination issue

2020-05-13 Thread Pinal Shah

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72489/
---

(Updated May 14, 2020, 5:46 a.m.)


Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath 
Subramanian.


Changes
---

added license to new file


Bugs: ATLAS-3788
https://issues.apache.org/jira/browse/ATLAS-3788


Repository: atlas


Description
---

When selecting ALL_CLASSIFICATION_TYPES, with system attribute Created Time < 
currentTimeStamp , it results to less or zero number of entities than actual.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 74c088c0e 
  repository/src/test/java/org/apache/atlas/BasicTestSetup.java 958781e41 
  
repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/72489/diff/3/

Changes: https://reviews.apache.org/r/72489/diff/2-3/


Testing
---

Manually Tested.
Will add Precommit Link.


Thanks,

Pinal Shah



Re: Review Request 72440: Support sort params for FreeTextSearchProcessor

2020-05-13 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72440/#review220753
---


Ship it!




Ship It!

- Madhan Neethiraj


On May 13, 2020, 10:51 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72440/
> ---
> 
> (Updated May 13, 2020, 10:51 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
> and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> No way to sort results by specified attribute while freetext search is 
> enabled. In our case we would like to enforce ordering by introducing custom 
> attribute definition e.g. popularity score from 
> https://github.com/dwarszawski/amundsen-atlas-types/blob/master/amundsenatlastypes/schema/01_2_table_schema.json
> 
> 
> Reference to jira https://issues.apache.org/jira/browse/ATLAS-3758
> Patched applied against master branch.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  fb12244ed 
>   
> repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java
>  9850d8ecf 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 11eb7ca49 
>   
> repository/src/test/java/org/apache/atlas/discovery/FreeTextSearchProcessorTest.java
>  PRE-CREATION 
>   test-tools/src/main/resources/solr/core-template/solrconfig.xml 9264f99d4 
> 
> 
> Diff: https://reviews.apache.org/r/72440/diff/2/
> 
> 
> Testing
> ---
> 
> Patch was applied on our dev env with custom entity definitions and 
> successfully verified if order is applied as specified in the search query.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



Re: Review Request 72440: Support sort params for FreeTextSearchProcessor

2020-05-13 Thread Damian Warszawski


> On May 7, 2020, 12:21 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
> > Lines 995 (patched)
> > 
> >
> > entityType could be null when called from FreeTextSearchProcessor. 
> > Please review and update to handle this condition.

Good point, additional check added.


- Damian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72440/#review220666
---


On May 13, 2020, 10:51 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72440/
> ---
> 
> (Updated May 13, 2020, 10:51 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
> and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> No way to sort results by specified attribute while freetext search is 
> enabled. In our case we would like to enforce ordering by introducing custom 
> attribute definition e.g. popularity score from 
> https://github.com/dwarszawski/amundsen-atlas-types/blob/master/amundsenatlastypes/schema/01_2_table_schema.json
> 
> 
> Reference to jira https://issues.apache.org/jira/browse/ATLAS-3758
> Patched applied against master branch.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  fb12244ed 
>   
> repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java
>  9850d8ecf 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 11eb7ca49 
>   
> repository/src/test/java/org/apache/atlas/discovery/FreeTextSearchProcessorTest.java
>  PRE-CREATION 
>   test-tools/src/main/resources/solr/core-template/solrconfig.xml 9264f99d4 
> 
> 
> Diff: https://reviews.apache.org/r/72440/diff/2/
> 
> 
> Testing
> ---
> 
> Patch was applied on our dev env with custom entity definitions and 
> successfully verified if order is applied as specified in the search query.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



Re: Review Request 72440: Support sort params for FreeTextSearchProcessor

2020-05-13 Thread Damian Warszawski

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72440/
---

(Updated May 13, 2020, 10:51 p.m.)


Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
and Sarath Subramanian.


Changes
---

add unit tests for freetext search processor


Repository: atlas


Description
---

No way to sort results by specified attribute while freetext search is enabled. 
In our case we would like to enforce ordering by introducing custom attribute 
definition e.g. popularity score from 
https://github.com/dwarszawski/amundsen-atlas-types/blob/master/amundsenatlastypes/schema/01_2_table_schema.json


Reference to jira https://issues.apache.org/jira/browse/ATLAS-3758
Patched applied against master branch.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
fb12244ed 
  
repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java
 9850d8ecf 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
11eb7ca49 
  
repository/src/test/java/org/apache/atlas/discovery/FreeTextSearchProcessorTest.java
 PRE-CREATION 
  test-tools/src/main/resources/solr/core-template/solrconfig.xml 9264f99d4 


Diff: https://reviews.apache.org/r/72440/diff/2/

Changes: https://reviews.apache.org/r/72440/diff/1-2/


Testing
---

Patch was applied on our dev env with custom entity definitions and 
successfully verified if order is applied as specified in the search query.


Thanks,

Damian Warszawski



Re: Review Request 72440: Support sort params for FreeTextSearchProcessor

2020-05-13 Thread Damian Warszawski


> On May 6, 2020, 6:57 a.m., Bolke de Bruin wrote:
> > Can you please update the tests?
> 
> Sarath Subramanian wrote:
> +1

introduced unit tests for FreetextSearchProcessor


- Damian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72440/#review220648
---


On May 13, 2020, 10:51 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72440/
> ---
> 
> (Updated May 13, 2020, 10:51 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, Madhan Neethiraj, 
> and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> No way to sort results by specified attribute while freetext search is 
> enabled. In our case we would like to enforce ordering by introducing custom 
> attribute definition e.g. popularity score from 
> https://github.com/dwarszawski/amundsen-atlas-types/blob/master/amundsenatlastypes/schema/01_2_table_schema.json
> 
> 
> Reference to jira https://issues.apache.org/jira/browse/ATLAS-3758
> Patched applied against master branch.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  fb12244ed 
>   
> repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java
>  9850d8ecf 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 11eb7ca49 
>   
> repository/src/test/java/org/apache/atlas/discovery/FreeTextSearchProcessorTest.java
>  PRE-CREATION 
>   test-tools/src/main/resources/solr/core-template/solrconfig.xml 9264f99d4 
> 
> 
> Diff: https://reviews.apache.org/r/72440/diff/2/
> 
> 
> Testing
> ---
> 
> Patch was applied on our dev env with custom entity definitions and 
> successfully verified if order is applied as specified in the search query.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



[jira] [Updated] (ATLAS-3798) Import Service: Percentage Display Problem with Large Datasets

2020-05-13 Thread Ashutosh Mestry (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-3798:
---
Description: 
*Steps to Duplicate*
 # Start migration import with 100M entities. Observe the log for 
_last-imported_.

_Expected output:_ Display should proceed in increasing order of %ages.

_Actual output_: Display starts showing -ve %ages after 37%.

*Root Cause*

Existing implementation uses _integer_ for storing _currentIndex_.

*Additional Information*
 * Start position computation is not impacted.
 * Resume is not impacted. 

  was:
*Steps to Duplicate*
 # Start migration import with 100M entities. Observe the log for 
_last-imported_.

_Expected output:_ Display should proceed in increasing order of %ages.

_Actual output_: Display starts showing -ve %ages after 37%.

*Root Cause*

Existing implementation uses _integer_ for storing _currentIndex_.


> Import Service: Percentage Display Problem with Large Datasets
> --
>
> Key: ATLAS-3798
> URL: https://issues.apache.org/jira/browse/ATLAS-3798
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Minor
> Fix For: trunk
>
>
> *Steps to Duplicate*
>  # Start migration import with 100M entities. Observe the log for 
> _last-imported_.
> _Expected output:_ Display should proceed in increasing order of %ages.
> _Actual output_: Display starts showing -ve %ages after 37%.
> *Root Cause*
> Existing implementation uses _integer_ for storing _currentIndex_.
> *Additional Information*
>  * Start position computation is not impacted.
>  * Resume is not impacted. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3798) Import Service: Percentage Display Problem with Large Datasets

2020-05-13 Thread Ashutosh Mestry (Jira)
Ashutosh Mestry created ATLAS-3798:
--

 Summary: Import Service: Percentage Display Problem with Large 
Datasets
 Key: ATLAS-3798
 URL: https://issues.apache.org/jira/browse/ATLAS-3798
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: trunk
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
 Fix For: trunk


*Steps to Duplicate*
 # Start migration import with 100M entities. Observe the log for 
_last-imported_.

_Expected output:_ Display should proceed in increasing order of %ages.

_Actual output_: Display starts showing -ve %ages after 37%.

*Root Cause*

Existing implementation uses _integer_ for storing _currentIndex_.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3796) Migration Import Display: StatusReporter Does not Remove Processed Items

2020-05-13 Thread Sarath Subramanian (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106704#comment-17106704
 ] 

Sarath Subramanian commented on ATLAS-3796:
---

+1 for the patch 
[^ATLAS-3796-Status-Reporter-processed-set-cleanup.patch]Thanks [~amestry].

> Migration Import Display: StatusReporter Does not Remove Processed Items
> 
>
> Key: ATLAS-3796
> URL: https://issues.apache.org/jira/browse/ATLAS-3796
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3796-Status-Reporter-processed-set-cleanup.patch
>
>
> *Background*
> _Migration Import_ implementation introduced _StatusReporter_. The class 
> maintains GUIDs of entities that are to be processed and entities that are 
> processed. When the entity is marked as processed, it needs to be removed 
> from the processed set.
> *Steps to Duplicate*
> _Pre-requisites_
> Large migration import dataset (over 50 M entities).
> _Steps to Duplicate_
>  * Start importing the dataset.
>  * After over 60%, the Atlas server will crash with out of memory exception.
> _Expected results_: Migration Import should continue without problems.
> _Actual results_: Atlas server crashes with out of memory exception, hprof 
> dump indicates large number of items allocated by 
> _StatusReporter.processedSet_.
> _Additional information_: This bug does not affect status reporting.
> _Work around_: Resume migration import. It will continue without problems.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72477: ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, UPDATE and DELETE

2020-05-13 Thread Mandar Ambawane

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72477/
---

(Updated May 13, 2020, 9:13 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
Sarath Subramanian, and Sidharth Mishra.


Bugs: ATLAS-3583
https://issues.apache.org/jira/browse/ATLAS-3583


Repository: atlas


Description
---

ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, 
UPDATE and DELETE


Diffs (updated)
-

  addons/models/-Area0/0010-base_model.json 9777b7a 
  intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java a95cf4e 
  intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java 
9120062 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 2b9cf6e 
  
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
 a0dc816 
  
repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 79f5270 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 0dc3193 
  repository/src/test/java/org/apache/atlas/TestModules.java a298934 
  webapp/src/test/java/org/apache/atlas/web/adapters/TestTypeDefsREST.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72477/diff/3/

Changes: https://reviews.apache.org/r/72477/diff/2-3/


Testing (updated)
---

Basic testing is done.

Pre-commit: 
https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1871/console

Pre-commit: 
https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1885/console

Pre-commit: 
https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1888/console


Thanks,

Mandar Ambawane



[jira] [Updated] (ATLAS-3786) ECCN for Apache Atlas

2020-05-13 Thread Gregory William (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory William updated ATLAS-3786:
---
Component/s:  atlas-core

> ECCN for Apache Atlas
> -
>
> Key: ATLAS-3786
> URL: https://issues.apache.org/jira/browse/ATLAS-3786
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Reporter: Gregory William
>Priority: Major
>  Labels: export_control, legal
>
> Apache Atlas is not listed in the ASF Export Classifications Matrix 
> ([http://www.apache.org/licenses/exports/#matrix).]  Please advise if this 
> product has been classified and assigned to an ECCN



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3797) Entity Creation: Refactoring: Improve Edge Creation

2020-05-13 Thread Ashutosh Mestry (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-3797:
---
Attachment: ATLAS-3797-Edge-creation-refactoring.patch

> Entity Creation: Refactoring: Improve Edge Creation
> ---
>
> Key: ATLAS-3797
> URL: https://issues.apache.org/jira/browse/ATLAS-3797
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3797-Edge-creation-refactoring.patch
>
>
> *Background*
> [ATLAS-3762|https://issues.apache.org/jira/browse/ATLAS-3762] introduced edge 
> creation improvement by improving on edge existence check.
> The same change could be applied to another area within the implementation.
> *Investigation*
> This was discovered as part of volume testing done for high-volume import. 
> This included importing over 100M entities. Various parts of the entity 
> creation process had probes added to check for performance.
> During this investigation the edge creation was doing the same existence 
> check that was addressed in earlier fixes. Re-using the same implementation 
> showed marked improvement in entity creation.
> These were the parameters used for during import.
> Node configuration: 128 GB RAM, 40 cores.
> {code:java}
> atlas.migration.mode.batch.size=1500
> atlas.migration.mode.workers=40{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72489: ATLAS-3788 : BasicSearch: Classification with System attribute(indexed) filters has pagination issue

2020-05-13 Thread Madhan Neethiraj

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72489/#review220742
---


Ship it!




Ship It!

- Madhan Neethiraj


On May 13, 2020, 5:04 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72489/
> ---
> 
> (Updated May 13, 2020, 5:04 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3788
> https://issues.apache.org/jira/browse/ATLAS-3788
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When selecting ALL_CLASSIFICATION_TYPES, with system attribute Created Time < 
> currentTimeStamp , it results to less or zero number of entities than actual.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  74c088c0e 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 958781e41 
>   
> repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72489/diff/2/
> 
> 
> Testing
> ---
> 
> Manually Tested.
> Will add Precommit Link.
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



[jira] [Created] (ATLAS-3797) Entity Creation: Refactoring: Improve Edge Creation

2020-05-13 Thread Ashutosh Mestry (Jira)
Ashutosh Mestry created ATLAS-3797:
--

 Summary: Entity Creation: Refactoring: Improve Edge Creation
 Key: ATLAS-3797
 URL: https://issues.apache.org/jira/browse/ATLAS-3797
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: trunk
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
 Fix For: trunk


*Background*

[ATLAS-3762|https://issues.apache.org/jira/browse/ATLAS-3762] introduced edge 
creation improvement by improving on edge existence check.

The same change could be applied to another area within the implementation.

*Investigation*

This was discovered as part of volume testing done for high-volume import. This 
included importing over 100M entities. Various parts of the entity creation 
process had probes added to check for performance.

During this investigation the edge creation was doing the same existence check 
that was addressed in earlier fixes. Re-using the same implementation showed 
marked improvement in entity creation.

These were the parameters used for during import.

Node configuration: 128 GB RAM, 40 cores.
{code:java}
atlas.migration.mode.batch.size=1500
atlas.migration.mode.workers=40{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3796) Migration Import Display: StatusReporter Does not Remove Processed Items

2020-05-13 Thread Ashutosh Mestry (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-3796:
---
Attachment: ATLAS-3796-Status-Reporter-processed-set-cleanup.patch

> Migration Import Display: StatusReporter Does not Remove Processed Items
> 
>
> Key: ATLAS-3796
> URL: https://issues.apache.org/jira/browse/ATLAS-3796
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3796-Status-Reporter-processed-set-cleanup.patch
>
>
> *Background*
> _Migration Import_ implementation introduced _StatusReporter_. The class 
> maintains GUIDs of entities that are to be processed and entities that are 
> processed. When the entity is marked as processed, it needs to be removed 
> from the processed set.
> *Steps to Duplicate*
> _Pre-requisites_
> Large migration import dataset (over 50 M entities).
> _Steps to Duplicate_
>  * Start importing the dataset.
>  * After over 60%, the Atlas server will crash with out of memory exception.
> _Expected results_: Migration Import should continue without problems.
> _Actual results_: Atlas server crashes with out of memory exception, hprof 
> dump indicates large number of items allocated by 
> _StatusReporter.processedSet_.
> _Additional information_: This bug does not affect status reporting.
> _Work around_: Resume migration import. It will continue without problems.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3796) Migration Import Display: StatusReporter Does not Remove Processed Items

2020-05-13 Thread Ashutosh Mestry (Jira)
Ashutosh Mestry created ATLAS-3796:
--

 Summary: Migration Import Display: StatusReporter Does not Remove 
Processed Items
 Key: ATLAS-3796
 URL: https://issues.apache.org/jira/browse/ATLAS-3796
 Project: Atlas
  Issue Type: Bug
  Components: atlas-intg
Affects Versions: trunk
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
 Fix For: trunk


*Background*

_Migration Import_ implementation introduced _StatusReporter_. The class 
maintains GUIDs of entities that are to be processed and entities that are 
processed. When the entity is marked as processed, it needs to be removed from 
the processed set.

*Steps to Duplicate*

_Pre-requisites_

Large migration import dataset (over 50 M entities).

_Steps to Duplicate_
 * Start importing the dataset.
 * After over 60%, the Atlas server will crash with out of memory exception.

_Expected results_: Migration Import should continue without problems.

_Actual results_: Atlas server crashes with out of memory exception, hprof dump 
indicates large number of items allocated by _StatusReporter.processedSet_.

_Additional information_: This bug does not affect status reporting.

_Work around_: Resume migration import. It will continue without problems.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72489: ATLAS-3788 : BasicSearch: Classification with System attribute(indexed) filters has pagination issue

2020-05-13 Thread Pinal Shah

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72489/
---

(Updated May 13, 2020, 5:04 p.m.)


Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath 
Subramanian.


Changes
---

Added unit test cases for basic search of classification


Bugs: ATLAS-3788
https://issues.apache.org/jira/browse/ATLAS-3788


Repository: atlas


Description
---

When selecting ALL_CLASSIFICATION_TYPES, with system attribute Created Time < 
currentTimeStamp , it results to less or zero number of entities than actual.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 74c088c0e 
  repository/src/test/java/org/apache/atlas/BasicTestSetup.java 958781e41 
  
repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/72489/diff/2/

Changes: https://reviews.apache.org/r/72489/diff/1-2/


Testing (updated)
---

Manually Tested.
Will add Precommit Link.


Thanks,

Pinal Shah



Re: Review Request 72477: ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, UPDATE and DELETE

2020-05-13 Thread Mandar Ambawane

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72477/
---

(Updated May 13, 2020, 3:02 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
Sarath Subramanian, and Sidharth Mishra.


Changes
---

Changes to store entire TypeDef json in AuditEntry.
Added Test cases.


Bugs: ATLAS-3583
https://issues.apache.org/jira/browse/ATLAS-3583


Repository: atlas


Description
---

ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, 
UPDATE and DELETE


Diffs (updated)
-

  addons/models/-Area0/0010-base_model.json 9777b7a 
  intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java a95cf4e 
  intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java 
9120062 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 2b9cf6e 
  
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
 a0dc816 
  
repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 79f5270 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 0dc3193 
  repository/src/test/java/org/apache/atlas/TestModules.java a298934 
  webapp/src/test/java/org/apache/atlas/web/adapters/TestTypeDefsREST.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72477/diff/2/

Changes: https://reviews.apache.org/r/72477/diff/1-2/


Testing (updated)
---

Basic testing is done.

Pre-commit: 
https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1871/console

Pre-commit: 
https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1885/console


Thanks,

Mandar Ambawane



[jira] [Assigned] (ATLAS-3551) Atlas unable to import entities if an HBase column family is deleted

2020-05-13 Thread chaitali borole (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chaitali borole reassigned ATLAS-3551:
--

Assignee: chaitali borole

> Atlas unable to import entities if an HBase column family is deleted
> 
>
> Key: ATLAS-3551
> URL: https://issues.apache.org/jira/browse/ATLAS-3551
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nixon Rodrigues
>Assignee: chaitali borole
>Priority: Minor
>
> *Description*
> Importing Hbase entities fails if an Hbase column family was previously 
> deleted.
> *Steps to reproduce:*
> kinit as user hbase
> create a table:
>  
> {noformat}
> $ echo "create 'default:hbase_table_csaba2' 
> ,{NAME=>'col_fam_csaba2_1'},{NAME=>'col_fam_csaba2_2'}" | 
> /usr/hdp/current/hbase-client/bin/hbase shell -n
> Created table default:hbase_table_csaba2
> Took 1.2510 seconds
> Hbase::Table - hbase_table_csaba2
> {noformat}
> import the table:
> {noformat}
> $ /usr/hdp/current/atlas-server/hook-bin/import-hbase.sh -t 
> hbase_table_csaba2 -Datlas.conf=/etc/hbase/conf
> HBase Data Model imported successfully!!!
>  
> {noformat}
> delete one of the column families:
> {noformat}
> $ echo "alter 'default:hbase_table_csaba2', NAME => 'col_fam_csaba2_1', 
> METHOD=>'delete'" | /usr/hdp/current/hbase-client/bin/hbase shell -n
> Updating all regions with the new schema...
>  1/1 regions updated.
>  Done.
>  Took 2.5881 seconds
>  
> {noformat}
> import the table again:
> {noformat}
> $ /usr/hdp/current/atlas-server/hook-bin/import-hbase.sh -t 
> hbase_table_csaba2 -Datlas.conf=/etc/hbase/conf
>  ImportHBaseEntities failed. Please check the log file for the detailed error 
> message
>  Failed to import HBase Data Model!!!
>  Log file content: (/grid/0/log/import-hbase.log):
> 2019-10-03 12:41:28,391 INFO - [main:] ~ Table already present in Atlas. 
> Updating it..: default:hbase_table_csaba2@cl1 (HBaseBridge:403)
>  2019-10-03 12:41:28,416 INFO - [main:] ~ method=POST 
> path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 
> accept=application/json status=404 (AtlasBaseClient:387)
>  2019-10-03 12:41:28,418 ERROR - [main:] ~ ImportHBaseEntities failed 
> (HBaseBridge:196)
>  org.apache.atlas.AtlasServiceException: Metadata service API 
> org.apache.atlas.AtlasClientV2$API_V2@2a389173 failed with status 404 (Not 
> Found) Response Body (
> {"errorCode":"ATLAS-404-00-007","errorMessage":"Invalid instance 
> creation/updation parameters passed : hbase_column_family.table: mandatory 
> attribute value missing in type hbase_column_family"}
> )
>  at 
> org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:427)
>  at 
> org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:353)
>  at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:229)
>  at org.apache.atlas.AtlasClientV2.updateEntity(AtlasClientV2.java:324)
>  at 
> org.apache.atlas.hbase.bridge.HBaseBridge.updateEntityInAtlas(HBaseBridge.java:615)
>  at 
> org.apache.atlas.hbase.bridge.HBaseBridge.createOrUpdateTable(HBaseBridge.java:409)
>  at 
> org.apache.atlas.hbase.bridge.HBaseBridge.importTable(HBaseBridge.java:282)
>  at 
> org.apache.atlas.hbase.bridge.HBaseBridge.importHBaseEntities(HBaseBridge.java:235)
>  at org.apache.atlas.hbase.bridge.HBaseBridge.main(HBaseBridge.java:186)
>  This causes the atlas split 2 failure: test_import_hbase_without_args
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3783) DSL query search should return results for both the relationship edge directions

2020-05-13 Thread chaitali borole (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106359#comment-17106359
 ] 

chaitali borole commented on ATLAS-3783:


Success

> DSL query search should return results for both the relationship edge 
> directions 
> -
>
> Key: ATLAS-3783
> URL: https://issues.apache.org/jira/browse/ATLAS-3783
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
>  Labels: AdvancedSearch, DSL
> Fix For: 2.1.0
>
>
> eg;hive_column where table.name="employee" doesn't give any results for 
> hive_column whereas 
> hive_table where columns.name="eid" gives results for hive_table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Issue Comment Deleted] (ATLAS-3783) DSL query search should return results for both the relationship edge directions

2020-05-13 Thread chaitali borole (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chaitali borole updated ATLAS-3783:
---
Comment: was deleted

(was: Success)

> DSL query search should return results for both the relationship edge 
> directions 
> -
>
> Key: ATLAS-3783
> URL: https://issues.apache.org/jira/browse/ATLAS-3783
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
>  Labels: AdvancedSearch, DSL
> Fix For: 2.1.0
>
>
> eg;hive_column where table.name="employee" doesn't give any results for 
> hive_column whereas 
> hive_table where columns.name="eid" gives results for hive_table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72474: ATLAS-3783 : DSL query search should return results for both the relationship edge directions

2020-05-13 Thread chaitali

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72474/
---

(Updated May 13, 2020, 12:37 p.m.)


Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Bugs: ATLAS-3783
https://issues.apache.org/jira/browse/ATLAS-3783


Repository: atlas


Description
---

Unable to get results for Dsl query eg;hive_column where table.name="employee".
Due to regression only one edge direction was giving results.
This patch gives a fix for results for both relationship edge direction queries.


Diffs (updated)
-

  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
04b177533 
  repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
6ab61e16b 
  repository/src/main/java/org/apache/atlas/query/Lookup.java 3c192d824 
  repository/src/main/java/org/apache/atlas/query/RegistryBasedLookup.java 
2b34abff8 
  repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 0bbff2f46 
  repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
6f4df88f0 


Diff: https://reviews.apache.org/r/72474/diff/10/

Changes: https://reviews.apache.org/r/72474/diff/9-10/


Testing
---

TestCases added.


Thanks,

chaitali