Re: Review Request 74954: ATLAS-4846 : Source the atlas-env file in ATLAS import script to get all the argument
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74954/#review226487 --- Ship it! Ship It! - Mandar Ambawane On April 12, 2024, 6:01 a.m., Paresh Devalia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74954/ > --- > > (Updated April 12, 2024, 6:01 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4846 > https://issues.apache.org/jira/browse/ATLAS-4846 > > > Repository: atlas > > > Description > --- > > User has to run manually export command to get all the argument which are > set, to run the Atlas import script. > > The problem is not with the Atlas service itself but with these scripts: > > /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-kafka.sh > /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hbase.sh > /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hive.sh > /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-impala.sh > > > Need to set argument before this script are being called. > > > Diffs > - > > addons/hbase-bridge/src/bin/import-hbase.sh 8ae735034 > addons/hive-bridge/src/bin/import-hive.sh 951be5ba2 > addons/impala-bridge/src/main/resources/import-impala.sh b440f2d05 > addons/kafka-bridge/src/bin/import-kafka.sh 8fe27748a > > > Diff: https://reviews.apache.org/r/74954/diff/3/ > > > Testing > --- > > Testing:- Manually create the cluster and ran the import script. > > > Thanks, > > Paresh Devalia > >
Review Request 74964: Atlas - Upgrade Nimbus-JOSE-JWT to 9.37.3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74964/ --- Review request for atlas, Jayendra Parab, Priyanshi Shah, and Sheetal Shah. Bugs: ATLAS-4855 https://issues.apache.org/jira/browse/ATLAS-4855 Repository: atlas Description --- Atlas is currently pulling in nimbus-jose-jwt 9.8.1. Upgrate it to 9.37.3 Diffs - webapp/pom.xml 7d2d4c952 Diff: https://reviews.apache.org/r/74964/diff/1/ Testing --- Thanks, Mandar Ambawane
[jira] [Updated] (ATLAS-4855) Atlas - Upgrade Nimbus-JOSE-JWT to 9.37.3
[ https://issues.apache.org/jira/browse/ATLAS-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane updated ATLAS-4855: --- Description: Atlas is currently pulling in nimbus-jose-jwt 9.8.1. Upgrate it to 9.37.3 > Atlas - Upgrade Nimbus-JOSE-JWT to 9.37.3 > - > > Key: ATLAS-4855 > URL: https://issues.apache.org/jira/browse/ATLAS-4855 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > > Atlas is currently pulling in nimbus-jose-jwt 9.8.1. Upgrate it to 9.37.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4855) Atlas - Upgrade Nimbus-JOSE-JWT to 9.37.3
Mandar Ambawane created ATLAS-4855: -- Summary: Atlas - Upgrade Nimbus-JOSE-JWT to 9.37.3 Key: ATLAS-4855 URL: https://issues.apache.org/jira/browse/ATLAS-4855 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Mandar Ambawane Assignee: Mandar Ambawane -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/#review226384 --- Ship it! Ship It! - Mandar Ambawane On April 9, 2024, 1:14 p.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74951/ > --- > > (Updated April 9, 2024, 1:14 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4845 > https://issues.apache.org/jira/browse/ATLAS-4845 > > > Repository: atlas > > > Description > --- > > Steps: > > 1. First export and import with changeMarker “0” - worked as expected. > 2. Try to run 2nd export + import without any changes in atlas, 2nd export > also worked and gave differentchangeMarker in output as expected. > > Expectation: > > a) Empty zip should be created if no entities is fetched. > > Problem: > > Export zip file was created with three json files(atlas-typesdef.json, > atlas-export-order.json and atlas-export-info.json) even though no entities > was fetched and hence, while importing these zip files the import was getting > failed. > > Solution: > > 1. The flag "omitZipResponseForEmptyExport" has been created which will > indicate whether to create an empty zip or not when no entities are fetched > during export operation. This flag is introduced to support backward > compatibility for export operation. The default value of this flag is false. > 2. If the "omitZipResponseForEmptyExport" flag is set to true and if no > entities are fetched during export, an empty zip file is created with status > code 204. Otherwise, if the flag value is false zip file is created with 3 > json files (atlas-typesdef.json, atlas-export-order.json and > atlas-export-info.json) with status code as 200. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java > b03b386c0 > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java > 5cec39dd6 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 3aa24 > > > Diff: https://reviews.apache.org/r/74951/diff/2/ > > > Testing > --- > > Manual testing for the same has been performed. > > PC is also successfull. > > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1603/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/#review226379 --- Ship it! Ship It! - Mandar Ambawane On April 4, 2024, 1:16 p.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74939/ > --- > > (Updated April 4, 2024, 1:16 p.m.) > > > Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. > > > Bugs: ATLAS-4842 > https://issues.apache.org/jira/browse/ATLAS-4842 > > > Repository: atlas > > > Description > --- > > Steps: > > create two hive_tables in same database: > > 1. create table hive1(id int); > 2. create table hive1_ctas as select * from hive1; > 3. create table hive2(name string); > 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > 5. Import the zip. > > Expectations: > > a) Expectation is only the requested hive_table hive1 and its related > entities should be exported. > > Problem: > > hive2 is also imported which is an independently created entity and not > related to hive1. > > Solution: > > 1. Earlier, the implementation for fetchType as INCREMENTAL was getting > satisfied only if the value of skipLineage was "true" otherwise it was > implementing the default fetchType i.e FULL fetch and hence, all the > independently created entities in same db was also getting fetched. > > 2. Now, the problem is fixed by adding additional condition which will handle > INCREMENTAL fetchType even if the value for skipLineage is false. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java > da5cf37c4 > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 65d7a1872 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 0e3955dcd > > repository/src/test/resources/json/stocksDB-Entities/typedef-new-classification-T3.json > PRE-CREATION > > repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification-T2.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/74939/diff/5/ > > > Testing > --- > > Manual testing has been done for the same. > > PC Link : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1594/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/#review226377 --- repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java Line 62 (original), 62 (patched) <https://reviews.apache.org/r/74939/#comment314601> Please import only those classes from package which you are going to use here. - Mandar Ambawane On April 4, 2024, 7:18 a.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74939/ > --- > > (Updated April 4, 2024, 7:18 a.m.) > > > Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. > > > Bugs: ATLAS-4842 > https://issues.apache.org/jira/browse/ATLAS-4842 > > > Repository: atlas > > > Description > --- > > Steps: > > create two hive_tables in same database: > > 1. create table hive1(id int); > 2. create table hive1_ctas as select * from hive1; > 3. create table hive2(name string); > 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > 5. Import the zip. > > Expectations: > > a) Expectation is only the requested hive_table hive1 and its related > entities should be exported. > > Problem: > > hive2 is also imported which is an independently created entity and not > related to hive1. > > Solution: > > 1. Earlier, the implementation for fetchType as INCREMENTAL was getting > satisfied only if the value of skipLineage was "true" otherwise it was > implementing the default fetchType i.e FULL fetch and hence, all the > independently created entities in same db was also getting fetched. > > 2. Now, the problem is fixed by adding additional condition which will handle > INCREMENTAL fetchType even if the value for skipLineage is false. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java > da5cf37c4 > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 65d7a1872 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 0e3955dcd > > repository/src/test/resources/json/stocksDB-Entities/typedef-new-classification-T3.json > PRE-CREATION > > repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification-T2.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/74939/diff/4/ > > > Testing > --- > > Manual testing has been done for the same. > > PC Link : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1594/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/#review226373 --- Ship it! Ship It! - Mandar Ambawane On April 2, 2024, 2:50 p.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74951/ > --- > > (Updated April 2, 2024, 2:50 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4845 > https://issues.apache.org/jira/browse/ATLAS-4845 > > > Repository: atlas > > > Description > --- > > Steps: > > 1. First export and import with changeMarker “0” - worked as expected. > 2. Try to run 2nd export + import without any changes in atlas, 2nd export > also worked and gave differentchangeMarker in output as expected. > > Expectation: > > a) Empty zip should be created if no entities is fetched. > > Problem: > > 2nd Import is getting failed. > > Solution: > > 1. Earlier export zip was created with default json files like > atlas-export-info.json, atlas-export-order.json and atlas-typesdef.json when > though no entities was fetched and hence, while performing import opertaion > failure was seen with Empty Zip File exception. > 2. Now, the problem is fixed by returning empty zip files if no entities was > fetched and the status code is set to 204. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java > 5cec39dd6 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 3aa24 > > > Diff: https://reviews.apache.org/r/74951/diff/1/ > > > Testing > --- > > Manual testing for the same has been performed. > > PC is also successfull. > > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1589/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74925: ATLAS-4225 : Support for Chinese character in entity data.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74925/#review226291 --- repository/src/test/java/org/apache/atlas/BasicTestSetup.java Lines 767 (patched) <https://reviews.apache.org/r/74925/#comment314526> Please use equals() method to compare two String objects. - Mandar Ambawane On March 4, 2024, 11:37 a.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74925/ > --- > > (Updated March 4, 2024, 11:37 a.m.) > > > Review request for atlas, Ashutosh Mestry, Deep Singh, Jayendra Parab, Pinal > Shah, and Sheetal Shah. > > > Bugs: ATLAS-4225 > https://issues.apache.org/jira/browse/ATLAS-4225 > > > Repository: atlas > > > Description > --- > > Updated latest patch with synchronized code > > Creating this review request to track and close > https://reviews.apache.org/r/73256/diff/3#1 as the patch here is not in sync > with latest code > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > 9be6517e9 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > c670740c9 > repository/src/test/java/org/apache/atlas/BasicTestSetup.java 593f8c19b > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 282a3fd4a > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java > 1489b2760 > > > Diff: https://reviews.apache.org/r/74925/diff/1/ > > > Testing > --- > > Testcases are added in the patch > > > Thanks, > > chaitali > >
Re: Review Request 74691: ATLAS-4802 : Atlas 'updateTime' parameter is not updated when business metadata is edited.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74691/#review226283
---
Ship it!
Ship It!
- Mandar Ambawane
On Jan. 2, 2024, 10:35 a.m., chaitali wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74691/
> ---
>
> (Updated Jan. 2, 2024, 10:35 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and
> Sheetal Shah.
>
>
> Bugs: ATLAS-4802
> https://issues.apache.org/jira/browse/ATLAS-4802
>
>
> Repository: atlas
>
>
> Description
> ---
>
> When updating the business metadata and Labels on an atlas entity, the
> 'updateTime' system attribute is not changed.
>
> Updates to other aspects, such as technical properties and user defined
> attributes does update the timestamp, but not the business metadata and
> labels.
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> c670740c9
>
>
> Diff: https://reviews.apache.org/r/74691/diff/2/
>
>
> Testing
> ---
>
> update time Without patch
> before business metadata update
>
> "createTime": 1703221384530,
> "updateTime": 1703221386989,
>
> after business metadata update
>
> "createTime": 1703221384530,
> "updateTime": 1703221386989,
>
>
> after applying PATCH
>
> before business metadata update
> "createTime": 1704190056433,
> "updateTime": 1704190059208,
>
> after business metadata update
> "createTime": 1704190056433,
> "updateTime": 1704190523323,
>
> Entire json for GET api on entity before applying patch :
>
> http://localhost:21000/api/atlas/v2/entity/guid/dc1baa88-3a07-4577-9b6b-48982253aa3d
>
> {
> "referredEntities": {
> "a1a27e35-aa0d-472d-8b94-2405d8001a84": {
> "typeName": "Column",
> "attributes": {
> "qualifiedName": "Sales.customer_dim.address@cl1",
> "name": "address",
> "comment": "customer address"
> },
> "guid": "a1a27e35-aa0d-472d-8b94-2405d8001a84",
> "isIncomplete": false,
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1704190056433,
> "updateTime": 1704190056433,
> "version": 0,
> "classifications": [
> {
> "typeName": "PII",
> "entityGuid": "a1a27e35-aa0d-472d-8b94-2405d8001a84",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "removePropagationsOnEntityDelete": false
> }
> ],
> "labels": []
> },
> "834ba687-c8f1-4a48-a6a0-ef3ddeadf7ee": {
> "typeName": "Column",
> "attributes": {
> "qualifiedName": "Sales.customer_dim.name@cl1",
> "name": "name",
> "comment": "customer name"
> },
> "guid": "834ba687-c8f1-4a48-a6a0-ef3ddeadf7ee",
> "isIncomplete": false,
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1704190056433,
> "updateTime": 1704190056433,
> "version": 0,
> "classifications": [
> {
> "typeName": "PII",
> "entityGuid": "834ba687-c8f1-4a48-a6a0-ef3ddeadf7ee",
> "entityStatus": "ACTIVE",
> "propagate": true,
>
Re: Review Request 74783: ATLAS-4819 Get default relationship-type of an entity-type which has relationships with different entity-types on same attribute
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74783/ --- (Updated Jan. 15, 2024, 11:43 a.m.) Review request for atlas, Jayendra Parab, Paresh Devalia, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4819 https://issues.apache.org/jira/browse/ATLAS-4819 Repository: atlas Description --- When an entity-type has relationships with other entity-types on same attribute and when the relationship type information is not provided. This might result in Atlas picking the random relationship-type for the current entity type. Which may lead to "Invalid relationshipDef" exceptions while creating entities / updating entities / migration import. This patch allows user to provide the default value for the relationship-type for particular entity-type. Diffs (updated) - distro/src/conf/atlas-application.properties b5734d7a8 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 60e57a379 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 591b31612 Diff: https://reviews.apache.org/r/74783/diff/2/ Changes: https://reviews.apache.org/r/74783/diff/1-2/ Testing --- Thanks, Mandar Ambawane
[jira] [Resolved] (ATLAS-4754) Download Search with Basic Search gives java.io.FileNotFoundException
[
https://issues.apache.org/jira/browse/ATLAS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mandar Ambawane resolved ATLAS-4754.
Resolution: Fixed
Default value for the download directory will be read from system property
"user.dir" instead of property "atlas.home"
> Download Search with Basic Search gives java.io.FileNotFoundException
> -
>
> Key: ATLAS-4754
> URL: https://issues.apache.org/jira/browse/ATLAS-4754
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Abhishek Pal
>Assignee: Mandar Ambawane
>Priority: Major
>
> Currently when trying to download search results, the download task is stuck
> on pending.
> Upon checking the logs the following error is being seen:
> {code:java|title=application.log}
> 2023-05-17 22:09:47,321 INFO - [etp1158258131-199 -
> a6302f98-7e3d-4cbc-bd06-883fbfa21d11:] ~ TaskManagement: Processing stats:
> total=1, sinceLastStatsReport=2 completedWithErrors=1, succeded=0
> (TaskExecutor$TaskLogger:166)
> 2023-05-17 22:09:47,330 ERROR - [atlas-task-0-etp1158258131-282 -
> b9a5a4ef-6e3f-440c-9f65-d446a6da9720:] ~ Task:
> f4e26459-746f-4f31-90db-144c31a9696f: Error performing task!
> (SearchResultDownloadTask:126)
> java.io.FileNotFoundException:
> /search_result_downloads//.csv (No such file
> or directory)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:237)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:187)
> at java.base/java.io.FileWriter.(FileWriter.java:96)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.generateCSVFileFromSearchResult(SearchResultDownloadTask.java:185)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.run(SearchResultDownloadTask.java:163)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.perform(SearchResultDownloadTask.java:122)
> at org.apache.atlas.tasks.AbstractTask.run(AbstractTask.java:33)
> at
> org.apache.atlas.tasks.TaskExecutor$TaskConsumer.performTask(TaskExecutor.java:150)
> at org.apache.atlas.tasks.TaskExecutor$TaskConsumer.run(TaskExecutor.java:109)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834){code}
> h2. Steps to Reproduce:
> h4. Create type: test_download_entity
> {code:java|title=/api/atlas/v2/types/typedefs}
> {
> "enumDefs": [],
> "structDefs": [],
> "classificationDefs": [],
> "entityDefs": [
> {
> "attributeDefs": [
> {
> "name": "name",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false
> },
> {
> "name": "type_str",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false
> }
> ],
> "description": "description",
> "name": "test_download_entity",
> "guid": "-32062751815011",
> "category": "ENTITY",
> "superTypes": []
> }
> ],
> "relationshipDefs": [],
> "businessMetadataDefs": []
> }
>
Re: [DRAFT] [REPORT] Apache Atlas - December 2023
+1, Thank you Madhan for putting this together. On Fri, Dec 15, 2023 at 11:01 AM Sarath Subramanian wrote: > +1. Thanks for the draft Madhan > > > Thanks, > Sarath > > On Wed, Dec 13, 2023 at 9:25 AM Madhan Neethiraj > wrote: > > > Atlas community, > > > > > > > > Please review the following draft report for this quarter. > > > > > > > > Thanks, > > > > Madhan > > > > > > > > ## Description: > > > > The mission of Apache Atlas is the creation and maintenance of software > > related > > > > to a scalable and extensible set of core foundational governance services > > - > > > > enabling enterprises to effectively and efficiently meet their compliance > > > > requirements > > > > > > > > ## Project Status: > > > > Current project status: Ongoing, with moderate activity > > > > Issues for the board: There are no issues requiring board attention. > > > > > > > > ## Membership Data: > > > > Apache Atlas was founded 2017-06-20 (6 years ago) > > > > There are currently 48 committers and 32 PMC members in this project. > > > > The Committer-to-PMC ratio is 3:2. > > > > > > > > Community changes, past quarter: > > > > - No new PMC members. Last addition was Ashutosh Mestry on 2019-04-16. > > > > - No new committers. Last addition was Umesh Padashetty on 2023-06-14. > > > > > > > > ## Project Activity: > > > > - enhancements to support aging of entity audits > > > > - added bridge for Couchbase > > > > - performance improvement in notification processing with optimization in > > > > fetching relationships > > > > - removed references to Python2 with Python3 > > > > - improvements in dynamic index recovery > > > > - fixes in Impala hook > > > > - published Atlas client Python library 0.0.15 > > > > - updated versions of Okio, Apache Ivy > > > > > > > > ## Community Health: > > > > - dev@atlas.apache.org had a 41% increase in traffic in the past quarter > > (247 > > > > emails compared to 174) > > > > - 17 issues opened in JIRA, past quarter (-48% change) > > > > - 14 issues closed in JIRA, past quarter (-30% change) > > > > - 33 commits in the past quarter (3% increase) > > > > - 11 code contributors in the past quarter (22% increase) > > > > - 2 PRs opened on GitHub, past quarter (-71% change) > > > > - 4 PRs closed on GitHub, past quarter (no change) > > > > > > > > ## Recent releases: > > > > Apache Atlas 2.3.0 was released on 2022-12-06. > > > > Apache Atlas 2.2.0 was released on 2021-08-17. > > > > Apache Atlas 2.1.0 was released on 2020-07-15. > > > > > > > > > > > > > -- Thanks and Regards, Mandar V. Ambawane.
Review Request 74783: ATLAS-4819 Get default relationship-type of an entity-type which has relationships with different entity-types on same attribute
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74783/ --- Review request for atlas, Jayendra Parab, Paresh Devalia, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4819 https://issues.apache.org/jira/browse/ATLAS-4819 Repository: atlas Description --- When an entity-type has relationships with other entity-types on same attribute and when the relationship type information is not provided. This might result in Atlas picking the random relationship-type for the current entity type. Which may lead to "Invalid relationshipDef" exceptions while creating entities / updating entities / migration import. This patch allows user to provide the default value for the relationship-type for particular entity-type. Diffs - distro/src/conf/atlas-application.properties b5734d7a8 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 60e57a379 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 591b31612 Diff: https://reviews.apache.org/r/74783/diff/1/ Testing --- Thanks, Mandar Ambawane
[jira] [Created] (ATLAS-4819) Get default relationship-type of an entity-type which has relationships with different entity-types on same attribute
Mandar Ambawane created ATLAS-4819: -- Summary: Get default relationship-type of an entity-type which has relationships with different entity-types on same attribute Key: ATLAS-4819 URL: https://issues.apache.org/jira/browse/ATLAS-4819 Project: Atlas Issue Type: Bug Components: atlas-core, atlas-intg Reporter: Mandar Ambawane Assignee: Mandar Ambawane When an entity-type has relationships with other entity-types on same attribute and when the relationship type information is not provided. This might result in Atlas picking the random relationship-type for the current entity type. Which may lead to "Invalid relationshipDef" exceptions while creating entities / updating entities / migration import. This patch allows user to provide the default value for the relationship-type for particular entity-type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74562: ATLAS-4788 : Kafka password is in clear text in application.properties
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74562/#review225936 --- Ship it! Ship It! - Mandar Ambawane On Nov. 6, 2023, 11:13 a.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74562/ > --- > > (Updated Nov. 6, 2023, 11:13 a.m.) > > > Review request for atlas, Jayendra Parab, Paresh Devalia, and Sheetal Shah. > > > Bugs: ATLAS-4788 > https://issues.apache.org/jira/browse/ATLAS-4788 > > > Repository: atlas > > > Description > --- > > atlas.jaas.KafkaClient.option.username=username > atlas.jaas.KafkaClient.option.password= > > We have to encrypt this passsword using jceks file > > ./cputil.py -k atlas.jaas.KafkaClient.option.password -p P@$$w0rd -r > jceks://file/home/project/atlas/kafka.jceks > > > Diffs > - > > common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 167442259 > common/src/test/java/org/apache/atlas/utils/KafkaUtilsTest.java 562e28ae1 > > > Diff: https://reviews.apache.org/r/74562/diff/10/ > > > Testing > --- > > test case added > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1498/consoleFull > > > Thanks, > > chaitali > >
[jira] [Resolved] (ATLAS-4666) Intermittently, the audits for creation of hive_db registered are different than expected.
[ https://issues.apache.org/jira/browse/ATLAS-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane resolved ATLAS-4666. Resolution: Fixed > Intermittently, the audits for creation of hive_db registered are different > than expected. > -- > > Key: ATLAS-4666 > URL: https://issues.apache.org/jira/browse/ATLAS-4666 > Project: Atlas > Issue Type: Bug >Reporter: Rahul Kurup > Assignee: Mandar Ambawane >Priority: Major > > Sometimes, when I create a hive database and then check for the same in > Atlas, the audits tab shows unexpected audit entries: it shows two audit > entries both marked as 'Entity Updated' rather than showing a 'Entity > Created' audit entry anywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ATLAS-4678) Restrict Relationship Typedef creation if "relationshipCategory" value is not provided
[ https://issues.apache.org/jira/browse/ATLAS-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane resolved ATLAS-4678. Resolution: Fixed > Restrict Relationship Typedef creation if "relationshipCategory" value is not > provided > -- > > Key: ATLAS-4678 > URL: https://issues.apache.org/jira/browse/ATLAS-4678 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (ATLAS-4678) Restrict Relationship Typedef creation if "relationshipCategory" value is not provided
[ https://issues.apache.org/jira/browse/ATLAS-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane reassigned ATLAS-4678: -- Assignee: Mandar Ambawane > Restrict Relationship Typedef creation if "relationshipCategory" value is not > provided > -- > > Key: ATLAS-4678 > URL: https://issues.apache.org/jira/browse/ATLAS-4678 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4754) Download Search with Basic Search gives java.io.FileNotFoundException
[
https://issues.apache.org/jira/browse/ATLAS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mandar Ambawane updated ATLAS-4754:
---
Component/s: atlas-core
(was: atlas-webui)
> Download Search with Basic Search gives java.io.FileNotFoundException
> -
>
> Key: ATLAS-4754
> URL: https://issues.apache.org/jira/browse/ATLAS-4754
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Abhishek Pal
> Assignee: Mandar Ambawane
>Priority: Major
>
> Currently when trying to download search results, the download task is stuck
> on pending.
> Upon checking the logs the following error is being seen:
> {code:java|title=application.log}
> 2023-05-17 22:09:47,321 INFO - [etp1158258131-199 -
> a6302f98-7e3d-4cbc-bd06-883fbfa21d11:] ~ TaskManagement: Processing stats:
> total=1, sinceLastStatsReport=2 completedWithErrors=1, succeded=0
> (TaskExecutor$TaskLogger:166)
> 2023-05-17 22:09:47,330 ERROR - [atlas-task-0-etp1158258131-282 -
> b9a5a4ef-6e3f-440c-9f65-d446a6da9720:] ~ Task:
> f4e26459-746f-4f31-90db-144c31a9696f: Error performing task!
> (SearchResultDownloadTask:126)
> java.io.FileNotFoundException:
> /search_result_downloads//.csv (No such file
> or directory)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:237)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:187)
> at java.base/java.io.FileWriter.(FileWriter.java:96)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.generateCSVFileFromSearchResult(SearchResultDownloadTask.java:185)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.run(SearchResultDownloadTask.java:163)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.perform(SearchResultDownloadTask.java:122)
> at org.apache.atlas.tasks.AbstractTask.run(AbstractTask.java:33)
> at
> org.apache.atlas.tasks.TaskExecutor$TaskConsumer.performTask(TaskExecutor.java:150)
> at org.apache.atlas.tasks.TaskExecutor$TaskConsumer.run(TaskExecutor.java:109)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834){code}
> h2. Steps to Reproduce:
> h4. Create type: test_download_entity
> {code:java|title=/api/atlas/v2/types/typedefs}
> {
> "enumDefs": [],
> "structDefs": [],
> "classificationDefs": [],
> "entityDefs": [
> {
> "attributeDefs": [
> {
> "name": "name",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false
> },
> {
> "name": "type_str",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false
> }
> ],
> "description": "description",
> "name": "test_download_entity",
> "guid": "-32062751815011",
> "category": "ENTITY",
> "superTypes": []
> }
> ],
> "relationshipDefs": [],
> "businessMetadataDefs": []
> }
> {code}
> h4. Create entity of type: test_download_entity
> {code:java|title=/api/atlas/v2
[jira] [Assigned] (ATLAS-4754) Download Search with Basic Search gives java.io.FileNotFoundException
[
https://issues.apache.org/jira/browse/ATLAS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mandar Ambawane reassigned ATLAS-4754:
--
Assignee: Mandar Ambawane
> Download Search with Basic Search gives java.io.FileNotFoundException
> -
>
> Key: ATLAS-4754
> URL: https://issues.apache.org/jira/browse/ATLAS-4754
> Project: Atlas
> Issue Type: Bug
> Components: atlas-webui
>Reporter: Abhishek Pal
> Assignee: Mandar Ambawane
>Priority: Major
>
> Currently when trying to download search results, the download task is stuck
> on pending.
> Upon checking the logs the following error is being seen:
> {code:java|title=application.log}
> 2023-05-17 22:09:47,321 INFO - [etp1158258131-199 -
> a6302f98-7e3d-4cbc-bd06-883fbfa21d11:] ~ TaskManagement: Processing stats:
> total=1, sinceLastStatsReport=2 completedWithErrors=1, succeded=0
> (TaskExecutor$TaskLogger:166)
> 2023-05-17 22:09:47,330 ERROR - [atlas-task-0-etp1158258131-282 -
> b9a5a4ef-6e3f-440c-9f65-d446a6da9720:] ~ Task:
> f4e26459-746f-4f31-90db-144c31a9696f: Error performing task!
> (SearchResultDownloadTask:126)
> java.io.FileNotFoundException:
> /search_result_downloads//.csv (No such file
> or directory)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:237)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:187)
> at java.base/java.io.FileWriter.(FileWriter.java:96)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.generateCSVFileFromSearchResult(SearchResultDownloadTask.java:185)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.run(SearchResultDownloadTask.java:163)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.searchdownload.SearchResultDownloadTask.perform(SearchResultDownloadTask.java:122)
> at org.apache.atlas.tasks.AbstractTask.run(AbstractTask.java:33)
> at
> org.apache.atlas.tasks.TaskExecutor$TaskConsumer.performTask(TaskExecutor.java:150)
> at org.apache.atlas.tasks.TaskExecutor$TaskConsumer.run(TaskExecutor.java:109)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834){code}
> h2. Steps to Reproduce:
> h4. Create type: test_download_entity
> {code:java|title=/api/atlas/v2/types/typedefs}
> {
> "enumDefs": [],
> "structDefs": [],
> "classificationDefs": [],
> "entityDefs": [
> {
> "attributeDefs": [
> {
> "name": "name",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false
> },
> {
> "name": "type_str",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false
> }
> ],
> "description": "description",
> "name": "test_download_entity",
> "guid": "-32062751815011",
> "category": "ENTITY",
> "superTypes": []
> }
> ],
> "relationshipDefs": [],
> "businessMetadataDefs": []
> }
> {code}
> h4. Create entity of type: test_download_entity
> {code:java|title=/api/atlas/v2/entity/bulk}
> {
> "entities": [{
>
Review Request 74452: ATLAS-4754 : Download Search with Basic Search gives java.io.FileNotFoundException
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74452/ --- Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4754 https://issues.apache.org/jira/browse/ATLAS-4754 Repository: atlas Description --- Default value for the download directory will be read from system property "user.dir" instead of property "atlas.home" Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java fd90fd440 Diff: https://reviews.apache.org/r/74452/diff/1/ Testing --- Thanks, Mandar Ambawane
Re: Review Request 74161: ATLAS-4687 : Atlas server doesn't come up due to circular dependency in TaskRegistry class
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74161/#review225419 --- Ship it! Ship It! - Mandar Ambawane On April 27, 2023, 7:34 a.m., Sheetal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74161/ > --- > > (Updated April 27, 2023, 7:34 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Prasad Pawar. > > > Bugs: ATLAS-4687 > https://issues.apache.org/jira/browse/ATLAS-4687 > > > Repository: atlas > > > Description > --- > > Following errors were found intermittently due to which Atlas server won’t > start up. These errors pointed to a circular dependency issue which was > caused by TaskRegistry > class(repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java) As a > resolution to this, @Lazy annotation is added on @Component definition of > this class. > (FYI : A @Lazy bean is not initialised until referenced by another bean or > explicitly retrieved from BeanFactory . Beans that are not annotated with > @Lazy are initialise eagerly.) > > > > org.springframework.beans.factory.BeanCurrentlyInCreationException: Error > creating bean with name 'taskRegistry': Bean with name 'taskRegistry' has > been injected into other beans [taskManagement] in its raw version as part of > a circular reference, but has eventually been wrapped. This means that said > other beans do not use the final version of the bean. This is often the > result of over-eager type matching - consider using 'getBeanNamesForType' > with the 'allowEagerInit' flag turned off, for example. > > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:649) > ~[spring-beans-5.3.21.jar:5.3.21] > > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) > ~[spring-beans-5.3.21.jar:5.3.21] > > at > org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) > ~[spring-beans-5.3.21.jar:5.3.21] > > at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) > ~[spring-beans-5.3.21.jar:5.3.21] > > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) > ~[spring-beans-5.3.21.jar:5.3.21] > > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) > ~[spring-beans-5.3.21.jar:5.3.21] > > at > org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) > ~[spring-beans-5.3.21.jar:5.3.21] > > at > org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) > ~[spring-context-5.3.21.jar:5.3.21] > > at > org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) > ~[spring-context-5.3.21.jar:5.3.21] > > at > org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) > ~[spring-web-5.3.21.jar:5.3.21] > > at > org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) > [spring-web-5.3.21.jar:5.3.21] > > at > org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) > [spring-web-5.3.21.jar:5.3.21] > > at > org.apache.atlas.web.setup.KerberosAwareListener.contextInitialized(KerberosAwareListener.java:31) > [atlas-webapp-3.0.0-SNAPSHOT.jar:?] > > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013) > [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723] > > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553) > [jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723] > > at > org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942) > [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723] > > at > org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782) > [jetty-servlet-9.4.31.v202
[jira] [Resolved] (ATLAS-4733) Download Basic and DSL search results
[ https://issues.apache.org/jira/browse/ATLAS-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane resolved ATLAS-4733. Resolution: Fixed > Download Basic and DSL search results > - > > Key: ATLAS-4733 > URL: https://issues.apache.org/jira/browse/ATLAS-4733 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > > This ticket tracks the Server side changes required for the Search result > download task. > From server side, each download request is considered as a separate thread > which will run in the background and will generate the csv file according to > the inputs provided. > The generated files will be stored in the specific location (which is > configurable). > A scheduler will be run to delete the old files. In this case, the scheduler > interval and file expiry time is also configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/
---
(Updated April 14, 2023, 8 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika
Kundam, Sheetal Shah, and Sidharth Mishra.
Changes
---
Addressed review comments
Bugs: ATLAS-4733
https://issues.apache.org/jira/browse/ATLAS-4733
Repository: atlas
Description
---
This ticket tracks the Server side changes required for the Search result
download task.
>From server side, each download request is considered as a separate thread
>which will run in the background and will generate the csv file according to
>the inputs provided.
The generated files will be stored in the specific location (which is
configurable).
A scheduler will be run to delete the old files. In this case, the scheduler
interval and file expiry time is also configurable.
Diffs (updated)
-
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
d94110004
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
582d97542
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
Diff: https://reviews.apache.org/r/74341/diff/7/
Changes: https://reviews.apache.org/r/74341/diff/6-7/
Testing
---
PreCommit:
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
Request to generate csv file with default columns seen on UI:
curl -u : --request POST
":/api/atlas/v2/search/basic/download/create_file" --header
'Content-Type: application/json' --data-raw
'{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
Request to generate csv file with additional columns selected from UI:
curl -u : --request POST
":/api/atlas/v2/search/basic/download/create_file" --header
'Content-Type: application/json' --data-raw
'{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
Thanks,
Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74341/ --- (Updated April 12, 2023, 6:21 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra. Changes --- Addressed review comments Bugs: ATLAS-4733 https://issues.apache.org/jira/browse/ATLAS-4733 Repository: atlas Description --- This ticket tracks the Server side changes required for the Search result download task. >From server side, each download request is considered as a separate thread >which will run in the background and will generate the csv file according to >the inputs provided. The generated files will be stored in the specific location (which is configurable). A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable. Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433 intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c Diff: https://reviews.apache.org/r/74341/diff/6/ Changes: https://reviews.apache.org/r/74341/diff/5-6/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull Thanks, Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74341/ --- (Updated April 10, 2023, 11:07 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra. Changes --- Addressed review comments Bugs: ATLAS-4733 https://issues.apache.org/jira/browse/ATLAS-4733 Repository: atlas Description --- This ticket tracks the Server side changes required for the Search result download task. >From server side, each download request is considered as a separate thread >which will run in the background and will generate the csv file according to >the inputs provided. The generated files will be stored in the specific location (which is configurable). A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable. Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433 intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c Diff: https://reviews.apache.org/r/74341/diff/5/ Changes: https://reviews.apache.org/r/74341/diff/4-5/ Testing (updated) --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull Thanks, Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74341/ --- (Updated March 30, 2023, 2:37 p.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra. Changes --- 1. Pending task limit added. 2. Search result limit added. 3. File name changed. 4. Code cleanup. Bugs: ATLAS-4733 https://issues.apache.org/jira/browse/ATLAS-4733 Repository: atlas Description --- This ticket tracks the Server side changes required for the Search result download task. >From server side, each download request is considered as a separate thread >which will run in the background and will generate the csv file according to >the inputs provided. The generated files will be stored in the specific location (which is configurable). A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable. Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433 intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c Diff: https://reviews.apache.org/r/74341/diff/4/ Changes: https://reviews.apache.org/r/74341/diff/3-4/ Testing --- Thanks, Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74341/ --- (Updated March 20, 2023, 6:37 p.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra. Changes --- 1. Code cleanup. 2. DSL query support. 3. CSV file name change. Bugs: ATLAS-4733 https://issues.apache.org/jira/browse/ATLAS-4733 Repository: atlas Description --- This ticket tracks the Server side changes required for the Search result download task. >From server side, each download request is considered as a separate thread >which will run in the background and will generate the csv file according to >the inputs provided. The generated files will be stored in the specific location (which is configurable). A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c Diff: https://reviews.apache.org/r/74341/diff/3/ Changes: https://reviews.apache.org/r/74341/diff/2-3/ Testing --- Thanks, Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74341/ --- (Updated March 9, 2023, 11:26 a.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. Changes --- Code cleanup Bugs: ATLAS-4733 https://issues.apache.org/jira/browse/ATLAS-4733 Repository: atlas Description --- This ticket tracks the Server side changes required for the Search result download task. >From server side, each download request is considered as a separate thread >which will run in the background and will generate the csv file according to >the inputs provided. The generated files will be stored in the specific location (which is configurable). A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c Diff: https://reviews.apache.org/r/74341/diff/2/ Changes: https://reviews.apache.org/r/74341/diff/1-2/ Testing --- Thanks, Mandar Ambawane
[jira] [Created] (ATLAS-4733) Download Basic and DSL search results
Mandar Ambawane created ATLAS-4733: -- Summary: Download Basic and DSL search results Key: ATLAS-4733 URL: https://issues.apache.org/jira/browse/ATLAS-4733 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Mandar Ambawane Assignee: Mandar Ambawane This ticket tracks the Server side changes required for the Search result download task. >From server side, each download request is considered as a separate thread >which will run in the background and will generate the csv file according to >the inputs provided. The generated files will be stored in the specific location (which is configurable). A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ATLAS-4576) Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
[ https://issues.apache.org/jira/browse/ATLAS-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane resolved ATLAS-4576. Resolution: Fixed > Backward compatibility for check provided for AttributeName in Parent and > Child TypeDef > --- > > Key: ATLAS-4576 > URL: https://issues.apache.org/jira/browse/ATLAS-4576 > Project: Atlas > Issue Type: Improvement > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > > Currently we have provided 2 checks for the Attribute Name: > [ATLAS-3872|https://issues.apache.org/jira/browse/ATLAS-3872] > Restrict typedef creation when a child type attribute conflicts with parent > type attribute of same name > [ATLAS-4522|https://issues.apache.org/jira/browse/ATLAS-4522] > Updating typedef with new supertype should be allowed only if attributes are > unique compared to other existing supertypes > But in the earlier versions these checks were not there. So there may be a > chance of Atlas environment having such data where these checks can cause > problems at the time: > 1. Atlas Startup. > 2. Creating new typedefs > This patch handles these scenarios and provides backward compatibility for > these 2 changes mentioned. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 73929: ATLAS-4576 Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73929/ --- (Updated Jan. 9, 2023, 8:16 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Paresh Devalia, Pinal Shah, Radhika Kundam, Sarath Subramanian, Sheetal Shah, and Sidharth Mishra. Changes --- Addressed review comments Bugs: ATLAS-4576 https://issues.apache.org/jira/browse/ATLAS-4576 Repository: atlas Description --- Currently we have provided 2 checks for the Attribute Name: https://issues.apache.org/jira/browse/ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name https://issues.apache.org/jira/browse/ATLAS-4522 Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes But in the earlier versions these checks were not there. So there may be a chance of Atlas environment having such data where these checks can cause problems at the time: 1. Atlas Startup. 2. Creating new typedefs This patch handles these scenarios and provides backward compatibility for these 2 changes mentioned. Diffs (updated) - distro/src/conf/atlas-application.properties c92c619c9 intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 17422cf95 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 76bee36f2 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 21ce23657 intg/src/test/java/org/apache/atlas/type/TestAtlasClassificationType.java b6055fa30 intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java 742970390 Diff: https://reviews.apache.org/r/73929/diff/6/ Changes: https://reviews.apache.org/r/73929/diff/5-6/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1084/consoleFull PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1293/consoleFull Thanks, Mandar Ambawane
Re: Review Request 73929: ATLAS-4576 Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73929/ --- (Updated Dec. 16, 2022, 6:41 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Paresh Devalia, Pinal Shah, Radhika Kundam, Sarath Subramanian, Sheetal Shah, and Sidharth Mishra. Changes --- Code cleanup and custom flag changes. Bugs: ATLAS-4576 https://issues.apache.org/jira/browse/ATLAS-4576 Repository: atlas Description --- Currently we have provided 2 checks for the Attribute Name: https://issues.apache.org/jira/browse/ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name https://issues.apache.org/jira/browse/ATLAS-4522 Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes But in the earlier versions these checks were not there. So there may be a chance of Atlas environment having such data where these checks can cause problems at the time: 1. Atlas Startup. 2. Creating new typedefs This patch handles these scenarios and provides backward compatibility for these 2 changes mentioned. Diffs (updated) - distro/src/conf/atlas-application.properties c92c619c9 intg/src/main/java/org/apache/atlas/ThreadContext.java PRE-CREATION intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 17422cf95 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 76bee36f2 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 21ce23657 intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 0c05bb399 intg/src/test/java/org/apache/atlas/type/TestAtlasClassificationType.java b6055fa30 intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java 742970390 webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3eab Diff: https://reviews.apache.org/r/73929/diff/5/ Changes: https://reviews.apache.org/r/73929/diff/4-5/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1084/consoleFull Thanks, Mandar Ambawane
Re: Review Request 73929: ATLAS-4576 Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73929/ --- (Updated Dec. 8, 2022, 7:52 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Changes --- Added more test cases and custom flag to skip checks. Bugs: ATLAS-4576 https://issues.apache.org/jira/browse/ATLAS-4576 Repository: atlas Description --- Currently we have provided 2 checks for the Attribute Name: https://issues.apache.org/jira/browse/ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name https://issues.apache.org/jira/browse/ATLAS-4522 Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes But in the earlier versions these checks were not there. So there may be a chance of Atlas environment having such data where these checks can cause problems at the time: 1. Atlas Startup. 2. Creating new typedefs This patch handles these scenarios and provides backward compatibility for these 2 changes mentioned. Diffs (updated) - distro/src/conf/atlas-application.properties c92c619c9 intg/src/main/java/org/apache/atlas/ThreadContext.java PRE-CREATION intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 17422cf95 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 76bee36f2 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 21ce23657 intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 0c05bb399 intg/src/test/java/org/apache/atlas/type/TestAtlasClassificationType.java b6055fa30 intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java 742970390 webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3eab Diff: https://reviews.apache.org/r/73929/diff/4/ Changes: https://reviews.apache.org/r/73929/diff/3-4/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1084/consoleFull Thanks, Mandar Ambawane
Re: Review Request 74240: ATLAS-4700: updated documentation for 2.3.0 release
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74240/#review224979 --- Ship it! Ship It! - Mandar Ambawane On Dec. 7, 2022, 8:08 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74240/ > --- > > (Updated Dec. 7, 2022, 8:08 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Mandar Ambawane, > Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Prasad Pawar, Radhika Kundam, > Sarath Subramanian, Sharmadha Sainath, and Sidharth Mishra. > > > Bugs: ATLAS-4700 > https://issues.apache.org/jira/browse/ATLAS-4700 > > > Repository: atlas > > > Description > --- > > updated documentation for 2.3.0 release > > > Diffs > - > > docs/docz-lib/config/menu.js 37acf46d4 > docs/docz-lib/config/versions.js 3ef55f681 > docs/src/documents/Downloads/Downloads.md e486a1d0b > docs/src/documents/Overview.md 8614106ab > docs/src/documents/Whats-New/WhatsNew-2.3.md PRE-CREATION > > > Diff: https://reviews.apache.org/r/74240/diff/1/ > > > Testing > --- > > verified the updated site with 'mvn site:run" in docs directory > > > Thanks, > > Madhan Neethiraj > >
[jira] [Resolved] (ATLAS-4495) Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
[ https://issues.apache.org/jira/browse/ATLAS-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane resolved ATLAS-4495. Resolution: Fixed > Glossary section takes longer time to load when Glossaries have large number > of Categories and Terms associated with them > - > > Key: ATLAS-4495 > URL: https://issues.apache.org/jira/browse/ATLAS-4495 > Project: Atlas > Issue Type: Improvement > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74205: ATLAS-4699: updated version in pom.xml from 2.3.0-SNAPSHOT to 2.3.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74205/#review224882 --- Ship it! Ship It! - Mandar Ambawane On Nov. 17, 2022, 8:52 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74205/ > --- > > (Updated Nov. 17, 2022, 8:52 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Mandar Ambawane, > Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Radhika Kundam, Sarath > Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-4699 > https://issues.apache.org/jira/browse/ATLAS-4699 > > > Repository: atlas > > > Description > --- > > updated Atlas version from 2.3.0-SNAPSHOT to 2.3.0 > > > Diffs > - > > addons/falcon-bridge-shim/pom.xml 981e76418 > addons/falcon-bridge/pom.xml 8e25111d4 > addons/hbase-bridge-shim/pom.xml ef3c1bad9 > addons/hbase-bridge/pom.xml 5a0f62fb3 > addons/hbase-testing-util/pom.xml 947ca477f > addons/hdfs-model/pom.xml 226b0d49b > addons/hive-bridge-shim/pom.xml cb4dbb3ae > addons/hive-bridge/pom.xml 80a2252c5 > addons/impala-bridge-shim/pom.xml 247dc10ae > addons/impala-bridge/pom.xml cacdf4a98 > addons/impala-hook-api/pom.xml 8b61994f7 > addons/kafka-bridge/pom.xml 31f5d5b3a > addons/sqoop-bridge-shim/pom.xml 4ba77d521 > addons/sqoop-bridge/pom.xml 62e2eee00 > addons/storm-bridge-shim/pom.xml 510718c8d > addons/storm-bridge/pom.xml a5017e0b2 > atlas-examples/pom.xml 12cd8c401 > atlas-examples/sample-app/pom.xml e2eb42f54 > authorization/pom.xml 177f2dde3 > build-tools/pom.xml 01c573258 > client/client-v1/pom.xml fc80a4294 > client/client-v2/pom.xml deb991001 > client/common/pom.xml e9959bf47 > client/pom.xml 5057e4c02 > common/pom.xml 554ab67d2 > dashboardv2/pom.xml 965e5856e > dashboardv3/pom.xml 2e1729988 > dev-support/atlas-docker/.env ca597ac04 > distro/pom.xml a892736de > docs/pom.xml 9b99d5caf > graphdb/api/pom.xml 741cf8d89 > graphdb/common/pom.xml 0e96d46f9 > graphdb/graphdb-impls/pom.xml 897a48dc8 > graphdb/janus-hbase2/pom.xml 4da5ae8e4 > graphdb/janus/pom.xml 75eed0f16 > graphdb/pom.xml 0fde3510a > intg/pom.xml 3a34983a5 > notification/pom.xml e8abd453e > plugin-classloader/pom.xml 2057d3554 > pom.xml 4cc2201c1 > repository/pom.xml eeb7af9de > server-api/pom.xml 0c5e66ec2 > test-tools/pom.xml 6622512fe > tools/atlas-index-repair/pom.xml 9298f91ba > tools/classification-updater/pom.xml 04e766bd2 > webapp/pom.xml 30c7b416b > > > Diff: https://reviews.apache.org/r/74205/diff/1/ > > > Testing > --- > > verified that Atlas builds successfully > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 74162: ATLAS-4610, ATLAS-4655: Classification/ Term: Add parent name for classification & term
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74162/#review224822 --- Ship it! Ship It! - Mandar Ambawane On Oct. 20, 2022, 10:30 a.m., Prasad Pawar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74162/ > --- > > (Updated Oct. 20, 2022, 10:30 a.m.) > > > Review request for atlas, Farhan Khan, Jayendra Parab, Mandar Ambawane, Pinal > Shah, Sharmadha Sainath, and Sheetal Shah. > > > Bugs: https://issues.apache.org/jira/browse/ATLAS-4610 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-4610 > > > Repository: atlas > > > Description > --- > > Backend: > Modified AtlasTermAssignmentHeader, AtlasRelatedObjectId and > AtlasRelatedTermHeader to display glossary details on entity search, dsl > search, basic search and term details pages. > > UI: > The classification and term assignment section should display their parents > if present, on the detail page. > > 1) Classification name will be displayed as "name"/ "name@parent"/ > "name@(parent1,parent2,...)", on search result page, entity detail page, > classification/term detail page (search result). > > 2)Term name will be displayed as "term@glossary", on search result page, > entity detail page, classification/term detail page (search result). > Same will be display on tooltip of classification and term. > > 3) Button Text on view related attribute change from "ok" to "Close" on term > detail page . fixed > > 4) Button Text while removing term from related terms columns is change from > "Ok" to "Cancle" on term detail page . fixed > > 5) New-ui, redirection to tag detail page from search result page & entity > detail page, fixed > > > Diffs > - > > dashboardv2/public/css/scss/common.scss 60b272214 > dashboardv2/public/js/templates/detail_page/DetailPageLayoutView_tmpl.html > ee4b9431f > dashboardv2/public/js/utils/CommonViewFunction.js f93b1788a > dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 73293507c > dashboardv2/public/js/views/glossary/GlossaryDetailLayoutView.js 02606067b > dashboardv2/public/js/views/glossary/TermRelationAttributeLayoutView.js > ded1ab943 > dashboardv2/public/js/views/schema/SchemaLayoutView.js efc8e9356 > dashboardv2/public/js/views/search/SearchResultLayoutView.js 66466e21d > dashboardv2/public/js/views/tag/TagLayoutView.js 7de7aea14 > dashboardv3/public/css/scss/common.scss c0f65fdab > dashboardv3/public/js/router/Router.js a48b4cc78 > dashboardv3/public/js/templates/detail_page/DetailPageLayoutView_tmpl.html > b209a49e4 > dashboardv3/public/js/utils/CommonViewFunction.js 5b2b0c2dc > dashboardv3/public/js/views/detail_page/DetailPageLayoutView.js a64032278 > dashboardv3/public/js/views/glossary/GlossaryDetailLayoutView.js 019cd9d3f > dashboardv3/public/js/views/glossary/TermRelationAttributeLayoutView.js > ded1ab943 > dashboardv3/public/js/views/schema/SchemaLayoutView.js 1ae75d922 > dashboardv3/public/js/views/search/SearchResultLayoutView.js d9d238353 > dashboardv3/public/js/views/search/tree/ClassificationTreeLayoutView.js > 13a5a899d > > intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasRelatedTermHeader.java > f16cf25ed > > intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasTermAssignmentHeader.java > c7de0638a > > intg/src/main/java/org/apache/atlas/model/instance/AtlasRelatedObjectId.java > ae6932de5 > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 110c447ad > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 0ee86735d > > > Diff: https://reviews.apache.org/r/74162/diff/2/ > > > Testing > --- > > On search result page: > > 1) Assigned classifications and terms are displayed as per requirement. > 2) Tooltip for assigned classifications and terms are displayed as per > requirement. > 3) Redirection to classification detail page & term detail page > > On entity detail page: > 1) Assigned classifications and terms are displayed as per requirement. > 2) Propagated classifications are displayed as per requirement. > 3) Tooltip for assigned classifications, terms and Propagated classific
Re: Review Request 74102: ATLAS-4666 : Intermittently, the audits for creation of hive_db registered are different than expected
> On Oct. 7, 2022, 4:23 a.m., Sidharth Mishra wrote: > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > > Lines 659 (patched) > > <https://reviews.apache.org/r/74102/diff/1/?file=2269810#file2269810line659> > > > > If someone set "atlas.rest.create.shell.entity.for.non-existing.ref" to > > true and used REST calls to create entities, it might end up creating > > multiple create audit records. Please try testing this. If it does I need > > to see how to fix this and have only one create entry. Hi Sidharth, we checked by setting this property "true" and creating Hive table entity from Atlas UI. Only one "Create" Audit entry got generated. - Mandar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74102/#review224762 --- On Sept. 6, 2022, 5:16 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74102/ > --- > > (Updated Sept. 6, 2022, 5:16 a.m.) > > > Review request for atlas, Jayendra Parab, Pinal Shah, Radhika Kundam, and > Sidharth Mishra. > > > Bugs: ATLAS-4666 > https://issues.apache.org/jira/browse/ATLAS-4666 > > > Repository: atlas > > > Description > --- > > Sometimes, when I create a hive database and then check for the same in > Atlas, the audits tab shows unexpected audit entries: it shows two audit > entries both marked as 'Entity Updated' rather than showing a 'Entity > Created' audit entry anywhere. > > This patch will check for the Shell entities created and then assign the > Entity status to the Audit entry. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > d48c91709 > > > Diff: https://reviews.apache.org/r/74102/diff/1/ > > > Testing > --- > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 74133: ATLAS-4678 : Restrict Relationship Typedef creation if relationshipCategory value is not provided
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74133/ --- (Updated Oct. 6, 2022, 9:42 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, Sheetal Shah, and Sidharth Mishra. Changes --- Addressed review comments Bugs: ATLAS-4678 https://issues.apache.org/jira/browse/ATLAS-4678 Repository: atlas Description --- When Relationship typedef is created without "relationshipCategory", the default "relationshipCategory" provided as "ASSOCIATION". For "relationshipCategory" "ASSOCIATION", it is required that both enddefs should have "isContainer" value as "false". while creating the Relationship type, Whenever user does not provide the "relationshipCategory" value and put one of the "isContainer" value as "true". Because of this, Relationship type gets created with "relationshipCategory" = "ASSOCIATION" and one of the Enddefs with "isContainer" = "true" When atlas service is restarted, it throws and exception for such Relationship type. and due to this, entire TypeRegistry does not get populated. So even after the Atlas service comes up, user cannot see any Typedefs. At the time of create operation, this patch checks Relationship type which has no "relationshipCategory", it also checks for the "isContainer" values for both the Enddefs. If both "isContainer" values are false, then only it sets the default "relationshipCategory" as "ASSOCIATION" otherwise it throws an exception. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipDefStoreV2.java 7bc4b28c3 Diff: https://reviews.apache.org/r/74133/diff/2/ Changes: https://reviews.apache.org/r/74133/diff/1-2/ Testing --- Thanks, Mandar Ambawane
Re: Review Request 74101: ATLAS-4668 : When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74101/ --- (Updated Oct. 6, 2022, 6:19 a.m.) Review request for atlas, Jayendra Parab, madhan, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra. Changes --- Addressed review comments Bugs: ATLAS-4668 https://issues.apache.org/jira/browse/ATLAS-4668 Repository: atlas Description --- When an entity type is created with multiple super types having same attribute name, Atlas throws exception. Ex: if there are 2 types: type1 - attr1, attr2 type2- attr1, attr2 Both types cant be added to super type to type3 since both type1 and type2 have same attribute - attr1 This patch will check for the exact scenario in case of Classification Typedef as well. Diffs (updated) - intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 5b86b808b intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java 85d041bdb Diff: https://reviews.apache.org/r/74101/diff/4/ Changes: https://reviews.apache.org/r/74101/diff/3-4/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1220/console Thanks, Mandar Ambawane
Review Request 74133: ATLAS-4678 : Restrict Relationship Typedef creation if relationshipCategory value is not provided
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74133/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, Sheetal Shah, and Sidharth Mishra. Bugs: ATLAS-4678 https://issues.apache.org/jira/browse/ATLAS-4678 Repository: atlas Description --- When Relationship typedef is created without "relationshipCategory", the default "relationshipCategory" provided as "ASSOCIATION". For "relationshipCategory" "ASSOCIATION", it is required that both enddefs should have "isContainer" value as "false". while creating the Relationship type, Whenever user does not provide the "relationshipCategory" value and put one of the "isContainer" value as "true". Because of this, Relationship type gets created with "relationshipCategory" = "ASSOCIATION" and one of the Enddefs with "isContainer" = "true" When atlas service is restarted, it throws and exception for such Relationship type. and due to this, entire TypeRegistry does not get populated. So even after the Atlas service comes up, user cannot see any Typedefs. At the time of create operation, this patch checks Relationship type which has no "relationshipCategory", it also checks for the "isContainer" values for both the Enddefs. If both "isContainer" values are false, then only it sets the default "relationshipCategory" as "ASSOCIATION" otherwise it throws an exception. Diffs - intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipDefStoreV2.java 7bc4b28c3 Diff: https://reviews.apache.org/r/74133/diff/1/ Testing --- Thanks, Mandar Ambawane
Re: Review Request 74101: ATLAS-4668 : When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74101/#review224686 --- intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java Lines 564 (patched) <https://reviews.apache.org/r/74101/#comment313482> We need this check in Diamond Inheritance. - Mandar Ambawane On Sept. 20, 2022, 10:12 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74101/ > --- > > (Updated Sept. 20, 2022, 10:12 a.m.) > > > Review request for atlas, Jayendra Parab, madhan, Pinal Shah, Radhika Kundam, > Sheetal Shah, and Sidharth Mishra. > > > Bugs: ATLAS-4668 > https://issues.apache.org/jira/browse/ATLAS-4668 > > > Repository: atlas > > > Description > --- > > When an entity type is created with multiple super types having same > attribute name, Atlas throws exception. > > Ex: > > if there are 2 types: > > type1 - attr1, attr2 > > type2- attr1, attr2 > > Both types cant be added to super type to type3 since both type1 and type2 > have same attribute - attr1 > > This patch will check for the exact scenario in case of Classification > Typedef as well. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 5b86b808b > intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java > 85d041bdb > > > Diff: https://reviews.apache.org/r/74101/diff/3/ > > > Testing > --- > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 74101: ATLAS-4668 : When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74101/ --- (Updated Sept. 20, 2022, 10:12 a.m.) Review request for atlas, Jayendra Parab, madhan, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra. Bugs: ATLAS-4668 https://issues.apache.org/jira/browse/ATLAS-4668 Repository: atlas Description --- When an entity type is created with multiple super types having same attribute name, Atlas throws exception. Ex: if there are 2 types: type1 - attr1, attr2 type2- attr1, attr2 Both types cant be added to super type to type3 since both type1 and type2 have same attribute - attr1 This patch will check for the exact scenario in case of Classification Typedef as well. Diffs (updated) - intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 5b86b808b intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java 85d041bdb Diff: https://reviews.apache.org/r/74101/diff/3/ Changes: https://reviews.apache.org/r/74101/diff/2-3/ Testing --- Thanks, Mandar Ambawane
Re: Review Request 74101: ATLAS-4668 : When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74101/ --- (Updated Sept. 20, 2022, 9:24 a.m.) Review request for atlas, Jayendra Parab, madhan, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra. Changes --- Addressed review comments. Added test cases. Bugs: ATLAS-4668 https://issues.apache.org/jira/browse/ATLAS-4668 Repository: atlas Description --- When an entity type is created with multiple super types having same attribute name, Atlas throws exception. Ex: if there are 2 types: type1 - attr1, attr2 type2- attr1, attr2 Both types cant be added to super type to type3 since both type1 and type2 have same attribute - attr1 This patch will check for the exact scenario in case of Classification Typedef as well. Diffs (updated) - intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 5b86b808b intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java 85d041bdb Diff: https://reviews.apache.org/r/74101/diff/2/ Changes: https://reviews.apache.org/r/74101/diff/1-2/ Testing --- Thanks, Mandar Ambawane
[jira] [Created] (ATLAS-4678) Restrict Relationship Typedef creation if "relationshipCategory" value is not provided
Mandar Ambawane created ATLAS-4678: -- Summary: Restrict Relationship Typedef creation if "relationshipCategory" value is not provided Key: ATLAS-4678 URL: https://issues.apache.org/jira/browse/ATLAS-4678 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Mandar Ambawane -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DRAFT] [REPORT] Apache Atlas - September 2022
Thank you Madhan for this report. +1 Thanks, Mandar On Wed, Sep 14, 2022 at 10:28 AM Sarath Subramanian wrote: > Thanks for putting this together Madhan. > > +1 > > Thanks, > Sarath > > On Tue, Sep 13, 2022 at 8:15 PM Madhan Neethiraj > wrote: > > > Atlas community, > > > > > > > > Please review the following draft report for this quarter. > > > > > > > > Thanks, > > > > Madhan > > > > > > > > ## Description: > > > > The mission of Apache Atlas is the creation and maintenance of software > > related > > > > to a scalable and extensible set of core foundational governance > services - > > > > enabling enterprises to effectively and efficiently meet their compliance > > > > requirements > > > > > > > > ## Issues: > > > > There are no issues requiring board attention. > > > > > > > > ## Membership Data: > > > > Apache Atlas was founded 2017-06-20 (5 years ago) There are currently 47 > > > > committers and 32 PMC members in this project. The Committer-to-PMC ratio > > is > > > > roughly 3:2. > > > > > > > > Community changes, past quarter: > > > > - No new PMC members. Last addition was Ashutosh Mestry on 2019-04-16. > > > > - Mandar Ambawane was added as committer on 2022-09-12 > > > > - Prasad Pawar was added as committer on 2022-09-12 > > > > - 4 new Atlas contributors in last quarter: > > > > - Abhishek Pal, Kundan Kumar Jha, Patrik Marton, Beakal Lemeneh > > > > > > > > ## Project Activity: > > > > - fixes in search API handling of pagination > > > > - updates to support authenticated Cassandra cluster as audit repository > > > > - performance improvements in Atlas lineage API and UI > > > > - removed Kafka core dependency in Atlas webapp > > > > - improvements in import-hive utility to delete entities for deleted > > objects > > > > in Hive > > > > - multiple UI fixes and improvements > > > > - dependent component version updates: Spring Framework 5.3.21, Spring > > > > Security 5.7.2, Netty Project 4.1.78.Final, POI 5.2.2, JUnit 4.13.2 > > > > - started a thread on Atlas 2.4.0 release. Looking to complete release by > > > > October 2022 > > > > > > > > > > > > ## Community Health: > > > > - dev@atlas.apache.org had a 22% decrease in traffic in the past quarter > > (680 > > > > emails compared to 868) > > > > - u...@atlas.apache.org had a 37% decrease in traffic in the past > quarter > > (12 > > > > emails compared to 19) > > > > - 48 issues opened in JIRA, past quarter (-12% change) > > > > - 29 issues closed in JIRA, past quarter (38% increase) > > > > - 51 commits in the past quarter (-15% change) > > > > - 13 code contributors in the past quarter (8% increase) > > > > - 16 PRs opened on GitHub, past quarter (-40% change) > > > > - 7 PRs closed on GitHub, past quarter (no change) > > > > > > > > ## Most Recent releases: > > > > 2.2.0 was released on 2021-08-17 > > > > 2.1.0 was released on 2020-07-15 > > > > 0.8.4 was released on 2019-06-21 > > > > > -- Thanks and Regards, Mandar V. Ambawane.
Re: Review Request 74101: ATLAS-4668 : When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74101/ --- (Updated Sept. 6, 2022, 5:36 a.m.) Review request for atlas, Jayendra Parab, madhan, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra. Bugs: ATLAS-4668 https://issues.apache.org/jira/browse/ATLAS-4668 Repository: atlas Description --- When an entity type is created with multiple super types having same attribute name, Atlas throws exception. Ex: if there are 2 types: type1 - attr1, attr2 type2- attr1, attr2 Both types cant be added to super type to type3 since both type1 and type2 have same attribute - attr1 This patch will check for the exact scenario in case of Classification Typedef as well. Diffs - intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 5b86b808b Diff: https://reviews.apache.org/r/74101/diff/1/ Testing --- Thanks, Mandar Ambawane
Review Request 74102: ATLAS-4666 : Intermittently, the audits for creation of hive_db registered are different than expected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74102/ --- Review request for atlas, Jayendra Parab, Pinal Shah, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4666 https://issues.apache.org/jira/browse/ATLAS-4666 Repository: atlas Description --- Sometimes, when I create a hive database and then check for the same in Atlas, the audits tab shows unexpected audit entries: it shows two audit entries both marked as 'Entity Updated' rather than showing a 'Entity Created' audit entry anywhere. This patch will check for the Shell entities created and then assign the Entity status to the Audit entry. Diffs - repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java d48c91709 Diff: https://reviews.apache.org/r/74102/diff/1/ Testing --- Thanks, Mandar Ambawane
[jira] [Created] (ATLAS-4668) When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
Mandar Ambawane created ATLAS-4668: -- Summary: When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception Key: ATLAS-4668 URL: https://issues.apache.org/jira/browse/ATLAS-4668 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Mandar Ambawane Assignee: Mandar Ambawane When an entity type is created with multiple super types having same attribute name, Atlas throws exception. Ex: if there are 2 types: type1 - attr1, attr2 type2- attr1, attr2 Both types cant be added to super type to type3 since both type1 and type2 have same attribute - attr1 But it is not the case with classification. Atlas allows to create a classification with multiple super types having same attribute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4668) When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
[ https://issues.apache.org/jira/browse/ATLAS-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane updated ATLAS-4668: --- Issue Type: Bug (was: Improvement) > When classification is created with multiple super types having same > attributes , Atlas doesn't throw an exception > -- > > Key: ATLAS-4668 > URL: https://issues.apache.org/jira/browse/ATLAS-4668 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > > When an entity type is created with multiple super types having same > attribute name, Atlas throws exception. > Ex: > if there are 2 types: > type1 - attr1, attr2 > type2- attr1, attr2 > Both types cant be added to super type to type3 since both type1 and type2 > have same attribute - attr1 > > But it is not the case with classification. Atlas allows to create a > classification with multiple super types having same attribute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (ATLAS-4666) Intermittently, the audits for creation of hive_db registered are different than expected.
[ https://issues.apache.org/jira/browse/ATLAS-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane reassigned ATLAS-4666: -- Assignee: Mandar Ambawane > Intermittently, the audits for creation of hive_db registered are different > than expected. > -- > > Key: ATLAS-4666 > URL: https://issues.apache.org/jira/browse/ATLAS-4666 > Project: Atlas > Issue Type: Bug >Reporter: Rahul Kurup > Assignee: Mandar Ambawane >Priority: Major > > Sometimes, when I create a hive database and then check for the same in > Atlas, the audits tab shows unexpected audit entries: it shows two audit > entries both marked as 'Entity Updated' rather than showing a 'Entity > Created' audit entry anywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74089: ATLAS-4655 Add Glossary details in Entity - Term Relationship attribute
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74089/ --- (Updated Aug. 16, 2022, 6:09 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, and Sarath Subramanian. Bugs: ATLAS-4655 https://issues.apache.org/jira/browse/ATLAS-4655 Repository: atlas Description --- Add "glossaryDetails" attribute in RelationshipTypeDef "AtlasGlossarySemanticAssignment". This will help in showing the Glossary details of the associated Term in the Entity details page on UI. Diffs (updated) - addons/models/-Area0/0011-glossary_model.json 3ea09bcf2 addons/models/-Area0/patches/007-glossary_model_add_additionalRelationshipAttributes.json PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 89e9422cd repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java a8fe5a762 Diff: https://reviews.apache.org/r/74089/diff/2/ Changes: https://reviews.apache.org/r/74089/diff/1-2/ Testing --- Thanks, Mandar Ambawane
Review Request 74089: ATLAS-4655 Add Glossary details in Entity - Term Relationship attribute
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74089/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian. Bugs: ATLAS-4655 https://issues.apache.org/jira/browse/ATLAS-4655 Repository: atlas Description --- Add "glossaryDetails" attribute in RelationshipTypeDef "AtlasGlossarySemanticAssignment". This will help in showing the Glossary details of the associated Term in the Entity details page on UI. Diffs - addons/models/-Area0/0011-glossary_model.json 3ea09bcf2 addons/models/-Area0/patches/007-glossary_model_add_additionalRelationshipAttributes.json PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 89e9422cd repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java a8fe5a762 Diff: https://reviews.apache.org/r/74089/diff/1/ Testing --- Thanks, Mandar Ambawane
[jira] [Updated] (ATLAS-4655) Add Glossary details in Entity - Term Relationship attribute
[
https://issues.apache.org/jira/browse/ATLAS-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mandar Ambawane updated ATLAS-4655:
---
Component/s: atlas-core
> Add Glossary details in Entity - Term Relationship attribute
>
>
> Key: ATLAS-4655
> URL: https://issues.apache.org/jira/browse/ATLAS-4655
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
> Reporter: Mandar Ambawane
> Assignee: Mandar Ambawane
>Priority: Major
>
> Add "{*}glossaryDetails{*}" attribute in RelationshipTypeDef
> "{*}AtlasGlossarySemanticAssignment{*}".
> This will help in showing the Glossary details of the associated Term in the
> Entity details page on UI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ATLAS-4655) Add Glossary details in Entity - Term Relationship attribute
[
https://issues.apache.org/jira/browse/ATLAS-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mandar Ambawane updated ATLAS-4655:
---
Description:
Add "{*}glossaryDetails{*}" attribute in RelationshipTypeDef
"{*}AtlasGlossarySemanticAssignment{*}".
This will help in showing the Glossary details of the associated Term in the
Entity details page on UI.
> Add Glossary details in Entity - Term Relationship attribute
>
>
> Key: ATLAS-4655
> URL: https://issues.apache.org/jira/browse/ATLAS-4655
> Project: Atlas
> Issue Type: Improvement
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
>
> Add "{*}glossaryDetails{*}" attribute in RelationshipTypeDef
> "{*}AtlasGlossarySemanticAssignment{*}".
> This will help in showing the Glossary details of the associated Term in the
> Entity details page on UI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (ATLAS-4655) Add Glossary details in Entity - Term Relationship attribute
Mandar Ambawane created ATLAS-4655: -- Summary: Add Glossary details in Entity - Term Relationship attribute Key: ATLAS-4655 URL: https://issues.apache.org/jira/browse/ATLAS-4655 Project: Atlas Issue Type: Improvement Reporter: Mandar Ambawane -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (ATLAS-4655) Add Glossary details in Entity - Term Relationship attribute
[ https://issues.apache.org/jira/browse/ATLAS-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane reassigned ATLAS-4655: -- Assignee: Mandar Ambawane > Add Glossary details in Entity - Term Relationship attribute > > > Key: ATLAS-4655 > URL: https://issues.apache.org/jira/browse/ATLAS-4655 > Project: Atlas > Issue Type: Improvement > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4605) Prevent NullPointerException when Atlas Vertex is empty because of corrupted data.
[ https://issues.apache.org/jira/browse/ATLAS-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane updated ATLAS-4605: --- Attachment: ATLAS-4605.patch > Prevent NullPointerException when Atlas Vertex is empty because of corrupted > data. > -- > > Key: ATLAS-4605 > URL: https://issues.apache.org/jira/browse/ATLAS-4605 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > Attachments: ATLAS-4605.patch > > > Because of corrupted data, there may be a chance of having AtlasVertex as an > empty, which means AtlasVertex does not have any property present in it. > This can cause NullPointerException while populating System attributes as > well Normal attributes for AtlasEntity object. > To avoid NullPointerException we need to skip such vertices, so that they > cannot be used to populate system attributes and normal attributes. -- This message was sent by Atlassian Jira (v8.20.7#820007)
Review Request 73991: ATLAS-4605 Prevent NullPointerException when Atlas Vertex is empty because of corrupted data
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73991/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4605 https://issues.apache.org/jira/browse/ATLAS-4605 Repository: atlas Description --- Because of corrupted data, there may be a chance of having AtlasVertex as an empty, which means AtlasVertex does not have any property present in it. This can cause NullPointerException while populating System attributes as well Normal attributes for AtlasEntity object. To avoid NullPointerException we need to skip such vertices, so that they cannot be used to populate system attributes and normal attributes. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java fcd8dfe7f Diff: https://reviews.apache.org/r/73991/diff/1/ Testing --- Sanity testing is done. Thanks, Mandar Ambawane
[jira] [Created] (ATLAS-4605) Prevent NullPointerException when Atlas Vertex is empty because of corrupted data.
Mandar Ambawane created ATLAS-4605: -- Summary: Prevent NullPointerException when Atlas Vertex is empty because of corrupted data. Key: ATLAS-4605 URL: https://issues.apache.org/jira/browse/ATLAS-4605 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Mandar Ambawane Assignee: Mandar Ambawane Because of corrupted data, there may be a chance of having AtlasVertex as an empty, which means AtlasVertex does not have any property present in it. This can cause NullPointerException while populating System attributes as well Normal attributes for AtlasEntity object. To avoid NullPointerException we need to skip such vertices, so that they cannot be used to populate system attributes and normal attributes. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (ATLAS-4572) Improved Soft Delete behaviour for relationships edges
[ https://issues.apache.org/jira/browse/ATLAS-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane resolved ATLAS-4572. Resolution: Fixed > Improved Soft Delete behaviour for relationships edges > -- > > Key: ATLAS-4572 > URL: https://issues.apache.org/jira/browse/ATLAS-4572 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > > Earlier the process for soft deleting relationships involved unnecessary > invocation of delete methods on already deleted Relationship edges. > This would consume a lot of time on an entity which has a long list of soft > deleted relationships. > This changes implements a check on relationship edges, identifying already > deleted relationship edges and avoiding invocation of delete method on them. > Thus only allowing deletion of active relationship edges. -- This message was sent by Atlassian Jira (v8.20.7#820007)
Re: Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73731/ --- (Updated April 28, 2022, 7:43 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Radhika Kundam, and Sarath Subramanian. Bugs: ATLAS-4495 https://issues.apache.org/jira/browse/ATLAS-4495 Repository: atlas Description --- It is observed that, due to existing framework Glossary GET api takes longer time. This patch avoids the unnecessary db calls and improve the performance when there is large number of data associated with the Glossary (say 10K terms or categories) Diffs - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java f81b53854 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java f902b2afb repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AbstractGlossaryDTO.java bee88c61f repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 9671df994 Diff: https://reviews.apache.org/r/73731/diff/4/ Testing (updated) --- Manual testing is done. PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1009/consoleFull PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1049/consoleFull PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1053/ There are total 11K Terms distributed among 11 Glossaries. For Glossary GET api without any performance changes it took: 1. 180 seconds 2. 168.814 seconds 3. 169.237 seconds AVERAGE: 172.683 seconds For Glossary GET api with performance changes it took: 1. 45 seconds 2. 45 seconds 3. 43.626 seconds AVERAGE: 44.542 seconds Thanks, Mandar Ambawane
Re: Review Request 73929: ATLAS-4576 Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73929/ --- (Updated April 26, 2022, 5:57 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Changes --- 1. Addressed review comments. 2. Added test cases for ClassificationType and EntityType. Bugs: ATLAS-4576 https://issues.apache.org/jira/browse/ATLAS-4576 Repository: atlas Description --- Currently we have provided 2 checks for the Attribute Name: https://issues.apache.org/jira/browse/ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name https://issues.apache.org/jira/browse/ATLAS-4522 Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes But in the earlier versions these checks were not there. So there may be a chance of Atlas environment having such data where these checks can cause problems at the time: 1. Atlas Startup. 2. Creating new typedefs This patch handles these scenarios and provides backward compatibility for these 2 changes mentioned. Diffs (updated) - intg/src/main/java/org/apache/atlas/ThreadContext.java PRE-CREATION intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 76bee36f2 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 21ce23657 intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 0c05bb399 intg/src/test/java/org/apache/atlas/type/TestAtlasClassificationType.java b6055fa30 intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java 742970390 webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3eab Diff: https://reviews.apache.org/r/73929/diff/3/ Changes: https://reviews.apache.org/r/73929/diff/2-3/ Testing (updated) --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1084/consoleFull Thanks, Mandar Ambawane
Re: Review Request 73917: ATLAS-4572 Improved Soft Delete behavior for relationships edges
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73917/ --- (Updated April 20, 2022, 6:20 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Changes --- Addressed review comments Bugs: ATLAS-4572 https://issues.apache.org/jira/browse/ATLAS-4572 Repository: atlas Description --- Earlier the process for soft deleting relationships involved unnecessary invocation of delete methods on already deleted Relationship edges. This would consume a lot of time on an entity which has a long list of soft deleted relationships. This changes implements a check on relationship edges, identifying already deleted relationship edges and avoiding invocation of delete method on them. Thus only allowing deletion of active relationship edges. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java f118ae69a Diff: https://reviews.apache.org/r/73917/diff/3/ Changes: https://reviews.apache.org/r/73917/diff/2-3/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1065/ We took 2 cluster, one with performance patch applied and other one without any changes. We loaded kafka dump on both the clusters. There were around 170k spark process entities. On cluster without any changes, it took more than 48 hours. On cluster with performance improvement changes, it took 25 hours to consume the entire kafka dump. Also, On cluster without changes it was taking approx 45 seconds to process each Kafka message. On cluster with performance improvement changes it was taking 3 to 5 seconds to process each Kafka message. Also we did sanity testing for Atlas. Thanks, Mandar Ambawane
Re: Review Request 73917: ATLAS-4572 Improved Soft Delete behavior for relationships edges
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73917/ --- (Updated April 20, 2022, 9:46 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4572 https://issues.apache.org/jira/browse/ATLAS-4572 Repository: atlas Description --- Earlier the process for soft deleting relationships involved unnecessary invocation of delete methods on already deleted Relationship edges. This would consume a lot of time on an entity which has a long list of soft deleted relationships. This changes implements a check on relationship edges, identifying already deleted relationship edges and avoiding invocation of delete method on them. Thus only allowing deletion of active relationship edges. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java f118ae69a Diff: https://reviews.apache.org/r/73917/diff/2/ Testing (updated) --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1065/ We took 2 cluster, one with performance patch applied and other one without any changes. We loaded kafka dump on both the clusters. There were around 170k spark process entities. On cluster without any changes, it took more than 48 hours. On cluster with performance improvement changes, it took 25 hours to consume the entire kafka dump. Also, On cluster without changes it was taking approx 45 seconds to process each Kafka message. On cluster with performance improvement changes it was taking 3 to 5 seconds to process each Kafka message. Also we did sanity testing for Atlas. Thanks, Mandar Ambawane
Re: Review Request 73917: ATLAS-4572 Improved Soft Delete behavior for relationships edges
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73917/ --- (Updated April 13, 2022, 5:44 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4572 https://issues.apache.org/jira/browse/ATLAS-4572 Repository: atlas Description --- Earlier the process for soft deleting relationships involved unnecessary invocation of delete methods on already deleted Relationship edges. This would consume a lot of time on an entity which has a long list of soft deleted relationships. This changes implements a check on relationship edges, identifying already deleted relationship edges and avoiding invocation of delete method on them. Thus only allowing deletion of active relationship edges. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java f118ae69a Diff: https://reviews.apache.org/r/73917/diff/2/ Changes: https://reviews.apache.org/r/73917/diff/1-2/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1065/ Thanks, Mandar Ambawane
Re: Review Request 73929: ATLAS-4576 Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73929/ --- (Updated April 12, 2022, 3:39 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Changes --- Added support for Update api. Code cleanup. Bugs: ATLAS-4576 https://issues.apache.org/jira/browse/ATLAS-4576 Repository: atlas Description --- Currently we have provided 2 checks for the Attribute Name: https://issues.apache.org/jira/browse/ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name https://issues.apache.org/jira/browse/ATLAS-4522 Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes But in the earlier versions these checks were not there. So there may be a chance of Atlas environment having such data where these checks can cause problems at the time: 1. Atlas Startup. 2. Creating new typedefs This patch handles these scenarios and provides backward compatibility for these 2 changes mentioned. Diffs (updated) - intg/src/main/java/org/apache/atlas/ThreadContext.java PRE-CREATION intg/src/main/java/org/apache/atlas/constants/ThreadContextConstants.java PRE-CREATION intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 76bee36f2 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 21ce23657 intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 0c05bb399 webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3eab Diff: https://reviews.apache.org/r/73929/diff/2/ Changes: https://reviews.apache.org/r/73929/diff/1-2/ Testing --- Thanks, Mandar Ambawane
Review Request 73929: ATLAS-4576 Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73929/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4576 https://issues.apache.org/jira/browse/ATLAS-4576 Repository: atlas Description --- Currently we have provided 2 checks for the Attribute Name: https://issues.apache.org/jira/browse/ATLAS-3872 Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name https://issues.apache.org/jira/browse/ATLAS-4522 Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes But in the earlier versions these checks were not there. So there may be a chance of Atlas environment having such data where these checks can cause problems at the time: 1. Atlas Startup. 2. Creating new typedefs This patch handles these scenarios and provides backward compatibility for these 2 changes mentioned. Diffs - intg/src/main/java/org/apache/atlas/ThreadContext.java PRE-CREATION intg/src/main/java/org/apache/atlas/constants/ThreadContextConstants.java PRE-CREATION intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 76bee36f2 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 21ce23657 intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 0c05bb399 webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3eab Diff: https://reviews.apache.org/r/73929/diff/1/ Testing --- Thanks, Mandar Ambawane
[jira] [Updated] (ATLAS-4572) Improved Soft Delete behaviour for relationships edges
[ https://issues.apache.org/jira/browse/ATLAS-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane updated ATLAS-4572: --- Summary: Improved Soft Delete behaviour for relationships edges (was: Improved Soft Delete behavior for relationships edges) > Improved Soft Delete behaviour for relationships edges > -- > > Key: ATLAS-4572 > URL: https://issues.apache.org/jira/browse/ATLAS-4572 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > > Earlier the process for soft deleting relationships involved unnecessary > invocation of delete methods on already deleted Relationship edges. > This would consume a lot of time on an entity which has a long list of soft > deleted relationships. > This changes implements a check on relationship edges, identifying already > deleted relationship edges and avoiding invocation of delete method on them. > Thus only allowing deletion of active relationship edges. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (ATLAS-4576) Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
[ https://issues.apache.org/jira/browse/ATLAS-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane reassigned ATLAS-4576: -- Assignee: Mandar Ambawane > Backward compatibility for check provided for AttributeName in Parent and > Child TypeDef > --- > > Key: ATLAS-4576 > URL: https://issues.apache.org/jira/browse/ATLAS-4576 > Project: Atlas > Issue Type: Improvement > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > > Currently we have provided 2 checks for the Attribute Name: > [ATLAS-3872|https://issues.apache.org/jira/browse/ATLAS-3872] > Restrict typedef creation when a child type attribute conflicts with parent > type attribute of same name > [ATLAS-4522|https://issues.apache.org/jira/browse/ATLAS-4522] > Updating typedef with new supertype should be allowed only if attributes are > unique compared to other existing supertypes > But in the earlier versions these checks were not there. So there may be a > chance of Atlas environment having such data where these checks can cause > problems at the time: > 1. Atlas Startup. > 2. Creating new typedefs > This patch handles these scenarios and provides backward compatibility for > these 2 changes mentioned. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (ATLAS-4576) Backward compatibility for check provided for AttributeName in Parent and Child TypeDef
[ https://issues.apache.org/jira/browse/ATLAS-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane updated ATLAS-4576: --- Summary: Backward compatibility for check provided for AttributeName in Parent and Child TypeDef (was: Backward compatibility for check provided for AttributeName in Parent and Child Type) > Backward compatibility for check provided for AttributeName in Parent and > Child TypeDef > --- > > Key: ATLAS-4576 > URL: https://issues.apache.org/jira/browse/ATLAS-4576 > Project: Atlas > Issue Type: Improvement > Reporter: Mandar Ambawane >Priority: Major > > Currently we have provided 2 checks for the Attribute Name: > [ATLAS-3872|https://issues.apache.org/jira/browse/ATLAS-3872] > Restrict typedef creation when a child type attribute conflicts with parent > type attribute of same name > [ATLAS-4522|https://issues.apache.org/jira/browse/ATLAS-4522] > Updating typedef with new supertype should be allowed only if attributes are > unique compared to other existing supertypes > But in the earlier versions these checks were not there. So there may be a > chance of Atlas environment having such data where these checks can cause > problems at the time: > 1. Atlas Startup. > 2. Creating new typedefs > This patch handles these scenarios and provides backward compatibility for > these 2 changes mentioned. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (ATLAS-4576) Backward compatibility for check provided for AttributeName in Parent and Child Type
[ https://issues.apache.org/jira/browse/ATLAS-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane updated ATLAS-4576: --- Summary: Backward compatibility for check provided for AttributeName in Parent and Child Type (was: Backward compatibility for check provided for Attribute Name in Parent and Child Type) > Backward compatibility for check provided for AttributeName in Parent and > Child Type > > > Key: ATLAS-4576 > URL: https://issues.apache.org/jira/browse/ATLAS-4576 > Project: Atlas > Issue Type: Improvement > Reporter: Mandar Ambawane >Priority: Major > > Currently we have provided 2 checks for the Attribute Name: > [ATLAS-3872|https://issues.apache.org/jira/browse/ATLAS-3872] > Restrict typedef creation when a child type attribute conflicts with parent > type attribute of same name > [ATLAS-4522|https://issues.apache.org/jira/browse/ATLAS-4522] > Updating typedef with new supertype should be allowed only if attributes are > unique compared to other existing supertypes > But in the earlier versions these checks were not there. So there may be a > chance of Atlas environment having such data where these checks can cause > problems at the time: > 1. Atlas Startup. > 2. Creating new typedefs > This patch handles these scenarios and provides backward compatibility for > these 2 changes mentioned. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (ATLAS-4576) Backward compatibility for check provided for Attribute Name in Parent and Child Type
Mandar Ambawane created ATLAS-4576: -- Summary: Backward compatibility for check provided for Attribute Name in Parent and Child Type Key: ATLAS-4576 URL: https://issues.apache.org/jira/browse/ATLAS-4576 Project: Atlas Issue Type: Improvement Reporter: Mandar Ambawane Currently we have provided 2 checks for the Attribute Name: [ATLAS-3872|https://issues.apache.org/jira/browse/ATLAS-3872] Restrict typedef creation when a child type attribute conflicts with parent type attribute of same name [ATLAS-4522|https://issues.apache.org/jira/browse/ATLAS-4522] Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes But in the earlier versions these checks were not there. So there may be a chance of Atlas environment having such data where these checks can cause problems at the time: 1. Atlas Startup. 2. Creating new typedefs This patch handles these scenarios and provides backward compatibility for these 2 changes mentioned. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Review Request 73917: ATLAS-4572 Improved Soft Delete behavior for relationships edges
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73917/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4572 https://issues.apache.org/jira/browse/ATLAS-4572 Repository: atlas Description --- Earlier the process for soft deleting relationships involved unnecessary invocation of delete methods on already deleted Relationship edges. This would consume a lot of time on an entity which has a long list of soft deleted relationships. This changes implements a check on relationship edges, identifying already deleted relationship edges and avoiding invocation of delete method on them. Thus only allowing deletion of active relationship edges. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java f118ae69a Diff: https://reviews.apache.org/r/73917/diff/1/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1065/ Thanks, Mandar Ambawane
Re: Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
> On March 5, 2022, 4:18 p.m., Ashutosh Mestry wrote: > > Can you please add metrics about the improvement you are seeing. There are total 11K Terms distributed among 11 Glossaries. For Glossary GET api without any performance changes it took: 1. 180 seconds 2. 168.814 seconds 3. 169.237 seconds AVERAGE: 172.683 seconds For Glossary GET api with performance changes it took: 1. 45 seconds 2. 45 seconds 3. 43.626 seconds AVERAGE: 44.542 seconds - Mandar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73731/#review224133 --- On March 7, 2022, 10:56 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73731/ > --- > > (Updated March 7, 2022, 10:56 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Radhika Kundam, > and Sarath Subramanian. > > > Bugs: ATLAS-4495 > https://issues.apache.org/jira/browse/ATLAS-4495 > > > Repository: atlas > > > Description > --- > > It is observed that, due to existing framework Glossary GET api takes longer > time. > > This patch avoids the unnecessary db calls and improve the performance when > there is large number of data associated with the Glossary (say 10K terms or > categories) > > > Diffs > - > > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > f81b53854 > repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java > f902b2afb > > repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AbstractGlossaryDTO.java > bee88c61f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 9671df994 > > > Diff: https://reviews.apache.org/r/73731/diff/4/ > > > Testing > --- > > Manual testing is done. > > PreCommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1009/consoleFull > > PreCommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1049/consoleFull > > PreCommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1053/ > > > Thanks, > > Mandar Ambawane > >
[jira] [Created] (ATLAS-4572) Improved Soft Delete behavior for relationships edges
Mandar Ambawane created ATLAS-4572: -- Summary: Improved Soft Delete behavior for relationships edges Key: ATLAS-4572 URL: https://issues.apache.org/jira/browse/ATLAS-4572 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Mandar Ambawane Assignee: Mandar Ambawane Earlier the process for soft deleting relationships involved unnecessary invocation of delete methods on already deleted Relationship edges. This would consume a lot of time on an entity which has a long list of soft deleted relationships. This changes implements a check on relationship edges, identifying already deleted relationship edges and avoiding invocation of delete method on them. Thus only allowing deletion of active relationship edges. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73731/ --- (Updated March 7, 2022, 10:56 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Radhika Kundam, and Sarath Subramanian. Changes --- Addressed review comments Bugs: ATLAS-4495 https://issues.apache.org/jira/browse/ATLAS-4495 Repository: atlas Description --- It is observed that, due to existing framework Glossary GET api takes longer time. This patch avoids the unnecessary db calls and improve the performance when there is large number of data associated with the Glossary (say 10K terms or categories) Diffs (updated) - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java f81b53854 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java f902b2afb repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AbstractGlossaryDTO.java bee88c61f repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 9671df994 Diff: https://reviews.apache.org/r/73731/diff/4/ Changes: https://reviews.apache.org/r/73731/diff/3-4/ Testing --- Manual testing is done. PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1009/consoleFull PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1049/consoleFull Thanks, Mandar Ambawane
Re: Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73731/ --- (Updated March 3, 2022, 12:44 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Radhika Kundam, and Sarath Subramanian. Changes --- Addressed review comments Bugs: ATLAS-4495 https://issues.apache.org/jira/browse/ATLAS-4495 Repository: atlas Description --- It is observed that, due to existing framework Glossary GET api takes longer time. This patch avoids the unnecessary db calls and improve the performance when there is large number of data associated with the Glossary (say 10K terms or categories) Diffs (updated) - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java f81b53854 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java f902b2afb repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AbstractGlossaryDTO.java bee88c61f repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 9671df994 Diff: https://reviews.apache.org/r/73731/diff/3/ Changes: https://reviews.apache.org/r/73731/diff/2-3/ Testing (updated) --- Manual testing is done. PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1009/consoleFull PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1049/consoleFull Thanks, Mandar Ambawane
[jira] [Updated] (ATLAS-4556) Add Ozone "ofs" scheme support for external tables created through hive.
[ https://issues.apache.org/jira/browse/ATLAS-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane updated ATLAS-4556: --- Issue Type: Bug (was: Improvement) > Add Ozone "ofs" scheme support for external tables created through hive. > > > Key: ATLAS-4556 > URL: https://issues.apache.org/jira/browse/ATLAS-4556 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > > When we provide ozone key as a path for an external table in hive, > corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", > "Ozone_Key" etc. get created. > Currently in Atlas, code supports the above in "o3fs" ozone scheme. > Need to provide improved support in "ofs" scheme, same as that of "o3fs". -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (ATLAS-4556) Add Ozone "ofs" scheme support for external tables created through hive.
[ https://issues.apache.org/jira/browse/ATLAS-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane resolved ATLAS-4556. Resolution: Fixed > Add Ozone "ofs" scheme support for external tables created through hive. > > > Key: ATLAS-4556 > URL: https://issues.apache.org/jira/browse/ATLAS-4556 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > > When we provide ozone key as a path for an external table in hive, > corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", > "Ozone_Key" etc. get created. > Currently in Atlas, code supports the above in "o3fs" ozone scheme. > Need to provide improved support in "ofs" scheme, same as that of "o3fs". -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Review Request 73855: ATLAS-4556 Add Ozone "ofs" scheme support for external tables created through hive.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73855/ --- (Updated Feb. 23, 2022, 5:50 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, Sarath Subramanian, and Sidharth Mishra. Changes --- Addressed review comments Bugs: ATLAS-4556 https://issues.apache.org/jira/browse/ATLAS-4556 Repository: atlas Description --- When we provide ozone key as a path for an external table in hive, corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", "Ozone_Key" etc. get created. Currently in Atlas, code supports the above in "o3fs" ozone scheme. Need to provide improved support in "ofs" scheme, same as that of "o3fs". Diffs (updated) - common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java cf8f262b9 common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java f35e9ae57 Diff: https://reviews.apache.org/r/73855/diff/3/ Changes: https://reviews.apache.org/r/73855/diff/2-3/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1046/consoleFull Thanks, Mandar Ambawane
Re: Review Request 73855: ATLAS-4556 Add Ozone "ofs" scheme support for external tables created through hive.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73855/ --- (Updated Feb. 21, 2022, 12:39 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, Sarath Subramanian, and Sidharth Mishra. Changes --- Addressed review comments Bugs: ATLAS-4556 https://issues.apache.org/jira/browse/ATLAS-4556 Repository: atlas Description --- When we provide ozone key as a path for an external table in hive, corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", "Ozone_Key" etc. get created. Currently in Atlas, code supports the above in "o3fs" ozone scheme. Need to provide improved support in "ofs" scheme, same as that of "o3fs". Diffs (updated) - common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java cf8f262b9 common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java f35e9ae57 Diff: https://reviews.apache.org/r/73855/diff/2/ Changes: https://reviews.apache.org/r/73855/diff/1-2/ Testing --- Thanks, Mandar Ambawane
Review Request 73855: ATLAS-4556 Add Ozone "ofs" scheme support for external tables created through hive.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73855/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4556 https://issues.apache.org/jira/browse/ATLAS-4556 Repository: atlas Description --- When we provide ozone key as a path for an external table in hive, corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", "Ozone_Key" etc. get created. Currently in Atlas, code supports the above in "o3fs" ozone scheme. Need to provide improved support in "ofs" scheme, same as that of "o3fs". Diffs - common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java cf8f262b9 common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java f35e9ae57 Diff: https://reviews.apache.org/r/73855/diff/1/ Testing --- Thanks, Mandar Ambawane
[jira] [Created] (ATLAS-4556) Add Ozone "ofs" scheme support for external tables created through hive.
Mandar Ambawane created ATLAS-4556: -- Summary: Add Ozone "ofs" scheme support for external tables created through hive. Key: ATLAS-4556 URL: https://issues.apache.org/jira/browse/ATLAS-4556 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Mandar Ambawane When we provide ozone key as a path for an external table in hive, corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", "Ozone_Key" etc. get created. Currently in Atlas, code supports the above in "o3fs" ozone scheme. Need to provide improved support in "ofs" scheme, same as that of "o3fs". -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (ATLAS-4556) Add Ozone "ofs" scheme support for external tables created through hive.
[ https://issues.apache.org/jira/browse/ATLAS-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane reassigned ATLAS-4556: -- Assignee: Mandar Ambawane > Add Ozone "ofs" scheme support for external tables created through hive. > > > Key: ATLAS-4556 > URL: https://issues.apache.org/jira/browse/ATLAS-4556 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > > When we provide ozone key as a path for an external table in hive, > corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", > "Ozone_Key" etc. get created. > Currently in Atlas, code supports the above in "o3fs" ozone scheme. > Need to provide improved support in "ofs" scheme, same as that of "o3fs". -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73731/ --- (Updated Dec. 21, 2021, 1:31 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Bugs: ATLAS-4495 https://issues.apache.org/jira/browse/ATLAS-4495 Repository: atlas Description --- It is observed that, due to existing framework Glossary GET api takes longer time. This patch avoids the unnecessary db calls and improve the performance when there is large number of data associated with the Glossary (say 10K terms or categories) Diffs - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java f81b53854 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java f902b2afb repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AtlasGlossaryDTO.java c525bdb99 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 794847555 Diff: https://reviews.apache.org/r/73731/diff/2/ Testing (updated) --- Manual testing is done. PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1009/consoleFull Thanks, Mandar Ambawane
Re: Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73731/ --- (Updated Dec. 6, 2021, 3 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Changes --- Some code cleanup and refactoring Bugs: ATLAS-4495 https://issues.apache.org/jira/browse/ATLAS-4495 Repository: atlas Description --- It is observed that, due to existing framework Glossary GET api takes longer time. This patch avoids the unnecessary db calls and improve the performance when there is large number of data associated with the Glossary (say 10K terms or categories) Diffs (updated) - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java f81b53854 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java f902b2afb repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AtlasGlossaryDTO.java c525bdb99 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 794847555 Diff: https://reviews.apache.org/r/73731/diff/2/ Changes: https://reviews.apache.org/r/73731/diff/1-2/ Testing --- Manual testing is done. Thanks, Mandar Ambawane
[jira] [Updated] (ATLAS-4473) Glossary Terms Bulk Create Performance Improvement
[ https://issues.apache.org/jira/browse/ATLAS-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane updated ATLAS-4473: --- Attachment: ATLAS-4473-V2.patch > Glossary Terms Bulk Create Performance Improvement > -- > > Key: ATLAS-4473 > URL: https://issues.apache.org/jira/browse/ATLAS-4473 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > Attachments: ATLAS-4473-V2.patch, ATLAS-4473.patch > > > While creating GlossaryTerms we need information about Glossary and > GlossaryCategory (if present) > As per current design, > When every Term is created, respective Glossary and Category objects are > fetched from db and they are populated with unwanted information. > In case of bulk creation, > Even if multiple Terms have same Glossary and Category associated with them, > every time these objects are fetched from db and they are populated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73731/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath Subramanian. Bugs: ATLAS-4495 https://issues.apache.org/jira/browse/ATLAS-4495 Repository: atlas Description --- It is observed that, due to existing framework Glossary GET api takes longer time. This patch avoids the unnecessary db calls and improve the performance when there is large number of data associated with the Glossary (say 10K terms or categories) Diffs - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 9c8459874 repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java f902b2afb repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AtlasGlossaryDTO.java c525bdb99 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 794847555 Diff: https://reviews.apache.org/r/73731/diff/1/ Testing --- Manual testing is done. Thanks, Mandar Ambawane
[jira] [Assigned] (ATLAS-4495) Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
[ https://issues.apache.org/jira/browse/ATLAS-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane reassigned ATLAS-4495: -- Assignee: Mandar Ambawane > Glossary section takes longer time to load when Glossaries have large number > of Categories and Terms associated with them > - > > Key: ATLAS-4495 > URL: https://issues.apache.org/jira/browse/ATLAS-4495 > Project: Atlas > Issue Type: Improvement > Reporter: Mandar Ambawane > Assignee: Mandar Ambawane >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (ATLAS-4495) Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them
Mandar Ambawane created ATLAS-4495: -- Summary: Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them Key: ATLAS-4495 URL: https://issues.apache.org/jira/browse/ATLAS-4495 Project: Atlas Issue Type: Improvement Reporter: Mandar Ambawane -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Review Request 73700: ATLAS-4473 GlossaryTerms Bulk Create Performance Improvement
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73700/ --- (Updated Nov. 16, 2021, 4:22 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian. Changes --- Addressed review comments Bugs: ATLAS-4473 https://issues.apache.org/jira/browse/ATLAS-4473 Repository: atlas Description --- While creating GlossaryTerms we need information about Glossary and GlossaryCategory (if present) As per current design, When every Term is created, respective Glossary and Category objects are fetched from db and they are populated with unwanted information. In case of bulk creation, Even if multiple Terms have same Glossary and Category associated with them, every time these objects are fetched from db and they are populated. This patch stores the optimum information of Glossaries and Categoies in the cache against their respective Guids. This helps in performance improvement of Bulk Term Creation. Diffs (updated) - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 9c8459874 repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java d92daeea5 Diff: https://reviews.apache.org/r/73700/diff/2/ Changes: https://reviews.apache.org/r/73700/diff/1-2/ Testing --- Earlier, it was taking more than a day to complete the creation process of the Terms. Now with the Performacne optimization, it is taking 5 to 6 minutes to complete. Thanks, Mandar Ambawane
Review Request 73700: ATLAS-4473 Glossary Terms Bulk Create Performance Improvement
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73700/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian. Bugs: ATLAS-4473 https://issues.apache.org/jira/browse/ATLAS-4473 Repository: atlas Description --- While creating GlossaryTerms we need information about Glossary and GlossaryCategory (if present) As per current design, When every Term is created, respective Glossary and Category objects are fetched from db and they are populated with unwanted information. In case of bulk creation, Even if multiple Terms have same Glossary and Category associated with them, every time these objects are fetched from db and they are populated. This patch stores the optimum information of Glossaries and Categoies in the cache against their respective Guids. This helps in performance improvement of Bulk Term Creation. Diffs - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 9c8459874 repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java d92daeea5 Diff: https://reviews.apache.org/r/73700/diff/1/ Testing --- Earlier, it was taking more than a day to complete the creation process of the Terms. Now with the Performacne optimization, it is taking 5 to 6 minutes to complete. Thanks, Mandar Ambawane
[jira] [Created] (ATLAS-4473) Glossary Terms Bulk Create Performance Improvement
Mandar Ambawane created ATLAS-4473: -- Summary: Glossary Terms Bulk Create Performance Improvement Key: ATLAS-4473 URL: https://issues.apache.org/jira/browse/ATLAS-4473 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Mandar Ambawane Assignee: Mandar Ambawane While creating GlossaryTerms we need information about Glossary and GlossaryCategory (if present) As per current design, When every Term is created, respective Glossary and Category objects are fetched from db and they are populated with unwanted information. In case of bulk creation, Even if multiple Terms have same Glossary and Category associated with them, every time these objects are fetched from db and they are populated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Review Request 73642: ATLAS-1558 Running Quickstart V2 multiple times throws an exception type already exists
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73642/#review223701 --- webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java Lines 226 (patched) <https://reviews.apache.org/r/73642/#comment312811> This is always going to be "True". Please retrieve error details from the Exception object and then proceed further. - Mandar Ambawane On Oct. 29, 2021, 7:27 a.m., Shraddha Pardeshi wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73642/ > --- > > (Updated Oct. 29, 2021, 7:27 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah. > > > Bugs: ATLAS-1558 > https://issues.apache.org/jira/browse/ATLAS-1558 > > > Repository: atlas > > > Description > --- > > When you try to run QuickStart V2 multiple times, after the first succesfull > run, it throws exception that types already exists. QuickStart should exit > gracefully with types already exist message. > > > Diffs > - > > webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 5309d1993 > > > Diff: https://reviews.apache.org/r/73642/diff/1/ > > > Testing > --- > > > Thanks, > > Shraddha Pardeshi > >
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
Hi Sidharth,
+1 for Apache Atlas 2.2.0 rc1
Verified following:
- md5, sha512 signatures
- mvn build with embedded-hbase-solr -DskipTests
- Started Atlas server and did run quick start
- Atlas UI loading cleanly
- Created entities using Atlas UI
- Validated basic search
- Validated Admin audits
Thanks,
Mandar
On Wed, Aug 11, 2021 at 12:06 PM Chaitali Borole <
chaitali.bor...@freestoneinfotech.com> wrote:
> Hello Sidharth,
>
> +1 for Apache Atlas 2.2.0 rc1
>
> Checked for below points:
> * md5, sha512 signatures.
> * Clean build with mvn clean package -Pdist,embedded-hbase-solr
> -DskipTests -Drat.skip.
> * Atlas server up and running.
> * Re-indexing with JAVA_PATCH.
> * Sanity checks :
> quick_search,entities,relationship,tags,glossaries,metrics api.
> * Quick start is successful.
>
> Thanks,
> Chaitali.
>
> On Tue, Aug 10, 2021 at 4:25 AM Sidharth Mishra
> wrote:
>
> > Atlas team,
> >
> > Apache Atlas 2.2.0 rc1, with the following fixes since rc0, is now
> > available for vote within the dev community.
> >
> > ATLAS-4390 - Changed Atlas branch to release-2.2.0-rc1
> > ATLAS-4391 - NoSuchFileException for
> {atlas.log.dir}/atlas-metrics.out
> > ATLAS-4343 - [Entity Audits] 'Propagated Classification Added'
> > Timestamp is < 'Entity
> > Created' Timestamp
> > ATLAS-4387 - UI: Propagated Classifications shows wrong count for
> > entity type on Entity detail
> > page
> > ATLAS-4386 - Relationship search: Sorting does not work when
> > relationship label is passed
> >
> > Links to the release artifacts are given below. Please review and vote.
> >
> > The vote will be open for at least 72 hours or until necessary votes
> > are reached.
> >
> > [ ] +1 Approve
> >
> > [ ] +0 No opinion
> >
> > [ ] -1 Disapprove (and reason why)
> >
> > Thanks,
> > Sid
> >
> > List of all issues addressed in this release:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
> >
> > Git tag for the release:
> > https://github.com/apache/atlas/tree/release-2.2.0-rc1
> >
> > Sources for the release:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz
> >
> > Source release verification:
> >
> > PGP Signature:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.asc
> >
> > SHA512 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.sha512
> >
> > MD5 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.md5
> >
> >
> > Keys to verify the signature of the release artifacts are available
> > at: https://dist.apache.org/repos/dist/release/atlas/KEYS
> >
> >
> > New features/enhancements:
> >
> > - Introduced Deferred Action - Tag propagation will be handled as
> > a background process (AtlasTask)
> >
> > - Added Re-indexing as JAVA_PATCH
> >
> > - Created JavaPatch to add new super types to existing entities
> >
> > - Supported for Business Metadata in Atlas Export API
> >
> > - Supported HA for admin/task API
> >
> > - Enhanced export service to export Terms
> >
> > - Added new models and addition attributes to Atlas for Apache
> > Flink, GCP and ADLS Gen 2
> >
> > - Added support to Configure HiveServer2 Hook to send Lineage-only
> > Messages
> >
> > - Improved and enhanced Bulk Glossary Import supporting import
> > with relations
> >
> > - Dependency between Atlas and Kafka Zookeeper removed
> >
> > - Improved performance of GraphHelper's guid and status getter
> methods
> >
> > - Added new privilege - "admin-audits" for Atlas Admin Audits
> > authorization
> >
> > - Basic search Improvements - search text with special characters,
> > optimize pagination
> >
> > - Updated free-text search processor to support Elasticsearch
> >
> > - Refactored and enhanced Atlas Python Client supporting Python 2.7
> >
> > - DSL Search enhancements - Support glossary terms and
> > relationship, can handle null attribute values, Java Tinkerpop
> > Traversals instead of GremlinScriptEngine, support search by
> > classification and its attribute, Caching mechanism for translated dsl
> > queries
> >
> > - Enhanced Export/Import Service to conditionally Support
> > Simultaneous Operations
> >
> > - Enhanced NotificationHookConsumer for Large Message Processing
> >
> > - Enhanced Atlas authorization for Add/Update/Remove
> > classification on entities
> >
> > - Provided option to add mandatory attribute to existing entity
> > definition
> >
> > - Fixed Atlas UI to load faster in case of large number of
> > classifications & entities
> >
> > - Updated versions of dependent libraries/components: JanusGraph,
> > elasticsearch, JQuery, Http core, Http Client, slf4j, log4j,
[jira] [Resolved] (ATLAS-3694) [Business Metadata] Max length defined for attributes of string type is not getting enforced
[
https://issues.apache.org/jira/browse/ATLAS-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mandar Ambawane resolved ATLAS-3694.
Resolution: Fixed
> [Business Metadata] Max length defined for attributes of string type is not
> getting enforced
>
>
> Key: ATLAS-3694
> URL: https://issues.apache.org/jira/browse/ATLAS-3694
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-webui
>Reporter: Umesh Padashetty
>Priority: Major
> Attachments: Screenshot 2020-03-26 at 7.49.51 PM.png, Screenshot
> 2020-03-26 at 7.51.13 PM.png
>
>
> Even when the max length applicable to an attribute is defined as 10, I was
> able to add all the alphabets of an english language i.e. a-z, 26 in length,
> as the attribute value.
> This is reproducible both from API as well as UI.
> {code:java}
> curl -X POST \
>
> 'https://:31443/api/atlas/v2/entity/guid/ab29815d-6d20-44ee-b426-8fba946b72a5/businessmetata?isOverwrite=true'
> \
> -H 'cache-control: no-cache' \
> -H 'content-type: application/json' \
> -H 'postman-token: 6c5a0476-ac64-57c9-5845-cbea945fdf99' \
> -H 'x-xsrf-header: \"\"' \
> -d '{"String Length":{"Name":"abcdefghijklmnopqrstuvwxyz"}}' {code}
> Screenshots attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Review Request 73509: ATLAS-4392 The displayed names in a relationship should be sorted
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73509/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4392 https://issues.apache.org/jira/browse/ATLAS-4392 Repository: atlas Description --- When we have a 1:N relationship then we able to see the list of connected entities in the Atlas UI. These entities don't seem to be sorted in a meaningful way. The correct way would be if they are sorted in alphabetical order. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 794847555 Diff: https://reviews.apache.org/r/73509/diff/1/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/796/consoleFull Thanks, Mandar Ambawane
