Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/17/ Changes: https://reviews.apache.org/r/71791/diff/16-17/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/16/ Changes: https://reviews.apache.org/r/71791/diff/15-16/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/15/ Changes: https://reviews.apache.org/r/71791/diff/14-15/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/14/ Changes: https://reviews.apache.org/r/71791/diff/13-14/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Jan. 22, 2020, 10:09 p.m., Sarath Subramanian wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 4 (patched) > > <https://reviews.apache.org/r/71791/diff/13/?file=2183621#file2183621line4> > > > > if this enum is used only for aud consider renaming to > > atlas_audit_operation Initially it was atlas_audit_operation, but later changed to atlas_operation as suggested by Madhan - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review219365 --- On Dec. 11, 2019, 1:17 a.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Dec. 11, 2019, 1:17 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js a7de0d451 > dashboardv3/public/js/utils/Enums.js a7de0d451 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > c9ed79750 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > > repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java > 1863b8d74 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java > d0c06a107 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > c14850f43 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java > 78fdaca8d > > repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java > 42b63535b > > repository/src/test/java/org/apache/atlas/repositor
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Jan. 22, 2020, 10:09 p.m., Sarath Subramanian wrote: > > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > > Lines 495 (patched) > > <https://reviews.apache.org/r/71791/diff/13/?file=2183622#file2183622line496> > > > > why return type is ArrayNode.class? can we not have return type to be - > > List ? if ti works we don't need ExtractOperation() and > > ExtractResults() It didn't work for me. Please let me know if you have any idea which will return List directly that way we can really get read of the two functions > On Jan. 22, 2020, 10:09 p.m., Sarath Subramanian wrote: > > intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java > > Lines 36 (patched) > > <https://reviews.apache.org/r/71791/diff/13/?file=2183626#file2183626line36> > > > > AuditSearchParameters class looks like a trimmed down version of > > SearchParameters class. Consider using SearchParameters instead and removeA > > uditSearchParameters class. As we initially decided this way user will have less flexibility and will understand what is required to be passed for audit filter not the wider SearchParameters > On Jan. 22, 2020, 10:09 p.m., Sarath Subramanian wrote: > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > > Line 85 (original), 86 (patched) > > <https://reviews.apache.org/r/71791/diff/13/?file=2183644#file2183644line86> > > > > why ImportServiceTest now need to extend from AtlasAuditTestBase? > > AtlasAuditTestBase should be specific to only atlas sudit tests. Move > > common methods to a new base class - AtlasBaseTest. > > > > Please review all tests which is modified now to extend from > > 'AtlasAuditTestBase' Moved to AtlasTestBase and all the impexp test and audit test are now inherited from this. The only function assertExportImportAuditEntry is part of this which I might have moved to another inherited class. But we have plan to unify the imp/exp audit soon, will take care later - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review219365 --- On Dec. 11, 2019, 1:17 a.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Dec. 11, 2019, 1:17 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js a7de0d451 > dashboardv3/public/js/utils/Enums.js a7de0d451 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > c9ed79750 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 13, 2020, 8:06 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/7/ Changes: https://reviews.apache.org/r/71922/diff/6-7/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 13, 2020, 5:40 a.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/6/ Changes: https://reviews.apache.org/r/71922/diff/5-6/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 11, 2020, 1:50 a.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/5/ Changes: https://reviews.apache.org/r/71922/diff/4-5/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 10, 2020, 7:13 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/4/ Changes: https://reviews.apache.org/r/71922/diff/3-4/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
ud/3030-aws_s3_typedefs_v2.json > > Lines 238 (patched) > > <https://reviews.apache.org/r/71922/diff/3/?file=2199453#file2199453line238> > > > > This should be part of base class. same as above. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/#review219215 --- On Jan. 8, 2020, 11:12 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71922/ > --- > > (Updated Jan. 8, 2020, 11:12 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and > Sarath Subramanian. > > > Bugs: ATLAS-3564 > https://issues.apache.org/jira/browse/ATLAS-3564 > > > Repository: atlas > > > Description > --- > > This is the new version) of aws s3 model at atlas to allow the hierarchical > structure and attributes same as AWS S3 Console. The existing aws s3 models > has limitations like pseudo directory not containing another pseudo > directory, pseudo directory doesn't have attributes of aws object (as some > case it gets populated), both object and bucket doesn't containing all the > attributes of AWS S3 etc. > > > Diffs > - > > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71922/diff/3/ > > > Testing > --- > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 8, 2020, 11:12 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/3/ Changes: https://reviews.apache.org/r/71922/diff/2-3/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 8, 2020, 6:54 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description (updated) --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/1/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 8, 2020, 6:54 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/2/ Changes: https://reviews.apache.org/r/71922/diff/1-2/ Testing --- Thanks, Sidharth Mishra
Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, Object doesn't have version information, both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/1/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/13/ Changes: https://reviews.apache.org/r/71791/diff/12-13/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/12/ Changes: https://reviews.apache.org/r/71791/diff/11-12/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/11/ Changes: https://reviews.apache.org/r/71791/diff/10-11/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/10/ Changes: https://reviews.apache.org/r/71791/diff/9-10/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/8/ Changes: https://reviews.apache.org/r/71791/diff/7-8/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > > Lines 124 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182059#file2182059line124> > > > > define equals() and hashCode() method as well. AtlasAuditEntry is derived from AtlasBaseModelObject and the base class contains both of these method with guid check. Please let me know do we still need these two to be overridden? > On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > > Lines 125 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182059#file2182059line125> > > > > incorrect toString() implementation. please format in alignment with > > other model classes. AtlasAuditEntry is derived from AtlasBaseModelObject and the base class contains the toString() which calls this toString(sb). At the end it returns the correct alignment. > On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > > Lines 128 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182060#file2182060line139> > > > > => StringUtils.join(entityIds, ",")) Fixed. Could you please let me know the advantage of StringUtils.join over String.join > On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > > Lines 76 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182062#file2182062line76> > > > > move debug log to beginning of method. This debug msg is to put debug message when we successfully added the audit entry > On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > > Lines 437 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182093#file2182093line462> > > > > shouldn't purge events be audited only on successful purge on entities? > > consider checking EntityMutationResponse for any purge events present and > > then audit them. I think its better to audit admin action purge even if we don't purge any entities. Please let me know if you still think we should not audit the unsuccessful purge action by admin. - Sidharth ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218927 --- On Dec. 4, 2019, 9:57 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Dec. 4, 2019, 9:57 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java > eb8634533 > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION >
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Dec. 5, 2019, 12:39 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> > Lines 1057 (patched)
> > <https://reviews.apache.org/r/71791/diff/7/?file=2182064#file2182064line1095>
> >
> > This will cause 'DELETED' entities to be skipped even though purge is
> > in progress. This isn't correct. Please review and update.
> >
> > final String guid = AtlasGraphUtilsV2.getIdFromVertex(vertex);
> >
> > ret = guid == null;
> >
> > if (!ret) {
> > if (reqContext.isPurgeRequested()) {
> > ret = requestContext.isPurgedEntity(guid) || getState(vertex) ==
> > ACTIVE;
> > } else {
> > ret = requestContext.isDeletedEntity(guid) || getState(vertex) ==
> > DELETED;
> > }
> > }
As the code for purge, hard delete is same, the requestContext still stores the
purged entities as part of deleteEntity List.
> On Dec. 5, 2019, 12:39 a.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Lines 436 (patched)
> > <https://reviews.apache.org/r/71791/diff/7/?file=2182093#file2182093line461>
> >
> > Audit generation should be done inside entityStore.purgeByIds(), in the
> > same graph transaction. Avoid spilling such details into REST layer.
As I earlier replied to your previous comment, Moving auditing to inside
AtlasEntityStore.purgeByIds() is causing circular Injection issue.
AtlasAuditService depends on DataAccess and DataAccess depends on
AtlasEntityStore. So AtlasEntityStore can't depends on AtlasAuditService again.
KIndly suggest is there any other option to avoid this circular injection issue.
- Sidharth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218929
---
On Dec. 4, 2019, 9:57 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
>
> (Updated Dec. 4, 2019, 9:57 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Right now for purge, delete entity, etc. at Atlas we create audit entry at
> HBase. User can go to entity and click on audit to see the audit information
> for that particular entity. But if user purge one entity there will not be
> any vertex at Janus graph and user will not able to get the purge audit which
> is important for governance purpose.
>
> As part of fix, implemented Atlas Audit Framework which is being used for
> Purge operation and can be extended for other operations. In Future once we
> change the AtlasAudit type as super type for Import export as well we can use
> this framework for all Atlas operations.
>
> admin/audit rest has been added which will return all the Atlas audit
> operations including import export at Atlas.
>
>
> Diffs
> -
>
> addons/models/-Area0/0010-base_model.json 2f5fdaf14
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
> 8c0a6407c
> dashboardv2/public/js/utils/Enums.js 58542ec59
> dashboardv3/public/js/utils/Enums.js 58542ec59
> intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
> PRE-CREATION
>
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
> b448d510b
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java
> eb8634533
>
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> c9ed79750
> repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java
> be1698fb4
> repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java
> 02f78b369
>
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
> PRE-CREATION
>
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditS
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/7/ Changes: https://reviews.apache.org/r/71791/diff/6-7/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 5 (patched) > > <https://reviews.apache.org/r/71791/diff/5/?file=2181198#file2181198line5> > > > > "Audit Operation defines the operations we support for admin audit" => > > "Defines admin operations in Apache Atlas" Removed the admin and changed it to "Defines operations in Apache Atlas which can be audited" > On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote: > > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > > Lines 436 (patched) > > <https://reviews.apache.org/r/71791/diff/5/?file=2181235#file2181235line461> > > > > Instead of generating audit in a seperate graph transaction here, > > consider generating in the transaction used by > > AtlasEntityStore.purgeByIds() i.e. move auditing to inside > > AtlasEntityStore.purgeByIds(). Moving auditing to inside AtlasEntityStore.purgeByIds() is causing circular Injection issue. AtlasAuditService depends on DataAccess and DataAccess depends on AtlasEntityStore. So AtlasEntityStore can't depends on AtlasAuditService again. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218865 --- On Dec. 4, 2019, 9:47 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Dec. 4, 2019, 9:47 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java > eb8634533 > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > c9ed79750 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/6/ Changes: https://reviews.apache.org/r/71791/diff/5-6/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 4 (patched) > > <https://reviews.apache.org/r/71791/diff/5/?file=2181198#file2181198line4> > > > > atlas_operation => atlas_admin_operation I think as you initially suggested to rename it to atlas_operation is more flexible than atlas_admin_operation as in future we may allow some more operation to be audited like other than admin operations. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218865 --- On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Nov. 30, 2019, 7:44 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > > repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java > 1863b8d74 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java > d0c06a107 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > c14850f43 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java > 78fdaca8d > > repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java > 42b63535b > > repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java > a1b512fb
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Nov. 27, 2019, 12:39 a.m., Madhan Neethiraj wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 442 (patched) > > <https://reviews.apache.org/r/71791/diff/4/?file=2179150#file2179150line442> > > > > Consider renaming 'clientHost' to 'clientId' - given this field is used > > to store AtlasServer name for import/export audit entries. > > > > This rename should be applied to AtlasAuditEntry.clientHost as well. I changed it to clientId as suggested, But please note this field contains Client IP Address. In future we may put host fqdn if available. In both the cases do you think client Id makes more sense? > On Nov. 27, 2019, 12:39 a.m., Madhan Neethiraj wrote: > > pom.xml > > Line 695 (original), 695 (patched) > > <https://reviews.apache.org/r/71791/diff/4/?file=2179156#file2179156line695> > > > > I suggest a separate patch for library version changes. This fix was to fix the precommit build and now the precommit is successful - https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1559/. I am having separate patch for the lib version change and reverted the changes - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218808 ------- On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Nov. 30, 2019, 7:44 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > >
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
- Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218808 --- On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Nov. 30, 2019, 7:44 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > > repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java > 1863b8d74 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java > d0c06a107 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > c14850f43 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java > 78fdaca8d > > repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java > 42b63535b > > repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java > a1b512fb9 > > repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java > 7a1ed18cb > > repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java > 48b958271 > > repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java > c9bb11c56 > > repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java > 0ffc3d595 > > r
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/5/ Changes: https://reviews.apache.org/r/71791/diff/4-5/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Nov. 25, 2019, 5:05 a.m., Madhan Neethiraj wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 397 (patched) > > <https://reviews.apache.org/r/71791/diff/3/?file=2174685#file2174685line397> > > > > - I suggest adding attribute 'serverName', that will be set to > > AtlasServer.name - similar to > > __ExportImportAuditEntry.sourceServerName/__ExportImportAuditEntry.targetServerName > > - consider adding an unique attribute, like __guid As we have __guid as system attribute and gets added to entity, not adding __guid as part of model. Do you think we need serverName as it will be inserted as current host all the time and it may not be usefull. For import export its needed and it may differ so we addded that. As part of migration I was planning to change the current import export audit model where it will be derived from this audit model type and add the source and destination server only for import export audit type. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218785 ------- On Nov. 26, 2019, 9:11 a.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Nov. 26, 2019, 9:11 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > pom.xml b2506e70e > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > > repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java > 1863b8d74 > > repository/src/test/java/org/apache/atlas/repo
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/4/ Changes: https://reviews.apache.org/r/71791/diff/3-4/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/3/ Changes: https://reviews.apache.org/r/71791/diff/2-3/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/2/ Changes: https://reviews.apache.org/r/71791/diff/1-2/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/1/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 13, 2019, 8:37 p.m., Sarath Subramanian wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > > Lines 484 (patched) > > <https://reviews.apache.org/r/71737/diff/4/?file=2173006#file2173006line485> > > > > should this be AtlasPrivilege.ADMIN_PURGE ? > > Sidharth Mishra wrote: > As we don't have the Ranger changes for time being its ADMIN_IMPORT. > Later ll change it to ADMIN_PURGE To track this created a JIRA Item - ATLAS-3524 - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71737/#review218622 --- On Nov. 14, 2019, 7:10 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71737/ > --- > > (Updated Nov. 14, 2019, 7:10 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3477 > https://issues.apache.org/jira/browse/ATLAS-3477 > > > Repository: atlas > > > Description > --- > > As part of this solution users can purge the entities which are already soft > deleted. This will be useful as someone may be interested to purge some list > of soft deleted entities from Atlas. Due to governance compliances, only > admin can perform purge of entities. > > Purge will also create separate Audit entry at HBase similar to delete entity > operation. > > > Diffs > - > > authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java > 59c596d66 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 199d6bf58 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > 444167eae > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > e9cc7cd3d > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > 7ace00dff > intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java > daf4ca88c > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 43a9b842c > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > b94590bf1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > d2544df4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 3389d2465 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17a96493f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 4d57d8bc8 > server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > 48f0cd3c2 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 464d46f1e > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > 5d506bbc2 > > > Diff: https://reviews.apache.org/r/71737/diff/6/ > > > Testing > --- > > IT has been added. > Validated manually by running below mentioned queries and REST: > > > a) create table thursday_07_11_1 (id int); >create table thursday_07_11_1_ctas as select * from thursday_07_11_1; > >create table thursday_07_11_2 (id int); >create table thursday_07_11_2_ctas as select * from thursday_07_11_2; > b) The above will create two lineages. Go to Atlas UI and get the vertices > for atleast any two. > > > c) Using rest delete those two entities or use below query: > drop table thursday_07_11_1; > drop table thursday_07_11_1_ctas; > > drop table thursday_07_11_2; > drop table thursday_07_11_2_ctas; > > REST delete example: > DELETE > http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c > > d) Now use the same guids and call purge REST: > DELETE > http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05=98bee463-8810-402a-bcdc-3eb00bf50365 > > Check the response which will return all the successfully purged entities > > e) For each guid check if the audit has separate new entry for purge. REST > used: > GET > http://localhost:31000/api/atlas/v2/entity/7b6806
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
plicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 14, 2019, 12:02 a.m., Ashutosh Mestry wrote: > > My thinking is that there should be a bulk API that should take JSON > > payload of entities to be purged. Generally, there are large number of > > deleted entities. There should be a convinent way to purge them. Right now > > the queryString has a smaller limit which is quickly exhausted due to > > length of GUIDs. Right now in the current code I have already Addressed this and changed it to body from queryString. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71737/#review218627 --- On Nov. 13, 2019, 9:04 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71737/ > --- > > (Updated Nov. 13, 2019, 9:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3477 > https://issues.apache.org/jira/browse/ATLAS-3477 > > > Repository: atlas > > > Description > --- > > As part of this solution users can purge the entities which are already soft > deleted. This will be useful as someone may be interested to purge some list > of soft deleted entities from Atlas. Due to governance compliances, only > admin can perform purge of entities. > > Purge will also create separate Audit entry at HBase similar to delete entity > operation. > > > Diffs > - > > authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java > 59c596d66 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 199d6bf58 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > 444167eae > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > e9cc7cd3d > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > 7ace00dff > intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java > daf4ca88c > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 43a9b842c > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > b94590bf1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > d2544df4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 3389d2465 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17a96493f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 4d57d8bc8 > server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > 48f0cd3c2 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 464d46f1e > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > 5d506bbc2 > > > Diff: https://reviews.apache.org/r/71737/diff/5/ > > > Testing > --- > > IT has been added. > Validated manually by running below mentioned queries and REST: > > > a) create table thursday_07_11_1 (id int); >create table thursday_07_11_1_ctas as select * from thursday_07_11_1; > >create table thursday_07_11_2 (id int); >create table thursday_07_11_2_ctas as select * from thursday_07_11_2; > b) The above will create two lineages. Go to Atlas UI and get the vertices > for atleast any two. > > > c) Using rest delete those two entities or use below query: > drop table thursday_07_11_1; > drop table thursday_07_11_1_ctas; > > drop table thursday_07_11_2; > drop table thursday_07_11_2_ctas; > > REST delete example: > DELETE > http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c > > d) Now use the same guids and call purge REST: > DELETE > http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05=98bee463-8810-402a-bcdc-3eb00bf50365 > > Check the response which will return all the successfully purged entities > > e) For each guid check if the audit has separate new entry for purge. REST > used: > GET > http://localhost:31000/api/atlas/v2/entity/7b6806c1-8daf-4e0f-8d10-672a192968f1/audit > > Example Response: &
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 14, 2019, midnight, Ashutosh Mestry wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > > Lines 159 (patched) > > <https://reviews.apache.org/r/71737/diff/5/?file=2173080#file2173080line159> > > > > Does this mean that if purge is requested, then even active edges are > > requested whereas if not deleted edges are requested. > > > > Shouldn't it be that when purge is requested, both active and deleted > > edges are requested? Right now purge will only be allowed if the entity is already soft deleted. This check skip the edges or vertices for the same reason. > On Nov. 14, 2019, midnight, Ashutosh Mestry wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > > Lines 200 (patched) > > <https://reviews.apache.org/r/71737/diff/5/?file=2173080#file2173080line200> > > > > Same comment as above. Right now purge will only be allowed if the entity is already soft deleted. This check skip the edges or vertices for the same reason. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71737/#review218626 --- On Nov. 13, 2019, 9:04 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71737/ > --- > > (Updated Nov. 13, 2019, 9:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3477 > https://issues.apache.org/jira/browse/ATLAS-3477 > > > Repository: atlas > > > Description > --- > > As part of this solution users can purge the entities which are already soft > deleted. This will be useful as someone may be interested to purge some list > of soft deleted entities from Atlas. Due to governance compliances, only > admin can perform purge of entities. > > Purge will also create separate Audit entry at HBase similar to delete entity > operation. > > > Diffs > - > > authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java > 59c596d66 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 199d6bf58 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > 444167eae > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > e9cc7cd3d > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > 7ace00dff > intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java > daf4ca88c > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 43a9b842c > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > b94590bf1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > d2544df4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 3389d2465 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17a96493f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 4d57d8bc8 > server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > 48f0cd3c2 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 464d46f1e > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > 5d506bbc2 > > > Diff: https://reviews.apache.org/r/71737/diff/5/ > > > Testing > --- > > IT has been added. > Validated manually by running below mentioned queries and REST: > > > a) create table thursday_07_11_1 (id int); >create table thursday_07_11_1_ctas as select * from thursday_07_11_1; > >create table thursday_07_11_2 (id int); >create table thursday_07_11_2_ctas as select * from thursday_07_11_2; > b) The above will create two lineages. Go to Atlas UI and get the vertices > for atleast any two. > > > c) Using rest delete those two entities or use below query: > drop table thursday_07_11_1; > drop table thursday_07_11_1_ctas; > > drop table thursday_07_11_2; > drop table thursday_07_1
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 13, 2019, 8:37 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
> > Lines 484 (patched)
> > <https://reviews.apache.org/r/71737/diff/4/?file=2173006#file2173006line485>
> >
> > should this be AtlasPrivilege.ADMIN_PURGE ?
As we don't have the Ranger changes for time being its ADMIN_IMPORT. Later ll
change it to ADMIN_PURGE
- Sidharth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71737/#review218622
---
On Nov. 13, 2019, 6:04 a.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71737/
> ---
>
> (Updated Nov. 13, 2019, 6:04 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-3477
> https://issues.apache.org/jira/browse/ATLAS-3477
>
>
> Repository: atlas
>
>
> Description
> ---
>
> As part of this solution users can purge the entities which are already soft
> deleted. This will be useful as someone may be interested to purge some list
> of soft deleted entities from Atlas. Due to governance compliances, only
> admin can perform purge of entities.
>
> Purge will also create separate Audit entry at HBase similar to delete entity
> operation.
>
>
> Diffs
> -
>
> authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java
> 59c596d66
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
> 199d6bf58
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java
> 444167eae
> intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
> e9cc7cd3d
>
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
> 7ace00dff
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java
> daf4ca88c
>
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
> 43a9b842c
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
> b94590bf1
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> d2544df4f
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
> 3389d2465
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
> 17a96493f
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
> 4d57d8bc8
> server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c
>
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
> 48f0cd3c2
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> 464d46f1e
>
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
> 5d506bbc2
>
>
> Diff: https://reviews.apache.org/r/71737/diff/4/
>
>
> Testing
> ---
>
> IT has been added.
> Validated manually by running below mentioned queries and REST:
>
>
> a) create table thursday_07_11_1 (id int);
>create table thursday_07_11_1_ctas as select * from thursday_07_11_1;
>
>create table thursday_07_11_2 (id int);
>create table thursday_07_11_2_ctas as select * from thursday_07_11_2;
> b) The above will create two lineages. Go to Atlas UI and get the vertices
> for atleast any two.
>
>
> c) Using rest delete those two entities or use below query:
> drop table thursday_07_11_1;
> drop table thursday_07_11_1_ctas;
>
> drop table thursday_07_11_2;
> drop table thursday_07_11_2_ctas;
>
> REST delete example:
> DELETE
> http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c
>
> d) Now use the same guids and call purge REST:
> DELETE
> http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05=98bee463-8810-402a-bcdc-3eb00bf50365
>
> Check the response which will return all the successfully purged entities
>
> e) For each guid check if the audit has separate new entry for purge. REST
> used:
> GET
> http://localhost:31000/api/atlas/v2/entity/7b6806c1-8daf-4e0f-8d10-672a192968f1/audit
>
> Example Response:
>
> [
> {
> "ent
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
plicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
edTo\":null,\"replicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
ot;:\"admin\",\"replicatedTo\":null,\"replicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 9, 2019, 7:03 a.m., Madhan Neethiraj wrote: > > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > > Lines 442 (patched) > > <https://reviews.apache.org/r/71737/diff/1/?file=2171841#file2171841line442> > > > > Consider adding another method in a subsequent patch, that take > > List as parameter. Such method can be useful in purging > > entities using typeName/uniqueAttributes. Created ATLAS-3522 to track this. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71737/#review218585 --- On Nov. 7, 2019, 11:54 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71737/ > --- > > (Updated Nov. 7, 2019, 11:54 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3477 > https://issues.apache.org/jira/browse/ATLAS-3477 > > > Repository: atlas > > > Description > --- > > As part of this solution users can purge the entities which are already soft > deleted. This will be useful as someone may be interested to purge some list > of soft deleted entities from Atlas. Due to governance compliances, only > admin can perform purge of entities. > > Purge will also create separate Audit entry at HBase similar to delete entity > operation. > > > Diffs > - > > authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java > 59c596d66 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 199d6bf58 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > 444167eae > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > e9cc7cd3d > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > 7ace00dff > intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java > daf4ca88c > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 43a9b842c > > repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java > 01e339d23 > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > b94590bf1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > d2544df4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 3389d2465 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17a96493f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 4d57d8bc8 > server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > 48f0cd3c2 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 464d46f1e > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > 5d506bbc2 > > > Diff: https://reviews.apache.org/r/71737/diff/1/ > > > Testing > --- > > IT has been added. > Validated manually by running below mentioned queries and REST: > > > a) create table thursday_07_11_1 (id int); >create table thursday_07_11_1_ctas as select * from thursday_07_11_1; > >create table thursday_07_11_2 (id int); >create table thursday_07_11_2_ctas as select * from thursday_07_11_2; > b) The above will create two lineages. Go to Atlas UI and get the vertices > for atleast any two. > > > c) Using rest delete those two entities or use below query: > drop table thursday_07_11_1; > drop table thursday_07_11_1_ctas; > > drop table thursday_07_11_2; > drop table thursday_07_11_2_ctas; > > REST delete example: > DELETE > http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c > > d) Now use the same guids and call purge REST: > DELETE > http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05=98bee463-8810-402a-bcdc-3eb00bf50365 > > Check the response which will return all the successfully purged entities > > e) For each guid check if the audit has separate new entry for purge. REST > used: >
Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
edTo\":null,\"replicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
Re: Review Request 71620: ATLAS-3461: Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71620/ --- (Updated Oct. 21, 2019, 10:45 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3461 https://issues.apache.org/jira/browse/ATLAS-3461 Repository: atlas Description --- Now even if the Impala queries contains comment/ extra spaces, Atlas will create the impala process approproately. Even "create external table" will create Impala process at Atlas and user can see lineage. Diffs (updated) - addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java b9dd894f8 addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java dc4d1333e addons/impala-bridge/src/test/resources/impalaAlterViewAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateTableAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateViewWithCommentSpaces.json PRE-CREATION Diff: https://reviews.apache.org/r/71620/diff/4/ Changes: https://reviews.apache.org/r/71620/diff/3-4/ Testing --- 1. Unit Tests added to ImpalaLineageToolIT and without fix it was failing. Now after the fix it works. 2. Validated manually by running below mentioned query from Impala:JDBC, Hue and Impala Shell a) create table testMon4 as select * from t1; b) creAte external table /*test*/ testMo3 aS -- temp fff /* temp */ -- sdfrnfkjef select select * from t1; c) Create /*dd*/ table testMon4 aS select * from t1; d) alter view testMonView2 as select * from t1; e) create view testMonView4 as select * from t1; f) alter view testMonView4 as select * from t1; g) create EXTERNAL TABLE IF NOT EXISTS thusday_ext1 (STUD_ID int, DEPT_ID int, NAME string ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/warehouse/tablespace/external/hive/db1.db/thusday_ext1'; Thanks, Sidharth Mishra
Re: Review Request 71620: ATLAS-3461: Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
> On Oct. 21, 2019, 10:02 p.m., Le Ma wrote: > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java > > Line 33 (original), 33 (patched) > > <https://reviews.apache.org/r/71620/diff/1-2/?file=2169132#file2169132line33> > > > > Hey Sid, thanks for the patch. I am not sure about the regex there "^[ > > ]*", does this mean the beginning of the line can have multiple white > > spaces? I saw you added this square brackets at diff2. '\bcreate\b' means > > exact match word "create", then if there are spaces before "create", will > > it still match? > > > > thanks, > > Le Hi Le, ^ is for line start and [ ]* is for zero or more white spaces at beginning. Yes \b\b will ensure the exact word match and even if it has multiple spaces it ll be taken care by [ ]* at beginning. Thanks, Sid - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71620/#review218312 --- On Oct. 21, 2019, 9:01 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71620/ > --- > > (Updated Oct. 21, 2019, 9:01 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3461 > https://issues.apache.org/jira/browse/ATLAS-3461 > > > Repository: atlas > > > Description > --- > > Now even if the Impala queries contains comment/ extra spaces, Atlas will > create the impala process approproately. > Even "create external table" will create Impala process at Atlas and user can > see lineage. > > > Diffs > - > > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java > b9dd894f8 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > dc4d1333e > > addons/impala-bridge/src/test/resources/impalaAlterViewAsSelectWithCommentSpaces.json > PRE-CREATION > > addons/impala-bridge/src/test/resources/impalaCreateTableAsSelectWithCommentSpaces.json > PRE-CREATION > > addons/impala-bridge/src/test/resources/impalaCreateViewWithCommentSpaces.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/71620/diff/3/ > > > Testing > --- > > 1. Unit Tests added to ImpalaLineageToolIT and without fix it was failing. > Now after the fix it works. > 2. Validated manually by running below mentioned query from Impala:JDBC, Hue > and Impala Shell > > a) create table testMon4 as select * from t1; > > b) creAte external table /*test*/ testMo3 aS -- temp fff > /* temp */ -- sdfrnfkjef select > select * from t1; > > > c) Create /*dd*/ table testMon4 aS select * from t1; > > d) alter view testMonView2 as select * from t1; > > e) create view testMonView4 as select * from t1; > > f) alter view testMonView4 as select * from t1; > > g) create EXTERNAL TABLE IF NOT EXISTS thusday_ext1 >(STUD_ID int, >DEPT_ID int, >NAME string >) >ROW FORMAT DELIMITED >FIELDS TERMINATED BY ',' >STORED AS TEXTFILE >LOCATION '/warehouse/tablespace/external/hive/db1.db/thusday_ext1'; > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 71620: ATLAS-3461: Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71620/ --- (Updated Oct. 21, 2019, 9:01 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3461 https://issues.apache.org/jira/browse/ATLAS-3461 Repository: atlas Description --- Now even if the Impala queries contains comment/ extra spaces, Atlas will create the impala process approproately. Even "create external table" will create Impala process at Atlas and user can see lineage. Diffs (updated) - addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java b9dd894f8 addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java dc4d1333e addons/impala-bridge/src/test/resources/impalaAlterViewAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateTableAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateViewWithCommentSpaces.json PRE-CREATION Diff: https://reviews.apache.org/r/71620/diff/3/ Changes: https://reviews.apache.org/r/71620/diff/2-3/ Testing --- 1. Unit Tests added to ImpalaLineageToolIT and without fix it was failing. Now after the fix it works. 2. Validated manually by running below mentioned query from Impala:JDBC, Hue and Impala Shell a) create table testMon4 as select * from t1; b) creAte external table /*test*/ testMo3 aS -- temp fff /* temp */ -- sdfrnfkjef select select * from t1; c) Create /*dd*/ table testMon4 aS select * from t1; d) alter view testMonView2 as select * from t1; e) create view testMonView4 as select * from t1; f) alter view testMonView4 as select * from t1; g) create EXTERNAL TABLE IF NOT EXISTS thusday_ext1 (STUD_ID int, DEPT_ID int, NAME string ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/warehouse/tablespace/external/hive/db1.db/thusday_ext1'; Thanks, Sidharth Mishra
Re: Review Request 71620: ATLAS-3461: Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71620/ --- (Updated Oct. 21, 2019, 6:24 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3461 https://issues.apache.org/jira/browse/ATLAS-3461 Repository: atlas Description --- Now even if the Impala queries contains comment/ extra spaces, Atlas will create the impala process approproately. Even "create external table" will create Impala process at Atlas and user can see lineage. Diffs (updated) - addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java b9dd894f8 addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java dc4d1333e addons/impala-bridge/src/test/resources/WALimpala.wal PRE-CREATION addons/impala-bridge/src/test/resources/impalaAlterViewAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateTableAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateViewWithCommentSpaces.json PRE-CREATION Diff: https://reviews.apache.org/r/71620/diff/2/ Changes: https://reviews.apache.org/r/71620/diff/1-2/ Testing --- 1. Unit Tests added to ImpalaLineageToolIT and without fix it was failing. Now after the fix it works. 2. Validated manually by running below mentioned query from Impala:JDBC, Hue and Impala Shell a) create table testMon4 as select * from t1; b) creAte external table /*test*/ testMo3 aS -- temp fff /* temp */ -- sdfrnfkjef select select * from t1; c) Create /*dd*/ table testMon4 aS select * from t1; d) alter view testMonView2 as select * from t1; e) create view testMonView4 as select * from t1; f) alter view testMonView4 as select * from t1; g) create EXTERNAL TABLE IF NOT EXISTS thusday_ext1 (STUD_ID int, DEPT_ID int, NAME string ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/warehouse/tablespace/external/hive/db1.db/thusday_ext1'; Thanks, Sidharth Mishra
Re: Review Request 71620: ATLAS-3461: Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71620/ --- (Updated Oct. 21, 2019, 6:22 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3461 https://issues.apache.org/jira/browse/ATLAS-3461 Repository: atlas Description --- Now even if the Impala queries contains comment/ extra spaces, Atlas will create the impala process approproately. Even "create external table" will create Impala process at Atlas and user can see lineage. Diffs - addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java b9dd894f8 addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java dc4d1333e addons/impala-bridge/src/test/resources/WALimpala.wal PRE-CREATION addons/impala-bridge/src/test/resources/impalaAlterViewAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateTableAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateViewWithCommentSpaces.json PRE-CREATION Diff: https://reviews.apache.org/r/71620/diff/1/ Testing (updated) --- 1. Unit Tests added to ImpalaLineageToolIT and without fix it was failing. Now after the fix it works. 2. Validated manually by running below mentioned query from Impala:JDBC, Hue and Impala Shell a) create table testMon4 as select * from t1; b) creAte external table /*test*/ testMo3 aS -- temp fff /* temp */ -- sdfrnfkjef select select * from t1; c) Create /*dd*/ table testMon4 aS select * from t1; d) alter view testMonView2 as select * from t1; e) create view testMonView4 as select * from t1; f) alter view testMonView4 as select * from t1; g) create EXTERNAL TABLE IF NOT EXISTS thusday_ext1 (STUD_ID int, DEPT_ID int, NAME string ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/warehouse/tablespace/external/hive/db1.db/thusday_ext1'; Thanks, Sidharth Mishra
Review Request 71620: ATLAS-3461: Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71620/ --- Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3461 https://issues.apache.org/jira/browse/ATLAS-3461 Repository: atlas Description --- Now even if the Impala queries contains comment/ extra spaces, Atlas will create the impala process approproately. Even "create external table" will create Impala process at Atlas and user can see lineage. Diffs - addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java b9dd894f8 addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java dc4d1333e addons/impala-bridge/src/test/resources/WALimpala.wal PRE-CREATION addons/impala-bridge/src/test/resources/impalaAlterViewAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateTableAsSelectWithCommentSpaces.json PRE-CREATION addons/impala-bridge/src/test/resources/impalaCreateViewWithCommentSpaces.json PRE-CREATION Diff: https://reviews.apache.org/r/71620/diff/1/ Testing --- 1. Unit Tests added to ImpalaLineageToolIT and without fix it was failing. Now after the fix it works. 2. Validated manually by running below mentioned query from Impala:JDBC: create table testMon4 as select * from t1; creAte external table /*test*/ testMo3 aS -- temp fff /* temp */ -- sdfrnfkjef select select * from t1; Create /*dd*/ table testMon4 aS select * from t1; creAte external table /*test*/ testMon10 aS -- temp fff /* temp */ -- sdfrnfkjef select select * from t1; alter view testMonView2 as select * from t1; create view testMonView4 as select * from t1; alter view testMonView4 as select * from t1; CREATE EXTERNAL TABLE IF NOT EXISTS mondaysecond1 LIKE t1 LOCATION '/warehouse/tablespace/external/hive/db1.db/mondaysecond1'; Thanks, Sidharth Mishra
Re: Request for adding me to contributor list
Thanks for adding me. Regards, Sid On Tue, Sep 24, 2019, 9:03 PM Madhan Neethiraj wrote: > Sid, > > Thanks for your interest in contributing to Apache Atlas. You have been > added as a contributor. Welcome to Apache Atlas community! > > Regards, > Madhan > > > > > On 9/24/19, 8:52 PM, "sidharth mishra" wrote: > > Hi All, > > I would like to contribute to Apache Atlas. Kindly add me to the > contributor list. > My git id is: sidharthkmishra > Email ID: sidharthkmis...@gmail.com > > Kindly let me know if you have any questions or concerns. > > Regards, > Sid > > > >
Request for adding me to contributor list
Hi All, I would like to contribute to Apache Atlas. Kindly add me to the contributor list. My git id is: sidharthkmishra Email ID: sidharthkmis...@gmail.com Kindly let me know if you have any questions or concerns. Regards, Sid
