Re: Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73140/ --- (Updated Jan. 13, 2021, 12:44 a.m.) Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, Radhika Kundam, and Sarath Subramanian. Bugs: ATLAS-4099 https://issues.apache.org/jira/browse/ATLAS-4099 Repository: atlas Description --- ATLAS-4099: Removed slash from adls_gen2_directory Diffs (updated) - common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 0e4b507ff common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 2d2ff2c47 Diff: https://reviews.apache.org/r/73140/diff/2/ Changes: https://reviews.apache.org/r/73140/diff/1-2/ Testing --- Already tested manually and precommit will be run Thanks, Sidharth Mishra
Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73140/ --- Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, Radhika Kundam, and Sarath Subramanian. Bugs: ATLAS-4099 https://issues.apache.org/jira/browse/ATLAS-4099 Repository: atlas Description --- ATLAS-4099: Removed slash from adls_gen2_directory Diffs - common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 0e4b507ff Diff: https://reviews.apache.org/r/73140/diff/1/ Testing --- Already tested manually and precommit will be run Thanks, Sidharth Mishra
Review Request 73093: ATLAS-4083: Change the existing ADLS Gen2 type with new attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73093/ --- Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, Radhika Kundam, and Sarath Subramanian. Bugs: ATLAS-4083 https://issues.apache.org/jira/browse/ATLAS-4083 Repository: atlas Description --- ATLAS-4083: Change the existing ADLS Gen2 type with new attributes Diffs - addons/models/3000-Cloud/3040-azure_adls_typedefs.json 2d44f4e07 Diff: https://reviews.apache.org/r/73093/diff/1/ Testing --- Thanks, Sidharth Mishra
Review Request 73091: ATLAS-4082: Add more attributes like create or last modify time, contentLength etc. to ADLS Gen2 types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73091/ --- Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, Radhika Kundam, and Sarath Subramanian. Bugs: ATLAS-4082 https://issues.apache.org/jira/browse/ATLAS-4082 Repository: atlas Description --- ATLAS-4082: Add more attributes like create or last modify time, contentLength etc. to ADLS Gen2 types Diffs - addons/models/3000-Cloud/patches/005-azure_adls_add_attributes.json PRE-CREATION Diff: https://reviews.apache.org/r/73091/diff/1/ Testing --- Applied the patch to my cluster successfully Thanks, Sidharth Mishra
Re: Review Request 73089: ATLAS-408: Atlas incorrectly reports successful change to glossary term
> On Dec. 15, 2020, 6:54 p.m., Deep Singh wrote:
> > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
> > Line 416 (original), 415 (patched)
> > <https://reviews.apache.org/r/73089/diff/1/?file=2243547#file2243547line416>
> >
> > What Exception is thrown here? Does it throw unauthorized error. And in
> > such cases, does the API sends 401 response ?
{"errorCode":"ATLAS-403-00-001","errorMessage":"sid is not authorized to
perform update entity: type=AtlasGlossaryTerm"} is been thrown and Status Code:
403 Forbidden
- Sidharth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73089/#review222345
-------
On Dec. 15, 2020, 5:45 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73089/
> ---
>
> (Updated Dec. 15, 2020, 5:45 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj,
> Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4081
> https://issues.apache.org/jira/browse/ATLAS-4081
>
>
> Repository: atlas
>
>
> Description
> ---
>
> ATLAS-4081: Atlas incorrectly reports successful change to glossary term
>
>
> Diffs
> -
>
> repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
> d630f669e
>
>
> Diff: https://reviews.apache.org/r/73089/diff/1/
>
>
> Testing
> ---
>
> Created one more user - sid which is not an admin user and reproduced the
> issue. Then patched the cluster with the fix and confirmed it works. Now the
> behaviour is same as GlossaryService.
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Review Request 73089: ATLAS-408: Atlas incorrectly reports successful change to glossary term
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73089/ --- Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, Radhika Kundam, and Sarath Subramanian. Bugs: ATLAS-4081 https://issues.apache.org/jira/browse/ATLAS-4081 Repository: atlas Description --- ATLAS-4081: Atlas incorrectly reports successful change to glossary term Diffs - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java d630f669e Diff: https://reviews.apache.org/r/73089/diff/1/ Testing --- Created one more user - sid which is not an admin user and reproduced the issue. Then patched the cluster with the fix and confirmed it works. Now the behaviour is same as GlossaryService. Thanks, Sidharth Mishra
Re: Review Request 72730: ATLAS-3875: Introduce AtlasClient in Python
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72730/#review221546 --- atlas-examples/sample-app/src/main/python/discovery_example.py Lines 3 (patched) <https://reviews.apache.org/r/72730/#comment310609> Remove two extra blank lines. Along with this please add content to README.md file as well atlas-examples/sample-app/src/main/python/discovery_example.py Lines 12 (patched) <https://reviews.apache.org/r/72730/#comment310601> should we rename it to something more meaningful? atlas-examples/sample-app/src/main/python/utils.py Lines 11 (patched) <https://reviews.apache.org/r/72730/#comment310600> I think it will be good to rename these const to have some prefix like CUSTOM or SAMPLE to differentiate between the existing table types. client/client-v2/src/main/python/atlas_client/base_client.py Lines 84 (patched) <https://reviews.apache.org/r/72730/#comment310602> Should we have custom exceptions like AtlasException instead of Exception thrown from most of the python client similar to Java client. This will help the users of the client to handle appropriately and distinguish from the generic python exceptions client/client-v2/src/main/python/atlas_client/discovery_client.py Lines 6 (patched) <https://reviews.apache.org/r/72730/#comment310608> Remove one extra blank line client/client-v2/src/main/python/atlas_client/discovery_client.py Lines 20 (patched) <https://reviews.apache.org/r/72730/#comment310603> Can we have vertical indentation like: DSL_SEARCH = API(DSL_SEARCH_URI, HttpMethod.GET, HTTPStatus.OK) FULL_TEXT_SEARCH = API(FULL_TEXT_SEARCH_URI, HttpMethod.GET, HTTPStatus.OK) BASIC_SEARCH = API(BASIC_SEARCH_URI, HttpMethod.GET, HTTPStatus.OK) FACETED_SEARCH = API(FACETED_SEARCH_URI, HttpMethod.POST, HTTPStatus.OK) ATTRIBUTE_SEARCH = API(DISCOVERY_URI + "/attribute", HttpMethod.GET, HTTPStatus.OK) In case you are changing please take care of this at other places client/client-v2/src/main/python/atlas_client/discovery_model.py Lines 18 (patched) <https://reviews.apache.org/r/72730/#comment310606> Add blank line after the header comment. Please do at other files as well client/client-v2/src/main/python/atlas_client/discovery_model.py Lines 22 (patched) <https://reviews.apache.org/r/72730/#comment310604> Please remove one extra blank line (we always keep one blank line as separator if needed) client/client-v2/src/main/python/atlas_client/discovery_model.py Lines 145 (patched) <https://reviews.apache.org/r/72730/#comment310607> Please remove if not needed - Sidharth Mishra On Aug. 10, 2020, 5:26 p.m., Jyoti Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72730/ > --- > > (Updated Aug. 10, 2020, 5:26 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath > Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3875 > https://issues.apache.org/jira/browse/ATLAS-3875 > > > Repository: atlas > > > Description > --- > > - Added basic structure for Python client > - This includes all classes for Entity and TypeDef > - Added all endpoints for typedef > > > Diffs > - > > atlas-examples/sample-app/src/main/python/discovery_example.py PRE-CREATION > atlas-examples/sample-app/src/main/python/entity_example.py PRE-CREATION > atlas-examples/sample-app/src/main/python/glossary_example.py PRE-CREATION > atlas-examples/sample-app/src/main/python/lineage_example.py PRE-CREATION > > atlas-examples/sample-app/src/main/python/request_json/entity_create_db.json > PRE-CREATION > > atlas-examples/sample-app/src/main/python/request_json/entity_create_process.json > PRE-CREATION > > atlas-examples/sample-app/src/main/python/request_json/entity_create_table_canada.json > PRE-CREATION > > atlas-examples/sample-app/src/main/python/request_json/entity_create_table_us.json > PRE-CREATION > atlas-examples/sample-app/src/main/python/request_json/typedef_create.json > PRE-CREATION > atlas-examples/sample-app/src/main/python/sample_client.py PRE-CREATION > atlas-examples/sample-app/src/main/python/typedef_example.py PRE-CREATION > atlas-examples/sample-app/src/main/python/utils.py PRE-CREATION > client/client-v2/src/main/python/LICENSE PRE-CREATION > client/client-v2/src/main/python/README.md PRE-CREATION > client/client-v2/src/main/pyt
Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72698/#review221369 --- Ship it! Ship It! - Sidharth Mishra On July 23, 2020, 6:10 p.m., Jyoti Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72698/ > --- > > (Updated July 23, 2020, 6:10 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath > Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3875 > https://issues.apache.org/jira/browse/ATLAS-3875 > > > Repository: atlas > > > Description > --- > > Using this project users can get an idea as how to integrate with Atlas using > AtlasCleint. This helps the user to understand the basic rest functionality > of Atlas such as > > - EntityRest > - TypeDefRest > - DiscoveryRest > - LineageRest > - GlossaryRest > > > Diffs > - > > atlas-examples/pom.xml PRE-CREATION > atlas-examples/sample-app/README.md PRE-CREATION > atlas-examples/sample-app/pom.xml PRE-CREATION > > atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/DiscoveryExample.java > PRE-CREATION > > atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/EntityExample.java > PRE-CREATION > > atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/GlossaryExample.java > PRE-CREATION > > atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/LineageExample.java > PRE-CREATION > > atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleApp.java > PRE-CREATION > > atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleAppConstants.java > PRE-CREATION > > atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java > PRE-CREATION > atlas-examples/sample-app/src/main/resources/atlas-application.properties > PRE-CREATION > pom.xml 5e0442ae5 > > > Diff: https://reviews.apache.org/r/72698/diff/3/ > > > Testing > --- > > > Thanks, > > Jyoti Singh > >
Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72698/#review221321
---
atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 63 (patched)
<https://reviews.apache.org/r/72698/#comment310200>
char[] is always preferred over String - check
https://stackoverflow.com/questions/8881291/why-is-char-preferred-over-string-for-passwords#:~:text=Since%20Strings%20are%20immutable%20there,2.
Please check if we can add a new contructor at client v2 and do this ow
lets file a jira and track this for client v2
atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 75 (patched)
<https://reviews.apache.org/r/72698/#comment310202>
Please check if we should change the code snippet from 72:84 to something
like:
public static void main(String[] args) throws Exception {
String[] basicAuthUsernamePassword = null;
String[] urls = null;
urls = getServerUrl();
if (!AuthenticationUtil.isKerberosAuthenticationEnabled()) {
basicAuthUsernamePassword = getUserInput();
new AtlasClientBaseExample(urls, basicAuthUsernamePassword);
} else {
new AtlasClientBaseExample(urls);
}
try {
// TypeDef
atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 83 (patched)
<https://reviews.apache.org/r/72698/#comment310201>
In case of kerberos urls will be null as we populate that if kerberos not
enabled.
atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 86 (patched)
<https://reviews.apache.org/r/72698/#comment310203>
Instead of commenting like typedef then entity examples etc. it would be
good to move these to separate private functions. Please refer -
FUNCTIONS SHOULD DO ONE THING. THEY SHOULD DO IT WELL. THEY SHOULD DO IT
ONLY (more details -
https://learning.oreilly.com/library/view/clean-code/9780136083238/chapter03.html#ch3)
https://softwareengineering.stackexchange.com/questions/137941/should-a-method-do-one-thing-and-be-good-at-it
atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 140 (patched)
<https://reviews.apache.org/r/72698/#comment310199>
Please use Console.readPassword instead
- Sidharth Mishra
On July 21, 2020, 4:24 p.m., Jyoti Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
>
> (Updated July 21, 2020, 4:24 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath
> Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Using this project users can get an idea as how to integrate with Atlas using
> AtlasCleint. This helps the user to understand the basic rest functionality
> of Atlas such as
>
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
>
>
> Diffs
> -
>
> atlas-examples/pom.xml PRE-CREATION
> atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
> PRE-CREATION
> atlas-examples/src/main/java/org/apache/atlas/DiscoveryExample.java
> PRE-CREATION
> atlas-examples/src/main/java/org/apache/atlas/EntityRestBaseExample.java
> PRE-CREATION
> atlas-examples/src/main/java/org/apache/atlas/GlossaryExample.java
> PRE-CREATION
> atlas-examples/src/main/java/org/apache/atlas/LineageExample.java
> PRE-CREATION
> atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java
> PRE-CREATION
> pom.xml 5e0442ae5
>
>
> Diff: https://reviews.apache.org/r/72698/diff/2/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Jyoti Singh
>
>
Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72638/#review221215 --- webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java Lines 38 (patched) <https://reviews.apache.org/r/72638/#comment310041> Please remove unused imports. webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java Lines 52 (patched) <https://reviews.apache.org/r/72638/#comment310042> Instead of import * use below: import static org.testng.Assert.assertNotNull; import static org.testng.Assert.fail; import static org.testng.Assert.assertNull; import static org.testng.Assert.assertEquals; - Sidharth Mishra On July 10, 2020, 12:05 a.m., Jyoti Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72638/ > --- > > (Updated July 10, 2020, 12:05 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath > Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3875 > https://issues.apache.org/jira/browse/ATLAS-3875 > > > Repository: atlas > > > Description > --- > > There are many missing API endpoints in AtlasClientV2. This solution is > adding functions corresponding to those missing APIs for the following Rest > endpoints. > > 1. TypeRest > 2. EntityRest > 3. LineageRest > 4. DiscoveryRest > 5. GlossaryRest > 6. RelationshipRest > > This will enable users to test and integrate with APIs more effectively via > AtlasCient. > > > Diffs > - > > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 7b6d1d0f3 > client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java e3d2ebc34 > webapp/pom.xml 3c55b4dd4 > webapp/src/test/java/org/apache/atlas/web/TestUtils.java e22a1c10d > webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java > 680028892 > webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java > 808f62354 > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > cc883d615 > > webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java > PRE-CREATION > > webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java > PRE-CREATION > > webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java > 331ea2c47 > webapp/src/test/resources/json/search-parameters/attribute-filters.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/72638/diff/4/ > > > Testing > --- > > IT has been added for newly created functions in AtlasClientV2. > Validated by running IT on local system. > > > Thanks, > > Jyoti Singh > >
Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72638/#review221214 --- client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java Lines 84 (patched) <https://reviews.apache.org/r/72638/#comment310038> Please remove one extra space added before String client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java Lines 107 (patched) <https://reviews.apache.org/r/72638/#comment310039> We don't have any v2 API at AtlasBaseClient. Could you please move this to AtlasClientV2. client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java Lines 489 (patched) <https://reviews.apache.org/r/72638/#comment310040> Please move this as well to v2 as this is the base client. - Sidharth Mishra On July 10, 2020, 12:05 a.m., Jyoti Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72638/ > --- > > (Updated July 10, 2020, 12:05 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath > Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3875 > https://issues.apache.org/jira/browse/ATLAS-3875 > > > Repository: atlas > > > Description > --- > > There are many missing API endpoints in AtlasClientV2. This solution is > adding functions corresponding to those missing APIs for the following Rest > endpoints. > > 1. TypeRest > 2. EntityRest > 3. LineageRest > 4. DiscoveryRest > 5. GlossaryRest > 6. RelationshipRest > > This will enable users to test and integrate with APIs more effectively via > AtlasCient. > > > Diffs > - > > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 7b6d1d0f3 > client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java e3d2ebc34 > webapp/pom.xml 3c55b4dd4 > webapp/src/test/java/org/apache/atlas/web/TestUtils.java e22a1c10d > webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java > 680028892 > webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java > 808f62354 > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > cc883d615 > > webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java > PRE-CREATION > > webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java > PRE-CREATION > > webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java > 331ea2c47 > webapp/src/test/resources/json/search-parameters/attribute-filters.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/72638/diff/4/ > > > Testing > --- > > IT has been added for newly created functions in AtlasClientV2. > Validated by running IT on local system. > > > Thanks, > > Jyoti Singh > >
Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72638/#review221213
---
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Line 40 (original), 57 (patched)
<https://reviews.apache.org/r/72638/#comment310029>
Please remove this one as its already imported above
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 103 (patched)
<https://reviews.apache.org/r/72638/#comment310030>
Please maintain the above alignment.
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 112 (patched)
<https://reviews.apache.org/r/72638/#comment310032>
Align below lines like:
//Glossary APIs
private static final String GLOSSARY_URI= BASE_URI +
"v2/glossary";
private static final String GLOSSARY_TERM = GLOSSARY_URI +
"/term";
private static final String GLOSSARY_TERMS = GLOSSARY_URI +
"/terms";
private static final String GLOSSARY_CATEGORY = GLOSSARY_URI +
"/category";
private static final String GLOSSARY_CATEGORIES = GLOSSARY_URI +
"/categories";
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 118 (patched)
<https://reviews.apache.org/r/72638/#comment310031>
Please have one blank line instead of two
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 220 (patched)
<https://reviews.apache.org/r/72638/#comment310033>
Please rename getRelationShipDefByName -> getRelationshipDefByName (As
RelationShip is a single word)
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 224 (patched)
<https://reviews.apache.org/r/72638/#comment310034>
getRelationShipDefByGuid -> getRelationshipDefByGuid
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 597 (patched)
<https://reviews.apache.org/r/72638/#comment310035>
Please remove these two extra blank lines
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 648 (patched)
<https://reviews.apache.org/r/72638/#comment310037>
Please remove the below lines as it seems its not bMultivaluedMap queryParams = new MultivaluedMapImpl();
queryParams.add("name", name);
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 650 (patched)
<https://reviews.apache.org/r/72638/#comment310036>
Please remove the commented line
- Sidharth Mishra
On July 10, 2020, 12:05 a.m., Jyoti Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72638/
> ---
>
> (Updated July 10, 2020, 12:05 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath
> Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
>
>
> Repository: atlas
>
>
> Description
> ---
>
> There are many missing API endpoints in AtlasClientV2. This solution is
> adding functions corresponding to those missing APIs for the following Rest
> endpoints.
>
> 1. TypeRest
> 2. EntityRest
> 3. LineageRest
> 4. DiscoveryRest
> 5. GlossaryRest
> 6. RelationshipRest
>
> This will enable users to test and integrate with APIs more effectively via
> AtlasCient.
>
>
> Diffs
> -
>
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
> 7b6d1d0f3
> client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java e3d2ebc34
> webapp/pom.xml 3c55b4dd4
> webapp/src/test/java/org/apache/atlas/web/TestUtils.java e22a1c10d
> webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
> 680028892
> webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java
> 808f62354
>
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
> cc883d615
>
> webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java
> PRE-CREATION
>
> webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java
> PRE-CREATION
>
> webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java
> 331ea2c47
> webapp/src/test/resources/json/search-parameters/attribute-filters.json
> PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/72638/diff/4/
>
>
> Testing
> ---
>
> IT has been added for newly created functions in AtlasClientV2.
> Validated by running IT on local system.
>
>
> Thanks,
>
> Jyoti Singh
>
>
Re: [VOTE] Release Apache Atlas version 2.1.0 - rc3
+1 for Apache Atlas 2.1 rc3 Validated the following: - Verified the MD5/sha512 hash - Verified PGP signature - No binaries in extracted source - Verified the build with embedded-hbase and solr - Ran quick start and tested some sanity test cases - Basic/advanced search, Classification, Glossary, tag propagation, Admin purge and audit, Lineage, Propagated from in case of purged entity On Tue, Jul 14, 2020 at 4:24 AM Keval Bhatt wrote: > +1 (binding) for Apache Atlas 2.1.0 RC3 > > Validated the following: > >- Verified the md5/sha checksum of the source bundle >- Verified the build with -Pdist,embedded-hbase-solr >- Created classification, updated the attributes and associated with the >entity and term >- Tested basic search with the type, classification, term and their >attribute filters >- Tested advance search >- Saved the search for later use. >- Created Business metadata and associated with an entity. >- Create Labels and user-define properties >- Tested Lineage and applied filter hide/show deleted entities and >process >- Checked the relationship diagram >- Entity create/update test from UI >- Checked the statistics page. >- checked the loader hide change for term association. > > > On Tue, Jul 14, 2020 at 12:01 PM vishal suvagia > wrote: > > > +1 for Apache Atlas 2.1.0 RC#3. > > Verified clean compilation and rat-checks.Validated the source checksums > > and hash.Verified quick-start data and UI to be working > > > > > > RegardsVishal Suvagia. > >On Monday, 13 July, 2020, 9:20:54 pm IST, Sharmadha Sainath > > wrote: > > > > +1 for Apache Atlas 2.1.0-rc3 release. > > > > - Validated MD5 and SHA512 checksum of source. > > - Verified PGP Signature > > - Brought up Atlas using embedded HBase and Solr. > > - Verified DSL , Basic Search , Quick search and Suggestions > > - Validated CRUD on labels, custom attributes , business metadata > > - Verified Basic search using system attributes filters (multiple > > classifications , labels , custom attributes , business metadata > search) > > - Added labels , user defined properties (custom attributes) , business > > metadata to entities and verified the entity properties and audits. > > - Purged Soft Deleted entity and verified the audits of purged entity. > > Deleted tag associated to the purged entity and verified there are no > > references. > > - Saved the Favorite searches and executed them. > > - Tested Atlas UI against XSS strings > > > > Thanks, > > Sharmadha Sainath > > > > On Mon, Jul 13, 2020 at 7:15 PM Nixon Rodrigues < > > nixon.rodrig...@freestoneinfotech.com> wrote: > > > > > +1 for Atlas 2.1.0 rc3 release. > > > > > > Verified signature and MD5/SHA keys. > > > Verified release tag on git repo. > > > Verified build with rc3 bits. > > > Verified UI with Quick Start data. > > > > > > Thanks & Regards > > > Nixon > > > > > > > > > > > > On Mon, Jul 13, 2020 at 12:44 PM Mandar Ambawane < > > > mandar.ambaw...@freestoneinfotech.com> wrote: > > > > > > > +1 for Apache Atlas 2.1.0 RC3 release > > > > > > > > Validated the following: > > > > 1. Verified the source SHA512 and MD5 checksum > > > > 2. Verified the build with -Pdist,embedded-hbase-solr > > > > 3. Started Atlas server successfully > > > > 4. Verified quick_start.py, all entities and tag get created > > successfully > > > > 5. Created Business metadata and associated with an entity. > > > > 6. Checked Basic and advanced search > > > > > > > > Thanks, > > > > Mandar > > > > > > > > On Mon, Jul 13, 2020 at 12:39 PM Jayendra Parab < > > > jayendra.pa...@gmail.com> > > > > wrote: > > > > > > > > > +1 for Apache Atlas 2.1.0 RC3 release > > > > > > > > > > Validated the following: > > > > > 1. Verified the source SHA512 and MD5 checksum > > > > > 2. Verified the build with -Pdist,embedded-hbase-solr > > > > > 3. Started Atlas server successfully > > > > > 4. Verified quick_start.py, all entities and tag get created > > > successfully > > > > > 5. Created entities, applied tags from UI > > > > > 6. Checked Basic and advanced search > > > > > > > > > > Regards, > > > > > Jayendra. > > > > > > > > > > On Mon, Jul 13, 2020 at 12:17 PM Sarath Subramanian < > > sar...@apache.org > > > > > > > > > wrote: > > > > > > > > > > > +1 for Apache Atlas 2.1 rc3 > > > > > > > > > > > > Validated the following: > > > > > > - Validated MD5 checksum > > > > > > - Validated sha512 hash > > > > > > - Verified PGP signature > > > > > > - No binaries in extracted source > > > > > > - Built successfully using embedded-hbase and solr profile > > > > > > - Ran quick start and tested basic sanity - saved search, > > > > basic/advanced > > > > > > search, Glossary, tag propagation, Admin purge, system attribute > > > search > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 9, 2020 at 9:59 PM Madhan Neethiraj < > mad...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > Atlas team, > > > > > > > > > > > > >
Re: Review Request 72658: ATLAS-3877: Audit API returns the audit information for an unauthorised entity
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72658/
---
(Updated July 8, 2020, 7:31 a.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
Bugs: ATLAS-3877
https://issues.apache.org/jira/browse/ATLAS-3877
Repository: atlas
Description
---
ATLAS-3877: Audit Fixed the authorization for audit both in case of non purged
entity to populate classifcation and purged entity by getting the header from
HBase Audit
Diffs (updated)
-
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
717310daf
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
bf1629cb3
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 88de8b679
Diff: https://reviews.apache.org/r/72658/diff/2/
Changes: https://reviews.apache.org/r/72658/diff/1-2/
Testing
---
#Deleted Entity
//Allow
curl --location --request GET
'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/196bf3a9-07c6-4152-82ad-10176c5df5b9/audit?count=1=ENTITY_DELETE'
\
-u admin:admin123
Response:
[{"entityId":"196bf3a9-07c6-4152-82ad-10176c5df5b9","timestamp":1594192518488,"user":"admin","action":"ENTITY_DELETE","details":"Deleted
entity","eventKey":"196bf3a9-07c6-4152-82ad-10176c5df5b9:1594192518488:1:1594192519277","entity":null,"type":null}]
//Deny
curl --location --request GET
'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/196bf3a9-07c6-4152-82ad-10176c5df5b9/audit?count=1=ENTITY_DELETE'
\
-u sid:sid
Response:
{"errorCode":"ATLAS-403-00-001","errorMessage":"sid is not authorized to
perform read entity: guid=196bf3a9-07c6-4152-82ad-10176c5df5b9"}
#Purged Entity
//Allow
curl --location --request GET
'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/def375bd-9bad-42f3-9fa7-9edc1c7d8d96/audit?count=1=ENTITY_PURGE'
\
-u admin:admin123
Response:
[{"entityId":"def375bd-9bad-42f3-9fa7-9edc1c7d8d96","timestamp":1594191071761,"user":"admin","action":"ENTITY_PURGE","details":"Purged:
{\"typeName\":\"hive_table\",\"attributes\":{\"owner\":\"admin\",\"createTime\":1594177077000,\"qualifiedName\":\"default.test_july_7_2020_ctas_10@cm\",\"name\":\"test_july_7_2020_ctas_10\"},\"guid\":\"def375bd-9bad-42f3-9fa7-9edc1c7d8d96\",\"isIncomplete\":false,\"provenanceType\":0,\"status\":\"DELETED\",\"version\":0,\"classifications\":[{\"typeName\":\"NOT_IMP\",\"entityGuid\":\"5f52ee87-85c8-46ac-ad38-e0e5be935bb2\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"validityPeriods\":[],\"removePropagationsOnEntityDelete\":false},{\"typeName\":\"PARENT\",\"entityGuid\":\"73cc5730-494b-4313-8747-9ccf8a1b0227\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"validityPeriods\":[],\"removePropagationsOnEntityDelete\":false}],\"meanings\":[],\"proxy\":false}","eventKey":"def375bd-9bad-42f3-9fa7-9edc1c7d8d96:1594191071761:3:1594191103634","entity":null
,"type":null}]
//Deny
curl --location --request GET
'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/def375bd-9bad-42f3-9fa7-9edc1c7d8d96/audit?count=1=ENTITY_PURGE'
\
-u sid:sid
Response:
{"errorCode":"ATLAS-403-00-001","errorMessage":"sid is not authorized to
perform read entity audit: guid=def375bd-9bad-42f3-9fa7-9edc1c7d8d96"}
Thanks,
Sidharth Mishra
Review Request 72658: ATLAS-3877: Audit API returns the audit information for an unauthorised entity
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72658/
---
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
Bugs: ATLAS-3877
https://issues.apache.org/jira/browse/ATLAS-3877
Repository: atlas
Description
---
ATLAS-3877: Audit Fixed the authorization for audit both in case of non purged
entity to populate classifcation and purged entity by getting the header from
HBase Audit
Diffs
-
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
717310daf
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
bf1629cb3
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 88de8b679
Diff: https://reviews.apache.org/r/72658/diff/1/
Testing
---
#Deleted Entity
//Allow
curl --location --request GET
'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/196bf3a9-07c6-4152-82ad-10176c5df5b9/audit?count=1=ENTITY_DELETE'
\
-u admin:admin123
Response:
[{"entityId":"196bf3a9-07c6-4152-82ad-10176c5df5b9","timestamp":1594192518488,"user":"admin","action":"ENTITY_DELETE","details":"Deleted
entity","eventKey":"196bf3a9-07c6-4152-82ad-10176c5df5b9:1594192518488:1:1594192519277","entity":null,"type":null}]
//Deny
curl --location --request GET
'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/196bf3a9-07c6-4152-82ad-10176c5df5b9/audit?count=1=ENTITY_DELETE'
\
-u sid:sid
Response:
{"errorCode":"ATLAS-403-00-001","errorMessage":"sid is not authorized to
perform read entity: guid=196bf3a9-07c6-4152-82ad-10176c5df5b9"}
#Purged Entity
//Allow
curl --location --request GET
'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/def375bd-9bad-42f3-9fa7-9edc1c7d8d96/audit?count=1=ENTITY_PURGE'
\
-u admin:admin123
Response:
[{"entityId":"def375bd-9bad-42f3-9fa7-9edc1c7d8d96","timestamp":1594191071761,"user":"admin","action":"ENTITY_PURGE","details":"Purged:
{\"typeName\":\"hive_table\",\"attributes\":{\"owner\":\"admin\",\"createTime\":1594177077000,\"qualifiedName\":\"default.test_july_7_2020_ctas_10@cm\",\"name\":\"test_july_7_2020_ctas_10\"},\"guid\":\"def375bd-9bad-42f3-9fa7-9edc1c7d8d96\",\"isIncomplete\":false,\"provenanceType\":0,\"status\":\"DELETED\",\"version\":0,\"classifications\":[{\"typeName\":\"NOT_IMP\",\"entityGuid\":\"5f52ee87-85c8-46ac-ad38-e0e5be935bb2\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"validityPeriods\":[],\"removePropagationsOnEntityDelete\":false},{\"typeName\":\"PARENT\",\"entityGuid\":\"73cc5730-494b-4313-8747-9ccf8a1b0227\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"validityPeriods\":[],\"removePropagationsOnEntityDelete\":false}],\"meanings\":[],\"proxy\":false}","eventKey":"def375bd-9bad-42f3-9fa7-9edc1c7d8d96:1594191071761:3:1594191103634","entity":null
,"type":null}]
//Deny
curl --location --request GET
'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/def375bd-9bad-42f3-9fa7-9edc1c7d8d96/audit?count=1=ENTITY_PURGE'
\
-u sid:sid
Response:
{"errorCode":"ATLAS-403-00-001","errorMessage":"sid is not authorized to
perform read entity audit: guid=def375bd-9bad-42f3-9fa7-9edc1c7d8d96"}
Thanks,
Sidharth Mishra
Re: [VOTE] Release Apache Atlas version 2.1.0 - rc1
+1 for Apache Atlas 2.1 RC1 Validated the following items. 1. Verified the md5/sha512 checksum of source bundle 2. Packaging with -Pdist,embedded-hbase-solr works fine 3. Starting Atlas server works fine 4. quick_start.py works fine and all the entities and tags get created successfully at UI 5. Tested basic and quick search flow 6. Tested entity and tag delete 7. Tested entities purge 8. Checked admin audit for entity purge Regards, Sid On Mon, Jun 8, 2020 at 11:41 AM Madhan Neethiraj wrote: > Bolke, > > Can you send specific security concerns that need to addressed in 2.1.0 > release? If this is about questions from Melinda Crane, I asked for Melinda > details to better understand the questions. I suggest we know the details > before treating them as release blockers. > > Thanks, > Madhan > > On 6/8/20, 9:56 AM, "Bolke de Bruin" wrote: > > This is my -1 (non-binding) which I will revert to +1 when the > security concerns have been addressed/clarified. > > Given that Atlas has pretty slow releases I think it smart to address > them now. > > Cheers > Bolke > > Sent from my iPhone > > > On 8 Jun 2020, at 18:18, Sharmadha Sainath > wrote: > > > > +1 for Apache Atlas 2.1 RC1 > > > > > > - Validated MD5 and SHA512 checksum of source. > > - Brought up Atlas using embedded HBase and Solr. > > - Verified DSL , Basic Search , Quick search and Suggestions > > - Verified Basic search using system attributes filters (multiple > > classification search , labels , custom attributes , business > metadata > > search) > > - Added labels , user defined properties (custom attributes) , > business > > metadata to entities and verified the entity properties and audits. > > - Purged Soft Deleted entity and verified purge audits. > > - Saved the Favorite searches and executed them. > > > > Thanks, > > Sharmadha S. > > > > > > > >> On Mon, Jun 8, 2020 at 6:23 PM Nixon Rodrigues < > >> nixon.rodrig...@freestoneinfotech.com> wrote: > >> > >> Thanks Madhan for putting 2.1 RC1 together for the vote. > >> > >> +1 for Apache Atlas 2.1 RC1 > >> > >> Validated the following items. > >> > >> 1. verified the md5/sha checksum of source bundle. > >> 2. packaging with -Pdist,embedded-hbase-solr is good. > >> 3. Unit testcase passes. > >> 4. Run quickstart and all the types and entities created are > loading > >> correctly on the UI. > >> 5. Create tags, glossary, term and its apply correctly on > entities. > >> 6. Created business metadata with attributes and applied it to > entities. > >> 7. tested basic and quick search flow. > >> 8. Verified statistics page details > >> > >> > >> Thanks & Regards > >> Nixon > >> > >> > >> > >> On Mon, Jun 8, 2020 at 11:17 AM Sarath Subramanian < > sar...@apache.org> > >> wrote: > >> > >>> +1 for Apache Atlas 2.1 RC1 > >>> > >>> Validated the following: > >>> > >>> - No binaries in source > >>> - Verified source checksum - md5 and sha512 > >>> - Verified signature > >>> - Build was successful on the source (mvn clean -DskipTests > package > >>> -Pdist,embedded-hbase-solr) > >>> - Ran quickStart and validated types and entities were created > >>> successfully. > >>> - Validated tag propagation works > >>> - Added/removed labels to an entity and verified audit is getting > >>> updated correctly > >>> - Added user-defined properties and business metadata attributes. > >>> - Deleted few entities and purged those entities - validated if > purge > >>> audit recorded correctly. > >>> - Basic sanity tests - basic search (with system attributes), > advanced > >>> search, quick search. > >>> > >>> > >>> > >>> Thanks, > >>> Sarath > >>> > >>> > >>> > >>> On Sat, Jun 6, 2020 at 10:22 AM Madhan Neethiraj < > mad...@apache.org> > >>> wrote: > >>> > (resending with plain-text format) > > Atlas team, > > Apache Atlas 2.1.0 rc1, with following fixes since rc0, is now > >> available > for vote within dev community. > > ATLAS-3770: UI(Classic): Active and Deleted hyperlinks for > certain > entities throwing error on click > ATLAS-3766: Stats modal not close issue #2 > ATLAS-3674: ZipFileMigationImporter: Migration status display > fix. > Part 2 > > Links to the release artifacts are given below. Please review and > vote. > > The vote will be open for at least 72 hours or until necessary > votes > >> are > reached. > [ ] +1 Approve > [ ] +0 No opinion > [ ] -1 Disapprove (and reason why) >
Re: Review Request 72540: ATLAS-3806: Classifications information missing in notification events during entity create/update
ionsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
{
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.168.128",
"msgCreatedBy": "",
"msgCreationTime": 1590181680710,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_process",
"attributes": {
"qualifiedName":
"sid_hive_db.sid_hive_table_ctas3@bulk:1590120919000",
"name": "create table sid_hive_table_ctas3 as select * from
sid_hive_table_1"
},
"guid": "36cc1a73-9c55-4a66-9d59-fa1a2f535791",
"status": "ACTIVE",
"displayText": "create table sid_hive_table_ctas3 as select * from
sid_hive_table_1",
"classificationNames": [
"ETL",
"PII",
"Fact"
],
"classifications": [
{
"typeName": "ETL",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
{
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.168.128",
"msgCreatedBy": "",
"msgCreationTime": 1590181680714,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "anonymous",
"createTime": 1590120919000,
"qualifiedName": "sid_hive_db.sid_hive_table_ctas3@bulk",
"name": "sid_hive_table_ctas3"
},
"guid": "ab302745-a843-4164-8166-017f85c7e15e",
"status": "ACTIVE",
"displayText": "sid_hive_table_ctas3",
"classificationNames": [
"ETL",
"PII",
"Fact"
],
"classifications": [
{
"typeName": "ETL",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
Thanks,
Sidharth Mishra
Re: Review Request 72540: ATLAS-3806: Classifications information missing in notification events during entity create/update
ionsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
{
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.168.128",
"msgCreatedBy": "",
"msgCreationTime": 1590181680710,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_process",
"attributes": {
"qualifiedName":
"sid_hive_db.sid_hive_table_ctas3@bulk:1590120919000",
"name": "create table sid_hive_table_ctas3 as select * from
sid_hive_table_1"
},
"guid": "36cc1a73-9c55-4a66-9d59-fa1a2f535791",
"status": "ACTIVE",
"displayText": "create table sid_hive_table_ctas3 as select * from
sid_hive_table_1",
"classificationNames": [
"ETL",
"PII",
"Fact"
],
"classifications": [
{
"typeName": "ETL",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
{
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.168.128",
"msgCreatedBy": "",
"msgCreationTime": 1590181680714,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "anonymous",
"createTime": 1590120919000,
"qualifiedName": "sid_hive_db.sid_hive_table_ctas3@bulk",
"name": "sid_hive_table_ctas3"
},
"guid": "ab302745-a843-4164-8166-017f85c7e15e",
"status": "ACTIVE",
"displayText": "sid_hive_table_ctas3",
"classificationNames": [
"ETL",
"PII",
"Fact"
],
"classifications": [
{
"typeName": "ETL",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
Thanks,
Sidharth Mishra
Re: Review Request 72540: ATLAS-3806: Classifications information missing in notification events during entity create/update
opagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
{
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.168.128",
"msgCreatedBy": "",
"msgCreationTime": 1590181680710,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_process",
"attributes": {
"qualifiedName":
"sid_hive_db.sid_hive_table_ctas3@bulk:1590120919000",
"name": "create table sid_hive_table_ctas3 as select * from
sid_hive_table_1"
},
"guid": "36cc1a73-9c55-4a66-9d59-fa1a2f535791",
"status": "ACTIVE",
"displayText": "create table sid_hive_table_ctas3 as select * from
sid_hive_table_1",
"classificationNames": [
"ETL",
"PII",
"Fact"
],
"classifications": [
{
"typeName": "ETL",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
{
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.168.128",
"msgCreatedBy": "",
"msgCreationTime": 1590181680714,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "anonymous",
"createTime": 1590120919000,
"qualifiedName": "sid_hive_db.sid_hive_table_ctas3@bulk",
"name": "sid_hive_table_ctas3"
},
"guid": "ab302745-a843-4164-8166-017f85c7e15e",
"status": "ACTIVE",
"displayText": "sid_hive_table_ctas3",
"classificationNames": [
"ETL",
"PII",
"Fact"
],
"classifications": [
{
"typeName": "ETL",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
Thanks,
Sidharth Mishra
Re: Review Request 72540: ATLAS-3806: Classifications information missing in notification events during entity create/update
> On May 26, 2020, 10:45 p.m., Madhan Neethiraj wrote:
> > Sid - I suggest to consider the following alternate approach to address
> > this issue:
> >
> > public class EntityGraphMapper{
> > private AtlasEntityHeader constructHeader(AtlasEntity entity, final
> > AtlasEntityType type, AtlasVertex vertex) {
> > /*
> > * replace current implementation with a call to
> > entityRetriever.toAtlasEntityHeaderWithClassifications()
> > *
> > AtlasEntityHeader header = new
> > AtlasEntityHeader(entity.getTypeName());
> >
> > header.setGuid(getIdFromVertex(vertex));
> > header.setStatus(entity.getStatus());
> > header.setIsIncomplete(entity.getIsIncomplete());
> >
> > for (AtlasAttribute attribute : type.getUniqAttributes().values()) {
> > header.setAttribute(attribute.getName(),
> > entity.getAttribute(attribute.getName()));
> > }
> >
> > return header;
> > *
> > */
> >
> > return
> > entityRetriever.toAtlasEntityHeaderWithClassifications(vertex);
> > }
> > }
> >
> > public class EntityMutationResponse {
> > JsonIgnore
> > public void addEntity(EntityOperation op, AtlasEntityHeader header) {
> > // if an entity is already included in CREATE, update the header,
> > to capture propagated classifications
> > if (op == EntityOperation.UPDATE || op ==
> > EntityOperation.PARTIAL_UPDATE) {
> > if (entityHeaderExists(getCreatedEntities(), header.getGuid()))
> > {
> > // return;
> > op = EntityOperaton.CREATE;
> > }
> > }
> >
> > ...
> > }
> > }
As you suggested I have changed the code. The issue is, we cache the entity at
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java:285
and 313 which is not getting updated with classification. At
org.apache.atlas.repository.store.graph.v2.AtlasEntityChangeNotifier#notifyPropagatedEntities
when we call fullTextMapperV2.getAndCacheEntity(guid) we still get the old one
without classification. So along with this change I added code to
constructHeader so that it will set classification to entity from entityHeader.
Now the updated solution works.
- Sidharth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72540/#review220873
---
On May 28, 2020, 10:58 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72540/
> ---
>
> (Updated May 28, 2020, 10:58 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-3806
> https://issues.apache.org/jira/browse/ATLAS-3806
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Classifications information missing in notification events during entity
> create/update
>
>
> Diffs
> -
>
>
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
> d83f0e0f1
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> 2ed524ff5
>
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
> a677b315c
>
>
> Diff: https://reviews.apache.org/r/72540/diff/2/
>
>
> Testing
> ---
>
> For add new ctas notification looks like (Just shown for ctas entity):
> {
> "version": {
> "version": "1.0.0",
> "versionParts": [
> 1
> ]
> },
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.168.128",
> "msgCreatedBy": "",
> "msgCreationTime": 1590176023734,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes": {
> "owner": "anonymous",
> "cre
Review Request 72540: ATLAS-3806: Classifications information missing in notification events during entity create/update
23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
{
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.168.128",
"msgCreatedBy": "",
"msgCreationTime": 1590181680710,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_process",
"attributes": {
"qualifiedName":
"sid_hive_db.sid_hive_table_ctas3@bulk:1590120919000",
"name": "create table sid_hive_table_ctas3 as select * from
sid_hive_table_1"
},
"guid": "36cc1a73-9c55-4a66-9d59-fa1a2f535791",
"status": "ACTIVE",
"displayText": "create table sid_hive_table_ctas3 as select * from
sid_hive_table_1",
"classificationNames": [
"ETL",
"PII",
"Fact"
],
"classifications": [
{
"typeName": "ETL",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
{
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.168.128",
"msgCreatedBy": "",
"msgCreationTime": 1590181680714,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "anonymous",
"createTime": 1590120919000,
"qualifiedName": "sid_hive_db.sid_hive_table_ctas3@bulk",
"name": "sid_hive_table_ctas3"
},
"guid": "ab302745-a843-4164-8166-017f85c7e15e",
"status": "ACTIVE",
"displayText": "sid_hive_table_ctas3",
"classificationNames": [
"ETL",
"PII",
"Fact"
],
"classifications": [
{
"typeName": "ETL",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "PII",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "Fact",
"entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
},
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1590181673717
}
}
Thanks,
Sidharth Mishra
Re: Review Request 72515: ATLAS-3785: Modify the AWS s3 v2 model to make relationship def end points as legacy attributes and objectPrefix as optional
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72515/ --- (Updated May 15, 2020, 6:50 a.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3785 https://issues.apache.org/jira/browse/ATLAS-3785 Repository: atlas Description --- Modify the aws s3 v2 model to make relationship def end points as legacy attributes and objectPrefix as optional Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json 676a2237c Diff: https://reviews.apache.org/r/72515/diff/2/ Changes: https://reviews.apache.org/r/72515/diff/1-2/ Testing --- Manually added to my cluster Thanks, Sidharth Mishra
Re: Review Request 72512: Edge Creation: Refactoring
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72512/#review220776 --- Ship it! Ship It! - Sidharth Mishra On May 14, 2020, 10:41 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72512/ > --- > > (Updated May 14, 2020, 10:41 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-3797 > https://issues.apache.org/jira/browse/ATLAS-3797 > > > Repository: atlas > > > Description > --- > > **Approach** > Modified: _GraphHelper.getOrCreateEdge_ to use > _AtlasJanusGraph.getEdgeBetweenVertices_. > Modified: _AtlasJanusGraph.getEdgeBetweenVertices_ modified fetch. > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 35274c9d1 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > b1368e5e9 > > > Diff: https://reviews.apache.org/r/72512/diff/2/ > > > Testing > --- > > **Pre-commit** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1896/ > > **Volume test** > Case edge creation involves existing entity where large number of existing > edges (200K), this change shows significant improvement. > > > Thanks, > > Ashutosh Mestry > >
Review Request 72515: ATLAS-3785: Modify the AWS s3 v2 model to make relationship def end points as legacy attributes and objectPrefix as optional
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72515/ --- Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3785 https://issues.apache.org/jira/browse/ATLAS-3785 Repository: atlas Description --- Modify the aws s3 v2 model to make relationship def end points as legacy attributes and objectPrefix as optional Diffs - addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json 676a2237c Diff: https://reviews.apache.org/r/72515/diff/1/ Testing --- Manually added to my cluster Thanks, Sidharth Mishra
Re: Review Request 72512: Edge Creation: Refactoring
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72512/#review220770 --- graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java Line 177 (original), 179 (patched) <https://reviews.apache.org/r/72512/#comment309462> As we are not returning ret, please rename this to something more meaningful like gremlinEdge ( As we are trying to find the gremlinEdge here) graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java Lines 183 (patched) <https://reviews.apache.org/r/72512/#comment309461> Kindly change this to somthing like below as second ret.property(STATE_PROPERTY_KEY).isPresent() check is not rquired: if (ret != null && (!ret.property(STATE_PROPERTY_KEY).isPresent() || ret.property(STATE_PROPERTY_KEY).value().equals(AtlasEntity.Status.ACTIVE.toString())) graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java Lines 195 (patched) <https://reviews.apache.org/r/72512/#comment309463> Just to avoid two returns statement as we do other places in atlas, we can do something like: Initialize variable AtlasEdge ret = null; At 192 line: ret = GraphDbObjectFactory.createEdge(this, ret); At the end return ret; - Sidharth Mishra On May 14, 2020, 6:19 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72512/ > --- > > (Updated May 14, 2020, 6:19 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-3797 > https://issues.apache.org/jira/browse/ATLAS-3797 > > > Repository: atlas > > > Description > --- > > **Approach** > Modified: _GraphHelper.getOrCreateEdge_ to use > _AtlasJanusGraph.getEdgeBetweenVertices_. > Modified: _AtlasJanusGraph.getEdgeBetweenVertices_ modified fetch. > > > Diffs > - > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 35274c9d1 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > b1368e5e9 > > > Diff: https://reviews.apache.org/r/72512/diff/1/ > > > Testing > --- > > **Pre-commit** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1896/ > > **Volume test** > Case edge creation involves existing entity where large number of existing > edges (200K), this change shows significant improvement. > > > Thanks, > > Ashutosh Mestry > >
Re: Review Request 72464: ATLAS-3778: Improve performance during classification delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72464/#review220605 --- Ship it! Ship It! - Sidharth Mishra On May 3, 2020, 11:04 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72464/ > --- > > (Updated May 3, 2020, 11:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, > and Sidharth Mishra. > > > Bugs: ATLAS-3778 > https://issues.apache.org/jira/browse/ATLAS-3778 > > > Repository: atlas > > > Description > --- > > When deleting a classification, we remove the following properties from > entity vertex and update these properties with the classification name > removed. > > PROPAGATED_TRAIT_NAMES_PROPERTY_KEY > TRAIT_NAMES_PROPERTY_KEY > > This will be an expensive operation during commit time when dealing with > large number of classifications. This improvement will only remove the > deleted classification name from both the above property keys and the commit > foot print will be less. > > > Diffs > - > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java > 4af39eded > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusElement.java > a9cc5a6ce > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 5643599a4 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 5564bc59a > > > Diff: https://reviews.apache.org/r/72464/diff/2/ > > > Testing > --- > > Manually tested on delete classification with 400 classifications associated > to entity. > > Performance improvement seen from 19 mins to 29 secs > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 72452: Efficiently Searching for Edges Between Vertices
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72452/#review220604
---
Ship it!
Ship It!
- Sidharth Mishra
On April 30, 2020, 4:04 a.m., Ashutosh Mestry wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72452/
> ---
>
> (Updated April 30, 2020, 4:04 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues,
> Sarath Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-3762
> https://issues.apache.org/jira/browse/ATLAS-3762
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Problem Definition**
> Please refer to JIRA for details.
>
> **Updates**
> - Modifed: _AtlasJanusGraph.wrapVertices_ and _AtlasJanusGraph.wrapEdges_ :
> Now uses genuine iterators. This reduces number of elements fetched, since
> the search is linear.
> - New: _AtlasVertex.getEdgeCount_ fetches edge count using iterator returned
> from _JanusVertex_. Fetching the count is effecient using stream support.
>
>
> **Metrics**
>
> This difference exists because earlier even _fromVertex_ did not have any
> edges, the search would iterate through all the edges of the _toVertex_ and
> return NULL. This condition is detected in the algorithm and used to reduce
> the search cycle.
>
> Before: "getRelationshipEdge":{"count":500,"timeTaken":37048}
> After: "getRelationshipEdge":{"count":500,"timeTaken":42}
>
>
> Diffs
> -
>
>
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
> 9406e26ff
>
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
> eb0206271
>
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java
> fdc9fd0b5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
> 2b8227a7e
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
> d1c1f1255
>
>
> Diff: https://reviews.apache.org/r/72452/diff/4/
>
>
> Testing
> ---
>
> **Volume testing**
> High volume testing makes the edge fetching effcient. Cases where incoming
> edges was in 1000s and outgoing edges was handful.
>
> Memory footprint has improved since JanusGrpah caches edges and then expires
> it. Fetching fewer edges will reduce number of items in memory.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1860/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 72452: Efficiently Searching for Edges Between Vertices
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72452/#review220557 --- Ship it! Ship It! - Sidharth Mishra On April 29, 2020, 9:17 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72452/ > --- > > (Updated April 29, 2020, 9:17 p.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, > Sarath Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3762 > https://issues.apache.org/jira/browse/ATLAS-3762 > > > Repository: atlas > > > Description > --- > > **Problem Definition** > Please refer to JIRA for details. > > **Updates** > - Modifed: _AtlasJanusGraph.wrapVertices_ and _AtlasJanusGraph.wrapEdges_ : > Now uses genuine iterators. This reduces number of elements fetched, since > the search is linear. > - New: _AtlasVertex.getEdgeCount_ fetches edge count using iterator returned > from _JanusVertex_. Fetching the count is effecient using stream support. > > > Diffs > - > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java > 9406e26ff > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > eb0206271 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java > fdc9fd0b5 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 2b8227a7e > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java > d1c1f1255 > > > Diff: https://reviews.apache.org/r/72452/diff/3/ > > > Testing > --- > > **Volume testing** > High volume testing makes the edge fetching effcient. Cases where incoming > edges was in 1000s and outgoing edges was handful. > > Memory footprint has improved since JanusGrpah caches edges and then expires > it. Fetching fewer edges will reduce number of items in memory. > > **Pre-commit Build** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1858/ > > > Thanks, > > Ashutosh Mestry > >
Re: Review Request 72452: Efficiently Searching for Edges Between Vertices
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72452/#review220553
---
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
Lines 780 (patched)
<https://reviews.apache.org/r/72452/#comment309045>
Just thinking the below might be more easy to understand if someone is not
aware of why we do both edge count.
AtlasPerfMetrics.MetricRecorder metric =
RequestContext.get().startMetricRecord("getRelationshipEdge");
AtlasEdge ret = null;
long toVertexIncomingEdgesCount =
graphHelper.getInComingEdgesByLabelCount(toVertex, relationshipLabel);
if(toVertexIncomingEdgesCount > 0) {
long fromVertexOutgoingEdgesCount =
graphHelper.getOutGoingEdgesByLabelCount(fromVertex, relationshipLabel);
if(toVertexIncomingEdgesCount < fromVertexOutgoingEdgesCount) {
Iterator edgesIterator =
graphHelper.getIncomingEdgesByLabel(toVertex, relationshipLabel);
ret = getActiveEdgeFromList(fromVertex.getId(),
edgesIterator);
} else if(fromVertexOutgoingEdgesCount > 0) {
Iterator edgesIterator =
graphHelper.getOutGoingEdgesByLabel(fromVertex, relationshipLabel);
ret = getActiveEdgeFromList(toVertex.getId(),
edgesIterator);
}
}
RequestContext.get().endMetricRecord(metric);
return ret;
- Sidharth Mishra
On April 29, 2020, 5:51 p.m., Ashutosh Mestry wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72452/
> ---
>
> (Updated April 29, 2020, 5:51 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues,
> Sarath Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-3762
> https://issues.apache.org/jira/browse/ATLAS-3762
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Problem Definition**
> Please refer to JIRA for details.
>
> **Updates**
> - Modifed: _AtlasJanusGraph.wrapVertices_ and _AtlasJanusGraph.wrapEdges_ :
> Now uses genuine iterators. This reduces number of elements fetched, since
> the search is linear.
> - New: _AtlasVertex.getEdgeCount_ fetches edge count using iterator returned
> from _JanusVertex_. Fetching the count is effecient using stream support.
>
>
> Diffs
> -
>
>
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
> 9406e26ff
>
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
> eb0206271
>
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java
> fdc9fd0b5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
> 2b8227a7e
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
> d1c1f1255
>
>
> Diff: https://reviews.apache.org/r/72452/diff/1/
>
>
> Testing
> ---
>
> **Volume testing**
> High volume testing makes the edge fetching effcient. Cases where incoming
> edges was in 1000s and outgoing edges was handful.
>
> Memory footprint has improved since JanusGrpah caches edges and then expires
> it. Fetching fewer edges will reduce number of items in memory.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1858/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 72411: ATLAS-3747: Atlas Admin Purge API should take list of guids as body with operation Put instead of Delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72411/ --- (Updated April 21, 2020, 9:45 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Summary (updated) - ATLAS-3747: Atlas Admin Purge API should take list of guids as body with operation Put instead of Delete Bugs: ATLAS-3747 https://issues.apache.org/jira/browse/ATLAS-3747 Repository: atlas Description --- ATLAS-3747: Atlas Admin Purge API should take list of guids as query param instead of body Diffs - client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 5a19a0b0e webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java a7d554a36 Diff: https://reviews.apache.org/r/72411/diff/1/ Testing (updated) --- Manual Testing: curl -u admin:admin123 --location --request PUT 'http://sid-ycloud-atlas2-1.sid-ycloud-atlas2.root.hwx.site:31000/api/atlas/admin/purge/' \ --header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \ --header 'Cookie: ATLASSESSIONID=node0g7w7rpfe6o0a1e4e7hk5cth4d2.node0' \ --data-raw '["bd7262f6-fed5-40f3-9f72-f57fba6b1193", "24c4df6e-b27a-4c60-b85a-67ed20210446", "24c4df6e-b27a-4c60-b85a-67ed20210446"]' Note: As local jetty was failing for couple of times, I was not able to run EntityV2JerseyResourceIT. precommit run will ensure if this passes. Thanks, Sidharth Mishra
Re: Review Request 72411: ATLAS-3747: Atlas Admin Purge API should take list of guids as body with operation Put instead of Delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72411/ --- (Updated April 21, 2020, 9:45 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3747 https://issues.apache.org/jira/browse/ATLAS-3747 Repository: atlas Description --- ATLAS-3747: Atlas Admin Purge API should take list of guids as query param instead of body Diffs (updated) - client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 5a19a0b0e webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java a7d554a36 Diff: https://reviews.apache.org/r/72411/diff/2/ Changes: https://reviews.apache.org/r/72411/diff/1-2/ Testing --- Manual Testing: curl -u admin:admin123 --location --request PUT 'http://sid-ycloud-atlas2-1.sid-ycloud-atlas2.root.hwx.site:31000/api/atlas/admin/purge/' \ --header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \ --header 'Cookie: ATLASSESSIONID=node0g7w7rpfe6o0a1e4e7hk5cth4d2.node0' \ --data-raw '["bd7262f6-fed5-40f3-9f72-f57fba6b1193", "24c4df6e-b27a-4c60-b85a-67ed20210446", "24c4df6e-b27a-4c60-b85a-67ed20210446"]' Note: As local jetty was failing for couple of times, I was not able to run EntityV2JerseyResourceIT. precommit run will ensure if this passes. Thanks, Sidharth Mishra
Review Request 72411: ATLAS-3747: Atlas Admin Purge API should take list of guids as query param instead of body
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72411/ --- Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3747 https://issues.apache.org/jira/browse/ATLAS-3747 Repository: atlas Description --- ATLAS-3747: Atlas Admin Purge API should take list of guids as query param instead of body Diffs - client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 5a19a0b0e webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java a7d554a36 Diff: https://reviews.apache.org/r/72411/diff/1/ Testing --- Manual Testing: curl -u admin:admin123 --location --request DELETE 'http://sid-ycloud-atlas2-1.sid-ycloud-atlas2.root.hwx.site:31000/api/atlas/admin/purge/?guid=82785128-778a-4e0d-aac0-8327b4ca1e87=f0cc9cfd-71af-47fc-9321-2970dfa5825f=f0cc9cfd-71af-47fc-9321-2970dfa5825f' --header 'Content-Type: application/json' Note: As local jetty was failing for couple of times, I was not able to run EntityV2JerseyResourceIT. precommit run will ensure if this passes. Thanks, Sidharth Mishra
Re: Review Request 72367: ATLAS-3709: fix to address skipped index initialization in earlier commit
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72367/#review220328 --- Ship it! Ship It! - Sidharth Mishra On April 15, 2020, 5:22 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72367/ > --- > > (Updated April 15, 2020, 5:22 p.m.) > > > Review request for atlas, Ashutosh Mestry, Mandar Ambawane, Mehul Parikh, > Nixon Rodrigues, Pinal Shah, Sarath Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3709 > https://issues.apache.org/jira/browse/ATLAS-3709 > > > Repository: atlas > > > Description > --- > > With the changes in earlier commit for ATLAS-3709, > notifyInitializationCompletion() was called without any entity-types which > resulted in index not getting initiaized. This issue is fixed in this patch. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > f505bee0a > > repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java > 0670d5000 > > > Diff: https://reviews.apache.org/r/72367/diff/1/ > > > Testing > --- > > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1810/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 72257: ATLAS-3583 Use Audit framework to generate audit entries for Classification CREATE, UPDATE and DELETE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72257/#review220322 --- repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AuditChangeListener.java Lines 71 (patched) <https://reviews.apache.org/r/72257/#comment308655> To get more details for these audit records, we need to change code simlar to the patch which I have - https://reviews.apache.org/r/72351/ (Look for AtlasAuditEntry.java changes. toEntityAuditActionV2() should return proper state which can be used to get more details from entity audit - HBase) - Sidharth Mishra On April 7, 2020, 6:22 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72257/ > --- > > (Updated April 7, 2020, 6:22 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3583 > https://issues.apache.org/jira/browse/ATLAS-3583 > > > Repository: atlas > > > Description > --- > > Added listener to listen create/update/delete operations of all type defs. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 351e07b > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > f2a8b3e > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java > 79f5270 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AuditChangeListener.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/72257/diff/1/ > > > Testing > --- > > Pre-commit build: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1747/console > > Manual testing is done. > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 72351: ATLAS-3727: Add get result details REST for admin audit record
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72351/
---
(Updated April 14, 2020, 12:46 a.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
Bugs: ATLAS-3727
https://issues.apache.org/jira/browse/ATLAS-3727
Repository: atlas
Description
---
ATLAS-3727: Add get result details REST for admin audit record
ATLAS-3729: Add audit operation as a parameter to v2/entity/{guid}/audit
Diffs (updated)
-
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
98e56f2bb
intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
bcfdd9466
repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
0deb7295e
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
35bf1d07e
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
b8131bd39
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
4c1e1a959
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
2a47e39ed
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
021ca43ab
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
ad6ec94c0
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
4bb68d502
repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
aa175a229
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
2ab8bdb03
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 402a323dd
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java
43b85efc0
Diff: https://reviews.apache.org/r/72351/diff/3/
Changes: https://reviews.apache.org/r/72351/diff/2-3/
Testing
---
Manual Testing:
curl -u admin:admin123 --location --request GET
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/audit/706e6e32-b619-43f2-8587-f2877ee27582/details?limit=100=0'
--header 'Content-Type: application/json'
curl -u admin:admin123 --location --request GET
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/0b10d06b-6740-4550-a571-5be8ba1ea3ae/audit?count=1=0b10d06b-6740-4550-a571-5be8ba1ea3ae:1586797534411:0:1586797535131=ENTITY_UPDATE'
curl -u admin:admin123 --location --request GET
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/0b10d06b-6740-4550-a571-5be8ba1ea3ae/audit?count=1=ENTITY_UPDATE'
Thanks,
Sidharth Mishra
Review Request 72358: ATLAS-3730: In case Hard delete and Purge of an Entity we should make the entity status as purged instead of deleted
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72358/
---
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
Bugs: ATLAS-3730
https://issues.apache.org/jira/browse/ATLAS-3730
Repository: atlas
Description
---
ATLAS-3730: Changed code to make entity status to PURGED at classification in
case of Hard delete or Purge of an entity
Diffs
-
intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java c534fd06f
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
7b438600d
Diff: https://reviews.apache.org/r/72358/diff/1/
Testing
---
1. Before Test when we do get we can see the classification contains entity
status as DELETED.
curl -u admin:admin123 --location --request GET
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/guid/b7d5d07c-fbaf-4a10-9c2e-c360d4ac0828?minExtInfo=true&_=1586810974769'
"classifications": [
{
"typeName": "TEST_TAG_2",
"entityGuid": "b7d5d07c-fbaf-4a10-9c2e-c360d4ac0828",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "TEST_TAG_1",
"entityGuid": "7d5dfc2d-bcb9-429a-b241-a34705777b6b",
"entityStatus": "DELETED",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "TEST_TAG_1",
"entityGuid": "a2d1cad9-d2a1-4fe4-8570-2e58e16877ca",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
2. Purge the entity:
curl -u admin:admin123 --location --request DELETE
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/purge/'
\
--header 'Content-Type: application/json' \
--data-raw ' ["7d5dfc2d-bcb9-429a-b241-a34705777b6b"]'
3. After purge of entity - 7d5dfc2d-bcb9-429a-b241-a34705777b6b we can see the
entity state changed to PURGED.
curl -u admin:admin123 --location --request GET
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/guid/b7d5d07c-fbaf-4a10-9c2e-c360d4ac0828?minExtInfo=true&_=1586810974769'
"classifications": [
{
"typeName": "TEST_TAG_2",
"entityGuid": "b7d5d07c-fbaf-4a10-9c2e-c360d4ac0828",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "TEST_TAG_1",
"entityGuid": "7d5dfc2d-bcb9-429a-b241-a34705777b6b",
"entityStatus": "PURGED",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
},
{
"typeName": "TEST_TAG_1",
"entityGuid": "a2d1cad9-d2a1-4fe4-8570-2e58e16877ca",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false
}
]
Thanks,
Sidharth Mishra
Re: Review Request 72351: ATLAS-3727: Add get result details REST for admin audit record
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72351/
---
(Updated April 13, 2020, 6:17 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
Bugs: ATLAS-3727
https://issues.apache.org/jira/browse/ATLAS-3727
Repository: atlas
Description (updated)
---
ATLAS-3727: Add get result details REST for admin audit record
ATLAS-3729: Add audit operation as a parameter to v2/entity/{guid}/audit
Diffs
-
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
98e56f2bb
intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
bcfdd9466
repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
0deb7295e
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
35bf1d07e
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
b8131bd39
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
4c1e1a959
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
2a47e39ed
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
021ca43ab
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
ad6ec94c0
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
4bb68d502
repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
aa175a229
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
2ab8bdb03
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 402a323dd
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java
43b85efc0
Diff: https://reviews.apache.org/r/72351/diff/2/
Testing (updated)
---
Manual Testing:
curl -u admin:admin123 --location --request GET
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/audit/706e6e32-b619-43f2-8587-f2877ee27582/details?limit=100=0'
--header 'Content-Type: application/json'
curl -u admin:admin123 --location --request GET
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/0b10d06b-6740-4550-a571-5be8ba1ea3ae/audit?count=1=0b10d06b-6740-4550-a571-5be8ba1ea3ae:1586797534411:0:1586797535131=ENTITY_UPDATE'
curl -u admin:admin123 --location --request GET
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/0b10d06b-6740-4550-a571-5be8ba1ea3ae/audit?count=1=ENTITY_UPDATE'
Thanks,
Sidharth Mishra
Re: Review Request 72351: ATLAS-3727: Add get result details REST for admin audit record
> On April 11, 2020, 1:35 a.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > > Lines 231 (patched) > > <https://reviews.apache.org/r/72351/diff/1/?file=2217432#file2217432line231> > > > > caching => limit I think this parameter is causing confusion and its been used for scan.setCaching(MAXPAYLOAD) which will increase the payload size to MAXPAYLOAD rows per remote call. I am removing this from function parameter as this will be always to a const value. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72351/#review220287 --- On April 13, 2020, 6:15 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72351/ > --- > > (Updated April 13, 2020, 6:15 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3727 > https://issues.apache.org/jira/browse/ATLAS-3727 > > > Repository: atlas > > > Description > --- > > ATLAS-3727: Add get result details REST for admin audit record > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > 98e56f2bb > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > bcfdd9466 > > repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java > 0deb7295e > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > 35bf1d07e > > repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java > b8131bd39 > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 4c1e1a959 > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java > 2a47e39ed > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > 021ca43ab > > repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java > ad6ec94c0 > > repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java > 4bb68d502 > > repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java > aa175a229 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 2ab8bdb03 > webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 402a323dd > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > 43b85efc0 > > > Diff: https://reviews.apache.org/r/72351/diff/2/ > > > Testing > --- > > Manual Testing: > > curl -u admin:admin123 --location --request GET > 'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/audit/706e6e32-b619-43f2-8587-f2877ee27582/details?limit=100=0' > --header 'Content-Type: application/json' > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 72351: ATLAS-3727: Add get result details REST for admin audit record
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72351/ --- (Updated April 13, 2020, 6:15 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3727 https://issues.apache.org/jira/browse/ATLAS-3727 Repository: atlas Description --- ATLAS-3727: Add get result details REST for admin audit record Diffs (updated) - intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 98e56f2bb intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java bcfdd9466 repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java 0deb7295e repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java 35bf1d07e repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java b8131bd39 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 4c1e1a959 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java 2a47e39ed repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 021ca43ab repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java ad6ec94c0 repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java 4bb68d502 repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java aa175a229 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 2ab8bdb03 webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 402a323dd webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 43b85efc0 Diff: https://reviews.apache.org/r/72351/diff/2/ Changes: https://reviews.apache.org/r/72351/diff/1-2/ Testing --- Manual Testing: curl -u admin:admin123 --location --request GET 'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/audit/706e6e32-b619-43f2-8587-f2877ee27582/details?limit=100=0' --header 'Content-Type: application/json' Thanks, Sidharth Mishra
Re: Review Request 72351: ATLAS-3727: Add get result details REST for admin audit record
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72351/ --- (Updated April 10, 2020, 10:44 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3727 https://issues.apache.org/jira/browse/ATLAS-3727 Repository: atlas Description --- ATLAS-3727: Add get result details REST for admin audit record Diffs - intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 98e56f2bb repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java 35bf1d07e repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java cab4e1e55 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java 2a47e39ed repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 021ca43ab webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 2ab8bdb03 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 43b85efc0 Diff: https://reviews.apache.org/r/72351/diff/1/ Testing (updated) --- Manual Testing: curl -u admin:admin123 --location --request GET 'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/audit/706e6e32-b619-43f2-8587-f2877ee27582/details?limit=100=0' --header 'Content-Type: application/json' Thanks, Sidharth Mishra
Review Request 72351: ATLAS-3727: Add get result details REST for admin audit record
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72351/ --- Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3727 https://issues.apache.org/jira/browse/ATLAS-3727 Repository: atlas Description --- ATLAS-3727: Add get result details REST for admin audit record Diffs - intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 98e56f2bb repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java 35bf1d07e repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java cab4e1e55 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java 2a47e39ed repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 021ca43ab webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 2ab8bdb03 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 43b85efc0 Diff: https://reviews.apache.org/r/72351/diff/1/ Testing --- Manual Testing: curl --location --request GET 'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/audit/706e6e32-b619-43f2-8587-f2877ee27582/details?limit=100=0' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' Thanks, Sidharth Mishra
Re: Review Request 72338: ATLAS-3726: Add result count to Atlas Admin Audit and enable filters on count for get audit
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72338/
---
(Updated April 9, 2020, 6:26 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
Bugs: ATLAS-3726
https://issues.apache.org/jira/browse/ATLAS-3726
Repository: atlas
Description
---
Added a new field for result count at model, Changed the DTO and service
accordingly. UT has been modified too. Now sort and filter will also work on
top of result count in admin audit
Diffs (updated)
-
addons/models/-Area0/0010-base_model.json 351e07bd3
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
f2a8b3e4d
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
590f7a034
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
8e4bdb523
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
0a2fc39a3
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
9bc70400e
Diff: https://reviews.apache.org/r/72338/diff/2/
Changes: https://reviews.apache.org/r/72338/diff/1-2/
Testing
---
Manual Testing:
curl --location --request POST
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/audits/'
\
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
--data-raw '{
"auditFilters": {
"condition": "AND",
"criterion": [
{
"attributeName": "userName",
"operator": "like",
"attributeValue": "admin"
},
{
"attributeName": "operation",
"operator": "like",
"attributeValue": "PURGE"
},
{
"attributeName": "clientId",
"operator": "eq",
"attributeValue": "10.96.94.236"
},
{
"attributeName": "startTime",
"operator": "gte",
"attributeValue": "1575958152162"
},
{
"attributeName": "endTime",
"operator": "gte",
"attributeValue": "1575958152184"
},
{
"attributeName": "resultCount",
"operator": "eq",
"attributeValue": "4"
}
]
},
"limit": 15,
"offset": 0,
"sortBy": "resultCount",
"sortOrder": "DESCENDING"
}'
Thanks,
Sidharth Mishra
Review Request 72338: ATLAS-3726: Add result count to Atlas Admin Audit and enable filters on count for get audit
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72338/
---
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
Bugs: ATLAS-3726
https://issues.apache.org/jira/browse/ATLAS-3726
Repository: atlas
Description
---
Added a new field for result count at model, Changed the DTO and service
accordingly. UT has been modified too. Now sort and filter will also work on
top of result count in admin audit
Diffs
-
addons/models/-Area0/0010-base_model.json 351e07bd3
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
f2a8b3e4d
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
590f7a034
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
8e4bdb523
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
0a2fc39a3
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
9bc70400e
Diff: https://reviews.apache.org/r/72338/diff/1/
Testing
---
Manual Testing:
curl --location --request POST
'http://sid-ycloud-atlas1-1.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/admin/audits/'
\
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
--data-raw '{
"auditFilters": {
"condition": "AND",
"criterion": [
{
"attributeName": "userName",
"operator": "like",
"attributeValue": "admin"
},
{
"attributeName": "operation",
"operator": "like",
"attributeValue": "PURGE"
},
{
"attributeName": "clientId",
"operator": "eq",
"attributeValue": "10.96.94.236"
},
{
"attributeName": "startTime",
"operator": "gte",
"attributeValue": "1575958152162"
},
{
"attributeName": "endTime",
"operator": "gte",
"attributeValue": "1575958152184"
},
{
"attributeName": "resultCount",
"operator": "eq",
"attributeValue": "4"
}
]
},
"limit": 15,
"offset": 0,
"sortBy": "resultCount",
"sortOrder": "DESCENDING"
}'
Thanks,
Sidharth Mishra
Re: Review Request 72317: ATLAS-3709 Issues with quick search/suggestions in conjunction with Business Metadata attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72317/#review220236 --- Ship it! Ship It! - Sidharth Mishra On April 7, 2020, 6:20 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72317/ > --- > > (Updated April 7, 2020, 6:20 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3709 > https://issues.apache.org/jira/browse/ATLAS-3709 > > > Repository: atlas > > > Description > --- > > Resolved issues in conjunction with Business Metadata attributes. > Also handled index populate for Business Metadata attributes at the time of > start up. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/listener/ChangedTypeDefs.java 58e889a > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 9df89a2 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 647e304 > > > Diff: https://reviews.apache.org/r/72317/diff/1/ > > > Testing > --- > > Pre-commit > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1784/console > > Manual testing done. > > CASE:1 > 1. create Busines meatadata "BM1" > 2. add one attribute "attr1" in "BM1" (search weight 9) > 3. create one entity "hdfsentity1" > 4. associate "attr1" to "hdfsentity1" > 5. check in quick search and suggestions for "attr1" value > > 6. add another attribute "attr2" in "BM1" (search weight 9) > 7. associate "attr2" to "hdfsentity1" > 8. check in quick search and suggestions > > > CASE:2 > 1. create Busines meatadata "BM1" > 2. add one attribute "attr1" in "BM1" (search weight 9) > 3. create one entity "hdfsentity1" > 4. associate "attr1" to "hdfsentity1" > 5. check in quick search and suggestions for "attr1" value > > 6. add another attribute "attr2" in "BM1" (search weight 10) > 7. associate "attr2" to "hdfsentity1" > 8. check in quick search and suggestions for "attr1" and "attr2" values > > > CASE:3 > 1. create Busines meatadata "BM1" > 2. add one attribute "attr1" in "BM1" (search weight 9) > 3. create one entity "hdfsentity1" > 4. associate "attr1" to "hdfsentity1" > 5. check in quick search and suggestions for "attr1" value > > 6. create another Busines meatadata "BM2" > 7. add attribute "attr2" in "BM2" (search weight 9) > 8. associate "attr2" to "hdfsentity1" > 9. check in quick search and suggestions for "attr1" and "attr2" values > > > Thanks, > > Mandar Ambawane > >
Review Request 72328: ATLAS-3719: Tag and Type search is not doing index search
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72328/
---
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
Bugs: ATLAS-3719
https://issues.apache.org/jira/browse/ATLAS-3719
Repository: atlas
Description
---
Changed the condition to add the classification processor in case of
Classification and Type
Diffs
-
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
2c2888e16
Diff: https://reviews.apache.org/r/72328/diff/1/
Testing
---
Manual testing -
curl --location --request POST
'http://sid-cdp-1-1.gce.cloudera.com:31000/api/atlas/v2/search/basic' \
--header 'Content-Type: application/json' \
--data-raw
'{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":100,"offset":0,"typeName":"hive_column","classification":"secu_rgpd_dcpc","termName":null}'
curl --location --request POST
'http://sid-cdp-1-1.gce.cloudera.com:31000/api/atlas/v2/search/basic' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw
'{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"limit":25,"offset":0,"typeName":null,"classification":"BEFORE_DEMO","termName":null}'
Thanks,
Sidharth Mishra
Re: Review Request 72284: ATLAS-3707: Atlas Purge entity is generating both purge and delete audit
> On March 30, 2020, 7:25 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> > Lines 325 (patched)
> > <https://reviews.apache.org/r/72284/diff/1/?file=2216465#file2216465line325>
> >
> > In case of purge, shouldn't these entities be not added to resp as
> > PURGEed entities? Like:
> >
> > EntityOperation deleteOperation = req.isPurgeRequested() ? PURGE :
> > DELETE;
> >
> > for (AtlasEntityHeader entity : req.getDeletedEntities()) {
> > resp.addEntity(deleteOperation, entity);
> > }
> >
> > Alternatively, consider adding RequestContext.purgedEntities and add to
> > this collection (instead of deletedEntities) while performing purge - from
> > DeleteHandlerV1.deleteEntities():
> >
> > for (GraphHelper.VertexInfo vertexInfo :
> > getOwnedVertices(instanceVertex)) {
> > if (requestContext.isPurgeRequested()) {
> > requestContext.recordEntityPurge(vertexInfo.getEntity());
> > } else {
> > requestContext.recordEntityDelete(vertexInfo.getEntity());
> > }
> >
> > deletionCandidateVertices.add(vertexInfo.getVertex());
> > }
At org/apache/atlas/web/resources/AdminResource.java:473,
EntityMutationResponse resp = entityStore.purgeByIds(guids); keeps the deleted
entities in requestContext and we get the same and assign it to response as you
correctly pointed above.
After that we again create atlas audit entity at
org/apache/atlas/web/resources/AdminResource.java:480 which eventually calls
org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2#createOrUpdate(org.apache.atlas.repository.store.graph.v2.EntityStream,
boolean, boolean, boolean) where we see if there are deleted entities at
request context then we generate the audit for those which we need to stop by
adding the check. To keep the hard delete and purge code same we are still
using the requestContext deletedEntities instead of having separate one for
purge.
- Sidharth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72284/#review220121
---
On March 30, 2020, 7:03 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72284/
> ---
>
> (Updated March 30, 2020, 7:03 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-3707
> https://issues.apache.org/jira/browse/ATLAS-3707
>
>
> Repository: atlas
>
>
> Description
> ---
>
> The check is been added, so that when its purge operation there will not be
> any entity delete notification sent to audit
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> 75b016cca
>
>
> Diff: https://reviews.apache.org/r/72284/diff/1/
>
>
> Testing
> ---
>
> Manual testing.
>
> curl --location --request GET
> 'http://sid-cdp-3-1.gce.cloudera.com:31000/api/atlas/v2/entity/d7d29f3e-1ef6-4baa-b66c-3f20d2839036/audit'
> \
> --header 'Authorization: Basic YWRtaW46YWRtaW4='
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 72265: ATLAS-3684: Atlas glossary get is taking more time when number of terms increase
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72265/ --- (Updated March 30, 2020, 7:05 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3684 https://issues.apache.org/jira/browse/ATLAS-3684 Repository: atlas Description --- ATLAS-3684: Atlas glossary get is taking more time when number of terms increase Diffs - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java d630f669e repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AbstractGlossaryDTO.java bee88c61f Diff: https://reviews.apache.org/r/72265/diff/1/ Testing --- Manually tested Thanks, Sidharth Mishra
Review Request 72284: ATLAS-3707: Atlas Purge entity is generating both purge and delete audit
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72284/ --- Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3707 https://issues.apache.org/jira/browse/ATLAS-3707 Repository: atlas Description --- The check is been added, so that when its purge operation there will not be any entity delete notification sent to audit Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 75b016cca Diff: https://reviews.apache.org/r/72284/diff/1/ Testing --- Manual testing. curl --location --request GET 'http://sid-cdp-3-1.gce.cloudera.com:31000/api/atlas/v2/entity/d7d29f3e-1ef6-4baa-b66c-3f20d2839036/audit' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' Thanks, Sidharth Mishra
Re: Review Request 72268: ATLAS-3688: updated dependencies to remove unnecessary references
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72268/#review220081 --- Ship it! Ship It! - Sidharth Mishra On March 25, 2020, 7:35 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72268/ > --- > > (Updated March 25, 2020, 7:35 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth > Mishra. > > > Bugs: ATLAS-3688 > https://issues.apache.org/jira/browse/ATLAS-3688 > > > Repository: atlas > > > Description > --- > > Unnecessary references to hdfs-model, hbase-common and hbase-server were > removed from modules > > > Diffs > - > > addons/falcon-bridge/pom.xml c53d0ef8b > addons/hbase-bridge/pom.xml ff88c8b1e > addons/hive-bridge/pom.xml 13384cb72 > addons/impala-bridge/pom.xml a684dd03d > addons/sqoop-bridge/pom.xml 0d89e98dc > addons/storm-bridge/pom.xml c282625bc > pom.xml 7e9c7e06a > tools/classification-updater/pom.xml 4adfb3d04 > > > Diff: https://reviews.apache.org/r/72268/diff/1/ > > > Testing > --- > > Pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1758/ > > > Thanks, > > Madhan Neethiraj > >
Review Request 72265: ATLAS-3684: Atlas glossary get is taking more time when number of terms increase
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72265/ --- Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Repository: atlas Description --- ATLAS-3684: Atlas glossary get is taking more time when number of terms increase Diffs - repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java d630f669e repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AbstractGlossaryDTO.java bee88c61f Diff: https://reviews.apache.org/r/72265/diff/1/ Testing --- Manually tested Thanks, Sidharth Mishra
Re: Review Request 71482: ATLAS-3423:-Import Glossary Terms CSV into a Glossary
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71482/#review220048
---
Ship it!
Ship It!
- Sidharth Mishra
On March 20, 2020, 2:06 p.m., mayank jain wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71482/
> ---
>
> (Updated March 20, 2020, 2:06 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, keval bhatt, Madhan Neethiraj,
> Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3423
> https://issues.apache.org/jira/browse/ATLAS-3423
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch consists implementation for 2 end points first for template
> download and other for csv file upload with term details also the Unit Test
> cases for both the end points.
>
> * The 1st endpoint {glossary/template} return template file this would be
> type of format of data that shows how the data needs to be populated by user
> in the file.
>
> http://localhost:21000/api/atlas/v2/glossary/import/template
>
> Template structure:-
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> Fruits,Apple5,SD4,LD4,"EXAMPLE","ABBREVIATION","USAGE",,"Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4"
>
>
> * The 2nd endpoint {glossary/importGlossaryData} (file upload) would actually
> parse the Data into AtlasObjects and further create the AtlasGlossaryTerms
> inside Glossary.
>
> curl -v -g POST -u admin:admin -H "Content-Type: multipart/form-data" -H
> "Cache-Control: no-cache" -F file=@template_6.csv
> "http://localhost:21000/api/atlas/v2/glossary/import
>
>
> Note:-
>
> While populating the data in the csv file each record should be maintained
> in single Line (enter command within the record would result in parsing the
> second line as a new record).
>
> The downloaded template needs to be saved as whateverTheFileNameIs.csv
> explicitly.
>
> If the file is been succefully uploaded then the AtlasGlossaryTerm would be
> returned or else List of Errors would returned for user to rectify them
> further.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 7c0fd56
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 1bd2fd0
>
> intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasGlossaryHeader.java
> 660514b
> pom.xml a6da9b8
> repository/pom.xml 802d587
> repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
> 9229d2d
> repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
> cdc3f07
> repository/src/main/java/org/apache/atlas/glossary/GlossaryUtils.java
> 9625f94
> repository/src/main/java/org/apache/atlas/util/FileUtils.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java
> 759dcdf
> repository/src/test/resources/csvFiles/empty.csv PRE-CREATION
> repository/src/test/resources/csvFiles/incorrectFile.csv PRE-CREATION
> repository/src/test/resources/csvFiles/template_1.csv PRE-CREATION
> repository/src/test/resources/excelFiles/template_1.xlsx PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/GlossaryREST.java 151aa6b
>
>
> Diff: https://reviews.apache.org/r/71482/diff/20/
>
>
> Testing
> ---
>
> Tested both the endpoint with curl call.
>
> Tested upload term csv with around 100 records from curl.
>
>
> Thanks,
>
> mayank jain
>
>
Re: Review Request 71482: ATLAS-3423:-Import Glossary Terms CSV into a Glossary
> On March 18, 2020, 11:55 p.m., Sidharth Mishra wrote:
> > repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
> > Lines 570 (patched)
> > <https://reviews.apache.org/r/71482/diff/18/?file=2213646#file2213646line570>
> >
> > Please remove these kind of extra blank lines from here as well all
> > other places
I can still see the extra blank lines other places. Kindly let me know if there
is any specific reason you add them?
> On March 18, 2020, 11:55 p.m., Sidharth Mishra wrote:
> > repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
> > Lines 690 (patched)
> > <https://reviews.apache.org/r/71482/diff/18/?file=2213646#file2213646line690>
> >
> > Instead of \n use System.lineSeparator(). Please do this at all other
> > places in entire patch
Please mention the reason for drop.
> On March 18, 2020, 11:55 p.m., Sidharth Mishra wrote:
> > repository/src/main/java/org/apache/atlas/util/FileUtils.java
> > Lines 99 (patched)
> > <https://reviews.apache.org/r/71482/diff/18/?file=2213648#file2213648line99>
> >
> > Try to create a const for 0 explaining what does the position contains
Please mention the reason for drop.
- Sidharth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71482/#review219988
---
On March 19, 2020, 5:38 a.m., mayank jain wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71482/
> ---
>
> (Updated March 19, 2020, 5:38 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, keval bhatt, Madhan Neethiraj,
> Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3423
> https://issues.apache.org/jira/browse/ATLAS-3423
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch consists implementation for 2 end points first for template
> download and other for csv file upload with term details also the Unit Test
> cases for both the end points.
>
> * The 1st endpoint {glossary/template} return template file this would be
> type of format of data that shows how the data needs to be populated by user
> in the file.
>
> http://localhost:21000/api/atlas/v2/glossary/import/template
>
> Template structure:-
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> Fruits,Apple5,SD4,LD4,"EXAMPLE","ABBREVIATION","USAGE",,"Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4"
>
>
> * The 2nd endpoint {glossary/importGlossaryData} (file upload) would actually
> parse the Data into AtlasObjects and further create the AtlasGlossaryTerms
> inside Glossary.
>
> curl -v -g POST -u admin:admin -H "Content-Type: multipart/form-data" -H
> "Cache-Control: no-cache" -F file=@template_6.csv
> "http://localhost:21000/api/atlas/v2/glossary/import
>
>
> Note:-
>
> While populating the data in the csv file each record should be maintained
> in single Line (enter command within the record would result in parsing the
> second line as a new record).
>
> The downloaded template needs to be saved as whateverTheFileNameIs.csv
> explicitly.
>
> If the file is been succefully uploaded then the AtlasGlossaryTerm would be
> returned or else List of Errors would returned for user to rectify them
> further.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 7c0fd56
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 1bd2fd0
>
> intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasGlossaryHeader.java
> 660514b
> pom.xml a6da9b8
> repository/pom.xml 802d587
> repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
> 9229d2d
> repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
> cdc3f07
> repository/src/main/java/org/apache/atlas/glossary/GlossaryUtils.java
> 962
Re: Review Request 71482: ATLAS-3423:-Import Glossary Terms CSV into a Glossary
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71482/#review219988
---
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 570 (patched)
<https://reviews.apache.org/r/71482/#comment308221>
Please remove these kind of extra blank lines from here as well all other
places
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 649 (patched)
<https://reviews.apache.org/r/71482/#comment308223>
Do something like:
Set ret = null;
if (StringUtils.isNotBlank(csvRecord)) {
...
ret = new HashSet();
...
}
return ret;
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 672 (patched)
<https://reviews.apache.org/r/71482/#comment308224>
there is on extra space after system msg and then you are again doing + "
". Please remove one. Same is there at other places. please take care of this
at all other locations
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 682 (patched)
<https://reviews.apache.org/r/71482/#comment308222>
you are already passing atlasGlossaryTerm as input to this function where
you are using set of this object and then retunring the same object
atlasGlossaryTerm and assigning it to same atlasGlossaryTerm at 573. Please
revisit the design and rename the function accordingly. The function should
always do one thing and should be named accordingly. Depending on that you
should design the input and output arguments
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 690 (patched)
<https://reviews.apache.org/r/71482/#comment308225>
Instead of \n use System.lineSeparator(). Please do this at all other
places in entire patch
repository/src/main/java/org/apache/atlas/util/FileUtils.java
Lines 50 (patched)
<https://reviews.apache.org/r/71482/#comment308226>
Please call readCSV function directly instead of using class name as its in
the same class
repository/src/main/java/org/apache/atlas/util/FileUtils.java
Lines 99 (patched)
<https://reviews.apache.org/r/71482/#comment308227>
Try to create a const for 0 explaining what does the position contains
repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java
Lines 960 (patched)
<https://reviews.apache.org/r/71482/#comment308228>
Please remove the extra space:
assertEquals(e.getMessage(), "No Data found in the uploaded file!");
- Sidharth Mishra
On March 13, 2020, 1 p.m., mayank jain wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71482/
> ---
>
> (Updated March 13, 2020, 1 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, keval bhatt, Madhan Neethiraj,
> Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3423
> https://issues.apache.org/jira/browse/ATLAS-3423
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch consists implementation for 2 end points first for template
> download and other for csv file upload with term details also the Unit Test
> cases for both the end points.
>
> * The 1st endpoint {glossary/template} return template file this would be
> type of format of data that shows how the data needs to be populated by user
> in the file.
>
> http://localhost:21000/api/atlas/v2/glossary/importHeaderRow
>
> Template structure:-
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> Fruits,Apple5,SD4,LD4,"EXAMPLE","ABBREVIATION","USAGE",,"Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4"
>
>
> * The 2nd endpoint {glossary/importGlossaryData} (file upload) would actually
> parse the Data into AtlasObjects and further create the AtlasGlossaryTerms
> inside Glossary.
>
> curl -v -g POST -u admin:admin -H "Content-Type: multipart/form-data" -H
> "Cache-Control: no-cache" -F file=@template_6.csv
> "http://localhost:21000/api/a
Re: Review Request 71482: ATLAS-3423:-Import Glossary Terms CSV into a Glossary
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71482/#review219912
---
repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 1124 (patched)
<https://reviews.apache.org/r/71482/#comment308149>
Remove extra blank line
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 549 (patched)
<https://reviews.apache.org/r/71482/#comment308151>
Please remove the extra blank lines like this even other places.
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 558 (patched)
<https://reviews.apache.org/r/71482/#comment308150>
Remove the extra blank line
- Sidharth Mishra
On March 12, 2020, 10:37 a.m., mayank jain wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71482/
> ---
>
> (Updated March 12, 2020, 10:37 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, keval bhatt, Madhan Neethiraj,
> Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3423
> https://issues.apache.org/jira/browse/ATLAS-3423
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch consists implementation for 2 end points first for template
> download and other for csv file upload with term details also the Unit Test
> cases for both the end points.
>
> * The 1st endpoint {glossary/template} return template file this would be
> type of format of data that shows how the data needs to be populated by user
> in the file.
>
> http://localhost:21000/api/atlas/v2/glossary/importHeaderRow
>
> Template structure:-
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> Fruits,Apple5,SD4,LD4,"EXAMPLE","ABBREVIATION","USAGE",,"Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4"
>
>
> * The 2nd endpoint {glossary/importGlossaryData} (file upload) would actually
> parse the Data into AtlasObjects and further create the AtlasGlossaryTerms
> inside Glossary.
>
> curl -v -g POST -u admin:admin -H "Content-Type: multipart/form-data" -H
> "Cache-Control: no-cache" -F file=@template_6.csv
> "http://localhost:21000/api/atlas/v2/glossary/import
>
>
> Note:-
>
> While populating the data in the csv file each record should be maintained
> in single Line (enter command within the record would result in parsing the
> second line as a new record).
>
> The downloaded template needs to be saved as whateverTheFileNameIs.csv
> explicitly.
>
> If the file is been succefully uploaded then the AtlasGlossaryTerm would be
> returned or else List of Errors would returned for user to rectify them
> further.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 7c0fd56
> dashboardv2/gruntfile.js fef4e08
> dashboardv2/package-lock.json 7f25b57
> dashboardv2/package.json e90040e
> dashboardv2/public/css/scss/theme.scss aad0dfa
> dashboardv2/public/index.html.tpl 7cd86bd
> dashboardv2/public/js/main.js 41af465
> dashboardv2/public/js/templates/glossary/GlossaryLayoutView_tmpl.html
> 1fa1e35
> dashboardv2/public/js/templates/glossary/ImportGlossaryLayoutView_tmpl.html
> PRE-CREATION
> dashboardv2/public/js/utils/UrlLinks.js 6c67e8c
> dashboardv2/public/js/views/glossary/GlossaryLayoutView.js 9b386f3
> dashboardv2/public/js/views/glossary/ImportGlossaryLayoutView.js
> PRE-CREATION
> dashboardv3/gruntfile.js f55ff0d
> dashboardv3/package-lock.json 3918ecc
> dashboardv3/package.json 5dc0510
> dashboardv3/public/css/scss/leftsidebar.scss bbdc5fb
> dashboardv3/public/css/scss/theme.scss 2b0c45d
> dashboardv3/public/index.html.tpl 1baeb12
> dashboardv3/public/js/main.js 96ef7ae
> dashboardv3/public/js/templates/glossary/ImportGlossaryLayoutView_tmpl.html
> PRE-CREATION
>
> dashboardv3/public/js/templates/search/tree/GlossaryTreeLayoutView_tmpl.html
> 83da9
Re: Review Request 71482: ATLAS-3423:-Import Glossary Terms CSV into a Glossary
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71482/#review219788
---
- Sidharth Mishra
On March 4, 2020, 10:07 a.m., mayank jain wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71482/
> ---
>
> (Updated March 4, 2020, 10:07 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues,
> and Sarath Subramanian.
>
>
> Bugs: ATLAS-3423
> https://issues.apache.org/jira/browse/ATLAS-3423
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch consists implementation for 2 end points first for template
> download and other for csv file upload with term details also the Unit Test
> cases for both the end points.
>
> * The 1st endpoint {glossary/template} return template file this would be
> type of format of data that shows how the data needs to be populated by user
> in the file.
>
> http://localhost:21000/api/atlas/v2/glossary/importHeaderRow
>
> Template structure:-
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> Fruits,Apple5,SD4,LD4,"EXAMPLE","ABBREVIATION","USAGE",,"Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4"
>
>
> * The 2nd endpoint {glossary/importGlossaryData} (file upload) would actually
> parse the Data into AtlasObjects and further create the AtlasGlossaryTerms
> inside Glossary.
>
> curl -v -g POST -u admin:admin -H "Content-Type: multipart/form-data" -H
> "Cache-Control: no-cache" -F file=@template_6.csv
> "http://localhost:21000/api/atlas/v2/glossary/import
>
>
> Note:-
>
> While populating the data in the csv file each record should be maintained
> in single Line (enter command within the record would result in parsing the
> second line as a new record).
>
> The downloaded template needs to be saved as whateverTheFileNameIs.csv
> explicitly.
>
> If the file is been succefully uploaded then the AtlasGlossaryTerm would be
> returned or else List of Errors would returned for user to rectify them
> further.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 7c0fd5601
> dashboardv2/gruntfile.js fef4e08c3
> dashboardv2/package-lock.json 7f25b5752
> dashboardv2/package.json e90040edb
> dashboardv2/public/css/scss/theme.scss 0589e0920
> dashboardv2/public/index.html.tpl a6a999e53
> dashboardv2/public/js/main.js 75e16c3aa
> dashboardv2/public/js/templates/glossary/GlossaryLayoutView_tmpl.html
> 1fa1e3540
> dashboardv2/public/js/templates/glossary/ImportGlossaryLayoutView_tmpl.html
> PRE-CREATION
> dashboardv2/public/js/utils/UrlLinks.js 6c67e8c37
> dashboardv2/public/js/views/glossary/GlossaryLayoutView.js 9b386f326
> dashboardv2/public/js/views/glossary/ImportGlossaryLayoutView.js
> PRE-CREATION
> dashboardv3/gruntfile.js f55ff0d5e
> dashboardv3/package-lock.json 3918eccaa
> dashboardv3/package.json 5dc05104f
> dashboardv3/public/css/scss/leftsidebar.scss bbdc5fb26
> dashboardv3/public/css/scss/theme.scss 2b0c45d6b
> dashboardv3/public/index.html.tpl 2edbb659d
> dashboardv3/public/js/main.js 26fd70991
> dashboardv3/public/js/templates/glossary/ImportGlossaryLayoutView_tmpl.html
> PRE-CREATION
>
> dashboardv3/public/js/templates/search/tree/GlossaryTreeLayoutView_tmpl.html
> 83da9c57c
> dashboardv3/public/js/utils/UrlLinks.js 2bbe6796f
> dashboardv3/public/js/views/glossary/ImportGlossaryLayoutView.js
> PRE-CREATION
> dashboardv3/public/js/views/search/tree/GlossaryTreeLayoutView.js 28c6a9e4a
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java c5bf50dca
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 04eb4a08e
>
> intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasGlossaryHeader.java
> 660514bc2
> pom.xml f76c6a05e
> repository/pom.xml 802d587a8
>
Re: Review Request 71482: ATLAS-3423:-Import Glossary Terms CSV into a Glossary
id out argument
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 676 (patched)
<https://reviews.apache.org/r/71482/#comment308010>
Please take care of indentation at most of the places in your patchl. Like
space before and after '+' etc.
- Sidharth Mishra
On March 4, 2020, 10:07 a.m., mayank jain wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71482/
> ---
>
> (Updated March 4, 2020, 10:07 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues,
> and Sarath Subramanian.
>
>
> Bugs: ATLAS-3423
> https://issues.apache.org/jira/browse/ATLAS-3423
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch consists implementation for 2 end points first for template
> download and other for csv file upload with term details also the Unit Test
> cases for both the end points.
>
> * The 1st endpoint {glossary/template} return template file this would be
> type of format of data that shows how the data needs to be populated by user
> in the file.
>
> http://localhost:21000/api/atlas/v2/glossary/importHeaderRow
>
> Template structure:-
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> Fruits,Apple5,SD4,LD4,"EXAMPLE","ABBREVIATION","USAGE",,"Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4"
>
>
> * The 2nd endpoint {glossary/importGlossaryData} (file upload) would actually
> parse the Data into AtlasObjects and further create the AtlasGlossaryTerms
> inside Glossary.
>
> curl -v -g POST -u admin:admin -H "Content-Type: multipart/form-data" -H
> "Cache-Control: no-cache" -F file=@template_6.csv
> "http://localhost:21000/api/atlas/v2/glossary/import
>
>
> Note:-
>
> While populating the data in the csv file each record should be maintained
> in single Line (enter command within the record would result in parsing the
> second line as a new record).
>
> The downloaded template needs to be saved as whateverTheFileNameIs.csv
> explicitly.
>
> If the file is been succefully uploaded then the AtlasGlossaryTerm would be
> returned or else List of Errors would returned for user to rectify them
> further.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 7c0fd5601
> dashboardv2/gruntfile.js fef4e08c3
> dashboardv2/package-lock.json 7f25b5752
> dashboardv2/package.json e90040edb
> dashboardv2/public/css/scss/theme.scss 0589e0920
> dashboardv2/public/index.html.tpl a6a999e53
> dashboardv2/public/js/main.js 75e16c3aa
> dashboardv2/public/js/templates/glossary/GlossaryLayoutView_tmpl.html
> 1fa1e3540
> dashboardv2/public/js/templates/glossary/ImportGlossaryLayoutView_tmpl.html
> PRE-CREATION
> dashboardv2/public/js/utils/UrlLinks.js 6c67e8c37
> dashboardv2/public/js/views/glossary/GlossaryLayoutView.js 9b386f326
> dashboardv2/public/js/views/glossary/ImportGlossaryLayoutView.js
> PRE-CREATION
> dashboardv3/gruntfile.js f55ff0d5e
> dashboardv3/package-lock.json 3918eccaa
> dashboardv3/package.json 5dc05104f
> dashboardv3/public/css/scss/leftsidebar.scss bbdc5fb26
> dashboardv3/public/css/scss/theme.scss 2b0c45d6b
> dashboardv3/public/index.html.tpl 2edbb659d
> dashboardv3/public/js/main.js 26fd70991
> dashboardv3/public/js/templates/glossary/ImportGlossaryLayoutView_tmpl.html
> PRE-CREATION
>
> dashboardv3/public/js/templates/search/tree/GlossaryTreeLayoutView_tmpl.html
> 83da9c57c
> dashboardv3/public/js/utils/UrlLinks.js 2bbe6796f
> dashboardv3/public/js/views/glossary/ImportGlossaryLayoutView.js
> PRE-CREATION
> dashboardv3/public/js/views/search/tree/GlossaryTreeLayoutView.js 28c6a9e4a
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java c5bf50dca
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 04eb4a08e
>
> intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasGlossaryHeader.java
> 660514bc2
> pom.xml f76c6a05e
Re: Review Request 71482: ATLAS-3423:-Import Glossary Terms CSV into a Glossary
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71482/#review219786
---
common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 205 (patched)
<https://reviews.apache.org/r/71482/#comment307986>
Please provide comments which will be useful to get some more context or
remove the comment. In this case the variable name is more informative than the
comment.
common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 207 (patched)
<https://reviews.apache.org/r/71482/#comment307985>
Please consider renaming GlossaryImportSupportedFileFormats to
GlossaryImportSupportedFileExtensions as you have used extensions at other
places and comments
dashboardv2/public/js/views/glossary/ImportGlossaryLayoutView.js
Lines 53 (patched)
<https://reviews.apache.org/r/71482/#comment307987>
This comment is not adding any useful information as the initialize block
is self explanatory.
- Sidharth Mishra
On March 4, 2020, 10:07 a.m., mayank jain wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71482/
> ---
>
> (Updated March 4, 2020, 10:07 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues,
> and Sarath Subramanian.
>
>
> Bugs: ATLAS-3423
> https://issues.apache.org/jira/browse/ATLAS-3423
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch consists implementation for 2 end points first for template
> download and other for csv file upload with term details also the Unit Test
> cases for both the end points.
>
> * The 1st endpoint {glossary/template} return template file this would be
> type of format of data that shows how the data needs to be populated by user
> in the file.
>
> http://localhost:21000/api/atlas/v2/glossary/importHeaderRow
>
> Template structure:-
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> Fruits,Apple5,SD4,LD4,"EXAMPLE","ABBREVIATION","USAGE",,"Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4"
>
>
> * The 2nd endpoint {glossary/importGlossaryData} (file upload) would actually
> parse the Data into AtlasObjects and further create the AtlasGlossaryTerms
> inside Glossary.
>
> curl -v -g POST -u admin:admin -H "Content-Type: multipart/form-data" -H
> "Cache-Control: no-cache" -F file=@template_6.csv
> "http://localhost:21000/api/atlas/v2/glossary/import
>
>
> Note:-
>
> While populating the data in the csv file each record should be maintained
> in single Line (enter command within the record would result in parsing the
> second line as a new record).
>
> The downloaded template needs to be saved as whateverTheFileNameIs.csv
> explicitly.
>
> If the file is been succefully uploaded then the AtlasGlossaryTerm would be
> returned or else List of Errors would returned for user to rectify them
> further.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 7c0fd5601
> dashboardv2/gruntfile.js fef4e08c3
> dashboardv2/package-lock.json 7f25b5752
> dashboardv2/package.json e90040edb
> dashboardv2/public/css/scss/theme.scss 0589e0920
> dashboardv2/public/index.html.tpl a6a999e53
> dashboardv2/public/js/main.js 75e16c3aa
> dashboardv2/public/js/templates/glossary/GlossaryLayoutView_tmpl.html
> 1fa1e3540
> dashboardv2/public/js/templates/glossary/ImportGlossaryLayoutView_tmpl.html
> PRE-CREATION
> dashboardv2/public/js/utils/UrlLinks.js 6c67e8c37
> dashboardv2/public/js/views/glossary/GlossaryLayoutView.js 9b386f326
> dashboardv2/public/js/views/glossary/ImportGlossaryLayoutView.js
> PRE-CREATION
> dashboardv3/gruntfile.js f55ff0d5e
> dashboardv3/package-lock.json 3918eccaa
> dashboardv3/package.json 5dc05104f
> dashboardv3/public/css/scss/leftsidebar.scss bbdc5fb26
> dashboardv3/public/css/scss/theme.scss 2b0c45d6b
>
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/18/ Changes: https://reviews.apache.org/r/71791/diff/17-18/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/17/ Changes: https://reviews.apache.org/r/71791/diff/16-17/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/16/ Changes: https://reviews.apache.org/r/71791/diff/15-16/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/15/ Changes: https://reviews.apache.org/r/71791/diff/14-15/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-search-parameter-purge.json PRE-CREATION repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/14/ Changes: https://reviews.apache.org/r/71791/diff/13-14/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Jan. 22, 2020, 10:09 p.m., Sarath Subramanian wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 4 (patched) > > <https://reviews.apache.org/r/71791/diff/13/?file=2183621#file2183621line4> > > > > if this enum is used only for aud consider renaming to > > atlas_audit_operation Initially it was atlas_audit_operation, but later changed to atlas_operation as suggested by Madhan - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review219365 --- On Dec. 11, 2019, 1:17 a.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Dec. 11, 2019, 1:17 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js a7de0d451 > dashboardv3/public/js/utils/Enums.js a7de0d451 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > c9ed79750 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > > repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java > 1863b8d74 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java > d0c06a107 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > c14850f43 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java > 78fdaca8d > > repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java > 42b63535b > > repository/src/test/java/org/apache/atlas/repositor
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Jan. 22, 2020, 10:09 p.m., Sarath Subramanian wrote: > > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > > Lines 495 (patched) > > <https://reviews.apache.org/r/71791/diff/13/?file=2183622#file2183622line496> > > > > why return type is ArrayNode.class? can we not have return type to be - > > List ? if ti works we don't need ExtractOperation() and > > ExtractResults() It didn't work for me. Please let me know if you have any idea which will return List directly that way we can really get read of the two functions > On Jan. 22, 2020, 10:09 p.m., Sarath Subramanian wrote: > > intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java > > Lines 36 (patched) > > <https://reviews.apache.org/r/71791/diff/13/?file=2183626#file2183626line36> > > > > AuditSearchParameters class looks like a trimmed down version of > > SearchParameters class. Consider using SearchParameters instead and removeA > > uditSearchParameters class. As we initially decided this way user will have less flexibility and will understand what is required to be passed for audit filter not the wider SearchParameters > On Jan. 22, 2020, 10:09 p.m., Sarath Subramanian wrote: > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > > Line 85 (original), 86 (patched) > > <https://reviews.apache.org/r/71791/diff/13/?file=2183644#file2183644line86> > > > > why ImportServiceTest now need to extend from AtlasAuditTestBase? > > AtlasAuditTestBase should be specific to only atlas sudit tests. Move > > common methods to a new base class - AtlasBaseTest. > > > > Please review all tests which is modified now to extend from > > 'AtlasAuditTestBase' Moved to AtlasTestBase and all the impexp test and audit test are now inherited from this. The only function assertExportImportAuditEntry is part of this which I might have moved to another inherited class. But we have plan to unify the imp/exp audit soon, will take care later - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review219365 --- On Dec. 11, 2019, 1:17 a.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Dec. 11, 2019, 1:17 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js a7de0d451 > dashboardv3/public/js/utils/Enums.js a7de0d451 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > c9ed79750 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 13, 2020, 8:06 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/7/ Changes: https://reviews.apache.org/r/71922/diff/6-7/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 13, 2020, 5:40 a.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/6/ Changes: https://reviews.apache.org/r/71922/diff/5-6/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 11, 2020, 1:50 a.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/5/ Changes: https://reviews.apache.org/r/71922/diff/4-5/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 10, 2020, 7:13 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/4/ Changes: https://reviews.apache.org/r/71922/diff/3-4/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
ud/3030-aws_s3_typedefs_v2.json > > Lines 238 (patched) > > <https://reviews.apache.org/r/71922/diff/3/?file=2199453#file2199453line238> > > > > This should be part of base class. same as above. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/#review219215 --- On Jan. 8, 2020, 11:12 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71922/ > --- > > (Updated Jan. 8, 2020, 11:12 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and > Sarath Subramanian. > > > Bugs: ATLAS-3564 > https://issues.apache.org/jira/browse/ATLAS-3564 > > > Repository: atlas > > > Description > --- > > This is the new version) of aws s3 model at atlas to allow the hierarchical > structure and attributes same as AWS S3 Console. The existing aws s3 models > has limitations like pseudo directory not containing another pseudo > directory, pseudo directory doesn't have attributes of aws object (as some > case it gets populated), both object and bucket doesn't containing all the > attributes of AWS S3 etc. > > > Diffs > - > > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71922/diff/3/ > > > Testing > --- > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 8, 2020, 11:12 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/3/ Changes: https://reviews.apache.org/r/71922/diff/2-3/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 8, 2020, 6:54 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description (updated) --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/1/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- (Updated Jan. 8, 2020, 6:54 p.m.) Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, pseudo directory doesn't have attributes of aws object (as some case it gets populated), both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs (updated) - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/2/ Changes: https://reviews.apache.org/r/71922/diff/1-2/ Testing --- Thanks, Sidharth Mishra
Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/ --- Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3564 https://issues.apache.org/jira/browse/ATLAS-3564 Repository: atlas Description --- This is the new version) of aws s3 model at atlas to allow the hierarchical structure and attributes same as AWS S3 Console. The existing aws s3 models has limitations like pseudo directory not containing another pseudo directory, Object doesn't have version information, both object and bucket doesn't containing all the attributes of AWS S3 etc. Diffs - addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION Diff: https://reviews.apache.org/r/71922/diff/1/ Testing --- Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/13/ Changes: https://reviews.apache.org/r/71791/diff/12-13/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/12/ Changes: https://reviews.apache.org/r/71791/diff/11-12/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/11/ Changes: https://reviews.apache.org/r/71791/diff/10-11/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
-parameter-without-filter.json PRE-CREATION server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION Diff: https://reviews.apache.org/r/71791/diff/10/ Changes: https://reviews.apache.org/r/71791/diff/9-10/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/8/ Changes: https://reviews.apache.org/r/71791/diff/7-8/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > > Lines 124 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182059#file2182059line124> > > > > define equals() and hashCode() method as well. AtlasAuditEntry is derived from AtlasBaseModelObject and the base class contains both of these method with guid check. Please let me know do we still need these two to be overridden? > On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > > Lines 125 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182059#file2182059line125> > > > > incorrect toString() implementation. please format in alignment with > > other model classes. AtlasAuditEntry is derived from AtlasBaseModelObject and the base class contains the toString() which calls this toString(sb). At the end it returns the correct alignment. > On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > > Lines 128 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182060#file2182060line139> > > > > => StringUtils.join(entityIds, ",")) Fixed. Could you please let me know the advantage of StringUtils.join over String.join > On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > > Lines 76 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182062#file2182062line76> > > > > move debug log to beginning of method. This debug msg is to put debug message when we successfully added the audit entry > On Dec. 5, 2019, 12:10 a.m., Sarath Subramanian wrote: > > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > > Lines 437 (patched) > > <https://reviews.apache.org/r/71791/diff/7/?file=2182093#file2182093line462> > > > > shouldn't purge events be audited only on successful purge on entities? > > consider checking EntityMutationResponse for any purge events present and > > then audit them. I think its better to audit admin action purge even if we don't purge any entities. Please let me know if you still think we should not audit the unsuccessful purge action by admin. - Sidharth ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218927 --- On Dec. 4, 2019, 9:57 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Dec. 4, 2019, 9:57 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java > eb8634533 > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION >
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Dec. 5, 2019, 12:39 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> > Lines 1057 (patched)
> > <https://reviews.apache.org/r/71791/diff/7/?file=2182064#file2182064line1095>
> >
> > This will cause 'DELETED' entities to be skipped even though purge is
> > in progress. This isn't correct. Please review and update.
> >
> > final String guid = AtlasGraphUtilsV2.getIdFromVertex(vertex);
> >
> > ret = guid == null;
> >
> > if (!ret) {
> > if (reqContext.isPurgeRequested()) {
> > ret = requestContext.isPurgedEntity(guid) || getState(vertex) ==
> > ACTIVE;
> > } else {
> > ret = requestContext.isDeletedEntity(guid) || getState(vertex) ==
> > DELETED;
> > }
> > }
As the code for purge, hard delete is same, the requestContext still stores the
purged entities as part of deleteEntity List.
> On Dec. 5, 2019, 12:39 a.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Lines 436 (patched)
> > <https://reviews.apache.org/r/71791/diff/7/?file=2182093#file2182093line461>
> >
> > Audit generation should be done inside entityStore.purgeByIds(), in the
> > same graph transaction. Avoid spilling such details into REST layer.
As I earlier replied to your previous comment, Moving auditing to inside
AtlasEntityStore.purgeByIds() is causing circular Injection issue.
AtlasAuditService depends on DataAccess and DataAccess depends on
AtlasEntityStore. So AtlasEntityStore can't depends on AtlasAuditService again.
KIndly suggest is there any other option to avoid this circular injection issue.
- Sidharth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218929
---
On Dec. 4, 2019, 9:57 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
>
> (Updated Dec. 4, 2019, 9:57 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Right now for purge, delete entity, etc. at Atlas we create audit entry at
> HBase. User can go to entity and click on audit to see the audit information
> for that particular entity. But if user purge one entity there will not be
> any vertex at Janus graph and user will not able to get the purge audit which
> is important for governance purpose.
>
> As part of fix, implemented Atlas Audit Framework which is being used for
> Purge operation and can be extended for other operations. In Future once we
> change the AtlasAudit type as super type for Import export as well we can use
> this framework for all Atlas operations.
>
> admin/audit rest has been added which will return all the Atlas audit
> operations including import export at Atlas.
>
>
> Diffs
> -
>
> addons/models/-Area0/0010-base_model.json 2f5fdaf14
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
> 8c0a6407c
> dashboardv2/public/js/utils/Enums.js 58542ec59
> dashboardv3/public/js/utils/Enums.js 58542ec59
> intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
> PRE-CREATION
>
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
> b448d510b
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java
> eb8634533
>
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> c9ed79750
> repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java
> be1698fb4
> repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java
> 02f78b369
>
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
> PRE-CREATION
>
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditS
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/7/ Changes: https://reviews.apache.org/r/71791/diff/6-7/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 5 (patched) > > <https://reviews.apache.org/r/71791/diff/5/?file=2181198#file2181198line5> > > > > "Audit Operation defines the operations we support for admin audit" => > > "Defines admin operations in Apache Atlas" Removed the admin and changed it to "Defines operations in Apache Atlas which can be audited" > On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote: > > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > > Lines 436 (patched) > > <https://reviews.apache.org/r/71791/diff/5/?file=2181235#file2181235line461> > > > > Instead of generating audit in a seperate graph transaction here, > > consider generating in the transaction used by > > AtlasEntityStore.purgeByIds() i.e. move auditing to inside > > AtlasEntityStore.purgeByIds(). Moving auditing to inside AtlasEntityStore.purgeByIds() is causing circular Injection issue. AtlasAuditService depends on DataAccess and DataAccess depends on AtlasEntityStore. So AtlasEntityStore can't depends on AtlasAuditService again. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218865 --- On Dec. 4, 2019, 9:47 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Dec. 4, 2019, 9:47 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java > eb8634533 > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > c9ed79750 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/6/ Changes: https://reviews.apache.org/r/71791/diff/5-6/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 4 (patched) > > <https://reviews.apache.org/r/71791/diff/5/?file=2181198#file2181198line4> > > > > atlas_operation => atlas_admin_operation I think as you initially suggested to rename it to atlas_operation is more flexible than atlas_admin_operation as in future we may allow some more operation to be audited like other than admin operations. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218865 --- On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Nov. 30, 2019, 7:44 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > > repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java > 1863b8d74 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java > d0c06a107 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > c14850f43 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java > 78fdaca8d > > repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java > 42b63535b > > repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java > a1b512fb
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Nov. 27, 2019, 12:39 a.m., Madhan Neethiraj wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 442 (patched) > > <https://reviews.apache.org/r/71791/diff/4/?file=2179150#file2179150line442> > > > > Consider renaming 'clientHost' to 'clientId' - given this field is used > > to store AtlasServer name for import/export audit entries. > > > > This rename should be applied to AtlasAuditEntry.clientHost as well. I changed it to clientId as suggested, But please note this field contains Client IP Address. In future we may put host fqdn if available. In both the cases do you think client Id makes more sense? > On Nov. 27, 2019, 12:39 a.m., Madhan Neethiraj wrote: > > pom.xml > > Line 695 (original), 695 (patched) > > <https://reviews.apache.org/r/71791/diff/4/?file=2179156#file2179156line695> > > > > I suggest a separate patch for library version changes. This fix was to fix the precommit build and now the precommit is successful - https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1559/. I am having separate patch for the lib version change and reverted the changes - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218808 ------- On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Nov. 30, 2019, 7:44 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > >
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
- Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218808 --- On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Nov. 30, 2019, 7:44 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > > repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java > 1863b8d74 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java > d0c06a107 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > c14850f43 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java > 78fdaca8d > > repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java > 42b63535b > > repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java > a1b512fb9 > > repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java > 7a1ed18cb > > repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java > 48b958271 > > repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java > c9bb11c56 > > repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java > 0ffc3d595 > > r
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/5/ Changes: https://reviews.apache.org/r/71791/diff/4-5/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
> On Nov. 25, 2019, 5:05 a.m., Madhan Neethiraj wrote: > > addons/models/-Area0/0010-base_model.json > > Lines 397 (patched) > > <https://reviews.apache.org/r/71791/diff/3/?file=2174685#file2174685line397> > > > > - I suggest adding attribute 'serverName', that will be set to > > AtlasServer.name - similar to > > __ExportImportAuditEntry.sourceServerName/__ExportImportAuditEntry.targetServerName > > - consider adding an unique attribute, like __guid As we have __guid as system attribute and gets added to entity, not adding __guid as part of model. Do you think we need serverName as it will be inserted as current host all the time and it may not be usefull. For import export its needed and it may differ so we addded that. As part of migration I was planning to change the current import export audit model where it will be derived from this audit model type and add the source and destination server only for import export audit type. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218785 ------- On Nov. 26, 2019, 9:11 a.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > --- > > (Updated Nov. 26, 2019, 9:11 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js 58542ec59 > intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java > PRE-CREATION > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > b448d510b > pom.xml b2506e70e > > repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > be1698fb4 > repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java > 02f78b369 > > repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java > 78865c569 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java > ba7a8a04b > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > 925b2a007 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 4d438525a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java > db8c6c83c > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > 25e0a5362 > > repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java > 1863b8d74 > > repository/src/test/java/org/apache/atlas/repo
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/4/ Changes: https://reviews.apache.org/r/71791/diff/3-4/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/3/ Changes: https://reviews.apache.org/r/71791/diff/2-3/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/2/ Changes: https://reviews.apache.org/r/71791/diff/1-2/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
/AdminResourceTest.java 563a16f64 Diff: https://reviews.apache.org/r/71791/diff/1/ Testing --- Two UT and one IT has been added to test functionality like: 1. Atlas Audit entry save and retrive 2. Atlas Purge Audit store, retrive and search (UT and IT) Thanks, Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 13, 2019, 8:37 p.m., Sarath Subramanian wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > > Lines 484 (patched) > > <https://reviews.apache.org/r/71737/diff/4/?file=2173006#file2173006line485> > > > > should this be AtlasPrivilege.ADMIN_PURGE ? > > Sidharth Mishra wrote: > As we don't have the Ranger changes for time being its ADMIN_IMPORT. > Later ll change it to ADMIN_PURGE To track this created a JIRA Item - ATLAS-3524 - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71737/#review218622 --- On Nov. 14, 2019, 7:10 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71737/ > --- > > (Updated Nov. 14, 2019, 7:10 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3477 > https://issues.apache.org/jira/browse/ATLAS-3477 > > > Repository: atlas > > > Description > --- > > As part of this solution users can purge the entities which are already soft > deleted. This will be useful as someone may be interested to purge some list > of soft deleted entities from Atlas. Due to governance compliances, only > admin can perform purge of entities. > > Purge will also create separate Audit entry at HBase similar to delete entity > operation. > > > Diffs > - > > authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java > 59c596d66 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 199d6bf58 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > 444167eae > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > e9cc7cd3d > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > 7ace00dff > intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java > daf4ca88c > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 43a9b842c > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > b94590bf1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > d2544df4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 3389d2465 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17a96493f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 4d57d8bc8 > server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > 48f0cd3c2 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 464d46f1e > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > 5d506bbc2 > > > Diff: https://reviews.apache.org/r/71737/diff/6/ > > > Testing > --- > > IT has been added. > Validated manually by running below mentioned queries and REST: > > > a) create table thursday_07_11_1 (id int); >create table thursday_07_11_1_ctas as select * from thursday_07_11_1; > >create table thursday_07_11_2 (id int); >create table thursday_07_11_2_ctas as select * from thursday_07_11_2; > b) The above will create two lineages. Go to Atlas UI and get the vertices > for atleast any two. > > > c) Using rest delete those two entities or use below query: > drop table thursday_07_11_1; > drop table thursday_07_11_1_ctas; > > drop table thursday_07_11_2; > drop table thursday_07_11_2_ctas; > > REST delete example: > DELETE > http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c > > d) Now use the same guids and call purge REST: > DELETE > http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05=98bee463-8810-402a-bcdc-3eb00bf50365 > > Check the response which will return all the successfully purged entities > > e) For each guid check if the audit has separate new entry for purge. REST > used: > GET > http://localhost:31000/api/atlas/v2/entity/7b6806
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
plicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 14, 2019, 12:02 a.m., Ashutosh Mestry wrote: > > My thinking is that there should be a bulk API that should take JSON > > payload of entities to be purged. Generally, there are large number of > > deleted entities. There should be a convinent way to purge them. Right now > > the queryString has a smaller limit which is quickly exhausted due to > > length of GUIDs. Right now in the current code I have already Addressed this and changed it to body from queryString. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71737/#review218627 --- On Nov. 13, 2019, 9:04 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71737/ > --- > > (Updated Nov. 13, 2019, 9:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3477 > https://issues.apache.org/jira/browse/ATLAS-3477 > > > Repository: atlas > > > Description > --- > > As part of this solution users can purge the entities which are already soft > deleted. This will be useful as someone may be interested to purge some list > of soft deleted entities from Atlas. Due to governance compliances, only > admin can perform purge of entities. > > Purge will also create separate Audit entry at HBase similar to delete entity > operation. > > > Diffs > - > > authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java > 59c596d66 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 199d6bf58 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > 444167eae > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > e9cc7cd3d > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > 7ace00dff > intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java > daf4ca88c > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 43a9b842c > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > b94590bf1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > d2544df4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 3389d2465 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17a96493f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 4d57d8bc8 > server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > 48f0cd3c2 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 464d46f1e > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > 5d506bbc2 > > > Diff: https://reviews.apache.org/r/71737/diff/5/ > > > Testing > --- > > IT has been added. > Validated manually by running below mentioned queries and REST: > > > a) create table thursday_07_11_1 (id int); >create table thursday_07_11_1_ctas as select * from thursday_07_11_1; > >create table thursday_07_11_2 (id int); >create table thursday_07_11_2_ctas as select * from thursday_07_11_2; > b) The above will create two lineages. Go to Atlas UI and get the vertices > for atleast any two. > > > c) Using rest delete those two entities or use below query: > drop table thursday_07_11_1; > drop table thursday_07_11_1_ctas; > > drop table thursday_07_11_2; > drop table thursday_07_11_2_ctas; > > REST delete example: > DELETE > http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c > > d) Now use the same guids and call purge REST: > DELETE > http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05=98bee463-8810-402a-bcdc-3eb00bf50365 > > Check the response which will return all the successfully purged entities > > e) For each guid check if the audit has separate new entry for purge. REST > used: > GET > http://localhost:31000/api/atlas/v2/entity/7b6806c1-8daf-4e0f-8d10-672a192968f1/audit > > Example Response: &
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 14, 2019, midnight, Ashutosh Mestry wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > > Lines 159 (patched) > > <https://reviews.apache.org/r/71737/diff/5/?file=2173080#file2173080line159> > > > > Does this mean that if purge is requested, then even active edges are > > requested whereas if not deleted edges are requested. > > > > Shouldn't it be that when purge is requested, both active and deleted > > edges are requested? Right now purge will only be allowed if the entity is already soft deleted. This check skip the edges or vertices for the same reason. > On Nov. 14, 2019, midnight, Ashutosh Mestry wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > > Lines 200 (patched) > > <https://reviews.apache.org/r/71737/diff/5/?file=2173080#file2173080line200> > > > > Same comment as above. Right now purge will only be allowed if the entity is already soft deleted. This check skip the edges or vertices for the same reason. - Sidharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71737/#review218626 --- On Nov. 13, 2019, 9:04 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71737/ > --- > > (Updated Nov. 13, 2019, 9:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3477 > https://issues.apache.org/jira/browse/ATLAS-3477 > > > Repository: atlas > > > Description > --- > > As part of this solution users can purge the entities which are already soft > deleted. This will be useful as someone may be interested to purge some list > of soft deleted entities from Atlas. Due to governance compliances, only > admin can perform purge of entities. > > Purge will also create separate Audit entry at HBase similar to delete entity > operation. > > > Diffs > - > > authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java > 59c596d66 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 199d6bf58 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > 444167eae > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > e9cc7cd3d > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > 7ace00dff > intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java > daf4ca88c > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 43a9b842c > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > b94590bf1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > d2544df4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 3389d2465 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17a96493f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 4d57d8bc8 > server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > 48f0cd3c2 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 464d46f1e > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > 5d506bbc2 > > > Diff: https://reviews.apache.org/r/71737/diff/5/ > > > Testing > --- > > IT has been added. > Validated manually by running below mentioned queries and REST: > > > a) create table thursday_07_11_1 (id int); >create table thursday_07_11_1_ctas as select * from thursday_07_11_1; > >create table thursday_07_11_2 (id int); >create table thursday_07_11_2_ctas as select * from thursday_07_11_2; > b) The above will create two lineages. Go to Atlas UI and get the vertices > for atleast any two. > > > c) Using rest delete those two entities or use below query: > drop table thursday_07_11_1; > drop table thursday_07_11_1_ctas; > > drop table thursday_07_11_2; > drop table thursday_07_1
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
> On Nov. 13, 2019, 8:37 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
> > Lines 484 (patched)
> > <https://reviews.apache.org/r/71737/diff/4/?file=2173006#file2173006line485>
> >
> > should this be AtlasPrivilege.ADMIN_PURGE ?
As we don't have the Ranger changes for time being its ADMIN_IMPORT. Later ll
change it to ADMIN_PURGE
- Sidharth
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71737/#review218622
---
On Nov. 13, 2019, 6:04 a.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71737/
> ---
>
> (Updated Nov. 13, 2019, 6:04 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-3477
> https://issues.apache.org/jira/browse/ATLAS-3477
>
>
> Repository: atlas
>
>
> Description
> ---
>
> As part of this solution users can purge the entities which are already soft
> deleted. This will be useful as someone may be interested to purge some list
> of soft deleted entities from Atlas. Due to governance compliances, only
> admin can perform purge of entities.
>
> Purge will also create separate Audit entry at HBase similar to delete entity
> operation.
>
>
> Diffs
> -
>
> authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java
> 59c596d66
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
> 199d6bf58
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java
> 444167eae
> intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
> e9cc7cd3d
>
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
> 7ace00dff
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java
> daf4ca88c
>
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
> 43a9b842c
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
> b94590bf1
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> d2544df4f
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
> 3389d2465
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
> 17a96493f
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
> 4d57d8bc8
> server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c
>
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
> 48f0cd3c2
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> 464d46f1e
>
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
> 5d506bbc2
>
>
> Diff: https://reviews.apache.org/r/71737/diff/4/
>
>
> Testing
> ---
>
> IT has been added.
> Validated manually by running below mentioned queries and REST:
>
>
> a) create table thursday_07_11_1 (id int);
>create table thursday_07_11_1_ctas as select * from thursday_07_11_1;
>
>create table thursday_07_11_2 (id int);
>create table thursday_07_11_2_ctas as select * from thursday_07_11_2;
> b) The above will create two lineages. Go to Atlas UI and get the vertices
> for atleast any two.
>
>
> c) Using rest delete those two entities or use below query:
> drop table thursday_07_11_1;
> drop table thursday_07_11_1_ctas;
>
> drop table thursday_07_11_2;
> drop table thursday_07_11_2_ctas;
>
> REST delete example:
> DELETE
> http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c
>
> d) Now use the same guids and call purge REST:
> DELETE
> http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05=98bee463-8810-402a-bcdc-3eb00bf50365
>
> Check the response which will return all the successfully purged entities
>
> e) For each guid check if the audit has separate new entry for purge. REST
> used:
> GET
> http://localhost:31000/api/atlas/v2/entity/7b6806c1-8daf-4e0f-8d10-672a192968f1/audit
>
> Example Response:
>
> [
> {
> "ent
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
plicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
edTo\":null,\"replicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
ot;:\"admin\",\"replicatedTo\":null,\"replicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
"entity": null,
"type": null
},
{
"entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
"timestamp": 1573161723180,
"user": "impala",
"action": "ENTITY_CREATE",
"details": "Created:
{\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
"eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
"entity": null,
"type": null
}
]
Thanks,
Sidharth Mishra
