> On March 22, 2021, 6:08 p.m., Madhan Neethiraj wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java
> > Line 173 (original), 175 (patched)
> > <https://reviews.apache.org/r/73239/diff/2/?file=2248185#file2248185line175>
> >
> >     if hive_process and hdfs_path entities are skipped here, then 
> > hive_process_execution should be skipped as well. Please review.

Including this causes few executions not to be included.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73239/#review222727
-----------------------------------------------------------


On March 22, 2021, 5:23 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73239/
> -----------------------------------------------------------
> 
> (Updated March 22, 2021, 5:23 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4204
>     https://issues.apache.org/jira/browse/ATLAS-4204
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please see JIRA.
> 
> Terms and abbreviations used:
> - Database objects: Database, table, views, etc.
> - DDL: Data Definition Langugage. Database parlance. Operations performed on 
> a database server that result in creation of objects that hold data.
> - DML: Data Manipulation Language. Database parlance. Operations performed on 
> a database server that result in manipulation (add, update, delete) of data. 
> These are performed using database objects that are already created.
> 
> Hive hook is implemented by:
> - _HiveHook_
> - _HiveMetastoreImpl_
> 
> - _BaseHiveEvent_ has 2 methods:
> - getHiveMetastoreEntities: Called by HMS events.
> - getHiveEntities: Called by HS2 events.
> 
> **Approach**
> - Introduce a flag _skipDDL_ indicating if entities generated as part of DDL 
> operation should be skipped by a hook. This flag is set as a configuration 
> parameters: atlas.hive.hook.ignore.ddl.operations
> - Skip creation of Atlas's DDL entities when HS2 event is used.
> - Continue using HMS event processing unchanged.
> 
> 
> Diffs
> -----
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  128647147 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 79e87c79d 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  f01419ce6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java
>  6961fa7c2 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRenameCol.java
>  756a6089e 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  7c269ce53 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
>  bf5f5620e 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java
>  63fc89424 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropDatabase.java
>  20019d2ef 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropTable.java
>  440c08aff 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> 7a11ec3b3 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/AlterDatabaseTest.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/CreateDBTest.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/CreateTableTest.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/DropDatabaseTest.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/DropTableTest.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/HiveHookEventFixtures.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 52684e75a 
>   addons/hive-bridge/src/test/resources/json/alter-db-hs2.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/alter-db.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/alter-table-hs2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/alter-table-rename.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/alter-table.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/create-db-hs2.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/create-db-table-col-hms.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/create-db-table-col.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/create-db.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/create-table-hs2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/create-table.json PRE-CREATION 
>   common/src/main/java/org/apache/atlas/utils/PathExtractorContext.java 
> ce688e4e4 
> 
> 
> Diff: https://reviews.apache.org/r/73239/diff/2/
> 
> 
> Testing
> -------
> 
> **Functional tests**
> - Used specific queries to exercise the affected code paths.
> - Verified messages going out of the hooks, onces posted on ATLAS_HOOK and 
> the entities created within Atlas.
> 
> **Volume test**
> Pending.
> 
> **Unit test**
> New tests added.
> 
> 
> File Attachments
> ----------------
> 
> HMS Create Table
>   
> https://reviews.apache.org/media/uploaded/files/2021/03/18/748a849a-c8f8-4869-bcb0-ee4ee50215d8__hms-create-table.json
> HS2 Create Table
>   
> https://reviews.apache.org/media/uploaded/files/2021/03/18/3f27aac6-a741-4758-9d6a-db67b05a63eb__hs2-create-table.json
> HS2 Create Database
>   
> https://reviews.apache.org/media/uploaded/files/2021/03/18/e0472b5d-3367-42d8-834a-7cfafa13aae6__hs2-create-db.json
> HMS Create Database
>   
> https://reviews.apache.org/media/uploaded/files/2021/03/18/bc34e035-f805-4ddd-abf3-a2b833ce3af0__hms-create-db.json
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Reply via email to