> On March 22, 2021, 6:08 p.m., Madhan Neethiraj wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > > Line 173 (original), 175 (patched) > > <https://reviews.apache.org/r/73239/diff/2/?file=2248185#file2248185line175> > > > > if hive_process and hdfs_path entities are skipped here, then > > hive_process_execution should be skipped as well. Please review.
Including this causes few executions not to be included. - Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73239/#review222727 ----------------------------------------------------------- On March 22, 2021, 5:23 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73239/ > ----------------------------------------------------------- > > (Updated March 22, 2021, 5:23 p.m.) > > > Review request for atlas, Madhan Neethiraj and Sarath Subramanian. > > > Bugs: ATLAS-4204 > https://issues.apache.org/jira/browse/ATLAS-4204 > > > Repository: atlas > > > Description > ------- > > **Background** > Please see JIRA. > > Terms and abbreviations used: > - Database objects: Database, table, views, etc. > - DDL: Data Definition Langugage. Database parlance. Operations performed on > a database server that result in creation of objects that hold data. > - DML: Data Manipulation Language. Database parlance. Operations performed on > a database server that result in manipulation (add, update, delete) of data. > These are performed using database objects that are already created. > > Hive hook is implemented by: > - _HiveHook_ > - _HiveMetastoreImpl_ > > - _BaseHiveEvent_ has 2 methods: > - getHiveMetastoreEntities: Called by HMS events. > - getHiveEntities: Called by HS2 events. > > **Approach** > - Introduce a flag _skipDDL_ indicating if entities generated as part of DDL > operation should be skipped by a hook. This flag is set as a configuration > parameters: atlas.hive.hook.ignore.ddl.operations > - Skip creation of Atlas's DDL entities when HS2 event is used. > - Continue using HMS event processing unchanged. > > > Diffs > ----- > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > 128647147 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 79e87c79d > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java > f01419ce6 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > 6961fa7c2 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRenameCol.java > 756a6089e > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > 7c269ce53 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java > bf5f5620e > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > 63fc89424 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropDatabase.java > 20019d2ef > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropTable.java > 440c08aff > addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java > 7a11ec3b3 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/AlterDatabaseTest.java > PRE-CREATION > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/CreateDBTest.java > PRE-CREATION > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/CreateTableTest.java > PRE-CREATION > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/DropDatabaseTest.java > PRE-CREATION > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/DropTableTest.java > PRE-CREATION > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/HiveHookEventFixtures.java > PRE-CREATION > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 52684e75a > addons/hive-bridge/src/test/resources/json/alter-db-hs2.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/alter-db.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/alter-table-hs2.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/alter-table-rename.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/alter-table.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/create-db-hs2.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/create-db-table-col-hms.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/create-db-table-col.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/create-db.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/create-table-hs2.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/create-table.json PRE-CREATION > common/src/main/java/org/apache/atlas/utils/PathExtractorContext.java > ce688e4e4 > > > Diff: https://reviews.apache.org/r/73239/diff/2/ > > > Testing > ------- > > **Functional tests** > - Used specific queries to exercise the affected code paths. > - Verified messages going out of the hooks, onces posted on ATLAS_HOOK and > the entities created within Atlas. > > **Volume test** > Pending. > > **Unit test** > New tests added. > > > File Attachments > ---------------- > > HMS Create Table > > https://reviews.apache.org/media/uploaded/files/2021/03/18/748a849a-c8f8-4869-bcb0-ee4ee50215d8__hms-create-table.json > HS2 Create Table > > https://reviews.apache.org/media/uploaded/files/2021/03/18/3f27aac6-a741-4758-9d6a-db67b05a63eb__hs2-create-table.json > HS2 Create Database > > https://reviews.apache.org/media/uploaded/files/2021/03/18/e0472b5d-3367-42d8-834a-7cfafa13aae6__hs2-create-db.json > HMS Create Database > > https://reviews.apache.org/media/uploaded/files/2021/03/18/bc34e035-f805-4ddd-abf3-a2b833ce3af0__hms-create-db.json > > > Thanks, > > Ashutosh Mestry > >
