> On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java > > Line 49 (original), 50 (patched) > > <https://reviews.apache.org/r/73239/diff/4/?file=2248405#file2248405line50> > > > > can we call ActiveEntityFilter.run() inside getNotificationMessages() > > to make it filter in a central place? Please review other commands and move > > logic to filter in getNotificationMessages() > > > > > > e.g. > > AtlasEntitiesWithExtInfo entities = context.isMetastoreHook() ? > > getHiveMetastoreEntities() : ActiveEntityFilter.run(getHiveEntities());
This is tricky to implement given the current state. I have partially implemented this commment. Please take a look. We may decide to move to earlier implementation. > On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java > > Lines 26 (patched) > > <https://reviews.apache.org/r/73239/diff/4/?file=2248410#file2248410line26> > > > > consider renaming ActiveEntityFilter -> HookEntityFilter since it deals > > with filtering entities in hook side and we have methods to filter out hook > > notification messages. ActiveEntityFilter is more appropriate since this class abstracts the filer that gets set as active. > On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java > > Lines 27 (patched) > > <https://reviews.apache.org/r/73239/diff/4/?file=2248411#file2248411line27> > > > > Consider renaming all 'run' methods to 'filter'. 'run' can be confused > > with starting some process or thread. > > > > e.g. EntityFilter.filter(entities) I am using apply instead of run. > On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java > > Lines 31 (patched) > > <https://reviews.apache.org/r/73239/diff/4/?file=2248411#file2248411line31> > > > > runObjectId -> filterObjectIds I have renamed this method to use apply instead of run. > On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/PassthroughFilter.java > > Lines 26 (patched) > > <https://reviews.apache.org/r/73239/diff/4/?file=2248413#file2248413line26> > > > > PassthroughFilter can be specific to hive, consider renaming to > > 'HiveEntityFilter' Passthrough is more appropriate since it does not do anything to the entities. - Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73239/#review222790 ----------------------------------------------------------- On April 2, 2021, 4:18 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73239/ > ----------------------------------------------------------- > > (Updated April 2, 2021, 4:18 p.m.) > > > Review request for atlas, Madhan Neethiraj and Sarath Subramanian. > > > Bugs: ATLAS-4204 > https://issues.apache.org/jira/browse/ATLAS-4204 > > > Repository: atlas > > > Description > ------- > > **Background** > Please see JIRA. > > Terms and abbreviations used: > - Database objects: Database, table, views, etc. > - DDL: Data Definition Langugage. Database parlance. Operations performed on > a database server that result in creation of objects that hold data. > - DML: Data Manipulation Language. Database parlance. Operations performed on > a database server that result in manipulation (add, update, delete) of data. > These are performed using database objects that are already created. > > Hive hook is implemented by: > - _HiveHook_ > - _HiveMetastoreImpl_ > > - _BaseHiveEvent_ has 2 methods: > - getHiveMetastoreEntities: Called by HMS events. > - getHiveEntities: Called by HS2 events. > > **Approach** > - Introduce a flag _skipDDL_ indicating if entities generated as part of DDL > operation should be skipped by a hook. This flag is set as a configuration > parameters: atlas.hive.hook.ignore.ddl.operations > - Continue using HMS event processing unchanged. > - New: _ActiveEntityFilter_: Initializes entity filter based on the skipDDL > setting. > - New: _LineageOnlyFilter_: Retains entities that participate in lineage. > - New: _PassthroughFilter_: Does not do any filtering. > > > Diffs > ----- > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > 128647147 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 79e87c79d > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTable.java > e164370ba > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > 6961fa7c2 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRenameCol.java > 756a6089e > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > 7c269ce53 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java > bf5f5620e > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java > 659417d82 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > 63fc89424 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropDatabase.java > 20019d2ef > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropTable.java > 440c08aff > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java > PRE-CREATION > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java > PRE-CREATION > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/LineageOnlyFilter.java > PRE-CREATION > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/PassthroughFilter.java > PRE-CREATION > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/ActiveEntityFilterTest.java > PRE-CREATION > addons/hive-bridge/src/test/resources/atlas-application.properties > 898b69c99 > addons/hive-bridge/src/test/resources/json/hs2-alter-view-v2.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-alter-view.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-create-db.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-create-process.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-create-table.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-drop-db-v2.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-drop-db.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-drop-table-v2.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-drop-table.json PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-table-rename-v2.json > PRE-CREATION > addons/hive-bridge/src/test/resources/json/hs2-table-rename.json > PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 > intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 > > > Diff: https://reviews.apache.org/r/73239/diff/4/ > > > Testing > ------- > > **Functional tests** > - Used specific queries to exercise the affected code paths. > - Verified messages going out of the hooks, onces posted on ATLAS_HOOK and > the entities created within Atlas. > > **Test data** > ``` > create database cadb02; > use cadb02; > > create external table hh6(col1 int) location '/tmp/external/hh6.csv'; > ALTER TABLE hh6 RENAME TO hh6_renamed; > > create view hh6_renamed_view as select * from hh6_renamed; > ALTER VIEW hh6_renamed_view RENAME TO hh6_renamed_view2; > > create external table bb1(col1 int) location '/tmp/bb1.csv'; > ALTER TABLE bb1 CHANGE COLUMN col1 col11 string; > ALTER TABLE bb1 SET SERDEPROPERTIES ('field.delim' = '|'); > ALTER TABLE bb1 ADD COLUMNS (dept STRING COMMENT 'Department name'); > ALTER TABLE bb1 RENAME TO cc1; > ``` > > **Volume test** > > +----------+-------------+--------------+ > | Hook | Baseline | New | > +----------+-------------+--------------+ > | HMS | 31 KB. | 31 KB | > +----------+-------------+--------------+ > | HS2 | 53 KB. | 12 KB | > +----------+-------------+--------------+ > > Even though the size of payload has reduced, the overall time taken for the > volumne of entities created has not changed. > > **Unit test** > New tests added. > > > Thanks, > > Ashutosh Mestry > >
