> On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
> > Line 49 (original), 50 (patched)
> > <https://reviews.apache.org/r/73239/diff/4/?file=2248405#file2248405line50>
> >
> >     can we call ActiveEntityFilter.run() inside getNotificationMessages() 
> > to make it filter in a central place? Please review other commands and move 
> > logic to filter in getNotificationMessages()
> >     
> >     
> >     e.g.
> >     AtlasEntitiesWithExtInfo entities = context.isMetastoreHook() ? 
> > getHiveMetastoreEntities() : ActiveEntityFilter.run(getHiveEntities());

This is tricky to implement given the current state. I have partially 
implemented this commment. Please take a look. We may decide to move to earlier 
implementation.


> On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
> > Lines 26 (patched)
> > <https://reviews.apache.org/r/73239/diff/4/?file=2248410#file2248410line26>
> >
> >     consider renaming ActiveEntityFilter -> HookEntityFilter since it deals 
> > with filtering entities in hook side and we have methods to filter out hook 
> > notification messages.

ActiveEntityFilter is more appropriate since this class abstracts the filer 
that gets set as active.


> On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java
> > Lines 27 (patched)
> > <https://reviews.apache.org/r/73239/diff/4/?file=2248411#file2248411line27>
> >
> >     Consider renaming all 'run' methods to 'filter'. 'run' can be confused 
> > with starting some process or thread.
> >     
> >     e.g. EntityFilter.filter(entities)

I am using apply instead of run.


> On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java
> > Lines 31 (patched)
> > <https://reviews.apache.org/r/73239/diff/4/?file=2248411#file2248411line31>
> >
> >     runObjectId -> filterObjectIds

I have renamed this method to use apply instead of run.


> On April 8, 2021, 11:26 p.m., Sarath Subramanian wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/PassthroughFilter.java
> > Lines 26 (patched)
> > <https://reviews.apache.org/r/73239/diff/4/?file=2248413#file2248413line26>
> >
> >     PassthroughFilter can be specific to hive, consider renaming to 
> > 'HiveEntityFilter'

Passthrough is more appropriate since it does not do anything to the entities.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73239/#review222790
-----------------------------------------------------------


On April 2, 2021, 4:18 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73239/
> -----------------------------------------------------------
> 
> (Updated April 2, 2021, 4:18 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4204
>     https://issues.apache.org/jira/browse/ATLAS-4204
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please see JIRA.
> 
> Terms and abbreviations used:
> - Database objects: Database, table, views, etc.
> - DDL: Data Definition Langugage. Database parlance. Operations performed on 
> a database server that result in creation of objects that hold data.
> - DML: Data Manipulation Language. Database parlance. Operations performed on 
> a database server that result in manipulation (add, update, delete) of data. 
> These are performed using database objects that are already created.
> 
> Hive hook is implemented by:
> - _HiveHook_
> - _HiveMetastoreImpl_
> 
> - _BaseHiveEvent_ has 2 methods:
> - getHiveMetastoreEntities: Called by HMS events.
> - getHiveEntities: Called by HS2 events.
> 
> **Approach**
> - Introduce a flag _skipDDL_ indicating if entities generated as part of DDL 
> operation should be skipped by a hook. This flag is set as a configuration 
> parameters: atlas.hive.hook.ignore.ddl.operations
> - Continue using HMS event processing unchanged.
> - New: _ActiveEntityFilter_: Initializes entity filter based on the skipDDL 
> setting.
> - New: _LineageOnlyFilter_: Retains entities that participate in lineage.
> - New: _PassthroughFilter_: Does not do any filtering.
> 
> 
> Diffs
> -----
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  128647147 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 79e87c79d 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTable.java
>  e164370ba 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java
>  6961fa7c2 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRenameCol.java
>  756a6089e 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  7c269ce53 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
>  bf5f5620e 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
>  659417d82 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java
>  63fc89424 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropDatabase.java
>  20019d2ef 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropTable.java
>  440c08aff 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/LineageOnlyFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/PassthroughFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/ActiveEntityFilterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/resources/atlas-application.properties 
> 898b69c99 
>   addons/hive-bridge/src/test/resources/json/hs2-alter-view-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-alter-view.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-db.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-process.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-table.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-db-v2.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-db.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-table-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-table.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-table-rename-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-table-rename.json 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 
>   intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 
> 
> 
> Diff: https://reviews.apache.org/r/73239/diff/4/
> 
> 
> Testing
> -------
> 
> **Functional tests**
> - Used specific queries to exercise the affected code paths.
> - Verified messages going out of the hooks, onces posted on ATLAS_HOOK and 
> the entities created within Atlas.
> 
> **Test data**
> ```
> create database cadb02;
> use cadb02;
> 
> create external table hh6(col1 int) location '/tmp/external/hh6.csv';
> ALTER TABLE hh6 RENAME TO hh6_renamed;
> 
> create view hh6_renamed_view as select * from hh6_renamed;
> ALTER VIEW hh6_renamed_view RENAME TO hh6_renamed_view2;
> 
> create external table bb1(col1 int) location '/tmp/bb1.csv';
> ALTER TABLE bb1 CHANGE COLUMN col1 col11 string;
> ALTER TABLE bb1 SET SERDEPROPERTIES ('field.delim' = '|');
> ALTER TABLE bb1 ADD COLUMNS (dept STRING COMMENT 'Department name');
> ALTER TABLE bb1 RENAME TO cc1;
> ```
> 
> **Volume test**
> 
> +----------+-------------+--------------+
> | Hook     | Baseline    | New          |
> +----------+-------------+--------------+
> | HMS      |      31 KB. |        31 KB |
> +----------+-------------+--------------+
> | HS2      |      53 KB. |        12 KB |
> +----------+-------------+--------------+
> 
> Even though the size of payload has reduced, the overall time taken for the 
> volumne of entities created has not changed.
> 
> **Unit test**
> New tests added.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Reply via email to