-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69632/#review211693
-----------------------------------------------------------


Ship it!




Ship It!

- Sarath Subramanian


On Dec. 26, 2018, 1:22 p.m., Madhan Neethiraj wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69632/
> -----------------------------------------------------------
> 
> (Updated Dec. 26, 2018, 1:22 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-3006
>     https://issues.apache.org/jira/browse/ATLAS-3006
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Introduced following configurations to specify temporary/staging Hive tables, 
> so that Hive hook/Atlas server can ignore or prune these tables. For pruned 
> tables, columns and column-lineage details will be ignored.
> 
> # configurations for Hive hook
>   atlas.hook.hive.ignore.hive_table.pattern=
>   atlas.hook.hive.prune.hive_table.pattern=
> 
> # configurations for Atlas server
>   atlas.notification.consumer.ignore.hive_table.pattern=
>   atlas.notification.consumer.prune.hive_table.pattern=
> 
> 
> Appropriate use of these configurations can avoid loading Atlas with 
> unnecessary metadata of trainsient tables.
> 
> 
> Diffs
> -----
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  23cb853ca 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 0f4857856 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java
>  35b058639 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  e4537b461 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java
>  442a0a0aa 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  003e5b01b 
>   
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
>  PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
>  PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69632/diff/2/
> 
> 
> Testing
> -------
> 
> Verified with the following script that Hive hook and Atlas server 
> ignore/prune metadata for specified Hive tables:
> 
> # configurations for Hive hook
> atlas.hook.hive.ignore.hive_table.pattern=temp\..*,test\..*
> atlas.hook.hive.prune.hive_table.pattern=staging\..*,.*_stg@.*
> 
> 
> # configurations for Atlas server
> atlas.notification.consumer.ignore.hive_table.pattern=temp\..*,test\..*
> atlas.notification.consumer.prune.hive_table.pattern=staging\..*,.*_stg@.*
> 
> CREATE DATABASE IF NOT EXISTS test;
> CREATE DATABASE IF NOT EXISTS temp;
> CREATE DATABASE IF NOT EXISTS staging;
> CREATE DATABASE IF NOT EXISTS prod;
> 
> DROP VIEW  IF EXISTS test.testView;
> DROP TABLE IF EXISTS test.testTable;
> 
> DROP VIEW  IF EXISTS temp.tempView;
> DROP TABLE IF EXISTS temp.tempTable;
> 
> DROP VIEW  IF EXISTS staging.stagingView;
> DROP TABLE IF EXISTS staging.stagingTable;
> 
> DROP VIEW  IF EXISTS prod.prodView;
> DROP TABLE IF EXISTS prod.prodTable;
> DROP TABLE IF EXISTS prod.prodSourceTable;
> 
> DROP VIEW  IF EXISTS prod.myTable_stg;
> DROP TABLE IF EXISTS prod.myView_stg;
> 
> CREATE TABLE test.testTable(id INT, name STRING);
> CREATE VIEW  test.testView AS SELECT * FROM test.testTable;
> 
> CREATE TABLE temp.tempTable(id INT, name STRING);
> CREATE VIEW  temp.tempView AS SELECT * FROM temp.tempTable;
> 
> CREATE TABLE staging.stagingTable(id INT, name STRING);
> CREATE VIEW  staging.stagingView AS SELECT * FROM staging.stagingTable;
> 
> CREATE TABLE prod.prodSourceTable(id INT, name STRING);
> CREATE TABLE prod.prodTable(id INT, name STRING);
> CREATE VIEW  prod.prodView AS SELECT * FROM prod.prodTable;
> 
> CREATE TABLE prod.myTable_stg(id INT, name STRING);
> CREATE VIEW  prod.myView_stg AS SELECT * FROM prod.prodTable;
> 
> INSERT INTO TABLE prod.prodTable SELECT * FROM staging.stagingTable;
> INSERT INTO TABLE prod.prodTable SELECT * FROM temp.tempTable;
> INSERT INTO TABLE prod.prodTable SELECT * FROM prod.myView_stg;
> INSERT INTO TABLE prod.prodTable SELECT * FROM staging.stagingView;
> INSERT INTO TABLE prod.prodTable SELECT * FROM prod.prodSourceTable;
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Reply via email to