Madhan Neethiraj created ATLAS-3006:
---------------------------------------
Summary: Option to ignore/prune metadata for temporary/staging
Hive tables
Key: ATLAS-3006
URL: https://issues.apache.org/jira/browse/ATLAS-3006
Project: Atlas
Issue Type: Improvement
Components: atlas-core
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj
It is not uncommon for a Hive deployment to use a large number of
staging/temporary tables, which are created periodically to load data into
target tables and deleted after completion of data load. A large number of
entities are created in Atlas for these staging/temporary tables
(tables/columns/column-lineage).
For staging tables, it is probably not useful to track details like columns and
column-lineage in Atlas. Not tracking these details in Atlas can significantly
reduce the time it takes to process notifications, and can help in improving
the performance overall. Only minimum details of these staging tables can be
stored in Atlas, to capture data lineage from source to target table via all
intermediate staging tables.
Also, it will be helpful to good to ignore tables that are created & deleted
during data loading i.e. temporary tables.
Configurations should be provided to specify which of the tables are
staging/temporary. In addition to supporting this in Hive hook (to avoid
generation of large messages for staging/temporary tables), Atlas server should
also be updated, to control this further at server side while processing
notifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)