----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/ -----------------------------------------------------------
(Updated May 8, 2019, 7:20 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath Subramanian, and Xinran Tinney. Repository: atlas Description ------- Impala generates lineage records for its commands. This new feature will read Impala lineage file, convert the lineage record to Atlas entities and send them to Atlas. In this way, Atlas can get lineage of Impala operation. The metadata referred in the lineage are captured in Hive Metastore hook and sent to Atlas. This work is done in ATLAS-3148 This jira only supports the Impala command "create view". Following jira will add support for more Impala commands. Diffs (updated) ----- addons/impala-bridge/pom.xml PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java PRE-CREATION addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java PRE-CREATION addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java PRE-CREATION addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java PRE-CREATION addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java PRE-CREATION addons/impala-bridge/src/test/resources/atlas-application.properties PRE-CREATION addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION addons/impala-bridge/src/test/resources/users-credentials.properties PRE-CREATION addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION pom.xml ae4dfdc Diff: https://reviews.apache.org/r/70512/diff/15/ Changes: https://reviews.apache.org/r/70512/diff/14-15/ Testing ------- Run the tool in real cluster that has Atlas server with Impala lineage file as input for creating view. The Atlas UI displays hive_lineage lineage and hive_column_lineage. Add new integration tests and they pass Thanks, Na Li
