> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 204 (patched)
> > <https://reviews.apache.org/r/70512/diff/13/?file=2142467#file2142467line204>
> >
> > dependencyType can not always be be SIMPLE;
> > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageInfo.java#L59
Impala dependencyType can only be Projection or Predicate. We should skip the
dependency for Predicate. So the dependency type for Impala column lineage will
always Projection. Impala does not differentiate simple, expression or script
as Hive does.
// based on
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java#L267
// There are two types of dependencies that are represented as
edges in the column
// lineage graph:
// a) Projection dependency: This is a dependency between a set
of source
// columns (base table columns) and a single target (result expr
or table column).
// This dependency indicates that values of the target depend on
the values of the source
// columns.
// b) Predicate dependency: This is a dependency between a set
of target
// columns (or exprs) and a set of source columns (base table
columns). It indicates that
// the source columns restrict the values of their targets (e.g.
by participating in
// WHERE clause predicates).
> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
> > Lines 26 (patched)
> > <https://reviews.apache.org/r/70512/diff/13/?file=2142471#file2142471line26>
> >
> > consider removing explicit "private" modifier.
done
> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
> > Lines 38 (patched)
> > <https://reviews.apache.org/r/70512/diff/13/?file=2142472#file2142472line38>
> >
> > make edgeType as enum
done
> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
> > Lines 36 (patched)
> > <https://reviews.apache.org/r/70512/diff/13/?file=2142474#file2142474line36>
> >
> > consider changing "id" type from Integer to Long.
good catch! done.
// https://github.com/apache/impala/blob/master/be/src/util/lineage-util.h#L40
// Impala id is int64. Therefore, define this field as Long
- Na
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215013
-----------------------------------------------------------
On May 2, 2019, 8:31 p.m., Na Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> -----------------------------------------------------------
>
> (Updated May 2, 2019, 8:31 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath
> Subramanian, and Xinran Tinney.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Impala generates lineage records for its commands. This new feature will read
> Impala lineage file, convert the lineage record to Atlas entities and send
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
>
> The metadata referred in the lineage are captured in Hive Metastore hook and
> sent to Atlas. This work is done in ATLAS-3148
>
> This jira only supports the Impala command "create view". Following jira will
> add support for more Impala commands.
>
>
> Diffs
> -----
>
> addons/impala-bridge-shim/pom.xml PRE-CREATION
>
> addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> PRE-CREATION
> addons/impala-bridge/pom.xml PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
> PRE-CREATION
>
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
> PRE-CREATION
> addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION
> addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION
>
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
> PRE-CREATION
>
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
> PRE-CREATION
>
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
> PRE-CREATION
> addons/impala-bridge/src/test/resources/atlas-application.properties
> PRE-CREATION
> addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION
> addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION
> addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION
> addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION
> addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION
> addons/impala-bridge/src/test/resources/users-credentials.properties
> PRE-CREATION
> pom.xml 7de5d31
>
>
> Diff: https://reviews.apache.org/r/70512/diff/13/
>
>
> Testing
> -------
>
> Run the tool in real cluster that has Atlas server with Impala lineage file
> as input for creating view. The Atlas UI displays hive_lineage lineage and
> hive_column_lineage.
> Add new integration tests and they pass
>
>
> Thanks,
>
> Na Li
>
>
