-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
-----------------------------------------------------------
(Updated May 21, 2019, 10:10 p.m.)
Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath
Subramanian.
Repository: atlas
Description
-------
Integrate with Impala hook using Impala private jar with post exec hook
changes:
1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex
{
"id":4,
"vertexType":"TABLE",
"vertexId":"db_1.table_1",
"createTime":1554750000
}
To embed the table create time inside column vertex
{
"id": 3,
"vertexType": "COLUMN",
"vertexId": "db_1.table_1.tinyint_col",
"metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
}
}
2) add Impala hook API code to allow Atlas integration code to compile without
Impala updates
Diffs
-----
addons/impala-bridge-shim/pom.xml PRE-CREATION
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
PRE-CREATION
addons/impala-bridge/pom.xml 8e7e952
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
PRE-CREATION
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
fdb6748
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
afe296c
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
e4a38a3
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
7c1103a
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
82672c9
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
PRE-CREATION
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
cc62955
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
6e4d321
addons/impala-bridge/src/test/resources/impala3.json 6a7d171
addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION
addons/impala-bridge/src/test/resources/impala5.json 854969b
addons/impala-bridge/src/test/resources/impala6.json f136180
addons/impala-bridge/src/test/resources/impala7.json f9ee670
addons/impala-hook-api/pom.xml PRE-CREATION
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
PRE-CREATION
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
PRE-CREATION
pom.xml 98c2d69
Diff: https://reviews.apache.org/r/70619/diff/8/
Testing
-------
add integration tests
ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the
situation that there is no createTime in lineage record
change the file content for other integration tests in ImpalaLineageToolIT to
handle new lineage record format
Thanks,
Na Li
