-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
-----------------------------------------------------------
(Updated May 22, 2019, 2:15 a.m.)
Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath
Subramanian.
Repository: atlas
Description
-------
Integrate with Impala hook using Impala private jar with post exec hook
changes:
1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex
{
"id":4,
"vertexType":"TABLE",
"vertexId":"db_1.table_1",
"createTime":1554750000
}
To embed the table create time inside column vertex
{
"id": 3,
"vertexType": "COLUMN",
"vertexId": "db_1.table_1.tinyint_col",
"metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
}
}
2) add Impala hook API code to allow Atlas integration code to compile without
Impala updates
Diffs (updated)
-----
addons/impala-bridge-shim/pom.xml PRE-CREATION
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
PRE-CREATION
addons/impala-bridge/pom.xml 8e7e952
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
7c9abc8
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
88faace
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
b9d6cbb
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
fdb6748
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
b9dd894
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
afe296c
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
e4a38a3
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
7c1103a
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
10ce448
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
892ee9b
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
a3ddf53
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
a893b88
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
27bdc72
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
8ec3f85
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
251507e
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
82672c9
addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDependencyType.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaNode.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaOperationType.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaQuery.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaVertexType.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageEdge.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertex.java
PRE-CREATION
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertexMetadata.java
PRE-CREATION
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
cc62955
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
6e4d321
addons/impala-bridge/src/test/resources/impala1.json 8f747f6
addons/impala-bridge/src/test/resources/impala2.json 239797b
addons/impala-bridge/src/test/resources/impala3.json 6a7d171
addons/impala-bridge/src/test/resources/impala5.json 854969b
addons/impala-bridge/src/test/resources/impala6.json f136180
addons/impala-bridge/src/test/resources/impala7.json f9ee670
addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json
PRE-CREATION
addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json
PRE-CREATION
addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION
addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json
PRE-CREATION
addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json
PRE-CREATION
addons/impala-hook-api/pom.xml PRE-CREATION
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
PRE-CREATION
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
PRE-CREATION
pom.xml 98c2d69
Diff: https://reviews.apache.org/r/70619/diff/9/
Changes: https://reviews.apache.org/r/70619/diff/8-9/
Testing
-------
add integration tests
ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the
situation that there is no createTime in lineage record
change the file content for other integration tests in ImpalaLineageToolIT to
handle new lineage record format
Thanks,
Na Li
