Re: Review Request 70505: CDPD-252:parse metadata/Lineage files generated by Impala

[Na Li via Review Board]

> On April 19, 2019, 3:27 p.m., Na Li wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
> > Lines 27 (patched)
> > 
> >
> > can you make the type to be Integer? So it is consistent with the id 
> > type ion LineageEdge "private List sources;"
> 
> Xinran Tinney wrote:
> Thre are 3 elements in LieageVetext 
> private String id;
> private String vertexType;
> private String vertexId;
> 
> The one in private List sources is using the first one "id", I 
> can not make vertexId an Integer because it is not a number. e.g. 
> "vertexId":"db_1.view_1.count"
> 
> Xinran Tinney wrote:
> Do you mean the change "id" to Integer?

sorry. I meant "id" should be Integer. vertexId should be String


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70505/#review214761
---


On April 18, 2019, 8:33 p.m., Xinran Tinney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70505/
> ---
> 
> (Updated April 18, 2019, 8:33 p.m.)
> 
> 
> Review request for atlas and Na Li.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> the code is implemented in Java.
> 
> 1) It takes Impala file location and file name prefix as input. 
> 2) It reads its own record-keeping file to get the offset of the Impala file 
> it has already read. Then read the new records from Impala file, update the 
> record-keeping file to append the new offset of the Impala file.
> 3) It parses the new records from Impala file into internal data structure as 
> output.
> 4) If the Impala file name has changed, start a new file for record-keeping 
> and remove the old record-keeping file.
> 5) Implemented unit test to verify the behavior.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/impala1.txt PRE-CREATION 
>   addons/impala-bridge/impala2.txt PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/ImpalaBridge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/bridge/TestImpalaBridge.java
>  PRE-CREATION 
>   pom.xml 76341388b 
> 
> 
> Diff: https://reviews.apache.org/r/70505/diff/1/
> 
> 
> Testing
> ---
> 
> unit testing and mvn clean install -DskipTests
> 
> 
> Thanks,
> 
> Xinran Tinney
> 
>

Re: Review Request 70505: CDPD-252:parse metadata/Lineage files generated by Impala

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70505/#review214765
---




addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageQuery.java
Lines 31 (patched)


Your code does not work for this lineage record. That is why it is critical 
to finish "Collect the lineage records for all operations that Impala generate 
lineage records" before you can claim you are done with this jira!

{  
   "1554132988258":{  
  "query_id":"544e8c5ab3b6a634:650a9d9e",
  "session_id":"044dbeb34e24aa5d:ad43167db6878180",
  "start_time":"2019-04-01 08:36:27.842860",
  "authorization_failure":false,
  "status":"",
  "user":"imp...@gce.cloudera.com",
  "impersonator":null,
  "statement_type":"CREATE_TABLE",
  "network_address":":::172.31.114.111:41200",
  "sql_statement":"CREATE EXTERNAL TABLE IF NOT EXISTS 
db_atlqkltata.tbl_atlqkltata_1(col_tbl_atlqkltata_1_id 
INT,col_tbl_atlqkltata_1_name STRING,col_tbl_atlqkltata_1_street 
STRING,col_tbl_atlqkltata_1_city STRING,col_tbl_atlqkltata_1_state 
STRING,col_tbl_atlqkltata_1_zipcode INT)",
  "catalog_objects":[  
 {  
"name":"db_atlqkltata.tbl_atlqkltata_1",
"object_type":"TABLE",
"privilege":"CREATE"
 }
  ]
   }
}


- Na Li


On April 19, 2019, 3:51 p.m., Xinran Tinney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70505/
> ---
> 
> (Updated April 19, 2019, 3:51 p.m.)
> 
> 
> Review request for atlas and Na Li.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> the code is implemented in Java.
> 
> 1) It takes Impala file location and file name prefix as input. 
> 2) It reads its own record-keeping file to get the offset of the Impala file 
> it has already read. Then read the new records from Impala file, update the 
> record-keeping file to append the new offset of the Impala file.
> 3) It parses the new records from Impala file into internal data structure as 
> output.
> 4) If the Impala file name has changed, start a new file for record-keeping 
> and remove the old record-keeping file.
> 5) Implemented unit test to verify the behavior.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/impala1.txt PRE-CREATION 
>   addons/impala-bridge/impala2.txt PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/ImpalaBridge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/bridge/TestImpalaBridge.java
>  PRE-CREATION 
>   pom.xml 76341388b 
> 
> 
> Diff: https://reviews.apache.org/r/70505/diff/2/
> 
> 
> Testing
> ---
> 
> unit testing and mvn clean install -DskipTests
> 
> 
> Thanks,
> 
> Xinran Tinney
> 
>

Re: Review Request 70505: CDPD-252:parse metadata/Lineage files generated by Impala

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70505/#review214761
---




addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
Lines 27 (patched)


can you make the type to be Integer? So it is consistent with the id type 
ion LineageEdge "private List sources;"


- Na Li


On April 18, 2019, 8:33 p.m., Xinran Tinney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70505/
> ---
> 
> (Updated April 18, 2019, 8:33 p.m.)
> 
> 
> Review request for atlas and Na Li.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> the code is implemented in Java.
> 
> 1) It takes Impala file location and file name prefix as input. 
> 2) It reads its own record-keeping file to get the offset of the Impala file 
> it has already read. Then read the new records from Impala file, update the 
> record-keeping file to append the new offset of the Impala file.
> 3) It parses the new records from Impala file into internal data structure as 
> output.
> 4) If the Impala file name has changed, start a new file for record-keeping 
> and remove the old record-keeping file.
> 5) Implemented unit test to verify the behavior.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/impala1.txt PRE-CREATION 
>   addons/impala-bridge/impala2.txt PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/ImpalaBridge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/bridge/TestImpalaBridge.java
>  PRE-CREATION 
>   pom.xml 76341388b 
> 
> 
> Diff: https://reviews.apache.org/r/70505/diff/1/
> 
> 
> Testing
> ---
> 
> unit testing and mvn clean install -DskipTests
> 
> 
> Thanks,
> 
> Xinran Tinney
> 
>