[jira] [Commented] (ATLAS-2903) Atlas should support get the change records of entities
[ https://issues.apache.org/jira/browse/ATLAS-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633034#comment-16633034 ] Ayush Nigam commented on ATLAS-2903: [~wangtao_20180929] So basically you want to get the history of type updations right? Not entity updations So as per my understanding you want some kind of list of types with their versions,basically a list of map of version and type definition? You want the change history of entitydef,enumdef etc? > Atlas should support get the change records of entities > --- > > Key: ATLAS-2903 > URL: https://issues.apache.org/jira/browse/ATLAS-2903 > Project: Atlas > Issue Type: Wish > Components: atlas-core >Reporter: wangtao >Priority: Major > > Atlas should support get the change records of entities. Such as to modify > the metadata of a property or to increase a field of a table, and later can > query the change history. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aflyary updated ATLAS-2893: --- Comment: was deleted (was: Could somebody help to review this patch ? Thanks a lot.:)) > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68883: ATLAS-2893: Atlas Column Lineage of Hive Hook to support the hive old version
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68883/ --- (Updated Sept. 29, 2018, 7:04 a.m.) Review request for atlas, Apoorv Naik, Ashutosh Mestry, keval bhatt, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-2893 https://issues.apache.org/jira/browse/ATLAS-2893 Repository: atlas Description --- Right now atlas hook for apache hive of column lineage is limited, only "Column level lineage works with Hive version 1.2.1 after the patch for HIVE-13112 is applied to Hive source" could support the hive column lineage. In some product env, hive version still very old, but we should also support the hive column lineage. Diffs - addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java ae01d504d Diff: https://reviews.apache.org/r/68883/diff/1/ Testing --- I refer to the hive source code, when we set the altas hook class at the hive-site.xml with the name 'org.apache.atlas.hive.hook.HiveHook', hive optimizer would not set the linegae info into the hookContext, after the patch HIVE-13112 already fix this. But early hive version still could not work, so i go ahead and dive into the hive lineageState of SessionState, maybe we could get the private info of the LineageCtx to work out the lineage dependency, then get the column lineage. Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the method to get the column lineage from the hive lineageState of SessionState, it could work with the pre-released hive version before hive-1.2.1. I test the code with the hive version 1.2.0 and 0.12.0, it could get the hive column lineage correctly. The hive sql which i test are below: CREATE TABLE table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; CREATE TABLE table2 AS SELECT id,name,age,address FROM table1; Thanks, aflyary
Re: Review Request 68883: ATLAS-2893: Atlas Column Lineage of Hive Hook to support the hive old version
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68883/ --- (Updated Sept. 29, 2018, 6:56 a.m.) Review request for atlas. Summary (updated) - ATLAS-2893: Atlas Column Lineage of Hive Hook to support the hive old version Repository: atlas Description --- Right now atlas hook for apache hive of column lineage is limited, only "Column level lineage works with Hive version 1.2.1 after the patch for HIVE-13112 is applied to Hive source" could support the hive column lineage. In some product env, hive version still very old, but we should also support the hive column lineage. Diffs - addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java ae01d504d Diff: https://reviews.apache.org/r/68883/diff/1/ Testing --- I refer to the hive source code, when we set the altas hook class at the hive-site.xml with the name 'org.apache.atlas.hive.hook.HiveHook', hive optimizer would not set the linegae info into the hookContext, after the patch HIVE-13112 already fix this. But early hive version still could not work, so i go ahead and dive into the hive lineageState of SessionState, maybe we could get the private info of the LineageCtx to work out the lineage dependency, then get the column lineage. Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the method to get the column lineage from the hive lineageState of SessionState, it could work with the pre-released hive version before hive-1.2.1. I test the code with the hive version 1.2.0 and 0.12.0, it could get the hive column lineage correctly. The hive sql which i test are below: CREATE TABLE table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; CREATE TABLE table2 AS SELECT id,name,age,address FROM table1; Thanks, aflyary
Review Request 68883: ATLAS-2893?Atlas Column Lineage of Hive Hook to support the hive old version
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68883/ --- Review request for atlas. Repository: atlas Description --- Right now atlas hook for apache hive of column lineage is limited, only "Column level lineage works with Hive version 1.2.1 after the patch for HIVE-13112 is applied to Hive source" could support the hive column lineage. In some product env, hive version still very old, but we should also support the hive column lineage. Diffs - addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java ae01d504d Diff: https://reviews.apache.org/r/68883/diff/1/ Testing --- I refer to the hive source code, when we set the altas hook class at the hive-site.xml with the name 'org.apache.atlas.hive.hook.HiveHook', hive optimizer would not set the linegae info into the hookContext, after the patch HIVE-13112 already fix this. But early hive version still could not work, so i go ahead and dive into the hive lineageState of SessionState, maybe we could get the private info of the LineageCtx to work out the lineage dependency, then get the column lineage. Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the method to get the column lineage from the hive lineageState of SessionState, it could work with the pre-released hive version before hive-1.2.1. I test the code with the hive version 1.2.0 and 0.12.0, it could get the hive column lineage correctly. The hive sql which i test are below: CREATE TABLE table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; CREATE TABLE table2 AS SELECT id,name,age,address FROM table1; Thanks, aflyary
[jira] [Commented] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632841#comment-16632841 ] aflyary commented on ATLAS-2893: Could somebody help to review this patch ? Thanks a lot.:) > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632816#comment-16632816 ] aflyary edited comment on ATLAS-2893 at 9/29/18 6:23 AM: - Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the method to get the column lineage from the hive lineageState of SessionState, it could work with the pre-released hive version before hive-1.2.1. I test the code with the hive version 1.2.0 and 0.12.0, it could get the hive column lineage correctly. The hive sql which i test are below: CREATE TABLE table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; CREATE TABLE table2 AS SELECT id,name,age,address FROM table1; was (Author: aflyary): Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the method to get the column lineage from the hive lineageState of SessionState, it could work with the pre-released hive version before hive-1.2.1. I test the code with the hive version 1.2.0 and 0.12.0, it could get the hive column lineage correctly. The hive sql which i test are below: create table table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; create table table2 as select id,name,age,address from table1; > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
