[jira] [Commented] (ATLAS-3969) Atlas1.2.0INSERT_SELECT Lineage can be retrieved but not CREATE_TABLE_AS_SELECT
[
https://issues.apache.org/jira/browse/ATLAS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419565#comment-17419565
]
aflyary commented on ATLAS-3969:
the sql u use maybe wrong, pls check.
{code:java}
// CREATE TABLE test.t5(
id String,
name String
)
CREATE TABLE test.t6 AS SELECT id,name,age,phone FROM test.t5;
{code}
> Atlas1.2.0INSERT_SELECT Lineage can be retrieved but not
> CREATE_TABLE_AS_SELECT
> ---
>
> Key: ATLAS-3969
> URL: https://issues.apache.org/jira/browse/ATLAS-3969
> Project: Atlas
> Issue Type: Bug
> Components: hive-integration
>Affects Versions: 1.2.0
>Reporter: zhangsan
>Priority: Major
> Labels: newbie
>
> CREATE TABLE test.t5(
> id String,
> name String
> )
> CREATE TABLE test.t6 AS SELECT id,name,age,phone FROM test.t5;
> The above operation cannot obtain the lineage
>
> INSERT OVERWRITE TABLE test.t6 SELECT id,name FROM test.t5;
> The above operation can obtain the Lineage
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3969) Atlas1.2.0INSERT_SELECT Lineage can be retrieved but not CREATE_TABLE_AS_SELECT
[
https://issues.apache.org/jira/browse/ATLAS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419565#comment-17419565
]
aflyary edited comment on ATLAS-3969 at 9/24/21, 5:21 AM:
--
the sql u use maybe wrong, pls check.
{code:java}
CREATE TABLE test.t5(
id String,
name String
)
CREATE TABLE test.t6 AS SELECT id,name,age,phone FROM test.t5;
{code}
was (Author: aflyary):
the sql u use maybe wrong, pls check.
{code:java}
// CREATE TABLE test.t5(
id String,
name String
)
CREATE TABLE test.t6 AS SELECT id,name,age,phone FROM test.t5;
{code}
> Atlas1.2.0INSERT_SELECT Lineage can be retrieved but not
> CREATE_TABLE_AS_SELECT
> ---
>
> Key: ATLAS-3969
> URL: https://issues.apache.org/jira/browse/ATLAS-3969
> Project: Atlas
> Issue Type: Bug
> Components: hive-integration
>Affects Versions: 1.2.0
>Reporter: zhangsan
>Priority: Major
> Labels: newbie
>
> CREATE TABLE test.t5(
> id String,
> name String
> )
> CREATE TABLE test.t6 AS SELECT id,name,age,phone FROM test.t5;
> The above operation cannot obtain the lineage
>
> INSERT OVERWRITE TABLE test.t6 SELECT id,name FROM test.t5;
> The above operation can obtain the Lineage
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419563#comment-17419563 ] aflyary commented on ATLAS-2893: [~xiaozhang_is] could u please show your sql which run under the hive version 1.2.0 and 0.12.0 ? > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aflyary updated ATLAS-2893: --- Comment: was deleted (was: Could somebody help to review this patch ? Thanks a lot.:)) > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632841#comment-16632841 ] aflyary commented on ATLAS-2893: Could somebody help to review this patch ? Thanks a lot.:) > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632816#comment-16632816 ] aflyary edited comment on ATLAS-2893 at 9/29/18 6:23 AM: - Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the method to get the column lineage from the hive lineageState of SessionState, it could work with the pre-released hive version before hive-1.2.1. I test the code with the hive version 1.2.0 and 0.12.0, it could get the hive column lineage correctly. The hive sql which i test are below: CREATE TABLE table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; CREATE TABLE table2 AS SELECT id,name,age,address FROM table1; was (Author: aflyary): Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the method to get the column lineage from the hive lineageState of SessionState, it could work with the pre-released hive version before hive-1.2.1. I test the code with the hive version 1.2.0 and 0.12.0, it could get the hive column lineage correctly. The hive sql which i test are below: create table table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; create table table2 as select id,name,age,address from table1; > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632816#comment-16632816 ] aflyary commented on ATLAS-2893: Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the method to get the column lineage from the hive lineageState of SessionState, it could work with the pre-released hive version before hive-1.2.1. I test the code with the hive version 1.2.0 and 0.12.0, it could get the hive column lineage correctly. The hive sql which i test are below: create table table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; create table table2 as select id,name,age,address from table1; > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aflyary updated ATLAS-2893: --- Attachment: ATLAS-2893.001.patch > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > Attachments: ATLAS-2893.001.patch > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aflyary updated ATLAS-2893: --- Attachment: (was: ATLAS-2893.001.patch) > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aflyary updated ATLAS-2893: --- Attachment: ATLAS-2893.001.patch > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
[ https://issues.apache.org/jira/browse/ATLAS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631283#comment-16631283 ] aflyary commented on ATLAS-2893: I refer to the hive source code, when we set the altas hook class at the hive-site.xml with the name 'org.apache.atlas.hive.hook.HiveHook', hive optimizer would not set the linegae info into the hookContext, after the patch HIVE-13112 already fix this. But early hive version still could not work, so i go ahead and dive into the hive lineageState of SessionState, maybe we could get the private info of the LineageCtx to work out the lineage dependency, then get the column lineage. > Atlas Column Lineage of Hive Hook to support the hive old version (before > hive version 1.2.1 after the patch for HIVE-13112) > > > Key: ATLAS-2893 > URL: https://issues.apache.org/jira/browse/ATLAS-2893 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 1.0.0 >Reporter: aflyary >Priority: Major > Fix For: 1.1.0 > > > Right now atlas hook for apache hive of column lineage is limited, > only "Column level lineage works with Hive version 1.2.1 after the patch for > HIVE-13112 is applied to Hive source" could support the hive column lineage. > In some product env, hive version still very old, but we should also > support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-2893) Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112)
aflyary created ATLAS-2893: -- Summary: Atlas Column Lineage of Hive Hook to support the hive old version (before hive version 1.2.1 after the patch for HIVE-13112) Key: ATLAS-2893 URL: https://issues.apache.org/jira/browse/ATLAS-2893 Project: Atlas Issue Type: Improvement Components: atlas-intg Affects Versions: 1.0.0 Reporter: aflyary Fix For: 1.1.0 Right now atlas hook for apache hive of column lineage is limited, only "Column level lineage works with Hive version 1.2.1 after the patch for HIVE-13112 is applied to Hive source" could support the hive column lineage. In some product env, hive version still very old, but we should also support the hive column lineage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ATLAS-2791) There is no column Lineage Information with atlas 1.0 and hive 2.1.1
[ https://issues.apache.org/jira/browse/ATLAS-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628333#comment-16628333 ] aflyary edited comment on ATLAS-2791 at 9/26/18 7:10 AM: - i thought it is very useful to support the column lineage of low version hive was (Author: aflyary): i thought it is very import to support the low version hive colmn lineage > There is no column Lineage Information with atlas 1.0 and hive 2.1.1 > > > Key: ATLAS-2791 > URL: https://issues.apache.org/jira/browse/ATLAS-2791 > Project: Atlas > Issue Type: Bug > Environment: Atlas 1.0 Hive 2.1.1 >Reporter: Helios.He >Priority: Major > > There is no column Lineage Information with atlas 1.0 and hive 2.1.1. but I > change hive version to 2.3.3 the Lineage Information is generated. Does > atlas 1.0 not match hive 2.1.1? If not, what version of Atlas should I use > which match hive 2.1.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2791) There is no column Lineage Information with atlas 1.0 and hive 2.1.1
[ https://issues.apache.org/jira/browse/ATLAS-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628333#comment-16628333 ] aflyary commented on ATLAS-2791: i thought it is very import to support the low version hive colmn lineage > There is no column Lineage Information with atlas 1.0 and hive 2.1.1 > > > Key: ATLAS-2791 > URL: https://issues.apache.org/jira/browse/ATLAS-2791 > Project: Atlas > Issue Type: Bug > Environment: Atlas 1.0 Hive 2.1.1 >Reporter: Helios.He >Priority: Major > > There is no column Lineage Information with atlas 1.0 and hive 2.1.1. but I > change hive version to 2.3.3 the Lineage Information is generated. Does > atlas 1.0 not match hive 2.1.1? If not, what version of Atlas should I use > which match hive 2.1.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2791) There is no column Lineage Information with atlas 1.0 and hive 2.1.1
[ https://issues.apache.org/jira/browse/ATLAS-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628278#comment-16628278 ] aflyary commented on ATLAS-2791: on the atlas apache web page site describe the column lineage with such info: * "Column level lineage works with Hive version 1.2.1 after the patch for HIVE-13112 is applied to Hive source" u could try this: create table table1(id int, name string, age int, address string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; create table table2 as select id,name,age,address from table1; create table table3 as select id,name,age,address from table2; then u could check the table2 whether has the column lineage > There is no column Lineage Information with atlas 1.0 and hive 2.1.1 > > > Key: ATLAS-2791 > URL: https://issues.apache.org/jira/browse/ATLAS-2791 > Project: Atlas > Issue Type: Bug > Environment: Atlas 1.0 Hive 2.1.1 >Reporter: Helios.He >Priority: Major > > There is no column Lineage Information with atlas 1.0 and hive 2.1.1. but I > change hive version to 2.3.3 the Lineage Information is generated. Does > atlas 1.0 not match hive 2.1.1? If not, what version of Atlas should I use > which match hive 2.1.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
