[jira] [Assigned] (ATLAS-3187) Regression, Relationship updates : In GET entity definition of a deleted table, columns,sd etc., are empty

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian reassigned ATLAS-3187:
-

Assignee: Madhan Neethiraj

> Regression, Relationship updates : In GET entity definition of a deleted 
> table, columns,sd etc., are empty
> --
>
> Key: ATLAS-3187
> URL: https://issues.apache.org/jira/browse/ATLAS-3187
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sharmadha Sainath
>Assignee: Madhan Neethiraj
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: table_entity_def.json
>
>
> Adding the table_entity_def["entity"]["attributes"]:
> Here , table is deleted , but the columns array and sd are [] and null 
> respectively. Expected is all entities in DELETED  state.
> Attaching the complete table definition :
> {code}
> attributes: {
> owner: "hrt_qa",
> temporary: false,
> lastAccessTime: 1556788686000,
> aliases: null,
> replicatedTo: null,
> replicatedFrom: null,
> qualifiedName: "default.table1@cl1",
> columns: [ ],
> description: null,
> viewExpandedText: null,
> tableType: "MANAGED_TABLE",
> sd: null,
> createTime: 1556788686000,
> name: "table1",
> comment: null,
> partitionKeys: [ ],
> parameters: {
> totalSize: "0",
> numRows: "0",
> rawDataSize: "0",
> transactional_properties: "default",
> COLUMN_STATS_ACCURATE: "{"BASIC_STATS":"true","COLUMN_STATS":{"id":"true"}}",
> numFiles: "0",
> transient_lastDdlTime: "1556788686",
> bucketing_version: "2",
> transactional: "true"
> },
> retention: 0,
> viewOriginalText: null,
> db: {
> guid: "ab02dd3b-1d1c-4522-8c2c-0fa60d82fcbe",
> typeName: "hive_db"
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3155) UI - IE-11 - Name entry not clickable

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832257#comment-16832257
 ] 

ASF subversion and git services commented on ATLAS-3155:


Commit 6da3bcab11b45df4960bbccff03ff5960741b266 in atlas's branch 
refs/heads/branch-1.0 from Keval Bhatt
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=6da3bca ]

ATLAS-3155: UI - IE-11 - Name entry not clickable (#2)

Signed-off-by: Madhan Neethiraj 
(cherry picked from commit bd7f5eecf8cadae392adb948e3492ce6c1c97bbf)


> UI - IE-11 - Name entry not clickable
> -
>
> Key: ATLAS-3155
> URL: https://issues.apache.org/jira/browse/ATLAS-3155
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
> Attachments: ATLAS-3155-2.patch, ATLAS-3155.patch, ie11_name_issue.jpg
>
>
> In the search results page, when we try to click the 'Name' column value of 
> any entry, nothing happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3155) UI - IE-11 - Name entry not clickable

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832256#comment-16832256
 ] 

ASF subversion and git services commented on ATLAS-3155:


Commit 37ae1d5d257bcf358a747659da3d20e038958ac8 in atlas's branch 
refs/heads/branch-2.0 from Keval Bhatt
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=37ae1d5 ]

ATLAS-3155: UI - IE-11 - Name entry not clickable (#2)

Signed-off-by: Madhan Neethiraj 
(cherry picked from commit bd7f5eecf8cadae392adb948e3492ce6c1c97bbf)


> UI - IE-11 - Name entry not clickable
> -
>
> Key: ATLAS-3155
> URL: https://issues.apache.org/jira/browse/ATLAS-3155
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
> Attachments: ATLAS-3155-2.patch, ATLAS-3155.patch, ie11_name_issue.jpg
>
>
> In the search results page, when we try to click the 'Name' column value of 
> any entry, nothing happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3155) UI - IE-11 - Name entry not clickable

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832255#comment-16832255
 ] 

ASF subversion and git services commented on ATLAS-3155:


Commit bd7f5eecf8cadae392adb948e3492ce6c1c97bbf in atlas's branch 
refs/heads/master from Keval Bhatt
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=bd7f5ee ]

ATLAS-3155: UI - IE-11 - Name entry not clickable (#2)

Signed-off-by: Madhan Neethiraj 


> UI - IE-11 - Name entry not clickable
> -
>
> Key: ATLAS-3155
> URL: https://issues.apache.org/jira/browse/ATLAS-3155
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
> Attachments: ATLAS-3155-2.patch, ATLAS-3155.patch, ie11_name_issue.jpg
>
>
> In the search results page, when we try to click the 'Name' column value of 
> any entry, nothing happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (ATLAS-3187) Regression, Relationship updates : In GET entity definition of a deleted table, columns,sd etc., are empty

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj resolved ATLAS-3187.
-
Resolution: Fixed

> Regression, Relationship updates : In GET entity definition of a deleted 
> table, columns,sd etc., are empty
> --
>
> Key: ATLAS-3187
> URL: https://issues.apache.org/jira/browse/ATLAS-3187
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sharmadha Sainath
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: table_entity_def.json
>
>
> Adding the table_entity_def["entity"]["attributes"]:
> Here , table is deleted , but the columns array and sd are [] and null 
> respectively. Expected is all entities in DELETED  state.
> Attaching the complete table definition :
> {code}
> attributes: {
> owner: "hrt_qa",
> temporary: false,
> lastAccessTime: 1556788686000,
> aliases: null,
> replicatedTo: null,
> replicatedFrom: null,
> qualifiedName: "default.table1@cl1",
> columns: [ ],
> description: null,
> viewExpandedText: null,
> tableType: "MANAGED_TABLE",
> sd: null,
> createTime: 1556788686000,
> name: "table1",
> comment: null,
> partitionKeys: [ ],
> parameters: {
> totalSize: "0",
> numRows: "0",
> rawDataSize: "0",
> transactional_properties: "default",
> COLUMN_STATS_ACCURATE: "{"BASIC_STATS":"true","COLUMN_STATS":{"id":"true"}}",
> numFiles: "0",
> transient_lastDdlTime: "1556788686",
> bucketing_version: "2",
> transactional: "true"
> },
> retention: 0,
> viewOriginalText: null,
> db: {
> guid: "ab02dd3b-1d1c-4522-8c2c-0fa60d82fcbe",
> typeName: "hive_db"
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3155) UI - IE-11 - Name entry not clickable

[Madhan Neethiraj (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832252#comment-16832252
 ] 

Madhan Neethiraj commented on ATLAS-3155:
-

+1 for  [^ATLAS-3155-2.patch]. Thanks [~kevalbhatt] for the patch.

> UI - IE-11 - Name entry not clickable
> -
>
> Key: ATLAS-3155
> URL: https://issues.apache.org/jira/browse/ATLAS-3155
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
> Attachments: ATLAS-3155-2.patch, ATLAS-3155.patch, ie11_name_issue.jpg
>
>
> In the search results page, when we try to click the 'Name' column value of 
> any entry, nothing happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3188) Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3188:
--
Fix Version/s: trunk

> Regression: Simple insert queries are being captured in ATLAS_HOOK topic
> 
>
> Key: ATLAS-3188
> URL: https://issues.apache.org/jira/browse/ATLAS-3188
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Padashetty
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.0.0, trunk
>
>
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3155) UI - IE-11 - Name entry not clickable

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-3155:
---
Attachment: ATLAS-3155-2.patch

> UI - IE-11 - Name entry not clickable
> -
>
> Key: ATLAS-3155
> URL: https://issues.apache.org/jira/browse/ATLAS-3155
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
> Attachments: ATLAS-3155-1.patch, ATLAS-3155-2.patch, 
> ATLAS-3155.patch, ie11_name_issue.jpg
>
>
> In the search results page, when we try to click the 'Name' column value of 
> any entry, nothing happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3188) Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832245#comment-16832245
 ] 

ASF subversion and git services commented on ATLAS-3188:


Commit 41e14045d2dadffc71d1fbd44e40b19d52caa2a1 in atlas's branch 
refs/heads/branch-2.0 from Sarath Subramanian
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=41e1404 ]

ATLAS-3188: Regression: Simple insert queries are being captured in ATLAS_HOOK 
topic

(cherry picked from commit fcca791b60364eb0fd5ca3b74c0f52f8f781cf64)


> Regression: Simple insert queries are being captured in ATLAS_HOOK topic
> 
>
> Key: ATLAS-3188
> URL: https://issues.apache.org/jira/browse/ATLAS-3188
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Padashetty
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.0.0
>
>
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3155) UI - IE-11 - Name entry not clickable

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-3155:
---
Attachment: (was: ATLAS-3155-1.patch)

> UI - IE-11 - Name entry not clickable
> -
>
> Key: ATLAS-3155
> URL: https://issues.apache.org/jira/browse/ATLAS-3155
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
> Attachments: ATLAS-3155-2.patch, ATLAS-3155.patch, ie11_name_issue.jpg
>
>
> In the search results page, when we try to click the 'Name' column value of 
> any entry, nothing happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3155) UI - IE-11 - Name entry not clickable

[Keval Bhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832246#comment-16832246
 ] 

Keval Bhatt commented on ATLAS-3155:


Added new patch [^ATLAS-3155-2.patch] please review

> UI - IE-11 - Name entry not clickable
> -
>
> Key: ATLAS-3155
> URL: https://issues.apache.org/jira/browse/ATLAS-3155
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
> Attachments: ATLAS-3155-2.patch, ATLAS-3155.patch, ie11_name_issue.jpg
>
>
> In the search results page, when we try to click the 'Name' column value of 
> any entry, nothing happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3188) Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832244#comment-16832244
 ] 

ASF subversion and git services commented on ATLAS-3188:


Commit fcca791b60364eb0fd5ca3b74c0f52f8f781cf64 in atlas's branch 
refs/heads/master from Sarath Subramanian
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=fcca791 ]

ATLAS-3188: Regression: Simple insert queries are being captured in ATLAS_HOOK 
topic


> Regression: Simple insert queries are being captured in ATLAS_HOOK topic
> 
>
> Key: ATLAS-3188
> URL: https://issues.apache.org/jira/browse/ATLAS-3188
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Padashetty
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.0.0
>
>
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70588: ATLAS-3188: Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70588/#review215012
---


Ship it!




Ship It!

- Madhan Neethiraj


On May 3, 2019, 4:36 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70588/
> ---
> 
> (Updated May 3, 2019, 4:36 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-3188
> https://issues.apache.org/jira/browse/ATLAS-3188
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  54cd48f04 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  f378d8457 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
>  cdb4cd218 
> 
> 
> Diff: https://reviews.apache.org/r/70588/diff/2/
> 
> 
> Testing
> ---
> 
> validated that "insert into values" query doesn't create hive_process entities
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 70588: ATLAS-3188: Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70588/
---

(Updated May 2, 2019, 9:36 p.m.)


Review request for atlas, Ashutosh Mestry and Madhan Neethiraj.


Bugs: ATLAS-3188
https://issues.apache.org/jira/browse/ATLAS-3188


Repository: atlas


Description
---

insert into  values () is triggering messages in ATLAS_HOOK 
topic. Due to this, Atlas is reporting the operations in Audits. 

Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
they do not play any role in lineage 

Inserting 100 rows in a table is generating 100 messages in the kafka topic.


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 54cd48f04 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 f378d8457 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
 cdb4cd218 


Diff: https://reviews.apache.org/r/70588/diff/2/

Changes: https://reviews.apache.org/r/70588/diff/1-2/


Testing
---

validated that "insert into values" query doesn't create hive_process entities


Thanks,

Sarath Subramanian

[jira] [Commented] (ATLAS-3188) Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Ashutosh Mestry (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832243#comment-16832243
 ] 

Ashutosh Mestry commented on ATLAS-3188:


[~sarath.ku...@gmail.com] / [~umesh.padashetty] Is there a bug that tracks the 
temp table creation?

> Regression: Simple insert queries are being captured in ATLAS_HOOK topic
> 
>
> Key: ATLAS-3188
> URL: https://issues.apache.org/jira/browse/ATLAS-3188
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Padashetty
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.0.0
>
>
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3182) Import Transforms: Hive Table Qualified Name is Modified Incorrectly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832238#comment-16832238
 ] 

ASF subversion and git services commented on ATLAS-3182:


Commit 8b58a326197e897a937d28f53076f385bb6710a7 in atlas's branch 
refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=8b58a32 ]

ATLAS-3182: Import transforms handling qualifiedName for hive_table entity.


> Import Transforms: Hive Table Qualified Name is Modified Incorrectly
> 
>
> Key: ATLAS-3182
> URL: https://issues.apache.org/jira/browse/ATLAS-3182
> Project: Atlas
>  Issue Type: Bug
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: 
> ATLAS-3182-Import-transforms-handling-qualifiedName-.patch
>
>
> *Background*
> Existing transformation framework that is used in _ImportService_ does not 
> correctly replace _qualifiedName_ for _hive_table_ entities in cases where 
> entity _name_ and _qualifiedName_ are different.
> *Steps to Duplicate*
>  # Create _hive_db_, say _db1_.
>  # Create _hive_table_ entity such that it has name = _testtable_ and 
> _qualifiedName = db1.testtable-123@cl1_.
>  # Use the transforms handler to convert _hive_db_ (_db1_) name to dbX and 
> _clusterName = clX_.
> *Expected Results*
> The resulting _hive_db_ should have _name_ and _qualifiedName_ changed to 
> _dbX._
> The _hive_table_ should have its name = _testtable_ and _qualifiedName = 
> dbX.testtable-123@clX._
> *Actual Results*
> The _hive_db_ entity is correctly transformed.
> The _hive_table_ entity's qualifiedName gets changed to _dbX.testtable@clX._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70588: ATLAS-3188: Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70588/#review215011
---


Ship it!




Ship It!

- Madhan Neethiraj


On May 3, 2019, 4:05 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70588/
> ---
> 
> (Updated May 3, 2019, 4:05 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-3188
> https://issues.apache.org/jira/browse/ATLAS-3188
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  54cd48f04 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  f378d8457 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
>  cdb4cd218 
> 
> 
> Diff: https://reviews.apache.org/r/70588/diff/1/
> 
> 
> Testing
> ---
> 
> validated that "insert into values" query doesn't create hive_process entities
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

[jira] [Commented] (ATLAS-3033) ATLAS Hive hook fails for CTAS query involving UDFs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832234#comment-16832234
 ] 

ASF subversion and git services commented on ATLAS-3033:


Commit 594c998037d55eb04b38c4c727d1f3419dfd4ec8 in atlas's branch 
refs/heads/branch-1.0 from Nixon Rodrigues
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=594c998 ]

ATLAS-3033: Skip hive temp table while getting Table object details from 
hiveContext.

Signed-off-by: Madhan Neethiraj 

(cherry picked from commit b5472ce313dcf24b0bffbe5201f0c95eec025848)
(cherry picked from commit 0323ce3825c63b06d7b0032edd10b73b84078a4a)
(cherry picked from commit 3e65639e2bf9155090a87bf05f16ef040ded14d1)


> ATLAS Hive hook fails for CTAS query involving UDFs
> ---
>
> Key: ATLAS-3033
> URL: https://issues.apache.org/jira/browse/ATLAS-3033
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 0.8.4, 1.2.0, 2.0.0
>
> Attachments: ATLAS-3033-branch-2.0.patch, ATLAS-3033-master.patch, 
> ATLAS-3033.patch
>
>
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> {code}
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> {code}
> complete stack trace.
> {code}
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: 
> _dummy_database._
> dummy_table table not found
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process 
> operation CREAT
> ETABLE_AS_SELECT
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-12-26 12:15:22,234 INFO  [HiveServer2-Background-Pool: Thread-339]: 
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) -  method=PostHook.org.apache.atlas.hive.hook.HiveHook start=154582659 
> end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
> {code}
> The problem looks to be appearing as result being stored for UDF in temporary 
> table _dummy_database._dummy_table and while creating a Lineage the process 
> fails.
> Steps to reproduce is running below SQL.
> {code}
> create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as 
> (name, ssn);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3188) Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832228#comment-16832228
 ] 

Sarath Subramanian commented on ATLAS-3188:
---

Code review: [https://reviews.apache.org/r/70588/]

> Regression: Simple insert queries are being captured in ATLAS_HOOK topic
> 
>
> Key: ATLAS-3188
> URL: https://issues.apache.org/jira/browse/ATLAS-3188
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Padashetty
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.0.0
>
>
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 70588: ATLAS-3188: Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70588/
---

Review request for atlas, Ashutosh Mestry and Madhan Neethiraj.


Bugs: ATLAS-3188
https://issues.apache.org/jira/browse/ATLAS-3188


Repository: atlas


Description
---

insert into  values () is triggering messages in ATLAS_HOOK 
topic. Due to this, Atlas is reporting the operations in Audits. 

Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
they do not play any role in lineage 

Inserting 100 rows in a table is generating 100 messages in the kafka topic.


Diffs
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 54cd48f04 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 f378d8457 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
 cdb4cd218 


Diff: https://reviews.apache.org/r/70588/diff/1/


Testing
---

validated that "insert into values" query doesn't create hive_process entities


Thanks,

Sarath Subramanian

[jira] [Updated] (ATLAS-3033) ATLAS Hive hook fails for CTAS query involving UDFs

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3033:

Attachment: ATLAS-3033-branch-2.0.patch

> ATLAS Hive hook fails for CTAS query involving UDFs
> ---
>
> Key: ATLAS-3033
> URL: https://issues.apache.org/jira/browse/ATLAS-3033
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 0.8.4
>
> Attachments: ATLAS-3033-branch-2.0.patch, ATLAS-3033-master.patch, 
> ATLAS-3033.patch
>
>
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> {code}
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> {code}
> complete stack trace.
> {code}
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: 
> _dummy_database._
> dummy_table table not found
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process 
> operation CREAT
> ETABLE_AS_SELECT
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-12-26 12:15:22,234 INFO  [HiveServer2-Background-Pool: Thread-339]: 
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) -  method=PostHook.org.apache.atlas.hive.hook.HiveHook start=154582659 
> end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
> {code}
> The problem looks to be appearing as result being stored for UDF in temporary 
> table _dummy_database._dummy_table and while creating a Lineage the process 
> fails.
> Steps to reproduce is running below SQL.
> {code}
> create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as 
> (name, ssn);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3033) ATLAS Hive hook fails for CTAS query involving UDFs

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3033:

Fix Version/s: 2.0.0
   1.2.0

> ATLAS Hive hook fails for CTAS query involving UDFs
> ---
>
> Key: ATLAS-3033
> URL: https://issues.apache.org/jira/browse/ATLAS-3033
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 0.8.4, 1.2.0, 2.0.0
>
> Attachments: ATLAS-3033-branch-2.0.patch, ATLAS-3033-master.patch, 
> ATLAS-3033.patch
>
>
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> {code}
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> {code}
> complete stack trace.
> {code}
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: 
> _dummy_database._
> dummy_table table not found
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process 
> operation CREAT
> ETABLE_AS_SELECT
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-12-26 12:15:22,234 INFO  [HiveServer2-Background-Pool: Thread-339]: 
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) -  method=PostHook.org.apache.atlas.hive.hook.HiveHook start=154582659 
> end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
> {code}
> The problem looks to be appearing as result being stored for UDF in temporary 
> table _dummy_database._dummy_table and while creating a Lineage the process 
> fails.
> Steps to reproduce is running below SQL.
> {code}
> create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as 
> (name, ssn);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3033) ATLAS Hive hook fails for CTAS query involving UDFs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832213#comment-16832213
 ] 

ASF subversion and git services commented on ATLAS-3033:


Commit 3e65639e2bf9155090a87bf05f16ef040ded14d1 in atlas's branch 
refs/heads/branch-2.0 from Nixon Rodrigues
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=3e65639 ]

ATLAS-3033: Skip hive temp table while getting Table object details from 
hiveContext.

Signed-off-by: Madhan Neethiraj 

(cherry picked from commit b5472ce313dcf24b0bffbe5201f0c95eec025848)
(cherry picked from commit 0323ce3825c63b06d7b0032edd10b73b84078a4a)


> ATLAS Hive hook fails for CTAS query involving UDFs
> ---
>
> Key: ATLAS-3033
> URL: https://issues.apache.org/jira/browse/ATLAS-3033
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 0.8.4
>
> Attachments: ATLAS-3033-master.patch, ATLAS-3033.patch
>
>
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> {code}
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> {code}
> complete stack trace.
> {code}
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: 
> _dummy_database._
> dummy_table table not found
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process 
> operation CREAT
> ETABLE_AS_SELECT
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-12-26 12:15:22,234 INFO  [HiveServer2-Background-Pool: Thread-339]: 
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) -  method=PostHook.org.apache.atlas.hive.hook.HiveHook start=154582659 
> end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
> {code}
> The problem looks to be appearing as result being stored for UDF in temporary 
> table _dummy_database._dummy_table and while creating a Lineage the process 
> fails.
> Steps to reproduce is running below SQL.
> {code}
> create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as 
> (name, ssn);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3033) ATLAS Hive hook fails for CTAS query involving UDFs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832198#comment-16832198
 ] 

ASF subversion and git services commented on ATLAS-3033:


Commit 0323ce3825c63b06d7b0032edd10b73b84078a4a in atlas's branch 
refs/heads/master from Nixon Rodrigues
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=0323ce3 ]

ATLAS-3033: Skip hive temp table while getting Table object details from 
hiveContext.

Signed-off-by: Madhan Neethiraj 

(cherry picked from commit b5472ce313dcf24b0bffbe5201f0c95eec025848)


> ATLAS Hive hook fails for CTAS query involving UDFs
> ---
>
> Key: ATLAS-3033
> URL: https://issues.apache.org/jira/browse/ATLAS-3033
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 0.8.4
>
> Attachments: ATLAS-3033-master.patch, ATLAS-3033.patch
>
>
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> {code}
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> {code}
> complete stack trace.
> {code}
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: 
> _dummy_database._
> dummy_table table not found
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process 
> operation CREAT
> ETABLE_AS_SELECT
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-12-26 12:15:22,234 INFO  [HiveServer2-Background-Pool: Thread-339]: 
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) -  method=PostHook.org.apache.atlas.hive.hook.HiveHook start=154582659 
> end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
> {code}
> The problem looks to be appearing as result being stored for UDF in temporary 
> table _dummy_database._dummy_table and while creating a Lineage the process 
> fails.
> Steps to reproduce is running below SQL.
> {code}
> create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as 
> (name, ssn);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3033) ATLAS Hive hook fails for CTAS query involving UDFs

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-3033:

Attachment: ATLAS-3033-master.patch

> ATLAS Hive hook fails for CTAS query involving UDFs
> ---
>
> Key: ATLAS-3033
> URL: https://issues.apache.org/jira/browse/ATLAS-3033
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 0.8.4
>
> Attachments: ATLAS-3033-master.patch, ATLAS-3033.patch
>
>
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> {code}
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> {code}
> complete stack trace.
> {code}
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: 
> _dummy_database._
> dummy_table table not found
> 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: 
> hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process 
> operation CREAT
> ETABLE_AS_SELECT
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
> at 
> org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
> at 
> org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-12-26 12:15:22,234 INFO  [HiveServer2-Background-Pool: Thread-339]: 
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) -  method=PostHook.org.apache.atlas.hive.hook.HiveHook start=154582659 
> end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
> {code}
> The problem looks to be appearing as result being stored for UDF in temporary 
> table _dummy_database._dummy_table and while creating a Lineage the process 
> fails.
> Steps to reproduce is running below SQL.
> {code}
> create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as 
> (name, ssn);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215010
---




addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 35 (patched)


Shim is not needed for ImpalaLineageTool. It is necessary only for the 
classes that run within host components (like Hive, HBase). Please remove this 
class.



addons/impala-bridge/pom.xml
Lines 47 (patched)


Following libraries are not needed, as the implementation uses 
notifications, instead of REST APIs. Please review and remove these and other 
unused libraries from here:
  - atlas-client-v1
  - atlas-client-v2
  - hdfs-model



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 52 (patched)


'implements Runnable' seems unnecessary here, as no threads are created to 
run the tool.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 54 (patched)


It may not be useful to designate a 'default-directoy'; the tool should 
process the files specified in the command-line. Please consider removing this.

Anyway, this doesn't seem to be used at all. Please review.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 59 (patched)


Instead of using 'static' fields (args, instance), consider using class 
members and instantiate the class from main().

public class ImpalaLineageTool {
  public static void main(String[] args) {
ImpalaLineageTool instance = new ImpalaLineageTool(args);

instance.run();
  }

  private ImpalaLineageTool(String[] args) {
this.args = args;
  }

  public void run() {
// ..
  }
}



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 61 (patched)


Consider avoiding test specific details (like 'isUnitTest') from product 
code.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 80 (patched)


log message seems incorrect. Please review the implementation to make sure 
that current directoy is read when arguments are not provided.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 87 (patched)


- args.length may be less than 2 here - see line #79 above
- instead of using args[1] here, consider initializing following variable 
after line #83 and use it here:
  String directoryName = args[1];



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 91 (patched)


info level log will be useful here.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 123 (patched)


It will help to use consistent stype in use of '{'. Please consider moving 
'{' to end of previous line. And review other such occurrences as well.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 124 (patched)


It will be useful to include name of the file in the log:
  LOG.info("deleted lineage file {}", currentFile.getAbsolutePath());
  
  LOG.info("failed to delete lineage file {}", 
currentFile.getAbsolutePath());

Please review and update other such occurrences.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 167 (patched)


info level log would be useful here.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
Lines 171 (patched)


Please handle 'listOfFiles == null' condition.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java
Lines 21 (patched)


ImpalaDataTypes doesn't seem to be used. If true, please consider removing 
this enum.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
Lines 35 (patched)


Consider renaming LineageQuery to ImpalaQuery or ImpalaProcess



addons/impala-bridge/src/main/resources/import-impala.sh
Lines 48 (patched)

Re: Review Request 70574: Import Transforms: Fix QualifiedName Transformation

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70574/#review215008
---


Fix it, then Ship it!





intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
Lines 52 (patched)


Consider marking tableNameFromQualifiedNameDifferent and 
tableNameFromQualifiedName as final.



intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
Lines 68 (patched)


Please handle cases where 'databaseSeparatorIdx == -1' and 
'clusterSeparatorIdx == -1'.



intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
Line 66 (original), 71 (patched)


initialize tableNameFromQualifiedNameDifferent and 
tableNameFromQualifiedName?


- Madhan Neethiraj


On May 2, 2019, 5:36 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70574/
> ---
> 
> (Updated May 2, 2019, 5:36 a.m.)
> 
> 
> Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3182
> https://issues.apache.org/jira/browse/ATLAS-3182
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Simple string manipulation.
> 
> **Transforms**
> 
> import-options.json
> ```
> {
> "options": {
> "transformers": "[{\"conditions\":{\"hive_db.clusterName\":\"EQUALS: 
> mycluster0\"},\"action\":{\"hive_db.clusterName\":\"SET: 
> mycluster1\"}},{\"conditions\":{\"hive_db.name\":\"EQUALS: 
> test_partition_bootstrap\"},\"action\":{\"hive_db.name\":\"SET: 
> test_partition_bootstrap_target\"}}]"
> }
> }
> ```
> **Import Zip**
> (Attached)
> 
> **CURL**
> 
> curl -X POST -u admin:admin123 -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F data=@./test_partition_bootstrap.zip -F 
> request=@./import-options.json http://localhost:21000/api/atlas/admin/import
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
>  9eb44d782 
>   
> intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java
>  d6b0ede2b 
>   intg/src/test/resources/json/entity1.json PRE-CREATION 
>   intg/src/test/resources/json/transform1.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70574/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Additional test.
> 
> **Functional test**
> Existing Import flow using test data.
> 
> 
> File Attachments
> 
> 
> Zip to Import
>   
> https://reviews.apache.org/media/uploaded/files/2019/04/30/44469b66-884c-4bde-8200-e41306a19b3b__test_partition_bootstrap.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

[jira] [Assigned] (ATLAS-3188) Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian reassigned ATLAS-3188:
-

Assignee: Sarath Subramanian

> Regression: Simple insert queries are being captured in ATLAS_HOOK topic
> 
>
> Key: ATLAS-3188
> URL: https://issues.apache.org/jira/browse/ATLAS-3188
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Padashetty
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.0.0
>
>
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3187) Regression, Relationship updates : In GET entity definition of a deleted table, columns,sd etc., are empty

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831951#comment-16831951
 ] 

ASF subversion and git services commented on ATLAS-3187:


Commit 114246d86da4e151393b0375337c164fb7fa2291 in atlas's branch 
refs/heads/branch-2.0 from Madhan Neethiraj
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=114246d ]

ATLAS-3187: columns attribute is empty for deleted hive_table entity

(cherry picked from commit aaf0ba23b418d1b5b128ec4b8854287e878e2b4e)


> Regression, Relationship updates : In GET entity definition of a deleted 
> table, columns,sd etc., are empty
> --
>
> Key: ATLAS-3187
> URL: https://issues.apache.org/jira/browse/ATLAS-3187
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sharmadha Sainath
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: table_entity_def.json
>
>
> Adding the table_entity_def["entity"]["attributes"]:
> Here , table is deleted , but the columns array and sd are [] and null 
> respectively. Expected is all entities in DELETED  state.
> Attaching the complete table definition :
> {code}
> attributes: {
> owner: "hrt_qa",
> temporary: false,
> lastAccessTime: 1556788686000,
> aliases: null,
> replicatedTo: null,
> replicatedFrom: null,
> qualifiedName: "default.table1@cl1",
> columns: [ ],
> description: null,
> viewExpandedText: null,
> tableType: "MANAGED_TABLE",
> sd: null,
> createTime: 1556788686000,
> name: "table1",
> comment: null,
> partitionKeys: [ ],
> parameters: {
> totalSize: "0",
> numRows: "0",
> rawDataSize: "0",
> transactional_properties: "default",
> COLUMN_STATS_ACCURATE: "{"BASIC_STATS":"true","COLUMN_STATS":{"id":"true"}}",
> numFiles: "0",
> transient_lastDdlTime: "1556788686",
> bucketing_version: "2",
> transactional: "true"
> },
> retention: 0,
> viewOriginalText: null,
> db: {
> guid: "ab02dd3b-1d1c-4522-8c2c-0fa60d82fcbe",
> typeName: "hive_db"
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3187) Regression, Relationship updates : In GET entity definition of a deleted table, columns,sd etc., are empty

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831950#comment-16831950
 ] 

ASF subversion and git services commented on ATLAS-3187:


Commit aaf0ba23b418d1b5b128ec4b8854287e878e2b4e in atlas's branch 
refs/heads/master from Madhan Neethiraj
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=aaf0ba2 ]

ATLAS-3187: columns attribute is empty for deleted hive_table entity


> Regression, Relationship updates : In GET entity definition of a deleted 
> table, columns,sd etc., are empty
> --
>
> Key: ATLAS-3187
> URL: https://issues.apache.org/jira/browse/ATLAS-3187
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sharmadha Sainath
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: table_entity_def.json
>
>
> Adding the table_entity_def["entity"]["attributes"]:
> Here , table is deleted , but the columns array and sd are [] and null 
> respectively. Expected is all entities in DELETED  state.
> Attaching the complete table definition :
> {code}
> attributes: {
> owner: "hrt_qa",
> temporary: false,
> lastAccessTime: 1556788686000,
> aliases: null,
> replicatedTo: null,
> replicatedFrom: null,
> qualifiedName: "default.table1@cl1",
> columns: [ ],
> description: null,
> viewExpandedText: null,
> tableType: "MANAGED_TABLE",
> sd: null,
> createTime: 1556788686000,
> name: "table1",
> comment: null,
> partitionKeys: [ ],
> parameters: {
> totalSize: "0",
> numRows: "0",
> rawDataSize: "0",
> transactional_properties: "default",
> COLUMN_STATS_ACCURATE: "{"BASIC_STATS":"true","COLUMN_STATS":{"id":"true"}}",
> numFiles: "0",
> transient_lastDdlTime: "1556788686",
> bucketing_version: "2",
> transactional: "true"
> },
> retention: 0,
> viewOriginalText: null,
> db: {
> guid: "ab02dd3b-1d1c-4522-8c2c-0fa60d82fcbe",
> typeName: "hive_db"
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70584: ATLAS-3187: legacy attributes do not include references to deleted entities

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70584/#review215003
---


Ship it!




Ship It!

- Ashutosh Mestry


On May 2, 2019, 6:26 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70584/
> ---
> 
> (Updated May 2, 2019, 6:26 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Le 
> Ma, Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, and 
> Umesh Padashetty.
> 
> 
> Bugs: ATLAS-3187
> https://issues.apache.org/jira/browse/ATLAS-3187
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> While populating legacyAttribute from its corresponding 
> relationshipAttribute, only objects with ACTIVE relationship-edge were 
> included. This caused the deleted contained entities to be skipped. Updated 
> this logic to skip only ACTIVE entities with DELETED relationship-edge (like 
> a hive_table moved from one-db to another).
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  3e1e023e9 
> 
> 
> Diff: https://reviews.apache.org/r/70584/diff/2/
> 
> 
> Testing
> ---
> 
> - verified that entity-get on a deleted hive_table includes all its columns 
> as well
> - pre-commit tests run: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1072/
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/
---

(Updated May 2, 2019, 8:31 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
Subramanian, and Xinran Tinney.


Repository: atlas


Description
---

Impala generates lineage records for its commands. This new feature will read 
Impala lineage file, convert the lineage record to Atlas entities and send them 
to Atlas. In this way, Atlas can get lineage of Impala operation.

The metadata referred in the lineage are captured in Hive Metastore hook and 
sent to Atlas. This work is done in ATLAS-3148

This jira only supports the Impala command "create view". Following jira will 
add support for more Impala commands.


Diffs
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 PRE-CREATION 
  addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-application.properties 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/users-credentials.properties 
PRE-CREATION 
  pom.xml 7de5d31 


Diff: https://reviews.apache.org/r/70512/diff/13/


Testing
---

Run the tool in real cluster that has Atlas server with Impala lineage file as 
input for creating view. The Atlas UI displays hive_lineage lineage and 
hive_column_lineage.
Add new integration tests and they pass


Thanks,

Na Li

[jira] [Updated] (ATLAS-3183) Read Impala lineage record for creating view and send to Atlas

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated ATLAS-3183:
-
Attachment: (was: ATLAS-3183.005.patch)

> Read Impala lineage record for creating view and send to Atlas
> --
>
> Key: ATLAS-3183
> URL: https://issues.apache.org/jira/browse/ATLAS-3183
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: ATLAS-3183.006.patch
>
>
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in 
> [ATLAS-3148|https://issues.apache.org/jira/browse/ATLAS-3148]
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3183) Read Impala lineage record for creating view and send to Atlas

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated ATLAS-3183:
-
Attachment: ATLAS-3183.006.patch

> Read Impala lineage record for creating view and send to Atlas
> --
>
> Key: ATLAS-3183
> URL: https://issues.apache.org/jira/browse/ATLAS-3183
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: ATLAS-3183.005.patch, ATLAS-3183.006.patch
>
>
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in 
> [ATLAS-3148|https://issues.apache.org/jira/browse/ATLAS-3148]
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3183) Read Impala lineage record for creating view and send to Atlas

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated ATLAS-3183:
-
Attachment: (was: ATLAS-3183.002.patch)

> Read Impala lineage record for creating view and send to Atlas
> --
>
> Key: ATLAS-3183
> URL: https://issues.apache.org/jira/browse/ATLAS-3183
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: ATLAS-3183.005.patch
>
>
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in 
> [ATLAS-3148|https://issues.apache.org/jira/browse/ATLAS-3148]
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3183) Read Impala lineage record for creating view and send to Atlas

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated ATLAS-3183:
-
Attachment: (was: ATLAS-3183.004.patch)

> Read Impala lineage record for creating view and send to Atlas
> --
>
> Key: ATLAS-3183
> URL: https://issues.apache.org/jira/browse/ATLAS-3183
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: ATLAS-3183.005.patch
>
>
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in 
> [ATLAS-3148|https://issues.apache.org/jira/browse/ATLAS-3148]
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3183) Read Impala lineage record for creating view and send to Atlas

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831885#comment-16831885
 ] 

Na Li commented on ATLAS-3183:
--

tests passed for Impala-bridge

> Read Impala lineage record for creating view and send to Atlas
> --
>
> Key: ATLAS-3183
> URL: https://issues.apache.org/jira/browse/ATLAS-3183
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: ATLAS-3183.002.patch, ATLAS-3183.004.patch, 
> ATLAS-3183.005.patch
>
>
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in 
> [ATLAS-3148|https://issues.apache.org/jira/browse/ATLAS-3148]
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3183) Read Impala lineage record for creating view and send to Atlas

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated ATLAS-3183:
-
Attachment: ATLAS-3183.005.patch

> Read Impala lineage record for creating view and send to Atlas
> --
>
> Key: ATLAS-3183
> URL: https://issues.apache.org/jira/browse/ATLAS-3183
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: ATLAS-3183.002.patch, ATLAS-3183.004.patch, 
> ATLAS-3183.005.patch
>
>
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in 
> [ATLAS-3148|https://issues.apache.org/jira/browse/ATLAS-3148]
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70584: ATLAS-3187: legacy attributes do not include references to deleted entities

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70584/
---

(Updated May 2, 2019, 6:26 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Le Ma, 
Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, and Umesh 
Padashetty.


Changes
---

addressed review comments


Bugs: ATLAS-3187
https://issues.apache.org/jira/browse/ATLAS-3187


Repository: atlas


Description
---

While populating legacyAttribute from its corresponding relationshipAttribute, 
only objects with ACTIVE relationship-edge were included. This caused the 
deleted contained entities to be skipped. Updated this logic to skip only 
ACTIVE entities with DELETED relationship-edge (like a hive_table moved from 
one-db to another).


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 3e1e023e9 


Diff: https://reviews.apache.org/r/70584/diff/2/

Changes: https://reviews.apache.org/r/70584/diff/1-2/


Testing
---

- verified that entity-get on a deleted hive_table includes all its columns as 
well
- pre-commit tests run: 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1072/


Thanks,

Madhan Neethiraj

Re: Review Request 70584: ATLAS-3187: legacy attributes do not include references to deleted entities

[Ashutosh Mestry]

> On May 2, 2019, 5:46 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
> > Line 1118 (original), 1114 (patched)
> > 
> >
> > Consider simplifying. I see 3 methods within method.
> 
> Madhan Neethiraj wrote:
> Splitting these into 3 methods doesn't see to be useful, as: 1) this is 
> already a small piece of code 2) this is a private method 3) doesn't have 
> having to know the various paths (relatedobjectid/map/..).
> 
> I suggest we keep the current implementation - for simplicity.

Disagree. This is too complicated for understanding.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70584/#review214999
---


On May 2, 2019, 4:56 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70584/
> ---
> 
> (Updated May 2, 2019, 4:56 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Le 
> Ma, Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, and 
> Umesh Padashetty.
> 
> 
> Bugs: ATLAS-3187
> https://issues.apache.org/jira/browse/ATLAS-3187
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> While populating legacyAttribute from its corresponding 
> relationshipAttribute, only objects with ACTIVE relationship-edge were 
> included. This caused the deleted contained entities to be skipped. Updated 
> this logic to skip only ACTIVE entities with DELETED relationship-edge (like 
> a hive_table moved from one-db to another).
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  3e1e023e9 
> 
> 
> Diff: https://reviews.apache.org/r/70584/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that entity-get on a deleted hive_table includes all its columns 
> as well
> - pre-commit tests run: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1072/
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 70584: ATLAS-3187: legacy attributes do not include references to deleted entities

[Madhan Neethiraj]

> On May 2, 2019, 5:46 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
> > Line 1118 (original), 1114 (patched)
> > 
> >
> > Consider simplifying. I see 3 methods within method.

Splitting these into 3 methods doesn't see to be useful, as: 1) this is already 
a small piece of code 2) this is a private method 3) doesn't have having to 
know the various paths (relatedobjectid/map/..).

I suggest we keep the current implementation - for simplicity.


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70584/#review214999
---


On May 2, 2019, 4:56 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70584/
> ---
> 
> (Updated May 2, 2019, 4:56 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Le 
> Ma, Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, and 
> Umesh Padashetty.
> 
> 
> Bugs: ATLAS-3187
> https://issues.apache.org/jira/browse/ATLAS-3187
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> While populating legacyAttribute from its corresponding 
> relationshipAttribute, only objects with ACTIVE relationship-edge were 
> included. This caused the deleted contained entities to be skipped. Updated 
> this logic to skip only ACTIVE entities with DELETED relationship-edge (like 
> a hive_table moved from one-db to another).
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  3e1e023e9 
> 
> 
> Diff: https://reviews.apache.org/r/70584/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that entity-get on a deleted hive_table includes all its columns 
> as well
> - pre-commit tests run: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1072/
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

[jira] [Updated] (ATLAS-3183) Read Impala lineage record for creating view and send to Atlas

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated ATLAS-3183:
-
Attachment: ATLAS-3183.004.patch

> Read Impala lineage record for creating view and send to Atlas
> --
>
> Key: ATLAS-3183
> URL: https://issues.apache.org/jira/browse/ATLAS-3183
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: ATLAS-3183.002.patch, ATLAS-3183.004.patch
>
>
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in 
> [ATLAS-3148|https://issues.apache.org/jira/browse/ATLAS-3148]
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70584: ATLAS-3187: legacy attributes do not include references to deleted entities

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70584/#review214999
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Line 1118 (original), 1114 (patched)


Consider simplifying. I see 3 methods within method.


- Ashutosh Mestry


On May 2, 2019, 4:56 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70584/
> ---
> 
> (Updated May 2, 2019, 4:56 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Le 
> Ma, Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, and 
> Umesh Padashetty.
> 
> 
> Bugs: ATLAS-3187
> https://issues.apache.org/jira/browse/ATLAS-3187
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> While populating legacyAttribute from its corresponding 
> relationshipAttribute, only objects with ACTIVE relationship-edge were 
> included. This caused the deleted contained entities to be skipped. Updated 
> this logic to skip only ACTIVE entities with DELETED relationship-edge (like 
> a hive_table moved from one-db to another).
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  3e1e023e9 
> 
> 
> Diff: https://reviews.apache.org/r/70584/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that entity-get on a deleted hive_table includes all its columns 
> as well
> - pre-commit tests run: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1072/
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 70574: Import Transforms: Fix QualifiedName Transformation

[Ashutosh Mestry]

> On May 2, 2019, 5:59 a.m., Madhan Neethiraj wrote:
> > intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
> > Lines 68 (patched)
> > 
> >
> > Should rename of table automatically update the qualifiedName? Or 
> > should the qualifiedName continue to use 'tableNameFromQualifiedName'?
> > 
> > Table rename will be performed by a transformer with a call to 
> > setAttribute(HIVE_TABLE_NAME_ATTRIBUTE, newName).
> > 
> > Also, how does this impact the 'tableName' in other entity-type's 
> > qualifiedName - like hive_column, hive_storagedesc?
> 
> Ashutosh Mestry wrote:
> The case is that name and qualifiedName are different. In my test case, 
> this is all hive generated. Also, there can be a case where from REST calls, 
> user can set name and qualifiedName to be different.
> 
> The patch handles this case.

>From AtlasHiveHookContext, this method is more complicated than simple concat 
>operation. It explains the format I was seeing in my tests. In short, this 
>makes it apparent that qualifiedName will not always be as straightforward as 
>we expect it to be.

   
   public String getQualifiedName(Table table) {
String tableName = table.getTableName();

if (table.isTemporary()) {
if (SessionState.get() != null && SessionState.get().getSessionId() 
!= null) {
tableName = tableName + TEMP_TABLE_PREFIX + 
SessionState.get().getSessionId();
} else {
tableName = tableName + TEMP_TABLE_PREFIX + 
RandomStringUtils.random(10);
}
}

return (table.getDbName() + QNAME_SEP_ENTITY_NAME + tableName + 
QNAME_SEP_CLUSTER_NAME).toLowerCase() + getClusterName();
}


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70574/#review214993
---


On May 2, 2019, 5:36 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70574/
> ---
> 
> (Updated May 2, 2019, 5:36 a.m.)
> 
> 
> Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3182
> https://issues.apache.org/jira/browse/ATLAS-3182
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Simple string manipulation.
> 
> **Transforms**
> 
> import-options.json
> ```
> {
> "options": {
> "transformers": "[{\"conditions\":{\"hive_db.clusterName\":\"EQUALS: 
> mycluster0\"},\"action\":{\"hive_db.clusterName\":\"SET: 
> mycluster1\"}},{\"conditions\":{\"hive_db.name\":\"EQUALS: 
> test_partition_bootstrap\"},\"action\":{\"hive_db.name\":\"SET: 
> test_partition_bootstrap_target\"}}]"
> }
> }
> ```
> **Import Zip**
> (Attached)
> 
> **CURL**
> 
> curl -X POST -u admin:admin123 -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F data=@./test_partition_bootstrap.zip -F 
> request=@./import-options.json http://localhost:21000/api/atlas/admin/import
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
>  9eb44d782 
>   
> intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java
>  d6b0ede2b 
>   intg/src/test/resources/json/entity1.json PRE-CREATION 
>   intg/src/test/resources/json/transform1.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70574/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Additional test.
> 
> **Functional test**
> Existing Import flow using test data.
> 
> 
> File Attachments
> 
> 
> Zip to Import
>   
> https://reviews.apache.org/media/uploaded/files/2019/04/30/44469b66-884c-4bde-8200-e41306a19b3b__test_partition_bootstrap.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 70574: Import Transforms: Fix QualifiedName Transformation

[Ashutosh Mestry]

> On May 2, 2019, 5:59 a.m., Madhan Neethiraj wrote:
> > intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
> > Lines 68 (patched)
> > 
> >
> > Should rename of table automatically update the qualifiedName? Or 
> > should the qualifiedName continue to use 'tableNameFromQualifiedName'?
> > 
> > Table rename will be performed by a transformer with a call to 
> > setAttribute(HIVE_TABLE_NAME_ATTRIBUTE, newName).
> > 
> > Also, how does this impact the 'tableName' in other entity-type's 
> > qualifiedName - like hive_column, hive_storagedesc?

The case is that name and qualifiedName are different. In my test case, this is 
all hive generated. Also, there can be a case where from REST calls, user can 
set name and qualifiedName to be different.

The patch handles this case.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70574/#review214993
---


On May 2, 2019, 5:36 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70574/
> ---
> 
> (Updated May 2, 2019, 5:36 a.m.)
> 
> 
> Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3182
> https://issues.apache.org/jira/browse/ATLAS-3182
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Simple string manipulation.
> 
> **Transforms**
> 
> import-options.json
> ```
> {
> "options": {
> "transformers": "[{\"conditions\":{\"hive_db.clusterName\":\"EQUALS: 
> mycluster0\"},\"action\":{\"hive_db.clusterName\":\"SET: 
> mycluster1\"}},{\"conditions\":{\"hive_db.name\":\"EQUALS: 
> test_partition_bootstrap\"},\"action\":{\"hive_db.name\":\"SET: 
> test_partition_bootstrap_target\"}}]"
> }
> }
> ```
> **Import Zip**
> (Attached)
> 
> **CURL**
> 
> curl -X POST -u admin:admin123 -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F data=@./test_partition_bootstrap.zip -F 
> request=@./import-options.json http://localhost:21000/api/atlas/admin/import
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
>  9eb44d782 
>   
> intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java
>  d6b0ede2b 
>   intg/src/test/resources/json/entity1.json PRE-CREATION 
>   intg/src/test/resources/json/transform1.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70574/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Additional test.
> 
> **Functional test**
> Existing Import flow using test data.
> 
> 
> File Attachments
> 
> 
> Zip to Import
>   
> https://reviews.apache.org/media/uploaded/files/2019/04/30/44469b66-884c-4bde-8200-e41306a19b3b__test_partition_bootstrap.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Review Request 70584: ATLAS-3187: legacy attributes do not include references to deleted entities

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70584/
---

Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Le Ma, 
Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, and Umesh 
Padashetty.


Bugs: ATLAS-3187
https://issues.apache.org/jira/browse/ATLAS-3187


Repository: atlas


Description
---

While populating legacyAttribute from its corresponding relationshipAttribute, 
only objects with ACTIVE relationship-edge were included. This caused the 
deleted contained entities to be skipped. Updated this logic to skip only 
ACTIVE entities with DELETED relationship-edge (like a hive_table moved from 
one-db to another).


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 3e1e023e9 


Diff: https://reviews.apache.org/r/70584/diff/1/


Testing
---

- verified that entity-get on a deleted hive_table includes all its columns as 
well
- pre-commit tests run: 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1072/


Thanks,

Madhan Neethiraj

[jira] [Created] (ATLAS-3189) Issue with Deletes

[Ayush Nigam (JIRA)]

Ayush Nigam created ATLAS-3189:
--

 Summary: Issue with Deletes
 Key: ATLAS-3189
 URL: https://issues.apache.org/jira/browse/ATLAS-3189
 Project: Atlas
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Ayush Nigam
 Attachments: Screen Shot 2019-05-02 at 9.59.10 PM.png, Screen Shot 
2019-05-02 at 9.59.24 PM.png

Hi,

We currently have a hive model in Atlas(with no data),we tried to delete it by 
curl request,

curl -negotiate -u admin:admin -X DELETE -H 'Content-Type: application/json' -H 
'Accept: application/json' "http:/localhost:21001/api/atlas/v2/types/typedefs" 
-d "@./1030-hive_model.json"

Above request was successful and after that we repopulated the new model with 
one attribute deleted from hive_db and made code changes accordingly.

 

But then while running hive bridge we found that entities in basic search UI 
are not there while they are present in Advanced search.While digging deeper we 
found out that docs are itself not created in Solr/Elasticsearch while indexes 
are present.

We tried this approach with other models too,but the issues persists.

So to reproduce the bug.
 * Delete a model by Curl command(Clear any data before deleting the model if 
present)
 * Re post the model by curl command with some attribute deleted.
 * Populate the data

  

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3188) Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Umesh Padashetty (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Umesh Padashetty updated ATLAS-3188:

Fix Version/s: 2.0.0

> Regression: Simple insert queries are being captured in ATLAS_HOOK topic
> 
>
> Key: ATLAS-3188
> URL: https://issues.apache.org/jira/browse/ATLAS-3188
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Padashetty
>Priority: Major
> Fix For: 2.0.0
>
>
> insert into  values () is triggering messages in ATLAS_HOOK 
> topic. Due to this, Atlas is reporting the operations in Audits. 
> Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
> they do not play any role in lineage 
> Inserting 100 rows in a table is generating 100 messages in the kafka topic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-3188) Regression: Simple insert queries are being captured in ATLAS_HOOK topic

[Umesh Padashetty (JIRA)]

Umesh Padashetty created ATLAS-3188:
---

 Summary: Regression: Simple insert queries are being captured in 
ATLAS_HOOK topic
 Key: ATLAS-3188
 URL: https://issues.apache.org/jira/browse/ATLAS-3188
 Project: Atlas
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Umesh Padashetty


insert into  values () is triggering messages in ATLAS_HOOK 
topic. Due to this, Atlas is reporting the operations in Audits. 

Ideally, insert queries should not generate any messages in ATLAS_HOOK since 
they do not play any role in lineage 

Inserting 100 rows in a table is generating 100 messages in the kafka topic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3155) UI - IE-11 - Name entry not clickable

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-3155:
---
Attachment: ATLAS-3155-1.patch

> UI - IE-11 - Name entry not clickable
> -
>
> Key: ATLAS-3155
> URL: https://issues.apache.org/jira/browse/ATLAS-3155
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 1.2.0, 2.0.0
>
> Attachments: ATLAS-3155-1.patch, ATLAS-3155.patch, ie11_name_issue.jpg
>
>
> In the search results page, when we try to click the 'Name' column value of 
> any entry, nothing happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-3187) Regression, Relationship updates : In GET entity definition of a deleted table, columns,sd etc., are empty

[Sharmadha Sainath (JIRA)]

Sharmadha Sainath created ATLAS-3187:


 Summary: Regression, Relationship updates : In GET entity 
definition of a deleted table, columns,sd etc., are empty
 Key: ATLAS-3187
 URL: https://issues.apache.org/jira/browse/ATLAS-3187
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Sharmadha Sainath
 Fix For: 2.0.0
 Attachments: table_entity_def.json

Adding the table_entity_def["entity"]["attributes"]:

Here , table is deleted , but the columns array and sd are [] and null 
respectively. Expected is all entities in DELETED  state.

Attaching the complete table definition :
{code}
attributes: {
owner: "hrt_qa",
temporary: false,
lastAccessTime: 1556788686000,
aliases: null,
replicatedTo: null,
replicatedFrom: null,
qualifiedName: "default.table1@cl1",
columns: [ ],
description: null,
viewExpandedText: null,
tableType: "MANAGED_TABLE",
sd: null,
createTime: 1556788686000,
name: "table1",
comment: null,
partitionKeys: [ ],
parameters: {
totalSize: "0",
numRows: "0",
rawDataSize: "0",
transactional_properties: "default",
COLUMN_STATS_ACCURATE: "{"BASIC_STATS":"true","COLUMN_STATS":{"id":"true"}}",
numFiles: "0",
transient_lastDdlTime: "1556788686",
bucketing_version: "2",
transactional: "true"
},
retention: 0,
viewOriginalText: null,
db: {
guid: "ab02dd3b-1d1c-4522-8c2c-0fa60d82fcbe",
typeName: "hive_db"
}
}
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-3186) UI: Background navigation is possible when "Advanced Search Queries" popup is active.

[Rahul Kurup (JIRA)]

Rahul Kurup created ATLAS-3186:
--

 Summary: UI: Background navigation is possible when "Advanced 
Search Queries" popup is active.
 Key: ATLAS-3186
 URL: https://issues.apache.org/jira/browse/ATLAS-3186
 Project: Atlas
  Issue Type: Bug
Reporter: Rahul Kurup


Steps:
 # From Atlas UI, navigate as follows: Search–>Classification–>Change button 
from Flat to Tree->Glossary–>Search
 #  Click on "?" icon in Search.
 # Click the back button of your browser

You will see that as you go on pressing the back button, navigation will occur 
in the background in the reverse order of menus mentioned in step 1 (and the 
button selection will be reversed from Tree to Flat).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Apache Atlas version 2.0.0 - rc1

[Madhan Neethiraj]

Sarath - thanks for the updated release candidate.

+1 for Apache Atlas 2.0.0 release candidate #1

Performed the following:
- downloaded tar.gz and verified the signature
- built and installed Atlas for embedded HBase, Solr profile
- started Atlas with embedded HBase and Solr
- ran QuickStart successfully
- added/updated/deleted few 'Favorite Searches' - both basic and DSL
- performed number of searches - both basic and DSL
- created glossaries with number of terms, associated terms with entities, 
associated terms with classifications
- verified classification propagations

Thanks,
Madhan




On 4/29/19, 11:56 PM, "Sarath Subramanian"  wrote:

Atlas team,

Apache Atlas 2.0.0 Release Candidate #1 is now available for a vote within
dev community.

Following commits went into branch-2.0 since the last release candidate
(rc0):

 - ATLAS-3168: Fix Intermittent UT failure:
KafkaNotificationTest.initNotificationService()
 - ATLAS-3148: Implement Hive Metastore hook for Atlas
 - ATLAS-3167: Exception while starting Atlas with Cassandra
 - ATLAS-3160: Remove unthrown exception specified in AtlasRelationshipDef
constructors
 - ATLAS-3157: Add Integration tests for Hive metastore hook
 - ATLAS-3174: UI - Exclude process or deleted entity from search drop-down
if respective options are chosen
 - ATLAS-3155: UI - IE-11 - Name entry not clickable
 - ATLAS-3168: PatchFix: Unit test fixes and optimization.
 - ATLAS-3168: PatchFix: Support for HA Mode
 - ATLAS-3129: Fix SSL Truststore reloader leak from AtlasBaseClient
 - ATLAS-3159: UI - Add Reset button for Lienage
 - ATLAS-3154: UI - Lineage PNG export not working on Firefox
 - ATLAS-3158: UI - Render schema tab using relationshipAttributes (Columns)
 - ATLAS-3145: UI - Entity showing Active even if the relationshipStatus is
Deleted.

Links to the release artifacts are given below. Please review and vote.

The vote will be open for at least 72 hours or until necessary votes are
reached.
  [ ] +1 approve
  [ ] +0 no opinion
  [ ] -1 disapprove (and reason why)


Thanks,
Sarath

List of issues addressed in this release:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20ATLAS%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.0.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
Git tag for the release:
https://github.com/apache/atlas/tree/release-2.0.0-rc1
Sources for the release:

https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz

Source release verification:
PGP Signature:

https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.asc
SHA512 Hash:

https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.sha512
MD5 Hash:

https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.md5

Keys to verify the signature of the release artifacts are available at:
https://dist.apache.org/repos/dist/release/atlas/KEYS

Re: Add me as a Contributor for Atlas

[Madhan Neethiraj]

Mayank,

Thanks for your interest in contributing to Apache Atlas. You have been added 
as a contributor. Welcome to Apache Atlas community!

Regards,
Madhan


On 5/1/19, 11:05 PM, "mayank jain"  wrote:

Atlas community,

I would like to be a contributor for Apache Atlas.  My jira id is :
*mayank_nj*


*Regards,*
*Mayank Jain*

Add me as a Contributor for Atlas

[mayank jain]

Atlas community,

I would like to be a contributor for Apache Atlas.  My jira id is :
*mayank_nj*


*Regards,*
*Mayank Jain*

Re: Review Request 70574: Import Transforms: Fix QualifiedName Transformation

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70574/#review214993
---




intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
Lines 68 (patched)


Should rename of table automatically update the qualifiedName? Or should 
the qualifiedName continue to use 'tableNameFromQualifiedName'?

Table rename will be performed by a transformer with a call to 
setAttribute(HIVE_TABLE_NAME_ATTRIBUTE, newName).

Also, how does this impact the 'tableName' in other entity-type's 
qualifiedName - like hive_column, hive_storagedesc?


- Madhan Neethiraj


On May 2, 2019, 5:36 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70574/
> ---
> 
> (Updated May 2, 2019, 5:36 a.m.)
> 
> 
> Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3182
> https://issues.apache.org/jira/browse/ATLAS-3182
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Simple string manipulation.
> 
> **Transforms**
> 
> import-options.json
> ```
> {
> "options": {
> "transformers": "[{\"conditions\":{\"hive_db.clusterName\":\"EQUALS: 
> mycluster0\"},\"action\":{\"hive_db.clusterName\":\"SET: 
> mycluster1\"}},{\"conditions\":{\"hive_db.name\":\"EQUALS: 
> test_partition_bootstrap\"},\"action\":{\"hive_db.name\":\"SET: 
> test_partition_bootstrap_target\"}}]"
> }
> }
> ```
> **Import Zip**
> (Attached)
> 
> **CURL**
> 
> curl -X POST -u admin:admin123 -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F data=@./test_partition_bootstrap.zip -F 
> request=@./import-options.json http://localhost:21000/api/atlas/admin/import
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
>  9eb44d782 
>   
> intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java
>  d6b0ede2b 
>   intg/src/test/resources/json/entity1.json PRE-CREATION 
>   intg/src/test/resources/json/transform1.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70574/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Additional test.
> 
> **Functional test**
> Existing Import flow using test data.
> 
> 
> File Attachments
> 
> 
> Zip to Import
>   
> https://reviews.apache.org/media/uploaded/files/2019/04/30/44469b66-884c-4bde-8200-e41306a19b3b__test_partition_bootstrap.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>