Re: Review Request 69822: ATLAS-3033 - ATLAS Hive hook fails for CTAS query involving UDFs

2019-01-29 Thread Nixon Rodrigues 


> On Jan. 28, 2019, 4:58 p.m., Madhan Neethiraj wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
> > Lines 151 (patched)
> > 
> >
> > Please replace hardcoded value "Values__Tmp__Table__" with constant 
> > SemanticAnalyzer.VALUES_TMP_TABLE_NAME_PREFIX.

Madhan,

For Hive 1.2.1 version the constant VALUES_TMP_TABLE_NAME_PREFIX is private,its 
causing compile time issue, so its hardcoded here. 

ERROR] COMPILATION ERROR :
[INFO] -
[ERROR] 
~/atlas-gitbox/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java:[151,140]
 VALUES_TMP_TABLE_NAME_PREFIX has private access in 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer
[INFO] 1 error


- Nixon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69822/#review212385
---


On Jan. 28, 2019, 12:21 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69822/
> ---
> 
> (Updated Jan. 28, 2019, 12:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3033
> https://issues.apache.org/jira/browse/ATLAS-3033
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> 
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table.
> 
> 
> FIX :-  Added fix to skip hive _dummy_table temp table while getting Table 
> object details from hiveContext during lineage input creation.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  249f48b5f 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 4a6b41786 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  5c52cf442 
> 
> 
> Diff: https://reviews.apache.org/r/69822/diff/2/
> 
> 
> Testing
> ---
> 
> Tested by creating hive table with CTAS which creates temp table.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Re: Review Request 69822: ATLAS-3033 - ATLAS Hive hook fails for CTAS query involving UDFs

2019-01-28 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69822/#review212385
---


Fix it, then Ship it!





addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
Lines 151 (patched)


Please replace hardcoded value "Values__Tmp__Table__" with constant 
SemanticAnalyzer.VALUES_TMP_TABLE_NAME_PREFIX.


- Madhan Neethiraj


On Jan. 28, 2019, 12:21 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69822/
> ---
> 
> (Updated Jan. 28, 2019, 12:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3033
> https://issues.apache.org/jira/browse/ATLAS-3033
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> 
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table.
> 
> 
> FIX :-  Added fix to skip hive _dummy_table temp table while getting Table 
> object details from hiveContext during lineage input creation.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  249f48b5f 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 4a6b41786 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  5c52cf442 
> 
> 
> Diff: https://reviews.apache.org/r/69822/diff/2/
> 
> 
> Testing
> ---
> 
> Tested by creating hive table with CTAS which creates temp table.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Re: Review Request 69822: ATLAS-3033 - ATLAS Hive hook fails for CTAS query involving UDFs

2019-01-28 Thread Nixon Rodrigues 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69822/
---

(Updated Jan. 28, 2019, 12:21 p.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
Subramanian.


Changes
---

This patch addresses review comments from Madhan N.


Bugs: ATLAS-3033
https://issues.apache.org/jira/browse/ATLAS-3033


Repository: atlas


Description
---

ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
with the exception as below

org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
_dummy_table.


FIX :-  Added fix to skip hive _dummy_table temp table while getting Table 
object details from hiveContext during lineage input creation.


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 249f48b5f 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
4a6b41786 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 5c52cf442 


Diff: https://reviews.apache.org/r/69822/diff/2/

Changes: https://reviews.apache.org/r/69822/diff/1-2/


Testing
---

Tested by creating hive table with CTAS which creates temp table.


Thanks,

Nixon Rodrigues

Re: Review Request 69822: ATLAS-3033 - ATLAS Hive hook fails for CTAS query involving UDFs

2019-01-26 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69822/#review212361
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
Lines 231 (patched)


This will ignore ValueTmpTable only if its in one of the dummy-database. I 
think such tables should be ignored irrespective of the database. Please review.

case TABLE:
case PARTITION: {
  String  dbName= entity.getTable().getDbName();
  String  tableName = entity.getTable().getTableName();
  boolean skipTable = 
StringUtils.isNotEmpty(context.getIgnoreValuesTmpTableNamePrefix()) && 
tableName.toLowerCase().startsWith(context.getIgnoreValuesTmpTableNamePrefix());

  if (!skipTable) {
skipTable = context.getIgnoreDummyTableName().contains(tableName) && 
context.getIgnoreDummyDatabaseName().contains(dbName);
  }

  if (!skipTable) {
Table table = getHive().getTable(dbName, tableName);

ret = toTableEntity(table, entityExtInfo);
  }
}
break;


- Madhan Neethiraj


On Jan. 23, 2019, 3:07 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69822/
> ---
> 
> (Updated Jan. 23, 2019, 3:07 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3033
> https://issues.apache.org/jira/browse/ATLAS-3033
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
> with the exception as below
> 
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
> _dummy_table.
> 
> 
> FIX :-  Added fix to skip hive _dummy_table temp table while getting Table 
> object details from hiveContext during lineage input creation.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  249f48b5f 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 4a6b41786 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  5c52cf442 
> 
> 
> Diff: https://reviews.apache.org/r/69822/diff/1/
> 
> 
> Testing
> ---
> 
> Tested by creating hive table with CTAS which creates temp table.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Review Request 69822: ATLAS-3033 - ATLAS Hive hook fails for CTAS query involving UDFs

2019-01-23 Thread Nixon Rodrigues 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69822/
---

Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
Subramanian.


Bugs: ATLAS-3033
https://issues.apache.org/jira/browse/ATLAS-3033


Repository: atlas


Description
---

ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails 
with the exception as below

org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found 
_dummy_table.


FIX :-  Added fix to skip hive _dummy_table temp table while getting Table 
object details from hiveContext during lineage input creation.


Diffs
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 249f48b5f 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
4a6b41786 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 5c52cf442 


Diff: https://reviews.apache.org/r/69822/diff/1/


Testing
---

Tested by creating hive table with CTAS which creates temp table.


Thanks,

Nixon Rodrigues