Nixon Rodrigues created ATLAS-3033:
--------------------------------------
Summary: ATLAS Hive hook fails for CTAS query involving UDFs
Key: ATLAS-3033
URL: https://issues.apache.org/jira/browse/ATLAS-3033
Project: Atlas
Issue Type: Bug
Affects Versions: 0.8.3
Reporter: Nixon Rodrigues
Assignee: Nixon Rodrigues
Fix For: 0.8.4
ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails
with the exception as below
{code}
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
_dummy_table
{code}
complete stack trace.
{code}
2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]:
metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found:
_dummy_database._
dummy_table table not found
2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]:
hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process
operation CREAT
ETABLE_AS_SELECT
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
_dummy_table
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
at
org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
at
org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
at
org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
at
org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
at
org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at
org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-12-26 12:15:22,234 INFO [HiveServer2-Background-Pool: Thread-339]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG
method=PostHook.org.apache.atlas.hive.hook.HiveHook start=1545826522229
end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
{code}
The problem looks to be appearing as result being stored for UDF in temporary
table _dummy_database._dummy_table and while creating a Lineage the process
fails.
Steps to reproduce is running below SQL.
{code}
create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as
(name, ssn);
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
