[jira] [Updated] (ATLAS-904) Hive hook fails due to session state not being set
[ https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated ATLAS-904: --- Attachment: ATLAS-904.4.patch There were minor import rebases in HiveHookIT > Hive hook fails due to session state not being set > -- > > Key: ATLAS-904 > URL: https://issues.apache.org/jira/browse/ATLAS-904 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Blocker > Fix For: 0.7-incubating > > Attachments: ATLAS-904.1.patch, ATLAS-904.2.patch, ATLAS-904.3.patch, > ATLAS-904.4.patch, ATLAS-904.patch > > > {noformat} > 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:normalize(557)) - Could not rewrite query due to error. > Proceeding with original query EXPORT TABLE test_export_table to > 'hdfs://localhost:9000/hive_tables/test_path1' > java.lang.NullPointerException: Conf non-local session path expected to be > non-null > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at > org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641) > at org.apache.hadoop.hive.ql.Context.(Context.java:133) > at org.apache.hadoop.hive.ql.Context.(Context.java:120) > at > org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44) > at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:run(184)) - Atlas hook failed due to error > java.lang.NullPointerException > at java.lang.StringBuilder.(StringBuilder.java:109) > at > org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-904) Hive hook fails due to session state not being set
[ https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-904: --- Attachment: ATLAS-904.3.patch Addressed review comments and rebased > Hive hook fails due to session state not being set > -- > > Key: ATLAS-904 > URL: https://issues.apache.org/jira/browse/ATLAS-904 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Blocker > Fix For: 0.7-incubating > > Attachments: ATLAS-904.1.patch, ATLAS-904.2.patch, ATLAS-904.3.patch, > ATLAS-904.patch > > > {noformat} > 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:normalize(557)) - Could not rewrite query due to error. > Proceeding with original query EXPORT TABLE test_export_table to > 'hdfs://localhost:9000/hive_tables/test_path1' > java.lang.NullPointerException: Conf non-local session path expected to be > non-null > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at > org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641) > at org.apache.hadoop.hive.ql.Context.(Context.java:133) > at org.apache.hadoop.hive.ql.Context.(Context.java:120) > at > org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44) > at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:run(184)) - Atlas hook failed due to error > java.lang.NullPointerException > at java.lang.StringBuilder.(StringBuilder.java:109) > at > org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-904) Hive hook fails due to session state not being set
[ https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-904: --- Attachment: ATLAS-904.2.patch > Hive hook fails due to session state not being set > -- > > Key: ATLAS-904 > URL: https://issues.apache.org/jira/browse/ATLAS-904 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Blocker > Fix For: 0.7-incubating > > Attachments: ATLAS-904.1.patch, ATLAS-904.2.patch, ATLAS-904.patch > > > {noformat} > 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:normalize(557)) - Could not rewrite query due to error. > Proceeding with original query EXPORT TABLE test_export_table to > 'hdfs://localhost:9000/hive_tables/test_path1' > java.lang.NullPointerException: Conf non-local session path expected to be > non-null > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at > org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641) > at org.apache.hadoop.hive.ql.Context.(Context.java:133) > at org.apache.hadoop.hive.ql.Context.(Context.java:120) > at > org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44) > at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:run(184)) - Atlas hook failed due to error > java.lang.NullPointerException > at java.lang.StringBuilder.(StringBuilder.java:109) > at > org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-904) Hive hook fails due to session state not being set
[ https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-904: --- Attachment: (was: ATLAS-904.2.patch) > Hive hook fails due to session state not being set > -- > > Key: ATLAS-904 > URL: https://issues.apache.org/jira/browse/ATLAS-904 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Blocker > Fix For: 0.7-incubating > > Attachments: ATLAS-904.1.patch, ATLAS-904.patch > > > {noformat} > 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:normalize(557)) - Could not rewrite query due to error. > Proceeding with original query EXPORT TABLE test_export_table to > 'hdfs://localhost:9000/hive_tables/test_path1' > java.lang.NullPointerException: Conf non-local session path expected to be > non-null > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at > org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641) > at org.apache.hadoop.hive.ql.Context.(Context.java:133) > at org.apache.hadoop.hive.ql.Context.(Context.java:120) > at > org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44) > at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:run(184)) - Atlas hook failed due to error > java.lang.NullPointerException > at java.lang.StringBuilder.(StringBuilder.java:109) > at > org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-904) Hive hook fails due to session state not being set
[ https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-904: --- Attachment: ATLAS-904.2.patch Changes to address [~yhemanth] review comments. 1. Process qualified name = HiveOperation.name + sorted inputs + sorted outputs 2. HiveOperation.name doesnt provide identifiers for identiifying INSERT, INSERT_OVERWRITE, UPDATE, DELETE etc separately . Hence adding WriteEntity.WriteType as well which exhibits the following behaviour a. If there are multiple outputs, for each output, adds the query type(WriteType) b. if query being run if is type INSERT [into/overwrite] TABLE [PARTITION], WriteType is INSERT/INSERT_OVERWRITE b. If query is of type INSERT OVERWRITE hdfs_path, adds WriteType as PATH_WRITE c. If query is of type UPDATE/DELETE, adds type as UPDATE/DELETE [ Note - linage is not available for this since this is single table operation] > Hive hook fails due to session state not being set > -- > > Key: ATLAS-904 > URL: https://issues.apache.org/jira/browse/ATLAS-904 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Blocker > Fix For: 0.7-incubating > > Attachments: ATLAS-904.1.patch, ATLAS-904.2.patch, ATLAS-904.patch > > > {noformat} > 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:normalize(557)) - Could not rewrite query due to error. > Proceeding with original query EXPORT TABLE test_export_table to > 'hdfs://localhost:9000/hive_tables/test_path1' > java.lang.NullPointerException: Conf non-local session path expected to be > non-null > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at > org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641) > at org.apache.hadoop.hive.ql.Context.(Context.java:133) > at org.apache.hadoop.hive.ql.Context.(Context.java:120) > at > org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44) > at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:run(184)) - Atlas hook failed due to error > java.lang.NullPointerException > at java.lang.StringBuilder.(StringBuilder.java:109) > at > org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-904) Hive hook fails due to session state not being set
[ https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-904: --- Summary: Hive hook fails due to session state not being set (was: Hive hook fails due to rewrite query error in some cases like export to hdfs path) > Hive hook fails due to session state not being set > -- > > Key: ATLAS-904 > URL: https://issues.apache.org/jira/browse/ATLAS-904 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Blocker > Fix For: 0.7-incubating > > Attachments: ATLAS-904.1.patch, ATLAS-904.patch > > > {noformat} > 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:normalize(557)) - Could not rewrite query due to error. > Proceeding with original query EXPORT TABLE test_export_table to > 'hdfs://localhost:9000/hive_tables/test_path1' > java.lang.NullPointerException: Conf non-local session path expected to be > non-null > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at > org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641) > at org.apache.hadoop.hive.ql.Context.(Context.java:133) > at org.apache.hadoop.hive.ql.Context.(Context.java:120) > at > org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44) > at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook > (HiveHook.java:run(184)) - Atlas hook failed due to error > java.lang.NullPointerException > at java.lang.StringBuilder.(StringBuilder.java:109) > at > org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738) > at > org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703) > at > org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596) > at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222) > at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77) > at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)