[jira] [Commented] (ATLAS-904) Hive hook fails due to session state not being set
[
https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342238#comment-15342238
]
ATLAS QA commented on ATLAS-904:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12812227/ATLAS-904.4.patch
against master revision 32a5b76.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
+1 checkstyle. The patch generated 0 code style errors.
{color:red}-1 findbugs{color}. The patch appears to introduce 378 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.atlas.repository.typestore.GraphBackedTypeStoreTest
Test results:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningsauthorization.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningsfalcon-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningscatalog.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/345//artifact/patchprocess/newPatchFindbugsWarningstitan.html
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/345//console
This message is automatically generated.
> Hive hook fails due to session state not being set
> --
>
> Key: ATLAS-904
> URL: https://issues.apache.org/jira/browse/ATLAS-904
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Blocker
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-904.1.patch, ATLAS-904.2.patch, ATLAS-904.3.patch,
> ATLAS-904.4.patch, ATLAS-904.patch
>
>
> {noformat}
> 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:normalize(557)) - Could not rewrite query due to error.
> Proceeding with original query EXPORT TABLE test_export_table to
> 'hdfs://localhost:9000/hive_tables/test_path1'
> java.lang.NullPointerException: Conf non-local session path expected to be
> non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641)
> at org.apache.hadoop.hive.ql.Context.(Context.java:133)
> at org.apache.hadoop.hive.ql.Context.(Context.java:120)
> at
> org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44)
> at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702)
> at
>
[jira] [Commented] (ATLAS-904) Hive hook fails due to session state not being set
[
https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339241#comment-15339241
]
Hemanth Yamijala commented on ATLAS-904:
[~suma.shivaprasad], have added some comments to review board. BTW, the patch
wasn't applying cleanly to master.. Can you please rebase?
> Hive hook fails due to session state not being set
> --
>
> Key: ATLAS-904
> URL: https://issues.apache.org/jira/browse/ATLAS-904
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Blocker
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-904.1.patch, ATLAS-904.2.patch, ATLAS-904.patch
>
>
> {noformat}
> 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:normalize(557)) - Could not rewrite query due to error.
> Proceeding with original query EXPORT TABLE test_export_table to
> 'hdfs://localhost:9000/hive_tables/test_path1'
> java.lang.NullPointerException: Conf non-local session path expected to be
> non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641)
> at org.apache.hadoop.hive.ql.Context.(Context.java:133)
> at org.apache.hadoop.hive.ql.Context.(Context.java:120)
> at
> org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44)
> at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702)
> at
> org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596)
> at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222)
> at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77)
> at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:run(184)) - Atlas hook failed due to error
> java.lang.NullPointerException
> at java.lang.StringBuilder.(StringBuilder.java:109)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703)
> at
> org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596)
> at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222)
> at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77)
> at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-904) Hive hook fails due to session state not being set
[
https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338991#comment-15338991
]
ATLAS QA commented on ATLAS-904:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12811744/ATLAS-904.2.patch
against master revision 436a524.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
+1 checkstyle. The patch generated 0 code style errors.
{color:red}-1 findbugs{color}. The patch appears to introduce 379 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.atlas.repository.typestore.GraphBackedTypeStoreTest
Test results:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningsauthorization.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningsfalcon-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningscatalog.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/325//artifact/patchprocess/newPatchFindbugsWarningstitan.html
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/325//console
This message is automatically generated.
> Hive hook fails due to session state not being set
> --
>
> Key: ATLAS-904
> URL: https://issues.apache.org/jira/browse/ATLAS-904
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Blocker
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-904.1.patch, ATLAS-904.2.patch, ATLAS-904.patch
>
>
> {noformat}
> 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:normalize(557)) - Could not rewrite query due to error.
> Proceeding with original query EXPORT TABLE test_export_table to
> 'hdfs://localhost:9000/hive_tables/test_path1'
> java.lang.NullPointerException: Conf non-local session path expected to be
> non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641)
> at org.apache.hadoop.hive.ql.Context.(Context.java:133)
> at org.apache.hadoop.hive.ql.Context.(Context.java:120)
> at
> org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44)
> at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702)
> at
> org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596)
> at
[jira] [Commented] (ATLAS-904) Hive hook fails due to session state not being set
[
https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338961#comment-15338961
]
Suma Shivaprasad commented on ATLAS-904:
https://reviews.apache.org/r/48939
> Hive hook fails due to session state not being set
> --
>
> Key: ATLAS-904
> URL: https://issues.apache.org/jira/browse/ATLAS-904
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Blocker
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-904.1.patch, ATLAS-904.2.patch, ATLAS-904.patch
>
>
> {noformat}
> 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:normalize(557)) - Could not rewrite query due to error.
> Proceeding with original query EXPORT TABLE test_export_table to
> 'hdfs://localhost:9000/hive_tables/test_path1'
> java.lang.NullPointerException: Conf non-local session path expected to be
> non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641)
> at org.apache.hadoop.hive.ql.Context.(Context.java:133)
> at org.apache.hadoop.hive.ql.Context.(Context.java:120)
> at
> org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44)
> at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702)
> at
> org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596)
> at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222)
> at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77)
> at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:run(184)) - Atlas hook failed due to error
> java.lang.NullPointerException
> at java.lang.StringBuilder.(StringBuilder.java:109)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703)
> at
> org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596)
> at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222)
> at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77)
> at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-904) Hive hook fails due to session state not being set
[
https://issues.apache.org/jira/browse/ATLAS-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333127#comment-15333127
]
Hemanth Yamijala commented on ATLAS-904:
[~suma.shivaprasad], I don't know the full details of the code - so please
treat my review comments with a pinch of salt.
There are 2 changes that this patch covers:
* ATLAS-877 - Here we switched from getting a DDL time to the create time. This
change seems right to me. +1.
* ATLAS-904 - What we have essentially done is that instead of fixing the bug
reported here, we have removed the cause of the bug - i.e the normalize method
itself.
So, let's focus on the removal of the normalization. From what I understand,
this was done because Atlas currently does not model partition level lineage.
Hence, by removing literals in queries involving 2 sets of DataSets (inputs &
outputs), we were 'normalizing' partition changes to become like table level
changes. Further, we were capturing the most recent query that ran on this set.
(It appears that this was an array of latest queries, but I don't know if we
were appending to the array, or would it be a replace - in which case we would
capture only the latest query).
I think until we support partition level lineage, sticking to the above model
is useful. If normalization is costly, as it seems the Hive SMEs are telling
us, then can we just make the process name very generic capturing {set of
inputs} -> {set of outputs} in sorted order of input and output names? We could
still store the actual query (unnormalized) into array of latest queries (I
would prefer this is a bounded array - say the last 100 or configurable number
of entries??).
I believe this is a more usable solution than showing all the original
unnormalized queries - which could be very large for all that we know. Please
let me know if this makes sense.
> Hive hook fails due to session state not being set
> --
>
> Key: ATLAS-904
> URL: https://issues.apache.org/jira/browse/ATLAS-904
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Blocker
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-904.1.patch, ATLAS-904.patch
>
>
> {noformat}
> 2016-06-15 11:34:30,423 WARN [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:normalize(557)) - Could not rewrite query due to error.
> Proceeding with original query EXPORT TABLE test_export_table to
> 'hdfs://localhost:9000/hive_tables/test_path1'
> java.lang.NullPointerException: Conf non-local session path expected to be
> non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:641)
> at org.apache.hadoop.hive.ql.Context.(Context.java:133)
> at org.apache.hadoop.hive.ql.Context.(Context.java:120)
> at
> org.apache.atlas.hive.rewrite.HiveASTRewriter.(HiveASTRewriter.java:44)
> at org.apache.atlas.hive.hook.HiveHook.normalize(HiveHook.java:554)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:702)
> at
> org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596)
> at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222)
> at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77)
> at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-06-15 11:34:30,423 ERROR [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:run(184)) - Atlas hook failed due to error
> java.lang.NullPointerException
> at java.lang.StringBuilder.(StringBuilder.java:109)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessQualifiedName(HiveHook.java:738)
> at
> org.apache.atlas.hive.hook.HiveHook.getProcessReferenceable(HiveHook.java:703)
> at
> org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:596)
> at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:222)
> at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:77)
> at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
