Madhan,
To reiterate what I’m trying to accomplish, hooking Apache Hive on the Atlas
which was installed on Azure HD Insight cluster.
I have changed the settings as stated in the document.
hive.exec.post.hooks: org.apache.atlas.hive.hook.HiveHook
export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}:/home/sshuser/
apache-atlas-sources-1.1.0/distro/target/apache-atlas-1.1.0-server/apache-atlas-1.1.0/hook/hive
However when I tried to view tables using HIVE Ambari view in HDInsight cluster
I’m getting the below error. However when I tried to run Hive from command
prompt I’m able to insert the data for the second time(I believe one of the
Hive metastore is corrupted) and was able to retrieve from atlas.
Any help in this regard is greatly appreciated.
java.sql.SQLException: Error running query: java.lang.NoClassDefFoundError:
Could not initialize class org.apache.atlas.hive.hook.HiveHook
java.sql.SQLException: Error running query: java.lang.NoClassDefFoundError:
Could not initialize class org.apache.atlas.hive.hook.HiveHook
at
org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:348)
at
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251)
at
org.apache.ambari.view.hive20.HiveJdbcConnectionDelegate.execute(HiveJdbcConnectionDelegate.java:49)
at
org.apache.ambari.view.hive20.actor.StatementExecutor.runStatement(StatementExecutor.java:91)
at
org.apache.ambari.view.hive20.actor.StatementExecutor.handleMessage(StatementExecutor.java:72)
at
org.apache.ambari.view.hive20.actor.HiveActor.onReceive(HiveActor.java:38)
at
akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
at
akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at
akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
(less...)<https://hdinsighttestmo.azurehdinsight.net/views/HIVE/2.0.0/AUTO_HIVE20_INSTANCE/>
Mahender
From: Madhan Neethiraj <[email protected]>
Sent: Tuesday, November 20, 2018 1:56 PM
To: Mahender Shankesi <[email protected]>
Cc: Shankesi Mahender <[email protected]>; dev <[email protected]>; Apoorv
Naik <[email protected]>
Subject: Re: Need Help!!! Atlas Installation Error
EXTERNAL
Mahender,
In that case, please look for other error messages in Hive log. It will help
troubleshooting, if you can provide complete logs.
Madhan
From: Mahender Shankesi
<[email protected]<mailto:[email protected]>>
Date: Tuesday, November 20, 2018 at 10:46 AM
To: Madhan Neethiraj <[email protected]<mailto:[email protected]>>
Cc: Shankesi Mahender <[email protected]<mailto:[email protected]>>, dev
<[email protected]<mailto:[email protected]>>, Apoorv Naik
<[email protected]<mailto:[email protected]>>
Subject: RE: Need Help!!! Atlas Installation Error
Madhan,
I’ve followed the same steps and getting the same error.
Mahender
From: Madhan Neethiraj <[email protected]<mailto:[email protected]>>
Sent: Tuesday, November 20, 2018 1:44 PM
To: Mahender Shankesi
<[email protected]<mailto:[email protected]>>
Cc: Shankesi Mahender <[email protected]<mailto:[email protected]>>; dev
<[email protected]<mailto:[email protected]>>; Apoorv Naik
<[email protected]<mailto:[email protected]>>
Subject: Re: Need Help!!! Atlas Installation Error
EXTERNAL
Mahender,
Please make sure to perform the following, in the hosts where Hive server run
(from https://atlas.apache.org/Hook-Hive.html):
· untar apache-atlas-${project.version}-hive-hook.tar.gz
· cd apache-atlas-hive-hook-${project.version}
· Copy entire contents of folder
apache-atlas-hive-hook-${project.version}/hook/hive to <atlas package>/hook/hive
· Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in
hive-env.sh of your hive configuration
Hope this helps.
Madhan
From: Mahender Shankesi
<[email protected]<mailto:[email protected]>>
Date: Tuesday, November 20, 2018 at 10:12 AM
To: Apoorv Naik <[email protected]<mailto:[email protected]>>
Cc: Shankesi Mahender <[email protected]<mailto:[email protected]>>, Madhan
Neethiraj <[email protected]<mailto:[email protected]>>, dev
<[email protected]<mailto:[email protected]>>
Subject: RE: RE: Need Help!!! Atlas Installation Error
Apoorv,
I’m trying to hook Hive to the apache and followed the instructions at the
below page..
https://atlas.apache.org/Hook-Hive.html
However I’m getting the below error while starting the hive server in HDInsight
cluster. Do you have any idea about this?
java.sql.SQLException: Error running query: java.lang.NoClassDefFoundError:
Could not initialize class org.apache.atlas.hive.hook.HiveHook
java.sql.SQLException: Error running query: java.lang.NoClassDefFoundError:
Could not initialize class org.apache.atlas.hive.hook.HiveHook
at
org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:348)
at
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251)
at
org.apache.ambari.view.hive20.HiveJdbcConnectionDelegate.execute(HiveJdbcConnectionDelegate.java:49)
at
org.apache.ambari.view.hive20.actor.StatementExecutor.runStatement(StatementExecutor.java:91)
at
org.apache.ambari.view.hive20.actor.StatementExecutor.handleMessage(StatementExecutor.java:72)
at
org.apache.ambari.view.hive20.actor.HiveActor.onReceive(HiveActor.java:38)
at
akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
at
akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at
akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Thanks,
Mahender
From: Apoorv Naik <[email protected]<mailto:[email protected]>>
Sent: Friday, November 2, 2018 2:22 PM
To: Mahender Shankesi
<[email protected]<mailto:[email protected]>>
Cc: Shankesi Mahender <[email protected]<mailto:[email protected]>>; Madhan
Neethiraj <[email protected]<mailto:[email protected]>>; dev
<[email protected]<mailto:[email protected]>>
Subject: Re: RE: Need Help!!! Atlas Installation Error
EXTERNAL
Not that I know of.
Apoorv Naik
On Fri, Nov 2, 2018 at 9:42 AM Mahender Shankesi
<[email protected]<mailto:[email protected]>> wrote:
Apoorv/Dev Team,
We are facing challenges on install Atlas on HDinsight cluster, did anyone able
to install it earlier on Azure HDInsight Clusters?
Mahender
From: Apoorv Naik <[email protected]<mailto:[email protected]>>
Sent: Wednesday, October 24, 2018 3:37 PM
To: Mahender Shankesi
<[email protected]<mailto:[email protected]>>
Cc: Shankesi Mahender <[email protected]<mailto:[email protected]>>; Madhan
Neethiraj <[email protected]<mailto:[email protected]>>; dev
<[email protected]<mailto:[email protected]>>
Subject: Re: RE: Need Help!!! Atlas Installation Error
EXTERNAL
There are a bunch of models that get loaded during the startup process.
typically the files will follow *_model.json pattern.
https://github.com/apache/atlas/tree/master/addons/models
Apoorv Naik
On Wed, Oct 24, 2018 at 12:22 PM Mahender Shankesi
<[email protected]<mailto:[email protected]>> wrote:
Apoorv,
My atlas is up and running. I tried to find a datamodel for the tables we
loaded as quick_start.py I couldn’t find one. I saw the data is coming up but
not sure how the backend data model works. If some one from your team has a
data model can you please share with me.
Thanks,
Mahender
From: Apoorv Naik <[email protected]<mailto:[email protected]>>
Sent: Friday, October 19, 2018 3:52 PM
To: Mahender Shankesi
<[email protected]<mailto:[email protected]>>; Apoorv Naik
<[email protected]<mailto:[email protected]>>
Cc: Shankesi Mahender <[email protected]<mailto:[email protected]>>; Madhan
Neethiraj <[email protected]<mailto:[email protected]>>; dev
<[email protected]<mailto:[email protected]>>
Subject: Re: RE: Need Help!!! Atlas Installation Error
EXTERNAL
Everything should be available on atlas.apache.org<http://atlas.apache.org> if
you get stuck please reach out to the community and mention any gaps in the
documentation (if any).
On Oct 19, 2018 at 12:48 PM, <Mahender
Shankesi<mailto:[email protected]>> wrote:
Apoorv,
Thanks for your email. I did installed one more time with only HBASE installed
this time it worked and was able to view through browser. I will get back to
you if I have any issues.
Do you have any other documentation on Atlas for further study.
Mahender
From: Apoorv Naik <[email protected]<mailto:[email protected]>>
Sent: Friday, October 19, 2018 2:55 PM
To: Mahender Shankesi
<[email protected]<mailto:[email protected]>>
Cc: Shankesi Mahender <[email protected]<mailto:[email protected]>>; Madhan
Neethiraj <[email protected]<mailto:[email protected]>>; dev
<[email protected]<mailto:[email protected]>>
Subject: Re: Need Help!!! Atlas Installation Error
EXTERNAL
I tried checking the classpath for duplicate netty dependency but couldn't find
it.
Can you clean your local maven repo and run the following command ? Please
share the output if you see multiple netty JARs in the output.
mvn dependency:build-classpath -Pdist,embedded-cassandra-solr -pl webapp
On Fri, Oct 19, 2018 at 9:19 AM Mahender Shankesi
<[email protected]<mailto:[email protected]>> wrote:
Apoorv,
Any update on this?
Mahender
From: Apoorv Naik <[email protected]<mailto:[email protected]>>
Sent: Wednesday, October 17, 2018 12:11 PM
To: Mahender Shankesi
<[email protected]<mailto:[email protected]>>
Cc: Shankesi Mahender <[email protected]<mailto:[email protected]>>; Madhan
Neethiraj <[email protected]<mailto:[email protected]>>; dev
<[email protected]<mailto:[email protected]>>
Subject: Re: Need Help!!! Atlas Installation Error
EXTERNAL
Hi Mahender,
Sorry I didn't have time to investigate this. I'll try looking at it today or
tomorrow.
On Wed, Oct 17, 2018 at 6:56 AM Mahender Shankesi
<[email protected]<mailto:[email protected]>> wrote:
Dev Team,
Any update on this?
Mahender
From: Shankesi Mahender <[email protected]<mailto:[email protected]>>
Sent: Monday, October 15, 2018 9:30 AM
To: [email protected]<mailto:[email protected]>; Mahender Shankesi
<[email protected]<mailto:[email protected]>>
Cc: Madhan Neethiraj <[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: Re: Need Help!!! Atlas Installation Error
EXTERNAL
Apoorv/Dev Team,
Any update on the below request? For tracking purpose I have created a JIRA
ticket with Apache.
https://issues.apache.org/jira/browse/ATLAS-2919
Meanwhile can anyone share the Datamodel of ATLAS.
Thanks & Regards,
Mahender
On Thu, Oct 11, 2018 at 6:44 PM Shankesi Mahender
<[email protected]<mailto:[email protected]>> wrote:
Thanks Apoorv.
Mahender
On Thu, Oct 11, 2018 at 4:21 PM Apoorv Naik
<[email protected]<mailto:[email protected]>> wrote:
This looks like an issue with the packaging, a wrong version of netty is
getting pulled in. I'll try to find the source and update the thread.
On Thu, Oct 11, 2018 at 1:10 PM Shankesi Mahender
<[email protected]<mailto:[email protected]>> wrote:
Please find attached the application log. Did some more debugging and found out
that starting cassandra is an issue.
Any help would be greatly appreciated.
I got at one situation where I can't proceed further.
Regards,
Mahender
On Thu, Oct 11, 2018 at 12:56 PM Shankesi Mahender
<[email protected]<mailto:[email protected]>> wrote:
Madhan/Dev team,
I'm able to install by changing the command mvn clean -DskipTests package
-Pdist,embedded-cassandra-solr
I faced the same issue with executing the curl command again..
$<mailto:idtooleval@data-management-tools-eval:~/mahen/apache-atlas-sources-1.1.0/distro/target/apache-atlas-1.1.0/bin$>
curl -u admin:admin
http://myip:21000/api/atlas/admin/version<http://10.41.64.19:21000/api/atlas/admin/version>
curl: (7) Failed to connect to myip port 21000: Connection refused
$<mailto:idtooleval@data-management-tools-eval:~/mahen/apache-atlas-sources-1.1.0/distro/target/apache-atlas-1.1.0/bin$>
curl -u admin:admin
https://myip/api/atlas/admin/version<https://10.41.64.19:21443/api/atlas/admin/version>
curl: (7) Failed to connect to myip port 21443: Connection refused
I went to atlas_config.py and change the DEFAULT_ATLAS_SERVER_HOST to my ip.
None of them seems to be working.
$ netstat -a | grep 21000
netstat -a | grep 21443
Didn't return anything.
Do I need to do anything extra which is not documented in the instructions?
Thanks for your help.
Mahender
On Wed, Oct 10, 2018 at 11:54 PM Shankesi Mahender
<[email protected]<mailto:[email protected]>> wrote:
Dev team,
I'm able to install when I changed the command like below.
mvn clean -DskipTests package -Pdist,embedded-cassandra-solr.
After that when I started apache server I'm getting the below error.
File "atlas_start.py", line 107
print "Cannot run setup when server is running."
^
SyntaxError: Missing parentheses in call to 'print'
I tried to stop the server if it is already started but I got the same kind of
error.
File "atlas_stop.py", line 63
print "Apache Atlas Server stopped!!!\n"
^
SyntaxError: Missing parentheses in call to 'print'
Can anyone help me on this?
Thanks,
Mahender
On Wed, Oct 10, 2018 at 4:23 PM Mahender Shankesi
<[email protected]<mailto:[email protected]>> wrote:
I changed it but still not working.
Mahender
Sent from my iPhone
On Oct 10, 2018, at 4:03 PM, Madhan Neethiraj
<[email protected]<mailto:[email protected]>> wrote:
(copying dev list)
Mahender – the port # to use is “21000”; the curl commands you listed show
“2100”. Please try after correcting this.
@Pierre – perhaps you can help with the question on Cassandra?
Madhan
From: Shankesi Mahender <[email protected]<mailto:[email protected]>>
Date: Wednesday, October 10, 2018 at 12:37 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Need Help!!! Atlas Installation Error
Madhan,
I followed the instructions in the below site..
https://atlas.apache.org/InstallationSteps.html
I did it in two machines one Azure VM and one Azure Hdinisights.
Hdinsights I'm able to install all steps correctly and run apache_start
properly. However I'm not able to run the curl command..
sshuser@XXX:~/xyz/apache-atlas-sources-1.1.0/distro/target/apache-atlas-1.1.0/bin$<mailto:sshuser@XXX:~/xyz/apache-atlas-sources-1.1.0/distro/target/apache-atlas-1.1.0/bin$>
curl -u admin:admin https://xxx.azurehdinsight.net:2100/api/atlas/admin/version
^C
sshuser@xxx:~/xxx/apache-atlas-sources-1.1.0/distro/target/apache-atlas-1.1.0/bin$<mailto:sshuser@xxx:~/xxx/apache-atlas-sources-1.1.0/distro/target/apache-atlas-1.1.0/bin$>
curl -u admin:admin https://localhost:2100/api/atlas/admin/version
curl: (7) Failed to connect to localhost port 2100: Connection refused
However I'm facing challenges in install the steps in VM..
mvn clean package -Pdist,embedded-cassandra-solr
I'm getting the below error and it is hanging there..
pache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:117)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:93)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:117)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:93)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:117)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:93)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:117)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:93)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:117)
at
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:93)
Any help is greatly appreciated..
Mahender
at org.a
On Tue, Oct 9, 2018 at 3:09 PM Madhan Neethiraj
<[email protected]<mailto:[email protected]>> wrote:
Mahender,
“port 80”? Atlas should be listening in port 21000 by default. Which URL are
you getting this error for?
Thanks,
Madhan
From: Shankesi Mahender <[email protected]<mailto:[email protected]>>
Date: Tuesday, October 9, 2018 at 11:56 AM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Need Help!!! Atlas Installation Error
Thanks Madhan..
When I tried to run the curl command with admin:admin I got below error.
curl: (7) Failed to connect to ipaddress port 80: Connection refused
When I tried to start atlas_start.py I got the following error
File "atlas_start.py", line 107
print "Cannot run setup when server is running."
^
SyntaxError: Missing parentheses in call to 'print'
Do you know anything about this?
Thanks for your help.
Mahender
On Tue, Oct 9, 2018 at 2:07 PM Madhan Neethiraj
<[email protected]<mailto:[email protected]>> wrote:
Mahender,
Default username:password is: admin:admin. Please try with these and let me
know.
Thanks,
Madhan
From: Shankesi Mahender <[email protected]<mailto:[email protected]>>
Date: Tuesday, October 9, 2018 at 6:03 AM
To: Shankesi Mahender <[email protected]<mailto:[email protected]>>
Subject: Need Help!!! Atlas Installation Error
I’m trying to install Apache Atlas on my Hdinsight cluster and following the
instructions from the below page.
https://atlas.apache.org/InstallationSteps.html
I was able to install correctly until the below point.
curl -u username:password http://localhost:21000/api/atlas/admin/version
When I execute this command it is throwing an error, I have replace the
localhost with my azure hdinsight one and error says
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>403 - Forbidden: Access is denied.</title>
<style type="text/css">
I’m not sure what should I provide for username and password, I tried different
ways username:password, admin:admin and my cluster id/password none of them
seems to be working.
Do you know any default userid and password is there for the installation? I
didn’t see anywhere in the documentation.
Thanks for your help in advance.
Mahender
GSK monitors email communications sent to and from GSK in order to protect GSK,
our employees, customers, suppliers and business partners, from cyber threats
and loss of GSK Information. GSK monitoring is conducted with appropriate
confidentiality controls and in accordance with local laws and after
appropriate consultation.
GSK monitors email communications sent to and from GSK in order to protect GSK,
our employees, customers, suppliers and business partners, from cyber threats
and loss of GSK Information. GSK monitoring is conducted with appropriate
confidentiality controls and in accordance with local laws and after
appropriate consultation.
GSK monitors email communications sent to and from GSK in order to protect GSK,
our employees, customers, suppliers and business partners, from cyber threats
and loss of GSK Information. GSK monitoring is conducted with appropriate
confidentiality controls and in accordance with local laws and after
appropriate consultation.
GSK monitors email communications sent to and from GSK in order to protect GSK,
our employees, customers, suppliers and business partners, from cyber threats
and loss of GSK Information. GSK monitoring is conducted with appropriate
confidentiality controls and in accordance with local laws and after
appropriate consultation.
GSK monitors email communications sent to and from GSK in order to protect GSK,
our employees, customers, suppliers and business partners, from cyber threats
and loss of GSK Information. GSK monitoring is conducted with appropriate
confidentiality controls and in accordance with local laws and after
appropriate consultation.
GSK monitors email communications sent to and from GSK in order to protect GSK,
our employees, customers, suppliers and business partners, from cyber threats
and loss of GSK Information. GSK monitoring is conducted with appropriate
confidentiality controls and in accordance with local laws and after
appropriate consultation.
GSK monitors email communications sent to and from GSK in order to protect GSK,
our employees, customers, suppliers and business partners, from cyber threats
and loss of GSK Information. GSK monitoring is conducted with appropriate
confidentiality controls and in accordance with local laws and after
appropriate consultation.
GSK monitors email communications sent to and from GSK in order to protect GSK,
our employees, customers, suppliers and business partners, from cyber threats
and loss of GSK Information. GSK monitoring is conducted with appropriate
confidentiality controls and in accordance with local laws and after
appropriate consultation.