Glad that you git it resolved :) On Tue, Oct 28, 2014 at 2:33 PM, Erin Boyd <[email protected]> wrote:
> Hi John, > So I got it running. > I restarted the server and agent and then restarted Hive and it suddenly > works. > I do believe increasing the heap size was necessary, I am just not sure it > was picking it up for the JVM until the server reboot. > > Erin > > > ----- Original Message ----- > From: "Erin Boyd" <[email protected]> > To: [email protected] > Sent: Tuesday, October 28, 2014 11:53:22 AM > Subject: Re: Hive metastore cannot start > > HI John, > I also tried to start it via the command line > [root@ibm-x3650m4-02-vm-02 hive]# hive --service metastore > Starting Hive Metastore Server > Exception in thread "pool-3-thread-31" java.lang.OutOfMemoryError: Java > heap space > at > org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215) > at > org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Exception in thread "pool-3-thread-33" java.lang.OutOfMemoryError: Java > heap space > at > org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215) > at > org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Exception in thread "pool-3-thread-36" java.lang.OutOfMemoryError: Java > heap space > at > org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215) > at > org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > > So from this I increased the heap size to 2048 (it was 1024). > > > It did however allow the listener to start: > [root@ibm-x3650m4-02-vm-02 conf]# netstat -an | grep 9083 > tcp 0 0 0.0.0.0:9083 0.0.0.0:* > LISTEN > tcp 0 0 10.16.40.239:59522 10.16.40.239:9083 > TIME_WAIT > tcp 0 0 10.16.40.239:59563 10.16.40.239:9083 > TIME_WAIT > [root@ibm-x3650m4-02-vm-02 conf]# > > If I start it via the UI, I get: > Python script has been killed due to timeout > > But if I look in the hiveserver2.log I see this: > 2014-10-28 12:53:13,615 INFO [main]: hive.metastore > (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with > URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083 > 2014-10-28 12:53:13,735 INFO [main]: hive.metastore > (HiveMetaStoreClient.java:open(385)) - Connected to metastore. > 2014-10-28 12:53:13,954 INFO [main]: glusterfs.GlusterVolume > (GlusterVolume.java:setConf(105)) - Initializing gluster volume.. > 2014-10-28 12:53:13,955 INFO [main]: glusterfs.GlusterFileSystem > (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS > 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem > (GlusterFileSystem.java:<init>(51)) - Initializing GlusterFS, CRC disabled. > 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem > (GlusterFileSystem.java:<init>(52)) - GIT > INFO={git.commit.id.abbrev=acca0e1, git.commit.user.email= > [email protected], git.commit.message.full=[update RPM > spec file/changelog] - 2.3.3 > , git.commit.id=acca0e11de5f49a67474af70f6758d4182852760, > git.commit.message.short=[update RPM spec file/changelog] - 2.3.3, > git.commit.user.name=Brad Childs, git.build.user.name=Unknown, > git.commit.id.describe=2.3.10-12-gacca0e1, git.build.user.email=Unknown, > git.branch=2.3.3, git.commit.time=29.07.2014 @ 10:52:06 EDT, > git.build.time=29.07.2014 @ 11:06:36 EDT} > 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem > (GlusterFileSystem.java:<init>(53)) - GIT_TAG=2.3.10 > 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem > (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS > 2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume > (GlusterVolume.java:setConf(105)) - Initializing gluster volume.. > 2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume > (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol1 at : > /mnt/glusterfs/HadoopVol1 > 2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume > (GlusterVolume.java:setConf(121)) - Gluster volume: gv0 at : /mnt/gv0 > 2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume > (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol2 at : > /mnt/glusterfs/HadoopVol2 > 2014-10-28 12:53:14,053 INFO [main]: glusterfs.GlusterVolume > (GlusterVolume.java:setConf(155)) - Working directory is : > glusterfs:/user/hive > 2014-10-28 12:53:14,054 INFO [main]: glusterfs.GlusterVolume > (GlusterVolume.java:setConf(164)) - Write buffer size : 131072 > 2014-10-28 12:53:14,055 INFO [main]: glusterfs.GlusterVolume > (GlusterVolume.java:setConf(175)) - Default block size : 67108864 > 2014-10-28 12:53:14,056 INFO [main]: session.SessionState > (SessionState.java:start(361)) - No Tez session required at this point. > hive.execution.engine=mr. > 2014-10-28 12:53:14,070 INFO [main]: service.CompositeService > (SessionManager.java:createBackgroundOperationPool(76)) - HiveServer2: > Background operation thread pool size: 100 > 2014-10-28 12:53:14,071 INFO [main]: service.CompositeService > (SessionManager.java:createBackgroundOperationPool(78)) - HiveServer2: > Background operation thread wait queue size: 100 > 2014-10-28 12:53:14,071 INFO [main]: service.CompositeService > (SessionManager.java:createBackgroundOperationPool(80)) - HiveServer2: > Background operation thread keepalive time: 10 > 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService > (AbstractService.java:init(89)) - Service:OperationManager is inited. > 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService > (AbstractService.java:init(89)) - Service:SessionManager is inited. > 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService > (AbstractService.java:init(89)) - Service:CLIService is inited. > 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService > (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited. > 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService > (AbstractService.java:init(89)) - Service:HiveServer2 is inited. > 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService > (AbstractService.java:start(104)) - Service:OperationManager is started. > 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService > (AbstractService.java:start(104)) - Service:SessionManager is started. > 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService > (AbstractService.java:start(104)) - Service:CLIService is started. > 2014-10-28 12:53:14,130 INFO [main]: hive.metastore > (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with > URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083 > 2014-10-28 12:53:14,133 INFO [main]: hive.metastore > (HiveMetaStoreClient.java:open(385)) - Connected to metastore. > 2014-10-28 12:53:14,209 INFO [main]: service.AbstractService > (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is > started. > 2014-10-28 12:53:14,210 INFO [main]: service.AbstractService > (AbstractService.java:start(104)) - Service:HiveServer2 is started. > 2014-10-28 12:53:14,399 INFO [Thread-4]: thrift.ThriftCLIService > (ThriftBinaryCLIService.java:run(97)) - ThriftBinaryCLIService listening on > 0.0.0.0/0.0.0.0:10000 > > > which makes it appear that it's working.... > > also this: > [root@ibm-x3650m4-02-vm-02 root]# ps -ef | grep hive > hcat 9251 1 0 Oct27 ? 00:02:31 > /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true > -Dwebhcat.log.dir=/var/log/webhcat/ > -Dlog4j.configuration=file:///usr/lib/hive-hcatalog/sbin/../etc/webhcat/webhcat-log4j.properties > -Dhadoop.log.dir=/var/log/hadoop/hcat -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hcat > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Dhadoop.security.logger=INFO,NullAppender > org.apache.hadoop.util.RunJar > /usr/lib/hive-hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-0.13.0.2.1.7.0-784.jar > org.apache.hive.hcatalog.templeton.Main > root 9808 10146 0 13:14 pts/1 00:00:10 > /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true > -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=root > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender > org.apache.hadoop.util.RunJar > /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar > org.apache.hadoop.hive.metastore.HiveMetaStore > root 20972 7782 0 13:32 pts/4 00:00:00 grep hive > hive 27185 1 0 12:49 ? 00:00:23 > /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true > -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hive > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender > org.apache.hadoop.util.RunJar > /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar > org.apache.hive.service.server.HiveServer2 -hiveconf hive.metastore.uris=" > " -hiveconf hive.log.file=hiveserver2.log -hiveconf > hive.log.dir=/var/log/hive > [root@ibm-x3650m4-02-vm-02 root]# > > > Hum........ > > But if you look at the UI, it still says stopped. > > Erin > > > > ----- Original Message ----- > From: "John Speidel" <[email protected]> > To: [email protected] > Sent: Tuesday, October 28, 2014 10:33:12 AM > Subject: Re: Hive metastore cannot start > > Erin, > > Please provide an exported blueprint for your cluster so that we can take a > look at your configs. > > http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME?format=blueprint > > Thanks, > John > > On Tue, Oct 28, 2014 at 11:52 AM, Erin Boyd <[email protected]> wrote: > > > Hi, > > Has anyone else had an issue where the Hive Server and mysql will start > > but not the hive metastore? > > I am getting this in the hiveserver2.log logs: > > > > java.lang.RuntimeException: java.lang.RuntimeException: Unable to > > instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > > at > > > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347) > > at > > > org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:96) > > at > > > org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:64) > > at > > org.apache.hive.service.CompositeService.init(CompositeService.java:59) > > at > org.apache.hive.service.cli.CLIService.init(CLIService.java:109) > > at > > org.apache.hive.service.CompositeService.init(CompositeService.java:59) > > at > > org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68) > > at > > > org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100) > > at > > org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > > Caused by: java.lang.RuntimeException: Unable to instantiate > > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > > at > > > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422) > > at > > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62) > > at > > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > > at > > > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457) > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469) > > at > > > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341) > > ... 13 more > > Caused by: java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > Method) > > at > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at > java.lang.reflect.Constructor.newInstance(Constructor.java:526) > > at > > > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420) > > ... 18 more > > Caused by: MetaException(message:Could not connect to meta store using > any > > of the URIs provided. Most recent failure: > > org.apache.thrift.transport.TTransportException: > java.net.ConnectException: > > Connection refused > > > > This is running local (not remote). THe URI has been the FQDN, locallost, > > 0.0.0.0 -> yet I still get the same error. > > I have tried changing ports... > > > > Anyone seen this and knows how to fix/debug it? > > > > Erin > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
