Hi John, So I got it running. I restarted the server and agent and then restarted Hive and it suddenly works. I do believe increasing the heap size was necessary, I am just not sure it was picking it up for the JVM until the server reboot.
Erin ----- Original Message ----- From: "Erin Boyd" <[email protected]> To: [email protected] Sent: Tuesday, October 28, 2014 11:53:22 AM Subject: Re: Hive metastore cannot start HI John, I also tried to start it via the command line [root@ibm-x3650m4-02-vm-02 hive]# hive --service metastore Starting Hive Metastore Server Exception in thread "pool-3-thread-31" java.lang.OutOfMemoryError: Java heap space at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Exception in thread "pool-3-thread-33" java.lang.OutOfMemoryError: Java heap space at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Exception in thread "pool-3-thread-36" java.lang.OutOfMemoryError: Java heap space at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) So from this I increased the heap size to 2048 (it was 1024). It did however allow the listener to start: [root@ibm-x3650m4-02-vm-02 conf]# netstat -an | grep 9083 tcp 0 0 0.0.0.0:9083 0.0.0.0:* LISTEN tcp 0 0 10.16.40.239:59522 10.16.40.239:9083 TIME_WAIT tcp 0 0 10.16.40.239:59563 10.16.40.239:9083 TIME_WAIT [root@ibm-x3650m4-02-vm-02 conf]# If I start it via the UI, I get: Python script has been killed due to timeout But if I look in the hiveserver2.log I see this: 2014-10-28 12:53:13,615 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083 2014-10-28 12:53:13,735 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(385)) - Connected to metastore. 2014-10-28 12:53:13,954 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(105)) - Initializing gluster volume.. 2014-10-28 12:53:13,955 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(51)) - Initializing GlusterFS, CRC disabled. 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(52)) - GIT INFO={git.commit.id.abbrev=acca0e1, [email protected], git.commit.message.full=[update RPM spec file/changelog] - 2.3.3 , git.commit.id=acca0e11de5f49a67474af70f6758d4182852760, git.commit.message.short=[update RPM spec file/changelog] - 2.3.3, git.commit.user.name=Brad Childs, git.build.user.name=Unknown, git.commit.id.describe=2.3.10-12-gacca0e1, git.build.user.email=Unknown, git.branch=2.3.3, git.commit.time=29.07.2014 @ 10:52:06 EDT, git.build.time=29.07.2014 @ 11:06:36 EDT} 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(53)) - GIT_TAG=2.3.10 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS 2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(105)) - Initializing gluster volume.. 2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol1 at : /mnt/glusterfs/HadoopVol1 2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: gv0 at : /mnt/gv0 2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol2 at : /mnt/glusterfs/HadoopVol2 2014-10-28 12:53:14,053 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(155)) - Working directory is : glusterfs:/user/hive 2014-10-28 12:53:14,054 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(164)) - Write buffer size : 131072 2014-10-28 12:53:14,055 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(175)) - Default block size : 67108864 2014-10-28 12:53:14,056 INFO [main]: session.SessionState (SessionState.java:start(361)) - No Tez session required at this point. hive.execution.engine=mr. 2014-10-28 12:53:14,070 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(76)) - HiveServer2: Background operation thread pool size: 100 2014-10-28 12:53:14,071 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(78)) - HiveServer2: Background operation thread wait queue size: 100 2014-10-28 12:53:14,071 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(80)) - HiveServer2: Background operation thread keepalive time: 10 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited. 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited. 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited. 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited. 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited. 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started. 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started. 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started. 2014-10-28 12:53:14,130 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083 2014-10-28 12:53:14,133 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(385)) - Connected to metastore. 2014-10-28 12:53:14,209 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is started. 2014-10-28 12:53:14,210 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started. 2014-10-28 12:53:14,399 INFO [Thread-4]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(97)) - ThriftBinaryCLIService listening on 0.0.0.0/0.0.0.0:10000 which makes it appear that it's working.... also this: [root@ibm-x3650m4-02-vm-02 root]# ps -ef | grep hive hcat 9251 1 0 Oct27 ? 00:02:31 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dwebhcat.log.dir=/var/log/webhcat/ -Dlog4j.configuration=file:///usr/lib/hive-hcatalog/sbin/../etc/webhcat/webhcat-log4j.properties -Dhadoop.log.dir=/var/log/hadoop/hcat -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hcat -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive-hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-0.13.0.2.1.7.0-784.jar org.apache.hive.hcatalog.templeton.Main root 9808 10146 0 13:14 pts/1 00:00:10 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar org.apache.hadoop.hive.metastore.HiveMetaStore root 20972 7782 0 13:32 pts/4 00:00:00 grep hive hive 27185 1 0 12:49 ? 00:00:23 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar org.apache.hive.service.server.HiveServer2 -hiveconf hive.metastore.uris=" " -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive [root@ibm-x3650m4-02-vm-02 root]# Hum........ But if you look at the UI, it still says stopped. Erin ----- Original Message ----- From: "John Speidel" <[email protected]> To: [email protected] Sent: Tuesday, October 28, 2014 10:33:12 AM Subject: Re: Hive metastore cannot start Erin, Please provide an exported blueprint for your cluster so that we can take a look at your configs. http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME?format=blueprint Thanks, John On Tue, Oct 28, 2014 at 11:52 AM, Erin Boyd <[email protected]> wrote: > Hi, > Has anyone else had an issue where the Hive Server and mysql will start > but not the hive metastore? > I am getting this in the hiveserver2.log logs: > > java.lang.RuntimeException: java.lang.RuntimeException: Unable to > instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347) > at > org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:96) > at > org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:64) > at > org.apache.hive.service.CompositeService.init(CompositeService.java:59) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:109) > at > org.apache.hive.service.CompositeService.init(CompositeService.java:59) > at > org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68) > at > org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100) > at > org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341) > ... 13 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420) > ... 18 more > Caused by: MetaException(message:Could not connect to meta store using any > of the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: java.net.ConnectException: > Connection refused > > This is running local (not remote). THe URI has been the FQDN, locallost, > 0.0.0.0 -> yet I still get the same error. > I have tried changing ports... > > Anyone seen this and knows how to fix/debug it? > > Erin > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
