[
https://issues.apache.org/jira/browse/TAJO-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818718#comment-13818718
]
Jihoon Son commented on TAJO-16:
--------------------------------
Thanks for your patch.
After applying your patch and adding the above catalog configuration, I met the
following errors.
{code}
tajo> \d
2013-11-11 14:56:32,710 INFO rpc.RpcConnectionPool
(RpcConnectionPool.java:closeConnection(78)) - CloseConnection [class
org.apache.tajo.ipc.TajoMasterClientProtocol,localhost/127.0.0.1:26002,false]
2013-11-11 14:56:32,713 ERROR rpc.RpcProtos
(BlockingRpcClient.java:exceptionCaught(205)) - RPC Exception:Exception
[org.apache.tajo.ipc.TajoMasterClientProtocol(127.0.0.1:26002)]:
org.apache.tajo.rpc.RetriesExhaustedException: Failed after attempts=3,
exceptions:
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:91)
at
org.apache.tajo.catalog.AbstractCatalogClient.getAllTableNames(AbstractCatalogClient.java:73)
at
org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.getTableList(TajoMasterClientService.java:273)
at
org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:244)
at
org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:91)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
{code}
The master log is as follows.
{code}
2013-11-11 14:55:36,538 INFO rm.TajoWorkerResourceManager
(TajoWorkerResourceManager.java:run(209)) - WorkerMonitor start
2013-11-11 14:55:36,540 INFO rm.TajoWorkerResourceManager
(TajoWorkerResourceManager.java:run(278)) - WorkerResourceAllocationThread start
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:{TAJO_HOME}/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:{HADOOP_HOME}/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2013-11-11 14:55:41,584 INFO mortbay.log
(NativeMethodAccessorImpl.java:invoke0(?)) - Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-11-11 14:55:41,626 INFO webapp.HttpServer (HttpServer.java:start(333)) -
Jetty bound to port 26080
2013-11-11 14:55:41,626 INFO mortbay.log
(NativeMethodAccessorImpl.java:invoke0(?)) - jetty-6.1.14
2013-11-11 14:55:45,426 INFO mortbay.log
(NativeMethodAccessorImpl.java:invoke0(?)) - Started
[email protected]:26080
2013-11-11 14:55:45,436 INFO master.TajoMaster
(TajoMaster.java:checkAndInitializeSystemDirectories(201)) - Tajo Root
Directory: hdfs://localhost:7020/tajo
2013-11-11 14:55:46,031 INFO master.TajoMaster
(TajoMaster.java:checkAndInitializeSystemDirectories(206)) - FileSystem
(hdfs://localhost:7020) is initialized.
2013-11-11 14:55:46,093 INFO master.TajoMaster
(TajoMaster.java:checkAndInitializeSystemDirectories(226)) - Tajo Warehouse
dir: hdfs://localhost:7020/tajo/warehouse
2013-11-11 14:55:46,095 INFO master.TajoMaster
(TajoMaster.java:checkAndInitializeSystemDirectories(235)) - Staging dir:
hdfs://localhost:7020/tajo/warehouse
2013-11-11 14:55:46,102 WARN storage.AbstractStorageManager
(AbstractStorageManager.java:<init>(85)) - does not support block metadata.
('dfs.datanode.hdfs-blocks-metadata.enabled')
2013-11-11 14:55:46,135 INFO service.AbstractService
(AbstractService.java:init(81)) - Service:Dispatcher is inited.
2013-11-11 14:55:46,138 INFO catalog.CatalogServer
(CatalogServer.java:init(108)) - Catalog Store Class:
org.apache.tajo.catalog.store.HCatalogStore
2013-11-11 14:55:46,141 INFO service.AbstractService
(AbstractService.java:init(81)) - Service:org.apache.tajo.catalog.CatalogServer
is inited.
2013-11-11 14:55:46,141 INFO service.AbstractService
(AbstractService.java:init(81)) - Service:org.apache.tajo.master.GlobalEngine
is inited.
2013-11-11 14:55:46,142 INFO event.AsyncDispatcher
(AsyncDispatcher.java:register(153)) - Registering class
org.apache.tajo.master.querymaster.QueryJobEvent$Type for class
org.apache.tajo.master.querymaster.QueryJobManager$QueryJobManagerEventHandler
2013-11-11 14:55:46,143 INFO service.AbstractService
(AbstractService.java:init(81)) - Service:Dispatcher is inited.
2013-11-11 14:55:46,143 INFO service.AbstractService
(AbstractService.java:init(81)) -
Service:org.apache.tajo.master.querymaster.QueryJobManager is inited.
2013-11-11 14:55:46,143 INFO service.AbstractService
(AbstractService.java:init(81)) -
Service:org.apache.tajo.master.TajoMasterClientService is inited.
2013-11-11 14:55:46,143 INFO service.AbstractService
(AbstractService.java:init(81)) -
Service:org.apache.tajo.master.TajoMasterService is inited.
2013-11-11 14:55:46,143 INFO service.AbstractService
(AbstractService.java:init(81)) - Service:org.apache.tajo.master.TajoMaster is
inited.
2013-11-11 14:55:46,143 INFO master.TajoMaster (TajoMaster.java:init(177)) -
Tajo Master is initialized.
2013-11-11 14:55:46,143 INFO master.TajoMaster (TajoMaster.java:start(412)) -
TajoMaster startup
2013-11-11 14:55:46,143 INFO service.AbstractService
(AbstractService.java:start(94)) - Service:Dispatcher is started.
2013-11-11 14:55:46,201 INFO rpc.NettyServerBase
(NettyServerBase.java:start(106)) - Rpc (CatalogProtocol) listens on
/127.0.0.1:26005
2013-11-11 14:55:46,201 INFO catalog.CatalogServer
(CatalogServer.java:start(156)) - Catalog Server startup (127.0.0.1:26005)
2013-11-11 14:55:46,201 INFO service.AbstractService
(AbstractService.java:start(94)) -
Service:org.apache.tajo.catalog.CatalogServer is started.
2013-11-11 14:55:46,247 INFO service.AbstractService
(AbstractService.java:start(94)) - Service:org.apache.tajo.master.GlobalEngine
is started.
2013-11-11 14:55:46,247 INFO service.AbstractService
(AbstractService.java:start(94)) - Service:Dispatcher is started.
2013-11-11 14:55:46,247 INFO service.AbstractService
(AbstractService.java:start(94)) -
Service:org.apache.tajo.master.querymaster.QueryJobManager is started.
2013-11-11 14:55:46,256 INFO rpc.NettyServerBase
(NettyServerBase.java:start(106)) - Rpc (TajoMasterClientProtocol) listens on
/127.0.0.1:26002
2013-11-11 14:55:46,256 INFO master.TajoMasterClientService
(TajoMasterClientService.java:start(90)) - Instantiated TajoMasterClientService
at /127.0.0.1:26002
2013-11-11 14:55:46,257 INFO service.AbstractService
(AbstractService.java:start(94)) -
Service:org.apache.tajo.master.TajoMasterClientService is started.
2013-11-11 14:55:46,265 INFO rpc.NettyServerBase
(NettyServerBase.java:start(106)) - Rpc (TajoMasterProtocol) listens on
/127.0.0.1:26001
2013-11-11 14:55:46,265 INFO master.TajoMasterService
(TajoMasterService.java:start(72)) - Instantiated TajoMasterService at
/127.0.0.1:26001
2013-11-11 14:55:46,266 INFO service.AbstractService
(AbstractService.java:start(94)) -
Service:org.apache.tajo.master.TajoMasterService is started.
2013-11-11 14:55:46,266 INFO service.AbstractService
(AbstractService.java:start(94)) - Service:org.apache.tajo.master.TajoMaster is
started.
2013-11-11 14:55:46,318 WARN conf.Configuration
(Configuration.java:warnOnceIfDeprecated(817)) - fs.default.name is deprecated.
Instead, use fs.defaultFS
2013-11-11 14:55:49,754 INFO rm.TajoWorkerResourceManager
(TajoWorkerResourceManager.java:workerHeartbeat(487)) -
TajoWorker:host:localhost, port=28093,28091,28092,50461, slots=1024:1:1,
used=0:0:0 added in live TajoWorker list
2013-11-11 14:56:22,035 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:22,049 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:22,049 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013-11-11 14:56:23,049 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:23,050 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:23,050 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013-11-11 14:56:24,051 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:24,052 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:24,052 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013-11-11 14:56:25,631 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:25,631 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:25,631 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013-11-11 14:56:26,631 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:26,632 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:26,632 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013-11-11 14:56:27,633 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:27,634 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:27,634 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013-11-11 14:56:29,693 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:29,694 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:29,694 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013-11-11 14:56:30,694 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:30,695 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:30,695 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013-11-11 14:56:31,695 INFO hive.metastore
(HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI
thrift://localhost:10001
2013-11-11 14:56:31,696 WARN hive.metastore
(HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore
Server...
2013-11-11 14:56:31,696 INFO hive.metastore
(HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection
attempt.
2013. 11. 11 오후 2:56:32 org.jboss.netty.channel.DefaultChannelPipeline
경고: An exception was thrown by a user handler while handling an exception event
([id: 0xbe6bfd26, /127.0.0.1:39838 => /127.0.0.1:26002] EXCEPTION:
org.apache.tajo.rpc.RemoteCallException: Remote call error occurs when
TajoMasterClientProtocolService.getTableListis called:)
org.apache.tajo.rpc.RemoteException: org.apache.tajo.rpc.RemoteCallException:
Remote call error occurs when TajoMasterClientProtocolService.getTableListis
called:
at
org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.exceptionCaught(BlockingRpcServer.java:117)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at
org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at
org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.tajo.rpc.RemoteCallException: Remote call error occurs
when TajoMasterClientProtocolService.getTableListis called:
at
org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:94)
... 16 more
Caused by: org.apache.tajo.rpc.RetriesExhaustedException: Failed after
attempts=3, exceptions:
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:91)
at
org.apache.tajo.catalog.AbstractCatalogClient.getAllTableNames(AbstractCatalogClient.java:73)
at
org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.getTableList(TajoMasterClientService.java:273)
at
org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:244)
at
org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:91)
... 16 more
2013-11-11 14:56:33,317 ERROR hive.metastore
(HiveMetaStoreClient.java:close(346)) - Unable to shutdown local metastore
client
org.apache.thrift.transport.TTransportException: Cannot write to null
outputStream
at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
at
org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
at
org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at
com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:431)
at
com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:425)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:343)
at
org.apache.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.tearDown(HiveClientCache.java:316)
at
org.apache.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.finalize(HiveClientCache.java:331)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
{code}
> Enable Tajo catalog to access Hive metastore.
> ---------------------------------------------
>
> Key: TAJO-16
> URL: https://issues.apache.org/jira/browse/TAJO-16
> Project: Tajo
> Issue Type: Improvement
> Reporter: JaeHwa Jung
> Assignee: JaeHwa Jung
> Fix For: 0.8-incubating
>
> Attachments: TAJO-16.patch, TAJO-16.pdf, TAJO-16_2.pdf
>
>
> Hive has been widely used in this area. Many users have maintained lots of
> big tables through Hive metastore. It would be great to enable Tajo catalog
> to access Hive metastore. Hive metastore already provides thrift API. We
> could employ it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
