Thanks Hyunsik,
I prepared the data file as given in the example and i put the file into
HDFS using hdfs put.
the file is now in HDFS at location /home/tanujit/tajo/data
Now when i run the create external table command, get the following error
tajo> create external table test (id int, name string, score float, type
string) using csv with ('csvfile.delimiter'='|') location
'hdfs:/home/tanujit/tajo/data/table1'
Parent path is not a directory: /home/tanujit/tajo/data/table1 table1
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:1467)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:249)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1878)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:1757)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1735)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:429)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:271)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40732)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729)
This is the log output on the server end.
2013-03-29 17:44:33,194 INFO master.GlobalEngine
(GlobalEngine.java:executeQuery(146)) - TQL: create external table test (id
int, name string, score float, type string) using csv with
('csvfile.delimiter'='|') location 'hdfs:/home/tanujit/tajo/data/table1'
2013-03-29 17:44:33,270 DEBUG parser.QueryAnalyzer
(QueryAnalyzer.java:parse(78)) - Analyzer: (CREATE_TABLE test external
(TABLE_DEF (FIELD_DEF id int) (FIELD_DEF name string) (FIELD_DEF score
float) (FIELD_DEF type string)) (using csv) (PARAMS (PARAM
csvfile.delimiter |)) (location hdfs:/home/tanujit/tajo/data/table1))
2013-03-29 17:44:33,271 INFO planner.LogicalPlanner
(LogicalPlanner.java:createPlanInternal(109)) - Planning store statement
2013-03-29 17:44:33,291 INFO master.GlobalEngine
(GlobalEngine.java:createLogicalPlan(237)) - LogicalPlan:
Logical Plan Root
"Store": {"table": "test","schema: "{{id (INT),name (STRING),score
(FLOAT),type (STRING)}},"storeType": "CSV,"path" :
"hdfs:/home/tanujit/tajo/data/table1",
"out schema": null,
"in schema": null}
2013-03-29 17:44:33,310 DEBUG ipc.Client (Client.java:<init>(299)) - The
ping interval is 60000 ms.
2013-03-29 17:44:33,318 DEBUG ipc.Client (Client.java:<init>(323)) - RPC
Server's Kerberos principal name for
protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is null
2013-03-29 17:44:33,321 DEBUG ipc.Client (Client.java:<init>(340)) - Use
SIMPLE authentication for protocol ClientNamenodeProtocolPB
2013-03-29 17:44:33,324 DEBUG ipc.Client (Client.java:setupIOstreams(597))
- Connecting to garfield/192.168.0.103:8020
2013-03-29 17:44:33,335 DEBUG ipc.Client (Client.java:run(839)) - IPC
Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit:
starting, having connections 1
2013-03-29 17:44:33,354 DEBUG ipc.Client (Client.java:run(899)) - IPC
Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit
sending #2
2013-03-29 17:44:33,358 DEBUG ipc.Client (Client.java:receiveResponse(954))
- IPC Client (2719739) connection to garfield/192.168.0.103:8020 from
tanujit got value #2
2013-03-29 17:44:33,358 DEBUG ipc.ProtobufRpcEngine
(ProtobufRpcEngine.java:invoke(217)) - Call: getFileInfo took 48ms
2013-03-29 17:44:33,362 DEBUG ipc.Client (Client.java:run(899)) - IPC
Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit
sending #3
2013-03-29 17:44:33,364 DEBUG ipc.Client (Client.java:receiveResponse(954))
- IPC Client (2719739) connection to garfield/192.168.0.103:8020 from
tanujit got value #3
2013-03-29 17:44:33,365 DEBUG ipc.ProtobufRpcEngine
(ProtobufRpcEngine.java:invoke(217)) - Call: getFileInfo took 6ms
2013-03-29 17:44:33,368 DEBUG ipc.Client (Client.java:run(899)) - IPC
Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit
sending #4
2013-03-29 17:44:33,369 DEBUG ipc.Client (Client.java:receiveResponse(954))
- IPC Client (2719739) connection to garfield/192.168.0.103:8020 from
tanujit got value #4
2013-03-29 17:44:33,370 DEBUG ipc.ProtobufRpcEngine
(ProtobufRpcEngine.java:invoke(217)) - Call: getListing took 3ms
2013-03-29 17:44:33,407 DEBUG hdfs.DFSClient (DFSClient.java:create(1267))
- /home/tanujit/tajo/data/table1/.meta: masked=rw-r--r--
2013-03-29 17:44:33,423 DEBUG hdfs.DFSClient
(DFSOutputStream.java:computePacketChunkSize(1365)) -
computePacketChunkSize: src=/home/tanujit/tajo/data/table1/.meta,
chunkSize=516, chunksPerPacket=127, packetSize=65532
2013-03-29 17:44:33,432 DEBUG ipc.Client (Client.java:run(899)) - IPC
Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit
sending #5
2013-03-29 17:44:33,669 DEBUG ipc.Client (Client.java:receiveResponse(954))
- IPC Client (2719739) connection to garfield/192.168.0.103:8020 from
tanujit got value #5
2013-03-29 17:44:43,432 DEBUG ipc.Client (Client.java:close(1025)) - IPC
Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit:
closed
2013-03-29 17:44:43,432 DEBUG ipc.Client (Client.java:run(857)) - IPC
Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit:
stopped, remaining connections 0
On Fri, Mar 29, 2013 at 5:32 PM, Hyunsik Choi <[email protected]>wrote:
> Hi Tanujit,
>
> The current Tajo cannot create an empty table. You can only create a table
> with an external dataset. Like the section 'Query Execution' in Getting
> Started (http://tajo.incubator.apache.org/getting_started.html), you
> firstly prepare an table data, and then you can create a table with some
> parameters as follows:
>
> create external table table1 (id int, name string, score float, type
> string) using csv with ('csvfile.delimiter'='|') location
> 'hdfs:/home/x/table1'
>
> However, the 'select into' statement and the feature for creating an empty
> table are must-have features. I'll create these features to Tajo Jira.
>
> Best regards,
> Hyunsik
>
>
>
> On Fri, Mar 29, 2013 at 8:48 PM, Tanujit Ghosh <[email protected]
> >wrote:
>
> > Hi,
> >
> > I have build tajo and installed the build and trying to run a simple
> create
> > table statement, but i get the following expception.
> >
> > I have turned on debug logging in log4j.
> >
> > [tanujit@garfield logs]$ tail -f tajo-tanujit-master-garfield.out
> > 2013-03-29 17:02:56,933 DEBUG ipc.Client (Client.java:run(899)) - IPC
> > Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit
> > sending #1
> > 2013-03-29 17:02:56,935 DEBUG ipc.Client
> (Client.java:receiveResponse(954))
> > - IPC Client (2719739) connection to garfield/192.168.0.103:8020 from
> > tanujit got value #1
> > 2013-03-29 17:02:56,936 DEBUG ipc.ProtobufRpcEngine
> > (ProtobufRpcEngine.java:invoke(217)) - Call: getFileInfo took 3ms
> > 2013-03-29 17:02:56,938 DEBUG ipc.YarnRPC (YarnRPC.java:create(57)) -
> > Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> > 2013-03-29 17:02:56,961 INFO master.TajoMaster
> (TajoMaster.java:init(144))
> > - Enabled Distributed Mode
> > 2013-03-29 17:02:57,110 INFO event.AsyncDispatcher
> > (AsyncDispatcher.java:register(153)) - Registering class
> > tajo.master.event.QueryEventType for class
> > tajo.master.TajoMaster$QueryEventDispatcher
> > 2013-03-29 17:02:57,129 INFO service.AbstractService
> > (AbstractService.java:init(81)) - Service:Dispatcher is inited.
> > 2013-03-29 17:02:57,136 INFO catalog.CatalogServer
> > (CatalogServer.java:init(102)) - Catalog Store Class:
> > tajo.catalog.store.DBStore
> > 2013-03-29 17:02:57,804 INFO store.DBStore (DBStore.java:<init>(76)) -
> > Loaded the JDBC driver (org.apache.derby.jdbc.EmbeddedDriver)
> > 2013-03-29 17:02:57,805 INFO store.DBStore (DBStore.java:<init>(82)) -
> > Trying to connect database
> (jdbc:derby:/home/tanujit/tajo/catalog_data/db)
> > 2013-03-29 17:02:59,576 INFO store.DBStore (DBStore.java:<init>(84)) -
> > Connected to database (jdbc:derby:/home/tanujit/tajo/catalog_data/db)
> > 2013-03-29 17:03:00,135 INFO store.DBStore (DBStore.java:<init>(95)) -
> The
> > base tables of CatalogServer already is initialized.
> > 2013-03-29 17:03:00,395 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function sum(INT) is
> > registered.
> > 2013-03-29 17:03:00,396 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function sum(LONG) is
> > registered.
> > 2013-03-29 17:03:00,396 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function sum(FLOAT) is
> > registered.
> > 2013-03-29 17:03:00,396 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function sum(DOUBLE) is
> > registered.
> > 2013-03-29 17:03:00,397 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function max(INT) is
> > registered.
> > 2013-03-29 17:03:00,397 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function max(LONG) is
> > registered.
> > 2013-03-29 17:03:00,397 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function max(FLOAT) is
> > registered.
> > 2013-03-29 17:03:00,397 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function max(DOUBLE) is
> > registered.
> > 2013-03-29 17:03:00,398 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function min(INT) is
> > registered.
> > 2013-03-29 17:03:00,398 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function min(LONG) is
> > registered.
> > 2013-03-29 17:03:00,398 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function min(FLOAT) is
> > registered.
> > 2013-03-29 17:03:00,398 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function min(DOUBLE) is
> > registered.
> > 2013-03-29 17:03:00,398 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function min(STRING) is
> > registered.
> > 2013-03-29 17:03:00,399 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function avg(INT) is
> > registered.
> > 2013-03-29 17:03:00,400 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function avg(LONG) is
> > registered.
> > 2013-03-29 17:03:00,400 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function avg(FLOAT) is
> > registered.
> > 2013-03-29 17:03:00,400 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function avg(DOUBLE) is
> > registered.
> > 2013-03-29 17:03:00,400 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function count(ANY) is
> > registered.
> > 2013-03-29 17:03:00,401 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function count() is
> > registered.
> > 2013-03-29 17:03:00,401 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function
> > in_country(STRING,STRING) is registered.
> > 2013-03-29 17:03:00,401 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function country(STRING) is
> > registered.
> > 2013-03-29 17:03:00,401 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function date(STRING) is
> > registered.
> > 2013-03-29 17:03:00,402 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function today() is
> > registered.
> > 2013-03-29 17:03:00,402 INFO catalog.CatalogServer
> > (CatalogServer.java:registerFunction(407)) - Function random(INT) is
> > registered.
> > 2013-03-29 17:03:00,402 INFO service.AbstractService
> > (AbstractService.java:init(81)) - Service:tajo.catalog.CatalogServer is
> > inited.
> > 2013-03-29 17:03:00,402 INFO service.AbstractService
> > (AbstractService.java:init(81)) - Service:tajo.master.GlobalEngine is
> > inited.
> > 2013-03-29 17:03:00,403 INFO service.AbstractService
> > (AbstractService.java:init(81)) - Service:tajo.master.ClientService is
> > inited.
> > 2013-03-29 17:03:00,403 INFO service.AbstractService
> > (AbstractService.java:init(81)) - Service:tajo.master.TajoMaster is
> inited.
> > 2013-03-29 17:03:00,403 INFO master.TajoMaster
> > (TajoMaster.java:start(296)) - TajoMaster startup
> > 2013-03-29 17:03:00,403 INFO service.AbstractService
> > (AbstractService.java:start(94)) - Service:Dispatcher is started.
> > 2013-03-29 17:03:00,782 INFO rpc.NettyServerBase
> > (NettyServerBase.java:start(84)) - RpcServer on /127.0.0.1:9002
> > 2013-03-29 17:03:00,782 INFO catalog.CatalogServer
> > (CatalogServer.java:start(142)) - Catalog Server startup (127.0.0.1:9002
> )
> > 2013-03-29 17:03:00,783 INFO service.AbstractService
> > (AbstractService.java:start(94)) - Service:tajo.catalog.CatalogServer is
> > started.
> > 2013-03-29 17:03:00,787 DEBUG ipc.HadoopYarnProtoRPC
> > (HadoopYarnProtoRPC.java:getProxy(45)) - Creating a HadoopYarnProtoRpc
> > proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> > 2013-03-29 17:03:00,996 DEBUG master.GlobalEngine
> > (GlobalEngine.java:connectRMClient(274)) - Connecting to ResourceManager
> at
> > /0.0.0.0:8032
> > 2013-03-29 17:03:01,076 INFO service.AbstractService
> > (AbstractService.java:start(94)) - Service:tajo.master.GlobalEngine is
> > started.
> > 2013-03-29 17:03:01,120 INFO rpc.NettyServerBase
> > (NettyServerBase.java:start(84)) - RpcServer on /0:0:0:0:0:0:0:0:9004
> > 2013-03-29 17:03:01,121 INFO master.ClientService
> > (ClientService.java:start(93)) - Instantiated ClientService at
> > /0:0:0:0:0:0:0:0:9004
> > 2013-03-29 17:03:01,121 INFO service.AbstractService
> > (AbstractService.java:start(94)) - Service:tajo.master.ClientService is
> > started.
> > 2013-03-29 17:03:01,121 INFO service.AbstractService
> > (AbstractService.java:start(94)) - Service:tajo.master.TajoMaster is
> > started.
> > 2013-03-29 17:03:06,933 DEBUG ipc.Client (Client.java:close(1025)) - IPC
> > Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit:
> > closed
> > 2013-03-29 17:03:06,935 DEBUG ipc.Client (Client.java:run(857)) - IPC
> > Client (2719739) connection to garfield/192.168.0.103:8020 from tanujit:
> > stopped, remaining connections 0
> > 2013-03-29 17:03:38,775 DEBUG store.DBStore
> > (DBStore.java:getAllTableNames(745)) - SELECT TABLE_ID from TABLES
> > 2013-03-29 17:03:59,233 INFO master.GlobalEngine
> > (GlobalEngine.java:executeQuery(146)) - TQL: create table test(x string)
> > 2013-03-29 17:03:59,488 DEBUG parser.QueryAnalyzer
> > (QueryAnalyzer.java:parse(78)) - Analyzer: (CREATE_TABLE test (TABLE_DEF
> > (FIELD_DEF x string)))
> > 2013-03-29 17:03:59,489 INFO planner.LogicalPlanner
> > (LogicalPlanner.java:createPlanInternal(109)) - Planning store statement
> > Mar 29, 2013 5:03:59 PM org.jboss.netty.channel.DefaultChannelPipeline
> > WARNING: An exception was thrown by a user handler while handling an
> > exception event ([id: 0x0194df96, /192.168.0.103:48058 => /
> > 192.168.0.103:9004] EXCEPTION: tajo.rpc.RemoteCallException: Remote call
> > error occurs when ClientProtocolService.submitQueryis called:)
> > tajo.rpc.RemoteException: tajo.rpc.RemoteCallException: Remote call error
> > occurs when ClientProtocolService.submitQueryis called:
> > at
> >
> >
> tajo.rpc.ProtoBlockingRpcServer$ServerHandler.exceptionCaught(ProtoBlockingRpcServer.java:121)
> > at
> >
> >
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:66)
> > at
> >
> >
> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
> > at
> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
> > at
> >
> >
> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
> > at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
> > at
> >
> >
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:76)
> > at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
> > at
> >
> >
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
> > at
> >
> >
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
> > at
> >
> >
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
> > at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
> > at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
> > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
> > at
> >
> >
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: tajo.rpc.RemoteCallException: Remote call error occurs when
> > ClientProtocolService.submitQueryis called:
> > at
> >
> >
> tajo.rpc.ProtoBlockingRpcServer$ServerHandler.messageReceived(ProtoBlockingRpcServer.java:97)
> > ... 14 more
> > Caused by: java.lang.NullPointerException
> > at
> >
> >
> tajo.client.ClientProtocol$SubmitQueryRespose$Builder.setErrorMessage(ClientProtocol.java:2932)
> > at
> >
> >
> tajo.master.ClientService$ClientProtocolHandler.submitQuery(ClientService.java:135)
> > at
> >
> >
> tajo.client.ClientProtocol$ClientProtocolService$2.callBlockingMethod(ClientProtocol.java:11912)
> > at
> >
> >
> tajo.rpc.ProtoBlockingRpcServer$ServerHandler.messageReceived(ProtoBlockingRpcServer.java:94)
> > ... 14 more
> >
> >
> > Is there some setting that i have done wrong ?
> > I cant quite figure it out.
> >
> >
> > Regards,
> > Tanujit
> >
>
--
Regards,
Tanujit
