[
https://issues.apache.org/jira/browse/TAJO-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13756684#comment-13756684
]
JaeHwa Jung commented on TAJO-144:
----------------------------------
I found two kinds of errors.
First, If I use "SELECT * FROM" clause, it made NullPointerException as follows:
{code:xml}
tajo> insert overwrite into table1 select * from table2;
java.lang.NullPointerException
at
org.apache.tajo.engine.planner.rewrite.ProjectionPushDownRule.rewrite(ProjectionPushDownRule.java:71)
at
org.apache.tajo.engine.planner.rewrite.BasicQueryRewriteEngine.rewrite(BasicQueryRewriteEngine.java:63)
at
org.apache.tajo.engine.planner.LogicalOptimizer.optimize(LogicalOptimizer.java:40)
at
org.apache.tajo.master.GlobalEngine.createLogicalPlan(GlobalEngine.java:189)
at
org.apache.tajo.master.GlobalEngine.executeQuery(GlobalEngine.java:110)
at
org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.submitQuery(TajoMasterClientService.java:135)
at
org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:241)
at
org.apache.tajo.rpc.ProtoBlockingRpcServer$ServerHandler.messageReceived(ProtoBlockingRpcServer.java:91)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)</test>
{code}
Second, If I write each columns, it can't progress more than zero percent as
follows:
{code:xml}
tajo> insert overwrite into table4 select id, name, score, type from table2;
2013-09-03 23:51:26,073 INFO client.TajoClient
(TajoClient.java:connectionToQueryMaster(190)) - Connected to Query Master
(qid=q_1378219667565_0004, addr=192.168.1.103:26121)
Progress: 0%, response time: 1.313 sec
Progress: 0%, response time: 2.315 sec
Progress: 0%, response time: 3.317 sec
Progress: 0%, response time: 4.318 sec
Progress: 0%, response time: 5.319 sec
Progress: 0%, response time: 6.321 sec
Progress: 0%, response time: 7.323 sec
Progress: 0%, response time: 8.325 sec
Progress: 0%, response time: 9.327 sec
Progress: 0%, response time: 10.328 sec
Progress: 0%, response time: 11.33 sec
{code}
In second case, Tajo master write log file as follows:
{code:xml}
2013-09-03 23:51:22,023 INFO master.GlobalEngine
(GlobalEngine.java:executeQuery(106)) - SQL: insert overwrite into table4
select id, name, score, type from table2
2013-09-03 23:51:22,041 INFO rewrite.BasicQueryRewriteEngine
(BasicQueryRewriteEngine.java:rewrite(64)) - The rule "ProjectionPushDown "
rewrites the query.
2013-09-03 23:51:22,047 INFO master.GlobalEngine
(GlobalEngine.java:createLogicalPlan(190)) - LogicalPlan:
Logical Plan Root
INSERT OVERWRITE INTO{
"tableName": "table4",
"uri": {},
"meta": {
"schema": {
"fields": [
{
"name": "table4.id",
"dataType": {}
},
{
"name": "table4.name",
"dataType": {}
},
{
"name": "table4.score",
"dataType": {}
},
{
"name": "table4.type",
"dataType": {}
}
],
"fieldsByName": {
"table4.score": 2,
"table4.name": 1,
"table4.id": 0,
"table4.type": 3
}
},
"storeType": "CSV",
"options": {
"keyVals": {
"csvfile.delimiter": "|"
}
},
"stat": {
"numRows": 0,
"numBytes": 0,
"numBlocks": 0,
"numPartitions": 0,
"avgRows": 0,
"columnStats": []
}
}
} [SUBQUERY]
2013-09-03 23:51:22,051 INFO service.AbstractService
(AbstractService.java:init(81)) -
Service:org.apache.tajo.master.TajoAsyncDispatcher is inited.
2013-09-03 23:51:22,051 INFO service.AbstractService
(AbstractService.java:init(81)) -
Service:org.apache.tajo.master.querymaster.QueryInProgress is inited.
2013-09-03 23:51:22,052 INFO service.AbstractService
(AbstractService.java:start(94)) -
Service:org.apache.tajo.master.TajoAsyncDispatcher is started.
2013-09-03 23:51:22,052 INFO master.TajoAsyncDispatcher
(TajoAsyncDispatcher.java:start(101)) - AsyncDispatcher
started:QueryInProgress:q_1378219667565_0004
2013-09-03 23:51:22,052 INFO service.AbstractService
(AbstractService.java:start(94)) -
Service:org.apache.tajo.master.querymaster.QueryInProgress is started.
2013-09-03 23:51:22,052 INFO querymaster.QueryInProgress
(QueryInProgress.java:startQueryMaster(145)) - Initializing QueryInProgress for
QueryID=q_1378219667565_0004
2013-09-03 23:51:22,052 INFO rm.YarnTajoResourceManager
(YarnTajoResourceManager.java:allocateAndLaunchQueryMaster(204)) - Allocate and
launch ApplicationMaster for QueryMaster: queryId=q_1378219667565_0004,
appId=application_1378219667565_0004
2013-09-03 23:51:22,053 INFO master.ContainerProxy
(YarnContainerProxy.java:createCommonContainerLaunchContext(233)) - Set the
environment for the application master
2013-09-03 23:51:22,053 INFO master.ContainerProxy
(YarnContainerProxy.java:createCommonContainerLaunchContext(301)) - defaultFS:
hdfs://localhost:9010
2013-09-03 23:51:22,053 INFO master.ContainerProxy
(YarnContainerProxy.java:createCommonContainerLaunchContext(302)) - defaultFS:
hdfs://localhost:9010
2013-09-03 23:51:22,055 INFO master.ContainerProxy
(YarnContainerProxy.java:createCommonContainerLaunchContext(316)) - Writing a
QueryConf to HDFS and add to local environment
2013-09-03 23:51:22,134 INFO rm.YarnTajoResourceManager
(YarnTajoResourceManager.java:allocateAndLaunchQueryMaster(265)) - Completed
setting up QueryMasterRunner command ${JAVA_HOME}/bin/java -Xmx2000m
org.apache.tajo.worker.TajoWorker qm q_1378219667565_0004
blrunnerui-MacBook-Pro.local:9005 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
2013-09-03 23:51:22,135 INFO rm.YarnTajoResourceManager
(YarnTajoResourceManager.java:allocateAndLaunchQueryMaster(283)) - Submitting
QueryMaster to ResourceManager
2013-09-03 23:51:22,137 INFO client.YarnClientImpl
(YarnClientImpl.java:submitApplication(134)) - Submitted application
application_1378219667565_0004 to ResourceManager at /0.0.0.0:8032
2013-09-03 23:51:22,139 INFO rm.YarnTajoResourceManager
(YarnTajoResourceManager.java:monitorApplication(303)) - Got application report
from ASM for, appId=4, appAttemptId=appattempt_1378219667565_0004_000001,
clientToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default,
appMasterRpcPort=0, appStartTime=1378219882136, yarnAppState=SUBMITTED,
distributedFinalState=UNDEFINED,
appTrackingUrl=blrunnerui-MacBook-Pro.local:8088/proxy/application_1378219667565_0004/,
appUser=blrunner
2013-09-03 23:51:22,242 INFO rm.YarnTajoResourceManager
(YarnTajoResourceManager.java:monitorApplication(303)) - Got application report
from ASM for, appId=4, appAttemptId=appattempt_1378219667565_0004_000001,
clientToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default,
appMasterRpcPort=0, appStartTime=1378219882136, yarnAppState=ACCEPTED,
distributedFinalState=UNDEFINED,
appTrackingUrl=blrunnerui-MacBook-Pro.local:8088/proxy/application_1378219667565_0004/,
appUser=blrunner
2013-09-03 23:51:22,242 INFO rm.YarnTajoResourceManager
(YarnTajoResourceManager.java:allocateAndLaunchQueryMaster(289)) - Launching
QueryMaster with appAttemptId: appattempt_1378219667565_0004_000001
2013-09-03 23:51:22,242 INFO querymaster.QueryInProgress
(QueryInProgress.java:submmitQueryToMaster(199)) - No QueryMaster conneciton
info.
2013-09-03 23:51:25,769 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:25,769 INFO querymaster.QueryInProgress
(QueryInProgress.java:connectQueryMaster(182)) - Connect to
QueryMaster:blrunnerui-MacBook-Pro.local/192.168.1.103:26122
2013-09-03 23:51:25,773 INFO querymaster.QueryInProgress
(QueryInProgress.java:submmitQueryToMaster(203)) - Call executeQuery to
:blrunnerui-MacBook-Pro.local:26122,q_1378219667565_0004
2013-09-03 23:51:27,496 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:29,497 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:31,498 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:33,499 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:35,500 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:37,502 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:39,503 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:41,504 INFO querymaster.QueryInProgress
(QueryInProgress.java:heartbeat(233)) - Received QueryMaster
heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
{code}
> Implement INSERT OVERWRITE clause
> ---------------------------------
>
> Key: TAJO-144
> URL: https://issues.apache.org/jira/browse/TAJO-144
> Project: Tajo
> Issue Type: New Feature
> Components: planner/optimizer
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2-incubating
>
> Attachments: TAJO-144.patch
>
>
> 'INSERT INTO' appends data into existing table. In contrast, 'INSERT
> OVERWRITE' will overwrite existing table data. It is used as an idiom in
> analytical fields based HDFS. We need it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
