[
https://issues.apache.org/jira/browse/TAJO-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923627#comment-13923627
]
Keuntae Park commented on TAJO-678:
-----------------------------------
I found an exception occurred before the above exception.
{noformat}
2014-03-07 15:11:41,585 INFO worker.Fetcher (Fetcher.java:get(129)) - Fetch:
http://skt-rf-88:39979/?qid=q_1394170620737_0001&sid=1&p=373&type=h&ta=7873_0,4879_0,4387_0,7875_0
100614 2014-03-07 15:11:41,741 INFO worker.Fetcher (Fetcher.java:get(129)) -
Fetch:
http://skt-rf-55:44134/?qid=q_1394170620737_0001&sid=1&p=428&type=h&ta=2690_0
100615 2014-03-07 15:11:41,756 INFO worker.Fetcher (Fetcher.java:get(129)) -
Fetch:
http://skt-rf-82:37300/?qid=q_1394170620737_0001&sid=1&p=154&type=h&ta=1149_0,2679_0,2455_0
100616 2014-03-07 15:11:42,535 INFO worker.Task
(Task.java:waitForFetch(355)) - ta_1394170620737_0001_000002_000000_00 All
fetches are done!
100617 2014-03-07 15:11:44,431 INFO planner.PhysicalPlannerImpl
(PhysicalPlannerImpl.java:createBestAggregationPlan(955)) - The planner chooses
[Hash Aggregation]
100618 2014-03-07 15:11:44,431 INFO planner.PhysicalPlannerImpl
(PhysicalPlannerImpl.java:createInMemoryHashAggregation(898)) - The planner
chooses [Hash Aggregation]
100619 2014-03-07 15:15:01,466 ERROR worker.Task (Task.java:run(392)) -
java.io.FileNotFoundException:
/data5/tajo/tajo-localdir/q_1394170620737_0001/in/eb_1394170620737_0001_000002/0/0/eb_1394170620737_0001_000001/in_22000
(Too many open files)
100620 at java.io.FileInputStream.open(Native Method)
100621 at java.io.FileInputStream.<init>(FileInputStream.java:120)
100622 at
org.apache.tajo.storage.RawFile$RawFileScanner.init(RawFile.java:85)
100623 at
org.apache.tajo.storage.MergeScanner.getNextScanner(MergeScanner.java:127)
100624 at
org.apache.tajo.storage.MergeScanner.next(MergeScanner.java:108)
100625 at
org.apache.tajo.engine.planner.physical.SeqScanExec.next(SeqScanExec.java:168)
100626 at
org.apache.tajo.engine.planner.physical.HashAggregateExec.compute(HashAggregateExec.java:51)
100627 at
org.apache.tajo.engine.planner.physical.HashAggregateExec.next(HashAggregateExec.java:77)
100628 at
org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:76)
100629 at org.apache.tajo.worker.Task.run(Task.java:383)
100630 at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:395)
100631 at java.lang.Thread.run(Thread.java:662)
100632
100633 2014-03-07 15:15:01,466 INFO worker.TaskAttemptContext
(TaskAttemptContext.java:setState(110)) - Query status of
ta_1394170620737_0001_000002_000000_00 is changed to TA_FAILED
100634 2014-03-07 15:15:01,476 INFO worker.Task (Task.java:run(446)) - Task
Counter - total:144, succeeded: 143, killed: 0, failed: 1
100635 2014-03-07 15:15:01,477 INFO worker.TaskRunner
(TaskRunner.java:run(336)) - Request GetTask:
eb_1394170620737_0001_000002,container_1394170620737_0001_01_000780
100636 2014-03-07 15:15:01,708 INFO worker.TaskRunner
(TaskRunner.java:run(374)) - Accumulated Received Task: 2
{noformat}
> Too many open files error
> -------------------------
>
> Key: TAJO-678
> URL: https://issues.apache.org/jira/browse/TAJO-678
> Project: Tajo
> Issue Type: Bug
> Reporter: Keuntae Park
>
> During the fetch phase, too many open file exception occurred.
> {noformat}
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-52:49073/?qid=q_1394170620737_0001&sid=1&p=486&type=h&ta=4390_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-55:44134/?qid=q_1394170620737_0001&sid=1&p=270&type=h&ta=9044_0,4744_0,51_0,7815_0,7619_0,6923_0,6735_0,3548_0,7607_0,4013_0,3234_0,2757_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-83:34835/?qid=q_1394170620737_0001&sid=1&p=111&type=h&ta=7644_0,7739_0,8937_0,4305_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-112:51947/?qid=q_1394170620737_0001&sid=1&p=30&type=h&ta=3224_0,6164_0,2627_0,2630_0,5929_0,9160_0,5506_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-64:59899/?qid=q_1394170620737_0001&sid=1&p=12&type=h&ta=7879_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-88:39979/?qid=q_1394170620737_0001&sid=1&p=373&type=h&ta=7873_0,4879_0,4387_0,7875_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-55:44134/?qid=q_1394170620737_0001&sid=1&p=428&type=h&ta=2690_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-82:37300/?qid=q_1394170620737_0001&sid=1&p=154&type=h&ta=1149_0,2679_0,2455_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(197)) - * Local
> task dir: file:/data1/tajo/tajo-localdir/q_1394170620737_0001/output/2/0_1
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(202)) -
> ==================================
> 2014-03-07 15:15:03,362 INFO worker.Task (Task.java:init(218)) - the
> directory is created
> file:/data3/tajo/tajo-localdir/q_1394170620737_0001/in/eb_1394170620737_0001_000002/0/1/eb_1394170620737_0001_000001
> 2014-03-07 15:15:03,365 ERROR worker.TaskRunner (TaskRunner.java:run(397)) -
> Failed to create a selector.
> org.jboss.netty.channel.ChannelException: Failed to create a selector.
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:337)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.<init>(NioClientBoss.java:63)
> at
> org.jboss.netty.channel.socket.nio.NioClientBossPool.newBoss(NioClientBossPool.java:61)
> at
> org.jboss.netty.channel.socket.nio.NioClientBossPool.newBoss(NioClientBossPool.java:27)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioBossPool.init(AbstractNioBossPool.java:65)
> at
> org.jboss.netty.channel.socket.nio.NioClientBossPool.<init>(NioClientBossPool.java:45)
> at
> org.apache.tajo.rpc.RpcChannelFactory.createClientChannelFactory(RpcChannelFactory.java:69)
> at org.apache.tajo.worker.Task.getFetchRunners(Task.java:626)
> at org.apache.tajo.worker.Task.localize(Task.java:236)
> at org.apache.tajo.worker.Task.init(Task.java:224)
> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:390)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Too many open files
> at sun.nio.ch.IOUtil.initPipe(Native Method)
> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> at java.nio.channels.Selector.open(Selector.java:209)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:335)
> ... 12 more
> 2014-03-07 15:15:03,366 INFO worker.TaskRunner (TaskRunner.java:run(336)) -
> Request GetTask:
> eb_1394170620737_0001_000002,container_1394170620737_0001_01_000780
> 2014-03-07 15:15:03,580 INFO worker.TaskRunner (TaskRunner.java:run(374)) -
> Accumulated Received Task: 3
> 2014-03-07 15:15:03,580 INFO worker.TaskRunner (TaskRunner.java:run(383)) -
> Initializing: ta_1394170620737_0001_000002_000000_02
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(182)) - Output
> File Path:
> hdfs://nameservice1/tmp/tajo-tajo/staging/q_1394170620737_0001/RESULT/part-02-000000
> 2014-03-07 15:15:03,583 INFO worker.TaskAttemptContext
> (TaskAttemptContext.java:setState(110)) - Query status of
> ta_1394170620737_0001_000002_000000_02 is changed to TA_PENDING
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(187)) -
> ==================================
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(188)) - *
> Subquery ta_1394170620737_0001_000002_000000_02 is initialized
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(189)) - *
> InterQuery: false
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(192)) - *
> Fragments (num: 1)
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(193)) - * Fetches
> (total:47922) :
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-75:39431/?qid=q_1394170620737_0001&sid=1&p=3&type=h&ta=2925_0,7796_0,9038_0,8593_0,6694_0,9229_0,8360_0,2903_0,574_0,4357_0,9022_0,3137_0
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-14:33755/?qid=q_1394170620737_0001&sid=1&p=168&type=h&ta=3103_0
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-63:49448/?qid=q_1394170620737_0001&sid=1&p=541&type=h&ta=9039_0,2625_0
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-09:38879/?qid=q_1394170620737_0001&sid=1&p=648&type=h&ta=2476_0,50_0,9063_0,4536_0,2685_0,4168_0
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id:
> eb_1394170620737_0001_000001, url:
> http://skt-rf-73:34510/?qid=q_1394170620737_0001&sid=1&p=453&type=h&ta=7168_0,3176_0,2626_0,3841_0,7926_0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
