[
https://issues.apache.org/jira/browse/TAJO-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876106#comment-13876106
]
Min Zhou commented on TAJO-525:
-------------------------------
The internal error is caused by a failure of an execution block. I changed some
code, and found something similar to TAJO-522
{noformat}
014-01-20 01:53:04,959 ERROR master.ContainerProxy
(TajoContainerProxy.java:assignExecutionBlock(88)) -
java.lang.OutOfMemoryError: unable to create new native thread
java.io.IOException: java.lang.OutOfMemoryError: unable to create new native
thread
at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76)
at org.apache.tajo.rpc.AsyncRpcClient.<init>(AsyncRpcClient.java:67)
at
org.apache.tajo.master.TajoContainerProxy.assignExecutionBlock(TajoContainerProxy.java:72)
at
org.apache.tajo.master.TajoContainerProxy.launch(TajoContainerProxy.java:59)
at
org.apache.tajo.worker.TajoResourceAllocator$LaunchRunner.run(TajoResourceAllocator.java:189)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
at
org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
at
org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at
org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
at
org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
at
org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
at
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
at
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54)
... 10 more
{noformat}
This process ate a lot of memory for threads, I even can't launch jstack on
that machine
{noformat}
$ jps
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /export/home/yarn/hs_err_pid19677.log
{noformat}
pstack shows the process started over 1000 threads, will submit the pstack
result soon.
> Concurrent queries hang
> -----------------------
>
> Key: TAJO-525
> URL: https://issues.apache.org/jira/browse/TAJO-525
> Project: Tajo
> Issue Type: Bug
> Affects Versions: 0.8-incubating
> Reporter: Min Zhou
> Priority: Critical
> Fix For: 0.8-incubating
>
>
> When I run a simple query "select count(1) from lineitem;" in one tsql
> session and after that run the 2nd query in another tsql session. The 2nd
> one hung.
> After the 1st one done, the 2nd one still stuck at 0%.
> After that, run the same query in the first tsql session, the 3rd query also
> hung.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
