Re: MapReduce Job stuck when using ignite hadoop accelerator
Hi Kaiming, ^-- Public thread pool [active=80, idle=0, qSize=944] There are long queue and 80 busy threads that seemd do no progress. It looks like all of threads are blocked. Please, attach a thread-dump. On Tue, Nov 29, 2016 at 6:40 AM, Kaiming Wan <344277...@qq.com> wrote: > I can find the WARN in logs. The stuck should be caused by a long running > cache operations. How to locate what cause the long running cache > operations. My map-reduce job can run successfully without ignite hadoop > accelerator. > > [GridCachePartitionExchangeManager] Found long running cache operations, > dump IO statistics. > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/MapReduce-Job-stuck-when-using-ignite- > hadoop-accelerator-tp9216p9251.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- С уважением, Машенков Андрей Владимирович Тел. +7-921-932-61-82 Best regards, Andrey V. Mashenkov Cerr: +7-921-932-61-82
Re: MapReduce Job stuck when using ignite hadoop accelerator
I can find the WARN in logs. The stuck should be caused by a long running cache operations. How to locate what cause the long running cache operations. My map-reduce job can run successfully without ignite hadoop accelerator. [GridCachePartitionExchangeManager] Found long running cache operations, dump IO statistics. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/MapReduce-Job-stuck-when-using-ignite-hadoop-accelerator-tp9216p9251.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
MapReduce Job stuck when using ignite hadoop accelerator
:40,668][INFO ][grid-timeout-worker-#201%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=1988aad6, name=null, uptime=01:27:00:430]
^-- H/N/C [hosts=3, nodes=3, CPUs=120]
^-- CPU [cur=0.03%, avg=0.18%, GC=0%]
^-- Heap [used=19992MB, free=63.39%, comm=39744MB]
^-- Non heap [used=148MB, free=98.58%, comm=153MB]
^-- Public thread pool [active=80, idle=0, qSize=944]
^-- System thread pool [active=0, idle=80, qSize=0]
^-- Outbound messages queue [size=0]
[18:26:40,674][INFO ][grid-timeout-worker-#201%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=1988aad6, name=null, uptime=01:28:00:439]
^-- H/N/C [hosts=3, nodes=3, CPUs=120]
^-- CPU [cur=0.03%, avg=0.18%, GC=0%]
^-- Heap [used=1MB, free=63.38%, comm=39744MB]
^-- Non heap [used=148MB, free=98.58%, comm=153MB]
^-- Public thread pool [active=80, idle=0, qSize=944]
^-- System thread pool [active=0, idle=80, qSize=0]
^-- Outbound messages queue [size=0]
How to solve this problem and what cause this?
>From the log info, I find free heap space diminish by 0.02% every several
minutes. And "thread pool starvation detected" warnning appeared now and
then.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/MapReduce-Job-stuck-when-using-ignite-hadoop-accelerator-tp9216.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
