Re: deadlock in system pool on meta update
Hi,
Here is reproducer for master :
https://github.com/macrergate/ignite/commit/9a7d2d27af30018a5f6faccb39176a35243ccfa2
Best regards,
Sergey Kosarev.
вт, 17 мар. 2020 г. в 12:27, Sergey-A Kosarev :
> Classification: Public
>
> Hi Sergey,
> Ticket is here https://issues.apache.org/jira/browse/IGNITE-12793
>
> I will try to make reproducer in the coming days.
>
> Kind regards,
> Sergey Kosarev
>
> -Original Message-
> From: Sergey Chugunov [mailto:sergey.chugu...@gmail.com]
> Sent: 17 March 2020 09:45
> To: dev
> Subject: Re: deadlock in system pool on meta update
>
> Hello Sergey,
>
> Your analysis looks valid to me, we definitely need to investigate this
> deadlock and find out how to fix it.
>
> Could you create a ticket and write a test that reproduces the issue with
> sufficient probability?
>
> Thanks!
>
> On Mon, Mar 16, 2020 at 8:22 PM Sergey-A Kosarev
> wrote:
>
> > Classification: Public
> >
> > Hi,
> > I've recently tried to apply Ilya's idea (
> > https://issues.apache.org/jira/browse/IGNITE-12663) of minimizing
> > thread pools and tried to set system pool to 3 in my own tests.
> > It caused deadlock on a client node and I think it can happen not only
> > on such small pool values.
> >
> > Details are following:
> > I'm not using persistence currently (if it matters).
> > On the client note I use ignite compute to call a job on every server
> > node (there are 3 server nodes in the tests).
> >
> > Then I've found in logs:
> >
> > [10:55:21] : [Step 1/1] [2020-03-13 10:55:21,773] {
> > grid-timeout-worker-#8} [WARN] [o.a.i.i.IgniteKernal] - Possible
> > thread pool starvation detected (no task completed in last 3ms, is
> > system thread pool size large enough?)
> > [10:55:21] : [Step 1/1] ^-- System thread pool [active=3, idle=0,
> > qSize=9]
> >
> >
> > I see in threaddumps that all 3 system pool workers do the same -
> > processing of job responses:
> >
> > "sys-#26" #605 daemon prio=5 os_prio=0 tid=0x64a0a800
> > nid=0x1f34 waiting on condition [0x7b91d000]
> >java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> > at
> >
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> > at
> >
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> > at
> >
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:749)
> > at
> >
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$1.metadata(CacheObjectBinaryProcessorImpl.java:250)
> > at
> >
> org.apache.ignite.internal.binary.BinaryContext.metadata(BinaryContext.java:1169)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:2005)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:285)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:184)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1797)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2160)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2091)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1914)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1982)
> > at
> >
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:702)
> > at
> >
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:187)
> > at
> >
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:887)
> > at
> >
>
RE: deadlock in system pool on meta update
Classification: Public
Hi Sergey,
Ticket is here https://issues.apache.org/jira/browse/IGNITE-12793
I will try to make reproducer in the coming days.
Kind regards,
Sergey Kosarev
-Original Message-
From: Sergey Chugunov [mailto:sergey.chugu...@gmail.com]
Sent: 17 March 2020 09:45
To: dev
Subject: Re: deadlock in system pool on meta update
Hello Sergey,
Your analysis looks valid to me, we definitely need to investigate this
deadlock and find out how to fix it.
Could you create a ticket and write a test that reproduces the issue with
sufficient probability?
Thanks!
On Mon, Mar 16, 2020 at 8:22 PM Sergey-A Kosarev
wrote:
> Classification: Public
>
> Hi,
> I've recently tried to apply Ilya's idea (
> https://issues.apache.org/jira/browse/IGNITE-12663) of minimizing
> thread pools and tried to set system pool to 3 in my own tests.
> It caused deadlock on a client node and I think it can happen not only
> on such small pool values.
>
> Details are following:
> I'm not using persistence currently (if it matters).
> On the client note I use ignite compute to call a job on every server
> node (there are 3 server nodes in the tests).
>
> Then I've found in logs:
>
> [10:55:21] : [Step 1/1] [2020-03-13 10:55:21,773] {
> grid-timeout-worker-#8} [WARN] [o.a.i.i.IgniteKernal] - Possible
> thread pool starvation detected (no task completed in last 3ms, is
> system thread pool size large enough?)
> [10:55:21] : [Step 1/1] ^-- System thread pool [active=3, idle=0,
> qSize=9]
>
>
> I see in threaddumps that all 3 system pool workers do the same -
> processing of job responses:
>
> "sys-#26" #605 daemon prio=5 os_prio=0 tid=0x64a0a800
> nid=0x1f34 waiting on condition [0x7b91d000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> at
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:749)
> at
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$1.metadata(CacheObjectBinaryProcessorImpl.java:250)
> at
> org.apache.ignite.internal.binary.BinaryContext.metadata(BinaryContext.java:1169)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:2005)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:285)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:184)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1797)
> at
> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2160)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2091)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1914)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1982)
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:702)
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:187)
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:887)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1797)
> at
> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2160)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2091)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1914)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(Bina
Re: deadlock in system pool on meta update
Hello Sergey,
Your analysis looks valid to me, we definitely need to investigate this
deadlock and find out how to fix it.
Could you create a ticket and write a test that reproduces the issue with
sufficient probability?
Thanks!
On Mon, Mar 16, 2020 at 8:22 PM Sergey-A Kosarev
wrote:
> Classification: Public
>
> Hi,
> I've recently tried to apply Ilya's idea (
> https://issues.apache.org/jira/browse/IGNITE-12663) of minimizing thread
> pools and tried to set system pool to 3 in my own tests.
> It caused deadlock on a client node and I think it can happen not only on
> such small pool values.
>
> Details are following:
> I'm not using persistence currently (if it matters).
> On the client note I use ignite compute to call a job on every server
> node (there are 3 server nodes in the tests).
>
> Then I've found in logs:
>
> [10:55:21] : [Step 1/1] [2020-03-13 10:55:21,773] {
> grid-timeout-worker-#8} [WARN] [o.a.i.i.IgniteKernal] - Possible thread
> pool starvation detected (no task completed in last 3ms, is system
> thread pool size large enough?)
> [10:55:21] : [Step 1/1] ^-- System thread pool [active=3, idle=0,
> qSize=9]
>
>
> I see in threaddumps that all 3 system pool workers do the same -
> processing of job responses:
>
> "sys-#26" #605 daemon prio=5 os_prio=0 tid=0x64a0a800 nid=0x1f34
> waiting on condition [0x7b91d000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> at
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:749)
> at
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$1.metadata(CacheObjectBinaryProcessorImpl.java:250)
> at
> org.apache.ignite.internal.binary.BinaryContext.metadata(BinaryContext.java:1169)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:2005)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:285)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:184)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1797)
> at
> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2160)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2091)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1914)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1982)
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:702)
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:187)
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:887)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1797)
> at
> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2160)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2091)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1914)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
> at
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:306)
> at
> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:100)
> at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
> at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10493)
> at
>
