RE: ignite server cpu is high
These threads are not actually running (i.e. are not consuming CPU) – they are
blocked on the socket IO.
This SO question explains why they’re RUNNABLE
https://stackoverflow.com/questions/20795295/why-jstack-out-says-thread-state-is-runnable-while-socketread.
> if disk IO is very slow, it will make CPU busy?
Generally, no, not in modern systems.
Another SO question explains this
https://stackoverflow.com/questions/13596997/why-is-the-cpu-not-needed-to-service-i-o-requests.
Stan
From: shawn.du
Sent: 12 июня 2018 г. 6:11
To: user
Subject: Re: ignite server cpu is high
Hi,
I see most running threads are doing below code:
"tcp-disco-sock-reader-#469" #82488 prio=10 os_prio=0
tid=0x7febf8308000 nid=0x3e8f runnable [0x7feb948ef000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x0005cee474e0> (a java.io.BufferedInputStream)
at
org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2657)
at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2673)
at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3150)
at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:859)
at java.io.ObjectInputStream.(ObjectInputStream.java:355)
at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMarshallerObjectInputStream.java:39)
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Our data into ignite are big(at most several Mega bytes.) binary data like
below:
class Data
{
byte[] data;
}
one question:
if disk IO is very slow, it will make CPU busy?
Thanks
Shawn
On 6/12/2018 00:01，Stanislav Lukyanov wrote：
Sorry, but there isn’t much else to be said without additional data.
If consistent usage of resources on that host is important, I suggest to setup
some monitoring so that if it happens again there’ll be at least some place to
start.
Stan
From: shawn.du
Sent: 11 июня 2018 г. 16:34
To: user@ignite.apache.org
Cc: user@ignite.apache.org
Subject: Re: ignite server cpu is high
the server is a dedicate ignite server。i am sure it is ignite consume cpu. I
can’t get more info now, the cpu issue is gone as now there are very few
traffic on our system.
For our case,we store data into ignite every 30 seconds. If no query, the cpu
should be very low. But yesterday it is a exception. Very strange.
shawn.du
邮箱：shawn...@neulion.com.cn
Signature is customized by Netease Mail Master
On 06/11/2018 17:43, Stanislav Lukyanov wrote:
How do you monitor your CPU usage? Do you know which processes consume CPU? Are
you sure it is Ignite’s process? Is CPU consumed more in user space or in
system space? Can you share the actual stats?
>From what I see in the thread dump, there is at least some activity on this
>Ignite: sys-stripe-5-#6 thread is processing an update.
In any case, that thread is the only one in the JVM that is actually performing
some work, so I’d assume that the CPU load comes from other processes.
Thanks,
Stan
From: shawn.du
Sent: 11 июня 2018 г. 5:38
To: user
Subject: ignite server cpu is high
Hi Community,
My single-node ignite cluster has started since Apr 23. In past couple weeks,
It worked fine.
For our case, Ignite server's CPU is very low at most time with exception that
there are complex/concurrent queries.
event in case of query, ignite server's CPU will be high for a very short
while. I think all above is normal behavior.
But yesterday, ignite server's CPU keep high for a long time event we don't
have query.
I use jstack to dump the threads(see the attachment), We don't find any our
business code.
please help and thanks in advance.
We use ignite 2.3.0.
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
Thanks
Shawn
Re: ignite server cpu is high
Hi,I see most running threads are doing below code: "tcp-disco-sock-reader-#469" #82488 prio=10 os_prio=0 tid=0x7febf8308000 nid=0x3e8f runnable [0x7feb948ef000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x0005cee474e0> (a java.io.BufferedInputStream) at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2657) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2673) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3150) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:859) at java.io.ObjectInputStream.(ObjectInputStream.java:355) at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMarshallerObjectInputStream.java:39) at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119) at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740) at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Our data into ignite are big(at most several Mega bytes.) binary data like below: class Data{ byte[] data; }one question:if disk IO is very slow, it will make CPU busy?
ThanksShawn
On 6/12/2018 00:01，Stanislav Lukyanov wrote：
Sorry, but there isn’t much else to be said without additional data.If consistent usage of resources on that host is important, I suggest to setup some monitoring so that if it happens again there’ll be at least some place to start. Stan From: shawn.duSent: 11 июня 2018 г. 16:34To: user@ignite.apache.orgCc: user@ignite.apache.orgSubject: Re: ignite server cpu is high the server is a dedicate ignite server。i am sure it is ignite consume cpu. I can’t get more info now, the cpu issue is gone as now there are very few traffic on our system. For our case,we store data into ignite every 30 seconds. If no query, the cpu should be very low. But yesterday it is a exception. Very strange. shawn.du邮箱：shawn...@neulion.com.cn Signature is customized by Netease Mail MasterOn 06/11/2018 17:43, Stanislav Lukyanov wrote: How do you monitor your CPU usage? Do you know which processes consume CPU? Are you sure it is Ignite’s process? Is CPU consumed more in user space or in system space? Can you share the actual stats? From what I see in the thread dump, there is at least some activity on this Ignite: sys-stripe-5-#6 thread is processing an update.In any case, that thread is the only one in the JVM that is actually performing some work, so I’d assume that the CPU load comes from other processes. Thanks,Stan From: shawn.duSent: 11 июня 2018 г. 5:38To: userSubject: ignite server cpu is high Hi Community, My single-node ignite cluster has started since Apr 23. In past couple weeks, It worked fine. For our case, Ignite server's CPU is very low at most time with exception that there are complex/concurrent queries.event in case of query, ignite server's CPU will be high for a very short while. I think all above is normal behavior.But yesterday, ignite server's CPU keep high for a long time event we don't have query.I use jstack to dump the threads(see the attachment), We don't find any our business code.please help and thanks in advance. We use ignite 2.3.0. java version "1.8.0_151"Java(TM) SE Runtime Environment (build 1.8.0_151-b12)Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) ThanksShawn
RE: ignite server cpu is high
Sorry, but there isn’t much else to be said without additional data. If consistent usage of resources on that host is important, I suggest to setup some monitoring so that if it happens again there’ll be at least some place to start. Stan From: shawn.du Sent: 11 июня 2018 г. 16:34 To: user@ignite.apache.org Cc: user@ignite.apache.org Subject: Re: ignite server cpu is high the server is a dedicate ignite server。i am sure it is ignite consume cpu. I can’t get more info now, the cpu issue is gone as now there are very few traffic on our system. For our case,we store data into ignite every 30 seconds. If no query, the cpu should be very low. But yesterday it is a exception. Very strange. shawn.du 邮箱：shawn...@neulion.com.cn Signature is customized by Netease Mail Master On 06/11/2018 17:43, Stanislav Lukyanov wrote: How do you monitor your CPU usage? Do you know which processes consume CPU? Are you sure it is Ignite’s process? Is CPU consumed more in user space or in system space? Can you share the actual stats? >From what I see in the thread dump, there is at least some activity on this >Ignite: sys-stripe-5-#6 thread is processing an update. In any case, that thread is the only one in the JVM that is actually performing some work, so I’d assume that the CPU load comes from other processes. Thanks, Stan From: shawn.du Sent: 11 июня 2018 г. 5:38 To: user Subject: ignite server cpu is high Hi Community, My single-node ignite cluster has started since Apr 23. In past couple weeks, It worked fine. For our case, Ignite server's CPU is very low at most time with exception that there are complex/concurrent queries. event in case of query, ignite server's CPU will be high for a very short while. I think all above is normal behavior. But yesterday, ignite server's CPU keep high for a long time event we don't have query. I use jstack to dump the threads(see the attachment), We don't find any our business code. please help and thanks in advance. We use ignite 2.3.0. java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) Thanks Shawn
Re: ignite server cpu is high
the server is a dedicate ignite server。i am sure it is ignite consume cpu. I can’t get more info now, the cpu issue is gone as now there are very few traffic on our system. For our case,we store data into ignite every 30 seconds. If no query, the cpu should be very low. But yesterday it is a exception. Very strange. shawn.du 邮箱：shawn...@neulion.com.cn Signature is customized by Netease Mail Master On 06/11/2018 17:43, Stanislav Lukyanov wrote: How do you monitor your CPU usage? Do you know which processes consume CPU? Are you sure it is Ignite’s process? Is CPU consumed more in user space or in system space? Can you share the actual stats? From what I see in the thread dump, there is at least some activity on this Ignite: sys-stripe-5-#6 thread is processing an update.In any case, that thread is the only one in the JVM that is actually performing some work, so I’d assume that the CPU load comes from other processes. Thanks,Stan From: shawn.duSent: 11 июня 2018 г. 5:38To: userSubject: ignite server cpu is high Hi Community, My single-node ignite cluster has started since Apr 23. In past couple weeks, It worked fine. For our case, Ignite server's CPU is very low at most time with exception that there are complex/concurrent queries.event in case of query, ignite server's CPU will be high for a very short while. I think all above is normal behavior.But yesterday, ignite server's CPU keep high for a long time event we don't have query.I use jstack to dump the threads(see the attachment), We don't find any our business code.please help and thanks in advance. We use ignite 2.3.0. java version "1.8.0_151"Java(TM) SE Runtime Environment (build 1.8.0_151-b12)Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) ThanksShawn
RE: ignite server cpu is high
How do you monitor your CPU usage? Do you know which processes consume CPU? Are you sure it is Ignite’s process? Is CPU consumed more in user space or in system space? Can you share the actual stats? >From what I see in the thread dump, there is at least some activity on this >Ignite: sys-stripe-5-#6 thread is processing an update. In any case, that thread is the only one in the JVM that is actually performing some work, so I’d assume that the CPU load comes from other processes. Thanks, Stan From: shawn.du Sent: 11 июня 2018 г. 5:38 To: user Subject: ignite server cpu is high Hi Community, My single-node ignite cluster has started since Apr 23. In past couple weeks, It worked fine. For our case, Ignite server's CPU is very low at most time with exception that there are complex/concurrent queries. event in case of query, ignite server's CPU will be high for a very short while. I think all above is normal behavior. But yesterday, ignite server's CPU keep high for a long time event we don't have query. I use jstack to dump the threads(see the attachment), We don't find any our business code. please help and thanks in advance. We use ignite 2.3.0. java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) Thanks Shawn
ignite server cpu is high
Hi Community,My single-node ignite cluster has started since Apr 23. In past couple weeks, It worked fine. For our case, Ignite server's CPU is very low at most time with exception that there are complex/concurrent queries.event in case of query, ignite server's CPU will be high for a very short while. I think all above is normal behavior.But yesterday, ignite server's CPU keep high for a long time event we don't have query.I use jstack to dump the threads(see the attachment), We don't find any our business code.please help and thanks in advance.We use ignite 2.3.0. java version "1.8.0_151"Java(TM) SE Runtime Environment (build 1.8.0_151-b12)Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) ThanksShawn 4215.1 Description: Binary data
