Hi,
We are using Apache Ignite 2.7.0 binary and servers are using Linux OS &
app servers are using Windows OS.We are using Apache Ignite .Net APIs.
Recently we have noticed that our application is not stopped with an OOM
error.
App server has 32 GB RAM & We are specifying JVM Heap = 8 GB
I am
On of these, depending on your query type:
* new ScanQuery() { PageSize = 5 }
* new SqlQuery() { PageSize = 5 }
On Wed, Oct 28, 2020 at 5:24 PM Ravi Makwana
wrote:
> Hi Paval,
>
> As we are not setting explicitly QueryBase.Pagesize for SqlQuery and
> SqlFieldQuery so default will be used as
Hi Paval,
As we are not setting explicitly QueryBase.Pagesize for SqlQuery and
SqlFieldQuery so default will be used as 1024.
We have not found so far any example by looking the same we can try to
explicitly set the Query base.Pagesize.
Can we have any reference by checking that we can try to
On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, wrote:
> I found a bug in Ignite [1] which probably causes the issue on your side.
>
> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and
> the size of one results page exceeds 2GB.
> Please try using a smaller value for
I found a bug in Ignite [1] which probably causes the issue on your side.
Looks like you are running a query (is it a ScanQuery or SqlQuery?) and the
size of one results page exceeds 2GB.
Please try using a smaller value for *QueryBase.PageSize*.
If you use the default value of 1024, your cache
Hi,
Our service is running with 64 bit and we have verified the same in our app
server too.
Any finding from the logs?
Is there any way to replicate it?
Thanks,
On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn wrote:
> Looks like the app is running in 32 bit mode, which can't use more than
>
Looks like the app is running in 32 bit mode, which can't use more than 2GB
of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to
use.
Please check the `Platform` column in the Task Manager - does it say `32
bit`?
If yes, then try disabling `Prefer 32 bit` in the
Hi Ravi,
The exception indicates that Ignite.NET failed to allocate unmanaged memory
on .NET side while trying to pass query data from Java to .NET.
This indicates that your system has run out of memory. Possible reasons are:
* Memory is consumed by other apps
* Memory is consumed by this app
*
Hi,
We are using Apache Ignite 2.7.0 binary and servers are using Linux OS &
app servers are using Windows OS.We are using Apache Ignite .Net APIs.
Recently we have noticed that our application is stopping due to a client
node throwing Out Of Memory error which we have seen in ignite
client node
Could a "high" packet loss cause issues?
netstat -i
Kernel Interface table
Iface MTURX-OK RX-ERR RX-DRP RX-OVRTX-OK TX-ERR TX-DRP TX-OVR
Flg
ens3 1500 230327840 0 2878982 0 195927467 0 0
0 BMRU
lo 6553632487 0 0 0 32487 0
We have done two things so far...
1- Disabled client metrics on all clients.
2- We noticed we had some DNS issues, Ubuntu introduced new DNS system, so
we made some fixes here. If somehow nodes where trying to reconnect to each
other using DNS, then this may help the situation.
So far it seems
Hello!
I think the node has determined that it need to shut down:
[22:26:31,800][SEVERE][disco-event-worker-#42%xx%][FailureProcessor]
Ignite node is in invalid state due to a critical failure.
You can tune failure handler to avoid this behavior:
Hi, the GC logs where also provided and we determined, there was no GC
pressure. At least what I understood from the thread above. We also enabled
some extra thread info for GC and that showed nothing also.
On the client side I also see this: Possible too long JVM pause: 25468
milliseconds. My
Hello!
Well, this is the relevant part:
[22:26:21,816][WARNING][jvm-pause-detector-worker][IgniteKernal%xx]
*Possible
too long JVM pause: 55705 milliseconds*.
[22:26:21,827][WARNING][tcp-disco-client-message-worker-#10%xx%][TcpDiscoverySpi]
Client node considered as unreachable and will
Hi, in my original post I had included full logs of the stopped node...
And when you say set to localhost you which property do you mean is it the
one IgniteConfiguration setLocalhost()?
On Tue, 29 Oct 2019 at 03:48, Ilya Kasnacheev
wrote:
> Hello!
>
> I think it is possible (such as, node
Hello!
I think it is possible (such as, node tries to connect to wrong address /
itself) but more detailed analysis of logs is needed.
You can try specifying localHost property in IgniteConfiguration to make
sure correct address is used.
Regards,
--
Ilya Kasnacheev
пн, 28 окт. 2019 г. в
Ok cool. Thanks. I'm just trying to figure scenarios where it may happen.
When I connect with ignite visor and run the "top" topology command. The
client node is bound to multiple addresses about 12...
Ignite is running on plain regular VMs while the client node is running in
a containerized env
Hello!
Sure, if nodes can't reach each other, eventually they may segment and stop.
Regards,
--
Ilya Kasnacheev
пт, 25 окт. 2019 г. в 00:08, John Smith :
> Is it possible this is somehow causing the issue of the node stopping?
>
> On Thu, 24 Oct 2019 at 11:24, Ilya Kasnacheev
> wrote:
>
>>
Is it possible this is somehow causing the issue of the node stopping?
On Thu, 24 Oct 2019 at 11:24, Ilya Kasnacheev
wrote:
> Hello!
>
> This likely means that you have reachability problems in your cluster,
> such as, xxx.xxx.xxx.68 can connect to xxx.xxx.xxx.82 (on range
> 47100-47200) but
Hello!
This likely means that you have reachability problems in your cluster, such
as, xxx.xxx.xxx.68 can connect to xxx.xxx.xxx.82 (on range 47100-47200) but
not the other way around.
Regards,
--
Ilya Kasnacheev
пн, 21 окт. 2019 г. в 19:36, John Smith :
> I also see this printing every few
I also see this printing every few seconds on my client application...
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi Accepted
incoming communication connection [locAddr=/xxx.xxx.xxx.68:47101,
rmtAddr=/xxx.xxx.xxx.82:49816
On Mon, 21 Oct 2019 at 12:04, John Smith wrote:
> Hi,
Hi, thanks. I already made sure that each ignite VM runs on a separate host
within our cloud. I'm not doing any of that migration stuff.
Also recently disabled metrics igniteConfig.setMetricsLogFrequency(0); Just
to make sure it doesn't get too chatty. But i doubt this would affect it...
Should
The following documentation page has some useful points on deployment in a
virtualised environment: https://apacheignite.readme.io/docs/vmware-deployment
Denis
On 17 Oct 2019, 17:41 +0300, John Smith , wrote:
> Ok I have metribeat running on the VM hopefully I will see something...
>
> > On Thu,
Ok I have metribeat running on the VM hopefully I will see something...
On Thu, 17 Oct 2019 at 05:09, Denis Mekhanikov
wrote:
> There are no long pauses in the GC logs, so it must be the whole VM pause.
>
> Denis
> On 16 Oct 2019, 23:07 +0300, John Smith , wrote:
>
> Sorry here is the gc logs
There are no long pauses in the GC logs, so it must be the whole VM pause.
Denis
On 16 Oct 2019, 23:07 +0300, John Smith , wrote:
> Sorry here is the gc logs for all 3 machines:
> https://www.dropbox.com/s/chbbxigahd4v9di/gc-logs.zip?dl=0
>
> > On Wed, 16 Oct 2019 at 15:49, John Smith wrote:
>
Sorry here is the gc logs for all 3 machines:
https://www.dropbox.com/s/chbbxigahd4v9di/gc-logs.zip?dl=0
On Wed, 16 Oct 2019 at 15:49, John Smith wrote:
> Hi, so it happened again here is my latest gc.log stats:
>
Hi, so it happened again here is my latest gc.log stats:
https://gceasy.io/diamondgc-report.jsp?oTxnId_value=a215d573-d1cf-4d53-acf1-9001432bb28e
Everything seems ok to me. I also have Elasticsearch Metricbeat running,
the CPU usage looked normal at the time.
On Thu, 10 Oct 2019 at 13:05, Denis
Unfortunately, I don’t.
You can ask the VM vendor or the cloud provider (if you use any) for a proper
tooling or logs.
Make sure, that there is no such step in the VM’s lifecycle that makes it
freeze for a minute.
Also make sure that the physical CPU is not overutilized and no VMs that run on
Do you know of any good tools I can use to check the VM?
On Thu, 10 Oct 2019 at 11:38, Denis Mekhanikov
wrote:
> > Hi Dennis, so are you saying I should enable GC logs + the safe point
> logs as well?
>
> Having safepoint statistics in your GC logs may be useful, so I recommend
> enabling them
> Hi Dennis, so are you saying I should enable GC logs + the safe point logs as
> well?
Having safepoint statistics in your GC logs may be useful, so I recommend
enabling them for troubleshooting purposes.
Check the lifecycle of your virtual machines. There is a high chance that the
whole
You are correct, it is running in a VM.
On Thu, 10 Oct 2019 at 10:11, Denis Mekhanikov
wrote:
> Hi!
>
> There are the following messages in the logs:
>
> [22:26:21,816][WARNING][jvm-pause-detector-worker][IgniteKernal%xx]
> Possible too long JVM pause: *55705 milliseconds*.
> ...
>
Hi Dennis, so are you saying I should enable GC logs + the safe point logs
as well?
On Thu, 10 Oct 2019 at 11:22, John Smith wrote:
> You are correct, it is running in a VM.
>
> On Thu, 10 Oct 2019 at 10:11, Denis Mekhanikov
> wrote:
>
>> Hi!
>>
>> There are the following messages in the logs:
Hi!
There are the following messages in the logs:
[22:26:21,816][WARNING][jvm-pause-detector-worker][IgniteKernal%xx]
Possible too long JVM pause: 55705 milliseconds.
...
[22:26:21,847][SEVERE][ttl-cleanup-worker-#48%xx%][G] Blocked
system-critical thread has been detected. This can
So the error sais to set clientFailureDetectionTimeout=3
1- Do I put a higher value than 3?
2- Do I do it on the client or the server nodes or all nodes?
3- Also if a client is misbehaving why shutoff the server node?
On Thu, 3 Oct 2019 at 21:02, John Smith wrote:
> But if it's the
But if it's the client node that's failing why is the server node stopping?
I'm pretty sure we do verry simple put and get operations. All the client
nodes are started as client=true
On Thu., Oct. 3, 2019, 4:18 p.m. Denis Magda, wrote:
> Hi John,
>
> I don't see any GC pressure or STW pauses
Hi John,
I don't see any GC pressure or STW pauses either. If not GC then it might
have been caused by a network glitch or some long-running operation started
by the app. These logs statement
[22:26:21,827][WARNING][tcp-disco-client-message-worker-#10%xx%][TcpDiscoverySpi]
Client node
So I have been monitoring my node and the same one seems to stop once in a
while.
https://www.dropbox.com/s/7n5qfsl5uyi1obt/ignite-logs.zip?dl=0
I have attached the GC logs and the ignite logs. From what I see from
gc.logs I don't see big pauses. I could be wrong.
The machine is 16GB and I have
For GC logs I think I can just change the ignite.sh script in
/usr/share/apache-ignite/bin ???
Like right now I have a "heavy" load running and everything seems to be
fine...
Ok I'll check if it happens again
On Wed, 28 Aug 2019 at 11:21, Ilya Kasnacheev
wrote:
> Hello!
>
> Well, my
Hello!
Well, my recommendation is to find a way to enable GC logs and collect
regular logs as well, from all nodes.
Regards,
--
Ilya Kasnacheev
ср, 28 авг. 2019 г. в 18:18, John Smith :
> The drop box link here:
> https://www.dropbox.com/s/etm61xeb9mghs9m/ignite-details.log?dl=0
>
> Didn't
The drop box link here:
https://www.dropbox.com/s/etm61xeb9mghs9m/ignite-details.log?dl=0
Didn't take any logs just some visor printout and some Linux command
printouts and cat some info/stats from the logs just to be sure I wasn't
reading the wrong values.
Everything else as far as am aware is
Hello!
Don't see any logs here.
This setting may be problematic because you consume 14 out of 16GB by a
single Ignite process so system may decide to swap something out. I
recommend decreasing heap to 2G if possible. Should also make GC faster.
I'm not sure how to enable GC logs when running a
Hi I have attached some details here:
https://www.dropbox.com/s/etm61xeb9mghs9m/ignite-details.log?dl=0
How do I enable GC logs? I'm running the Debian package.
In summary:
1- Only ignite is running on the host
2- Ignite is configured to use 4GB heap
3- Host has 16GB total
4- 10GB off-heap
Hello!
It's hard to say what happens here. Do you have GC log? Please make sure to
collect it.
Is there anything running in the same JVM with Ignite?
Regards,
--
Ilya Kasnacheev
ср, 28 авг. 2019 г. в 15:13, John Smith :
> I'm not doing anything fancy with the cache I have 3 million records
I'm not doing anything fancy with the cache I have 3 million records
partitioned cache over 3 servers. And all I do is some put and gets. Unless
I have a bad config?
On Wed., Aug. 28, 2019, 6:32 a.m. Ilya Kasnacheev, <
ilya.kasnach...@gmail.com> wrote:
> Hello!
>
> Well, it's pretty descriptive.
Hello!
Well, it's pretty descriptive. Node was dropped from topology because of
long GC pauses.
Either find ways to decrease GC pauses, or increase failureDetectionTimeout.
Regards,
--
Ilya Kasnacheev
ср, 28 авг. 2019 г. в 00:18, John Smith :
> Hi, running 2.7.0
>
> I noticed one of my
Hi, running 2.7.0
I noticed one of my nodes was down. It seems to have turned itself off,
because of: Ignite node is in invalid state due to a critical failure.
I attached logs here:
https://www.dropbox.com/s/82li1020a5ig4ty/ignite-failled.log?dl=0
Venkat,
Please don't paste your logs into the body of your message, it makes them
look unreadable.
Use file attachment or provide a link instead.
Node on host 1 got segmented and killed according to the configured
segmentation policy.
It may happen due to a network problem or long GC pause.
Please could you provide some pointers, so that I can look deeper into the
issue.
1) Does PageMemory cause any issues
as per the metrics, one node is contains only 70 and other node is
PageMemory [pages=390178]
2) Why whole cluster is going down, when one node is down
3) Can I restart the
Ignite related only that log,
After that only cache stopped exception in the log.
Even we have enabled -DIGNITE_QUIET=false, but it is printing that much
only.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hi, kvenkatramtreddy!
Could you please provide a full log, not only stack trace with an exception?
On Sat, Jun 30, 2018 at 3:24 PM kvenkatramtreddy
wrote:
> Hi,
>
> Node stopped automatically after 48 hours uptime. Please find the logs
> below.
>
> I got 2 hosts and 3 Ignit
Hi,
Node stopped automatically after 48 hours uptime. Please find the logs
below.
I got 2 hosts and 3 Ignite nodes
Host 1, Node 1
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=5582ed60, uptime=48:00:04.981]
^-- H/N/C [hosts=2, nodes=3, CPUs=4
tl=64 time=0.134 ms 14:15:56
> 64 bytes from 172.21.0.181: icmp_seq=51353 ttl=64 time=0.133 ms 14:15:57
> 64 bytes from 172.21.0.181: icmp_seq=51354 ttl=64 time=0.177 ms 14:15:58
> 64 bytes from 172.21.0.181: icmp_seq=51355 ttl=64 time=0.127 ms 14:15:59
>
> 3:Faced with the problem，I plan
://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6962.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
ogs.bz2>
>
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6917.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6917.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
.nabble.com/Ignite-Cluster-node-stopped-tp6608p6893.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
t; intOrder=260, lastExchangeTime=1469676024755, loc=false,
> ver=1.6.0#20160518-sha1:0b22c45b, isClient=true]
> [22:03:38,159][WARNING][disco-event-worker-#144%null%][
> GridDiscoveryManager]
> Node FAILED: TcpDiscoveryNode [id=bf64226a-7597-4bc8-866b-6d99c7e9f2aa,
> addrs=[124.250.36.47, 127.0.0.1, 172.21.0.37, 33.33.33.1],
> sockAddrs=[/33.33.33.1:0, /124.250.36.47:0, /172.21.0.37:0, /127.0.0.1:0,
> /124.250.36.47:0, /172.21.0.37:0, /33.33.33.1:0], discPort=0, order=503,
> intOrder=264, lastExchangeTime=1469698198271, loc=false,
> ver=1.6.0#20160518-sha1:0b22c45b, isClient=true]
> [22:03:38,561][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> bjdqAppRecommendingCache
> [22:03:38,563][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> bjdqAppRecommendedCache
> [22:03:38,564][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> yiCheAppRecommendingCache
> [22:03:38,564][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> yiCheAppRecommendedCache
> [22:03:38,565][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> idfaCache
> [22:03:38,566][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> clickedArticleCache
> [22:03:38,566][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> deviceCache
> [22:03:38,567][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> ignite-marshaller-sys-cache
> [22:03:38,567][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> ignite-sys-cache
> [22:03:38,568][INFO][Thread-117][GridCacheProcessor] Stopped cache:
> ignite-atomics-sys-cache
> [22:03:38,571][INFO][Thread-117][GridDeploymentLocalStore] Removed
> undeployed class: GridDeployment [ts=1469624393224, depMode=SHARED,
> clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2,
> clsLdrId=a1d707c2651-424ef276-c1b6-48b0-9ded-c6fca0997502, userVer=0,
> loc=true,
> sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.
> preloader.GridDhtPartitionFullMap,
> pendingUndeploy=false, undeployed=true, usage=0]
> [22:03:38,585][INFO][Thread-117][IgniteKernal]
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6865.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
ndeploy=false, undeployed=true, usage=0]
[22:03:38,585][INFO][Thread-117][IgniteKernal]
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6865.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
, 2016 at 6:12 AM, suhuadong <suhuadong...@163.com> wrote:
> hi vkulichenko,
> Can you find out reason about the node stoped?
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6842.html
&g
hi vkulichenko,
Can you find out reason about the node stoped?
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6842.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Hi Kevin,
The node was segmented. This usually happens either due to networks
problems, or because of memory issues. Can you check the logs of other
nodes?
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/ignite-node-stopped-abnormally-tp6781p6819.html
Hi there,
When I am using ignite server, I notice that sometimes one node failed
abnormally, and please see the attachment for detail error message.
Best regards,
Kevin
test.log
Description: test.log
I attach gc log in ignite server node. gc-logs.bz2
<http://apache-ignite-users.70518.x6.nabble.com/file/n6758/gc-logs.bz2>
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6758.html
Sent from the Apache Ignite Users m
518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6757.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Got it. Please attach *full* log files from all the nodes.
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6729.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
/cluster-config
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6725.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Hi,
Generally, 60GB is too much. Can you try to give about 10GB of heap and
switch caches to off-heap mode [1]?
[1] https://apacheignite.readme.io/docs/off-heap-memory
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped
ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped-tp6608p6631.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
The first thing I would check is the memory consumption. Is it possible that
some the nodes run out of memory or sit in long GC pauses? Do you have
enough heap memory allocated?
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-node-stopped
69 matches
Mail list logo
