Hello,
I have a medium cluster set up for testings - 3 x r4.8xlarge EC2 nodes. It
has persistence enabled, and zero backup.
- Full configs are attached.
- JVM settings are: JVM_OPTS="-Xms16g -Xmx64g -server -XX:+AggressiveOpts
-XX:MaxMetaspaceSize=256m -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC"
The table has 145M rows, and takes up about 180G of memory
I testing 2 things
1) Writing SQL tables from Spark
2) Performing large SQL queries (from the web console): for example Select
COUNT (*) FROM (SELECT customer_id FROM MyTable where dt > '2018-05-12'
GROUP BY customer_id having SUM(column1) > 2 AND MAX(column2) < 1)
Most of the times I run the query it fails after one of the nodes crashes
(it has finished a few times, and then crashed the next time). I have also
similar stability issues when writing from Spark - at some point, one of
the nodes crashes. All I can see in the logs is
[21:51:58,548][SEVERE][disco-event-worker-#101%Server%][] Critical system
error detected. Will be handled accordingly to configured handler
[hnd=class o.a.i.failure.StopNodeFailureHandler, failureCtx=FailureContext
[type=SEGMENTATION, err=null]]
[21:51:58,549][SEVERE][disco-event-worker-#101%Server%][FailureProcessor]
Ignite node is in invalid state due to a critical failure.
[21:51:58,549][SEVERE][node-stopper][] Stopping local node on Ignite
failure: [failureCtx=FailureContext [type=SEGMENTATION, err=null]]
[21:52:03] Ignite node stopped OK [name=Server, uptime=00:07:06.780]
My questions are:
1) What is causing the issue?
2) How can I debug it better?
The rate of crashes and our lack of ability to debug them is becoming quite
a concern.
Cheers,
Eugene
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd";>
<!--
Alter configuration below as needed.
-->
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="Server"/>
<property name="gridLogger">
<bean class="org.apache.ignite.logger.log4j.Log4JLogger">
<constructor-arg type="java.lang.String" value="ignite-log4j.xml"/>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.zk.ZookeeperDiscoverySpi">
<property name="zkConnectionString" value="172.21.85.10:2181,172.21.85.11:2181,172.21.86.10:2181"/>
<property name="sessionTimeout" value="30000"/>
<property name="zkRootPath" value="/apacheIgnite"/>
<property name="joinTimeout" value="10000"/>
</bean>
</property>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="walPath" value="/ignite_data/wal"/>
<property name="walArchivePath" value="/ignite_data/wal/archive"/>
<property name="storagePath" value="/ignite_data/persistence"/>
<property name="pageSize" value="#{4 * 1024}"/>
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="initialSize" value="#{32L * 1024 * 1024 * 1024}"/>
<property name="maxSize" value="#{130L * 1024 * 1024 * 1024}"/>
<property name="metricsEnabled" value="true"/>
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean parent="cache-template">
<property name="name" value="ga_template*"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="copyOnRead" value="true"/>
<property name="queryDetailMetricsSize" value="10"/>
<property name="queryParallelism" value="32"/>
<property name="statisticsEnabled" value="true"/>
<property name="managementEnabled" value="true"/>
</bean>
</list>
</property>
<property name="transactionConfiguration">
<bean class="org.apache.ignite.configuration.TransactionConfiguration">
<property name="TxTimeoutOnPartitionMapExchange" value="20000"/>
</bean>
</property>
</bean>
<bean id="cache-template" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="rebalanceMode" value="SYNC"/>
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="backups" value="0"/>
<property name="eagerTtl" value="true"/>
</bean>
</beans>
