Mideen Abdul Gaffoor created IGNITE-16497:
---------------------------------------------
Summary: Data streaming in Apache-Ignite through Apache-Kafka
consumes high CPU
Key: IGNITE-16497
URL: https://issues.apache.org/jira/browse/IGNITE-16497
Project: Ignite
Issue Type: Bug
Reporter: Mideen Abdul Gaffoor
Attachments: main idc cpu consumption issue.png
I am using Ignite Source Connector (Single Kafka connector node) to export the
Ignite Events from my ignite cluster(2 Nodes) to Kafka broker nodes (2 Nodes)
with a single topic and 29 partitions.
I have processed 0.1 Million events (PUT, DELETE) per minute with the 1k size
messages(in Avg).
My connector node consumes 90% of the CPU (you can see it below).
!main idc cpu consumption issue.png!
My Connector node machine config.
RAM --> 30 GB
ROM --> 1TB
Configured Heap --> 15GB
No of CPU Core --> 40 (20 x 2)
# connector
name=my-ignite-source-connector
connector.class=class of my source connector
tasks.max=1
topicNames=ignite-data
# cache
cacheName=DOCIDS
cacheAllowOverwrite=true
cacheEvts=put , removed
evtBatchSize=100
numberOfPartitions=29
igniteCfg=/myconfig/ignite-config.xml
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:util="http://www.springframework.org/schema/util";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd";>
<bean abstract="true" id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples,
default is false. -->
<property name="communicationSpi"><bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"><property
name="socketWriteTimeout" value="60000"/></bean></property><property
name="peerClassLoadingEnabled" value="true"/><property
name="clientFailureDetectionTimeout" value="10000"/><property
name="dataStorageConfiguration"><bean
class="org.apache.ignite.configuration.DataStorageConfiguration"><property
name="walPath" value="wal_data/"/><property name="walArchivePath"
value="wal_data/"/><property name="defaultDataRegionConfiguration"><bean
class="org.apache.ignite.configuration.DataRegionConfiguration"><property
name="persistenceEnabled" value="true"/><property name="maxSize" value="#\{15L
* 1024 * 1024 * 1024}"/></bean></property></bean></property>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<!--Task execution events-->
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/>
<!--Cache events-->
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes.
-->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery
that can be used
instead os static IP based discovery. For information
on all options refer
to our documentation:
http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based
discovery of initial nodes. -->
<!--<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with
actual host IP address. -->
<value>ignite-server-node1:47500..47509</value><value>ignite-server-node2:47500..47509</value><value>kafka-conect-node:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
The data transfer process is working as intended only for a few hours because
of this high CPU consumption, after that the Kafka connector node will be
OFFLINE in ignite server topology.
Any pointers will help.
Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
