DC replication via Kafka Connect for Apache Ignite

We are trying to implement the replication of data between two Ignite data clusters. For this purpose, we are using Kafka Connect. We have followed the things mentioned in this document -> https://dzone.com/articles/linking-apache-ignite-and-apache-kafka-for-highly Everything is working fine

Re: A question regarding cluster groups

Thanks for the reply denis. A few follow up questions. 1. Is there any way I can move the data from one cluster node to other cluster node. Like exporting work folder or something like that periodically. 2. Is there any snapshot of data concept that can be done periodically and push it to

Re: A question regarding cluster groups

Can someone please reply to this thread. It's kind of urgency. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

A question regarding cluster groups

Let us say I've two server nodes in one data center DC1 and two more server nodes in another data center DC2. Two data centers have some network delay. Now I'm using SQL select statements on caches which are replicated. Now those caches' write synchronization mode is FULL_SYNC. Now at a time we

Re: Too much network latency issue

Can someone please reply to this urgently -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Too much network latency issue

We are having a cluster of 9 nodes. All connected fine in the same network. There are 5 clients and 4 server nodes. All the client nodes and 2 of 4 server nodes are in the same data center. The other two server nodes are in different date center (but in the same network). There can be some

Re: Setting expiry on the fly via Data Streamers

Thanks for the solution. I will check this. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Setting expiry on the fly via Data Streamers

I created cache in this way. CreatedExpiryPolicy cep = new CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 10)); CacheConfiguration C_CONFIG = new CacheConfiguration(); C_CONFIG.setName("temp"); IgniteCache cache = ignite.getOrCreateCache(C_CONFIG);

Re: Thread starvations

Sorry the above log file doesn't contain all info. Please check this file Logs.zip -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Thread starvations

Thanks for the reply. We've done query analysis and found that pretty much all queries are taking around 1-5ms execution time for our operations which we consider it to be fine. But we have one query which is adding of rows in a table. We get maximum 5000 records at a time and we use

Thread starvations

We are getting these warnings quite often ``` ./ignite-3fb86334.0.log:[09:48:02,663][WARNING][grid-timeout-worker-#71][IgniteKernal] Possible thread pool starvation detected (no task completed in last 3ms, is query thread pool size large enough?)

Re: Multi-threaded JDBC driver issue/concern

This is my SQL table schema ID (Long), URL (Varchar), SCORE (Double), APPNAME_ID (Long) We have composite index on Score, Appname_Id. Based on your answer I've two questions. 1. How can I insert SQL rows using JCache data streamer API (if possible, with example)? Currently I'm using jdbc

Re: Multi-threaded JDBC driver issue/concern

Hi stephen. Thanks so much for reply. This is my SQL table schema ID (Long), URL (Varchar), SCORE (Double), APPNAME_ID (Long) We have composite index on Score, Appname_Id. Based on your answer I've two questions. 1. How can I insert SQL rows using JCache data streamer API (if possible, with

Re: Multi-threaded JDBC driver issue/concern

Hi stephen. Thanks so much for reply. This is my SQL table schema ID (Long), URL (Varchar), SCORE (Double), APPNAME_ID (Long) We have composite index on Score, Appname_Id. Based on your answer I've two questions. 1. How can I insert SQL rows using JCache data streamer API (if possible, with

Multi-threaded JDBC driver issue/concern

We have an SQL table which we need because for normal JCache K-V we cannot sort on some column's data. We need that sort feature. That's why we chose SQL table representation. Our application is heavily multi-threaded. Now when trying to insert rows in that table, each thread simultaneously

Re: Data Consistency Question

Can you please suggest an efficient solution? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Question regarding topology

Can someone please help with this case? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Question regarding topology

Also, In SQL query, we are having a semaphore lock where all the threads stay there and only one thread can execute the code. It's because in the code, we have jdbc connection query. And it's not thread safe. Sometimes we even get thread hanging here if we keep increasing client worker threads.

Re: Question regarding topology

Thanks for reply. Our clients are connected in both ways one is JCache way and other is SQL way each for various caches. -> For get/put we use normal getAll() and normal putAll(). The thing is here all the threads will have their own data and the data is not shared. So no need to worry about

Question regarding topology

If I have a cache named "XYZ" with replication mode ON then what happens for the queries if the topology configuration is as below. 1) Two app servers (clients) which make total 100 connections (50 each) in parallel as it's multi-threaded application. Both are 64 core machines. 2) Two DB servers

Re: Suggest a better way to access a particular K-V store

Can someone please help regarding this issue? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Two specific warnings everytime

Can someone please help regarding this issue? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data Consistency Question

Can someone please help regarding this issue? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Semaphore Lock Not Acquired. Race issue?

When I removed the stopped Server node IP from Client IP list and server IP list and start both clients, it is working fine. Although I couldn't reproduce it again. If I found it again, I'll try taking dumps and reply it in this thread. Java version -> jdk 1.8.0_152 Ignite version -> 8.7.10

Semaphore Lock Not Acquired. Race issue?

Server 1 -> Let's say IP is X. Status is running. Server 2 -> Let's say IP is Y. Status is not running. We stopped this node for various reasons. Client1 -> TcpDiscoveryVmIpFinder ipAddress list has two server IPs - X,Y Client 2 -> TcpDiscoveryVmIpFinder ipAddress list has two server IPs -

Re: Suggest a better way to access a particular K-V store

Thanks for the reply. For your reference I'm sharing the code.. Please have a look at it.. ``` public Set execQuery(Object... args){ Set urls = new LinkedHashSet<>(); Long start = (Long)objects[2]; Long end = (Long)objects[3]; int i = 0;

Re: Data Consistency Question

Thanks for the reply. But I've two doubts here. 1. If you check the code, I used ignite semaphore and if i remove that sempahore checking, then I'm getting JDBC connection issues as multiple threads access that particular code. We run 100s of threads. Is there any way where we can run in

Re: Two specific warnings everytime

Thanks for the reply. But I'm not using those. How can I disable those? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Data Consistency Question

Can anyone tell if this code guarantees all the rows to be inserted right after all the statements are executed? IgniteSemaphore semaphore = null; try { semaphore = cacheHolder.getJDBCSemaphore(IgniteLocks.JDBC_LOCK.getLockValue());

Re: Two specific warnings everytime

I'm sorry I don't have any clue on "continous" queries. What are they? And I don't use it as per my knowledge. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Suggest a better way to access a particular K-V store

My query is SELECT * FROM URLS WHERE APPNAME_ID = ? ORDER BY SCORE LIMIT ? That is giving me 0.7-1.0 msec read performance for a test run for about some time. But the index with URLS (SCORE ASC, APPNAME_ID), I'm getting a read performance of about 0.3 to 0.5 msec. i found it to be constant when

Two specific warnings everytime

I always get these two warnings. Persistence is on. Warning 1: ``` [WARNING][checkpoint-runner-#569][GridCacheDatabaseSharedManager] 3 checkpoint pages were not written yet due to unsuccessful page write lock acquisition and will be retried

Calculation of Size

I have persistence enabled. How much size do I need for Off-Heap if I wish not to have "too many dirty pages" checkpointing? Our application has many threads which simultaneously write/read data. With total 1L records in all caches I'm seeing 770MB of size occupied in Off-heap while running for 1

Re: How many Caches/Tables we can create/query in parallel?

I thought it's obvious that, SQL table return in O(log n) with n being number of elements in BTree. Our task being write intensive, If all the threads are inserting data to same tree then rebalancing and accessing time would be affected right? So for this sake, if we divide each thread's SQL data

How many Caches/Tables we can create/query in parallel?

We created a task which crawls website. We have an SQL table, where the application writes data while crawling website. But the application has several threads running in parallel, each thread working on a certain website. The amount of data that is being generated on that SQL table is huge and we

Re: Suggest a better way to access a particular K-V store

We've already tried that but we are receiving these errors. We are using index in a correct way only We are using Ignite persistence. ``` [15:32:19,481][WARNING][long-qry-#110][LongRunningQueryManager] Query execution is too long [duration=3898ms, type=MAP, distributedJoin=false,

Re: Apache Ignite Persistence Issues/Warnings

Hi Alex, Are you saying if we disable WAL Archiving, will there be any problem? Let's say if I disable WAL archiving and if 6 out of 10 WAL segments are filled then what happens during OS crash? When we restart the server, will all the operations in those 6 segments be updated on the disc data

Suggest a better way to access a particular K-V store

The K is key with type String. The V is the value with type JSONObject/HashMap. V is having the following structure: {"agentId":,"score":,"url":} Now the query on this will be in such a way that -> Get top 50 values of "K" where values in that K-V store are sorted based on "score". Provided

Re: Apache Ignite Persistence Issues/Warnings

Hi Alex, thanks for the reply. But as our application is write/update intensive, we want a good performance/high availability application solution along with disk persistence. In that case, what configuration would you suggest for the following case, 1. Disk storing functionality can run

Apache Ignite Persistence Issues/Warnings

We are running a write-intensive load test on Apache Ignite. We are also doing reads. We are getting the following warning messages from sys-stripe threads quite often after a few hours of testing. DB Config -> RAM - 8GB, CPU cores - 64, Persistence - ON, Heap - 2GB, Durable memory Off-heap -

Re: Serialisation issue.

Team, is there anyone who could help on this? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Serialisation issue.

cache = ignite.getOrCreateCache(new CacheConfiguration<>() .setName("DOCIDS") .setStoreKeepBinary(true) .setQueryEntities(Collections.unmodifiableList(new ArrayList(){{ QueryEntity entity = new QueryEntity();

Re: Getting No space left on device exception when persistence is enabled

The whole machine is getting hung up, when I try to start the node. Every app I open is saying not enough memory. But when I restart and see I could see still 115 GB of space free. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Getting No space left on device exception when persistence is enabled

We don't have any user limitations. And currently out of 300GB, we have 115 GB free space left. What's with having 100 GB free space? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Getting No space left on device exception when persistence is enabled

Can someone please help? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Getting No space left on device exception when persistence is enabled

Hi, We are using Apache Ignite's persistence enabled mode. The following is the only configuration we made. An

Re: Doubt regarding CACHE Replication.

Thanks.. I've added to baseline and it started replicating.. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Doubt regarding CACHE Replication.

I am using Ignite with persistence mode enabled. I created a cache and let it grow in one server node till the data is upto 2 GB. After some time, if I start a new server node and since the cache is in replicated mode, why the data is not replicated in the new server node? Any idea? It's showing 0