Re: Ignite with Spring Cache on K8S, eviction problem

[lukaszbyjos]

I have something like in this gist. I cant share everything but in get there
is just request to DB. 
Evict is the same and I need to have this in the separate method because
sometimes I need to evict for list of ids. I know that I can't call this
method directly from the same class because of how Spring AOP works so
that's not a problem. Also, this method works for another cache where I have
only one key. 
I was looking at debug log level for Ignite and it was printed correctly
like 
[part=0, val=[PCdXo1, 0], hasValBytes=true]

https://gist.github.com/Mistic92/5043b682aa5742a7d7fad68c7ad8722c



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Enable Kerberos on Ignite Kafka streamer / support for new consumer config

[jackson.dickson]

Thanks Andrew.

The new consumer config enable Kafka clients to connect directly to the
Kafka brokers. kafka-clients 0.10.x support both style. I believe you would
need another ConsumerConfig type like ZKConfig.

-Jackson



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Unable to identify root cause for node failure

[Rahul Pandey]

Hi,
Yes, those logs are full.

Also found that root cause was O.S killing the process, the two nodes were
given 48gb each heap size, on reducing it to 32gb O.S did not kill the
process, but getting OutOfMemoryError while data loading.

Since Ignite Persistance is turned on doesn't data should be spill over to
disk when heap is full?

Thanks,
Rahul Pandey

On Mon, Feb 19, 2018 at 8:52 PM, Mikhail 
wrote:

> Hi Rahul,
>
> at 12:25:03 first node detected that second one is failed:
>
> [12:45:03,251][INFO][exchange-worker-#182][GridCachePartitionExchangeMana
> ger]
> Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
> [topVer=3, minorTopVer=0], evt=NODE_FAILED,
> node=bbdd7792-1309-4941-a7ad-05ed642f62b5]
>
> however, logs for ignite-bbdd7792 is ended at 12:44:14,167
>
> so, something interesting should be in logs after that time, are you sure
> that it's full log?
>
> Thanks,
> Mike.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

-- 


--

The content of this e-mail is confidential and intended solely for the use 
of the addressee(s). The text of this email (including any attachments) may 
contain information, belonging to Pragmatix Services Private Limited, 
and/or its associates/ group companies/ subsidiaries (the Company), which 
is proprietary and/or confidential or legally privileged in nature or 
exempt from disclosure under applicable law. If you are not the addressee, 
or the person responsible for delivering it to the addressee, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it is prohibited and may be unlawful. If you have 
received this e-mail in error, please notify the sender and remove this 
communication entirely from your system. The recipient acknowledges that no 
guarantee or any warranty is given as to completeness and accuracy of the 
content of the email. The recipient further acknowledges that the views 
contained in the email message are those of the sender and may not 
necessarily reflect those of the Company. Before opening and accessing the 
attachment please check and scan for virus. Thank you.

WARNING: Computer viruses can be transmitted via email. The recipient 
should check this email and any attachments for the presence of viruses. 
The sender or the Company accepts no liability for any damage caused by any 
virus transmitted by this email or errors or omissions.

Node failed to join cluster with error related to : Memory configuration mismatch (fix configuration or set -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true system property) [rmtNodeId=47e76683-39ec

[aMark]

Hi,

Before I go into error, here is our production configuration : 
We are using Ignite as persistent store, in Partitioned Mode having 6 data
node, each running on different server.  Atomicity mode is ATOMIC, and
Rebalance mode is ASYNC while CacheWriteSynchronizationMode is FULL_SYNC. 

We got the following errors when we upgraded our production environment from
Ignite 2.1 to Ignite 2.3.

Out of six nodes , we got page corrupt error only on one of the node (Though
all the nodes shares same set of ) , below is the stack trace for the error: 

java.lang.IllegalStateException: Failed to get page IO instance (page
content is corrupted)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:219)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.FreeListImpl.(FreeListImpl.java:358)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore$1.(GridCacheOffheapManager.java:923)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:915)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.initialUpdateCounter(GridCacheOffheapManager.java:1202)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.onPartitionInitialCounterUpdated(GridCacheOffheapManager.java:471)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restorePartitionState(GridCacheDatabaseSharedManager.java:1700)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1657)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1072)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:863)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1019)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:651)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2279)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)


Thinking that page is corrupt, we cleaned various Ignite directories
IgnitePersistentStore/WalStore/WalArchive/Work from the same machine.  After
cleanup, when we started the node, we started getting different error : 
org.apache.ignite.internal.IgniteKernal%eb4de377-2601-47c6-9b13-fd6ee02cb0fe
- Got exception while starting (will rollback startup routine). class
org.apache.ignite.IgniteCheckedException: Memory configuration mismatch (fix
configuration or set -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
system property) [rmtNodeId=47e76683-39ec-4d22-b9c8-eadd6dad7fdc,
locPageSize = 4096, rmtPageSize = 2048]
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.checkMemoryConfiguration(GridCacheProcessor.java:3088)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.checkConsistency(GridCacheProcessor.java:825)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:763)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1060)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1909)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1652)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1080)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:998)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:783)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:653)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:622)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
com.msci.datagrid.AlcyoneDataGridServer.main(AlcyoneDataGridServer.java:63)


We know that page size has changed from 2KB to 4 KB in Ignite 2.3, but as
there was no way to migrate existing persisted data and we didnot 

Re: Large durable caches

[Ivan Rakov]

Hello,

Seems like checkpoint-runner thread is idle. There's no active 
checkpoint at the moment of dump, and thus there's no work for 
checkpoint thread.
If the problem persists, please share full thread dump. It would be 
better to take 2-3 consecutive thread dumps when your data streamer stops.


Best Regards,
Ivan Rakov

On 19.02.2018 0:41, lawrencefinn wrote:

I am testing a very large durable cache that mostly cannot fit into memory.
I start loading in a lot of data via a data streamer.  At some point the
data becomes too large to fit into memory so ignite starts writing a lot to
disk during checkpoints.  But after some point after that, the datastreamer
stops streaming.  It's stuck.  Doing a jstack on ignite i see the
datastreamer threads are all stuck in a parked state.  Any thoughts?

"data-streamer-stripe-4-#37" #62 prio=5 os_prio=0 tid=0x7fe8c212d000
nid=0x1410 waiting on condition [0x7fe73d1d6000]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
 at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
 at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
 at java.lang.Thread.run(Thread.java:745)

The checkpoint thread seems to be parked too
"checkpoint-runner-#354" #397 prio=5 os_prio=0 tid=0x7fe49c04f000
nid=0x177c waiting on condition [0x7fdc54a1c000]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x0002c08a6ef8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
 - None


My configuration is as follows:

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
   
 
 
 
 
 
 
   






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Snapshot backup of Ignite native persistance data

[arunkjn]

Hi,

I have an ignite cluster which has some durable caches. These caches are
using amazon EBS volumes. I want to backup each EBS node daily for disaster
recovery since we cannot afford losing the data and want disaster recovery
capability.

I was looking around to see if ignite offers a native way of taking reliable
snapshots. I could see its support in the grid gain
distribution(https://www.gridgain.com/products/software/ultimate-edition/cluster-snapshots).
Is there anything available in ignite to do the same?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Stale class on ignite data node

[Alexey Kukushkin]

Right now the only way to avoid re-deploying entity classes when they
change is using Ignite Binary Object API

to work with entities.

Ignite with mariadb

[Sumanth Sumanth]

Hi,

Can Ignite be used with Mariadb?
When attempted (via jdbc url:
jdbc:mariadb://localhost:3306/db1?user=root=password) Ignite
throws exception that it is not able to find suitable jdbc driver for the
Mariadb.
What is the recommended way of integrating Mariadb with Ignite?

Thanks,
Sumanth

Re: Ignite with mariadb

[Alexey Kukushkin]

Do you want to use mariadb as Ignite's cache store (3-rd party
persistence)? You can use whatever you want if you implement your custom
cache store. You might also try using out-of-the-box CacheJdbcPojoStore. It
already supports DB2, Oracle, SQK Server, H2 and MySQL. It looks like
mariadb should be detected as MySQL - see getDatabaseProductName()

.

As for your "not able to find suitable jdbc driver for the Mariadb"
exception - Ignite does not include any 3rd party drivers. You must
download mariadb driver and put it on Ignite's classpath.

Stale class on ignite data node

[arunkjn]

Hi Guys,

I have an application in which I segregate ignite nodes into data and
service nodes.

The data node is responsible for hosting cache and queues. The service nodes
host an ignite service which uses cache defined in data nodes. These nodes
are separated because during my application development lifecycle I want to
redeploy service nodes containing new code with each release but I want to
keep the data nodes the same.

I am experiencing an issue where the data node has a stale version of an
entity class in its classpath and is thus not able to execute a service
method. I have to redeploy the data node with updated class for it to work.
What are the suggested ways of getting around this problem? Should I deploy
data nodes when an entity structure changes? What is the downside to doing
this?  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Rebalancing in Ignite (Cache loader are executing instead of just rebalancing)

[Prasad Bhalerao]

Hi Alex,

How do you load the multiple caches in prod env? Do you always call the
loadCache method from client node. What if I do not have client server mode
configuration? I just have server nodes. How does one load the cache in
this scenario? Is it mandatory to use client-server node configuration?
How does one know that the cluster is formed and now its time to call
loadCache method?

I had used hazelcast in past, in case of hazel cast I just used to invoke
the loader using map.get method at server startup and the node which starts
first used to invoke the data loaders and all other nodes used to just
accept the data as a part of rebalancing (other nodes do not execute data
loader code).

Do we have some thing like this in ignite?


Thanks,
Prasad


On Mon, Feb 19, 2018 at 3:41 PM, Alexey Kukushkin  wrote:

> I am not sure if it helps - please note that:
>
>- loadCache() shall be called once. Call it on a client node and not
>on the server nodes. What loadCache() does it just broadcasts request to
>invoke CacheStore's loadCache() on every server node.
>- loadCache() is optional - you use it if you want to "warm up" the
>data. You still can start without calling loadCache() having no pre-loaded
>data.
>- First start the cluster and wait your final topology. Then connect a
>client utility that would call loadCache().
>
>

Re: Enable Kerberos on Ignite Kafka streamer / support for new consumer config

[jackson.dickson]

Hi Andrew,

Yes, "bootstrap.server" option is not working for me.

-Jackson



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Rebalancing in Ignite (Cache loader are executing instead of just rebalancing)

[Alexey Kukushkin]

Prasad, I am not aware about anything like what you described in Ignite
(may be Community will help). To me Ignite servers work as a cluster and
having a server doing something that other servers do not breaks the cluster
definition . I would really
consider developing a simple one-line-of-code "warm-ignite-data" utility
that would start as Ignite client node, call loadCache and exit. Please
note that Ignite client nodes do not necessarily have to be always part of
Ignite topology. Ignite clients are apps and utilities to execute business
(or utility) tasks on the Ignite cluster and leave the cluster when the job
is done.

Re: Ignite with Spring Cache on K8S, eviction problem

[lukaszbyjos]

I still have a problem with evicting one cache. What's recommended setting
for partitioned cache?
>From service B I want to evict "ch-current" cache but service A still see
old values. 
At the same time just after evicting this cache the other one is evicted and
looks like it's working. 
Cache "ch-current" keys are created like this 
"@CacheEvict(value = "ch-current", key = "{#uid.concat('-').concat(0)}"),"
"@Cacheable(cacheNames = "ch-current", key =
"{#userId.name().concat('-').concat(#page)}")"
I changed key creation after your suggestions. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: And again... Failed to get page IO instance (page content is corrupted)

[Mikhail]

Hi Sergey,

The release of 2.4 should be soon, in a week or couple, however, there's no
strong schedule for Apache releases.

Could you please share a reproducer for the issue? Might be you can share a
storage on which the issue can be reproduced?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: QueryEntities in Ingnite

[Mikhail]

Hi,

there's two way to index objects in cache:
1. by annotations:
https://apacheignite.readme.io/docs/cache-queries#section-query-configuration-by-annotations
2. by QueryEntity:
https://apacheignite.readme.io/docs/cache-queries#section-query-configuration-using-queryentity

But you can't mix both, please define fields in QueryEntity or set indexed
objects by: CacheConfiguration.setIndexedTypes(MyKey.class, MyValue.class) 

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Enable Kerberos on Ignite Kafka streamer / support for new consumer config

[jackson.dickson]

Hi Andrew,

Yes, "bootstrap.server" option is not working for you?

-Jackson



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Snapshot backup of Ignite native persistance data

[arunkjn]

Hi Ivan,

Thanks for you reply.

I have some follow up questions.
1. What is the behaviour of cluster when it is in
ignite.cluster.active(false) state? Does it throw error when a cache seek
operation is performed? Or does it wait for cluster to become active again? 
2. How can I reliably know if cluster has become inactive before taking
backup? Is cluster.active(false) a synchronous operation?
3. What do you mean by "consistent ID of your nodes"? Can you please
elaborate?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Enable Kerberos on Ignite Kafka streamer / support for new consumer config

[Andrey Mashenkov]

Hi Jackson,

Got it. I've created a ticket for this [1].
I'm not too close familiar with Kafka. Please, let me know if I missed smth.

Seems, the fix is trivial, we've just create a KafkaStreamer directly
instead of using zookeeper kafka connector.

[1] https://issues.apache.org/jira/browse/IGNITE-7752

On Mon, Feb 19, 2018 at 2:04 PM, jackson.dickson 
wrote:

> Hi Andrew,
>
> Yes, "bootstrap.server" option is not working for me.
>
> -Jackson
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Best regards,
Andrey V. Mashenkov

Re: Unable to identify root cause for node failure

[Mikhail]

Hi Rahul,

at 12:25:03 first node detected that second one is failed:

[12:45:03,251][INFO][exchange-worker-#182][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=3, minorTopVer=0], evt=NODE_FAILED,
node=bbdd7792-1309-4941-a7ad-05ed642f62b5]

however, logs for ignite-bbdd7792 is ended at 12:44:14,167

so, something interesting should be in logs after that time, are you sure
that it's full log?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

QueryEntities in Ingnite

[Prasad Bhalerao]

Hi,

As per this link (
https://stackoverflow.com/questions/36773011/what-is-the-difference-between-a-primary-key-and-a-surrogate-key),
I have created the QueryEntities as follows. I have not set the fields and
indexes explicitly as it is already define in my value Pojo class.

private static QueryEntity createIPV4RangeDataEntity() {
QueryEntity ipV4RangeDataEntity = new QueryEntity();
ipV4RangeDataEntity.setValueType(IPV4RangeData.class.getName());
ipV4RangeDataEntity.setKeyType(IPRangeDataKey.class.getName());
return ipV4RangeDataEntity;
}

I am using following code to get the number of records for a given groupId.

try (QueryCursor cursor = ipV4RangeDataCache.query(new
SqlFieldsQuery("select count(1) from IPV4RANGEDATA where assetGroupId
= 10")) )  {
for (List row : cursor)
System.out.println("count=" + row.get(0));
}

But when execute above code, I get following exception. Do I have to
redefine all the fields,qury fields and indexes in QueryEntity for each
cache? Can some one please advise?

 Caused by: org.h2.jdbc.JdbcSQLException: Column "GROUPID" not found; SQL
statement:
select count(1) from IPV4RANGEDATA where groupId = 10 [42122-196]


Pojo:

public class IPV4RangeData implements Serializable {

private long id;

@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ipv4_range_idx", order = 0)})
private long groupId;

private int assetType;

@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ipv4_range_idx", order = 1)})
private long startIp;

@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ipv4_range_idx", order = 1)})
private long endIp;






Thanks,
Prasad

Re: Discovery on Google Cloud

[mcherkasov]

Hi,

Could you please show us ignites logs?

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Only one node has a high number of SystemExecutor CompletedTask

[dark]

Thanks to your kind reply.

We use Ignite to Group By Query about time series data. However, I have seen
a lot of data, so when I create TimeWindow through Expiry Strategy in one
cache, I see that it uses too many resources to expire entries. 

As a result, I have adopted a method of simply destroying the cache and
erasing all entries in that time window. Therefore, creating and destroying
multiple Cache seems inevitable. :(

I first noticed that the oldest node acts as a coordinator. 
Thank you for your valuable information. :)




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Snapshot backup of Ignite native persistance data

[Ivan Rakov]

Arun,

1. First option - you'll get something like this

Method threw 'org.apache.ignite.IgniteException' exception: Can not 
perform the operation because the cluster is inactive. Note, that the 
cluster is considered inactive by default if Ignite Persistent Store 
is used to let all the nodes join the cluster. To activate the cluster 
call Ignite.active(true). 
2. active(false) is synchronous. You can consider that the cluster is 
deactivated when ignite.cluster().active(false) call returned.
3. Every node in cluster has unique consistent ID. It should remain 
unchanged after node restart. By default, it's generated automatically 
from something like host:port, but you can override it via 
IgniteConfiguration#setConsistentId.
This is necessary because {$IGNITE_HOME}\work\db and 
{$IGNITE_HOME}\work\db\wal has subfolders with names of consistent IDs 
ever started locally. If you change consistent ID, old Ignite Native 
Persistence files won't be found and node will start from scratch.


Best Regards,
Ivan Rakov

On 19.02.2018 15:22, arunkjn wrote:

Hi Ivan,

Thanks for you reply.

I have some follow up questions.
1. What is the behaviour of cluster when it is in
ignite.cluster.active(false) state? Does it throw error when a cache seek
operation is performed? Or does it wait for cluster to become active again?
2. How can I reliably know if cluster has become inactive before taking
backup? Is cluster.active(false) a synchronous operation?
3. What do you mean by "consistent ID of your nodes"? Can you please
elaborate?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Write ahead log and early eviction of new elements

[Dmitry Pavlov]

Hi Michail,

To avoid confusion between real evictions and PDS-enabled region pages
rotation with disk we've desided to call second process as 'page
replacement'.

In future releases all messages related to page purge to disk will contain
'page replacement' instead of eviction. Hope it helps to separate these two
process.

Sincerely,
Dmitriy Pavlov

ср, 14 февр. 2018 г. в 19:14, Mikhail :

> Hi Raymond,
>
> >I understand when I add an element to a cache that element is serialized,
> >placed into the local memory for the cache on that server and then placed
> >into the WAL pending checkpointing (merging into the persistence store).
>
> First, the update will be written into WAL and only then into local memory.
>
>
> >What happens if the newly added element is evicted and
> > then re-read from the cache by the client before the next checkpoint
> > occurs?
>
> What do you mean by "evicted"? Ignite evicts memory pages to a disk if
> there's not enough space to save new record or it needs to load a page from
> disk and for this purpose, it will evict some page from memory.
> But it will evict the only page that is already saved to the disk.
>
> Thanks,
> Mike.
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: QueryEntities in Ingnite

[Prasad Bhalerao]

Does setting index of cache configuration solve the coumn Groupid not found
problem.
Cause I am not setting fields explicitly in QueryEntity. Is it required to
set fields in QueryEntity when annotations are used in Pojo?

On Feb 19, 2018 8:45 PM, "Mikhail"  wrote:

Hi,

there's two way to index objects in cache:
1. by annotations:
https://apacheignite.readme.io/docs/cache-queries#section-qu
ery-configuration-by-annotations
2. by QueryEntity:
https://apacheignite.readme.io/docs/cache-queries#section-qu
ery-configuration-using-queryentity

But you can't mix both, please define fields in QueryEntity or set indexed
objects by: CacheConfiguration.setIndexedTypes(MyKey.class, MyValue.class)

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[Webinar] Caching technologies evolution

[Denis Magda]

Igniters,

Curious to see how the caching technologies evolved over the time? Where
Ignite, Redis or Memcached fits? Join my tomorrow webinar at 11.00 PT or
bookmark and get a recording later:

Redis Replaced: Why Companies Now Choose Apache® Ignite™ to Improve
Application Speed and Scale


--
Denis

Re: Large durable caches

[lawrencefinn]

Okay I am trying to reproduce.  It hasn't got stuck yet, but the client got
disconnected and reconnected recently.  I don't think it is related to GC
because I am recording GC times and it does not jump up that much.  Could
the system get slow on a lot of io?  i see this in the ignite log:

[19:13:01,988][WARNING][grid-timeout-worker-#71][diagnostic] Found long
running cache future [startTime=19:11:56.656, curTime=19:13:01.911,
 fut=GridNearAtomicSingleUpdateFuture [reqState=Primary
[id=62a2a255-3320-4040-aa23-ffb86dec7586, opRes=false, expCnt=-1, rcvdCnt=0,
primaryRes=false, done=false, waitFor=null, rcvd=null],
super=GridNearAtomicAbstractUpdateFuture [remapCnt=100,
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=14], remapTopVer=null,
err=null, futId=313296239, super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=1229092316
[19:13:01,988][WARNING][grid-timeout-worker-#71][diagnostic] Found long
running cache future [startTime=19:11:39.917, curTime=19:13:01.911,
fut=GridNearAtomicSingleUpdateFuture [reqState=Primary
[id=62a2a255-3320-4040-aa23-ffb86dec7586, opRes=false, expCnt=-1, rcvdCnt=0,
primaryRes=false, done=false, waitFor=null, rcvd=null],
super=GridNearAtomicAbstractUpdateFuture [remapCnt=100,
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=14], remapTopVer=null,
err=null, futId=312914655, super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=15435296
[19:13:51,057][INFO][db-checkpoint-thread-#110][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=77744626-04e6-4e17-bda7-23ecb50bbe19,
startPtr=FileWALPointer [idx=9600, fileOffset=35172819, len=124303,
forceFlush=true], checkpointLockWait=57708ms, checkpointLockHoldTime=64ms,
pages=3755135, reason='too many dirty pages']
[19:14:01,919][INFO][grid-timeout-worker-#71][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=62a2a255, uptime=01:42:41.752]
^-- H/N/C [hosts=2, nodes=3, CPUs=64]
^-- CPU [cur=77.83%, avg=39.11%, GC=0.13%]
^-- PageMemory [pages=5111642]
^-- Heap [used=11669MB, free=43.02%, comm=20480MB]
^-- Non heap [used=67MB, free=95.56%, comm=69MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
^-- Outbound messages queue [size=0]
[19:15:03,470][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=33542]
[19:15:03,470][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=33542]


My app log has:
2018-02-19 19:15:02,176 [WARN] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-reconnector-#5%cabooseGrid% - Timed out waiting for message
to be read (most probably, the reason is long GC pauses on remote node)
[curTimeout=5000, rmtAddr=/127.0.0.1:47500, rmtPort=47500]
2018-02-19 19:15:02,176 [ERROR] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-reconnector-#5%cabooseGrid% - Exception on joining: Failed
to deserialize object with given class loader:
sun.misc.Launcher$AppClassLoader@28d93b30
org.apache.ignite.IgniteCheckedException: Failed to deserialize object with
given class loader: sun.misc.Launcher$AppClassLoader@28d93b30
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.readMessage(TcpDiscoverySpi.java:1590)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:627)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:524)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:124)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1377)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.net.SocketTimeoutException: Read timed out



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Issues trying to force redeployment in shared mode

[Dave Harvey]

I was trying to demonstrate changing classes on a client node so that classes
on servers get replaced with new code, but the symptoms make me believe that
I don't understand the rules at all.  

The deployment mode is SHARED.   I read the instructions and creates an
ignite.xml with a different userVersion.

and install this on the client node in order to force the cause previously
loaded code versions to be reloaded.   But instead, I get this failure even
if I restart both server and client.


Caused by: class org.apache.ignite.IgniteDeploymentException: Task was not
deployed or was redeployed since task execution
[taskName=org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor$ClientChangeGlobalStateComputeRequest,
taskClsName=org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor$ClientChangeGlobalStateComputeRequest,
codeVer=3, clsLdrId=ff87e49a161-695076b6-c96f-40ec-beb6-9897f3210dee,
seqNum=1518963947775, depMode=SHARED, dep=null]

at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1160)

at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1913)

at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)

at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)

at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)

at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

which seems to be due to "codeVer=3".   I can change the 3 to 0, and it goes
back to working.


The original problem I had was I changed a field in a static statistics
class deployed as part of a [Data]StreamReciever from Long to AtomicLong,
and the BinaryObject schema merging complained about this.  So I renamed the
field, but kept getting the same error with the old field name, so I assumed
that the code was not getting replaced on the server, because I needed to
communicate that the version changed.  The error was being thrown as
part of responding to a call for statistics. Because we are in SHARED
mode, as I read this now, restarting the client should have been sufficient
to replace the code.   So perhaps instead something has remembered the
intermediate schema that could not be merged? I would have assumed that the
intermediate unmerged schema would have been discarded.   




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Large durable caches

[Ivan Rakov]

It's hard to determine the problem by these messages. I don't see 
anything unhealthy regarding persistence - checkpoint start is a regular 
event.


There were cases when excessive GC load on client side seriously 
affected throughput/latency of data streaming. You may consider playing 
with the following data streamer parameters:


public int perNodeBufferSize(int bufSize) - defines how many items 
should be saved in buffer before send it to server node
 public void perNodeParallelOperations(int parallelOps) - defines how 
many buffers can be sent to the node without acknowledge that it was 
processed


Best Regards,
Ivan Rakov

On 19.02.2018 22:24, lawrencefinn wrote:

Okay I am trying to reproduce.  It hasn't got stuck yet, but the client got
disconnected and reconnected recently.  I don't think it is related to GC
because I am recording GC times and it does not jump up that much.  Could
the system get slow on a lot of io?  i see this in the ignite log:

[19:13:01,988][WARNING][grid-timeout-worker-#71][diagnostic] Found long
running cache future [startTime=19:11:56.656, curTime=19:13:01.911,
  fut=GridNearAtomicSingleUpdateFuture [reqState=Primary
[id=62a2a255-3320-4040-aa23-ffb86dec7586, opRes=false, expCnt=-1, rcvdCnt=0,
primaryRes=false, done=false, waitFor=null, rcvd=null],
super=GridNearAtomicAbstractUpdateFuture [remapCnt=100,
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=14], remapTopVer=null,
err=null, futId=313296239, super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=1229092316
[19:13:01,988][WARNING][grid-timeout-worker-#71][diagnostic] Found long
running cache future [startTime=19:11:39.917, curTime=19:13:01.911,
fut=GridNearAtomicSingleUpdateFuture [reqState=Primary
[id=62a2a255-3320-4040-aa23-ffb86dec7586, opRes=false, expCnt=-1, rcvdCnt=0,
primaryRes=false, done=false, waitFor=null, rcvd=null],
super=GridNearAtomicAbstractUpdateFuture [remapCnt=100,
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=14], remapTopVer=null,
err=null, futId=312914655, super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=15435296
[19:13:51,057][INFO][db-checkpoint-thread-#110][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=77744626-04e6-4e17-bda7-23ecb50bbe19,
startPtr=FileWALPointer [idx=9600, fileOffset=35172819, len=124303,
forceFlush=true], checkpointLockWait=57708ms, checkpointLockHoldTime=64ms,
pages=3755135, reason='too many dirty pages']
[19:14:01,919][INFO][grid-timeout-worker-#71][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
 ^-- Node [id=62a2a255, uptime=01:42:41.752]
 ^-- H/N/C [hosts=2, nodes=3, CPUs=64]
 ^-- CPU [cur=77.83%, avg=39.11%, GC=0.13%]
 ^-- PageMemory [pages=5111642]
 ^-- Heap [used=11669MB, free=43.02%, comm=20480MB]
 ^-- Non heap [used=67MB, free=95.56%, comm=69MB]
 ^-- Public thread pool [active=0, idle=0, qSize=0]
 ^-- System thread pool [active=0, idle=6, qSize=0]
 ^-- Outbound messages queue [size=0]
[19:15:03,470][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=33542]
[19:15:03,470][INFO][tcp-disco-srvr-#2][TcpDiscoverySpi] TCP discovery
spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=33542]


My app log has:
2018-02-19 19:15:02,176 [WARN] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-reconnector-#5%cabooseGrid% - Timed out waiting for message
to be read (most probably, the reason is long GC pauses on remote node)
[curTimeout=5000, rmtAddr=/127.0.0.1:47500, rmtPort=47500]
2018-02-19 19:15:02,176 [ERROR] from
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi in
tcp-client-disco-reconnector-#5%cabooseGrid% - Exception on joining: Failed
to deserialize object with given class loader:
sun.misc.Launcher$AppClassLoader@28d93b30
org.apache.ignite.IgniteCheckedException: Failed to deserialize object with
given class loader: sun.misc.Launcher$AppClassLoader@28d93b30
 at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
 at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
 at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
 at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.readMessage(TcpDiscoverySpi.java:1590)
 at
org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:627)
 at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:524)
 at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:124)
 at
org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1377)
 at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.net.SocketTimeoutException: Read timed out



--
Sent from: