Re: Ignite memory memory-architecture with cache partitioned mode

[Mikael]

Hi!

You may get worse performance with a thin client compared to an ordinary 
client because as the thin client works through an intermediary node, so 
your request will always go to node A in your case and then it will be 
handled there, a normal client would go straight to the node where the 
data is while the thin client first has to go to node A and then from 
there to the B node where the data is.


The index data is stored where the data is (B,C) so if your primary data 
and backup data is not on node A it will have to pass on the request to 
B, Ignite may not have index data on node A but it does know on what 
node your data is located so it will go from A to B, it does not have 
the request it from C also.


So with an ordinary client your request will go straight to B and 
response back to client.


With a thin client your request will go to A and from there to B, 
resoponse back to A and then to your client.


Mikael

Den 2020-06-03 kl. 07:22, skrev kay:

Hello, I read this page,

https://apacheignite.readme.io/docs/memory-architecture.

and I would like to know what is going to be happen if there are 3 remote
server nodes(A,B,C)
, and cache mode is partitioned and backups 1.

If I wanna get '1' cache data and my application is connected with node A
using thin client but '1' data is allocated in node B(primary), C(backup).

In these case, I want to know Ignite operating principle for data get.

All node have a index page all data??
I want to know..

I look forward to reply.
Thank you so much.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite memory memory-architecture with cache partitioned mode

[kay]

Hello, I read this page,

https://apacheignite.readme.io/docs/memory-architecture.

and I would like to know what is going to be happen if there are 3 remote
server nodes(A,B,C)
, and cache mode is partitioned and backups 1.

If I wanna get '1' cache data and my application is connected with node A
using thin client but '1' data is allocated in node B(primary), C(backup). 

In these case, I want to know Ignite operating principle for data get.

All node have a index page all data?? 
I want to know..

I look forward to reply.
Thank you so much.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

ignite node heap size

[kay]

Hello, I have 4 nodes for server node and I'd like to allocate 4G each node.
but I don't know how to calculate heap size set for each node of percentage.

what percentage is proper for node (JVM -Xmx) if I use only off-heap cache.

Is there any best practice for heap size for node?? or reference link??


I'll wait for reply.
Thank you so much .




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: connection refused

[Clay Teahouse]

Thanks Wesley, Alex. Attached is a snapshot of the client log. I don't see
anything on the ignite server side.
I am/was trying to do get data from a replicated cache.

On Tue, Jun 2, 2020 at 10:36 AM Alexandr Shapkin  wrote:

> Hi,
>
>
>
> As Wesley mentioned, It’s quite difficult to say something specific about
> without the logs.
>
>
>
> As for the additional questions, can you explain, what Ignite API is being
> used? Is it a Compute API call or
>
> a regular cache GETs?
>
>
>
> It’s totally fine to spawn several user threads and access the server
> simultaneously.
>
> Regarding the node connectivity, by default there is a 1 connection per
> node, you can control it
>
> using the TcpCommunicationSpi#setConnectionsPerNode [1]
>
>
>
> You can check the pool and adjust the sizes using the following docs [2].
>
> If you perform much compute API calls, you might be interested in
> increasing the server public pool size,
>
> striped pool for cache accesses and so on.
>
>
>
> In any case, it’s better to check your system resources utilization
> (CPU/memory/etc) if it’s about 100% on a client
>
> and 20% on a server, you might want to add several more clients to a
> cluster.
>
>
>
> [1] -
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/communication/tcp/TcpCommunicationSpi.html#setConnectionsPerNode-int-
>
>
>
> [2] - https://apacheignite.readme.io/docs/thread-pools
>
>
>
>
>
> *From: *Clay Teahouse 
> *Sent: *Tuesday, June 2, 2020 5:20 PM
> *To: *user@ignite.apache.org
> *Subject: *Re: connection refused
>
>
>
> It turned out if I inject a pause of 4-5 seconds, then the client
> connection goes through.
>
> Why the service grid requests go through  immediately but the regular
> cache access requires a pause from the time an ignite instance is acquired?
> Subsequent calls are fast. The caches are replicated.
>
> Another question, probably trivial:
>
> Can you instantiate multiple concurrent connections from a client to the
> compute nodes? Is the number of connections limited by the number of
> threads? Or do I need to create separate client instances for each request?
>
>
>
>
>
> On Tue, Jun 2, 2020 at 5:46 AM Wesley Peng  wrote:
>
> Does either client or server have any logs?
>
>
>
> Thanks
>
>
>
> On Tue, Jun 2, 2020 at 6:37 PM Clay Teahouse 
> wrote:
>
> I'd appreciate your help with this issue.
>
> I have a server and a client node, the latter running in tomcat. I get
> connection refused if I try to connect to the server node from the client,
> although the server shows the client has joined the cluster. I don't have a
> problem running a service on the server's service grid from the client node.
>
> I don't have any issue connecting from any client anywhere, if the client
> is not running in tomcat.
>
> what could be the problem?
>
>
>
>
>


log2
Description: Binary data

Re: Node is unable to join cluster because it has destroyed caches

[xero]

Hi, thanks for the prompt response
We can have several of these caches, one for each query (is an exceptional
case but, with load, there can be several simultaneously) that is being
executed so we would like to preserve the persistence to take advantage of
the swapping in case the amount of memory is not enough. That's the reason
we didn't try to move these caches to a region without persistence.

Do you think that this scenario will only happen with persistence enabled?
is it possible to remove the validation?

Is this an expected behavior when persistence is enabled? I thought the node
would know that those caches were removed in its absence and would delete
the data accordingly





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Node is unable to join cluster because it has destroyed caches

[Aleksandr Shapkin]

Hi,

Have you tried to put the temp cache into the different, non persisted,
memory region?

You can also try to use a node filter to control what nodes should store
the cache.


On Tue, Jun 2, 2020, 19:24 xero  wrote:

> Hi Ignite team, We have a use case where a small portion of the dataset
> must answer successive queries that could be relatively expensive. For
> this, we create a temporary cache with that small subset of the dataset and
> operate on that new cache. At the end of the process, that cache is
> destroyed. The caches are created *partitioned* and *persistence* is
> enabled. The problem we are facing is the following. If during the reboot
> of a node, one of those "temporal" caches is destroyed (which is very
> likely since they have a short life), the node is unable to rejoin the
> cluster because it contains information about a cache that no longer
> exists. This is the exception: 2020-06-02T02: 56: 22.326 + 00: 00
> fa72f64b5d0f ignite: Caused by: class
> org.apache.ignite.spi.IgniteSpiException: Joining node has caches with data
> which are not presented on cluster, it could mean that they were already
> destroyed, to add the node to cluster - remove directories with the caches
> [cache-tempX, cache-tempY] Is this an expected behavior? Is it
> possible to skip this validation? Is there any way to indicate that it is a
> temporary cache? Thanks
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>

Node is unable to join cluster because it has destroyed caches

[xero]

Hi Ignite team, We have a use case where a small portion of the dataset must
answer successive queries that could be relatively expensive. For this, we
create a temporary cache with that small subset of the dataset and operate
on that new cache. At the end of the process, that cache is destroyed. The
caches are created *partitioned* and *persistence* is enabled.The problem we
are facing is the following. If during the reboot of a node, one of those
"temporal" caches is destroyed (which is very likely since they have a short
life), the node is unable to rejoin the cluster because it contains
information about a cache that no longer exists. This is the
exception:2020-06-02T02: 56: 22.326 + 00: 00 fa72f64b5d0f ignite: Caused by:
class org.apache.ignite.spi.IgniteSpiException: Joining node has caches with
data which are not presented on cluster, it could mean that they were
already destroyed, to add the node to cluster - remove directories with the
caches [cache-tempX, cache-tempY]Is this an expected behavior? Is it
possible to skip this validation? Is there any way to indicate that it is a
temporary cache?Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Semaphore Lock Not Acquired. Race issue?

[adipro]

Server 1 -> Let's say IP is X. Status is running.

Server 2 -> Let's say IP is Y. Status is not running. We stopped this node
for various reasons.

Client1 -> TcpDiscoveryVmIpFinder ipAddress list has two server IPs - X,Y

Client 2 -> TcpDiscoveryVmIpFinder ipAddress list has two server IPs - X,Y


When we start Client1, and Client 2 App machines both at a time. In Client2
App machine, we found the following thread trace. A thread is kept waiting
here since that App started.


```
"1 - 1591083466169" #168 prio=5 os_prio=0 tid=0x7f60e416b800 nid=0x8034
waiting on condition [0x7f5fde005000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000739a4eb78> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:999)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at 
org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7824)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.awaitInitialization(DataStructuresProcessor.java:1161)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:505)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.semaphore(DataStructuresProcessor.java:1270)
at
org.apache.ignite.internal.IgniteKernal.semaphore(IgniteKernal.java:3930)
at
com.a.b.common.ignite.IgniteConnectionHandler.getSemaphore(IgniteConnectionHandler.java:110)
at
com.a.b.common.ignite.IgniteConnectionHandler.startIgniteObject(IgniteConnectionHandler.java:78)
at
com.a.b.common.ignite.IgniteConnectionHandler.getConnection(IgniteConnectionHandler.java:138)
at
com.a.b.common.ignite.IgniteConnectionHandler.(IgniteConnectionHandler.java:32)
at com.a.b.common.ignite.IgniteUtil.keys(IgniteUtil.java:489)
```


Can someone tell why it is WAITING here in Client2? In Client1 there were no
issues and everything is working fine. All the threads in Client2 are in
waiting state as "IgniteConnectionHandler" class's static block code is not
completed.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.8.0: ConcurrentModificationException on attempt to do put into cache

[Ilya Kasnacheev]

Hello!

I have merged this fix to master.

Regards,
-- 
Ilya Kasnacheev


ср, 27 мая 2020 г. в 16:14, Ivan Fedorenkov :

> Hello, Ilya!
>
> Yes, there is an existing ticket and I thought that it hasn’t been merged
> yet for a reason. If it was just missed and it should be merged, then
> should I do something about it? Vote somewhere or something like that?
>
> Thanks for the workaround suggestion! :)
>
> Regards,
> Ivan
>
> On Wed, 27 May 2020 at 15:10, Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> I can see the following ticket:
>> https://issues.apache.org/jira/browse/IGNITE-5214
>>
>> Since it was never resolved, I think the proper workaround will be
>> disabling DEBUG logging, especially for
>> org.apache.ignite.internal.processors.cache.query.continuous
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 27 мая 2020 г. в 09:17, Ivan Fedorenkov :
>>
>>> Thank you, Alex!
>>>
>>> There are no other warning or error messages in ignite log around that
>>> time. Nodes were stable. Though I can provide a full log and the source
>>> code of the entity, I can see that the "pendingEvts" field of the "
>>> CacheContinuousQueryHandler" class is being used without a lock. Could
>>> it be the reason of ConcurrentModificationException?
>>>
>>> Thanks,
>>> Ivan
>>>
>>> ср, 27 мая 2020 г. в 01:28, akorensh :
>>>
 Hi,

 looks like there is an error inside ContinousQuery response collection

 Can you provide more context, and possibly a reproducer project.


 at
 java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
 at
 java.util.TreeMap$EntryIterator.next(TreeMap.java:1247)
 at
 java.util.TreeMap$EntryIterator.next(TreeMap.java:1242)
 at
 java.util.AbstractMap.toString(AbstractMap.java:554)
 at
 java.lang.String.valueOf(String.java:2994)
 at
 java.lang.StringBuilder.append(StringBuilder.java:131)
 at

 org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryPartitionRecovery.collectEntries(CacheContinuousQueryPartitionRecovery.java:261)

 Thanks, Alex



 --
 Sent from: http://apache-ignite-users.70518.x6.nabble.com/

>>>

Re: Semaphore Lock Not Acquired. Race issue?

[akorensh]

Also wanted to mention that there is a working semaphore example available:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datastructures/IgniteSemaphoreExample.java



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Semaphore Lock Not Acquired. Race issue?

[akorensh]

Hi,
  In an older version there was an issue that caused this condition
  https://issues.apache.org/jira/browse/IGNITE-8987

  Send a reproducer, mention the version you are using, along with Ignite
logs(from both nodes) and a stack trace of both servers when this condition
occurs.  

  How to take thread dump:
https://stackoverflow.com/questions/10756105/how-to-get-a-complete-stack-trace-of-a-running-java-program-that-is-taking-100

  Also mention what java version you are using.
Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: connection refused

[Alexandr Shapkin]

Hi, As Wesley mentioned, It’s quite difficult to say something specific about without the logs.  As for the additional questions, can you explain, what Ignite API is being used? Is it a Compute API call or a regular cache GETs? It’s totally fine to spawn several user threads and access the server simultaneously.Regarding the node connectivity, by default there is a 1 connection per node, you can control it using the TcpCommunicationSpi#setConnectionsPerNode [1]  You can check the pool and adjust the sizes using the following docs [2].If you perform much compute API calls, you might be interested in increasing the server public pool size, striped pool for cache accesses and so on. In any case, it’s better to check your system resources utilization (CPU/memory/etc) if it’s about 100% on a client and 20% on a server, you might want to add several more clients to a cluster. [1] - https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/communication/tcp/TcpCommunicationSpi.html#setConnectionsPerNode-int- [2] - https://apacheignite.readme.io/docs/thread-pools  From: Clay TeahouseSent: Tuesday, June 2, 2020 5:20 PMTo: user@ignite.apache.orgSubject: Re: connection refused It turned out if I inject a pause of 4-5 seconds, then the client connection goes through.Why the service grid requests go through  immediately but the regular cache access requires a pause from the time an ignite instance is acquired? Subsequent calls are fast. The caches are replicated.Another question, probably trivial:Can you instantiate multiple concurrent connections from a client to the compute nodes? Is the number of connections limited by the number of threads? Or do I need to create separate client instances for each request?   On Tue, Jun 2, 2020 at 5:46 AM Wesley Peng  wrote:Does either client or server have any logs? Thanks  On Tue, Jun 2, 2020 at 6:37 PM Clay Teahouse  wrote:I'd appreciate your help with this issue.I have a server and a client node, the latter running in tomcat. I get connection refused if I try to connect to the server node from the client, although the server shows the client has joined the cluster. I don't have a problem running a service on the server's service grid from the client node.I don't have any issue connecting from any client anywhere, if the client is not running in tomcat.what could be the problem?  

Re: Unable to enable ML inference storage

[akorensh]

Courtney,
  This config should be enough
  







  
  It will load the ML plugin and initialize model storage.

  You can send me a reproducer and I'll take a look.

Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: connection refused

[Clay Teahouse]

It turned out if I inject a pause of 4-5 seconds, then the client
connection goes through.
Why the service grid requests go through  immediately but the regular cache
access requires a pause from the time an ignite instance is acquired?
Subsequent calls are fast. The caches are replicated.
Another question, probably trivial:
Can you instantiate multiple concurrent connections from a client to the
compute nodes? Is the number of connections limited by the number of
threads? Or do I need to create separate client instances for each request?


On Tue, Jun 2, 2020 at 5:46 AM Wesley Peng  wrote:

> Does either client or server have any logs?
>
> Thanks
>
> On Tue, Jun 2, 2020 at 6:37 PM Clay Teahouse 
> wrote:
>
>> I'd appreciate your help with this issue.
>> I have a server and a client node, the latter running in tomcat. I get
>> connection refused if I try to connect to the server node from the client,
>> although the server shows the client has joined the cluster. I don't have a
>> problem running a service on the server's service grid from the client node.
>> I don't have any issue connecting from any client anywhere, if the client
>> is not running in tomcat.
>> what could be the problem?
>>
>>

Re: connection refused

[Wesley Peng]

Does either client or server have any logs?

Thanks

On Tue, Jun 2, 2020 at 6:37 PM Clay Teahouse  wrote:

> I'd appreciate your help with this issue.
> I have a server and a client node, the latter running in tomcat. I get
> connection refused if I try to connect to the server node from the client,
> although the server shows the client has joined the cluster. I don't have a
> problem running a service on the server's service grid from the client node.
> I don't have any issue connecting from any client anywhere, if the client
> is not running in tomcat.
> what could be the problem?
>
>

connection refused

[Clay Teahouse]

I'd appreciate your help with this issue.
I have a server and a client node, the latter running in tomcat. I get
connection refused if I try to connect to the server node from the client,
although the server shows the client has joined the cluster. I don't have a
problem running a service on the server's service grid from the client node.
I don't have any issue connecting from any client anywhere, if the client
is not running in tomcat.
what could be the problem?