Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

2019-08-15 Thread Pavel Tupitsyn
Hi Eduard,

Yes, that is the same issue.
The workaround above is not enough, because SetSynchronizationContext
affects only the current thread.
After the first async operation completes, the continuation is dispatched
to some ThreadPool thread,
which may not have a SynchronizationContext set.

Updated workaround:

class ThreadPoolSynchronizationContext : SynchronizationContext
{
public override void Post(SendOrPostCallback d, object state)
{
ThreadPool.QueueUserWorkItem(s =>
{
SetSynchronizationContext(this);
d(s);
}, state);
}


Bear in mind, though - this can affect your entire application performance.

We are discussing Ignite-wide fix for this on the dev list.



On Wed, Aug 14, 2019 at 1:21 PM e.llull  wrote:

> Hi guys,
>
> We are also facing a similar problem, if not the same. Our main difference
> with the initial reproducer is that we are using the Thick Client. We
> applied the suggested fix of setting the SynchronizationContext, but we
> also
> perform a GetAsync after the initial PutAsync. Also, I added a loop around
> the Replace because sometimes it takes several iterations to block.
>
> Here is the code:
> using System;
> using System.Collections.Generic;
> using System.Threading;
> using System.Threading.Tasks;
> using Apache.Ignite.Core;
> using Apache.Ignite.Core.Cache.Configuration;
> using Apache.Ignite.Core.Transactions;
>
> namespace IgniteHangTest
> {
> class Program : IDisposable
> {
> protected readonly IIgnite server;
>
> protected readonly IIgnite client;
>
> public static async Task Main(string[] args)
> {
> SynchronizationContext.SetSynchronizationContext(new
> ThreadPoolSynchronizationContext());
>
> using (var program = new Program())
> {
> await program.Run();
> }
> }
>
> public Program() {
> server = Ignition.Start(IgniteConfiguration("server"));
> server.GetOrCreateCache(new
> CacheConfiguration("TestCache")
> {
> AtomicityMode = CacheAtomicityMode.Transactional,
>
> });
>
> var clientConfiguration = IgniteConfiguration("client");
> clientConfiguration.ClientMode = true;
> client = Ignition.Start(clientConfiguration);
> }
>
> private async Task Run() {
> var cache = client.GetCache("TestCache");
>
> Console.WriteLine("Put initial value");
> await cache.PutAsync(0, "Test");
>
> Console.WriteLine("Get initial value");
> string initialValue = await cache.GetAsync(0);  // if removed,
> it works
>
> Console.WriteLine("Entering Replace loop");
> for(int i = 0; i < 100; i++)
> {
> cache.Replace(0, "Replace " + i);  // It blocks here
> Console.WriteLine("Loop: i = {0}", i);
> }
>
> Console.WriteLine("End");
> }
>
> public void Dispose() {
> Ignition.Stop("client", true);
> Ignition.Stop("server", true);
> }
>
> private IgniteConfiguration IgniteConfiguration(string
> instanceName)
> {
> return new IgniteConfiguration
> {
> IgniteInstanceName = instanceName,
> JvmOptions = new List { "-DIGNITE_QUIET=false", },
> TransactionConfiguration = new TransactionConfiguration
> {
> DefaultTimeout = TimeSpan.FromSeconds(5),
> DefaultTransactionConcurrency =
> TransactionConcurrency.Optimistic,
> DefaultTransactionIsolation =
> TransactionIsolation.Serializable
> },
> };
> }
> }
>
> class ThreadPoolSynchronizationContext : SynchronizationContext { }
> }
>
> In the reproducer we are starting two Ignite nodes, one as the server and
> one with ClientMode = true. This is only in the reproducer, in the real use
> case the server Ignite node is started in a different machine but the
> problem also arises with the "external" server Ignite node.
>
> If the line `string initialValue = await cache.GetAsync(0);` is removed the
> programs finishes successfully.
>
> In the console, the relevant logs are:
>
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
> o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]]
> class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]
> at
> 

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

2019-08-14 Thread e.llull
Hi guys, 

We are also facing a similar problem, if not the same. Our main difference
with the initial reproducer is that we are using the Thick Client. We
applied the suggested fix of setting the SynchronizationContext, but we also
perform a GetAsync after the initial PutAsync. Also, I added a loop around
the Replace because sometimes it takes several iterations to block. 

Here is the code: 
using System; 
using System.Collections.Generic; 
using System.Threading; 
using System.Threading.Tasks; 
using Apache.Ignite.Core; 
using Apache.Ignite.Core.Cache.Configuration; 
using Apache.Ignite.Core.Transactions; 

namespace IgniteHangTest 
{ 
class Program : IDisposable 
{ 
protected readonly IIgnite server; 

protected readonly IIgnite client; 

public static async Task Main(string[] args) 
{ 
SynchronizationContext.SetSynchronizationContext(new
ThreadPoolSynchronizationContext()); 

using (var program = new Program()) 
{ 
await program.Run(); 
} 
} 

public Program() { 
server = Ignition.Start(IgniteConfiguration("server")); 
server.GetOrCreateCache(new
CacheConfiguration("TestCache") 
{ 
AtomicityMode = CacheAtomicityMode.Transactional,   
  
}); 

var clientConfiguration = IgniteConfiguration("client"); 
clientConfiguration.ClientMode = true; 
client = Ignition.Start(clientConfiguration); 
} 

private async Task Run() { 
var cache = client.GetCache("TestCache"); 

Console.WriteLine("Put initial value"); 
await cache.PutAsync(0, "Test"); 

Console.WriteLine("Get initial value"); 
string initialValue = await cache.GetAsync(0);  // if removed,
it works 

Console.WriteLine("Entering Replace loop"); 
for(int i = 0; i < 100; i++) 
{ 
cache.Replace(0, "Replace " + i);  // It blocks here 
Console.WriteLine("Loop: i = {0}", i); 
} 

Console.WriteLine("End"); 
} 

public void Dispose() { 
Ignition.Stop("client", true); 
Ignition.Stop("server", true); 
} 

private IgniteConfiguration IgniteConfiguration(string instanceName) 
{ 
return new IgniteConfiguration 
{ 
IgniteInstanceName = instanceName, 
JvmOptions = new List { "-DIGNITE_QUIET=false", }, 
TransactionConfiguration = new TransactionConfiguration 
{ 
DefaultTimeout = TimeSpan.FromSeconds(5), 
DefaultTransactionConcurrency =
TransactionConcurrency.Optimistic, 
DefaultTransactionIsolation =
TransactionIsolation.Serializable 
}, 
}; 
} 
} 

class ThreadPoolSynchronizationContext : SynchronizationContext { } 
} 

In the reproducer we are starting two Ignite nodes, one as the server and
one with ClientMode = true. This is only in the reproducer, in the real use
case the server Ignite node is started in a different machine but the
problem also arises with the "external" server Ignite node. 

If the line `string initialValue = await cache.GetAsync(0);` is removed the
programs finishes successfully. 

In the console, the relevant logs are: 

Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]] 
class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
igniteInstanceName=client, finished=false, heartbeatTs=1565776844696] 
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
 
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
 
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
 
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) 
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
 
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) 
at java.lang.Thread.run(Thread.java:748) 

And the stack trace of the sys-stripe-6 thread is: 
Thread [name="sys-stripe-6-#52%client%", id=82, state=WAITING, blockCnt=0,
waitCnt=11] 
at sun.misc.Unsafe.park(Native Method) 
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) 
at

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

2019-08-06 Thread Pavel Tupitsyn
Sorry guys, I've completely missed this thread, and the topic is very
important.

First, a simple fix for the given example. Add the following on the first
line of Main:
SynchronizationContext.SetSynchronizationContext(new
ThreadPoolSynchronizationContext());

And put the ThreadPoolSynchronizationContext class somewhere:
class ThreadPoolSynchronizationContext : SynchronizationContext
{
// No-op.
}


Now, detailed explanation. The problem exists forever in Ignite and is
mentioned in the docs briefly [1].
Also mentioned in .NET docs (I've updated them a bit) [2].

Breakdown:
* Ignite (Java side) runs async callbacks (continuations) on system
threads, and those threads have limitations (you should not call Ignite
APIs from them in general)
* Ignite.NET wraps async operations into native .NET Tasks
* Usually `await ...` call in .NET will continue execution on the original
Thread (simply put, actually it is more complex), so Ignite system thread
issue is avoided
* However, Console applications have no `SynchronizationContext`, so the
continuation can't be dispatched to original thread, and is executed on
current (Ignite) thread
* Setting custom SynchronizationContext fixes the issue: all async
continuations will be dispatched to .NET thread pool and never executed on
Ignite threads

However, dispatching callbacks to a different thread causes performance
hit, and Ignite favors performance over usability right now.
So it is up to the user to configure desired behavior.

Let me know if you need more details.

Thanks

[1] https://apacheignite.readme.io/docs/async-support
[2] https://apacheignite-net.readme.io/docs/asynchronous-support

On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> I have filed a ticket about this issue so it won't get lost.
> https://issues.apache.org/jira/browse/IGNITE-12033
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 2 мая 2019 г. в 10:53, Barney Pippin :
>
>> Thanks for the response Ilya. Did you get a chance to look at this Pavel?
>> Thanks.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>


Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

2019-08-01 Thread Ilya Kasnacheev
Hello!

I have filed a ticket about this issue so it won't get lost.
https://issues.apache.org/jira/browse/IGNITE-12033

Regards,
-- 
Ilya Kasnacheev


чт, 2 мая 2019 г. в 10:53, Barney Pippin :

> Thanks for the response Ilya. Did you get a chance to look at this Pavel?
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

2019-05-02 Thread Barney Pippin
Thanks for the response Ilya. Did you get a chance to look at this Pavel?
Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

2019-04-25 Thread Ilya Kasnacheev
Hello!

Yes, I can reproduce this scenario.

I attribute it to async/await, where there is a .Net callback which is
invoked from cache operation and which calls cache operations, and this
leads to attempts of scheduling striped pool operations to same stripe as
cuirrently occupied.

Pavel, what do you think of that? Stack trace below:

"sys-stripe-0-#1" #12 prio=5 os_prio=0 tid=0x55cecf9e2800 nid=0x3358
waiting on condition [0x7f25b85c7000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2470)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2468)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4233)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2468)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.replace(GridCacheAdapter.java:2896)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.replace(IgniteCacheProxyImpl.java:1294)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.replace(GatewayProtectedCacheProxy.java:1012)
at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:483)
at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
at
org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongOutLong(Native
Method)
at
org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.futureNullResult(PlatformCallbackGateway.java:643)
at
org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:208)
at
org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:189)
at
org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:382)
at
org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:377)
at
org.apache.ignite.internal.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:215)
at
org.apache.ignite.internal.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:179)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
at
org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
at
org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
at
org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:5022)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:5017)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
at

RE: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

2019-04-25 Thread Barney Pippin
Hi, is anyone else seeing this or able to reproduce? 

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


RE: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

2019-04-15 Thread James PRINCE
Hi,

Thanks for looking. You need to run two instances of the reproducer. Let the 
first run until you can see "Wait" on the console then run the second. For me 
the second instance won't get past the Replace call in either 2.6 or 2.7.

It's using the default config with nothing else set up over and above the code 
you can see on the first post.

Thanks

-Original Message-
From: Alexandr Shapkin  
Sent: 15 April 2019 11:46
To: user@ignite.apache.org
Subject: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Hi,

I took a look at the reproducer and it works just fine with different Ignite 
and .net versions.

Is there just a single Ignite server with the default config?









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.