Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Igniters,
Bump this thread up. It seems the issue [1] is important but still
unassigned.
[1] https://issues.apache.org/jira/browse/IGNITE-12033
On Thu, 15 Aug 2019 at 15:01, Pavel Tupitsyn wrote:
> Hi Eduard,
>
> Yes, that is the same issue.
> The workaround above is not enough, because SetSynchronizationContext
> affects only the current thread.
> After the first async operation completes, the continuation is dispatched
> to some ThreadPool thread,
> which may not have a SynchronizationContext set.
>
> Updated workaround:
>
> class ThreadPoolSynchronizationContext : SynchronizationContext
> {
> public override void Post(SendOrPostCallback d, object state)
> {
> ThreadPool.QueueUserWorkItem(s =>
> {
> SetSynchronizationContext(this);
> d(s);
> }, state);
> }
>
>
> Bear in mind, though - this can affect your entire application performance.
>
> We are discussing Ignite-wide fix for this on the dev list.
>
>
>
> On Wed, Aug 14, 2019 at 1:21 PM e.llull wrote:
>
>> Hi guys,
>>
>> We are also facing a similar problem, if not the same. Our main difference
>> with the initial reproducer is that we are using the Thick Client. We
>> applied the suggested fix of setting the SynchronizationContext, but we
>> also
>> perform a GetAsync after the initial PutAsync. Also, I added a loop around
>> the Replace because sometimes it takes several iterations to block.
>>
>> Here is the code:
>> using System;
>> using System.Collections.Generic;
>> using System.Threading;
>> using System.Threading.Tasks;
>> using Apache.Ignite.Core;
>> using Apache.Ignite.Core.Cache.Configuration;
>> using Apache.Ignite.Core.Transactions;
>>
>> namespace IgniteHangTest
>> {
>> class Program : IDisposable
>> {
>> protected readonly IIgnite server;
>>
>> protected readonly IIgnite client;
>>
>> public static async Task Main(string[] args)
>> {
>> SynchronizationContext.SetSynchronizationContext(new
>> ThreadPoolSynchronizationContext());
>>
>> using (var program = new Program())
>> {
>> await program.Run();
>> }
>> }
>>
>> public Program() {
>> server = Ignition.Start(IgniteConfiguration("server"));
>> server.GetOrCreateCache(new
>> CacheConfiguration("TestCache")
>> {
>> AtomicityMode = CacheAtomicityMode.Transactional,
>>
>> });
>>
>> var clientConfiguration = IgniteConfiguration("client");
>> clientConfiguration.ClientMode = true;
>> client = Ignition.Start(clientConfiguration);
>> }
>>
>> private async Task Run() {
>> var cache = client.GetCache("TestCache");
>>
>> Console.WriteLine("Put initial value");
>> await cache.PutAsync(0, "Test");
>>
>> Console.WriteLine("Get initial value");
>> string initialValue = await cache.GetAsync(0); // if removed,
>> it works
>>
>> Console.WriteLine("Entering Replace loop");
>> for(int i = 0; i < 100; i++)
>> {
>> cache.Replace(0, "Replace " + i); // It blocks here
>> Console.WriteLine("Loop: i = {0}", i);
>> }
>>
>> Console.WriteLine("End");
>> }
>>
>> public void Dispose() {
>> Ignition.Stop("client", true);
>> Ignition.Stop("server", true);
>> }
>>
>> private IgniteConfiguration IgniteConfiguration(string
>> instanceName)
>> {
>> return new IgniteConfiguration
>> {
>> IgniteInstanceName = instanceName,
>> JvmOptions = new List { "-DIGNITE_QUIET=false",
>> },
>> TransactionConfiguration = new TransactionConfiguration
>> {
>> DefaultTimeout = TimeSpan.FromSeconds(5),
>> DefaultTransactionConcurrency =
>> TransactionConcurrency.Optimistic,
>> DefaultTransactionIsolation =
>> TransactionIsolation.Serializable
>> },
>> };
>> }
>> }
>>
>> class ThreadPoolSynchronizationContext : SynchronizationContext { }
>> }
>>
>> In the reproducer we are starting two Ignite nodes, one as the server and
>> one with ClientMode = true. This is only in the reproducer, in the real
>> use
>> case the server Ignite node is started in a different machine but the
>> problem also arises with the "external" server Ignite node.
>>
>> If the line `string initialValue = await cache.GetAsync(0);` is removed
>> the
>> programs finishes successfully.
>>
>> In the console, the relevant logs are:
>>
>> Critical system error detected. Will be handled accordingly to configured
>> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
>> super=AbstractFailureHandler
>> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
>>
Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Hi Eduard,
Yes, that is the same issue.
The workaround above is not enough, because SetSynchronizationContext
affects only the current thread.
After the first async operation completes, the continuation is dispatched
to some ThreadPool thread,
which may not have a SynchronizationContext set.
Updated workaround:
class ThreadPoolSynchronizationContext : SynchronizationContext
{
public override void Post(SendOrPostCallback d, object state)
{
ThreadPool.QueueUserWorkItem(s =>
{
SetSynchronizationContext(this);
d(s);
}, state);
}
Bear in mind, though - this can affect your entire application performance.
We are discussing Ignite-wide fix for this on the dev list.
On Wed, Aug 14, 2019 at 1:21 PM e.llull wrote:
> Hi guys,
>
> We are also facing a similar problem, if not the same. Our main difference
> with the initial reproducer is that we are using the Thick Client. We
> applied the suggested fix of setting the SynchronizationContext, but we
> also
> perform a GetAsync after the initial PutAsync. Also, I added a loop around
> the Replace because sometimes it takes several iterations to block.
>
> Here is the code:
> using System;
> using System.Collections.Generic;
> using System.Threading;
> using System.Threading.Tasks;
> using Apache.Ignite.Core;
> using Apache.Ignite.Core.Cache.Configuration;
> using Apache.Ignite.Core.Transactions;
>
> namespace IgniteHangTest
> {
> class Program : IDisposable
> {
> protected readonly IIgnite server;
>
> protected readonly IIgnite client;
>
> public static async Task Main(string[] args)
> {
> SynchronizationContext.SetSynchronizationContext(new
> ThreadPoolSynchronizationContext());
>
> using (var program = new Program())
> {
> await program.Run();
> }
> }
>
> public Program() {
> server = Ignition.Start(IgniteConfiguration("server"));
> server.GetOrCreateCache(new
> CacheConfiguration("TestCache")
> {
> AtomicityMode = CacheAtomicityMode.Transactional,
>
> });
>
> var clientConfiguration = IgniteConfiguration("client");
> clientConfiguration.ClientMode = true;
> client = Ignition.Start(clientConfiguration);
> }
>
> private async Task Run() {
> var cache = client.GetCache("TestCache");
>
> Console.WriteLine("Put initial value");
> await cache.PutAsync(0, "Test");
>
> Console.WriteLine("Get initial value");
> string initialValue = await cache.GetAsync(0); // if removed,
> it works
>
> Console.WriteLine("Entering Replace loop");
> for(int i = 0; i < 100; i++)
> {
> cache.Replace(0, "Replace " + i); // It blocks here
> Console.WriteLine("Loop: i = {0}", i);
> }
>
> Console.WriteLine("End");
> }
>
> public void Dispose() {
> Ignition.Stop("client", true);
> Ignition.Stop("server", true);
> }
>
> private IgniteConfiguration IgniteConfiguration(string
> instanceName)
> {
> return new IgniteConfiguration
> {
> IgniteInstanceName = instanceName,
> JvmOptions = new List { "-DIGNITE_QUIET=false", },
> TransactionConfiguration = new TransactionConfiguration
> {
> DefaultTimeout = TimeSpan.FromSeconds(5),
> DefaultTransactionConcurrency =
> TransactionConcurrency.Optimistic,
> DefaultTransactionIsolation =
> TransactionIsolation.Serializable
> },
> };
> }
> }
>
> class ThreadPoolSynchronizationContext : SynchronizationContext { }
> }
>
> In the reproducer we are starting two Ignite nodes, one as the server and
> one with ClientMode = true. This is only in the reproducer, in the real use
> case the server Ignite node is started in a different machine but the
> problem also arises with the "external" server Ignite node.
>
> If the line `string initialValue = await cache.GetAsync(0);` is removed the
> programs finishes successfully.
>
> In the console, the relevant logs are:
>
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
> o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]]
> class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]
> at
>
Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Hi guys,
We are also facing a similar problem, if not the same. Our main difference
with the initial reproducer is that we are using the Thick Client. We
applied the suggested fix of setting the SynchronizationContext, but we also
perform a GetAsync after the initial PutAsync. Also, I added a loop around
the Replace because sometimes it takes several iterations to block.
Here is the code:
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Apache.Ignite.Core;
using Apache.Ignite.Core.Cache.Configuration;
using Apache.Ignite.Core.Transactions;
namespace IgniteHangTest
{
class Program : IDisposable
{
protected readonly IIgnite server;
protected readonly IIgnite client;
public static async Task Main(string[] args)
{
SynchronizationContext.SetSynchronizationContext(new
ThreadPoolSynchronizationContext());
using (var program = new Program())
{
await program.Run();
}
}
public Program() {
server = Ignition.Start(IgniteConfiguration("server"));
server.GetOrCreateCache(new
CacheConfiguration("TestCache")
{
AtomicityMode = CacheAtomicityMode.Transactional,
});
var clientConfiguration = IgniteConfiguration("client");
clientConfiguration.ClientMode = true;
client = Ignition.Start(clientConfiguration);
}
private async Task Run() {
var cache = client.GetCache("TestCache");
Console.WriteLine("Put initial value");
await cache.PutAsync(0, "Test");
Console.WriteLine("Get initial value");
string initialValue = await cache.GetAsync(0); // if removed,
it works
Console.WriteLine("Entering Replace loop");
for(int i = 0; i < 100; i++)
{
cache.Replace(0, "Replace " + i); // It blocks here
Console.WriteLine("Loop: i = {0}", i);
}
Console.WriteLine("End");
}
public void Dispose() {
Ignition.Stop("client", true);
Ignition.Stop("server", true);
}
private IgniteConfiguration IgniteConfiguration(string instanceName)
{
return new IgniteConfiguration
{
IgniteInstanceName = instanceName,
JvmOptions = new List { "-DIGNITE_QUIET=false", },
TransactionConfiguration = new TransactionConfiguration
{
DefaultTimeout = TimeSpan.FromSeconds(5),
DefaultTransactionConcurrency =
TransactionConcurrency.Optimistic,
DefaultTransactionIsolation =
TransactionIsolation.Serializable
},
};
}
}
class ThreadPoolSynchronizationContext : SynchronizationContext { }
}
In the reproducer we are starting two Ignite nodes, one as the server and
one with ClientMode = true. This is only in the reproducer, in the real use
case the server Ignite node is started in a different machine but the
problem also arises with the "external" server Ignite node.
If the line `string initialValue = await cache.GetAsync(0);` is removed the
programs finishes successfully.
In the console, the relevant logs are:
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]]
class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
And the stack trace of the sys-stripe-6 thread is:
Thread [name="sys-stripe-6-#52%client%", id=82, state=WAITING, blockCnt=0,
waitCnt=11]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Sorry guys, I've completely missed this thread, and the topic is very
important.
First, a simple fix for the given example. Add the following on the first
line of Main:
SynchronizationContext.SetSynchronizationContext(new
ThreadPoolSynchronizationContext());
And put the ThreadPoolSynchronizationContext class somewhere:
class ThreadPoolSynchronizationContext : SynchronizationContext
{
// No-op.
}
Now, detailed explanation. The problem exists forever in Ignite and is
mentioned in the docs briefly [1].
Also mentioned in .NET docs (I've updated them a bit) [2].
Breakdown:
* Ignite (Java side) runs async callbacks (continuations) on system
threads, and those threads have limitations (you should not call Ignite
APIs from them in general)
* Ignite.NET wraps async operations into native .NET Tasks
* Usually `await ...` call in .NET will continue execution on the original
Thread (simply put, actually it is more complex), so Ignite system thread
issue is avoided
* However, Console applications have no `SynchronizationContext`, so the
continuation can't be dispatched to original thread, and is executed on
current (Ignite) thread
* Setting custom SynchronizationContext fixes the issue: all async
continuations will be dispatched to .NET thread pool and never executed on
Ignite threads
However, dispatching callbacks to a different thread causes performance
hit, and Ignite favors performance over usability right now.
So it is up to the user to configure desired behavior.
Let me know if you need more details.
Thanks
[1] https://apacheignite.readme.io/docs/async-support
[2] https://apacheignite-net.readme.io/docs/asynchronous-support
On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev
wrote:
> Hello!
>
> I have filed a ticket about this issue so it won't get lost.
> https://issues.apache.org/jira/browse/IGNITE-12033
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 2 мая 2019 г. в 10:53, Barney Pippin :
>
>> Thanks for the response Ilya. Did you get a chance to look at this Pavel?
>> Thanks.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Hello! I have filed a ticket about this issue so it won't get lost. https://issues.apache.org/jira/browse/IGNITE-12033 Regards, -- Ilya Kasnacheev чт, 2 мая 2019 г. в 10:53, Barney Pippin : > Thanks for the response Ilya. Did you get a chance to look at this Pavel? > Thanks. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Thanks for the response Ilya. Did you get a chance to look at this Pavel? Thanks. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Hello! Yes, I can reproduce this scenario. I attribute it to async/await, where there is a .Net callback which is invoked from cache operation and which calls cache operations, and this leads to attempts of scheduling striped pool operations to same stripe as cuirrently occupied. Pavel, what do you think of that? Stack trace below: "sys-stripe-0-#1" #12 prio=5 os_prio=0 tid=0x55cecf9e2800 nid=0x3358 waiting on condition [0x7f25b85c7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2470) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2468) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4233) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2468) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.replace(GridCacheAdapter.java:2896) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.replace(IgniteCacheProxyImpl.java:1294) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.replace(GatewayProtectedCacheProxy.java:1012) at org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:483) at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67) at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongOutLong(Native Method) at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.futureNullResult(PlatformCallbackGateway.java:643) at org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:208) at org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:189) at org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:382) at org.apache.ignite.internal.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:377) at org.apache.ignite.internal.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:215) at org.apache.ignite.internal.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:179) at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453) at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78) at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70) at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30) at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:5022) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:5017) at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385) at
RE: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Hi, is anyone else seeing this or able to reproduce? Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Hi, Thanks for looking. You need to run two instances of the reproducer. Let the first run until you can see "Wait" on the console then run the second. For me the second instance won't get past the Replace call in either 2.6 or 2.7. It's using the default config with nothing else set up over and above the code you can see on the first post. Thanks -Original Message- From: Alexandr Shapkin Sent: 15 April 2019 11:46 To: user@ignite.apache.org Subject: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang Hi, I took a look at the reproducer and it works just fine with different Ignite and .net versions. Is there just a single Ignite server with the default config? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ ___ This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is prohibited. Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional disclosures.
