Hi Eduard,

Yes, that is the same issue.
The workaround above is not enough, because SetSynchronizationContext
affects only the current thread.
After the first async operation completes, the continuation is dispatched
to some ThreadPool thread,
which may not have a SynchronizationContext set.

Updated workaround:

class ThreadPoolSynchronizationContext : SynchronizationContext
{
    public override void Post(SendOrPostCallback d, object state)
    {
        ThreadPool.QueueUserWorkItem(s =>
        {
            SetSynchronizationContext(this);
            d(s);
        }, state);
    }


Bear in mind, though - this can affect your entire application performance.

We are discussing Ignite-wide fix for this on the dev list.



On Wed, Aug 14, 2019 at 1:21 PM e.llull <edu...@llull.net> wrote:

> Hi guys,
>
> We are also facing a similar problem, if not the same. Our main difference
> with the initial reproducer is that we are using the Thick Client. We
> applied the suggested fix of setting the SynchronizationContext, but we
> also
> perform a GetAsync after the initial PutAsync. Also, I added a loop around
> the Replace because sometimes it takes several iterations to block.
>
> Here is the code:
> using System;
> using System.Collections.Generic;
> using System.Threading;
> using System.Threading.Tasks;
> using Apache.Ignite.Core;
> using Apache.Ignite.Core.Cache.Configuration;
> using Apache.Ignite.Core.Transactions;
>
> namespace IgniteHangTest
> {
>     class Program : IDisposable
>     {
>         protected readonly IIgnite server;
>
>         protected readonly IIgnite client;
>
>         public static async Task Main(string[] args)
>         {
>             SynchronizationContext.SetSynchronizationContext(new
> ThreadPoolSynchronizationContext());
>
>             using (var program = new Program())
>             {
>                 await program.Run();
>             }
>         }
>
>         public Program() {
>             server = Ignition.Start(IgniteConfiguration("server"));
>             server.GetOrCreateCache<int, string>(new
> CacheConfiguration("TestCache")
>             {
>                 AtomicityMode = CacheAtomicityMode.Transactional,
>
>             });
>
>             var clientConfiguration = IgniteConfiguration("client");
>             clientConfiguration.ClientMode = true;
>             client = Ignition.Start(clientConfiguration);
>         }
>
>         private async Task Run() {
>             var cache = client.GetCache<int, string>("TestCache");
>
>             Console.WriteLine("Put initial value");
>             await cache.PutAsync(0, "Test");
>
>             Console.WriteLine("Get initial value");
>             string initialValue = await cache.GetAsync(0);  // if removed,
> it works
>
>             Console.WriteLine("Entering Replace loop");
>             for(int i = 0; i < 100; i++)
>             {
>                 cache.Replace(0, "Replace " + i);  // It blocks here
>                 Console.WriteLine("Loop: i = {0}", i);
>             }
>
>             Console.WriteLine("End");
>         }
>
>         public void Dispose() {
>             Ignition.Stop("client", true);
>             Ignition.Stop("server", true);
>         }
>
>         private IgniteConfiguration IgniteConfiguration(string
> instanceName)
>         {
>             return new IgniteConfiguration
>             {
>                 IgniteInstanceName = instanceName,
>                 JvmOptions = new List<string> { "-DIGNITE_QUIET=false", },
>                 TransactionConfiguration = new TransactionConfiguration
>                 {
>                     DefaultTimeout = TimeSpan.FromSeconds(5),
>                     DefaultTransactionConcurrency =
> TransactionConcurrency.Optimistic,
>                     DefaultTransactionIsolation =
> TransactionIsolation.Serializable
>                 },
>             };
>         }
>     }
>
>     class ThreadPoolSynchronizationContext : SynchronizationContext { }
> }
>
> In the reproducer we are starting two Ignite nodes, one as the server and
> one with ClientMode = true. This is only in the reproducer, in the real use
> case the server Ignite node is started in a different machine but the
> problem also arises with the "external" server Ignite node.
>
> If the line `string initialValue = await cache.GetAsync(0);` is removed the
> programs finishes successfully.
>
> In the console, the relevant logs are:
>
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
> o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]]
> class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]
>         at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
>
>         at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
>
>         at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
>
>         at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
>
>         at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
>
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
>         at java.lang.Thread.run(Thread.java:748)
>
> And the stack trace of the sys-stripe-6 thread is:
> Thread [name="sys-stripe-6-#52%client%", id=82, state=WAITING, blockCnt=0,
> waitCnt=11]
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
>         at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
>         at
> o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2470)
>
>         at
> o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2468)
>
>         at
> o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4233)
>
>         at
> o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2468)
>         at
> o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
>         at
> o.a.i.i.processors.cache.GridCacheAdapter.replace(GridCacheAdapter.java:2896)
>
>         at
> o.a.i.i.processors.cache.IgniteCacheProxyImpl.replace(IgniteCacheProxyImpl.java:1294)
>
>         at
> o.a.i.i.processors.cache.GatewayProtectedCacheProxy.replace(GatewayProtectedCacheProxy.java:1012)
>
>         at
> o.a.i.i.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:483)
>
>         at
> o.a.i.i.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
>
>         at
>
> o.a.i.i.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native
> Method)
>         at
> o.a.i.i.processors.platform.callback.PlatformCallbackGateway.futureObjectResult(PlatformCallbackGateway.java:626)
>
>         at
> o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:219)
>
>         at
> o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:189)
>
>         at
> o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:382)
>
>         at
> o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:377)
>
>         at
> o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:215)
>
>         at
> o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:179)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
>         at
> o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
>         at
> o.a.i.i.processors.cache.GridCacheFutureAdapter.onDone(GridCacheFutureAdapter.java:55)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
>         at
> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onDone(GridPartitionedSingleGetFuture.java:774)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
>         at
> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.setResult(GridPartitionedSingleGetFuture.java:696)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:551)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:368)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.access$100(GridDhtColocatedCache.java:87)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:132)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:130)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
>
>         at
> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>
>         at
> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>
>         at
> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>
>         at
> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>
>         at
> o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
>         at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
>         at java.lang.Thread.run(Thread.java:748)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Reply via email to