Hi Pablo,

If you want to have more logging for async io, please pass 
`MONO_LOG_MASK=io-threadpool MONO_LOG_LEVEL=debug` as environment variables. 
You can send us the output for your run, and we will take a look.

Also, on 4.6 the default backend for our async io is poll (not epoll or 
kqueue), but you can force their use by passing `MONO_ENABLE_AIO=1` as an 
environment variable.

I hope it helps,
Ludovic

From: Mono-devel-list <mono-devel-list-boun...@lists.dot.net> on behalf of Ivo 
Smits <i...@ufo-net.nl>
Date: Tuesday, 13 September 2016 at 09:10
To: "mono-devel-list@lists.dot.net" <mono-devel-list@lists.dot.net>
Subject: Re: [Mono-dev] Potential issue with async sockets in Raspberry ARM – 
Mono 4.4.0


Hi Pablo,

You can send a SIGQUIT signal to mono (killall -SIGQUIT mono or ctrl+\ might 
work as well) to get a stacktraces for all live threads. This might show 
possible problems with threadpool exhaustion or other blocking calls.

Have you checked with netstat if there's data in de receive queue?
Ivo
Op 13-9-2016 om 0:45 schreef 
psant...@codicesoftware.com<mailto:psant...@codicesoftware.com>:
Hi Alan,

I'll try to create a small test case. We are experiencing this running a 
Plastic SCM server on raspberry, and a client on a different machine. It will 
take me a few hours to isolate a test case taking just the required parts from 
the plastic network protocol.

Meanwhile: do you have any suggestion to try to check whether my assumption is 
correct? I mean, I suspect somehow the BeginReceive never awakes. I guess it is 
using epoll down the stairs. Any log or something I can build to make sure the 
issue is on my code and not on Mono?

Thanks,

pablo

On 9/12/2016 11:37, Alan McGovern wrote:
Can you share the code to repro the issue?

Sent from my iPhone

On 12 Sep 2016, at 09:39, 
"psant...@codicesoftware.com<mailto:psant...@codicesoftware.com>" 
<psant...@codicesoftware.com<mailto:psant...@codicesoftware.com>> wrote:
Thanks for the hint Chris,

We are actually enqueuing the request and doing the actual read on a different 
threadpool.

That being said, we are reproducing this issue with a single client, I mean, 
single thread attending on the server. Not sure what can go wrong, tcp 
connection is still there, client can send, it simply looks like the server 
never wakes up to attend it, randomly.

Any other hints would be appreciated.

Thanks,

pablo
On 9/9/2016 18:11, Chris Swiedler wrote:
From what I understand it’s dangerous to do blocking reads on the callback from 
something like BeginReceive. The callback will happen on a threadpool thread, 
and the blocking reads could then cause the threadpool to be exhausted. I don’t 
know if that’s causing your specific problem (it could, if the reads really do 
block and you have enough of them) but it’s something to watch out for.

chris

From: Mono-devel-list [mailto:mono-devel-list-boun...@lists.dot.net] On Behalf 
Of psant...@codicesoftware.com<mailto:psant...@codicesoftware.com>
Sent: Friday, September 09, 2016 2:42 AM
To: mono-devel-list@lists.dot.net<mailto:mono-devel-list@lists.dot.net>
Cc: dude <rdea...@codicesoftware.com><mailto:rdea...@codicesoftware.com>
Subject: [Mono-dev] Potential issue with async sockets in Raspberry ARM – Mono 
4.4.0

Hi,

I’m posting here hoping someone can throw some light into the problem. I'm a 
little bit lost now.

Our server code (Plastic SCM) running on Raspberry on Mono 4.4.0 (exact version 
below) ends up not awaking from socket.BeginReceive after a while.

I mean, client connects and requests data in 4MB chunks, and depending on the 
run, it can transfer a few GB but it ends up not awaking. The client just sits 
waiting on a “socket recv” but the server doesn’t answer. Connection is 
established (can be checked at OS level).

The code could be simplified as follows:

            mSocket.BeginReceive(buffer, 0, 0, SocketFlags.None,
                OnMessageReceived, null);

OnMessageReceived => does the EndReceive and then reads data and enqueues the 
request for a threadpool to attend it. Once the request is attented and the 
response sent, BeginReceive is invoked again. Important: all “BeginReceive()” 
calls are done from the same thread which NEVER dies.

All we use the BeginReceive for is to decouple socket from thread, so we do not 
have a 1-1. You see we do pass “zero” as bytes to read, because all we want to 
achieve is to get awakened when new data is received, then just read using 
blocking calls (no async).

I’m asking if there could be something about Mono because I read 4.6 release 
notes 
(http://www.mono-project.com/docs/about-mono/releases/4.6.0/<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.mono-project.com%2fdocs%2fabout-mono%2freleases%2f4.6.0%2f&data=02%7c01%7cluhenry%40microsoft.com%7cf762cd1214474206d05208d3dba522b8%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636093474720622808&sdata=mGh0hVzd%2fICsVT%2fc7NdzFgK80JysAfsWc5J9z%2bSYckI%3d>)
 and the “atomic fixes for ARM64”. Could it be related somehow.

The same code runs on Windows and Linux PCs (even Macs) without issues. We use 
the same code on all our production servers and even our Cloud roles, and we 
are not aware of issues.
Now we are testing a new faster storage and using Raspberry to check how fast 
it goes on slower hardware.

Complete Mono version:
Mono JIT compiler version 4.4.0 (tarball Tue Jun 14 13:44:06 UTC 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. 
www.mono-project.com<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.mono-project.com&data=02%7c01%7cluhenry%40microsoft.com%7cf762cd1214474206d05208d3dba522b8%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636093474720622808&sdata=3GWB27ZrX61ErNX5sotWFjo4SG1stTju3PdxhDlwYo8%3d>
        TLS:           __thread
        SIGSEGV:       normal
        Notifications: epoll
        Architecture:  armel,vfp+hard
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen
.

Thanks!

Pablo Santos
www.plasticscm.com<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.plasticscm.com&data=02%7c01%7cluhenry%40microsoft.com%7cf762cd1214474206d05208d3dba522b8%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636093474720622808&sdata=F2tQB4ySYUJfos5VISIxbCyvODwUCPkqigsv7UsXF9U%3d>

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@lists.dot.net<mailto:Mono-devel-list@lists.dot.net>
http://lists.dot.net/mailman/listinfo/mono-devel-list<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.dot.net%2fmailman%2flistinfo%2fmono-devel-list&data=02%7c01%7cluhenry%40microsoft.com%7cf762cd1214474206d05208d3dba522b8%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636093474720622808&sdata=b7mAiXwdY8V8MkFRWPouijLQb%2bupY8FdXZX75KsLzXk%3d>





_______________________________________________

Mono-devel-list mailing list

Mono-devel-list@lists.dot.net<mailto:Mono-devel-list@lists.dot.net>

http://lists.dot.net/mailman/listinfo/mono-devel-list<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.dot.net%2fmailman%2flistinfo%2fmono-devel-list&data=02%7c01%7cluhenry%40microsoft.com%7cf762cd1214474206d05208d3dba522b8%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636093474720622808&sdata=b7mAiXwdY8V8MkFRWPouijLQb%2bupY8FdXZX75KsLzXk%3d>


_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@lists.dot.net
http://lists.dot.net/mailman/listinfo/mono-devel-list

Reply via email to