Re: [Mono-dev] FastCGI Performance
I've been experimenting with hello world requests running under self-hosted Nancy and I'm getting about 2000 requests/second and sometimes locks up after a long period of time. When running similar test using Golang and Martini I'm getting around 4700 on the same hardware. I'm concerned why there is such a big difference between Mono/HttpListener and Go. It shouldn't be that much slower. Anyway gonna try your FastCGI implementation with Nginx in front and see how it performs. Keep these benchmarks coming, that's the only way Webstuff on Mono will improve! We need to get Mono higher up in http://www.techempower.com/benchmarks/#section=data-r8hw=i7test=jsons=2p=13ydj4-0 . On Mon, Apr 7, 2014 at 3:20 PM, Marcelo Zabani mzab...@gmail.com wrote: I have not compared the fastcgi implementation per se, because it is not very easy to test only the underlying fastcgi implementations and I never had the time for that. Fos, however, is a highly asynchronous server implementation, and I've seen it dealing with a lot of connections simultaneously. I haven't benchmarked it properly and compared it to other servers yet, but I'll try to do that in the next two weeks. I run a website with Fos and I get 10-20ms average response time (measured as Fos - Nginx, that is, not counting the time it takes for the response to reach the user) with static pages. In case you want to take a better look at these numbers, take a look at http://beeder.com.br/_stats Don't be scared by the large response times for some of the URLs, as those are usually contacting Facebook through Fb's API. On Mon, Apr 7, 2014 at 8:02 AM, Giuliano Barberi gbarb...@aotaonline.comwrote: Have you benchmarked it to see how it compares to the existing FastCGI implementation? Regards On Sun, Apr 6, 2014 at 11:03 PM, Marcelo Zabani mzab...@gmail.comwrote: In case you want to host an OWIN application with Mono and FastCgi, you may wanna take a look at a project of mine, Fos: https://github.com/mzabani/Fos It is also available at NuGet. -- From: Greg Najda gregna...@gmail.com Sent: 06/04/2014 22:43 To: Giuliano Barberi gbarb...@aotaonline.com Cc: Mono Developer List mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] FastCGI Performance Someone looked into Mono FastCGI performance a couple months ago and made a series of blog posts: http://forcedtoadmin.blogspot.com/2013/11/servicestack-performance-in-mono-p1.html http://forcedtoadmin.blogspot.com/2013/11/servicestack-performance-in-mono-p2.html http://forcedtoadmin.blogspot.com/2013/12/servicestack-performance-in-mono-p3.html He ended up writing a replacement for the Mono FastCGI server instead of changing it because of architectural changes: https://github.com/xplicit/HyperFastCgi On Sun, Apr 6, 2014 at 7:43 PM, Giuliano Barberi gbarb...@aotaonline.com wrote: After looking at some of the Mono web benchmarks ( http://www.techempower.com/benchmarks/#section=data-r8hw=i7test=jsons=2p=13ydj4-0 ) I got very curious as to why FastCGI performance was so much lower than when using a C# libevent implementation. If you look at nancy-libevent2 vs nancy benchmarks, the only difference is a C# wrapper around libevent ( https://github.com/kekekeks/evhttp-sharp ) vs Mono FastCGI. Since Mono uses epoll underneath which is what libevent uses afaik, I would not expect there to be such a gap in performance. I'm curious whether performance of FastCGI is being looked at or if this is expected. Mono when using FastCGI benchmarks almost at the bottom of the list when compared to many other technologies ( http://www.techempower.com/benchmarks/#section=data-r8hw=i7test=json ). I've done a bit of analysis on where the bottleneck is and everything I've found is pointing to the FastCGI implementation. Regards -- Giuliano Barberi ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list -- Giuliano Barberi ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
So far so good, done few tests and no lockups.
-Original message-
From: Nikita Tsukanov kek...@gmail.com
Sent: Wednesday 14th August 2013 15:04
To: mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Should be fixed with
https://github.com/mono/mono/commit/a31b580fdcbaa9a8a16d59ffb12d04f5872f54e8
Please, try to compile a fresh version of mono and check if the issue is still
present.
2013/8/13 Nikita Tsukanov kek...@gmail.com mailto:kek...@gmail.com
I've filled a bug report: https://bugzilla.xamarin.com/show_bug.cgi?id=13933
(also increased thread count from 2 to 10 to make it easier to reproduce the
issue).
2013/8/12 Nikita Tsukanov kek...@gmail.com mailto:kek...@gmail.com
Oh, great. I've got lockup and segfault (on different runs) with this code:
static INancyEngine _engine;
public static void Main(string[] args)
{
var pool = new SmartThreadPool ();
var bt = new DefaultNancyBootstrapper ();
bt.Initialise ();
_engine = bt.GetEngine ();
for (int c=0; c2; c++)
new Thread (ThreadProc).Start ();
Thread.Sleep (-1);
}
static void ThreadProc ()
{
int crid = 0;
while (true) {
crid ++;
var resp = _engine.HandleRequest (
new Request (GET, /,
new Dictionarystring,
IEnumerablestring (),
RequestStream.FromStream (new
MemoryStream (), 0, 0), http, , 127.0.0.1));
resp.Response.Contents (new MemoryStream ());
Console.WriteLine (Request #{0} done, crid);
}
}
It just uses nancyfx engine from 2 threads.
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Nikita: Have you filed a bug for this yet? If so please give me the bug number
so I can follow it.
-Original message-
From:quot;Andrés G. Aragonesesquot; kno...@gmail.com
mailto:kno...@gmail.com
Sent: Wednesday 7th August 2013 13:15
To: mono-devel-list@lists.ximian.com
mailto:mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on
linux
If you're able to reproduce it with 3.3 and not with 2.10.x, you should
definitely open a bug report for it in http://bugzilla.xamarin.com
http://bugzilla.xamarin.com /
stating [regression] in the bug summary.
Also, I would open a separate bug report for the segfault you're getting
when running with MONO_DISABLE_AIO (pasting the backtrace of the
segfault, with debug symbols installed).
On 07/08/13 14:02, Alfred Hall wrote:
Tried running it with sgen or boehm on 2.10? Worth trying both I think.
Also how about 3.3 (master) ?
-Original message-
*From:* Nikita Tsukanov kek...@gmail.com mailto:kek...@gmail.com
*Sent:* Wednesday 7th August 2013 12:54
*To:* mono-devel-list@lists.ximian.com
mailto:mono-devel-list@lists.ximian.com
*Subject:* Re: [Mono-dev] NancyFX self hosting (HttpListener)
locking up on linux
I've rewritten my SCGI server to work with TPL directly instead of
using async/await to make it run on mono 2.10. Then I've tried to
run it with mono 2.10.8.1 and mono 3.2 with System.Net.Sockets
backend and to hammer it with jmeter. 500K requests without any
lockups on Mono 2.10, lockup at 22164th request on mono 3.2.
Server source code is still on GitHub -
https://github.com/kekekeks/scgi-sharp
2013/8/7 Greg Young gregoryyou...@gmail.com
mailto:gregoryyou...@gmail.com
mailto:gregoryyou...@gmail.com mailto:gregoryyou...@gmail.com
I believe attaching a debugger changes things like optimizations
from occurring (not positive but it does in clr)
On Wednesday, August 7, 2013, Nikita Tsukanov wrote:
Huh, it doesn't require debugger to be _attched_, just
debugging subsystem initialized i. e. if I launch this
program as a debugger it doesn't lock up.
publicstaticvoidMain(string[]args)
{
intport=27042;
if(args.Length !=0)
port=int.Parse(args[0]);
while(true)
{
varvm=
Mono.Debugger.Soft.VirtualMachineManager.Listen(newIPEndPoint(IPAddress.Loopback,port));
vm.Resume();
vm.Detach();
}
}
I'll use running with
--debugger-agent=transport=dt_socket,address=127.0.0.1:27042
http://127.0.0.1:27042 as a temporary workaround since
performance doesn't degrade a lot.
2013/8/7 Nikita Tsukanov kek...@gmail.com
mailto:kek...@gmail.com
I suspect that the problem is actually with thread pool
itself. I've created socket layer implementation using
libevent (wrapped with Oars) and send/recv that utilizes
thread pool for cases when it's unable to complete
operation synchronously. It survives longer, but still
locks up after a while. Same behavior with debugger -
I'm unable to reproduce the issue when running under it.
I also unable to grab thread stack traces, it prints
Full thread dump: and nothing else.
2013/8/7 Greg Young gregoryyou...@gmail.com
mailto:gregoryyou...@gmail.com
We will see your test then as it will probably
affect us as well
On Tuesday, August 6, 2013, Nikita Tsukanov wrote:
Greg, I've tried running my server with mono
compiled from master (with pull request #703
merged in), still freezes after a while.
2013/8/7 Greg Young gregoryyou...@gmail.com
mailto:gregoryyou...@gmail.com
Do you have our pull req? We are stable
after (and seriously read history of this list)
On Tuesday, August 6, 2013, Nikita Tsukanov
wrote:
https://github.com/kekekeks/scgi-sharp -
here is my SCGI server with host for
NancyFx. If you run Sandbox.exe with
--echo-server it will not use nancy
infrastructure and will respond
directly. It locks up after several
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Tried running it with sgen or boehm on 2.10? Worth trying both I think. Also
how about 3.3 (master) ?
-Original message-
From: Nikita Tsukanov kek...@gmail.com
Sent: Wednesday 7th August 2013 12:54
To: mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
I've rewritten my SCGI server to work with TPL directly instead of using
async/await to make it run on mono 2.10. Then I've tried to run it with mono
2.10.8.1 and mono 3.2 with System.Net.Sockets backend and to hammer it with
jmeter. 500K requests without any lockups on Mono 2.10, lockup at 22164th
request on mono 3.2.
Server source code is still on GitHub - https://github.com/kekekeks/scgi-sharp
2013/8/7 Greg Young gregoryyou...@gmail.com mailto:gregoryyou...@gmail.com
I believe attaching a debugger changes things like optimizations from occurring
(not positive but it does in clr)
On Wednesday, August 7, 2013, Nikita Tsukanov wrote:
Huh, it doesn't require debugger to be _attched_, just debugging subsystem
initialized i. e. if I launch this program as a debugger it doesn't lock up.
public static void Main (string[] args)
{
int port = 27042;
if (args.Length != 0)
port = int.Parse (args [0]);
while (true)
{
var vm = Mono.Debugger.Soft.VirtualMachineManager.Listen (new IPEndPoint
(IPAddress.Loopback, port));
vm.Resume ();
vm.Detach ();
}
}
I'll use running with
--debugger-agent=transport=dt_socket,address=127.0.0.1:27042
http://127.0.0.1:27042 as a temporary workaround since performance doesn't
degrade a lot.
2013/8/7 Nikita Tsukanov kek...@gmail.com
I suspect that the problem is actually with thread pool itself. I've created
socket layer implementation using libevent (wrapped with Oars) and send/recv
that utilizes thread pool for cases when it's unable to complete operation
synchronously. It survives longer, but still locks up after a while. Same
behavior with debugger - I'm unable to reproduce the issue when running under
it. I also unable to grab thread stack traces, it prints Full thread dump:
and nothing else.
2013/8/7 Greg Young gregoryyou...@gmail.com
We will see your test then as it will probably affect us as well
On Tuesday, August 6, 2013, Nikita Tsukanov wrote:
Greg, I've tried running my server with mono compiled from master (with pull
request #703 merged in), still freezes after a while.
2013/8/7 Greg Young gregoryyou...@gmail.com
Do you have our pull req? We are stable after (and seriously read history of
this list)
On Tuesday, August 6, 2013, Nikita Tsukanov wrote:
https://github.com/kekekeks/scgi-sharp - here is my SCGI server with host for
NancyFx. If you run Sandbox.exe with --echo-server it will not use nancy
infrastructure and will respond directly. It locks up after several thousands
of requests under jmeter.
Simple nginx configuration:
location /
{
include /etc/nginx/scgi_params;
scgi_pass 127.0.0.1:10081 http://127.0.0.1:10081 ;
}
Now I'm looking for alternative socket library to use it as a replacement for
System.Net.Sockets.
2013/8/6 Greg Young gregoryyou...@gmail.com
Actually not that surprised we also found out file stream.flush(true) only
works sometimes and ms never back supported it to actually work :)
On Tuesday, August 6, 2013, Alfred Hall wrote:
Yeah you're having exactly the same issues as I am. I'm surprised others
haven't had this problem before. Not sure who works on this area of the mono
codebase these days. If you got a minimal test case it may be worth us raising
a Xamarin bug in bugzilla.
-Original message-
From: Nikita Tsukanov kek...@gmail.com
Sent: Tuesday 6th August 2013 20:18
To: mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Running with mono from master haven't helped.
And I'm not sure what the hell is going on, but I cann't reproduce the issue
when running under... Monodevelop's debugger. It runs perfectly under it, but
when I try to run the same binary from console (even with --debug option) it
locks up or segfaults. Does anyone know what does it mean?
2013/8/6 Nikita Tsukanov kek...@gmail.com
Great. It locked up with my more complex logic.
Funny fact: NancyFx increases request processing time from 2ms to 70ms with the
same echo response.
Another funny fact: with MONO_DISABLE_AIO I've got segfault.
Now I'll try to use build patched mono. Not sure that it's the same issue,
because in my case it never tries to read and write simultane
--
Le doute n'est pas une condition agréable, mais la certitude est absurde.
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Yeah you're having exactly the same issues as I am. I'm surprised others haven't had this problem before. Not sure who works on this area of the mono codebase these days. If you got a minimal test case it may be worth us raising a Xamarin bug in bugzilla. -Original message- From: Nikita Tsukanov kek...@gmail.com Sent: Tuesday 6th August 2013 20:18 To: mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux Running with mono from master haven't helped. And I'm not sure what the hell is going on, but I cann't reproduce the issue when running under... Monodevelop's debugger. It runs perfectly under it, but when I try to run the same binary from console (even with --debug option) it locks up or segfaults. Does anyone know what does it mean? 2013/8/6 Nikita Tsukanov kek...@gmail.com mailto:kek...@gmail.com Great. It locked up with my more complex logic. Funny fact: NancyFx increases request processing time from 2ms to 70ms with the same echo response. Another funny fact: with MONO_DISABLE_AIO I've got segfault. Now I'll try to use build patched mono. Not sure that it's the same issue, because in my case it never tries to read and write simultaneously on the same socket. 2013/8/6 Greg Young gregoryyou...@gmail.com mailto:gregoryyou...@gmail.com There are many cases the patch we provided does not affect eg no overlap in io between send/receive On Tuesday, August 6, 2013, Nikita Tsukanov wrote: Interesting... I've written a simple server using only Socket.BeginRecieve and Socket.BeginSend. It just reads 100 bytes and then sends hardcoded HTTP response. Now jmeter is working for 5 minutes and it still responds with Lorem ipsum ... perfectly. I'll try to port my SCGI server logic from NetworkStream to Socket and see what will happen. 2013/8/6 Andrés G. Aragoneses kno...@gmail.com On 06/08/13 18:42, Nikita Tsukanov wrote: Ubuntu 13.04, Mono JIT compiler version 3.2.0 (tarball Tue Jul 30 21:08:00 UTC 2013) Mono 3.2.0 does *not* have Yuri's patch. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list http://lists.ximian.com/mailman/listinfo/mono-devel-list -- Le doute n'est pas une condition agréable, mais la certitude est absurde. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Getting similar issues when using FastCGI/XSP proxied via Nginx using tcp and
unix sockets (tried both). After hammering it with jmeter it ends up locking
up. I'm wondering if other mono webapps have this issue. Would be useful if
someone else here could do a similar loadtest using jmeter and let me know if
it happens to them.
-Original message-
From: Alfred Hall ah...@ahall.org
Sent: Sunday 4th August 2013 17:41
To: Nikita Tsukanov kek...@gmail.com; mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi Nikita.
Full thread dump:
threadpool thread tid=0x0x7fc4ad29d700 this=0x0x7fc4ad584c70 thread handle
0x80f state : not waiting owns ()
IO Threadpool worker tid=0x0x7fc4ad25c700 this=0x0x7fc4ad584dd0 thread handle
0x810 state : interrupted state owns ()
IO Threadpool worker tid=0x0x7fc4a7567700 this=0x0x7fc4a741d350 thread handle
0x845 state : interrupted state owns ()
Threadpool worker tid=0x0x7fc4ac39a700 this=0x0x7fc4a6192270 thread handle
0x837 state : interrupted state owns ()
at unknown 0x
at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector
(intptr,intptr,intptr) 0x
at (wrapper alloc) object.AllocVector (intptr,intptr) 0x
at System.Net.HttpConnection.BeginReadRequest () 0x0003a
at System.Net.EndPointListener.OnAccept (object,System.EventArgs) 0x00357
at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted
(System.Net.Sockets.SocketAsyncEventArgs) 0x0002e
at System.Net.Sockets.SocketAsyncEventArgs.AcceptCallback
(System.IAsyncResult) 0x00336
at System.Net.Sockets.SocketAsyncEventArgs.DispatcherCB (System.IAsyncResult)
0x0010f
at (wrapper runtime-invoke) Module.runtime_invoke_void__this___object
(object,intptr,intptr,intptr) 0x
I then tried the same again and only got this in my trace:
Full thread dump:
threadpool thread tid=0x0x7f31b8ac5700 this=0x0x7f31b8da4c70 thread handle
0x80e state : not waiting owns ()
IO Threadpool worker tid=0x0x7f31b8a84700 this=0x0x7f31b8da4dd0 thread handle
0x80f state : interrupted state owns ()
Not sure why I'm not getting any dump here. Any more debugging I can do on
there?
What seems to happen is its coping well initially with the requests and then in
all of a sudden it stops accepting connections and existing connections dont
die off.
Cheers,
Alf
-Original message-
From: Nikita Tsukanov kek...@gmail.com
Sent: Sunday 4th August 2013 16:13
To: mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Alfred, please, try to send SIGQUIT to mono (i. e. kill -SIGQUIT {PID}) when it
stops processing requests. It will force mono to write thread stack traces to
stdout. Grab them and post here, I suspect that the issue is simular to one
experienced by me.
2013/8/4 Nikita Tsukanov kek...@gmail.com mailto:kek...@gmail.com
Alfred, please, try to send SIGQUIT to mono (i. e. kill -SIGQUIT
{PID}) when it stops processing requests. It will force mono to write
thread stack traces to stdout. Grab them and post here, I suspect that
the issue is simular to one experienced by me.
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
I meant to say I tried master too:
root@mulder:~# /opt/ahall-mono/bin/mono -V
Mono JIT compiler version 3.3.0 (master/2354865 Sun Aug 4 00:42:51 BST 2013)
Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors.
www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: sgen
That should have the fix in from https://github.com/ysw, but setting
MONO_DISABLE_AIO should have worked around that anyway, as its meant to bypass
the epoll backend.
My Nancy service is literally just returning a very simple JSON:
public class HelloWorldModule : NancyModule
{
public HelloWorldModule()
{
Get[/] = parameters = {
return Response.AsJson(new HomeResponse { Message = Test });
};
}
}
In JMeter I'm using 100 threads and loop count of 100 and it locks up after
like 15 seconds even over the network. Very odd.
-Original message-
From:quot;Andrés G. Aragonesesquot; kno...@gmail.com
mailto:kno...@gmail.com
Sent: Sunday 4th August 2013 10:03
To: mono-devel-list@lists.ximian.com
mailto:mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on
linux
On 04/08/13 03:07, Alfred Hall wrote:
Hi there.
I'm running the NancyFX web framework in self hosting mode which is
using HttpListener which basically means I deploy an executable and run
it and it will start a webserver on localhost on a TCP port of choice. I
then use nginx to proxy to it.
I first ran my executable on my macbook pro and bombarded it with jmeter
and it coped fine and no issues.
I then deployed on my debian wheezy 64 bit linux box running on top of
KVM using mono 3.2.0 and performed the same jmeter experiment. It locks
up fairly quickly and wont take any new requests. I've tried using both
sgen and boehm but they behave similarly although it seems to lock up
faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it
doesn't make any difference.
Anyone had similar issues?
I tried using self hosted ServiceStack which also uses HttpListener and
had similar issues so I'm finding it unlikely that the bug is in NancyFX
itself.
I tried installing mono 2.10.8 to check if this is a regression, but
getting exactly the same results.
Any idea how I can debug this further or what could be going on.
Hey Alfred.
Could you try mono master (3.3) instead of 3.2? I mention this because
this pull request [1] has been merged recently, which could help in this
situation (and wouldn't make much difference in Mac since the problem in
this platform is worked-around by avoiding kqueue [2]).
[1] https://github.com/mono/mono/pull/703
[2] https://github.com/mono/mono/blob/master/configure.in#L1823
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com mailto:Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi Esben.
Certainly a good suggestion. I've tried increasing it:
nitoback@mulder:~$ ulimit -Sn
20
nitoback@mulder:~$ ulimit -Hn
20
Doesn't really have any effect. Here's what happens when I start hammering it -
regarding connections - The self host is running on port 9591:
root@mulder:~# lsof -n | grep -i 9591 | wc -l
93
root@mulder:~# lsof -n | grep -i 9591 | wc -l
642
root@mulder:~# lsof -n | grep -i 9591 | wc -l
573
root@mulder:~# lsof -n | grep -i 9591 | wc -l
677
root@mulder:~# lsof -n | grep -i 9591 | wc -l
1270
root@mulder:~# lsof -n | grep -i 9591 | wc -l
1270
The connection count goes up to 1270 and at that point it hangs, this is
slightly different each time I run this.
After having stopped the tests and 10 minutes after it:
root@mulder:~# lsof -n | grep -i 9591 | wc -l
1170
Here is an example of the connections that are in the CLOSE_WAIT state:
mono 2788 2840 nitoback 71u IPv4 65081 0t0
TCP 127.0.0.1:9591-127.0.0.1:40372 (CLOSE_WAIT)
mono 2788 2840 nitoback 72u IPv4 65082 0t0
TCP 127.0.0.1:9591-127.0.0.1:40373 (CLOSE_WAIT)
They seem to be stuck in CLOSE_WAIT which I guess means that there is unread
data on the socket. When it gets into this situation its not recoverable, just
hangs and keeps the connections in CLOSE_WAIT state.
-Original message-
From: hy...@hyber.dk hy...@hyber.dk
Sent: Sunday 4th August 2013 14:39
To: Alfred Hall ah...@ahall.org
Cc: mono-devel-list@lists.ximian.com
Subject: SV: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi Alfred
Are you sure its not a ulimit issue? I have had this issue before with the
nofile limit as each tcp session is a file under /proc and when you have
1024 open (or close wait) sessions the OS does not allow you to open more files
and the code seems to hang for a bit until a session is completely gone. Eg.
Timed out and you can create files again...
--
Esben
(on mobile)
- Reply message -
Fra: Alfred Hall ah...@ahall.org
Til: Andrés G. Aragoneses kno...@gmail.com,
mono-devel-list@lists.ximian.com, mono-devel-list@lists.ximian.com
Emne: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Dato: søn., aug. 4, 2013 13:17
I meant to say I tried master too:
root@mulder:~# /opt/ahall-mono/bin/mono -V
Mono JIT compiler version 3.3.0 (master/2354865 Sun Aug 4 00:42:51 BST 2013)
Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors.
www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: sgen
That should have the fix in from https://github.com/ysw, but setting
MONO_DISABLE_AIO should have worked around that anyway, as its meant to bypass
the epoll backend.
My Nancy service is literally just returning a very simple JSON:
public class HelloWorldModule : NancyModule
{
public HelloWorldModule()
{
Get[/] = parameters = {
return Response.AsJson(new HomeResponse { Message = Test });
};
}
}
In JMeter I'm using 100 threads and loop count of 100 and it locks up after
like 15 seconds even over the network. Very odd.
-Original message-
From:quot;Andrés G. Aragonesesquot; kno...@gmail.com
mailto:kno...@gmail.com
Sent: Sunday 4th August 2013 10:03
To: mono-devel-list@lists.ximian.com
mailto:mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on
linux
On 04/08/13 03:07, Alfred Hall wrote:
Hi there.
I'm running the NancyFX web framework in self hosting mode which is
using HttpListener which basically means I deploy an executable and run
it and it will start a webserver on localhost on a TCP port of choice. I
then use nginx to proxy to it.
I first ran my executable on my macbook pro and bombarded it with jmeter
and it coped fine and no issues.
I then deployed on my debian wheezy 64 bit linux box running on top of
KVM using mono 3.2.0 and performed the same jmeter experiment. It locks
up fairly quickly and wont take any new requests. I've tried using both
sgen and boehm but they behave similarly although it seems to lock up
faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it
doesn't make any difference.
Anyone had similar issues?
I tried using self hosted ServiceStack which also uses HttpListener and
had similar issues so I'm finding it unlikely that the bug is in NancyFX
itself.
I tried installing mono 2.10.8 to check if this is a regression, but
getting exactly the same results.
Any idea how I can debug this further or what could be going on.
Hey Alfred.
Could you try mono master (3.3) instead of 3.2? I mention this because
this pull request [1] has been merged recently, which could help in this
situation
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi Nikita.
Full thread dump:
threadpool thread tid=0x0x7fc4ad29d700 this=0x0x7fc4ad584c70 thread handle
0x80f state : not waiting owns ()
IO Threadpool worker tid=0x0x7fc4ad25c700 this=0x0x7fc4ad584dd0 thread handle
0x810 state : interrupted state owns ()
IO Threadpool worker tid=0x0x7fc4a7567700 this=0x0x7fc4a741d350 thread handle
0x845 state : interrupted state owns ()
Threadpool worker tid=0x0x7fc4ac39a700 this=0x0x7fc4a6192270 thread handle
0x837 state : interrupted state owns ()
at unknown 0x
at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector
(intptr,intptr,intptr) 0x
at (wrapper alloc) object.AllocVector (intptr,intptr) 0x
at System.Net.HttpConnection.BeginReadRequest () 0x0003a
at System.Net.EndPointListener.OnAccept (object,System.EventArgs) 0x00357
at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted
(System.Net.Sockets.SocketAsyncEventArgs) 0x0002e
at System.Net.Sockets.SocketAsyncEventArgs.AcceptCallback
(System.IAsyncResult) 0x00336
at System.Net.Sockets.SocketAsyncEventArgs.DispatcherCB (System.IAsyncResult)
0x0010f
at (wrapper runtime-invoke) Module.runtime_invoke_void__this___object
(object,intptr,intptr,intptr) 0x
I then tried the same again and only got this in my trace:
Full thread dump:
threadpool thread tid=0x0x7f31b8ac5700 this=0x0x7f31b8da4c70 thread handle
0x80e state : not waiting owns ()
IO Threadpool worker tid=0x0x7f31b8a84700 this=0x0x7f31b8da4dd0 thread handle
0x80f state : interrupted state owns ()
Not sure why I'm not getting any dump here. Any more debugging I can do on
there?
What seems to happen is its coping well initially with the requests and then in
all of a sudden it stops accepting connections and existing connections dont
die off.
Cheers,
Alf
-Original message-
From: Nikita Tsukanov kek...@gmail.com
Sent: Sunday 4th August 2013 16:13
To: mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Alfred, please, try to send SIGQUIT to mono (i. e. kill -SIGQUIT {PID}) when it
stops processing requests. It will force mono to write thread stack traces to
stdout. Grab them and post here, I suspect that the issue is simular to one
experienced by me.
2013/8/4 Nikita Tsukanov kek...@gmail.com mailto:kek...@gmail.com
Alfred, please, try to send SIGQUIT to mono (i. e. kill -SIGQUIT
{PID}) when it stops processing requests. It will force mono to write
thread stack traces to stdout. Grab them and post here, I suspect that
the issue is simular to one experienced by me.
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Cheers, Alf ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] EV
I think https://github.com/txdv/LibuvSharp is more up to date, Bentkus seems to be working on this actively.. On 25 Jul 2012, at 12:17, Greg Young wrote: action jackson to the rescue as always! If anyone is looking for similar check out https://github.com/kersny/libuv-csharp from google summer of code student last year On Wed, Jul 25, 2012 at 8:01 AM, Greg Young gregoryyou...@gmail.com wrote: Yeah I figured someone might have already built one and could save us a bit of time :) Rather generic code. If we decide to build we will OSS. On Wed, Jul 25, 2012 at 7:45 AM, Jérémie Laval jeremie.la...@gmail.com wrote: You can look at manos code https://github.com/jacksonh/manos -- Jérémie Laval http://neteril.org 2012/7/25 Greg Young gregoryyou...@gmail.com Does anyone already have/know of a socket lib on top of EV for eventing? Cheers, Greg -- Le doute n'est pas une condition agréable, mais la certitude est absurde. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list -- Le doute n'est pas une condition agréable, mais la certitude est absurde. -- Le doute n'est pas une condition agréable, mais la certitude est absurde. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-dev] P/Invoking libsmbclient from OSX
Good afternoon!
I have been trying to P/Invoke into libsmbclient on OSX 10.6 running
64 bit kernel. I compile the samba library myself (version 3.6.0) with
-m32 and managed to get a simple stub working in C/C++:
static void
auth_fn(const char *server, const char *share,
char *workgroup, int wgmaxlen, char *username, int unmaxlen,
char *password, int pwmaxlen)
{
}
smbc_init(auth_fn, 10);
smbc_opendir(smb://myuser:mypass@192.168.x.x/c$);
This works fine, then I created the P/Invokes:
public static void SmbInit()
{
smbc_init(CallBackAuth, 0);
}
public static void CallBackAuth(IntPtr server,
IntPtr share,
IntPtr workgroup,
int wgmaxlen,
IntPtr username,
int unmaxlen,
IntPtr password,
int pwmaxlen)
{
Console.WriteLine(MeeMa);
}
public delegate void SmbCGetAuthDataFn(IntPtr server,
IntPtr share,
IntPtr workgroup,
int wgmaxlen,
IntPtr username,
int unmaxlen,
IntPtr password,
int pwmaxlen);
[DllImport(libsmbclient)] extern internal static int
smbc_init(SmbCGetAuthDataFn callBackAuth, int debug);
[DllImport(libsmbclient)] extern internal static int
smbc_opendir([In, MarshalAs(UnmanagedType.LPStr)]string durl);
Then simple calling into:
SambaWrapper.SmbInit();
SambaWrapper.smbc_opendir(@smb://username:password@somehostname/someshare);
Then I got this beautiful stacktrace here:
http://paste.pocoo.org/show/517187/. I have made sure it's calling
into the libsmbclient library I compiled and not the one that comes
with OSX. I have also tried
it against the one that comes with OSX but that one just blows up with
a different trace.
I was debugging this with Alan on IRC today and we thought it might be
the delete being free'd but we tried adding a new smbc_init function
into smb that
looks like this:
void
myauth_fn(const char *server, const char *share,
char *workgroup, int wgmaxlen, char *username, int unmaxlen,
char *password, int pwmaxlen)
{
}
int
smbc_init2(int debug)
{
smbc_init(myauth_fn, debug);
smbc_opendir(smb://user:pass@192.168.212.133/c$);
}
And then tried to P/Invoke smbc_init2 only and P/Invoke:
[DllImport(libsmbclient, CallingConvention =
CallingConvention.Cdecl)] extern internal static int smbc_init2(int
debug);
When calling into smbc_init2() it blows up with exactly the same
stacktrace. I tried again a C++ program that calls into smbc2_init()
and it worked fine.
Anyone here got any idea how to proceed further with this?
Many thanks,
Alfred
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] P/Invoking libsmbclient from OSX
I actually used that library as a reference. My code also works fine
on linux, it's just OSX that's not working.
Cheers,
Alfred
On Tue, Dec 6, 2011 at 7:55 PM, Slide slide.o@gmail.com wrote:
On Tue, Dec 6, 2011 at 12:53 PM, Alfred Hall ah...@ahall.org wrote:
Good afternoon!
I have been trying to P/Invoke into libsmbclient on OSX 10.6 running
64 bit kernel. I compile the samba library myself (version 3.6.0) with
-m32 and managed to get a simple stub working in C/C++:
static void
auth_fn(const char *server, const char *share,
char *workgroup, int wgmaxlen, char *username, int unmaxlen,
char *password, int pwmaxlen)
{
}
smbc_init(auth_fn, 10);
smbc_opendir(smb://myuser:mypass@192.168.x.x/c$);
This works fine, then I created the P/Invokes:
public static void SmbInit()
{
smbc_init(CallBackAuth, 0);
}
public static void CallBackAuth(IntPtr server,
IntPtr share,
IntPtr workgroup,
int wgmaxlen,
IntPtr username,
int unmaxlen,
IntPtr password,
int pwmaxlen)
{
Console.WriteLine(MeeMa);
}
public delegate void SmbCGetAuthDataFn(IntPtr server,
IntPtr share,
IntPtr workgroup,
int wgmaxlen,
IntPtr username,
int unmaxlen,
IntPtr password,
int pwmaxlen);
[DllImport(libsmbclient)] extern internal static int
smbc_init(SmbCGetAuthDataFn callBackAuth, int debug);
[DllImport(libsmbclient)] extern internal static int
smbc_opendir([In, MarshalAs(UnmanagedType.LPStr)]string durl);
Then simple calling into:
SambaWrapper.SmbInit();
SambaWrapper.smbc_opendir(@smb://username:password@somehostname/someshare);
Then I got this beautiful stacktrace here:
http://paste.pocoo.org/show/517187/. I have made sure it's calling
into the libsmbclient library I compiled and not the one that comes
with OSX. I have also tried
it against the one that comes with OSX but that one just blows up with
a different trace.
I was debugging this with Alan on IRC today and we thought it might be
the delete being free'd but we tried adding a new smbc_init function
into smb that
looks like this:
void
myauth_fn(const char *server, const char *share,
char *workgroup, int wgmaxlen, char *username, int unmaxlen,
char *password, int pwmaxlen)
{
}
int
smbc_init2(int debug)
{
smbc_init(myauth_fn, debug);
smbc_opendir(smb://user:pass@192.168.212.133/c$);
}
And then tried to P/Invoke smbc_init2 only and P/Invoke:
[DllImport(libsmbclient, CallingConvention =
CallingConvention.Cdecl)] extern internal static int smbc_init2(int
debug);
When calling into smbc_init2() it blows up with exactly the same
stacktrace. I tried again a C++ program that calls into smbc2_init()
and it worked fine.
Anyone here got any idea how to proceed further with this?
Many thanks,
Alfred
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Check out this library and see if your p/invoke is setup the same way.
http://code.google.com/p/cifsclient/
Thanks,
slide
--
slide-o-blog
http://slide-o-blog.blogspot.com/
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] P/Invoking libsmbclient from OSX
Hi,
This may explain why smbc_init blows up, but does not explain why
smbc_init2() blows up. In smbc_init2()
the callback is in the unmanaged code. It still blows up.
Cheers,
Alfred
On Tue, Dec 6, 2011 at 8:40 PM, Robert Jordan robe...@gmx.net wrote:
Hi,
On 06.12.2011 20:53, Alfred Hall wrote:
Good afternoon!
I have been trying to P/Invoke into libsmbclient on OSX 10.6 running
64 bit kernel. I compile the samba library myself (version 3.6.0) with
-m32 and managed to get a simple stub working in C/C++:
static void
auth_fn(const char *server, const char *share,
char *workgroup, int wgmaxlen, char *username, int unmaxlen,
char *password, int pwmaxlen)
{
}
smbc_init(auth_fn, 10);
smbc_opendir(smb://myuser:mypass@192.168.x.x/c$);
This works fine, then I created the P/Invokes:
public static void SmbInit()
{
smbc_init(CallBackAuth, 0);
You must pin the delegate passed to smbc_init, either using
a field or a GCHandle.
Robert
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
