Re: [prosody-dev] possible issue with event loop block
On 19 September 2017 at 13:42, Harsha Bellur wrote: > On Tuesday, September 19, 2017 at 5:00:21 AM UTC-4, Matthew Wild wrote: >> How are you sending this HTTP request? and where? Did you modify Prosody's >> code? > > I am using the request method in net/http.lua to post HTTPS/HTTP requests to > an external http server. I did modify the method to handle POST as well as > PUT - that's about it. net.http already supports POST - you simply specify a 'body' in the options and it will change the default method to POST automatically. To override this and use a different method, simply set 'method' in the options table. >> A segmentation fault is a very different thing to just a slow server. >> Is this reproducible? > > At my end, it is intermittent but reproducible. The way I do it is setup > say 40 thousand WS connections and stop nginx. Basically, I am simulating a > use case where the web socket server has lost connectivity to all clients. Without seeing your code or the strace output I asked for earlier in the thread, I'm afraid we really don't have enough information to diagnose the issue you're having. Regards, Matthew -- You received this message because you are subscribed to the Google Groups "prosody-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to prosody-dev+unsubscr...@googlegroups.com. To post to this group, send email to prosody-dev@googlegroups.com. Visit this group at https://groups.google.com/group/prosody-dev. For more options, visit https://groups.google.com/d/optout.
Re: [prosody-dev] possible issue with event loop block
On Tuesday, September 19, 2017 at 5:00:21 AM UTC-4, Matthew Wild wrote:
>
> Hi,
>
> On 18 September 2017 at 19:33, Harsha Bellur > wrote:
> > Hi Mathew,
> >
> > I guess I should have mentioned this earlier - when websocket is closed,
> I
> > send a http request to a microservice. I believe I am establishing
> multiple
> > connections that is either slowing things down or causing segmentation
> fault
> > based on strace.
>
> How are you sending this HTTP request? and where? Did you modify Prosody's
> code?
>
I am using the request method in net/http.lua to post HTTPS/HTTP requests
to an external http server. I did modify the method to handle POST as well
as PUT - that's about it.
>
> >
> > 15:16:50.130413 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR,
> > si_addr=0x10} ---
> > 15:16:50.238181 +++ killed by SIGSEGV +++
> > -
>
> A segmentation fault is a very different thing to just a slow server.
> Is this reproducible?
>
At my end, it is intermittent but reproducible. The way I do it is setup
say 40 thousand WS connections and stop nginx. Basically, I am simulating a
use case where the web socket server has lost connectivity to all clients.
>
> Regards,
> Matthew
>
--
You received this message because you are subscribed to the Google Groups
"prosody-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prosody-dev+unsubscr...@googlegroups.com.
To post to this group, send email to prosody-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/prosody-dev.
For more options, visit https://groups.google.com/d/optout.
Re: [prosody-dev] possible issue with event loop block
Hi,
On 18 September 2017 at 19:33, Harsha Bellur wrote:
> Hi Mathew,
>
> I guess I should have mentioned this earlier - when websocket is closed, I
> send a http request to a microservice. I believe I am establishing multiple
> connections that is either slowing things down or causing segmentation fault
> based on strace.
How are you sending this HTTP request? and where? Did you modify Prosody's code?
>
> 15:16:50.130413 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR,
> si_addr=0x10} ---
> 15:16:50.238181 +++ killed by SIGSEGV +++
> -
A segmentation fault is a very different thing to just a slow server.
Is this reproducible?
Regards,
Matthew
--
You received this message because you are subscribed to the Google Groups
"prosody-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prosody-dev+unsubscr...@googlegroups.com.
To post to this group, send email to prosody-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/prosody-dev.
For more options, visit https://groups.google.com/d/optout.
Re: [prosody-dev] possible issue with event loop block
To clarify, each websocket disconnect results in a HTTP request being sent
out. Since I am sending the request to the same endpoint, I am wondering if
I can stay connected. Please advice if I can do that.
On Monday, September 18, 2017 at 2:33:52 PM UTC-4, Harsha Bellur wrote:
>
> Hi Mathew,
>
> I guess I should have mentioned this earlier - when websocket is closed, I
> send a http request to a microservice. I believe I am establishing multiple
> connections that is either slowing things down or causing segmentation
> fault based on strace.
>
>
> 15:16:50.130413 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR,
> si_addr=0x10} ---
> 15:16:50.238181 +++ killed by SIGSEGV +++
> -
>
> Again, this happens when a large number of websockets close at the same
> time
>
> Thanks
>
> On Saturday, September 16, 2017 at 4:16:57 AM UTC-4, Matthew Wild wrote:
>>
>> Hey,
>>
>> On 16 September 2017 at 01:27, Harsha Bellur wrote:
>> > Well, besides setting use_libevent to true in prosody config, is there
>> > anything else to do?
>> > I am using Jitsi components for audio/video conferencing - so NGINX
>> proxies
>> > APIs to these components as well.
>> > When connected the clients send ping messages (XEP-0199) periodically -
>> that
>> > is about it
>> > Basically, I am validating how the websocket server handles loss of
>> > connectivity from all clients at once - this is where I am seeing the
>> issue.
>>
>> Can you provide the output of `strace -Ttt -p PROSODY_PID` for the
>> duration of the socket-closing operation?
>>
>> Regards,
>> Matthew
>>
>
--
You received this message because you are subscribed to the Google Groups
"prosody-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prosody-dev+unsubscr...@googlegroups.com.
To post to this group, send email to prosody-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/prosody-dev.
For more options, visit https://groups.google.com/d/optout.
Re: [prosody-dev] possible issue with event loop block
Hi Mathew,
I guess I should have mentioned this earlier - when websocket is closed, I
send a http request to a microservice. I believe I am establishing multiple
connections that is either slowing things down or causing segmentation
fault based on strace.
15:16:50.130413 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR,
si_addr=0x10} ---
15:16:50.238181 +++ killed by SIGSEGV +++
-
Again, this happens when a large number of websockets close at the same time
Thanks
On Saturday, September 16, 2017 at 4:16:57 AM UTC-4, Matthew Wild wrote:
>
> Hey,
>
> On 16 September 2017 at 01:27, Harsha Bellur > wrote:
> > Well, besides setting use_libevent to true in prosody config, is there
> > anything else to do?
> > I am using Jitsi components for audio/video conferencing - so NGINX
> proxies
> > APIs to these components as well.
> > When connected the clients send ping messages (XEP-0199) periodically -
> that
> > is about it
> > Basically, I am validating how the websocket server handles loss of
> > connectivity from all clients at once - this is where I am seeing the
> issue.
>
> Can you provide the output of `strace -Ttt -p PROSODY_PID` for the
> duration of the socket-closing operation?
>
> Regards,
> Matthew
>
--
You received this message because you are subscribed to the Google Groups
"prosody-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prosody-dev+unsubscr...@googlegroups.com.
To post to this group, send email to prosody-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/prosody-dev.
For more options, visit https://groups.google.com/d/optout.
Re: [prosody-dev] possible issue with event loop block
Hey, On 16 September 2017 at 01:27, Harsha Bellur wrote: > Well, besides setting use_libevent to true in prosody config, is there > anything else to do? > I am using Jitsi components for audio/video conferencing - so NGINX proxies > APIs to these components as well. > When connected the clients send ping messages (XEP-0199) periodically - that > is about it > Basically, I am validating how the websocket server handles loss of > connectivity from all clients at once - this is where I am seeing the issue. Can you provide the output of `strace -Ttt -p PROSODY_PID` for the duration of the socket-closing operation? Regards, Matthew -- You received this message because you are subscribed to the Google Groups "prosody-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to prosody-dev+unsubscr...@googlegroups.com. To post to this group, send email to prosody-dev@googlegroups.com. Visit this group at https://groups.google.com/group/prosody-dev. For more options, visit https://groups.google.com/d/optout.
Re: [prosody-dev] possible issue with event loop block
Well, besides setting use_libevent to true in prosody config, is there anything else to do? I am using Jitsi components for audio/video conferencing - so NGINX proxies APIs to these components as well. When connected the clients send ping messages (XEP-0199) periodically - that is about it Basically, I am validating how the websocket server handles loss of connectivity from all clients at once - this is where I am seeing the issue. - Harsha On Monday, August 28, 2017 at 6:46:55 AM UTC-4, Kim Alvefur wrote: > > Hey, > > On Mon, Aug 28, 2017 at 01:52:42AM -0700, Harsha Bellur wrote: > > I am using nginx as the websocket proxy. Prosody version 0.9.8 > > > > I am making 1500 websocket connections using a script and when all > > connections are done - I close them all. At this point, based on the > logs, > > it seems like the event loop is (presumably) blocked till all the > > connections are closed (until listener's ondisconnect function is called > > for all connections). The *main* observation is that there is a delay in > > handling the events - *about 3 - 5 seconds between handling each > disconnect*. > > This renders prosody unusable for a long time until all the disconnect > > events are processed, > > > > What I want to understand is the delay between each disconnect. Is there > a > > way to reduce this delay? > > > > Note that I do not observe this delay (between disconnects) if I run the > > same test with say up to 500 connections. The *total* delay was about 2 > - 3 > > seconds during which time the event loop is ( presumably) blocked till > all > > the connections are closed. > > This all sounds very weird. Can you describe your (Prosody) > configuration in more detail? Especially whether you use libevent or the > default select network backed, and if you have any custom settings for > it. Have you tried connecting directly to Prosody, without nginx? Have > you tried something other than websocket connections? What was sent on > the connections? > > > -- > Zash > -- You received this message because you are subscribed to the Google Groups "prosody-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to prosody-dev+unsubscr...@googlegroups.com. To post to this group, send email to prosody-dev@googlegroups.com. Visit this group at https://groups.google.com/group/prosody-dev. For more options, visit https://groups.google.com/d/optout.
Re: [prosody-dev] possible issue with event loop block
Well, besides setting use_libevent to true in prosody config, is there anything else to do? I am using Jitsi components for audio/video conferencing - so NGINX proxies APIs to these components as well. When connected the clients send ping messages (XEP-0199) periodically - that is about it Basically, I am validating how the websocket server handles loss of connection connectivity from all clients at once - this is where I am seeing the issue. - Harsha On Monday, August 28, 2017 at 6:46:55 AM UTC-4, Kim Alvefur wrote: > > Hey, > > On Mon, Aug 28, 2017 at 01:52:42AM -0700, Harsha Bellur wrote: > > I am using nginx as the websocket proxy. Prosody version 0.9.8 > > > > I am making 1500 websocket connections using a script and when all > > connections are done - I close them all. At this point, based on the > logs, > > it seems like the event loop is (presumably) blocked till all the > > connections are closed (until listener's ondisconnect function is called > > for all connections). The *main* observation is that there is a delay in > > handling the events - *about 3 - 5 seconds between handling each > disconnect*. > > This renders prosody unusable for a long time until all the disconnect > > events are processed, > > > > What I want to understand is the delay between each disconnect. Is there > a > > way to reduce this delay? > > > > Note that I do not observe this delay (between disconnects) if I run the > > same test with say up to 500 connections. The *total* delay was about 2 > - 3 > > seconds during which time the event loop is ( presumably) blocked till > all > > the connections are closed. > > This all sounds very weird. Can you describe your (Prosody) > configuration in more detail? Especially whether you use libevent or the > default select network backed, and if you have any custom settings for > it. Have you tried connecting directly to Prosody, without nginx? Have > you tried something other than websocket connections? What was sent on > the connections? > > > -- > Zash > -- You received this message because you are subscribed to the Google Groups "prosody-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to prosody-dev+unsubscr...@googlegroups.com. To post to this group, send email to prosody-dev@googlegroups.com. Visit this group at https://groups.google.com/group/prosody-dev. For more options, visit https://groups.google.com/d/optout.
Re: [prosody-dev] possible issue with event loop block
Hey, On Mon, Aug 28, 2017 at 01:52:42AM -0700, Harsha Bellur wrote: > I am using nginx as the websocket proxy. Prosody version 0.9.8 > > I am making 1500 websocket connections using a script and when all > connections are done - I close them all. At this point, based on the logs, > it seems like the event loop is (presumably) blocked till all the > connections are closed (until listener's ondisconnect function is called > for all connections). The *main* observation is that there is a delay in > handling the events - *about 3 - 5 seconds between handling each disconnect*. > This renders prosody unusable for a long time until all the disconnect > events are processed, > > What I want to understand is the delay between each disconnect. Is there a > way to reduce this delay? > > Note that I do not observe this delay (between disconnects) if I run the > same test with say up to 500 connections. The *total* delay was about 2 - 3 > seconds during which time the event loop is ( presumably) blocked till all > the connections are closed. This all sounds very weird. Can you describe your (Prosody) configuration in more detail? Especially whether you use libevent or the default select network backed, and if you have any custom settings for it. Have you tried connecting directly to Prosody, without nginx? Have you tried something other than websocket connections? What was sent on the connections? -- Zash -- You received this message because you are subscribed to the Google Groups "prosody-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to prosody-dev+unsubscr...@googlegroups.com. To post to this group, send email to prosody-dev@googlegroups.com. Visit this group at https://groups.google.com/group/prosody-dev. For more options, visit https://groups.google.com/d/optout.
[prosody-dev] possible issue with event loop block
I am using nginx as the websocket proxy. Prosody version 0.9.8 I am making 1500 websocket connections using a script and when all connections are done - I close them all. At this point, based on the logs, it seems like the event loop is (presumably) blocked till all the connections are closed (until listener's ondisconnect function is called for all connections). The *main* observation is that there is a delay in handling the events - *about 3 - 5 seconds between handling each disconnect*. This renders prosody unusable for a long time until all the disconnect events are processed, What I want to understand is the delay between each disconnect. Is there a way to reduce this delay? Note that I do not observe this delay (between disconnects) if I run the same test with say up to 500 connections. The *total* delay was about 2 - 3 seconds during which time the event loop is ( presumably) blocked till all the connections are closed. Please advise -- You received this message because you are subscribed to the Google Groups "prosody-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to prosody-dev+unsubscr...@googlegroups.com. To post to this group, send email to prosody-dev@googlegroups.com. Visit this group at https://groups.google.com/group/prosody-dev. For more options, visit https://groups.google.com/d/optout.
