Hi Aaron,

On 22 March 2016 at 21:46, Aaron van Meerten <avanmeer...@atlassian.com> wrote:
> Hi Prosody People,
>
> I have prosody set up for bosh, and am forwarding requests to /http-bind in
> nginx to prosody on 5280.  When using this configuration with
> prosody-0.10-nightly181, I have had no issues running the service.
>
> However, now I have been experiencing this strange issue on all builds of
> prosody since prosody-0.10-1nightly181-1, up to and including prosody-trunk
> build 645.

Just to confirm, are you saying 0.10 build 181 works, and 182 does
not? (for the record, that seems unlikely) or is the exact build that
breaks unknown?

> The issue is that upon reaching a certain condition (not clear if it’s load,
> number of concurrent connections, or what), prosody stops responding to BOSH
> requests in a timely fashion.  The error I see in nginx is as follows:
>
> 2016/03/22 21:12:45 [error] 29130#0: *12378 upstream timed out (110:
> Connection timed out) while reading response header from upstream, client:
> 52.90.26.24, server: , request: "POST /http-bind/?room=ip1065189341
> HTTP/1.1", upstream: "http://127.0.0.1:5280/http-bind/?room=ip1065189341";,
> host: "lonely.hipchat.me:443”
>
> There are no equivalent lines in the prosody logs.  It’s as if prosody is
> blocking and fails to respond to the request, closing the connection
> silently within a few seconds of initiation.
>
> The errors only occur when enough clients are connected and communicating,
> so I suspect it’s a resource management problem or some kind of locking
> issue.  When the load reduces again, BOSH connections work properly without
> having to restart the prosody process so it does seem to be related to
> ongoing connections.

What authentication and storage backends is Prosody configured to use?
These are the most likely candidates for this kind of behaviour.

In particular, we have been making some changes to our SQL code in the
0.10 branch recently, so if you are using SQL then we might need to
investigate some of our changes further.

> All this is blocking me from upgrading to the latest prosody, so I am using
> the last one that didn’t have this issue, prosody-0.10-1nightly181-1.
> However, this doesn’t contain the support I need for token authentication,
> and also requires me to check out separate repos of modules which have since
> been integrated into the mainline.  So I’d like to get to the bottom of this
> and start using the latest prosody again.

You mention using other modules (I guess the prosody-modules repo?) -
which of these modules are you using, and any chance that it could be
an issue in one of these rather than Prosody itself?

> Does anyone have any ideas on what I could do to investigate further?  Or
> does anyone have any idea what might have changed in prosody to permit such
> a failure mode?

No particular change springs to mind (apart from SQL changes, as
mentioned above).

Prosody appearing to freeze momentarily will either be caused by
something CPU intensive (e.g. a loop), or by some syscall that blocks.
The former should be detectable with most standard monitoring tools.
The latter could be caught with something like strace or sysdig.

Regards,
Matthew

-- 
You received this message because you are subscribed to the Google Groups 
"prosody-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prosody-dev+unsubscr...@googlegroups.com.
To post to this group, send email to prosody-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/prosody-dev.
For more options, visit https://groups.google.com/d/optout.

Reply via email to