Re: [Interest] QWebSocketServer - server randomly stops acceptingconnections

[Jason H]

So full disclosure it's been a while. But it does look like the connection was accepted by the server, the client sent 334 bytes of data and 33 seconds later the client disconnected.

 

To me this would mean that the OS accepted the connection because Qt was listening on it, and Qt failed to emit incommingConnection() or  emit readyRead(). Looking at the source, this class is a very thin wrapper around QTcpServer. In the docs, it says if you have more than the maxPendingConnections() limit (30) it will start emitting errors. " QWebSocketServer will emit the error() signal with the QWebSocketProtocol::CloseCodeAbnormalDisconnection close code when the maximum of connections has been reached."

Given that it takes time for this to show up, this may be it. Also if you have a lot of clients, you might take a while, make sure that you socket->deleteLater()  when the connection is closed. 

 

I can't be sure how you are using it, but my next steps would be to:

- QWebSocketServer::error()

- make sure idle (implement your own timeout mechanism) and closed connections are deleteLater()ed.

- maybe maintain a list of connections and log that?

- Check your threading model. If you're not multithreaded maybe some previous client is blocking you? (Log incommingConnection, with object addresses)

 

 

 

 

 

Sent: Wednesday, April 03, 2019 at 1:32 PM
From: "Jakub Narolewski" <izow...@gmail.com>
To: "Jason H" <jh...@gmx.com>
Cc: "interest@qt-project.org" <interest@qt-project.org>
Subject: RE: [Interest] QWebSocketServer - server randomly stops acceptingconnections

Hello Jason,

 

Thank you very much for your reply.

 

I’m not sure if I can / should directly attach binaries here. I’ve uploaded a wireshark dump of the connection attempt at:

 

https://drive.google.com/open?id=1juNXd7pkI9Xryt_enjh6i2LWm3RL5iQV

 

I call listen() method with QHostAddress::Any parameter value:

 

m_wsServer->listen(QHostAddress::Any, m_port)

 

No proxy is used.

No other custom, network oriented software is running on this box. Only vanilla stuff that came post VPS provisioning.

 

 

From: Jason H
Sent: 03 April 2019 18:23
To: Jakub Narolewski
Cc: interest@qt-project.org
Subject: Re: [Interest] QWebSocketServer - server randomly stops acceptingconnections

 

What happens when you telnet to it?

Are you getting stuff (SYN_ACK) rom the OS, is the OS resetting (RST) the connection?

Is this listening on localhost, virtual or a physical interface?

Linux is generally bad at power managment and networking and maybe that has something to do with it? There is/was an issue with QNetworkAccessManager which after the task got suspended, would not work anymore (I saw this mostly on mobile OSs)

Is it proxied? i.e. behind NGINX?

 

Wireshark dump of a connection attemp while it's dead would really help.

Sent: Wednesday, April 03, 2019 at 11:59 AM
From: "Jakub Narolewski" <izow...@gmail.com>
To: "interest@qt-project.org" <interest@qt-project.org>
Subject: [Interest] QWebSocketServer - server randomly stops accepting connections

I have this one, strange problem with QWebSocketServer that is driving me insane.

My project is a very simple, websocket-based online game server. It uses Qt SQL modules and makes use of some light QEvent'ing.

Up until recently I used uWebSockets to drive my wss communications. To make matters simpler to me I decided to reimplement 'network' part of the application using Qt native module - QWebSocketServer.

It generally works nice - it survives my naive fuzzing tech and cleanly integrates with Qt's event loop.

 

But here is the thing.

 

It randomly, without any error, stops accepting incoming connections. This is VERY hard for me to debug. It happens anywhere between 8 to about 60ish hours after I start my debian unit with systemctl and is fixed immediately when I restart it's service.

I'm connected to every possible error related signal from QWebSocketServer interface, ssl or otherwise - nothing is emited when this happens.

When I ssh to my server I see nominal CPU and memory usage - no leaks, infinite loops or other fun things.

I recently used QObject's timerEvent to spam some qInfos into sys logs. I log:

 

void i9ms::Server::dumpServerStatus() const

{

                qInfo() << "[Server status]"

                                                << "isListening:" << m_wsServer->isListening()

                                                << "hasPendingConnections:" << m_wsServer->hasPendingConnections()

                                                << "errorString:" << m_wsServer->errorString();

}

 

Right now, at the time of writing this message, the server is 'frozen' for about 5 hours and all I see in logs are lines:

 

Apr 03 16:56:12 vps504293 mrserver_release[22148]: [info    ]         [Server status] isListening: true hasPendingConnections: false errorString: ""

 

Problem goes away when I return to uWebSockets-based wss implementation. I let the test server run on it for weeks without problems.

I have tested this on Ubuntu 18.04.2 LTS and Debian 9 both provided 'as-is' from my VPS provider - OVH.

 

I appreciate any feedback or help, have no idea where to go from here. I could possibly try master or 5.12.X branches but this will force me to add 'another configure step' for deployment and is not trival.

_______________________________________________ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest

 

_______________________________________________
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest